CN119007132A

CN119007132A - Unmanned aerial vehicle-based road traffic inspection method, unmanned aerial vehicle, equipment and medium

Info

Publication number: CN119007132A
Application number: CN202410983291.4A
Authority: CN
Inventors: 张玉新; 王子煜; 全威; 吕周杭; 寇洪瑞; 徐楚焱; 陈龙
Original assignee: Jilin University
Current assignee: Jilin University
Priority date: 2024-07-22
Filing date: 2024-07-22
Publication date: 2024-11-22

Abstract

The present application discloses a road traffic inspection method based on drones, drones, equipment and storage media, the method comprising: performing vehicle detection and road infrastructure detection on road traffic images, obtaining static vehicle information of at least one vehicle to be tracked in the target road section, and abnormal target road infrastructure in the target road section; performing vehicle tracking according to the static vehicle information of each vehicle to be tracked, obtaining vehicle dynamic information of each vehicle to be tracked; determining a lane semantic map of the target road section, performing traffic violation detection according to the lane semantic map and the static vehicle information and vehicle dynamic information of each vehicle to be tracked, obtaining traffic violation detection results, the traffic violation detection results including the target vehicle that has committed a traffic violation; determining the road traffic inspection results according to the target road infrastructure and the target vehicle. The above-mentioned road traffic inspection method can improve the efficiency of traffic road inspections.

Description

Unmanned aerial vehicle-based road traffic inspection method, unmanned aerial vehicle, equipment and medium

Technical Field

The application relates to the technical field of road traffic inspection, in particular to a road traffic inspection method based on an unmanned aerial vehicle, the unmanned aerial vehicle, computer equipment and a computer readable storage medium.

Background

At present, when road traffic inspection and supervision are performed, a camera on a road is generally utilized to shoot road traffic images, and then human eyes are manually used for identifying the road traffic images. Because the cameras are arranged on each section of road and shoot and generate a large number of road traffic images, the efficiency of road traffic inspection is greatly reduced by adopting a mode of identifying the road traffic images by human eyes.

Disclosure of Invention

The application provides a road traffic inspection method based on an unmanned aerial vehicle, the unmanned aerial vehicle, computer equipment and a computer readable storage medium, which can solve the problem that the efficiency of road traffic inspection is low due to the fact that a human eye is adopted to identify road traffic images in the related technology.

In a first aspect, the application provides a road traffic inspection method based on an unmanned aerial vehicle, which comprises the following steps:

Acquiring a road traffic image of a target road section to be inspected acquired by an unmanned aerial vehicle; detecting vehicles and road infrastructures in the road traffic image to obtain vehicle static information of at least one vehicle to be tracked in the target road section and abnormal target road infrastructures in the target road section; carrying out vehicle tracking according to the vehicle static information corresponding to each vehicle to be tracked, and obtaining vehicle dynamic information corresponding to each vehicle to be tracked; determining a lane semantic map corresponding to the target road section, and detecting traffic illegal behaviors according to the lane semantic map and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked to obtain traffic illegal behavior detection results, wherein the traffic illegal behavior detection results comprise target vehicles with traffic illegal behaviors; and determining a road traffic inspection result corresponding to the target road section according to the target road foundation facility and the target vehicle.

In a second aspect, the application further provides an unmanned aerial vehicle, which comprises a controller, a shooting device and a voice playing device;

the shooting device is used for acquiring road traffic images of a target road section to be patrolled and examined;

the voice playing device is used for outputting warning prompt information which is used for prompting traffic illegal behaviors of the traffic illegal vehicles;

the controller is used for executing the road traffic inspection method based on the unmanned aerial vehicle.

In a third aspect, the present application also provides a computer device comprising a memory and a processor;

The memory is used for storing a computer program;

the processor is used for executing the computer program and realizing the road traffic inspection method based on the unmanned aerial vehicle when the computer program is executed.

In a fourth aspect, the present application also provides a computer readable storage medium, where a computer program is stored, where the computer program when executed by a processor causes the processor to implement the unmanned aerial vehicle-based road traffic inspection method as described above.

The application discloses a road traffic inspection method based on unmanned aerial vehicle, computer equipment and computer readable storage medium, which can obtain the vehicle static information of at least one vehicle to be tracked in a target road section and the abnormal target road infrastructure in the target road section by firstly collecting the road traffic image of the target road section by the unmanned aerial vehicle, then carrying out vehicle tracking according to the vehicle static information corresponding to each vehicle to be tracked, carrying out traffic illegal action detection according to the lane semantic map corresponding to the target road section and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, and finally, determining a road traffic inspection result corresponding to the target road section according to the target road foundation facility and the target vehicle, and rapidly and widely inspecting the road traffic of the target road section by utilizing the road traffic image collected by the unmanned aerial vehicle and the lane semantic map of the target road section, thereby solving the problem of low road traffic inspection efficiency caused by the mode that the related technology adopts the human eyes to identify the road traffic image, and efficiently inspecting the road traffic in a large range, and further improving the traffic road inspection efficiency.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic structural diagram of an unmanned aerial vehicle according to an embodiment of the present application;

Fig. 2 is a schematic block diagram of a unmanned aerial vehicle provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of a computer device according to an embodiment of the present application;

Fig. 4 is a schematic flow chart of a road traffic inspection method based on an unmanned aerial vehicle according to an embodiment of the present application;

FIG. 5 is a schematic flow chart of a sub-step of determining a lane semantic map provided by an embodiment of the present application;

FIG. 6 is a lane semantic map provided by an embodiment of the present application;

FIG. 7 is a schematic flow chart of sub-steps of traffic violation detection provided by an embodiment of the present application;

Fig. 8 is a schematic flow chart of sub-steps for determining road traffic inspection results provided by an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The flow diagrams depicted in the figures are merely illustrative and not necessarily all of the elements and operations/steps are included or performed in the order described. For example, some operations/steps may be further divided, combined, or partially combined, so that the order of actual execution may be changed according to actual situations.

It is to be understood that the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should also be understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

At present, when road traffic inspection and supervision are performed, a camera on a road is generally utilized to shoot road traffic images, and then human eyes are manually used for identifying the road traffic images. Because the cameras are arranged on each section of road and shoot a large number of road traffic images, the efficiency and accuracy of road traffic inspection can be greatly reduced by adopting a mode of identifying the road traffic images by human eyes.

Therefore, the embodiment of the application provides a road traffic inspection method based on an unmanned aerial vehicle, the unmanned aerial vehicle, computer equipment and a computer readable storage medium, which can obtain the vehicle static information of at least one vehicle to be tracked in a target road section and the abnormal target road infrastructure in the target road section by firstly acquiring a road traffic image of the target road section through the unmanned aerial vehicle, then carrying out vehicle tracking according to the vehicle static information corresponding to each vehicle to be tracked, carrying out traffic illegal action detection according to a lane semantic map corresponding to the target road section and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, finally establishing a road traffic inspection result corresponding to the target vehicle according to the target road foundation, realizing the road traffic inspection of the target road section in a rapid and large-scale way by utilizing the road traffic image acquired by the unmanned aerial vehicle and the lane semantic of the target road section, solving the problem that the road traffic inspection efficiency is low due to the adoption of a human eye recognition traffic image mode in the related technology, and effectively improving the road traffic inspection efficiency. The following will describe in detail how the road traffic inspection is performed based on the unmanned aerial vehicle.

Referring to fig. 1 and fig. 2, fig. 1 is a schematic structural diagram of a unmanned aerial vehicle 10 according to an embodiment of the present application, and fig. 2 is a schematic block diagram of the unmanned aerial vehicle 10 according to an embodiment of the present application.

As shown in fig. 1 and 2, the unmanned aerial vehicle 10 may include a body 11, a cradle head 12, a photographing device 13, a power system 14, a control system 15, a voice playing device 16, and the like.

The body 11 may include a fuselage and a foot rest (also referred to as landing gear). The fuselage may include a center frame and one or more arms coupled to the center frame, the one or more arms extending radially from the center frame. The foot rest is connected with the fuselage for supporting the unmanned aerial vehicle 10 when landing.

The cradle head 12 is mounted on the body 11 and is used for mounting the photographing device 13. The pan-tilt 12 may include three motors, that is, the pan-tilt 12 is a three-axis pan-tilt, and under the control of the control system 15 of the unmanned aerial vehicle 10, the shooting angle of the shooting device 13 may be adjusted, where the shooting angle may be understood as an angle of a direction of a lens of the shooting device 13 toward a target to be shot relative to a horizontal direction or a vertical direction.

In some embodiments, the pan-tilt head 12 may further include a controller for controlling the movement of the pan-tilt head 12 by controlling the motor of the pan-tilt head, so as to adjust the shooting angle of the shooting device 13. It should be appreciated that the pan-tilt 12 may be independent of the drone 10 or may be part of the drone 10. It should also be appreciated that the motor may be a direct current motor or an alternating current motor; or the motor may be a brushless motor or a brushed motor.

The photographing device 13 may be, for example, a device for capturing an image, such as a camera or a video camera, and the photographing device 13 may communicate with the control system 15 and perform photographing under the control of the control system 15. In the embodiment of the present application, the camera 13 is mounted on the body 11 of the unmanned aerial vehicle 10 through the cradle head 12. It is understood that the camera 13 may be directly fixed to the body 11 of the unmanned aerial vehicle 10, so that the cradle head 12 may be omitted.

In some embodiments, the photographing device 13 may be controlled to photograph the target link in a depression angle, resulting in a road traffic image of the target link. The depression angle is a direction of an optical axis of a lens of the photographing device 13 perpendicular to a target link to be photographed or a direction of the optical axis of the lens is substantially perpendicular to the target link to be photographed, and the substantially perpendicular is, for example, 88 degrees or 92 degrees, and may be any angle value, which is not limited herein.

In some embodiments, the capturing device 13 may include a monocular camera or a binocular camera for capturing different functions, for example, the monocular camera is used for capturing road traffic images of a target road segment, the binocular camera may obtain a depth image of the target road segment, the depth image includes distance information of a target object on the target road segment, and the target object, such as other common vehicles, pedestrians, road infrastructure, and the like, and the depth image may also be used as one of traffic scene data.

The power system 14 may include one or more electronic speed governors (simply referred to as electric governors), one or more propellers, and one or more motors corresponding to the one or more propellers, wherein the motors are connected between the electronic speed governors and the propellers, the motors and propellers being disposed on a horn of the unmanned aerial vehicle 10. The electronic speed regulator is configured to receive a driving signal generated by the control system 15, and provide a driving current to the motor according to the driving signal, so as to control a rotation speed of the motor and further drive the propeller to rotate, thereby providing power for flight of the unmanned aerial vehicle 10, and the power enables the unmanned aerial vehicle 10 to realize movement in one or more degrees of freedom. In certain embodiments, the drone 10 may rotate about one or more axes of rotation.

For example, the rotation shaft may include a Roll shaft (Roll), a Yaw shaft (Yaw), and a pitch shaft (pitch). It should be appreciated that the motor may be a direct current motor or an alternating current motor. The motor may be a brushless motor or a brushed motor.

The control system 15 may include a controller and a sensing system. The controller is configured to control the flight of the unmanned aerial vehicle 10, for example, the flight of the unmanned aerial vehicle 10 may be controlled according to gesture information measured by the sensing system. It should be appreciated that the controller may control the drone 10 in accordance with preprogrammed instructions or may control the drone 10 in response to one or more control instructions from a control terminal.

The sensing system is used for measuring attitude information of the unmanned aerial vehicle 10, namely position information and state information of the unmanned aerial vehicle 10 in space, such as three-dimensional position, three-dimensional angle, three-dimensional speed, three-dimensional acceleration, three-dimensional angular speed and the like.

The sensing system may include, for example, at least one of a gyroscope, an ultrasonic sensor, an electronic compass, an inertial measurement unit (Inertial Measurement Unit, IMU), a vision sensor, a global navigation satellite system, and a barometer. For example, the global navigation satellite system may be a global positioning system (Global Positioning System, GPS).

The voice playing device 16 may be a device for playing sound, such as a loudspeaker or a speaker, and the voice playing device 16 may be in communication with the control system 15 and perform voice playing under the control of the control system 15. In the embodiment of the present application, the voice playing device 16 is mounted on the body 11 of the unmanned aerial vehicle 10 through the cradle head 12.

In the embodiment of the present application, the position information of the target object such as the vehicle or the road infrastructure to be tracked in the image can be calculated by using the position information and the state information of the unmanned aerial vehicle 10 in the space and combining the image shot by the unmanned aerial vehicle 10. For example, according to the flying height of the unmanned aerial vehicle, the field angle of the shot image and the position information of the unmanned aerial vehicle, the position information of the target object in the image can be calculated through coordinate transformation, triangle relation and the pixel position of the target object in the image.

The controller may include one or more processors and memory. The Processor may be, for example, a Micro-controller Unit (MCU), a central processing Unit (Central Processing Unit, CPU), or a digital signal Processor (DIGITAL SIGNAL Processor, DSP), or the like. The Memory may be a Flash chip, a Read-Only Memory (ROM) disk, an optical disk, a U-disk, a removable hard disk, or the like.

In some embodiments, the unmanned aerial vehicle 10 may further include a radar device mounted on the unmanned aerial vehicle 10, in particular, on the body 11 of the unmanned aerial vehicle 10, for measuring the surrounding environment of the unmanned aerial vehicle 10, such as an obstacle, etc., during the flight of the unmanned aerial vehicle 10, to ensure the safety of the flight.

The radar device is mounted on a foot rest of the unmanned aerial vehicle 10, and is in communication connection with the control system 15, and the radar device transmits acquired observation data to the control system and is processed by the control system 15. The unmanned aerial vehicle 10 may include two or more foot stands, and the radar apparatus is mounted on one of the foot stands. The radar device may be mounted at another position of the unmanned aerial vehicle 10, and is not particularly limited.

The unmanned aerial vehicle 10 may be a rotor unmanned aerial vehicle, such as a four-rotor unmanned aerial vehicle, a six-rotor unmanned aerial vehicle, or an eight-rotor unmanned aerial vehicle, or may be a fixed-wing unmanned aerial vehicle, or may be a combination of a rotor wing type and a fixed-wing unmanned aerial vehicle, which is not limited herein.

In the embodiment of the present application, the controller in the unmanned aerial vehicle 10 is configured to execute the unmanned aerial vehicle-based road traffic inspection method described in any of the embodiments. The controller may include, among other things, a processor, a memory, a communication interface, and an I/O interface. The processor, memory, communication interface, and I/O interface communicate over a bus. The processor may be a central processing unit, but may also be other general purpose processors, digital signal processors, application SPECIFIC INTEGRATED Circuits (ASICs), field-Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

In some embodiments, the controller is configured to implement the following method steps, and in particular, the processor of the controller invokes a computer program in the memory to implement the following method steps:

Acquiring a road traffic image of a target road section to be inspected acquired by an unmanned aerial vehicle; detecting vehicles and road infrastructures in the road traffic image to obtain vehicle static information of at least one vehicle to be tracked in a target road section and abnormal target road infrastructures in the target road section; carrying out vehicle tracking according to the vehicle static information corresponding to each vehicle to be tracked, and obtaining vehicle dynamic information corresponding to each vehicle to be tracked; determining a lane semantic map corresponding to the target road section, and detecting traffic violation according to the lane semantic map, the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, so as to obtain a traffic violation detection result, wherein the traffic violation detection result comprises a target vehicle with traffic violation; and determining a road traffic inspection result corresponding to the target road section according to the target road foundation facility and the target vehicle.

In some embodiments, the vehicle static information includes location information, vehicle profile information, vehicle identification information, vehicle type; the controller is used for realizing vehicle tracking according to the vehicle static information corresponding to each vehicle to be tracked, and when obtaining the vehicle dynamic information corresponding to each vehicle to be tracked:

And inputting the position information, the vehicle contour information, the vehicle identification information and the vehicle type of each vehicle to be tracked into the trained multi-target tracking model to track the vehicle, so as to obtain the vehicle dynamic information corresponding to each vehicle to be tracked.

In some embodiments, when implementing traffic violation detection according to the lane semantic map and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, the controller is configured to implement:

sequentially determining each vehicle to be tracked as a current vehicle; detecting traffic violation behaviors of the current vehicle according to the lane semantic map, the vehicle static information and the vehicle dynamic information corresponding to the current vehicle; if at least one traffic violation behavior of overspeed, illegal lane changing, reverse running and illegal parking exists in the current vehicle, determining the current vehicle as a target vehicle; and generating a traffic violation detection result according to the target vehicle and the traffic violation of the target vehicle.

In some embodiments, the vehicle dynamics information includes relative vehicle speeds for each of the plurality of time frames; the lane semantic map comprises a plurality of lanes, and a lane line type and a driving direction corresponding to each lane; the vehicle static information comprises position information, vehicle contour information, vehicle identification information and vehicle type; when the controller detects traffic violation of the current vehicle according to the lane semantic map, the vehicle static information and the vehicle dynamic information corresponding to the current vehicle, the controller is used for realizing:

If the real vehicle speed of the current vehicle is detected to be greater than the preset speed threshold value according to the relative vehicle speed of the current vehicle, determining that the current vehicle is overspeed; or if the lane line type of the lane where the current vehicle is located is the lane change prohibition type, determining that the current vehicle is illegal to change the lane; or if the running direction of the current vehicle is opposite to the running direction corresponding to the lane where the current vehicle is located, determining that the current vehicle is in reverse running, and determining the running direction of the current vehicle according to the vehicle positions corresponding to the current vehicle in a plurality of time sequences; or if the lane where the current vehicle is parked is the parking prohibition area, determining that the current vehicle is illegally parked.

In some embodiments, the controller is configured to, when implementing that if it is detected from the relative vehicle speed of the current vehicle that the actual vehicle speed of the current vehicle is greater than the preset speed threshold, determine that the current vehicle is overspeed:

acquiring the flight speed and the flight direction of the unmanned aerial vehicle; according to the flying speed and the flying direction of the unmanned aerial vehicle, carrying out speed compensation on the relative vehicle speed of the current vehicle to obtain the real vehicle speed corresponding to the current vehicle; when the actual vehicle speed of the current vehicle is greater than the speed threshold, determining that the current vehicle is overspeed.

In some embodiments, the vehicle dynamics information includes vehicle travel trajectory data; before realizing traffic violation detection according to the lane semantic map and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, the controller is further configured to realize:

And carrying out forward filtering and backward filtering on the vehicle running track data corresponding to each vehicle to be tracked to obtain the filtered vehicle running track data of each vehicle to be tracked.

and detecting traffic illegal behaviors according to the lane semantic map, the vehicle static information of each vehicle to be tracked and the filtered vehicle running track data, and obtaining a traffic illegal behavior detection result.

In some embodiments, after implementing traffic violation detection according to the lane semantic map and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, the controller is further configured to implement:

And the control unmanned aerial vehicle outputs warning prompt information to the target vehicle, wherein the warning prompt information is used for prompting the target vehicle to generate traffic illegal behaviors.

In some embodiments, the controller is configured to, when implementing the road traffic inspection result corresponding to the target road segment according to the target road infrastructure and the target vehicle, implement:

Determining road infrastructure information of a target road infrastructure; determining illegal behavior information corresponding to a target vehicle; and generating a road traffic inspection result according to the road infrastructure information and the illegal action information.

In some implementations, the controller, when implementing the determination of the road infrastructure information for the target road infrastructure, is to implement:

Acquiring an infrastructure image corresponding to an acquisition target road infrastructure of the unmanned aerial vehicle; determining infrastructure location information corresponding to the target road infrastructure; and generating road infrastructure information according to the infrastructure image and the infrastructure position information.

In some embodiments, the controller, when implementing determining infrastructure location information corresponding to the target road infrastructure, is configured to implement:

Acquiring a first image coordinate and a coordinate conversion coefficient of the unmanned aerial vehicle under an image coordinate system, and acquiring a second image coordinate of the target road infrastructure under the image coordinate system, wherein the first image coordinate is a center point of the image coordinate system; acquiring a first satellite positioning coordinate of the unmanned aerial vehicle; performing coordinate conversion according to the first satellite positioning coordinates, the first image coordinates, the coordinate conversion coefficient and the second image coordinates to obtain second satellite positioning coordinates corresponding to the target road infrastructure; infrastructure location information is generated based on the second satellite positioning coordinates.

In some embodiments, when implementing determining the corresponding illegal action information of the target vehicle, the controller is configured to implement:

acquiring illegal action video data corresponding to an unmanned aerial vehicle acquisition target vehicle; the license plate number identification is carried out on the target vehicle, and the license plate number corresponding to the target vehicle is obtained; and generating illegal action information according to the illegal action video data of the target vehicle and the license plate number.

In some embodiments, when implementing determining the lane semantic map corresponding to the target road segment, the controller is configured to implement:

acquiring target aerial survey data of a target road section; based on the trained lane line detection model, lane line detection is carried out on the target navigation measurement data, and a lane line detection result of a target road section is obtained, wherein the lane line detection result comprises a multi-frame lane line semantic graph and a multi-frame central line semantic graph; and carrying out lane positioning according to the multi-frame lane line semantic graph and the multi-frame central line semantic graph to obtain a lane semantic map of the target road section.

Referring to fig. 3, fig. 3 is a schematic structural diagram of a computer device 20 according to an embodiment of the application. The computer device 20 may include a processor 201 and a memory 202, wherein the processor 201 and the memory 202 may be connected by a bus, which may be any suitable bus such as an integrated circuit (Inter-INTEGRATED CIRCUIT, I2C) bus.

By way of example, the computer device 20 may be a server or a terminal. The server may be an independent server, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be electronic equipment such as a smart phone, a tablet computer, a notebook computer, a desktop computer and the like.

For example, when the computer device 20 is a server or a terminal, the computer device 20 may establish a wireless communication connection with the unmanned aerial vehicle, control the unmanned aerial vehicle to take an aerial photograph of the target road segment, and receive aerial survey data returned by the unmanned aerial vehicle, such as road traffic images of the target road segment.

By way of example, memory 202 may include storage media and internal memory. The storage medium may store an operating system and a computer program. The computer program comprises program instructions that, when executed, cause the processor 201 to perform the unmanned vehicle-based road traffic inspection method described in any of the embodiments.

The processor 201 is used to provide computing and control capabilities, supporting the operation of the overall computer device 20.

The Processor 201 may be a central processing unit (Central Processing Unit, CPU), which may also be a general purpose Processor, a digital signal Processor (DIGITAL SIGNAL Processor, DSP), an Application SPECIFIC INTEGRATED Circuit (ASIC), a Field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The general purpose processor may be a microprocessor, or the general purpose processor may be any conventional processor or the like.

Wherein in some embodiments the processor 201 is configured to run a computer program stored in the memory 202 to implement the steps of:

Some embodiments of the present application are described in detail below with reference to the accompanying drawings. The following embodiments and features of the embodiments may be combined with each other without conflict. Referring to fig. 4, fig. 4 is a schematic flow chart of a road traffic inspection method based on an unmanned aerial vehicle according to an embodiment of the application. As shown in fig. 4, the unmanned aerial vehicle-based road traffic inspection method may include steps S301 to S305.

Step S301, acquiring a road traffic image of a target road section to be patrolled and examined acquired by an unmanned aerial vehicle.

In the embodiment of the application, road section planning can be performed on the road needing to be inspected, the target road section needing to be inspected is determined, the unmanned aerial vehicle is controlled to perform aerial photography on the target road section, and the road traffic image of the target road section obtained by the unmanned aerial vehicle is obtained. The whole road can be used as a target road section to be patrolled and examined, and one part of the road can be used as the target road section to be patrolled and examined. It should be noted that, in the embodiment of the present application, the schematic flowchart of the road traffic inspection method based on the unmanned aerial vehicle provided in the embodiment of the present application may be executed by an unmanned aerial vehicle, or the schematic flowchart of the road traffic inspection method based on the unmanned aerial vehicle provided in the embodiment of the present application may be executed by a computer device in communication with the unmanned aerial vehicle.

For example, when controlling the unmanned aerial vehicle to take photo by plane or move the photo by plane to the target section, the shooting parameter of the unmanned aerial vehicle can be set, and the shooting parameter and the position coordinate of the target section are sent to the unmanned aerial vehicle, so that the unmanned aerial vehicle can take photo by plane or move the photo by plane to the target section according to the shooting parameter and the position coordinate of the target section. The shooting parameters may include flight height of the unmanned aerial vehicle, unmanned aerial vehicle position, pan-tilt angle, video format, and the like. For example, the flying height may be 100 meters hover aerial, the unmanned aerial vehicle position may be a preset longitude and latitude, the pan-tilt angle may be-90 ° look down angle to take a ground road, and the video format may be 4k 30fps MP4.

And step S302, detecting vehicles and road infrastructures in the road traffic image to obtain vehicle static information of at least one vehicle to be tracked in the target road section and abnormal target road infrastructures in the target road section.

For example, after the unmanned aerial vehicle acquires the road traffic image of the target road section to be inspected, the vehicle detection and the road infrastructure detection can be performed on the road traffic image to obtain the vehicle static information of at least one vehicle to be tracked in the target road section and the abnormal target road infrastructure in the target road section.

It should be noted that, in the embodiment of the present application, the vehicle detection and the road infrastructure detection may be performed on the road traffic image simultaneously through the trained target detection model. The vehicle detection means detecting which vehicles pass through in the target road section so as to be convenient for tracking the passing vehicles subsequently; the road infrastructure detection refers to detecting whether an abnormality or damage occurs to the road infrastructure within the target road section, wherein the road infrastructure may include, but is not limited to, a street lamp, a guideboard, a traffic light, a lane line, a road surface, a lane marking, etc., for example, whether a crack occurs to the road surface, whether abrasion occurs to the lane line, etc.

By way of example, the vehicle static information may include location information, vehicle profile information, vehicle identification information, vehicle type, and the like. The location information may include, among other things, the location coordinates of the vehicle on the road. The vehicle profile information may include the outer dimensions of the vehicle, such as the length of the vehicle and the width of the vehicle. The vehicle identification information is an ID (IDentity) of the vehicle, and for example, the vehicle may be numbered to obtain the ID of the vehicle. Vehicle types may include cars, buses, trucks, and the like.

By way of example, the object detection model may include, but is not limited to, a Mask R-CNN (fast R-CNN with added masking layer) model, YOLO (You Only Look Once) model, SSD (Single Shot MultiBox Detector) model, and so forth. For example, the Mask R-CNN model may be used to perform vehicle detection and road infrastructure detection on the road traffic image, to obtain location information, vehicle profile information, vehicle identification information, and vehicle type of at least one vehicle to be tracked in the target link, and a target road infrastructure in which an abnormality occurs in the target link. The specific detection process may be referred to in the related art, and is not limited herein.

In the embodiment of the application, the sample image can be used for carrying out vehicle detection training and road infrastructure detection training on the target detection model in advance, so that the trained target detection model not only can detect vehicles, but also can detect whether the road infrastructure is abnormal or damaged. The sample image may include normal road infrastructure and abnormal/damaged road infrastructure, and the specific model training process may refer to the related art, which is not described herein.

In the above embodiment, by adopting the target detection model to detect the vehicle and detect the road infrastructure simultaneously on the road traffic image, it is possible to automatically detect the vehicle static information of at least one vehicle to be tracked in the target road section and the abnormal target road infrastructure in the target road section.

Step S303, vehicle tracking is carried out according to the vehicle static information corresponding to each vehicle to be tracked, and vehicle dynamic information corresponding to each vehicle to be tracked is obtained.

For example, vehicle tracking may be performed according to vehicle static information corresponding to each vehicle to be tracked, so as to obtain vehicle dynamic information corresponding to each vehicle to be tracked.

It should be noted that, because the vehicle static information reflects the vehicle position and state of the vehicle to be tracked at a certain moment, but cannot reflect the dynamic change of the vehicle to be tracked along with time, the vehicle to be tracked needs to be further tracked to acquire the vehicle dynamic information of the vehicle to be tracked.

In some embodiments, vehicle tracking is performed according to vehicle static information corresponding to each vehicle to be tracked, and vehicle dynamic information corresponding to each vehicle to be tracked is obtained, including: and inputting the position information, the vehicle contour information, the vehicle identification information and the vehicle type of each vehicle to be tracked into the trained multi-target tracking model to track the vehicle, so as to obtain the vehicle dynamic information corresponding to each vehicle to be tracked.

The multi-target tracking model can be a model obtained by training a SORT (Simple Online AND REALTIME TRACKING) algorithm, and is used for tracking vehicles. It should be noted that the SORT algorithm is a multi-target tracking algorithm of the base Yu Erman filtering and hungarian algorithm, and can process the multi-target tracking problem in the video in real time. The main idea of the SORT algorithm is to predict the motion state of the target by using a kalman filtering algorithm, calculate the similarity of the detection frame and the tracking frame by using an IOU (Intersection over Union, cross-over ratio) algorithm, and finally find out the best matching result by using a hungarian algorithm.

Illustratively, the main flow of the SORT algorithm includes:

1. splitting video frames: the incoming video stream is split into individual frames, providing input for subsequent frame-by-frame analysis and object detection.

2. Target detection and initialization: in each frame, all possible targets are identified by the target detection algorithm and assigned a unique ID. At the same time, a Kalman filter is initialized for each detected object, which is used to dynamically predict the position and velocity of the object in the next frame based on the current state and motion of the object.

3. Position prediction: the next position of each target is predicted using a kalman filter.

4. Target matching and tracking: the targets of the current frame and the previous frame are matched using the hungarian algorithm. The hungarian algorithm correlates the targets by constructing a cost matrix and finding the lowest cost matching method, and compares the positions of the targets in the current and predicted frames, as well as other attributes (e.g., size and shape), during the matching process to ensure continuity and accuracy.

5. And (5) updating the state: according to the matching result, the state of each object, including its position, speed, etc. in the image is updated, and if a certain object is not matched in several consecutive frames, the object is considered to have left the picture or lost, and the object state is updated accordingly.

The Kalman filtering algorithm is an algorithm applied to dynamic system state estimation, and can perform optimal estimation of the system state by analyzing input and output observation data of a system based on a linear system state equation. The Kalman filtering algorithm not only can improve tracking accuracy, but also can effectively process noise interference. The hungarian algorithm is used to find the best match in a bipartite graph, which means that all nodes in the graph can be divided into two separate sets, with edges connecting the nodes between the two sets. In the SORT algorithm, the hungarian matching algorithm is mainly used to achieve correct matching of the detection box and the box predicted by the kalman filter.

For example, a SORT algorithm may be used to track the vehicle with respect to the position information, the vehicle contour information, the vehicle identification information, and the vehicle type of each vehicle to be tracked, so as to obtain the vehicle dynamic information corresponding to each vehicle to be tracked. The specific process of vehicle tracking may be referred to in the detailed description of the related art, and the specific process is not described herein.

For example, the vehicle dynamic information may include vehicle travel track data, which may include relative vehicle speed and position information corresponding to each of a plurality of time frames, and may also include a movement direction, acceleration, and the like of the vehicle.

According to the embodiment, the vehicle tracking is performed by inputting the position information, the vehicle contour information, the vehicle identification information and the vehicle type of each vehicle to be tracked into the trained multi-target tracking model, so that the vehicle dynamic information corresponding to each vehicle to be tracked can be accurately and efficiently obtained, and the traffic illegal action can be detected based on the vehicle dynamic information.

Step S304, determining a lane semantic map corresponding to the target road section, and detecting traffic illegal behaviors according to the lane semantic map and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked to obtain traffic illegal behavior detection results, wherein the traffic illegal behavior detection results comprise target vehicles with traffic illegal behaviors.

In the embodiment of the application, when detecting traffic illegal behaviors, in order to improve the accuracy of detecting the traffic illegal behaviors, a lane semantic map of a target road section is required to be used. The lane semantic map is a map obtained by carrying out lane line detection and lane positioning on navigation data of a target road section, and can comprise a plurality of lanes, lane line types corresponding to each lane and a driving direction. The lane semantic map can subdivide and define the illegal behaviors of the targets such as vehicles through lane lines, lane line types and the driving directions of the lane lines, and the illegal behaviors of the targets such as vehicles are prevented from being judged by adopting the satellite map, so that the accuracy of detecting the illegal behaviors of the targets such as vehicles can be effectively improved.

Referring to fig. 5, fig. 5 is a schematic flow chart of a substep of determining a lane semantic map according to an embodiment of the present application. As shown in fig. 5, determining the lane semantic map corresponding to the target link in step S304 may include the following steps S3041 to 3043.

Step S3041, obtaining target aerial survey data of a target road section.

For example, the unmanned aerial vehicle may be controlled to perform suspension aerial photography on the target road section, receive initial aerial survey data captured by the unmanned aerial vehicle, and determine the initial aerial survey data as target aerial survey data.

Step S3042, based on the trained lane line detection model, lane line detection is carried out on the target navigation data, and a lane line detection result of the target road section is obtained, wherein the lane line detection result comprises a multi-frame lane line semantic graph and a multi-frame central line semantic graph.

For example, after the target navigation data of the target road section is acquired, lane line detection may be performed on the target navigation data based on the trained lane line detection model, so as to obtain a lane line detection result of the target road section. The lane line detection result comprises a multi-frame lane line semantic graph and a multi-frame central line semantic graph. The lane line semantic graph can comprise information such as lane line type, lane line angle, lane line offset and the like, and the center line semantic graph can comprise information such as center line type, center line angle, center line offset and the like.

The lane line detection model can be a pre-trained model, and is used for detecting lane lines and central lines and outputting a lane line semantic graph and a central line semantic graph.

In the embodiment of the application, the lane line detection model can comprise a U-Net3+ network model, and the U-Net3+ network model comprises a plurality of coding layers and a plurality of decoding layers. It should be noted that, the U-net3+ network model adopts a symmetric U-shaped structure with downsampling and upsampling fused, and uses semantic segmentation to process the input image, and finally forms a semantic segmentation graph. Wherein the decoding layer of the penultimate layer is configured as the output layer; the output layers may include a lane line probability output layer, a lane line type output layer, a lane line angle output layer, a lane line offset output layer, a lane centerline probability output layer, a centerline type output layer, a centerline angle output layer, and a centerline offset output layer.

The lane line probability output layer adopts a specific convolution structure and is used for generating probability mapping that each pixel point belongs to a lane line. By classifying the images pixel by pixel, the model can generate a probability distribution map of the lane lines, thereby effectively distinguishing the lane lines from the background.

The lane line type output layer is used for distinguishing different types of lane lines, such as single broken line, double solid line and the like, through multi-classification.

The lane line angle output layer is used for regarding angles as classification problems, classifying the angles into 18 classes, and classifying the angles into one class every 10 degrees, so that the complexity of the problems can be simplified, and the stability of a model and the robustness to noise and data inaccuracy are improved.

And the lane line offset output layer is responsible for accurately predicting the offset of the lane line relative to the grid center of the lane line. The offset refers to the distance of the reference point from the point on the lane line, which is discretized into grid pixels in the model training, so that the offset represents the offset between the grid center and the point on the lane line. This is critical for accurately locating the lane line position, especially in the case of lane line width variations or curves.

The lane center line probability output layer is the same as the lane line probability output layer and is used for distinguishing the lane center line and the background.

The lane center line type output layer classifies lane center lines, and in the embodiment of the application, the center line classification only sets one center line type.

And the lane center line angle output layer returns the angle of the point of the lane center line by utilizing the thought for solving the classification problem.

And the lane center line offset output layer is the same as the lane line offset output layer and outputs the offset of the lane center line relative to the grid center of the lane center line.

In the embodiment of the application, the lane line detection model can provide more comprehensive road information by arranging the plurality of output layers, and more reference basis is provided for subsequent lane line post-processing. Meanwhile, the multi-dimensional output design enables the lane line detection model to adapt to various road conditions, and the robustness and applicability of the lane line detection model in practical application are enhanced. In addition, the accuracy and the efficiency of the segmentation task can be improved by the U-Net3+ network, so that the accuracy and the efficiency of lane line detection can be effectively improved by adopting the U-Net3+ network for model training.

And step S3043, carrying out lane positioning according to the multi-frame lane line semantic map and the multi-frame central line semantic map to obtain a lane semantic map of the target road section.

In some embodiments, lane positioning is performed according to the multi-frame lane line semantic graph and the multi-frame center line semantic graph to obtain a lane semantic map of the target road section, which may include: carrying out lane line recognition on the multi-frame lane line semantic graph to obtain a lane line recognition graph; performing central line recognition on the multi-frame central line semantic graph to obtain a central line recognition graph; carrying out lane positioning according to the lane line identification diagram and the central line identification diagram to obtain lane positions of all lanes of the target road; and drawing a lane region according to the lane position and the lane line type of each lane to obtain a lane semantic map.

The lane line recognition is similar to the center line recognition, and the application is described in detail by taking the lane line recognition as an example.

In an embodiment, lane line recognition is performed on a multi-frame lane line semantic graph to obtain a lane line recognition graph, which may include: image stacking is carried out on the multi-frame lane line semantic graphs to obtain lane line semantic segmentation graphs; performing binarization processing on the lane line semantic segmentation map to obtain a binarized lane line semantic segmentation map; carrying out connected domain analysis on the binarized lane line semantic segmentation map to obtain a connected domain analysis result map, wherein the connected domain analysis result map comprises a plurality of connected domains; and carrying out lane line identification on the connected domain analysis result graph to obtain a lane line identification graph.

For example, multiple lane line semantic graphs may be stacked to obtain a comprehensive lane line semantic segmentation graph. It should be noted that, by stacking the multiple frames of lane line semantic graphs, it is possible to enhance lane line information by using multiple frames of images and reduce possible uncertainty in a single frame of lane line semantic graph.

For example, binarization processing can be performed on the lane line semantic segmentation map to obtain a binarized lane line semantic segmentation map. The binarization processing is used for carrying out binarization processing on the stacked lane line semantic segmentation graphs so as to strengthen the representation of the lane lines. The specific operation is that the position with the confidence degree larger than 0 in the lane line semantic segmentation map is set as1, and the position with the confidence degree smaller than 0 is set as 0, so that lane lines and backgrounds in the lane line semantic segmentation map can be clearly distinguished.

It should be noted that the connected domain refers to a set of all connected pixels with the same pixel value in the binary image. Connected domain analysis not only helps identify individual objects or regions in an image, but is also critical to understanding and parsing the image content.

For example, the connected domain analysis may be performed on the binarized lane line semantic segmentation map to obtain a connected domain analysis result map, where the connected domain analysis result map includes a plurality of connected domains. The connected domain analysis comprises the following steps:

(1) Initializing a marking matrix: and creating a matrix with the same size as the lane line semantic segmentation map, setting all elements to 0, and recording the connected domain number of each pixel. At the beginning, the connected domain number i is set to 1.

(2) Pixel-by-pixel scanning and labeling:

a. If the periphery of a pixel is not marked (i.e., all the periphery are 0), it is marked as a new connected domain, numbered i, and incremented by 1.

B. if all marked points within the neighborhood have the same label, then the point inherits this label.

C. If different labels exist in the neighborhood, the smallest label is selected, and the equivalence relation of all the different labels is recorded.

(3) And (3) secondary scanning: and rescanning each pixel point, updating the labels of all equivalent connected domains, and ensuring that all equivalent connected domains use the smallest connected domain number, so that the uniformity and the accuracy of the connected domain labels can be improved.

(4) Generating a connected domain: based on the finally obtained connected domain labels, each independent connected domain can be extracted from the lane line semantic segmentation map.

According to the embodiment, through the connected domain analysis of the binarized lane line semantic segmentation map, the independent objects or areas in the lane line semantic segmentation map can be identified, and the accurate identification of the independent lane lines in the complex road image is facilitated.

In some embodiments, lane line recognition is performed on the connected domain analysis result graph to obtain a lane line recognition graph, which may include: performing adjacent point matching on each connected domain in the filtered connected domain analysis result graph to obtain a connected domain analysis result graph after adjacent point matching; connecting the connected domains to the connected domain analysis result graph after the adjacent points are matched, and obtaining the connected domain analysis result graph after the connected domains are connected; and performing polynomial fitting on the connected domains in the connected domain analysis result graph to obtain a lane line identification graph.

For example, when performing proximity matching on each connected domain in the filtered connected domain analysis result graph, the following steps may be performed for each connected domain: first, the connected domain is scanned from left to right, a point where the first pixel value is not zero is found, and the angle thereof is calculated. Then, the point is marked as processed and its pixel value is set to zero. Next, in a surrounding area (extended 1 pixel) centered on the point, adjacent non-zero pixel points are searched. For each found adjacent point, calculating the angle difference between the adjacent point and the current point, and selecting the point with the smallest angle difference as the matching point. At the same time, the matching point is marked as processed and its pixel value is set to zero. This process is repeated until the full-portion connected domain is searched.

The method for connecting the connected domain to the connected domain analysis result graph after the adjacent point matching comprises the following steps: determining the distance between two adjacent connected domains in the connected domain analysis result diagram; if two adjacent connected domains with the distance smaller than the preset distance value exist, connecting the two adjacent connected domains.

For example, a distance between two adjacent connected domains may be calculated, and for two adjacent connected domains having a distance smaller than a preset distance value, the two adjacent connected domains are connected. The preset distance value may be set according to practical situations, and specific numerical values are not limited herein.

According to the embodiment, the two adjacent communicating domains with the distance smaller than the preset distance value are connected, the broken line position of the lane line can be supplemented, the integrity of the lane line can be enhanced, the lane line in a complex road scene can be processed better, and therefore the accuracy and the stability of lane line detection are improved.

For example, polynomial fitting can be performed on the connected domain in the connected domain analysis result graph after the connected domain is connected, so as to obtain a lane line identification graph. For example, the connected domain in the connected domain analysis result graph may be fitted with a polynomial for three times, where the specific fitting process may be referred to the detailed description of the related art, and will not be described herein.

Illustratively, the lane line identification map includes lane line offsets corresponding to lane lines and the centerline identification map includes centerline offsets corresponding to centerlines. The lane line offset refers to the shortest distance between the lane line and the center of the image in the image, and can be understood as the vertical distance between the center of the image and the lane line. In the embodiment of the application, the position relation between the lane line and the central line in the image can be determined according to the lane line offset, so that the upper lane line and the lower lane line of the central line can be further determined. In some embodiments, performing lane positioning according to the lane line identification map and the center line identification map to obtain lane positions of each lane of the target road section may include: extracting at least one lane line in the lane line identification map, and extracting at least one center line in the center line identification map; and carrying out lane line matching on each central line according to the lane line offset corresponding to each lane line and the central line offset corresponding to each central line, and determining the lane position of each lane according to the lane line matched with each central line.

For example, since the lane lines are located at both sides of the center line, each center line may be sequentially processed, and the upper boundary of the found lane may be searched upward along a point on the center line according to the center line offset of the center line, and the lower boundary of the found lane may be searched downward, so that the lane position of the lane may be rapidly and accurately located.

For example, after determining the lane position of each lane of the target road, the lane region may be drawn according to the lane position of each lane and the lane line type, so as to obtain the lane semantic map. As shown in fig. 6, fig. 6 is a lane semantic map provided by the embodiment of the present application, where each lane in the lane semantic map may be divided and numbered. The lane semantic map may include information such as lane lines, lane line types, driving directions of the lane lines, center line types, and the like.

In the above embodiment, lane line detection is performed on the target navigation data of the target road section, and lane positioning is performed according to the detected multi-frame lane line semantic map and multi-frame central line semantic map to obtain the lane semantic map of the target road section, and since the lane semantic map can subdivide and define the illegal behaviors of the targets such as vehicles through the lane lines, the lane line types and the driving directions of the lane lines, and in the follow-up road traffic inspection of the target road section, the lane semantic map can be used for judging the illegal behaviors of the targets such as the vehicle and the like, and the satellite map is prevented from being adopted for judging the illegal behaviors of the targets such as the vehicle and the like, so that the accuracy of detecting the illegal behaviors of the targets such as the vehicle and the like can be effectively improved.

In some embodiments, after determining the lane semantic map corresponding to the target road section, traffic violation detection may be performed according to the lane semantic map and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, so as to obtain a traffic violation detection result. The following will describe in detail how traffic violations are detected.

Referring to fig. 7, fig. 7 is a schematic flow chart of sub-steps of traffic offence detection provided by an embodiment of the present application. As shown in fig. 7, the following steps S3044 to 3047 may be included.

Step S3044, determining each vehicle to be tracked as a current vehicle in turn.

For example, since there may be a plurality of vehicles to be tracked in the target road section, it is required to detect whether each vehicle to be tracked has traffic offence, and for convenience of explanation, each vehicle to be tracked may be determined as a current vehicle in turn, and traffic offence detection is performed on each current vehicle.

And step S3045, detecting traffic illegal behaviors of the current vehicle according to the lane semantic map, the vehicle static information and the vehicle dynamic information corresponding to the current vehicle.

In the embodiment of the application, whether the current vehicle has traffic illegal behaviors such as overspeed, illegal lane change, reverse running, illegal parking and the like can be detected at the same time.

In some embodiments, the traffic violation detection for the current vehicle according to the lane semantic map and the vehicle static information and the vehicle dynamic information corresponding to the current vehicle may include: if the real vehicle speed of the current vehicle is detected to be greater than the preset speed threshold value according to the relative vehicle speed of the current vehicle, determining that the current vehicle is overspeed; or if the lane line type of the lane where the current vehicle is located is the lane change prohibition type, determining that the current vehicle is illegal to change the lane; or if the running direction of the current vehicle is opposite to the running direction corresponding to the lane where the current vehicle is located, determining that the current vehicle is in reverse running, and determining the running direction of the current vehicle according to the vehicle positions corresponding to the current vehicle in a plurality of time sequences; or if the lane where the current vehicle is parked is the parking prohibition area, determining that the current vehicle is illegally parked.

In the embodiment of the application, besides detecting whether the current vehicle has overspeed, illegal lane change, reverse running and illegal parking, the traffic illegal behaviors such as whether the current vehicle presses the line according to the lane type and whether the current vehicle runs in the limited lane according to the vehicle type and the position information can be detected. The types of traffic offences are not limited herein.

In some embodiments, if the actual vehicle speed of the current vehicle is detected to be greater than the preset speed threshold according to the relative vehicle speed of the current vehicle, determining that the current vehicle is overspeed may include: acquiring the flight speed and the flight direction of the unmanned aerial vehicle; according to the flying speed and the flying direction of the unmanned aerial vehicle, carrying out speed compensation on the relative vehicle speed of the current vehicle to obtain the real vehicle speed corresponding to the current vehicle; when the actual vehicle speed of the current vehicle is greater than the speed threshold, determining that the current vehicle is overspeed.

The relative vehicle speed refers to a speed of the vehicle relative to the unmanned aerial vehicle. It will be appreciated that since the travel speed of the vehicle is detected from an aerial image or video of the drone, the drone has a certain flight speed, and the detected travel speed is the speed of the vehicle relative to the drone and not the true vehicle speed of the vehicle. At this time, it is necessary to compensate for the relative vehicle speed of the detected vehicle.

For example, when the speed compensation is performed on the relative vehicle speed of the current vehicle according to the flight speed and the flight direction of the unmanned aerial vehicle, whether the flight direction of the unmanned aerial vehicle is the same as the travel direction of the current vehicle can be judged first, and when the flight direction of the unmanned aerial vehicle is the same as the travel direction of the current vehicle, the flight speed of the unmanned aerial vehicle and the relative vehicle speed of the current vehicle can be added to obtain the real vehicle speed corresponding to the current vehicle; when the flight direction of the unmanned aerial vehicle is opposite to the running direction of the current vehicle, the flight speed of the unmanned aerial vehicle and the relative vehicle speed of the current vehicle can be subtracted to obtain the real vehicle speed corresponding to the current vehicle.

For example, the current vehicle overspeed is determined when the actual vehicle speed of the current vehicle is greater than the speed threshold. The speed threshold may be set according to practical situations, and specific values are not limited herein. For example, the speed threshold may be an upper speed limit of the target road segment.

According to the embodiment, the relative vehicle speed of the current vehicle is compensated according to the flying speed and the flying direction of the unmanned aerial vehicle, so that the real vehicle speed corresponding to the current vehicle can be obtained, and the accuracy of judging whether the current vehicle exceeds the overspeed can be improved.

And step S3046, if at least one traffic violation behavior of overspeed, illegal lane changing, reverse running and illegal parking exists in the current vehicle, determining the current vehicle as a target vehicle.

For example, after detecting traffic illegal behaviors of the current vehicle according to the lane semantic map and the vehicle static information and the vehicle dynamic information corresponding to the current vehicle, if at least one traffic illegal behavior of overspeed, illegal lane changing, reverse driving and illegal parking exists in the current vehicle, determining the current vehicle as a target vehicle. For example, for the a vehicle, if there is overspeed of the a vehicle, the a vehicle is determined as the target vehicle. For another example, with respect to the B vehicle, if there is an illegal lane change, the B vehicle is also determined as the target vehicle.

Step S3047, according to the target vehicle and the traffic illegal behaviors of the target vehicle, generating a traffic illegal behavior detection result.

For example, after determining that a traffic offence target vehicle exists in the target road section, a traffic offence detection result may be generated according to the target vehicle and the traffic offence of the target vehicle. For example, the vehicle identification information of the target vehicle and the type of traffic offence may be determined as the traffic offence detection result.

According to the embodiment, the traffic illegal action detection is carried out on the current vehicle according to the lane semantic map and the vehicle static information and the vehicle dynamic information corresponding to the current vehicle, so that the automatic traffic illegal action detection can be realized, the problem that the road traffic inspection efficiency is low due to the fact that a human eye is adopted to identify a road traffic image in the related technology is solved, the road traffic inspection efficiency can be effectively improved, and the lane semantic map can be used for subdividing and defining the illegal actions of targets such as vehicles through lane lines, lane line types and the driving directions of the lane lines, so that the traffic illegal action detection is carried out through the lane semantic map, the illegal actions of targets such as vehicles can be prevented from being judged through the satellite map, and the accuracy of detecting the illegal actions of the targets such as the vehicles can be effectively improved.

In some embodiments, the method further includes, before the detecting traffic violation according to the lane semantic map and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, obtaining a traffic violation detection result: and carrying out forward filtering and backward filtering on the vehicle running track data corresponding to each vehicle to be tracked to obtain the filtered vehicle running track data of each vehicle to be tracked.

It should be noted that, the position stability of the track point in the vehicle running track data has a great influence on the data precision, and the track point is generally the center position of the detection frame.

For example, an RTS (Rauch-tune-Striebel smoother) algorithm may be used to perform forward filtering and backward filtering on the vehicle running track data corresponding to each vehicle to be tracked, so as to obtain filtered vehicle running track data of each vehicle to be tracked. For specific processes of forward filtering and backward filtering, reference may be made to the related art, which is not limited herein. Of course, other types of smoothing algorithms may be used to smooth the vehicle track data, and the type of the smoothing algorithm is not limited in the embodiment of the present application.

It should be noted that, the RTS algorithm is a fixed interval smoother based on a maximum likelihood estimation criterion, and the RTS smoothing algorithm is based on kalman filtering and includes two stages of forward filtering and backward filtering. The forward filtering adopts traditional Kalman filtering, and the backward filtering recycles data on the basis of the traditional Kalman filtering to improve the accuracy of state estimation.

According to the embodiment, the forward filtering and the backward filtering are carried out on the vehicle running track data corresponding to each vehicle to be tracked by adopting the RTS algorithm, so that the accuracy of the speed and the acceleration in the vehicle running track data can be improved, and the accuracy and the reliability of the vehicle running track data can be remarkably improved.

In some embodiments, detecting traffic violation according to a lane semantic map and vehicle static information and vehicle dynamic information of each vehicle to be tracked, to obtain a traffic violation detection result, including: and detecting traffic illegal behaviors according to the lane semantic map, the vehicle static information of each vehicle to be tracked and the filtered vehicle running track data, and obtaining a traffic illegal behavior detection result.

Exemplary, traffic violation detection is performed according to the lane semantic map and the vehicle static information and the filtered vehicle driving track data of each vehicle to be tracked, and the specific process of traffic violation detection for the current vehicle according to the lane semantic map and the vehicle static information and the vehicle dynamic information corresponding to the current vehicle in the above embodiment is similar and is not described herein.

According to the embodiment, the traffic violation detection is carried out according to the lane semantic map, the vehicle static information of each vehicle to be tracked and the filtered vehicle running track data, and the accuracy and the reliability of the filtered vehicle running track data are greatly improved, so that the accuracy of the traffic violation detection can be effectively improved.

In some embodiments, the method further includes, after performing traffic violation detection according to the lane semantic map and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, obtaining a traffic violation detection result: and the control unmanned aerial vehicle outputs warning prompt information to the target vehicle, wherein the warning prompt information is used for prompting the target vehicle to generate traffic illegal behaviors.

The warning prompt information can be voice information. For example, traffic offenses of the target vehicle may be played through a voice playing device in the unmanned aerial vehicle.

According to the embodiment, the unmanned aerial vehicle is controlled to output the warning prompt information to the target vehicle, so that the vehicle owner of the target vehicle can timely learn the violated traffic behavior, and is supervised to timely correct, the safety can be improved, and the unmanned aerial vehicle is more humanized.

And step S305, determining a road traffic inspection result corresponding to the target road section according to the target road foundation facility and the target vehicle.

For example, after traffic offence is detected according to the lane semantic map and the vehicle static information and the vehicle dynamic information of each vehicle to be tracked, and a target vehicle with traffic offence is determined, a road traffic inspection result corresponding to a target road section can be determined according to the target road foundation and the target vehicle. The following will explain in detail how to determine the road traffic inspection result.

Referring to fig. 8, fig. 8 is a schematic flow chart of a sub-step of determining a road traffic inspection result according to an embodiment of the present application. As shown in fig. 8, step S305 may include the following steps S3051 to 3053.

Step S3051, determining road infrastructure information of the target road infrastructure.

For example, the road infrastructure information may include an image and a location corresponding to the target road infrastructure.

In some embodiments, determining the road infrastructure information for the target road infrastructure may include: acquiring an infrastructure image corresponding to an acquisition target road infrastructure of the unmanned aerial vehicle; determining infrastructure location information corresponding to the target road infrastructure; and generating road infrastructure information according to the infrastructure image and the infrastructure position information.

Wherein, determining the infrastructure location information corresponding to the target road infrastructure may include: acquiring a first image coordinate and a coordinate conversion coefficient of the unmanned aerial vehicle under an image coordinate system, and acquiring a second image coordinate of the target road infrastructure under the image coordinate system, wherein the first image coordinate is a center point of the image coordinate system; acquiring a first satellite positioning coordinate of the unmanned aerial vehicle; performing coordinate conversion according to the first satellite positioning coordinates, the first image coordinates, the coordinate conversion coefficient and the second image coordinates to obtain second satellite positioning coordinates corresponding to the target road infrastructure; infrastructure location information is generated based on the second satellite positioning coordinates.

It should be noted that, since the coordinates of the target road infrastructure are determined to be the image coordinates in the image coordinate system based on the road traffic image captured by the unmanned aerial vehicle, and are not the actual satellite positioning coordinates in the world coordinate system, it is necessary to convert the image coordinates of the target road infrastructure to the satellite positioning coordinates.

The first satellite positioning coordinates of the unmanned aerial vehicle can be determined through a GPS system, and coordinate conversion is performed according to the first satellite positioning coordinates, the first image coordinates, the coordinate conversion coefficients and the second image coordinates based on a GPS conversion tool, so that second satellite positioning coordinates corresponding to the target road infrastructure are obtained. The GPS conversion tool may calculate the second satellite positioning coordinates corresponding to the target road infrastructure according to the coordinate conversion coefficients. The coordinate conversion coefficient refers to the real world size represented by each pixel point of the image photographed at the height of the unmanned aerial vehicle, for example, the coordinate conversion coefficient may be 1 pixel=3.75 cm in one image 3840×2160 photographed at the height of 100m hovering of the unmanned aerial vehicle. Of course, the coordinate conversion coefficient may be another numerical value, and the present application is not limited thereto.

For example, after the second satellite positioning coordinates corresponding to the target road infrastructure are calculated, the second satellite positioning coordinates may be used as the infrastructure location information. The infrastructure location information may include, among other things, the latitude and longitude of the target road infrastructure.

In some embodiments, when the target road infrastructure is configured with the positioning device, the position coordinates of the positioning device may be directly determined as the infrastructure position information corresponding to the target road infrastructure.

For example, the road infrastructure information may be generated from the infrastructure image and the infrastructure location information.

According to the embodiment, the road infrastructure information is generated according to the infrastructure image and the infrastructure position information of the target road infrastructure, and the target road infrastructure can be quickly and accurately positioned according to the infrastructure image and the infrastructure position information, so that maintenance staff can maintain the target road infrastructure conveniently.

Step S3052, determining the information of the illegal actions corresponding to the target vehicle.

In some embodiments, determining the corresponding offence information of the target vehicle may include: acquiring illegal action video data corresponding to an unmanned aerial vehicle acquisition target vehicle; the license plate number identification is carried out on the target vehicle, and the license plate number corresponding to the target vehicle is obtained; and generating illegal action information according to the illegal action video data of the target vehicle and the license plate number.

For example, after the traffic offence occurs to the target vehicle, the unmanned aerial vehicle may be controlled to collect the offence video data corresponding to the target vehicle. And then, adopting a pre-trained license plate number recognition model to carry out license plate number recognition on the target vehicle. The license plate number recognition model can be adopted to recognize license plate numbers of the illegal activity video data, or the video or the image can be collected again to recognize license plate numbers. For example, when license plate numbers cannot be identified according to the illegal activity video data, videos or images of different angles can be acquired to carry out license plate number identification.

The license plate number recognition model may include, but is not limited to, a Mask R-CNN model, a YOLO model, an SSD model, and the like, and the type of the license plate number recognition model is not limited in the embodiment of the present application.

In the embodiment, by acquiring the illegal activity video data corresponding to the target vehicle and identifying the license plate number of the target vehicle, the illegal activity information can be generated according to the illegal activity video data and the license plate number of the target vehicle, so that follow-up tracking and processing of the illegal activity of the target vehicle are facilitated.

And step S3053, generating a road traffic inspection result according to the road infrastructure information and the illegal action information.

The road traffic inspection result can be generated according to the road infrastructure information and the illegal action information, and then can be stored in a local database or a local disk, and of course, the road traffic inspection result can be uploaded to a background road traffic inspection platform so that related personnel can further process the road traffic inspection result.

According to the embodiment, the road infrastructure information of the target road infrastructure is determined, the illegal action information corresponding to the target vehicle is determined, and the road traffic inspection result is generated according to the road infrastructure information and the illegal action information, so that related personnel can conveniently further process the road traffic inspection result.

The embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, the computer program comprises program instructions, and a processor executes the program instructions to realize any unmanned aerial vehicle-based road traffic inspection method.

For example, the computer program is loaded by a processor, the following steps may be performed:

The specific implementation of each operation above may be referred to the previous embodiments, and will not be described herein.

The computer readable storage medium may be an internal storage unit of the unmanned aerial vehicle or the computer device of the foregoing embodiment, for example, a hard disk or a memory of the unmanned aerial vehicle or the computer device. The computer readable storage medium may also be an external storage device of the drone or computer device, such as a plug-in hard disk provided on the drone or computer device, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), or the like.

The present application is not limited to the above embodiments, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the present application, and these modifications and substitutions are intended to be included in the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.

Claims

1. A road traffic inspection method based on an unmanned aerial vehicle, characterized in that the method comprises:

Acquire the road traffic images of the target road section to be inspected collected by the drone;

Performing vehicle detection and road infrastructure detection on the road traffic image to obtain static vehicle information of at least one vehicle to be tracked in the target road section and abnormal target road infrastructure in the target road section;

Perform vehicle tracking according to the vehicle static information corresponding to each of the vehicles to be tracked, and obtain the vehicle dynamic information corresponding to each of the vehicles to be tracked;

Determine a lane semantic map corresponding to the target road section, perform traffic violation detection according to the lane semantic map and the static vehicle information and the dynamic vehicle information of each of the vehicles to be tracked, and obtain a traffic violation detection result, wherein the traffic violation detection result includes the target vehicle that has committed the traffic violation;

A road traffic inspection result corresponding to the target road section is determined according to the target road infrastructure and the target vehicle.

2. The road traffic inspection method based on drone according to claim 1 is characterized in that the static vehicle information includes location information, vehicle profile information, vehicle identification information, and vehicle type; the vehicle tracking is performed according to the static vehicle information corresponding to each of the vehicles to be tracked to obtain the dynamic vehicle information corresponding to each of the vehicles to be tracked, including:

The position information, vehicle profile information, vehicle identification information and vehicle type of each vehicle to be tracked are input into the trained multi-target tracking model to perform vehicle tracking, so as to obtain the vehicle dynamic information corresponding to each vehicle to be tracked.

3. The method for road traffic inspection based on drone according to claim 1 is characterized in that the traffic violation detection is performed according to the lane semantic map and the static information and dynamic information of each vehicle to be tracked to obtain the traffic violation detection result, including:

sequentially determining each of the to-be-tracked vehicles as a current vehicle;

Performing traffic violation detection on the current vehicle according to the lane semantic map and the vehicle static information and vehicle dynamic information corresponding to the current vehicle;

If the current vehicle has at least one traffic violation of speeding, illegal lane change, wrong-way driving, and illegal parking, the current vehicle is determined as the target vehicle;

The traffic violation detection result is generated according to the target vehicle and the traffic violation of the target vehicle.

4. The method for road traffic inspection based on drone according to claim 3 is characterized in that the vehicle dynamic information includes the relative vehicle speed corresponding to each of the multiple time frames; the lane semantic map includes multiple lanes and the lane line type and driving direction corresponding to each lane; the vehicle static information includes location information, vehicle profile information, vehicle identification information, and vehicle type;

The detecting of traffic violations on the current vehicle according to the lane semantic map and the vehicle static information and vehicle dynamic information corresponding to the current vehicle includes:

If it is detected according to the relative vehicle speed of the current vehicle that the actual vehicle speed of the current vehicle is greater than a preset speed threshold, then determining that the current vehicle is speeding; or

If the lane line type of the lane where the current vehicle is located when changing lanes is a lane change prohibited type, then determining that the current vehicle has changed lanes illegally; or

If the driving direction of the current vehicle is opposite to the driving direction corresponding to the lane where the current vehicle is currently located, it is determined that the current vehicle is traveling in the wrong direction, and the driving direction of the current vehicle is determined according to the vehicle positions corresponding to the current vehicle in multiple time series; or

If the lane where the current vehicle is parked is a prohibited parking area, it is determined that the current vehicle is parked illegally.

5. The method for road traffic inspection based on a drone according to claim 4, characterized in that if the actual vehicle speed of the current vehicle is detected to be greater than a preset speed threshold according to the relative vehicle speed of the current vehicle, then determining that the current vehicle is speeding comprises:

Obtaining the flight speed and flight direction of the UAV;

Performing speed compensation on the relative vehicle speed of the current vehicle according to the flight speed and flight direction of the UAV to obtain a real vehicle speed corresponding to the current vehicle;

When the actual vehicle speed of the current vehicle is greater than the speed threshold, it is determined that the current vehicle is speeding.

6. The method for road traffic inspection based on drone according to claim 1, characterized in that the vehicle dynamic information includes vehicle driving trajectory data; before performing traffic violation detection based on the lane semantic map and the vehicle static information and vehicle dynamic information of each vehicle to be tracked, obtaining the traffic violation detection result, it also includes:

Performing forward filtering and backward filtering on the vehicle driving trajectory data corresponding to each of the vehicles to be tracked to obtain filtered vehicle driving trajectory data of each of the vehicles to be tracked;

The traffic violation detection is performed according to the lane semantic map and the static vehicle information and the dynamic vehicle information of each vehicle to be tracked to obtain the traffic violation detection result, including:

Traffic violation detection is performed based on the lane semantic map, the vehicle static information of each vehicle to be tracked, and the filtered vehicle driving trajectory data to obtain the traffic violation detection result.

7. The method for road traffic inspection based on drone according to claim 1 is characterized in that after performing traffic violation detection according to the lane semantic map and the static and dynamic information of each vehicle to be tracked, and obtaining the traffic violation detection result, it further comprises:

The drone is controlled to output warning information to the target vehicle, where the warning information is used to prompt the target vehicle that a traffic violation has occurred.

8. The method for road traffic inspection based on drone according to claim 1, characterized in that the step of determining the road traffic inspection result corresponding to the target road section according to the target road infrastructure and the target vehicle comprises:

Determining road infrastructure information of the target road infrastructure;

Determining the illegal behavior information corresponding to the target vehicle;

The road traffic inspection result is generated according to the road infrastructure information and the illegal behavior information.

9. The method for road traffic inspection based on an unmanned aerial vehicle according to claim 8, characterized in that the step of determining the road infrastructure information of the target road infrastructure comprises:

Acquire the infrastructure image corresponding to the target road infrastructure collected by the drone;

Determining infrastructure location information corresponding to the target road infrastructure;

The road infrastructure information is generated according to the infrastructure image and the infrastructure location information.

10. The road traffic inspection method based on drone according to claim 9, characterized in that the determining of the infrastructure location information corresponding to the target road infrastructure comprises:

Acquire a first image coordinate and a coordinate conversion coefficient of the UAV in an image coordinate system, and acquire a second image coordinate of the target road infrastructure in the image coordinate system, wherein the first image coordinate is a center point of the image coordinate system;

Obtaining first satellite positioning coordinates of the UAV;

Performing coordinate conversion according to the first satellite positioning coordinates, the first image coordinates, the coordinate conversion coefficient, and the second image coordinates to obtain second satellite positioning coordinates corresponding to the target road infrastructure;

The infrastructure location information is generated according to the second satellite positioning coordinates.

11. The method for road traffic inspection based on drone according to claim 8, characterized in that the step of determining the illegal behavior information corresponding to the target vehicle comprises:

Obtaining the illegal behavior video data corresponding to the target vehicle collected by the drone;

Performing license plate number recognition on the target vehicle to obtain the license plate number corresponding to the target vehicle;

The illegal behavior information is generated according to the illegal behavior video data of the target vehicle and the license plate number.

12. The method for road traffic inspection based on drone according to claim 1, characterized in that the determining of the lane semantic map corresponding to the target road section comprises:

Acquire target aerial survey data of the target road section;

Based on the trained lane line detection model, lane line detection is performed on the target aerial survey data to obtain a lane line detection result of the target road section, wherein the lane line detection result includes a multi-frame lane line semantic map and a multi-frame center line semantic map;

Lane positioning is performed based on the lane line semantic maps of multiple frames and the center line semantic maps of multiple frames to obtain a lane semantic map of the target road section.

13. A drone, characterized in that the drone comprises a controller, a shooting device and a voice playing device;

The photographing device is used to collect road traffic images of the target road section to be inspected;

The voice playback device is used to output warning information, and the warning information is used to prompt the traffic violation vehicle to have committed a traffic violation;

The controller is used to execute the drone-based road traffic inspection method as described in any one of claims 1 to 12.

14. A computer device, characterized in that the computer device comprises a processor and a memory;

The memory is used to store computer programs;

The processor is used to execute the computer program and implement the drone-based road traffic inspection method as described in any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the road traffic inspection method based on an unmanned aerial vehicle as described in any one of claims 1 to 12.