CN118865175A - Automatic labeling system and method of UAV aerial survey data based on artificial intelligence - Google Patents

Abstract

The invention relates to the technical field of aerial survey labeling, in particular to an unmanned aerial vehicle aerial survey data automatic labeling system and method based on artificial intelligence, comprising the following steps: recording shooting environment data and shooting attitude data corresponding to each aerial survey image during unmanned aerial vehicle navigation; extracting i areas which cannot be automatically marked in the aerial survey image, and marking the i areas as unrecognized areas; inputting shooting environment data and shooting gesture data corresponding to areas which cannot be automatically marked into a first machine learning model after training is completed, and outputting the corresponding standard pixel block number; dividing i unidentified regions in turn according to the number of standard pixel blocks to obtain sub-regions corresponding to each unidentified region; sequentially inputting shooting environment data, shooting gesture data and n sub-areas corresponding to the aerial survey image into a second machine learning model after training is completed, and obtaining a labeling result corresponding to each sub-area; the marking strength of the person to the unrecognized area is effectively reduced.

Description

Automatic labeling system and method for UAV aerial survey data based on artificial intelligence

Technical Field

The present invention relates to the field of aerial survey annotation technology, and in particular to an automatic annotation system and method for unmanned aerial survey data based on artificial intelligence.

Background Art

With the rapid development of drone technology, drone aerial surveys have been widely used in the fields of geographic information, urban planning, agricultural monitoring, etc. However, a large amount of aerial survey data needs to be accurately and efficiently labeled to extract valuable information. In the existing technology, although artificial intelligence is used to automatically label drone aerial survey data, there are still unidentified areas. These unidentified areas usually need to be reported directly and manually identified and labeled by staff, resulting in the workload and intensity of manual labeling being still high and not effectively reduced.

Summary of the invention

In order to solve the above problems, the present invention provides an automatic labeling system and method for UAV aerial survey data based on artificial intelligence.

The present invention adopts the following technical scheme, a method for automatic labeling of drone aerial survey data based on artificial intelligence, including:

Record the shooting environment data and shooting posture data corresponding to each aerial survey image during drone aerial survey;

Extract i areas in the aerial survey image that have not been automatically annotated and mark them as unrecognized areas;

Input the shooting environment data and shooting posture data corresponding to the area that cannot be automatically labeled into the first machine learning model that has been trained, and output the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks that are adjacent in sequence;

Divide the i unrecognized regions in turn by the number of standard pixel blocks to obtain sub-regions corresponding to each unrecognized region;

The shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image are input into the trained second machine learning model in sequence to obtain the labeling results corresponding to each sub-area, and the labeling results include the labeling name and labeling failure.

As a further description of the above technical solution: the method of dividing the sub-regions is: a square region is obtained with the number of standard pixel blocks as the length, and the unidentified region is divided into n sub-regions by the square region, where n is an integer greater than or equal to 1; a region that is not large enough to form a square region is also regarded as a sub-region.

As a further description of the above technical solution: the training method of the first machine learning model includes:

Pre-collect k groups of recognition unit training data, the recognition unit training data including recognition unit feature data and the number of standard pixel blocks corresponding to the recognition unit feature data, the recognition unit feature data including shooting environment data and shooting posture data, and convert the recognition unit training data into a first feature vector;

Each group of first feature vectors is used as the input of the first machine learning model. The first machine learning model uses the predicted standard pixel block number vector corresponding to each group of identification unit feature data vectors as the output, and the actual standard pixel block number vector corresponding to each group of identification unit feature data vectors as the prediction target; minimizing the sum of the first prediction errors of all identification unit feature data vectors is used as the training target; the first machine learning model is trained until the sum of the first prediction errors reaches convergence, and the training is stopped; the first machine learning model is a random forest model, a gradient boosting tree model or a recurrent neural network model.

As a further description of the above technical solution: the calculation formula of the first prediction error is Z _K =(α _K -μ _K ) ² , where Z _K is the first prediction error, K is the group number of the identification unit feature data vector corresponding to the first feature vector, α _K is the predicted standard pixel block number vector corresponding to the Kth group of identification unit feature data vectors, and μ _K is the actual standard pixel block number vector corresponding to the Kth group of identification unit feature data vectors.

As a further description of the above technical solution: the shooting environment data includes season, ambient brightness and weather, and the shooting attitude data includes the height above the ground, tilt angle and speed of the drone: the tilt angle includes pitch, yaw and roll.

As a further description of the above technical solution: the labeled names are highways, rivers, buildings, lakes, farmlands, airports, industrial areas and parks.

As a further description of the above technical solution: it also includes:

Add the annotation name to the corresponding sub-region position in the aerial survey image to form a secondary annotation image;

Input the secondary annotated images into the trained rationality recognition model and output the unreasonable annotation names.

As a further description of the above technical solution: the secondary labeled image includes a labeled name and an area corresponding to each labeled name.

As a further description of the above technical solution: the training method of the rationality recognition model includes:

Collect recognition data in advance, the recognition data includes secondary annotated pictures and unreasonable annotated names, convert the recognition data into a second feature vector, divide the second feature vector into a training set and a test set, use the secondary annotated picture vector in the second feature data as the input of the rationality recognition model, use the unreasonable annotated name vector in the training set as the output of the rationality recognition model, train the rationality recognition model to obtain an initial rationality recognition model, test the initial rationality recognition model with the test set, and output a rationality recognition model that meets the preset accuracy. The rationality recognition model is a convolutional neural network model.

The automatic labeling system for drone aerial survey data based on artificial intelligence is used to implement the automatic labeling method for drone aerial survey data based on artificial intelligence. The system includes:

The recording module is used to record the shooting environment data and shooting posture data corresponding to each aerial survey image during the UAV aerial survey;

The first processing module extracts i areas in the aerial survey image that have not been automatically marked and marks them as unrecognized areas;

The second processing module inputs the shooting environment data and shooting posture data corresponding to the area that cannot be automatically marked into the trained first machine learning model, and outputs the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks that are adjacent in sequence;

The third processing module is used to divide the i unrecognized areas in sequence according to the number of standard pixel blocks to obtain sub-areas corresponding to each unrecognized area;

A fourth processing module is used to input the shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image into the trained second machine learning model in sequence, and obtain a labeling result corresponding to each sub-area, where the labeling result includes a labeling name and a labeling failure;

A fifth processing module is used to add the annotation name to the corresponding sub-region position in the aerial survey image to form a secondary annotation image;

The rationality processing module is used to input the secondary labeled images into the trained rationality recognition model and output the unreasonable labeling names.

Beneficial effects:

For areas that cannot be automatically identified, the shooting environment data and shooting posture data of the aerial survey image are obtained, and the appropriate number of standard pixel blocks is selected to improve the accuracy of subsequent annotation results; selecting the appropriate number of standard pixel blocks can effectively improve the recognition accuracy.

First, the unrecognized area is divided into square sub-regions according to the number of standard pixel blocks to obtain several sub-regions. This detailed division can pay more attention to details and improve the recognition ability of tiny objects and complex scenes. Each sub-region is processed using a standard recognition model. Since the sub-region is small, the model can focus more on the features within the region, reduce interference factors, and improve recognition accuracy.

Taking the shooting environment data, shooting posture data and several sub-areas corresponding to the aerial survey image as input data, the model can analyze the unidentified areas more comprehensively and accurately label the names. Through automated processing, it reduces the intensity of manual labeling of unidentified areas, and also improves the completeness and accuracy of the overall labeling of the aerial survey image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a module connection diagram of an automatic annotation system for drone aerial survey data based on artificial intelligence in Embodiment 1 of the present invention;

FIG2 is a module connection diagram of an automatic annotation system for drone aerial survey data based on artificial intelligence in Embodiment 2 of the present invention;

FIG3 is a schematic diagram of a flow chart of an automatic labeling method for drone aerial survey data based on artificial intelligence in Embodiment 3 of the present invention.

DETAILED DESCRIPTION

In order to make the technical means, creative features, objectives and effects of the present invention easy to understand, the present invention is further described below with reference to specific diagrams. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

Please refer to Figure 1. An embodiment of the present invention provides an automatic annotation system for drone aerial survey data based on artificial intelligence, including: a recording module, a first processing module, a second processing module, a third processing module and a fourth processing module, and each module is connected by wire and/or wirelessly.

The recording module is used to record the shooting environment data and shooting attitude data corresponding to each aerial survey image during drone aerial survey; the shooting environment data includes season, ambient brightness and weather, and the shooting attitude data includes the drone's height from the ground, tilt angle and speed: the tilt angle includes pitch, yaw and roll.

The first processing module extracts i areas in the aerial survey image that cannot be automatically marked and marks them as unrecognized areas.

The second processing module inputs the shooting environment data and shooting posture data corresponding to the area that cannot be automatically marked into the trained first machine learning model, and outputs the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks adjacent to each other in sequence.

For areas that have not been automatically labeled, how to divide them into small areas for re-identification, that is, sub-areas below. The size of the sub-areas will directly affect the accuracy of the subsequent re-labeling results. The inventor uses season, ambient brightness, weather, altitude, tilt angle and speed of the drone as inputs to the recognition unit model for the following reasons:

Weather conditions such as sunny, cloudy, and haze will affect the clarity and contrast of the image. For example, a cloudy day will cause the image to be dark overall, while haze will cause the image to be blurred. When the image quality is low, it is divided into smaller areas to extract more details and reduce the negative impact of weather on the recognition results. The ambient brightness affects the shadows and brightness of the image. For example, strong sunlight will cast obvious shadows on the objects, while low light may make some objects difficult to identify. In poor lighting conditions, dividing into smaller areas can reduce the impact of shadows and lighting changes on recognition and improve recognition accuracy. Seasonal changes can cause changes in the appearance of objects, such as plant growth and withering, snow coverage, etc. For example, there is a big visual difference between farmland in summer and farmland covered with snow in winter. In the case of obvious seasonal changes, dividing into smaller areas can capture more details and help distinguish the characteristics of objects in different seasons.

The tilt angle of the drone will cause geometric distortion of the image, such as stretching, compression, rotation, etc. A larger tilt angle needs to be divided into smaller areas to reduce the impact of distortion on recognition. Using smaller areas can help better correct the image and maintain recognition accuracy. The speed of the drone will affect the clarity and overlap of the image. A faster speed will cause the image to be blurred, while a slower speed can obtain a clearer image. At a faster speed, dividing into smaller areas can reduce the impact of blur on recognition and ensure that the model can capture more details, and vice versa. The number of standard pixel blocks corresponding to dividing into smaller areas is smaller, and vice versa.

In summary, the shooting environment data and the shooting posture data are closely related to the number of selected unit number pixel blocks.

The training method of the first machine learning model includes:

Collect k groups of recognition unit training data in advance, the recognition unit training data includes recognition unit feature data and the number of standard pixel blocks corresponding to the recognition unit feature data, the recognition unit feature data includes shooting environment data and shooting posture data, and convert the recognition unit training data into a first feature vector.

Each group of first feature vectors is used as the input of the first machine learning model. The first machine learning model uses the predicted standard pixel block number vector corresponding to each group of identification unit feature data vectors as the output, and uses the actual standard pixel block number vector corresponding to each group of identification unit feature data vectors as the prediction target; minimizing the sum of the first prediction errors of all identification unit feature data vectors is used as the training target; wherein the calculation formula of the first prediction error is Z _K = (α _K - μ _K ) ² , wherein Z _K is the first prediction error, K is the group number of the first feature vector corresponding to the identification unit feature data vector, α _K is the predicted standard pixel block number vector corresponding to the Kth group of identification unit feature data vectors, and μ _K is the actual standard pixel block number vector corresponding to the Kth group of identification unit feature data vectors; the first machine learning model is trained until the sum of the first prediction errors converges and the training is stopped; the first machine learning model is a random forest model, a gradient boosting tree model (such as XGBoost, LightGBM) or a recurrent neural network model.

The third processing module is used to divide the i unidentified areas in sequence according to the number of standard pixel blocks to obtain sub-areas corresponding to each unidentified area.

The method of dividing the sub-regions is: a square region is obtained with the number of standard pixel blocks as the length, and the unrecognized region is divided into n sub-regions by the square region, where n is an integer greater than or equal to 1; a region that is not enough to form a square region is also regarded as a sub-region.

The fourth processing module is used to input the shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image into the trained second machine learning model in sequence to obtain the labeling results corresponding to each sub-area, and the labeling results include labeling names and labeling failures; the sub-areas corresponding to the labeling failures are reported and manually reviewed and labeled; the labeling names include highways, rivers, buildings, lakes, farmlands, airports, industrial areas, parks, etc.

In this embodiment, for areas that cannot be automatically identified, the shooting environment data and shooting posture data corresponding to the aerial survey image are obtained to select an appropriate number of standard pixel blocks. The appropriate number of standard pixel blocks effectively improves the accuracy of subsequent re-annotation results; a square area is then obtained with the number of standard pixel blocks as the length, and the unidentified area is divided into n sub-areas by the square area. The identification of sub-areas in a small range can pay more attention to details and improve the recognition ability of tiny objects and complex scenes. During processing, the first machine learning model can focus more on the features in the area, reduce interference factors, and improve recognition accuracy; the shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image are then used as input data to obtain the labeling name of the unidentified area, thereby reducing the intensity of manual labeling of the unidentified area, and at the same time effectively improving the integrity and accuracy of the overall labeling of the aerial survey image.

Example 2

Please refer to FIG. 2 , the automatic annotation system for drone aerial survey data based on artificial intelligence provided in this embodiment also includes:

The fifth processing module is used to add the annotation name to the corresponding sub-region position in the aerial survey image to form a secondary annotation image, which includes the annotation name and the region corresponding to each annotation name.

The rationality processing module is used to input the secondary labeled images into the trained rationality recognition model, output the unreasonable labeling names, report the unreasonable labeling names manually, check and re-label them, effectively improving the accuracy of automatic labeling and the efficiency of rationality screening.

For example, in a large farmland marking area, a high-rise building marking appears. There are obvious geographical and functional differences between farmland and high-rise buildings. It is impossible for a high-rise building to appear in the middle of farmland. The reason for the error may be that some structures in the farmland (such as irrigation facilities) are mistaken for buildings; in a highway marking area, an airport runway marking appears. The construction and use of highways and airport runways are completely different. It is impossible for an airport runway to be adjacent to a highway. The reason for the error is that the model confuses the characteristics of linear features, such as similar lengths, widths, and directions; in a park green space marking area, an industrial area marking appears. The interior of a park is usually green space, trails, and entertainment facilities. It is impossible for there to be large-scale industrial facilities. The reason for the error is that the model confuses some green areas of the park and the industrial area.

The training methods of the rationality recognition model include:

Collect recognition data in advance, the recognition data includes secondary annotated pictures and unreasonable annotated names, convert the recognition data into a second feature vector, divide the second feature vector into a training set and a test set, use the secondary annotated picture vector in the second feature data as the input of the rationality recognition model, use the unreasonable annotated name vector in the training set as the output of the rationality recognition model, train the rationality recognition model to obtain an initial rationality recognition model, test the initial rationality recognition model with the test set, and output a rationality recognition model that meets the preset accuracy. The rationality recognition model is a convolutional neural network model, such as VGG16, ResNet, etc.

Example 3

As shown in FIG3 , this embodiment provides an automatic labeling method for drone aerial survey data based on artificial intelligence, including:

Record the shooting environment data and shooting posture data corresponding to each aerial survey image during drone aerial survey;

Extract i areas in the aerial survey image that have not been automatically annotated and mark them as unrecognized areas;

Input the shooting environment data and shooting posture data corresponding to the area that cannot be automatically labeled into the first machine learning model that has been trained, and output the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks that are adjacent in sequence;

Divide the i unrecognized regions in turn by the number of standard pixel blocks to obtain sub-regions corresponding to each unrecognized region;

The shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image are input into the trained second machine learning model in sequence to obtain the labeling results corresponding to each sub-area, and the labeling results include the labeling name and labeling failure.

The basic principles, main features and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and the description in the specification are only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims (10)

1. The unmanned aerial vehicle aerial survey data automatic labeling method based on artificial intelligence is characterized by comprising the following steps of:

recording shooting environment data and shooting attitude data corresponding to each aerial survey image during unmanned aerial vehicle navigation;

extracting i areas which cannot be automatically marked in the aerial survey image, and marking the i areas as unrecognized areas;

Inputting shooting environment data and shooting gesture data corresponding to areas which cannot be automatically marked into a first machine learning model after training is completed, and outputting the corresponding standard pixel block number, wherein the pixel blocks are sequentially adjacent pixel blocks;

dividing i unidentified regions in turn according to the number of standard pixel blocks to obtain sub-regions corresponding to each unidentified region;

And sequentially inputting shooting environment data, shooting gesture data and n sub-areas corresponding to the aerial survey image into a second machine learning model after training is completed, and obtaining a labeling result corresponding to each sub-area, wherein the labeling result comprises a labeling name and labeling failure.

2. The unmanned aerial vehicle aerial survey data automatic labeling method based on artificial intelligence according to claim 1, wherein the method for dividing the subareas is as follows: the method comprises the steps of obtaining square areas by taking the number of standard pixel blocks as the length, dividing unidentified areas by the square areas to obtain n sub-areas, wherein n is an integer greater than or equal to 1; it is not sufficient to form square areas as well as a sub-area.

3. The unmanned aerial vehicle aerial survey data automatic labeling method based on artificial intelligence of claim 1, wherein the training method of the first machine learning model comprises:

K groups of recognition unit training data are collected in advance, the recognition unit training data comprise recognition unit feature data and standard pixel block numbers corresponding to the recognition unit feature data, the recognition unit feature data comprise shooting environment data and shooting gesture data, and the recognition unit training data are converted into first feature vectors;

taking each group of first feature vectors as input of a first machine learning model, wherein the first machine learning model takes the number vectors of the prediction standard pixel blocks corresponding to each group of recognition unit feature data vectors as output, and takes the number vectors of the actual standard pixel blocks corresponding to each group of recognition unit feature data vectors as a prediction target; taking the sum of the first prediction errors of all the recognition unit characteristic data vectors as a training target; training the first machine learning model until the sum of the first prediction errors reaches convergence, and stopping training; the first machine learning model is a random forest model, a gradient lifting tree model or a cyclic neural network model.

4. The automatic labeling method for aerial survey data of an unmanned aerial vehicle based on artificial intelligence according to claim 3, wherein the calculation formula of the first prediction error is Z _K＝(α_K-μ_K)², wherein Z _K is the first prediction error, K is the group number of the first feature vector corresponding to the identification unit feature data vector, alpha _K is the prediction standard pixel block number vector corresponding to the identification unit feature data vector of the K-th group, and mu _K is the actual standard pixel block number vector corresponding to the identification unit feature data vector of the K-th group.

5. The automatic labeling method for unmanned aerial vehicle aerial survey data based on artificial intelligence according to claim 1, wherein the shooting environment data comprises seasons, environment brightness and weather, and the shooting gesture data comprises ground altitude, inclination angle and speed of the unmanned aerial vehicle: the tilt angle includes pitch, yaw and roll.

6. The automatic labeling method for unmanned aerial vehicle aerial survey data based on artificial intelligence according to claim 1, wherein the labeling names are expressways, rivers, buildings, lakes, farmlands, airports, industrial areas and parks.

7. The automatic labeling method for unmanned aerial vehicle aerial survey data based on artificial intelligence according to claim 1, further comprising:

adding the labeling name to the position of the corresponding subarea in the aerial survey image to form a secondary labeling picture;

And inputting the secondary labeling picture into a rationality recognition model after training is completed, and outputting an unreasonable labeling name.

8. The automatic labeling method for unmanned aerial vehicle aerial survey data based on artificial intelligence according to claim 7, wherein the secondary labeling picture comprises labeling names and areas corresponding to each labeling name.

9. The unmanned aerial vehicle aerial survey data automatic labeling method based on artificial intelligence of claim 7, wherein the training method of the rationality recognition model comprises the following steps:

the method comprises the steps of collecting identification data in advance, wherein the identification data comprise secondary labeling pictures and unreasonable labeling names, converting the identification data into second feature vectors, dividing the second feature vectors into a training set and a testing set, using the secondary labeling picture vectors in the second feature data as the input of a rationality identification model, using the unreasonable labeling name vectors in the training set as the output of the rationality identification model, training the rationality identification model to obtain an initial rationality identification model, testing the initial rationality identification model by using the testing set, outputting the rationality identification model meeting the preset accuracy, and enabling the rationality identification model to be a convolutional neural network model.

10. The unmanned aerial vehicle aerial survey data automatic labeling system based on artificial intelligence is characterized in that the unmanned aerial vehicle aerial survey data automatic labeling system based on artificial intelligence is used for realizing the unmanned aerial vehicle aerial survey data automatic labeling method based on artificial intelligence as set forth in any one of 1-9, and the system comprises:

the recording module is used for recording shooting environment data and shooting gesture data corresponding to each aerial survey image when the unmanned aerial vehicle is in navigation;

the first processing module is used for extracting i areas which cannot be automatically marked in the aerial survey image and marking the i areas as unidentified areas;

The second processing module inputs shooting environment data and shooting attitude data corresponding to areas which cannot be automatically marked into a first machine learning model after training is completed, and outputs the corresponding standard pixel block number, wherein the pixel blocks are sequentially adjacent pixel blocks;

The third processing module is used for dividing the i unidentified areas in sequence according to the number of the standard pixel blocks to obtain sub-areas corresponding to each unidentified area;

the fourth processing module is used for sequentially inputting shooting environment data, shooting gesture data and n sub-areas corresponding to the aerial survey image into the second machine learning model after training is completed, and obtaining a labeling result corresponding to each sub-area, wherein the labeling result comprises a labeling name and labeling failure;

the fifth processing module is used for adding the labeling name to the position of the corresponding subarea in the aerial survey image to form a secondary labeling picture;

and the rationality processing module is used for inputting the secondary labeling picture into the rationality recognition model after training is completed and outputting the unreasonable labeling name.