WO2023120908A1 - Drone having onboard flight control computer, and system for obtaining positional coordinates of drone camera video object by using drone - Google Patents

Abstract

The present invention provides a system for obtaining positional coordinates of a camera video object by using a drone having an onboard flight control computer, the system comprising: a global positioning system (GPS) configured to use a global navigation satellite system (GNSS) receiver and receive RTK/DGPS calibration information to perform GPS calibration; an onboard flight control computer which receives gimbal information and video information (H.264 image) from a camera gimbal, obtains coordinates of the camera gimbal to generate meta information in a frame when image-synchronized MPEG transmission is performed, and transmits the meta information by using a frame-adjusted image processing algorithm; a smart controller configured to include the onboard flight control computer, an image communication unit, and a dual IP relay unit; an artificial intelligence learner which calculates positional coordinates of the time of a video and provides positional coordinates of a drone camera video object to a mixed reality monitor; and the mixed reality monitor configured to include smart glasses and a monitor to display the positional coordinates of the drone camera video object.

Description

A drone equipped with an on-board flight control computer and a drone camera video object location coordinate acquisition system using the same

The present invention relates to a drone equipped with an on-board flight control computer, and more particularly, when the coordinates of a video object viewed by a drone camera are acquired during the drone's high-speed flight and the position of an object extracted by artificial intelligence is determined, the drone camera acquires the coordinates. Real-time video, drone flight information, and gimbal attitude information are processed by the on-board flight control computer, and the drone's on-board flight control computer converts them into global coordinates and standard attitude values to form a frame in a container format that fuses video and meta information. It is stored and transmitted to the smart controller, and the MPEG video format received from the ground control system (GCS) of the smart controller is separated (parsed) into video and meta information, displayed in geographic information, and transmitted from the smart controller to the artificial intelligence processor. An on-board flight control computer that can process the determined position and time information by separating the extracted MPEG and identifying the video through AI-learned object recognition and synchronizing the time coordinates of the object calculated as meta information for the identified location coordinates. It relates to a drone equipped with and a drone camera video object location coordinate acquisition system using the same.

Image acquisition technology using drones is continuously evolving. In addition, as artificial intelligence technology is applied to drones, the importance of object identification of real-time drone video location and time information is emerging. Demand for service technology is expanding.

Until now, it is common for the drone's video to receive the video and GPS flight information separately using the drone's GPS value as the drone's flying location, and to match it with the image storage to obtain the location coordinates filmed by the drone, and to store and utilize the GPS flight information. am.

However, until now, errors in the position of drones, distances, angles and attitudes that the camera is looking at, and transmission delays due to irregular sizes of data in video transmission have occurred, causing difficulties due to too large errors.

As a way to solve this difficulty, research is being conducted to synchronize the position of the camera the drone is looking at in real time by mounting a computer on the drone or installing an expensive image processor and signal processor. There was a problem that the size should be large, and signal delay occurred in the process of connecting each device, so the size was too large, and it was difficult to operate with expensive equipment.

In addition, the real-time frame characteristics of the video in the drone show that the drone flight posture and data expression value and the camera gimbal attitude expression value are moving and output differently. should be converted into a global standard posture value and stored in meta information, and the altitude value that the camera is looking at is very important. DGPS, which corrects the GPS, corrects the altitude within 1m, and RTK correction, which corrects the altitude within 10cm. This is required, so it is not popularized due to the difficulty of miniaturization and light weight method in drones.

In particular, drone video characteristics change more severely than video frames that are generally fixed as the flight path changes, and the video frame compresses and transmits only the changed information of the previous frame and the current frame. When the change is large, the video frame data increases, and when transmitted wirelessly from the drone, there is a characteristic that transmission delay time occurs due to irregular specifications of the amount of data, and even if the drone operator identifies an object in the video received from the drone, at any point in time There is a problem that it is difficult to check whether it is a video or only a frame video.

Recently, as drone images are grafted onto artificial intelligence, the demand for real-time location and time information of object identification in drone images is increasing. Therefore, in order to acquire the positional coordinates of the time of the drone video while the drone is in flight, a real-time synchronization method is required for the video frame through the information of the gimbal posture that the camera is looking at and the information of the flying posture of the drone, the camera angle, and the precise altitude information.

In addition, the video obtained from the drone must be wirelessly transmitted to the drone operator. The video transmission causes a large change in data increase, and the barge communication transmits video and meta information of position and attitude including camera specification information at a constant speed according to limited frequency resources. Data transmission control is required to ensure lossless transmission.

In addition, the video MPEG transmitted from the drone is received by the onboard computer of the smart controller and has a computer function that is simultaneously transmitted to the ground control system (GCS) and artificial intelligence processor. Since the information is used together with the geographic information system, the artificial intelligence processor can recognize the object of the video and acquire the location coordinates of the meta information through the learning engine. Since it is a method of expressing video location identification synchronized with the position of the drone camera at the time of the location viewed by the drone camera, the frame in which the real-time video is expressed in the coordinates of the actual location is transferred between the on-board flight control computer and smart controller mounted on the drone. It is necessary to implement it as an on-board server and provide a service with coordinate synchronization technology at the time of video shooting.

On the other hand, the conventional technology of Patent No. 10-1943823 is a method for determining the synchronization of UAV mission equipment for obtaining precise position data during high-speed flight and UAV mission equipment for obtaining precise position data during high-speed flight. It is an invention related to and the prior art is a synchronization method in photography necessary for 3D modeling using a drone.

The existing shooting method generally describes a method of shooting by fixing the camera gimbal mounted on a drone in a direct downward direction and synchronizing the drone's GPS coordinates and the camera gimbal's posture information with photos and meta information, and 3D modeling This is a method of generating orthoimages with stereo imaging techniques according to the degree of overlap of the photos. To improve precision, the operator shoots the coordinates of the ground control point (GCS) on the ground for post-processing correction with a drone and commercialized 3D modeling software This is a photographing method that improves accuracy through orthogonal mosaic processing based on the ground reference point.

This method is a method for determining the position of a video taken by a drone by synchronizing the determination of a position for taking a picture with GPS information according to a photographing route, and is a technology for increasing the accuracy of the position of taking a picture.

And Registered Patent No. 10-2242366 is a drone-based ground control point placement automation device for digital map creation at earthwork sites. Because of the large GPS error, RTK correction technology with cm precision is used to place virtual control points to create a 3D digital map. A method for taking pictures using RTK technology is presented.

In particular, analog or HDMI communication methods are widely used as existing video transmission methods. This is an image dump method, which has a problem of not including meta information as a method of transmitting video in the same standard to solve the problem of increasing data during drone movement. When the information is converted to analog HDMI, meta information included in the video frame has a problem of being deleted, so it is a video format that is not used in coordinate recognition.

In addition, RTK synchronization technology, a separate satellite navigation (GNSS) correction technology, is applied to increase the accuracy of location information in the video of the drone, but this only has the effect of improving the precision of the drone location, and calculates the location that the real-time camera is looking at. When connecting different devices such as cameras, gimbals, airframes, flight controls, etc. to determine positional coordinates, camera information, standard attitude information, precise angle and altitude that are fused with drone attitude and gimbal attitude, without signal processing loss, at the GPS time required for Integration and accumulation errors of acquired information occur, and it has technical limitations due to data delay.

In addition, the characteristic of video in the prior art is that the current video frame is information transmitted by compressing the changed information of the previous frame, so each frame has a feature that the exact time and place are not known, and the amount of data of a specific frame increases when wirelessly transmitted from a drone. Due to the problem of delay time, it is difficult to accurately identify anomalies even after detecting object identification, resulting in an error of hundreds of meters. Even if there is a continuous image of the value, the problem of not being able to know the location and time point with only the video frame that the operator looks at after the video transmission is always on the rise.

The convergence of video processing and meta information requires a high-performance computer to be installed on the drone, so synchronization including precise posture and angle of view and video convergence with meta information are possible. It is difficult to implement as much computer convergence technology as possible, and when signal processing is dispersed, the delay problem and when the capacity of a large amount of video frames change greatly, the video is delayed with a limited frequency in the wireless communication section or delayed due to CPU processing limitations of video transmission. A problem is occurring.

In this way, conventionally, the problem of image processing that is not synchronized due to data integration from drones to various peripheral device interfaces and the problem of video and meta information being damaged when digital images of meta information containers are converted to HDMI or analog formats. There was a problem of automatic control of video wireless traffic transmitted from drones and communication delays when large-capacity data was transmitted in limited communication frequencies.

In the present invention, a redundant server computer configuration method for transmitting the video of a smart controller transmitted from a drone without loss of time coordinates of video digital information in a video frame to ground control equipment and an artificial intelligence processor and a ground control system in a smart controller ( GCS) displays the degree of confinement seen by drones in the geographic information system (GIS) and how videos are expressed in real-time coordinates, and artificial intelligence processors learn object identification through video and meta information through MPEG and object identification We are trying to solve the way in which services are provided by calculating the coordinates and realizing mixed reality of artificial intelligence through smart glasses or monitors.

Therefore, the present invention is to solve various disadvantages and problems of the prior art as described above, and configures a Linux operating system server on an on-board flight controller (FC) and a flight control computer board (FCC). Real-time camera image acquisition, gimbal control information, attitude information of the flight controller, aircraft altitude angle sensor information are synchronized with GPS time information to process real-time video optimized for the camera, and signal synchronization optimization of the onboard flight controller A drone equipped with an on-board flight control computer with an interface that synchronizes and integrates all signal processing operated in the drone with a Linux-based operating system and includes a flight control computer as a single signal processing board integrated for the purpose of operation and drone camera video using the same. Its purpose is to provide an object position coordinate acquisition system.

In addition, the present invention interlocks the posture of the drone body and the posture of the camera gimbal in real time through the interface of the camera gimbal and the flight controller, obtains standard attitude information in the earth coordinate system through an algorithm, obtains information on the angle of view and resolution of the camera, and the altimeter of the drone is inexpensive. Since it uses a sensor, it interlocks RTK (RTK) of less than 10 cm or DGPS (DGPS) of less than 1 m to link the corrected signal to GPS, and the flight control computer uses real-time camera video Based on this, meta information is created in the frame, and it is an MPEG structure in which a container structure is inserted in a time synchronization frame. In order to solve the problem that the data transmission speed is delayed according to the frame capacity when transmitting to the smart controller, if the frame size is large according to the video transmission time synchronization, the image processing algorithm that adjusts the frame so that it does not exceed the frequency set bandwidth by automatically adjusting the frame runs. , The smart controller operated by the drone operator receives and processes MPEG video from the drone. It is simultaneously transmitted to the control system (GCS) and artificial intelligence processor, and the ground control system (GCS) receives the video MPEG, separates the video and meta information, interprets the meta information on the GIS map, and drones It mimics the camera direction and geographical position of the camera, and the video is displayed in real time at the location. Recognizes an object by recognizing an object, and the recognized object calculates the location coordinates of time based on meta information and displays the coordinates of the object location of the drone camera video, respectively, to provide a method for determining the location of time for artificial intelligence object identification. Another object is to provide a drone equipped with an on-board flight control computer and a drone camera video object location coordinate acquisition system using the same.

In order to achieve the above object, the present invention uses a global positioning system (GNSS) receiver, receives RTK / DGPS correction information, and is configured to perform GPS correction. A Linux operating system server is configured on the flight controller (FC) and the flight control computer board (FCC), and real-time camera image acquisition, gimbal control information, flight controller attitude information, aircraft altitude angle sensor information are stored in GPS time It synchronizes the clock with information, processes real-time video optimized for the camera, and integrates a signal processing board to optimize the signal synchronization of the onboard flight controller. It includes a flight control computer, and the camera gimbal 300 and flight controller interface Through real-time linkage of the posture of the drone body and the posture of the camera gimbal, obtaining standard posture information in the Earth coordinate system through a preset algorithm, obtaining information on the angle of view and resolution of the camera of the camera gimbal 300, based on real-time camera video Meta information is created in the frame, and it is an MPEG structure in which a container structure is inserted into the time synchronization frame. When the frame size is large according to the video transmission time synchronization, the on-board flight control computer is configured to include an on-board flight control computer 200 using an image processing algorithm that adjusts the frame so that the frame does not exceed the set frequency band width by automatically adjusting the frame A drone is provided.

Here, the on-board flight control computer 200 includes an attitude/altitude/angle sensor unit, an ESC (Electronic Speed Controller) power control unit for driving a motor, a video communication unit for transmitting image data to the smart controller 400, and the above It is characterized in that it is configured to include a control communication unit for receiving a control signal from the smart controller 400 and a flight control unit for controlling flight by control of the control communication unit.

And in order to achieve the above object, the present invention uses a global positioning system (GNSS) receiver, and is configured to receive RTK / DGPS correction information and perform GPS correction (GPS) (100); A Linux operating system server is configured on the on-board flight controller (FC) and flight control computer board (FCC), real-time camera image acquisition, gimbal control information, flight controller attitude information, aircraft altitude angle Synchronizes sensor information with GPS time information, processes real-time videos optimized for cameras, and integrates a flight control computer with a single signal processing board to optimize signal synchronization of the on-board flight controller, and includes a camera gimbal ( 300) and the flight controller interface to link the posture of the drone body and the posture of the camera gimbal in real time, obtain standard posture information in the earth coordinate system through a preset algorithm, obtain angle of view and resolution information of the camera of the camera gimbal 300, Based on the real-time camera video, meta information is created in the frame, and it is an MPEG structure with a container structure inserted in the time synchronization frame. On-board flight control computer 200 using an image processing algorithm that adjusts the frame so that the frame does not exceed the frequency set bandwidth by automatic frame adjustment when the frame size is large according to the video transmission time synchronization when transmitted to the controller 400; The controller on-board computer is driven by Linux operating system, includes a control controller, includes a ground control system, and provides a control communication unit for controlling the on-board flight control computer 200 and video communication with the on-board flight control computer 200. A smart controller 400 comprising a video communication unit and a dual IP relay unit to perform; By receiving MPEG data from the smart controller 400, video and meta information are separated, and the video recognizes an object with a learned artificial intelligence algorithm, and the recognized object is positioned in time based on pre-learned meta information. An artificial intelligence learner 500 that calculates coordinates and provides drone camera video object position coordinates to the mixed reality monitor 600; and a mixed room monitor 600 comprising smart glasses or a monitor and displaying the drone camera video object position coordinates provided by the artificial intelligence learner 500; A camera video object location coordinate acquisition system using a drone is provided.

Here, the smart controller 400 is a smart controller operated by a drone pilot, and the on-board computer that receives and processes MPEG video from the on-board flight control computer 200 relays a Linux-based dual IP. Composed of a server (dual IP relay unit), MPEG streaming is simultaneously transmitted to the ground control system (GCS) and the artificial intelligence processor composed of the mixed reality monitor 600, and in the ground control system (GCS), video MPG It is characterized by being configured to receive MPEG, separate video and meta information, interpret meta information on a GIS map, simulate the drone's camera direction and geographical position, and display the video in real time at the location. do.

According to an embodiment of the present invention, there are the following effects.

First, stable real-time camera image acquisition, gimbal control information, posture information of the flight controller, aircraft by configuring the Linux operating system server on-board on the flight controller (FC) and flight control computer board (FCC). By synchronizing the altitude and angle sensor information with the GPS time information, real-time videos optimized for the camera can be processed, and the size of the drone can be drastically reduced by onboarding.

Second, the time coordinates of video digital information are configured with a redundant server computer to transmit the image of the smart controller transmitted from the drone without loss in the video frame to the ground control equipment and artificial intelligence processor, and the smart controller to the ground control system (GCS) ), the viewing angle of the drone is displayed in the geographic information system (GIS) and the video is expressed as real-time coordinates, and the artificial intelligence processor calculates the object identification and coordinates of the object learned through the video and meta information through MPEG. And by realizing the mixed reality of artificial intelligence through smart glasses or monitors, the video object location coordinates are stably acquired for the videos captured and transmitted by the drone camera moving in the air.

1 is a diagram for explaining the operation outline of a drone equipped with an on-board flight control computer and a system for obtaining location coordinates of a drone camera video object using the same according to the present invention;

2 is a block diagram for explaining an embodiment of a drone camera video object location coordinate acquisition system according to the present invention;

3 is a diagram for explaining an embodiment of a technology for precise position estimation and change detection based on object recognition learned in the drone camera video object location coordinate acquisition system according to the present invention;

4 and 5 are views for explaining a drone equipped with an on-board flight control computer according to the present invention;

6 is a diagram for explaining artificial intelligence learning object recognition and artificial intelligence learning results in the drone camera video object location coordinate acquisition system according to the present invention;

7 is a diagram for explaining the MPEG dual service without meta information loss in the smart controller and artificial intelligence learner of the drone camera video object location coordinate acquisition system according to the present invention;

8 is a diagram showing an example of executing a tank object recognition learning engine in an artificial intelligence learner of a drone camera video object position coordinate acquisition system according to the present invention and displaying it on a mixed reality monitor.

9 and 10 are diagrams showing an example of executing a human object recognition learning engine in an artificial intelligence learner of a drone camera video object position coordinate acquisition system according to the present invention and displaying it on a mixed reality monitor.

A preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

In addition, the terms used in the present invention have been selected from general terms that are currently widely used as much as possible, but in certain cases, there are terms arbitrarily selected by the applicant. It is intended to clarify that the present invention should be understood as the meaning of the term, not the name of. In addition, in describing the embodiments, descriptions of technical details that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to more clearly convey the gist of the present invention without obscuring it by omitting unnecessary description.

The present invention configures a Linux operating system server on an on-board flight controller (FC) and a flight control computer board (FCC), and provides real-time camera image acquisition and gimbal control information, flight controller attitude information, airframe Synchronizes altitude angle sensor information with GPS time information, processes real-time video optimized for cameras, and integrates a flight control computer into one signal processing board to optimize signal synchronization of the on-board flight controller. It has an interface that synchronizes and integrates all signal processing operated in drones with a Linux-based operating system. to obtain the camera's angle of view and resolution information, and since the drone's altimeter uses a low-cost sensor, RTK (RTK) of less than 10 cm is interlocked with the corrected signal to GPS to solve the problem of large errors. Alternatively, DGPS correction information of less than 1m is linked, and the flight control computer generates meta information in the frame based on the real-time camera video, and the drone flies with the MPEG structure in which the container structure is inserted in the time-synchronized frame. It includes the information that calculates the coordinates the camera is looking at from the position, and the frame size is adjusted to match the video transmission time synchronization to solve the problem that the data transmission speed is delayed depending on the frame capacity when transmitting video from the flight control computer to the smart controller. If the frame is large, the image processing algorithm that adjusts the frame so that the frequency does not exceed the set bandwidth is driven by automatic frame adjustment, and the smart controller operated by the drone operator is a controller that receives and processes MPEG video from the drone. Composed of a (Dual IP) relay server, MPEG streaming received from the smart controller is simultaneously transmitted to the ground control system (GCS) and artificial intelligence processor, and the ground control system (GCS) transmits video MPEG (MPEG) It receives and separates video and meta information, interprets meta information on GIS map, simulates the drone's camera direction and terrain position, displays video in real time at the location, and artificial intelligence processor transmits MPEG data. Recognizes the video and meta information, separates the video and recognizes the object with the learned artificial intelligence algorithm (to find a person, tank, etc.), and the recognized object calculates the location coordinates of the time based on the meta information, Provided is a drone equipped with an on-board flight control computer that displays object position coordinates and provides a time positioning method for artificial intelligence object identification, and a drone camera video object position coordinate acquisition system using the same.

1 is a diagram for explaining the operation outline of a drone equipped with an on-board flight control computer and a drone camera video object location coordinate acquisition system using the same according to the present invention, and FIG. 2 is a drone camera video object location coordinate acquisition system according to the present invention. It is a block configuration diagram for explaining an embodiment of the system.

The operation outline of a drone equipped with an on-board flight control computer and a drone camera video object location coordinate acquisition system using the same according to the present invention is shown in FIG. ), camera gimbal 300, smart controller 400, artificial intelligence learner 500, and mixed reality monitor 600, and flight control for the camera gimbal 300 in the drone camera video object location coordinate acquisition system. Coordinates are obtained from the computer 200, images and meta information are synchronized, and spiritually synchronized MPEG is transmitted from the mission computer (on-board flight control computer 200) provided in the drone. That is, transmission management of MPEG traffic is performed, the smart controller 400 is composed of dual IP relay servers for dual video streaming, and the mixed reality monitor 600 performs mixed reality by artificial intelligence learning (meta). Provide object recognition coordinate service by monitoring.

An embodiment of the drone camera video object position coordinate acquisition system for this purpose is as shown in FIG. It is configured to make GPS corrections.

The on-board flight control computer 200 configures a Linux operating system server on an on-board flight controller (FC) and a flight control computer board (FCC), and provides real-time camera image acquisition, gimbal control information, and flight controller Synchronizes the clock with the GPS time information of attitude information and aircraft altitude angle sensor information to process real-time video optimized for the camera, and to optimize the signal synchronization of the onboard flight controller. It is a Linux-based operating system and has an interface that synchronizes and integrates all signal processing operated in drones.

In particular, through the interface between the camera gimbal 300 and the flight controller, the posture of the drone body and the posture of the camera gimbal are interlocked in real time, and standard posture information in the Earth coordinate system is obtained through a preset algorithm, and information on angle of view and resolution of the camera of the camera gimbal 300 are obtained. Since the drone's altimeter uses a low-cost sensor, in order to solve the problem of large errors, RTK (RTK) less than 10 cm or DGPS (DGPS) correction information less than 1 m is interlocked to link the corrected signal to GPS. configured to do

In addition, the on-board flight control computer 200 generates meta information in frames based on real-time camera videos, and calculates the coordinates of the camera at the location where the drone flies with an MPEG structure in which a container structure is inserted into a time-synchronized frame. In order to solve the problem that the data transmission speed is delayed according to the frame capacity when the video is transmitted from the on-board flight control computer 200 to the smart controller 400, the frame size is automatically adjusted according to the video transmission time synchronization when the frame size is large. An image processing algorithm that adjusts the frame so as not to exceed the frequency set bandwidth was used.

This on-board flight control computer 200 is composed of a posture/altitude/angle sensor unit and an ESC (Electronic Speed Controller) power control unit for driving a motor, and is composed of a video communication unit, a control communication unit, and a flight control unit. do. The on-board flight control computer 200 receives precise location/altitude/time information from the global positioning system (GPS) 100 and gimbal information and video information (H.264 video) from the camera gimbal 300, and receives smart on the ground. It performs video communication and control communication with the controller 400 . That is, the on-board flight control computer 200 includes an attitude/altitude/angle sensor unit, an ESC (Electronic Speed Controller) power control unit for driving a motor, a video communication unit for transmitting image data to the smart controller 400, It is configured to include a control communication unit for receiving a control signal from the smart controller 400 and a flight control unit for controlling flight by control of the control communication unit.

The smart controller 400 is a smart controller operated by a drone operator, and the controller on-board computer that receives and processes MPEG video from the on-board flight control computer 200 of the drone is a Linux-based dual IP relay. Composed of a server (dual IP relay) and received from the smart controller, MPEG streaming is simultaneously performed by the ground control system (GCS) provided in the smart controller 400 and the artificial intelligence processor configured in the mixed reality monitor 600. The ground control system (GCS) receives the video MPEG, separates the video and meta information, interprets the meta information on the GIS map, and simulates the drone's camera direction and terrain position. It is displayed in real time at the location.

The smart controller 400 for this purpose is driven by the Linux operating system as a remote controller on-board computer, includes a control controller, includes a ground control system (GCS (tablet PC, etc.)), and controls the on-board flight control computer 200 It is configured to include a control communication unit, a video communication unit performing video communication with the on-board flight control computer 200, and a dual IP relay unit.

The artificial intelligence learner 500 receives MPEG data from the smart controller 400, separates the video and meta information, and the video is a learned artificial intelligence algorithm (eg, to find people, tanks, nematodes, etc.) ), and the recognized object calculates the location coordinates of the time based on the pre-learned meta information, and provides the drone camera video object location coordinates to the mixed reality monitor 600.

Here, the mixed room monitor 600 is composed of smart glasses or a monitor and displays object location coordinates of the drone camera video provided by the artificial intelligence learner 500.

3 is a diagram for explaining an embodiment of a technology for estimating a precise position and detecting a change based on object recognition learned in a drone camera video object position coordinate acquisition system according to the present invention.

As shown in FIG. 3, the embodiment of the object recognition-based precise position estimation and change detection technology learned in the drone camera video object position coordinate acquisition system according to the present invention first recognizes a deep learning object and estimates the distance. and relative. Absolute position is reported.

Here, deep learning object recognition,

Same as Equation 1,

Here, deep learning object recognition: L is loss, p is predicted class scores, u is true class scores, t ^u is true box coordinates, and v is predicted box coordinates, which is the difference between the actual position and the estimated position It is a formula that means learning to lower it.

And the distance estimation is as Equation 2.

Here, Distance on ground (D) is a target distance search on the ground, and Target global position is an object recognition location coordinate.

And the relative absolute position report is to estimate the precise position based on object recognition and detect the change change.

4 and 5 are views for explaining a drone equipped with an on-board flight control computer according to the present invention.

4 and 5 are views for explaining a drone equipped with an on-board flight control computer according to the present invention, the sensor unit configured in the on-board flight control computer 200 as shown in FIG. 2, flight control, and a Linux operating system. It shows that the operating flight control computer, control communication unit, video communication unit and power control unit are turned on on one board. Through this, in the conventional drone on the left side of FIG. 5, several boards such as sensors and main boards, flight control computer boards, gimbal control boards (gimbal controllers), battery packs made of battery cells, and power control boards (power controllers) are integrated into one drone. While the drone has become larger as the core parts are configured, the present invention, as shown in the right side, shows that it is possible to control the small and lightweight on-board flight by being on-boarded with one on-board flight control computer 200.

6 is a diagram for explaining artificial intelligence learning object recognition and artificial intelligence learning result processing in the drone camera video object location coordinate acquisition system according to the present invention, and FIG. 7 is a drone camera video object location coordinate acquisition system according to the present invention. Figure 8 is a diagram for explaining the MPEG dual service without meta information loss in the smart controller and artificial intelligence learner of the present invention. It is a drawing showing an example.

In the drone camera video object location coordinate acquisition system according to the present invention, artificial intelligence learning object recognition and processing of artificial intelligence learning results are the same as in FIGS. 6 and 7, but the GPS location in the GPS (100) Information, altitude/angle information, gimbal posture information (camera view angle information) and video information from the camera gimbal 300 are synchronized in the on-board flight control computer 200, and in the on-board flight control computer 200, the synchronized meta information and Meta information is created in the frame based on the real-time camera video of the video information, and the MPEG structure in which the container structure is inserted in the time synchronization frame calculates the coordinates of the camera at the location where the drone flies (Mission Computer Operation Department) . At this time, as described above, in order to solve the problem that the data transmission speed is delayed according to the frame capacity, if the frame size is large according to the video transmission time synchronization, the image processing algorithm is operated to adjust the frame so that the frame does not exceed the frequency set bandwidth by automatically adjusting the frame.

And when it is transmitted to the smart controller 400, the controller on-board computer consists of a Linux-based dual IP relay server (dual IP relay unit), and the smart controller 400 receives and processes the MPEG frame, The MPEG streaming received from 400) is simultaneously transmitted to the ground control system (GCS) (tablet PC, etc.) and artificial intelligence learning.

The ground control system (GCS) receives the video MPEG, separates the video and meta information, interprets the meta information on the GIS map, simulates the drone's camera direction and terrain location, and the video is stored at the location. express in real time.

The artificial intelligence learner 500 receives the MPEG data and separates the video (H.264) and meta information (separating the MPEG packet), and the video is converted into a learned artificial intelligence algorithm (to find people, tanks, etc.) The object is recognized, and the recognized object calculates the location coordinates of time based on the meta information and displays the drone camera video object location coordinates respectively.

The mixed reality monitor 600 detects an object with the tablet GIS through mixed reality artificial intelligence (MR), and displays it on the monitor (smart glasses) according to the tablet GIS coordinate information.

To explain this step by step, real-time camera gimbal 300 video drone flight meta information synchronization SW is implemented in one step. At this time, real-time video acquisition is synchronized, and real-time camera view angle gimbal attitude + position angle altitude meta information is synchronized.

Then, in the second step, the mission computer synchronized H.26x video + meta information MPEG frame is generated. At this time, an H.26x video + meta information container MPEG frame is generated, and encryption encoding and decoding operations are performed on the MPEG by applying an encryption algorithm.

Next, in the third step, port the dual service IP relay server. At this time, the video Dual IP relay server is ported (based on Linux Ubuntu), and the smart controller includes MPEG video ground control system (GCS) and artificial intelligence learner.

And in step 4, an artificial intelligence learner coordinate acquisition algorithm for object identification for surveillance and reconnaissance is performed, and object recognition pre-learning deep learning is performed to perform artificial intelligence surveillance and reconnaissance object recognition (real-time battlefield object identification and location estimation).

In the next 5 steps, create video object identification mixed reality smart glasses content service. At this time, video object mixed reality (MR) content for smart glasses is created (including UI).

Next, in the sixth step, geographic information system (GIS)-based AI object information monitoring is implemented. Accordingly, a video mixed reality content service is provided.

8 shows an example of executing a tank object recognition learning engine in an artificial intelligence learner and displaying it on a mixed reality monitor.

In the example of executing the tank object recognition learning engine in the artificial intelligence learner, object recognition is performed for three tanks, and the frame meta information coordinates are calculated in the lower left corner.

9 and 10 are diagrams showing an example of executing a human object recognition learning engine in an artificial intelligence learner and displaying it on a mixed reality monitor.

In the example of executing the human object recognition learning engine in the artificial intelligence learner, object recognition for three people is performed in FIG. 9 and object recognition for one person is performed in FIG. 10, and the recognized object is displayed as a red circle. An example is shown.

Although the present invention has been described with the above examples, the present invention is not necessarily limited to these examples, and may be variously modified and implemented without departing from the technical spirit of the present invention. Therefore, the examples disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these examples. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

The present invention is a camera video object location coordinate acquisition system using a drone, and specifically, a global positioning system (GPS) (100) configured to use a global positioning system (GNSS) receiver, receive RTK / DGPS correction information, and perform GPS correction. ), an on-board flight control computer 200 that receives gimbal information and video information (H.264 video) from the camera gimbal 300 and transmits them using an image processing algorithm that is frame-adjusted, an on-board flight control computer 200 and an image A smart controller 400 composed of a communication unit and a dual IP relay unit, an artificial intelligence learner 500 that receives MPEG data from the smart controller and provides drone camera video object location coordinates to a mixed reality monitor, and an artificial intelligence learner It is about a drone equipped with an on-board flight control computer and a drone camera video object positioning system using the same, characterized in that it is configured to include a mixed-room monitor 600 displaying the coordinates of the drone camera video object location provided in (500), so the coordinates are obtained. Since this service can be performed stably, there is industrial applicability.

Claims (2)

A global positioning system (GPS) 100 configured to use a global positioning system (GNSS) receiver, receive RTK/DGPS correction information, and perform GPS correction; An ESC (Electronic Speed Controller) power control unit for driving a motor, a video communication unit for transmitting video data, a control communication unit for receiving control signals, a flight control unit for controlling flight by control of the control communication unit, Receives precise position/altitude/time information from the attitude/altitude/angle sensor unit and the global positioning system (GPS) 100, and gimbal information and video information (H.264 video) from the camera gimbal 300, Performs video communication and control communication with the smart controller 400, acquires coordinates for the camera gimbal 300, synchronizes video and meta information, and transmits video-synchronized MPEG based on real-time camera video Meta information is created in the frame, and it is an MPEG structure in which a container structure is inserted into the time synchronization frame. If the frame size is large, the on-board flight control computer 200 transmits the frame using an image processing algorithm that adjusts the frame so that the frame does not exceed the set frequency band width by automatically adjusting the frame; The controller is an on-board computer driven by Linux operating system, includes a control controller and a ground control system, and provides a control communication unit for controlling the on-board flight control computer 200 and video communication with the on-board flight control computer 200. A smart controller 400 comprising a video communication unit and a dual IP relay unit to perform; MPEG data is received from the smart controller 400, video and meta information are separated, and the video recognizes an object with a learned artificial intelligence algorithm, and the recognized object is positioned in time based on pre-learned meta information. An artificial intelligence learner 500 that calculates coordinates and provides drone camera video object position coordinates to the mixed reality monitor 600; and A drone equipped with an on-board flight control computer, characterized in that it is configured to include a mixed room monitor 600 comprising smart glasses or a monitor and displaying the drone camera video object position coordinates provided by the artificial intelligence learner 500 A camera video object location coordinate acquisition system using . According to claim 1, The smart controller 400 is a smart controller operated by a drone controller, It includes a controller on-board computer driven by the Linux operating system, a control controller and a ground control system (GCS), a control communication unit for controlling the on-board flight control computer 200, and video communication with the on-board flight control computer 200. It is composed of a video communication unit and a dual IP relay unit to perform, The on-board flight control computer 200 receives the MPEG video and the MPEG streaming is simultaneously transmitted to the artificial intelligence processor configured in the ground control system (GCS) and the mixed reality monitor 600, The ground control system (GCS) receives the video MPEG and separates the video and meta information to simulate the drone's camera direction and terrain position on the GIS map, and the video is configured to be displayed in real time at the location. A camera video object position coordinate acquisition system using a drone equipped with an on-board flight control computer.

Images