WO2025135608A1 - Drone and control method therefor - Google Patents

Abstract

The embodiments of the present invention provide a drone and a control method therefor, in which, by converting a motor signal to a motor control signal in a general situation, the drone is controlled so as to fly along a specific flight trajectory, a specific situation is recognized by measuring attitude on the basis of sensor data, the difference between an input value and an estimated input value is estimated as estimated disturbance, the input value being outputted from a policy-based reinforcement learning network which has undergone reinforcement learning so as to output the input value by receiving a desired current state, and the estimated input value being outputted from a Bayesian network trained so as to output the estimated input value by receiving the current state and the next state, and the drone is controlled so as to fly while maintaining attitude by adjusting the motor control signal in the specific situation by means of a control signal in which disturbance that can actually occur is canceled by subtracting the estimated disturbance from the input value.

Description

Drone and method of controlling the drone

The present embodiments relate to a drone for controlling an aircraft in a specific situation and a method for controlling the drone.

For example, in order to produce brine lithium in Argentina, the target concentration must be maintained for each pond, and as a management task to maintain the concentration, there are tasks such as collecting water and measuring water depth for each brine pond. Drones can be used as one way to increase the efficiency of this task, but the lithium salt lake is located in a windy place to maximize evaporation.

Drones are a typical under-actuated system and are vulnerable to external disturbances such as wind. In addition, the most typically used PID controllers are vulnerable to gusts because their hyperparameters are tuned for path following.

Most drones pose a risk of drifting or tilting under certain conditions, such as when subjected to gusts of wind, which can lead to a crash or loss of control.

The present embodiments can provide a drone and a method for controlling the drone, which stably control the drone with low delay in specific situations such as a gust of wind.

Embodiments of the present invention provide a drone and a method for controlling the drone to fly while maintaining the attitude by adjusting the motor control signal in a specific situation as a control signal by converting a motor signal into a motor control signal in a general situation and controlling the drone to fly along a specific flight trajectory, measuring an attitude based on sensor data to recognize a specific situation, and estimating the difference between an input value output from a policy-based reinforcement learning network that receives a desired current state as an input and outputs an input value and an estimated input value output from a Bayesian network that is trained to receive a current state and a next state as inputs and output an estimated input value, and controlling the drone to fly while maintaining the attitude by adjusting the motor control signal in a specific situation as a control signal by subtracting the estimated disturbance from the input value and offsetting the disturbance that may actually occur.

In one aspect, the present embodiments provide a drone including a drone sensor unit that provides sensor data, a drone flight unit that provides a motor signal, and a drone control unit that controls the drone to fly along a specific flight trajectory by converting the motor signal into a motor control signal in a general situation, measures an attitude based on the sensor data to recognize a specific situation, and estimates the difference between an input value output from a policy-based reinforcement learning network that receives a desired current state as an input and outputs an input value and an estimated input value output from a Bayesian network that is trained to receive a current state and a next state as an input and output an estimated input value, and controls the drone to fly while maintaining the attitude by adjusting the motor control signal in a specific situation as a control signal that offsets a disturbance that may actually occur by subtracting the estimated disturbance from the input value.

In another aspect, the present embodiments can provide a method for controlling a drone, including a first step for controlling the drone to fly along a specific flight trajectory by converting a motor signal into a motor control signal in a general situation, and a second step for controlling the drone to fly while maintaining the attitude by measuring the attitude based on sensor data, recognizing a specific situation, estimating the difference between an input value output from a policy-based reinforcement learning network that receives a desired current state as an input and outputs an input value, and an estimated input value output from a Bayesian network that is trained to receive a next state as an input and output an estimated input value, and subtracting the estimated disturbance from the input value to cancel out a disturbance that may actually occur.

According to the drone and the method for controlling the drone according to the present embodiments, the drone can be stably controlled with low delay in specific situations such as a gust of wind.

Figure 1 is a perspective view of a drone to which embodiments are applied.

Figure 2 is a configuration diagram of a drone according to one embodiment.

Figure 3 is a configuration diagram of an example of the drone control unit of Figure 2.

Figure 4 is a relationship diagram between the first control unit and the drone drive unit of Figures 2 and 3.

Figure 5 is a relationship diagram between the second control unit and the drone drive unit of Figures 2 and 3.

Figure 6 is a conceptual diagram of a policy-based reinforcement learning network used in the second control unit (154) of Figure 5.

Figure 7 illustrates an example of the relationship between the policy-based reinforcement learning network of the second control unit of Figure 5 and the Bayesian network.

Figure 8 illustrates another example of the relationship between the policy-based reinforcement learning network of the second control unit of Figure 5 and the Bayesian network.

Figure 9 is an example of application of the second control unit and drone drive unit of Figures 2 and 3 in the learning stage.

Fig. 10 is an application example of the second control unit and the drone drive unit of Figs. 2 and 3 in the use stage.

Figure 11 is a conceptual diagram of the drone driving unit of Figures 2 and 3.

Figure 12 illustrates an example of a simulator of the drone of Figures 2 and 3.

Figure 13 is a flowchart of a drone control method according to another embodiment.

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, it should be noted that the same components are given the same numerals as much as possible even if they are shown in different drawings. In addition, when describing the present disclosure, if it is determined that a specific description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description thereof will be omitted.

Also, in describing components of the present disclosure, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by the terms. When it is described that a component is "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but another component may also be "connected," "coupled," or "connected" between each component.

In embodiments of the present invention, the form of energy may be electrical energy, thermal energy, light energy, etc. Hereinafter, embodiments of the present invention will mainly describe a case where the form of energy is electrical energy, but the form of energy is not limited thereto.

Hereinafter, an energy management device, a multi-agent based battery cooling plate design device, and a multi-agent based battery cooling plate design device according to embodiments of the present invention will be described with reference to related drawings.

The embodiments are described in detail with reference to the drawings below.

Figure 1 is a front view of a drone according to one embodiment.

Referring to FIG. 1, a drone (100) according to one embodiment can fly using a propeller (110) and an electric motor (120). The electric motor (120) is used to convert electric power into electrical energy to rotate the propeller (110). This rotational motion pushes out air, creating a force that propels the drone (100) upward. The propeller (110) helps to move the air using this rotational motion, and the drone (100) can fly using this principle.

Drones (100) are equipped with various sensors, cameras, GPS, communication systems, etc. and are used for flight control and data collection.

The drone (100) has multiple (e.g., four in FIG. 1) propellers (110), and each propeller (110) can be controlled to perform ascent, descent, forward, backward, left and right movement, rotation, etc. To this end, the direction and height of the drone (100) are controlled by controlling the rotation speed of each propeller (110).

The drone (100) has a built-in system for controlling flight. This system controls the speed of each electric motor (120) and adjusts the attitude to move in the desired direction according to the command entered by the user.

A drone (100) according to one embodiment stably controls attitude and speed without drifting, tilting, crashing or losing control with low delay in specific situations such as gusts.

Figure 2 is a configuration diagram of the drone of Figure 1.

Referring to FIG. 2, a drone (100) according to one embodiment includes a drone sensor unit (130) that provides sensor data, a drone flight unit (140) that provides motor signals, and a drone control unit that converts the motor signals into motor control signals and controls the drone to fly along a specific flight trajectory.

The drone control unit (150) controls the drone to fly along a specific flight trajectory by converting a motor signal into a motor control signal in a general situation, recognizes a specific situation by measuring an attitude based on sensor data, and estimates the difference between an input value output from a policy-based reinforcement learning network that receives a desired current state as input and outputs an input value and an estimated input value output from a Bayesian network that is trained to receive a current state and a next state as inputs and outputs an estimated input value as an estimated disturbance, and controls the drone to fly while maintaining an attitude by adjusting the motor control signal in a specific situation as a control signal that offsets disturbances that may actually occur by subtracting the estimated disturbance from the input value.

Hereinafter, a specific situation is exemplified by a gust situation as described above, but is not limited thereto, and generally includes all cases other than a general situation in which a drone (100) may drift or tilt, crash, or lose control. That is, this specification defines two situations in which a drone (100) may perform a general control operation in a first situation such as a general situation, and perform a special control operation in a second situation such as a specific situation. As described below, the drone (100) may perform a software control operation or a hardware control operation in the general situation and the special situation.

The drone sensor unit (130) includes a gyro sensor (132) and an acceleration sensor (134), and the sensor data may be, but is not limited to, angular velocity sensed by the gyro sensor (132) and acceleration sensed by the acceleration sensor (134).

The gyro sensor (132) can measure angular velocity. An example of the gyro sensor (132) may be a gyroscope that measures angular velocity. The acceleration sensor (134) is a sensor that measures acceleration. The acceleration sensor (132) can measure acceleration in the x-axis, y-axis, and z-axis directions, for example.

The drone flight unit (140) can convert a motor control signal adjusted by the drone control unit (150) into a specific PWM signal that adjusts the motor speed and direction.

The drone flight unit (140) converts the motor control signal adjusted by the drone control unit (150) into a corresponding PWM signal that adjusts the motor speed and direction. The control logic of the drone flight unit (140) uses aerodynamic and flight dynamics models to compensate for wind disturbance and maintain the desired flight trajectory.

The drone flight unit (140) provides the converted PWM signal to the electric motor (120) and controls the electric motor (120), thereby inducing the flight of the drone (200).

The drone control unit (150) includes a microprocessor, and the drone (100) may additionally include a memory (not shown). The memory stores various sensor data and various programs. The memory may be a volatile memory (e.g. SRAM, DRAM) or a nonvolatile memory (e.g. NAND Flash).

The drone (100) according to the above-described embodiment learns the model dynamics of the commercial drone to be used through a Bayesian network by utilizing a simulator of the commercial drone to be used.

Bayesian Network is one of the probabilistic graphical models, used to express probabilistic dependencies between variables. The parameters of the Bayesian Network are the conditional probabilities of each variable that constitutes the network, and the relationships between variables are modeled through this. For example, the method of estimating parameters in the learning process of the Bayesian Network utilizes a method called Maximum Likelihood Estimation (MLE). This is a method of finding the most probable parameters from data.

A trained Bayesian network uses learned parameters to calculate probabilities when making predictions on new data.

Bayesian networks in a broad sense exist in various types, including Bayesian networks, which represent conditional dependencies between general variables as a graph, Markov networks, or Markov Random Fields, which are expressed as graph structures consisting of nodes and edges, where edges represent conditional independence, Dynamic Bayesian Networks, which are used to model dependencies between variables in situations that change over time, and Structural Bayesian Networks, which automatically learn structures from data to create networks.

In addition, the Bayesian network used may be a sparse Bayesian network that calculates the signal-to-noise ratio for the parameter distribution at each step during learning, and if the signal-to-noise ratio value is small, it is considered a parameter with little influence on the result value, so it may be pruned to make it lightweight.

For example, the drone (100) according to the above-described embodiment considers the difference between the system input estimate output from the learned sparse Bayesian network and the intended system input as a disturbance and offsets it in the next input. The sparse Bayesian network compresses the network during the learning process to reduce the capacity and inference speed. The drone (100) according to the above-described embodiment can prevent crashes due to gusts when using the drone in the relatively windy airspace over Argentina.

In certain situations, such as a gust of wind, when a drone crashes, it can be controlled to hover (suspend in the air) using reinforcement learning. However, due to the nature of reinforcement learning, it is vulnerable to disturbances not experienced in the learning environment, so a method to improve the robustness of the policy-based reinforcement learning network is needed to apply it in a real environment. To improve the robustness of reinforcement learning, a disturbance observer can be applied after estimating the model dynamics through a general artificial neural network. However, general artificial neural networks have limited use in performance-limited environments such as embedded systems due to their unique large capacity and relatively slow inference speed.

The drone (100) according to the above-described embodiment can increase the robustness of a policy-based reinforcement learning network and reduce network capacity and inference speed by utilizing a Bayesian network, for example, a sparse Bayesian network and a disturbance observer.

Figure 3 is a configuration diagram of an example of the drone control unit of Figure 2.

Referring to FIG. 3, the drone control unit (150) may include a first control unit (152) that controls the drone to fly along a specific flight trajectory by converting a motor signal into a motor control signal in a general situation, and a second control unit (154) that controls the drone to fly while maintaining an attitude by adjusting the motor control signal in a specific situation with a control signal that offsets a disturbance that may actually occur by subtracting an estimated disturbance, which is a difference between an input value output from a policy-based reinforcement learning network and an estimated input value output from a Bayesian network, from the input value.

The first control unit (152) and the second control unit (154) may be implemented as separate hardware, may be implemented as software, or may be implemented as one hardware and the other software.

Fig. 4 is a relationship diagram between the first control unit and the drone drive unit of Figs. 2 and 3. Fig. 5 is a relationship diagram between the second control unit and the drone drive unit of Figs. 2 and 3.

Referring to FIG. 4, the first control unit (152) converts a motor signal into a motor control signal in a general situation and controls the flight to a specific flight trajectory. The first control unit (152) receives a desired current state (Sr) in a general situation and outputs a motor control signal (u) to the drone drive unit (140). The drone drive unit (140) drives the electric motor with the received motor control signal (u).

)을 출력하도록 학습된 베이지안 네트워크(158)로부터 출력된 추정 입력값(

)의 차이(

-u)를 추정 외란(

)으로 추정하고 입력값(u)에 추정 외란(

)을 빼서 실제 발생할 수 있는 외란(d)을 상쇄한 실제 제어신호(u_real)로 특정 상황에서 모터 제어 신호를 조정하여 자세를 유지하면서 비행하도록 제어할 수 있다. Referring to FIG. 5, the second control unit (154) receives the difference value (e s ), that is, the error value (e s ) between the desired current state (s _r ) and the output value ( _s ) of the drone drive unit (140), and outputs a control signal value (u) from a policy-based reinforcement learning network (policy network, 156) that has undergone reinforcement learning, and receives the control signal (u) and the current state (s _r ) and the next state (s) as inputs to output an estimated input value (

) is output from a Bayesian network (158) trained to output the estimated input value (

) difference(

-u) is estimated as a disturbance (

) is estimated and the input value (u) is estimated as a disturbance (

) can be used to control flight while maintaining attitude by adjusting the motor control signal in a specific situation with the real control signal (u _real ) that compensates for the disturbance (d) that may actually occur.

)이 200RPM이라면 드론 구동부(140)에 입력되는 실제 제어신호값은 3800RPM이 될 수 있다. For a simple example, if the control signal of the drone drive unit (140) is an RPM value, it is assumed that the control signal (u) output from the policy-based reinforcement learning network (policy network, 156) is 4000 RPM. At this time, the estimated disturbance (

) is 200 RPM, the actual control signal value input to the drone drive unit (140) can be 3800 RPM.

Unlike the general method of calculating the inverse function of a dynamic model to estimate input values, the Bayesian network (158) is trained to estimate input values through supervised learning using training data pairs in a buffer of reinforcement learning, as described below. The second control unit (154) can implement a disturbance observer in the policy-based reinforcement learning network (154) without physical modeling by replacing the artificial neural network inverse model dynamics described above with the trained Bayesian network (158).

Below, the general reinforcement learning and the policy-based reinforcement learning network and Bayesian network (158) used in the second control unit (154) of the drone (100) according to the above-described embodiment are described in detail with reference to FIGS. 6 to 10.

Figure 6 is a conceptual diagram of a policy-based reinforcement learning network used in the second control unit (154) of Figure 5.

Referring to FIG. 6, the policy-based reinforcement learning network (158) used in the second control unit (154) is a system in which an environment (156a) and an agent (156b) exchange states/rewards and actions. The environment (156a) may include various forms of simulation environments or simulators as described below. In this specification, the environment (156a) and simulator are used interchangeably, but are not limited thereto.

The environment (simulator) (156a) can input the current state (Current state, S _t ) and action (action, A _t ) and output the next state (Next State, S _t+1 ) and reward (reward, R _t+1 ). In this process, the agent (156b) learns a policy (policy, π(a/s): S ==> R∈{0,1}).

Figure 7 illustrates an example of the relationship between the policy-based reinforcement learning network of the second control unit of Figure 5 and the Bayesian network.

Referring to Fig. 7, in a policy-based reinforcement learning network (156) used in the second control unit (154), an agent (156b) repeatedly performs the next action based on the state and reward generated in the environment (156a), and updates the action decision so that the sum of accumulated rewards is maximized.

, s_t+1)을 버퍼(159)에 저장한 후, 미니 배치 크기(M)만큼 샘플링하여 학습될 수 있다. The Bayesian network (158) used in the second control unit (154) is a learning data pair (s _t , s t ) generated from policy-based reinforcement learning of the policy-based reinforcement learning network (156).

, s _t+1 ) can be stored in a buffer (159) and then learned by sampling as much as the mini batch size (M).

Mathematical expression 1 is an example of the objective formula of a Bayesian network (158).

_, θ 및 M은 각각 베이지안 네트워크의 로스 함수(Loss function), 베이지안 네트워크의 훈련 매개변수, 미니 배치 크기을 나타낸다. r_sbl은 스케링값, D_KL은 후술하는 바와 같이 KL 발산 항을 나타낸다.

을 학습하기 위한 데이터는 강화학습의 버퍼(159)에서 미니 배치 크기(M)만큼 샘플링된다. In mathematical expression 1,

_, θ, and M represent the loss function of the Bayesian network, the training parameters of the Bayesian network, and the mini-batch size, respectively. r _sbl represents a scaling value, and D _KL represents a KL divergence term as described below.

Data for learning is sampled from the buffer (159) of reinforcement learning in amounts equal to the mini-batch size (M).

로 표현할 수 있다. For example, a mini-batch with a mini-batch size (M)

can be expressed as

)와 정책 기반 강화학습 네트워크(156)에서 출력된 추정 행동(

) 사이의 평균 제곱 오차 (mean square error (MSE))를 최소화하도록 설계한다. 수학식 1의 두번째 항목인 KL 발산 항은 사후 분포가 사전 분포에 가깝게 유지되도록 장려하며, 영향력이 적은 네트워크 매개변수를 제거하기 위해 각 단계에서 신호 대 잡음비(Q(W))를 계산한다.The first item in Equation 1 is the behavioral data (

) and the estimated action output from the policy-based reinforcement learning network (156).

) is designed to minimize the mean square error (MSE) between the two networks. The second term in Equation 1, the KL divergence term, encourages the posterior distribution to remain close to the prior distribution, and the signal-to-noise ratio (Q(W)) is calculated at each step to remove less influential network parameters.

Figure 8 illustrates another example of the relationship between the policy-based reinforcement learning network of the second control unit of Figure 5 and the Bayesian network.

Referring to FIG. 8, the Bayesian network (158) used in the second control unit (154) calculates a signal-to-noise ratio for parameter distribution at each step during learning, and if the corresponding signal-to-noise ratio value (SNR(Q(W))) is small, it is considered a parameter that has little influence on the result value, so it can be pruned to make it lightweight. The network that is made lightweight by making pruning in the Bayesian network (158) can be the sparse Bayesian network (158b) illustrated in FIG. 8. Hereinafter, the Bayesian network (158) can be a Bayesian network (158) that is not made lightweight, or can be a sparse Bayesian network (158b). Hereinafter, the sparse Bayesian network (158b) will be described as an example of a Bayesian network (158).

The learned policy-based reinforcement learning network (156) and the sparse Bayesian network (158) are designed as a structure of a disturbance observer consisting of a second control unit (154) that observes disturbances and a drone driving unit (140).

Figure 9 is an example of application of the second control unit and drone drive unit of Figures 2 and 3 in the learning stage.

)을 출력하면 시뮬레이터 환경 또는 시뮬레이터(156a)가 이 입력값(

)을 입력받아 다음 상태(S_t+1)를 생성한다. 희소 베이지안 네트워크(158)은 현재 상태와 다음 상태를 입력받아 추정 입력값을 출력하고, 비교기에서 입력값과 추정 입력값의 차이값을 계산하고 이 과정이 반복적으로 수행되는 수학식 1에 따라 축적된 차이값이 최소화되도록 회소 베이지안 네트워크(158)이 학습된다. Referring to Figures 8 and 9, in the policy-based reinforcement learning network (156) used in the second control unit (154), an agent (156b) implemented as an artificial neural network receives a desired current state (s _t ) and inputs an input value (

) outputs the simulator environment or simulator (156a) to this input value (

) is input and generates the next state (S _t+1 ). The sparse Bayesian network (158) receives the current state and the next state, outputs an estimated input value, calculates the difference between the input value and the estimated input value in the comparator, and the sparse Bayesian network (158) is trained so that the accumulated difference value is minimized according to mathematical expression 1 in which this process is performed repeatedly.

, s_t+1)을, 미니 배치 크기(M)만큼 샘플링하여 학습될 수 있다. As described above, the Bayesian network (158) used in the second control unit (154) is a learning data pair (s _t , s t ) generated from policy-based reinforcement learning of the policy-based reinforcement learning network (156).

, s _t+1 ) can be learned by sampling as much as the mini-batch size (M).

Fig. 10 is an application example of the second control unit and the drone drive unit of Figs. 2 and 3 in the use stage.

Referring to FIG. 10, the learned sparse Bayesian network (158) estimates a system input paired with the current state (s _t ) and the next state (s _t+1 ) measured from the drone actuator (140) of the actual drone (100).

)과 정책 기반 강화학습 네트워크(156)에서 얻은 출력(

)의 차이는 알려지지 않은 외란(d)과 모델 불확실성에 대한 영향으로 간주된다. 결과적으로, 다음 제어 입력에 해당 추정치를 반영함으로써 의도하지 않은 외란(d)을 상쇄할 수 있다.The output of the sparse Bayesian network (158)

) and the output obtained from the policy-based reinforcement learning network (156) (

) is considered as an effect of unknown disturbance (d) and model uncertainty. Consequently, the unintended disturbance (d) can be offset by incorporating its estimate into the next control input.

)을 출력하도록 강화학습된 정책 기반 강화학습 네트워크(policy network)로부터 출력되는 입력값(

)과 현재 상태(s_t)와 다음 상태(S_t+1)를 입력받아 추정 입력값(

)을 출력하도록 학습된 베이지안 네트워크로부터 출력된 추정 입력값(

)의 차이(

-

)를 추정 외란(

)으로 추정하고 입력값(

)에 추정 외란(

)을 빼서 실제 발생할 수 있는 외란(d)을 상쇄한 제어신호로 특정 상황에서 모터 제어 신호를 조정하여 자세를 유지하면서 비행하도록 제어할 수 있다. In other words, the second control unit (154) receives the desired current state (s _t ) and inputs the input value (

) is the input value output from a policy-based reinforcement learning network that has been reinforced to output a policy (

) and the current state (s _t ) and the next state (S _t+1 ) are input to estimate the input value (

) from the Bayesian network trained to output the estimated input values (

) difference(

-

) to estimate the disturbance (

) and the input value (

) estimated external disturbance(

) can be controlled to maintain attitude and fly by adjusting the motor control signal in a specific situation as a control signal that offsets the disturbance (d) that may actually occur.

Figure 11 is a conceptual diagram of the drone driving unit of Figures 2 and 3.

)를 출력하도록 학습될 수 있다. Referring to Fig. 11, the policy-based reinforcement learning network (156) of the second control unit, when applied to a reinforcement learning problem, the state S is a three-dimensional position (Px, Py, Pz) and a three-dimensional rotation (roll, pitch, yaw), the action A is the thrust of each motor (F1, F2, F3, F4), and the sparse Bayesian network (158) receives the current state (s _t ) and the next state (s _t+1 ) as inputs and the current input (which is the thrust of each motor)

) can be learned to output.

Figure 12 illustrates an example of a simulator of the drone of Figures 2 and 3.

Referring to Fig. 12, as a simulator of a commercial drone, reinforcement learning is performed to output the input of the drone (100) based on the state data coming from the simulator during learning. The data is stored in a buffer (159) and used later when learning the model with a sparse Bayesian network (158).

According to the drone (100) according to the other embodiment described above, it is possible to stably control with low delay in specific situations such as a gust of wind.

Figure 13 is a flowchart of a drone control method according to another embodiment.

Referring to FIG. 13, a drone control method (200) according to another embodiment includes a first step (S210) for controlling the drone to fly along a specific flight trajectory by converting a motor signal into a motor control signal in a general situation, and a second step (S210) for controlling the drone to fly while maintaining the attitude by adjusting the motor control signal in a specific situation by measuring the attitude based on sensor data, recognizing a specific situation, estimating the difference between an input value output from a policy-based reinforcement learning network that receives a desired current state as an input and outputs an input value, and an estimated input value output from a Bayesian network that is trained to receive a next state as an input and output an estimated input value, and subtracting the estimated disturbance from the input value to offset a disturbance that may actually occur.

In the second step (S220), the adjusted motor control signal can be converted into a specific PWM signal that adjusts the motor speed and direction.

As described above with reference to FIGS. 4 and 5, the second step (S220) controls flight while maintaining attitude by adjusting the motor control signal in a specific situation by subtracting the estimated disturbance, which is the difference between the input value output from the policy-based reinforcement learning network and the estimated input value output from the Bayesian network, from the input value and canceling out the disturbance that may actually occur.

)을 출력하도록 강화학습된 정책 기반 강화학습 네트워크(policy network)로부터 출력되는 입력값(

)과 제t상태(s_t)와 제t상태(S_t)의 다음 상태인 제t+1상태(S_t+1)를 입력받아 추정 입력값(

)을 출력하도록 학습된 베이지안 네트워크로부터 출력된 추정 입력값(

)의 차이(

-

)를 추정 외란(

)으로 추정하고 입력값(

)에 추정 외란(

)을 빼서 실제 발생할 수 있는 외란(d)을 상쇄한 제어신호로 특정 상황에서 모터 제어 신호를 조정하여 자세를 유지하면서 비행하도록 제어할 수 있다. Step 2 (S220) receives the desired t-state (s _t ) and inputs the input value (

) is the input value output from a policy-based reinforcement learning network that has been reinforced to output a policy (

) and the t-th state (s _t ) and the t+1 state (S _{t +1} ), which is the next state of the t-th state (S t ), are input to estimate the input values (

) from the Bayesian network trained to output the estimated input values (

) difference(

-

) to estimate the disturbance (

) and the input value (

) estimated external disturbance(

) can be controlled to maintain attitude and fly by adjusting the motor control signal in a specific situation as a control signal that offsets the disturbance (d) that may actually occur.

, s_t+1)을 버퍼에 저장한 후, 미니 배치 크기만큼 샘플링하여 학습될 수 있다. As described above with reference to FIGS. 6 and 7, the Bayesian network used in the second step (S220) is a learning data pair (s _t ,

, s _t+1 ) can be stored in a buffer and then learned by sampling as much as the mini-batch size.

As described above with reference to FIGS. 8 and 9, the Bayesian network used in the second step (S220) calculates a signal-to-noise ratio for parameter distribution at each step during learning, and if the signal-to-noise ratio value is small, it is considered a parameter with little influence on the result value, so pruning can be performed to make it lighter.

)을 출력하도록 강화학습된 정책 기반 강화학습 네트워크(policy network)로부터 출력되는 입력값(

)과 제t상태(s_t)와 제t상태(S_t)의 다음 상태인 제t+1상태(S_t+1)를 입력받아 추정 입력값(

)을 출력하도록 학습된 베이지안 네트워크로부터 출력된 추정 입력값(

)의 차이(

-

)를 추정 외란(

)으로 추정하고 입력값(

)에 추정 외란(

)을 빼서 실제 발생할 수 있는 외란(d)을 상쇄한 제어신호로 특정 상황에서 모터 제어 신호를 조정하여 자세를 유지하면서 비행하도록 제어할 수 있다. As described above with reference to Fig. 10, the second step (S220) receives a desired t-state (s _t ) and inputs the input value (

) is the input value output from a policy-based reinforcement learning network that has been reinforced to output a policy (

) and the t-th state (s _t ) and the t+1 state (S _{t +1} ), which is the next state of the t-th state (S t ), are input to estimate the input values (

) from the Bayesian network trained to output the estimated input values (

) difference(

-

) to estimate the disturbance (

) and the input value (

) estimated external disturbance(

) can be controlled to maintain attitude and fly by adjusting the motor control signal in a specific situation as a control signal that offsets the disturbance (d) that may actually occur.

)를 출력하도록 학습될 수 있다. As described above with reference to FIG. 11, the policy-based reinforcement learning network of the second stage (S220), when applied to a reinforcement learning problem, the state S is a three-dimensional position (Px, Py, Pz) and a three-dimensional rotation (roll, pitch, yaw), the action A is the thrust of each motor (F1, F2, F3, F4), and the sparse Bayesian network receives the current state (s _t ) and the next state (s _t+1 ) as inputs and the current input (

) can be learned to output.

According to the drone control method (200) according to the other embodiment described above, the drone can be stably controlled with low delay in specific situations such as a gust of wind.

Although the drone and the control method thereof according to the embodiments have been described with reference to the above drawings, the present invention is not limited thereto.

The drone (100) described above may be implemented by a computing device including at least some of a processor, a memory, a user input device, and a presentation device. The memory is a medium that stores computer-readable software, applications, program modules, routines, instructions, and/or data, etc. that are coded to perform a specific task when executed by the processor. The processor can read and execute computer-readable software, applications, program modules, routines, instructions, and/or data stored in the memory. The user input device may be a means for allowing a user to input a command to cause the processor to perform a specific task or input data necessary for executing a specific task. The user input device may include a physical or virtual keyboard or keypad, key buttons, a mouse, a joystick, a trackball, a touch-sensitive input means, or a microphone. The presentation device may include a display, a printer, a speaker, or a vibration device.

Computing devices may include a variety of devices, such as smartphones, tablets, laptops, desktops, servers, and clients. A computing device may be a single, stand-alone device, or it may include multiple computing devices operating in a distributed environment with multiple computing devices cooperating with each other via a communications network.

In addition, the drone (100) described above may be executed by a computing device having a processor and a memory storing computer-readable software, applications, program modules, routines, instructions, and/or data structures coded to perform the drone control method (200) described above when executed by the processor.

The above-described drone control method (200) can be implemented through various means. For example, the above-described drone control method (200) can be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the drone control method (200) described above can be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, or microprocessors.

For example, the above-described drone control method (200) according to the embodiments may be implemented using an artificial intelligence semiconductor device in which neurons and synapses of a deep neural network are implemented using semiconductor elements. At this time, the semiconductor elements may be currently used semiconductor elements, such as SRAM, DRAM, NAND, etc., or may be next-generation semiconductor elements, such as RRAM, STT MRAM, PRAM, etc., or may be a combination thereof.

When implementing the above-described drone control method (200) using an artificial intelligence semiconductor device, the results (weights) of learning a deep learning model using software can be transferred to synapse-mimicking elements arranged in an array, or learning can be performed in the artificial intelligence semiconductor device.

In the case of implementation by firmware or software, the above-described drone control method (200) may be implemented in the form of a device, procedure, or function that performs the functions or operations described above. The software code may be stored in a memory unit and driven by a processor. The memory unit may be located inside or outside the processor and may exchange data with the processor by various means already known.

Additionally, the terms "system", "processor", "controller", "component", "module", "interface", "model", or "unit" as described above may generally refer to a computer-related entity hardware, a combination of hardware and software, software, or software in execution. For example, the aforementioned components may be, but are not limited to, a process driven by a processor, a processor, a controller, a control processor, an object, a thread of execution, a program, and/or a computer. For example, an application running on a controller or a processor and the controller or the processor can both be components. One or more components may be within a process and/or thread of execution, and the components may be located on a single device (e.g., a system, a computing device, etc.) or distributed across two or more devices.

Meanwhile, a computer program stored in a computer storage medium is provided for performing the above-described drone control method (200). In addition, another embodiment provides a computer-readable storage medium having recorded thereon a program for realizing the above-described multi-agent based battery cooling plate design device.

The program recorded on the recording medium can be read, installed and executed by a computer, thereby executing the steps described above.

In this way, in order for the computer to read the program recorded on the recording medium and execute the functions implemented by the program, the above-mentioned program may include code coded in a computer language such as C, C++, JAVA, or machine language that can be read by the computer's processor (CPU) through the computer's device interface.

Such code may include functional code related to functions defining the aforementioned functions, and may also include control code related to execution procedures necessary for the computer's processor to execute the aforementioned functions according to a predetermined procedure.

Additionally, such code may further include memory reference related code regarding where in the internal or external memory of the computer the additional information or media required for the computer's processor to execute the aforementioned functions should be referenced.

In addition, if the computer's processor needs to communicate with another computer or server located remotely in order to execute the functions described above, the code may further include communication-related code regarding how the computer's processor should communicate with another computer or server located remotely using the computer's communication module, and what information or media should be sent and received during the communication.

As described above, a computer-readable recording medium having recorded thereon a program may include, for example, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical media storage device, etc., and may also include one implemented in the form of a carrier wave (e.g., transmission via the Internet).

Additionally, computer-readable recording media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner.

In addition, the functional program for implementing the present invention and the codes and code segments related thereto may be easily inferred or changed by programmers in the technical field to which the present invention belongs, taking into consideration the system environment of the computer that reads the recording medium and executes the program.

The above-described drone control method (200) may also be implemented in the form of a recording medium including computer-executable commands, such as an application or program module executed by a computer. The computer-readable medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include all computer storage media. The computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable commands, data structures, program modules, or other data.

The drone control method (200) described above can be executed by an application that is basically installed on the terminal (which may include a program included in a platform or operating system basically installed on the terminal), and can also be executed by an application (i.e., a program) that the user directly installs on the master terminal through an application providing server such as an application store server, an application, or a web server related to the corresponding service. In this sense, the multi-agent-based battery cooling plate design device described above can be implemented as an application (i.e., a program) that is basically installed on the terminal or directly installed by the user, and can be recorded on a computer-readable recording medium such as the terminal.

In the above, even though all the components constituting the embodiments of the present disclosure have been described as being combined as one or combined and operating, the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the purpose of the present disclosure, all of the components may be selectively combined and operating one or more times. In addition, although all of the components may be implemented as independent hardware, some or all of the components may be selectively combined and implemented as a computer program having a program module that performs some or all of the functions combined in one or more hardware. The codes and code segments constituting the computer program may be easily inferred by a person skilled in the art of the present disclosure. Such a computer program may be stored in a computer-readable storage medium and read and executed by a computer, thereby implementing the embodiments of the present disclosure. The storage medium of the computer program may include a magnetic recording medium, an optical recording medium, etc.

In addition, the terms "include," "comprise," or "have" described above, unless otherwise specifically stated, mean that the corresponding component can be included, and therefore should be interpreted as including other components rather than excluding other components. All terms, including technical or scientific terms, unless otherwise defined, have the same meaning as commonly understood by a person of ordinary skill in the art to which this disclosure belongs. Commonly used terms, such as terms defined in a dictionary, should be interpreted as being consistent with the contextual meaning of the relevant technology, and shall not be interpreted in an ideal or overly formal sense, unless explicitly defined in this disclosure.

The above description is merely an illustrative description of the technical idea of the present disclosure, and those skilled in the art to which the present disclosure pertains may make various modifications and variations without departing from the essential characteristics of the present disclosure. Therefore, the embodiments disclosed in the present disclosure are not intended to limit the technical idea of the present disclosure but to explain it, and the scope of the technical idea of the present disclosure is not limited by these embodiments. The protection scope of the present disclosure should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present disclosure.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims the benefit of priority under 35 U.S.C. § 119(a) to Korean patent application No. 10-2023-0184113, filed December 18, 2023, the entire contents of which are incorporated herein by reference. In addition, this patent application claims priority for other countries than the United States for the same reasons, the entire contents of which are incorporated herein by reference.

Claims (14)

Drone sensor section that provides sensor data; A drone flight section providing motor signals; and A drone comprising a drone control unit that controls the drone to fly along a specific flight trajectory by converting the motor signal into the motor control signal in a general situation, and measures the attitude based on the sensor data to recognize a specific situation, and estimates the difference between the input value output from a policy-based reinforcement learning network trained to receive a desired current state as an input and output an input value and the estimated input value output from a Bayesian network trained to receive the current state and the next state as inputs, and controls the drone to fly while maintaining the attitude by adjusting the motor control signal in the specific situation as a control signal that offsets the disturbance that may actually occur by subtracting the estimated disturbance from the input value. In the first paragraph, A drone wherein the above drone flight section converts the adjusted motor control signal into a specific PWM signal that adjusts the motor speed and direction. In the first paragraph, The above drone control unit In the above general situation, a first control unit that converts the motor signal into the motor control signal and controls the flight to a specific flight trajectory; and A drone including a second control unit that controls the drone to fly while maintaining the attitude by adjusting the motor control signal in the specific situation with the control signal that offsets the disturbance that may actually occur by subtracting the estimated disturbance, which is the difference between the input value output from the policy-based reinforcement learning network and the estimated input value output from the Bayesian network, from the input value. In the third paragraph, The Bayesian network used in the second control unit above is a learning data pair (s _t ,

, s _t+1 ) are stored in the buffer, and then trained drones are sampled by the mini-batch size. In paragraph 4, The Bayesian network used in the second control unit calculates the signal-to-noise ratio for the parameter distribution at each step during learning, and if the signal-to-noise ratio value is small, it is considered a parameter with little influence on the result value, so pruning is performed to make the drone lightweight. In the third paragraph, The above second control unit receives the desired current state (s _t ) and inputs the input value (

) is output from the above policy-based reinforcement learning network (policy network) that has been reinforced to output the input value (

) and the current state (s _t ) and the next state (s _t+1 ) are input to estimate the input value (

) from the Bayesian network trained to output the estimated input value (

) difference(

-

) to estimate the disturbance (

) is estimated as the input value (

) in the above estimated disturbance (

) is controlled to fly while maintaining the attitude by adjusting the motor control signal in the specific situation with a control signal that offsets the disturbance (d) that may actually occur. In the third paragraph, The policy-based reinforcement learning network of the second control unit, when applied to a reinforcement learning problem, the state S is a three-dimensional position (Px, Py, Pz) and a three-dimensional rotation (roll, pitch, yaw), and the action A is the thrust of each motor (F1, F2, F3, F4). The above sparse Bayesian network receives the current state (s _t ) and the next state (s _t+1 ) as inputs and calculates the current input ( , which is the thrust of each motor)

) drone trained to output A first step for controlling the flight along a specific flight trajectory by converting a motor signal into the motor control signal in a general situation; and A method for controlling a drone, comprising a second step of controlling the drone to fly while maintaining the attitude by adjusting the motor control signal in the specific situation by estimating the difference between the input value output from a policy-based reinforcement learning network trained to receive a desired current state as input and output an input value and the estimated input value output from a Bayesian network trained to receive a next state as input and output an estimated input value as an estimated disturbance, and controlling the drone to fly while maintaining the attitude by subtracting the estimated disturbance from the input value and using a control signal that offsets disturbance that may actually occur. In Article 8, A method for controlling a drone by converting, in the second step, the adjusted motor control signal into a specific PWM signal for adjusting the motor speed and direction. In Article 8, The second step is a method for controlling a drone to fly while maintaining the attitude by adjusting the motor control signal in the specific situation with the control signal that offsets the disturbance that may actually occur by subtracting the estimated disturbance, which is the difference between the input value output from the policy-based reinforcement learning network and the estimated input value output from the Bayesian network, from the input value. In Article 10, The Bayesian network used in the second step above is a learning data pair (s _t ,

, s _t+1 ) in a buffer, and then a method for controlling a learned drone by sampling the size of the mini batch. In Article 11, The Bayesian network used in the second step calculates the signal-to-noise ratio for the parameter distribution at each step during learning, and if the signal-to-noise ratio value is small, it is considered a parameter with little influence on the result value, so pruning is performed to control a lightweight drone. In Article 10, The second step above receives the desired t-state (s _t ) and inputs the input value (

) is output from the above policy-based reinforcement learning network (policy network) that has been reinforced to output the input value (

) and the t-th state (s _t ) and the t+1-th state (s _{t +1} ), which is the next state of the t-th state (s t ), are input to estimate the input value (

) from the Bayesian network trained to output the estimated input value (

) difference(

-

) to estimate the disturbance (

) is estimated as the input value (

) in the above estimated disturbance (

) is subtracted from the drone to control the drone so that it can fly while maintaining the attitude by adjusting the motor control signal in the specific situation with a control signal that offsets the disturbance (d) that may actually occur. In Article 10, The above second stage policy-based reinforcement learning network, when applied to a reinforcement learning problem, the state S is a three-dimensional position (Px, Py, Pz) and a three-dimensional rotation (roll, pitch, yaw), and the action A is the thrust of each motor (F1, F2, F3, F4). The above sparse Bayesian network takes as input the current state (s _t ) and the next state (s _t+1 ) and outputs the current input ( , which is the thrust of each motor)

) A method for controlling a drone learned to output.

Images