WO2024197457A1 - Closed-loop adaptive brain‑machine interface decoding method and device based on reinforcement learning - Google Patents

Abstract

The present invention relates to the field of neural signal decoding, and in particular to a closed-loop adaptive brain‑machine interface decoding method and device based on reinforcement learning. The method and device comprise: pre-processing a collected neural signal of a test animal body; inputting the pre-processed neural signal into a decoder and mapping same into a specific action instruction, and the test animal body performing visual feedback on the executed specific action instruction to generate a new neural signal; and inputting the new neural signal into the decoder, enabling the decoder to generate a new action instruction to form a closed loop, and fusing a reinforcement learning theory and a brain-machine interface decoding paradigm. The brain and the decoder perform mutual learning and mutual adaptation, label signals are not needed, and this is also in line with the law of biological learning.

Description

A closed-loop adaptive brain-computer interface decoding method and device based on reinforcement learning Technical Field

The present invention relates to the field of neural signal decoding, and in particular to a closed-loop adaptive brain-computer interface decoding method and device based on reinforcement learning.

Background Art

A brain-machine interface decoder is a computational model designed to translate neural activity into human-understandable instructions for controlling external devices, such as prosthetic limbs, wheelchairs, computer games, etc. This technology is an important component of brain-machine interface (BMI). The brain-machine interface decoder works by converting neural signals from the brain into instructions for controlling external devices. These neural signals are usually recorded through implanted electrodes or using non-invasive neuroimaging techniques such as electroencephalography. By decoding and analyzing these signals, the computational model can recognize human intentions and behaviors and convert them into instructions that the machine can understand, thereby achieving the purpose of controlling external devices. Brain-machine interface decoders have applications in both clinical and laboratory settings. In the clinic, brain-machine interface decoders can help people who have lost motor skills regain the ability to control prosthetic limbs or wheelchairs. In the laboratory, it can help neuroscientists better understand how the brain controls behavior and movement, and provide basic research support for the development of new brain-machine interface technologies.

Closed-loop adaptive decoder (closed-loop decoder adaptation) is a brain-computer interface decoder paradigm that can make real-time adjustments based on feedback signals to improve the accuracy and reliability of the decoder. Compared with traditional open-loop decoders, it can better adapt to different neural signals and actual usage situations, thereby improving the performance and stability of brain-computer interface systems. Closed-loop adaptive decoders usually consist of two main parts: a decoder and a feedback control loop. The decoder is responsible for converting neural signals into control signals, while the feedback control loop provides real-time feedback signals for adjustment. This adjustment can be achieved in a variety of ways, such as adjusting decoder parameters, selecting different neural signal features, or adjusting the time delay of the feedback signal. Closed-loop adaptive decoders can help overcome many common problems, such as unstable signals, signal loss, nonlinear neural responses, etc. It can also better adapt to individual differences and Dynamically changing brain activities, thereby improving the accuracy and reliability of the decoder, so as to better control external devices or perform behavioral tasks. Closed-loop adaptive decoders have been widely used in brain-computer interface research and have been used in many application scenarios, such as prosthetic control, brain-controlled games, etc. In the future, with the continuous development of brain-computer interface technology, closed-loop adaptive decoders will continue to play an important role to help humans better control and interact with external devices, as well as provide a deeper understanding of neuroscience research. The current decoders are mainly divided into traditional direct decoding and new paradigm decoding using closed-loop adaptive decoders. Among them, the traditional decoder that directly decodes neural signals is mainly used for offline decoding and decoding offline data. The decoding method is mostly used for RNN networks, such as using long short-term memory networks (LSTM) as decoders to extract timing information from neural signals. Due to the rise of closed-loop adaptive decoders and the excellent results achieved, more and more studies have begun to pay attention to this emerging paradigm. For example, Wu, Zhang and others use this paradigm for decoding, in which the decoder module mainly uses methods such as Kalman filtering or support vector regression.

Traditional decoders that directly decode are often used in offline scenarios. Since the decoder only unilaterally learns the brain's neural signals, the learning efficiency is lower, and the decoding accuracy is also worse than that of the closed-loop adaptive decoder paradigm. In the closed-loop adaptive decoder paradigm, there are currently closed-loop adaptive decoders based on Kalman filter design and methods based on traditional machine learning such as support vector regression and random forests. However, these methods can learn more superficial content than artificial neural network methods and cannot learn deep features well. Therefore, in terms of effect, they are not as effective as deep learning methods.

Summary of the invention

The embodiment of the present invention provides a closed-loop adaptive brain-computer interface decoding method and device based on reinforcement learning, so as to design a decoder that can both quickly decode and ensure decoding accuracy for application in the brain-computer interface.

According to one embodiment of the present invention, a closed-loop adaptive brain-computer interface decoding method based on reinforcement learning is provided, comprising the following steps:

S101: preprocessing the collected neural signals of the test animal;

S102: inputting the preprocessed neural signal into the decoder to map it into a specific action instruction, and the test animal performs visual feedback on the executed specific action instruction to generate a new neural signal;

S103: The new neural signal is input to the decoder, and the decoder generates new action instructions, forming a closed loop that integrates reinforcement learning theory and brain-computer interface decoding paradigm.

Furthermore, the method further comprises:

S100: Collecting neural signals from the test animal.

Furthermore, in step S101, preprocessing includes amplification, filtering, and denoising.

Furthermore, the specific action instruction includes moving a cursor to a specified position. After the decoder generates a new action instruction, it assists the test animal in moving the cursor to a fixed position by setting constraints.

Further, the method includes a training part and a testing part;

In the training part, the monkey is trained to push the joystick to control the cursor to the specified position, and the monkey is rewarded at this time; the supervised learning neural network is used as a decoder in the training stage, and the target is used as a supervised learning label. Through training, a mapping relationship between neural signals and actions is established, and the trained model is saved;

During the testing phase, the monkey controlled the cursor movement through imagination. The monkey generated the intention to control the cursor movement by observing the position of the cursor on the screen. At this time, the decoder was replaced with a reinforcement learning algorithm to act as a decoder for decoding.

Furthermore, the reinforcement learning algorithm is DQN, in which the network that maps states to actions is directly loaded into the training phase using model parameters trained by a neural network with supervised learning.

Furthermore, in the training phase, the monkey used a joystick to control the cursor on the page to move. At this time, a circle of a different color appeared in each trial. After training, the monkey knew that moving the current cursor to the target position would be rewarded;

At the beginning of each trial, the neural signal is collected. After the signal is collected, it needs to be amplified and denoised. The processed data is 256 channels, and each channel has a half-precision floating-point data with a sampling rate of 30kHz. The collected data is then downsampled to 1kHz using a fixed interval sampling method. After that, the data is converted to the frequency domain and the specific frequency band features are extracted. The task sets the frequency band, uses these features as neural network input, and then uses the target position as the label to implement the classification task on the above data using a multi-layer perceptron network, and saves the trained model as a .pth file.

Furthermore, in the test phase, the monkeys used imagination to control the cursor movement to eventually reach the target area;

In this process, the decoder first loads the previously saved PTH model, specifically the DQN network acting as a decoder. The DQN network uses the same network structure as the previous supervised deep learning. Then, the trained parameter model is loaded to extract and decode the input neural signal features and output the corresponding action instructions.

The output instruction updates the cursor position on the page. At this time, the monkey knows the cursor change through visual feedback and generates new neural signals. The new neural signals are input into the decoder, and the decoder generates new actions, thus forming a closed loop.

According to another embodiment of the present invention, a closed-loop adaptive brain-computer interface decoding device based on reinforcement learning is provided, comprising:

A preprocessing unit, used for preprocessing the collected neural signals of the test animal;

A new neural signal generating unit is used to input the pre-processed neural signal into the decoder and map it into a specific action instruction, and the test animal body generates a new neural signal by providing visual feedback to the specific action instruction executed;

The new action command generation unit is used to input new neural signals into the decoder, and the decoder generates new action commands, forming a closed loop that integrates reinforcement learning theory and brain-computer interface decoding paradigm.

Furthermore, the device also includes:

The signal acquisition unit is used to collect nerve signals from the test animal.

A storage medium stores a program file capable of implementing any one of the above-mentioned closed-loop adaptive brain-computer interface decoding methods based on reinforcement learning.

A processor is used to run a program, wherein when the program is running, any one of the above-mentioned closed-loop adaptive brain-computer interface decoding methods based on reinforcement learning is executed.

The closed-loop adaptive brain-computer interface decoding method and device based on reinforcement learning in the embodiment of the present invention, The preprocessed neural signals are input into the decoder and mapped into specific action instructions. The test animal provides visual feedback on the specific action instructions to generate new neural signals. The new neural signals are input into the decoder, and the decoder generates new action instructions, forming a closed loop. The reinforcement learning theory and the brain-computer interface decoding paradigm are integrated. The brain and the decoder learn from each other and adapt to each other without the need for label signals, which is in line with the laws of biological learning.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

FIG1 is a flow chart of a closed-loop adaptive brain-computer interface decoding method based on reinforcement learning of the present invention;

FIG2 is a preferred flow chart of a closed-loop adaptive brain-computer interface decoding method based on reinforcement learning of the present invention;

FIG3 is a schematic diagram of a closed-loop adaptive decoder in the present invention;

FIG4 is a diagram of a reinforcement learning model in the present invention;

FIG5 is a demonstration diagram of a closed-loop adaptive decoder in the present invention;

FIG6 is a scene diagram of the macaque training stage in the present invention;

FIG7 is a module diagram of a closed-loop adaptive brain-computer interface decoding device based on reinforcement learning of the present invention;

FIG8 is a preferred module diagram of a closed-loop adaptive brain-computer interface decoding device based on reinforcement learning of the present invention.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. The described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

Example 1

According to an embodiment of the present invention, a closed-loop adaptive brain-computer interface decoding method based on reinforcement learning is provided, referring to FIG1 , comprising the following steps:

S101: preprocessing the collected neural signals of the test animal;

S102: inputting the preprocessed neural signal into the decoder to map it into a specific action instruction, and the test animal performs visual feedback on the executed specific action instruction to generate a new neural signal;

S103: The new neural signal is input to the decoder, and the decoder generates new action instructions, forming a closed loop that integrates reinforcement learning theory and brain-computer interface decoding paradigm.

The closed-loop adaptive brain-computer interface decoding method based on reinforcement learning in the embodiment of the present invention inputs the preprocessed neural signal into the decoder and maps it into specific action instructions. The test animal body generates new neural signals by providing visual feedback on the specific action instructions executed. The new neural signals are input into the decoder, and the decoder generates new action instructions, forming a closed loop. The reinforcement learning theory and the brain-computer interface decoding paradigm are integrated. The brain and the decoder learn from each other and adapt to each other without the need for label signals, which is in line with the law of biological learning.

Wherein, referring to FIG2 , the method further includes:

S100: Collecting neural signals from the test animal.

Wherein, in step S101, preprocessing includes amplification, filtering, and denoising.

The specific action instruction includes moving the cursor to a specified position. After the decoder generates a new action instruction, it assists the test animal to move the cursor to a fixed position by setting constraints.

Among them, the method includes a training part and a testing part;

In the training part, the monkey is trained to push the joystick to control the cursor to the specified position, and the monkey is rewarded at this time; the supervised learning neural network is used as a decoder in the training stage, and the target is used as a supervised learning label. Through training, a mapping relationship between neural signals and actions is established, and the trained model is saved;

During the testing phase, the monkey controlled the cursor movement through imagination. The monkey generated the intention to control the cursor movement by observing the position of the cursor on the screen. At this time, the decoder was replaced with a reinforcement learning algorithm to act as a decoder for decoding.

Among them, the reinforcement learning algorithm is DQN, in which the network that maps states to actions is directly loaded into the training phase using the model parameters trained by the neural network with supervised learning.

In other embodiments, the reinforcement learning algorithm may also adopt: Q-learning, SARSA, deep reinforcement learning algorithm.

In the training phase, the monkey used a joystick to control the cursor on the page to move. At this time, a circle of a different color appeared in each trial. After training, the monkey knew that moving the current cursor to the target position would be rewarded.

At the beginning of each trial, neural signals are collected. After the signals are collected, they need to be amplified and denoised. The processed data is 256 channels, half-precision floating-point data with a sampling rate of 30kHz for each channel. The collected data is then downsampled to 1kHz using a fixed interval sampling method. The data is then converted to the frequency domain, and specific frequency band features are extracted. The frequency band is set according to the specific task. These features are used as neural network inputs, and the target position is used as a label. A multi-layer perceptron network is used to implement the classification task for the above data, and the trained model is saved as a .pth file.

Among them, in the test phase, the monkey controlled the cursor movement through imagination and finally reached the target area;

In this process, the decoder first loads the previously saved PTH model, specifically the DQN network acting as a decoder. The DQN network uses the same network structure as the previous supervised deep learning. Then, the trained parameter model is loaded to extract and decode the input neural signal features and output the corresponding action instructions.

The output instruction updates the cursor position on the page. At this time, the monkey knows the cursor change through visual feedback and generates new neural signals. The new neural signals are input into the decoder, and the decoder generates new actions, thus forming a closed loop.

The following is a detailed description of the closed-loop adaptive brain-computer interface decoding method based on reinforcement learning of the present invention with a specific embodiment:

Neural decoding refers to recording brain neural activity and using computational models to analyze the activity in order to infer the cognitive or behavioral process that the observer is experiencing or performing. The most critical part of the neural decoding process is the decoder design. How to design a decoder that can decode quickly and ensure decoding accuracy is very important in brain-computer interface applications.

Based on the problems existing in the above decoders, the present invention designs a new decoding paradigm based on deep learning and reinforcement learning, and designs a brain-computer interface decoder based on the closed-loop decoding idea, which controls the movement of the cursor on the screen by decoding neural signals. The decoder combines the characteristics of fast convergence of supervised deep learning methods and the advantage of reinforcement learning that does not require labels. At the same time, both algorithms use the same neural network for learning. The later reinforcement learning algorithm directly uses the network trained by the supervised deep learning algorithm before. In this way, the reinforcement learning algorithm does not need to perform too much random exploration in the early stage, which greatly saves convergence time.

The basic contents of the technical solution of the present invention include:

As shown in Figure 3, neural signals are collected from the brain of an animal such as a monkey through neural microelectrodes and sent to the PC through operations such as amplification, filtering, and denoising. The PC starts to execute the decoder code, and the decoder maps the neural signal to a specific action instruction, such as moving the cursor up or down. The actuator updates the screen in real time after executing the movement instruction, and the updated page is displayed on the monitor. The monkey observes the corresponding new neural signal generated after the display page is updated, and the decoding forms a closed loop. Because the monkey's neurons are plastic, they will autonomously adjust in this process to move the cursor to the specified position. The decoder can set constraints to help the monkey move the cursor to a fixed position. The monkey and the decoder adapt to each other and learn from each other to complete the same task.

The present invention mainly includes two parts, namely, a training part and a testing part. In the training part, the monkey pushes the rocker to control the cursor to reach the specified position, and the monkey is rewarded with juice at this time. In the training stage, a supervised learning neural network is used as a decoder, because there are labels in the training part, and the target can be used as a supervised learning label in this process. Through training, a mapping relationship between neural signals and actions can be established, and the trained model can be saved. In the test stage, the monkey controls the movement of the cursor by imagination, and the monkey does not push the rocker. Instead, the monkey generates the intention of controlling the movement of the cursor by watching the cursor position on the screen. At this time, the decoder is replaced by a reinforcement learning algorithm to act as a decoder for decoding, and the reinforcement learning algorithm used here is a DQN algorithm. In the algorithm, the network that maps the state to the action is directly loaded into the model parameters trained by the neural network with supervised learning in the training stage, so that trial and error learning is avoided, thereby improving efficiency and accuracy, and making the system more practical.

First, let’s introduce reinforcement learning. Reinforcement learning is an important branch of machine learning that aims to learn how to make optimal decisions through interaction with the environment. Reinforcement learning models usually consist of the following five basic elements:

State: Represents the state of the environment, such as the state of a board in a game or the sensor readings of a robot.

Action: represents the operation performed by the agent in a certain state, such as the movement of chess pieces in a game or the movement of a robot.

Reward: Represents the feedback obtained by performing an action in a certain state, usually a scalar value. Rewards can be positive, negative, or zero, reflecting the quality of the action.

Environment: The environment refers to the external environment in which the agent is located, including all possible states and actions, as well as the feedback signals from the environment to the agent. The environment can be a real physical environment or a simulated environment, such as a game environment.

Agent: An agent is an algorithm or model that learns decision-making strategies. The agent interacts with the environment, constantly observes the current state, makes decisions and performs actions, and receives reward or punishment signals from the environment to provide feedback on the quality of the decision. The goal of the agent is to learn an optimal decision-making strategy that Maximize the long-term accumulated rewards.

Based on these elements, the reinforcement learning model can be described by a mathematical model. This model can calculate the next action based on the current state and historical actions and rewards, as well as the feedback signals provided by the environment, thereby achieving autonomous decision-making and action. Common reinforcement learning algorithms include Q-learning, SARSA, deep reinforcement learning, etc. These algorithms can select different strategies and value functions according to different tasks to achieve optimal decisions. Reinforcement learning models have been widely used in many fields, such as robot control, game AI, autonomous driving, etc. Through reinforcement learning, the intelligent agent can learn the optimal strategy from the interaction with the environment and continuously optimize its performance and performance.

As shown in FIG5 , in the present invention, the decoder in FIG5 is regarded as the intelligent agent in FIG4 , the decoder generates an action, and the monkey brain and external devices such as the display are the external environment for the decoder. The neural signal transmitted by the monkey brain to the decoder in FIG5 corresponds to the state received by the intelligent agent in FIG4 . When the decoder receives the neural signal, it generates a corresponding action instruction. After the host receives the action instruction, it updates the display state. After the display state changes, the monkey observes the corresponding change and the brain generates a new neural signal through encoding. The new neural signal can be regarded as a new state, thus forming a closed loop. Here, the reinforcement learning theory and the brain-computer interface decoding paradigm are integrated. The brain and the decoder learn from each other and adapt to each other without the need for label signals, which is in line with the law of biological learning.

As shown in Figure 6, during the training phase, the monkey used the joystick to control the cursor on the page to move. At this time, a circle of different colors appeared in each trial. After training, the monkey knew that moving the current cursor to the target position would result in a juice reward. At the beginning of each trial, the neural signal was collected. Here, the Utah electrode was used for collection. After the signal was collected, the neural signal needed to be amplified, denoised, and processed. The processed data was 256 channels, and each channel had a 30kHz sampling rate and half-precision floating-point data. The collected data was then downsampled and reduced to a sampling rate of 1kHz using a fixed interval sampling method. After that, the data was converted to the frequency domain, and specific frequency band features were extracted. The frequency band was set according to the specific task. These features were used as neural network inputs, and the target position was used as a label. The multi-layer perceptron network was used to implement the classification task for the above data, and the trained model was saved as a .pth file.

In the test phase, the monkey needs to use imagination to control the cursor movement and finally reach the target area. In this process, the decoder first loads the previously saved pth model. This part is the (Deep Q Learning, DQN) network acting as a decoder. The DQN network uses the same supervised deep learning as before. The network structure is then loaded with the trained parameter model to extract features and decode the input neural signals, and output the corresponding action instructions. The output instructions update the cursor position on the page. At this time, the monkey knows the cursor change through visual feedback and generates new neural signals. The new neural signals are input into the decoder, and the decoder generates new actions, thus forming a closed loop.

The key points and intended protection points of the present invention are:

The supervised deep learning scheme is combined with the reinforcement learning scheme that does not require labels. In the training phase (joystick control), the supervised deep learning decoding ensures that training can be carried out in a short time (establishing the mapping relationship between neural signals and actions). In the testing phase (through intention control), the reinforcement learning scheme ensures that learning and decoding can be carried out well without the need for a label decoder.

Compared with the prior art, the advantages of the present invention are:

Compared with the traditional direct decoding method, the present invention has better decoding effect and can achieve online decoding effect. Compared with the general closed-loop adaptive decoder, the present invention can quickly establish the mapping relationship between neural signals and actions in the initial stage, and can also be applied to unlabeled scenarios.

The present invention has been tested, simulated, and the model has been tested, and the technical effect of the present invention can be basically achieved. The present invention can be used not only in implantable brain-computer interfaces, but also in non-implantable brain-computer interfaces based on EEG data.

Example 2

According to another embodiment of the present invention, a closed-loop adaptive brain-computer interface decoding device based on reinforcement learning is provided, referring to FIG7 , comprising:

A preprocessing unit 201 is used to preprocess the collected neural signals of the test animal;

A new neural signal generating unit 202 is used to input the pre-processed neural signal into the decoder and map it into a specific action instruction, and the test animal body performs visual feedback on the executed specific action instruction to generate a new neural signal;

The new action instruction generating unit 203 is used to input new neural signals to the decoder, and the decoder generates new action instructions, forming a closed loop, integrating the reinforcement learning theory and the brain-computer interface decoding paradigm.

The closed-loop adaptive brain-computer interface decoding device based on reinforcement learning in the embodiment of the present invention inputs the preprocessed neural signal into the decoder to map it into specific action instructions. The test animal body provides visual feedback on the executed specific action instructions to generate new neural signals. The new neural signals are input to the decoder, and the decoder generates new action instructions to form a closed loop. The reinforcement learning theory and the brain-computer interface decoding paradigm are integrated. The brain and the decoder learn from each other and adapt to each other without the need for label signals, which is in line with the law of biological learning.

Wherein, referring to FIG8 , the device further includes:

The signal collection unit 200 is used to collect nerve signals from the test animal.

The following is a detailed description of the closed-loop adaptive brain-computer interface decoding device based on reinforcement learning of the present invention with a specific embodiment:

Neural decoding refers to recording brain neural activity and using computational models to analyze the activity in order to infer the cognitive or behavioral process that the observer is experiencing or performing. The most critical part of the neural decoding process is the decoder design. How to design a decoder that can decode quickly and ensure decoding accuracy is very important in brain-computer interface applications.

Based on the problems existing in the above decoders, the present invention designs a new decoding paradigm based on deep learning and reinforcement learning, and designs a brain-computer interface decoder based on the closed-loop decoding idea, which controls the movement of the cursor on the screen by decoding neural signals. The decoder combines the characteristics of fast convergence of supervised deep learning methods and the advantage of reinforcement learning that does not require labels. At the same time, both algorithms use the same neural network for learning. The later reinforcement learning algorithm directly uses the network trained by the supervised deep learning algorithm before. In this way, the reinforcement learning algorithm does not need to perform too much random exploration in the early stage, which greatly saves convergence time.

The basic contents of the technical solution of the present invention include:

As shown in Figure 3, neural signals are collected from the brain of an animal such as a monkey through neural microelectrodes and sent to the PC through operations such as amplification, filtering, and denoising. The PC starts to execute the decoder code, and the decoder maps the neural signal to a specific action instruction, such as moving the cursor up or down. The actuator updates the screen in real time after executing the movement instruction, and the updated page is displayed on the monitor. The monkey observes the corresponding new neural signal generated after the display page is updated, and the decoding forms a closed loop. Because the monkey's neurons are plastic, they will autonomously adjust in this process to move the cursor to the specified position. The decoder can set constraints to help the monkey move the cursor to a fixed position. The monkey and the decoder adapt to each other and learn from each other to complete the same task.

The present invention mainly includes two parts, namely, a training part and a testing part. In the training part, the monkey pushes the rocker to control the cursor to reach the specified position, and the monkey is rewarded with juice at this time. In the training stage, a supervised learning neural network is used as a decoder, because there are labels in the training part, and the target can be used as a supervised learning label in this process. Through training, a mapping relationship between neural signals and actions can be established, and the trained model can be saved. In the test stage, the monkey controls the movement of the cursor by imagination, and the monkey does not push the rocker. Instead, the monkey generates the intention of controlling the movement of the cursor by observing the position of the cursor on the screen. At this time, the decoder is replaced by a reinforcement learning algorithm to act as a decoder for decoding, and the reinforcement learning algorithm used here is a DQN algorithm. In the algorithm, the network that maps the state to the action is directly loaded into the model parameters trained by the neural network with supervised learning in the training stage, so that trial and error learning is avoided, thereby improving efficiency and accuracy, and making the system more practical.

First, let’s introduce reinforcement learning. Reinforcement learning is an important branch of machine learning that aims to learn how to make optimal decisions through interaction with the environment. Reinforcement learning models usually consist of the following five basic elements:

State: Represents the state of the environment, such as the state of a board in a game or the sensor readings of a robot.

Action: represents the operation performed by the agent in a certain state, such as the movement of chess pieces in a game or the movement of a robot.

Reward: Represents the feedback obtained by performing an action in a certain state, usually a scalar value. Rewards can be positive, negative, or zero, reflecting the quality of the action.

Environment: The environment refers to the external environment in which the agent is located, including all possible states and actions, as well as the feedback signals from the environment to the agent. The environment can be a real physical environment or a simulated environment, such as a game environment.

Agent: An agent is an algorithm or model that learns decision-making strategies. The agent interacts with the environment, constantly observes the current state, makes decisions and performs actions, and receives reward or punishment signals from the environment to provide feedback on the quality of the decision. The goal of the agent is to learn an optimal decision-making strategy that Maximize the long-term accumulated rewards.

Based on these elements, the reinforcement learning model can be described by a mathematical model. This model can calculate the next action based on the current state and historical actions and rewards, as well as the feedback signals provided by the environment, thereby achieving autonomous decision-making and action. Common reinforcement learning algorithms include Q-learning, SARSA, deep reinforcement learning, etc. These algorithms can select different strategies and value functions according to different tasks to achieve optimal decisions. Reinforcement learning models have been widely used in many fields, such as robot control, game AI, autonomous driving, etc. Through reinforcement learning, the intelligent agent can learn the optimal strategy from the interaction with the environment and continuously optimize its performance and performance.

As shown in FIG5 , in the present invention, the decoder in FIG5 is regarded as the intelligent agent in FIG4 , the decoder generates an action, and the monkey brain and external devices such as the display are the external environment for the decoder. The neural signal transmitted by the monkey brain to the decoder in FIG5 corresponds to the state received by the intelligent agent in FIG4 . When the decoder receives the neural signal, it generates a corresponding action instruction. After the host receives the action instruction, it updates the display state. After the display state changes, the monkey observes the corresponding change and the brain generates a new neural signal through encoding. The new neural signal can be regarded as a new state, thus forming a closed loop. Here, the reinforcement learning theory and the brain-computer interface decoding paradigm are integrated. The brain and the decoder learn from each other and adapt to each other without the need for label signals, which is in line with the law of biological learning.

As shown in Figure 6, during the training phase, the monkey used the joystick to control the cursor on the page to move. At this time, a circle of different colors appeared in each trial. After training, the monkey knew that moving the current cursor to the target position would result in a juice reward. At the beginning of each trial, the neural signal was collected. Here, the Utah electrode was used for collection. After the signal was collected, the neural signal needed to be amplified, denoised, and processed. The processed data was 256 channels, and each channel had a 30kHz sampling rate and half-precision floating-point data. The collected data was then downsampled and reduced to a sampling rate of 1kHz using a fixed interval sampling method. After that, the data was converted to the frequency domain, and specific frequency band features were extracted. The frequency band was set according to the specific task. These features were used as neural network inputs, and the target position was used as a label. The multi-layer perceptron network was used to implement the classification task for the above data, and the trained model was saved as a .pth file.

In the test phase, the monkey needs to use imagination to control the cursor movement and finally reach the target area. In this process, the decoder first loads the previously saved pth model. This part is the (Deep Q Learning, DQN) network acting as a decoder. The DQN network uses the same supervised deep learning as before. The network structure is then loaded with the trained parameter model to extract features and decode the input neural signals, and output the corresponding action instructions. The output instructions update the cursor position on the page. At this time, the monkey knows the cursor change through visual feedback and generates new neural signals. The new neural signals are input into the decoder, and the decoder generates new actions, thus forming a closed loop.

The key points and intended protection points of the present invention are:

The supervised deep learning scheme is combined with the reinforcement learning scheme that does not require labels. In the training phase (joystick control), the supervised deep learning decoding ensures that training can be carried out in a short time (establishing the mapping relationship between neural signals and actions). In the testing phase (through intention control), the reinforcement learning scheme ensures that learning and decoding can be carried out well without the need for a label decoder.

Compared with the prior art, the advantages of the present invention are:

Compared with the traditional direct decoding method, the present invention has better decoding effect and can achieve online decoding effect. Compared with the general closed-loop adaptive decoder, the present invention can quickly establish the mapping relationship between neural signals and actions in the initial stage, and can also be applied to unlabeled scenarios.

The present invention has been tested, simulated, and the model has been tested, and the technical effect of the present invention can be basically achieved. The present invention can be used not only in implantable brain-computer interfaces, but also in non-implantable brain-computer interfaces based on EEG data.

Example 3

A storage medium stores a program file capable of implementing any one of the above-mentioned closed-loop adaptive brain-computer interface decoding methods based on reinforcement learning.

Example 4

A processor is used to run a program, wherein when the program is running, any one of the above-mentioned closed-loop adaptive brain-computer interface decoding methods based on reinforcement learning is executed.

The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the system embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims (10)

A closed-loop adaptive brain-computer interface decoding method based on reinforcement learning, characterized in that it comprises the following steps: S101: preprocessing the collected neural signals of the test animal; S102: inputting the preprocessed neural signal into the decoder to map it into a specific action instruction, and the test animal performs visual feedback on the executed specific action instruction to generate a new neural signal; S103: The new neural signal is input to the decoder, and the decoder generates new action instructions, forming a closed loop that integrates reinforcement learning theory and brain-computer interface decoding paradigm. The closed-loop adaptive brain-computer interface decoding method based on reinforcement learning according to claim 1, characterized in that the method also includes: S100: Collecting neural signals from the test animal. The closed-loop adaptive brain-computer interface decoding method based on reinforcement learning according to claim 1 is characterized in that, in step S101, the preprocessing includes amplification, filtering, and denoising. According to the closed-loop adaptive brain-computer interface decoding method based on reinforcement learning in claim 1, it is characterized in that the specific action instruction includes moving a cursor to a specified position, and after the decoder generates a new action instruction, it assists the test animal in moving the cursor to a fixed position by setting constraints. The closed-loop adaptive brain-computer interface decoding method based on reinforcement learning according to claim 1 is characterized in that the method includes a training part and a testing part; In the training part, the monkey is trained to push the joystick to control the cursor to the specified position, and the monkey is rewarded at this time; the supervised learning neural network is used as a decoder in the training stage, and the target is used as a supervised learning label. Through training, a mapping relationship between neural signals and actions is established, and the trained model is saved; During the testing phase, the monkey controlled the cursor movement through imagination. The monkey generated the intention to control the cursor movement by observing the position of the cursor on the screen. At this time, the decoder was replaced with a reinforcement learning algorithm to act as a decoder for decoding. According to the closed-loop adaptive brain-computer interface decoding method based on reinforcement learning in claim 5, it is characterized in that the reinforcement learning algorithm is DQN, in which the network that maps states to actions is directly loaded into the training phase using model parameters trained by a neural network with supervised learning. The closed-loop adaptive brain-computer interface decoding method based on reinforcement learning according to claim 5 is characterized in that, in the training stage, the monkey controls the cursor on the page to move by means of a joystick, at which time a circle of a different color appears in each trial, and the monkey knows after training that there will be a reward for moving the cursor at the current position to the target position; At the beginning of each trial, neural signals are collected. After the signals are collected, they need to be amplified and denoised. The processed data is 256 channels, half-precision floating-point data with a sampling rate of 30kHz for each channel. The collected data is then downsampled to 1kHz using a fixed interval sampling method. The data is then converted to the frequency domain, and specific frequency band features are extracted. The frequency band is set according to the specific task. These features are used as neural network inputs, and the target position is used as a label. A multi-layer perceptron network is used to implement the classification task for the above data, and the trained model is saved as a .pth file. The closed-loop adaptive brain-computer interface decoding method based on reinforcement learning according to claim 5 is characterized in that, during the test phase, the monkey controls the cursor movement by imagination to finally reach the target area; In this process, the decoder first loads the previously saved PTH model, specifically the DQN network acting as a decoder. The DQN network uses the same network structure as the previous supervised deep learning. Then, the trained parameter model is loaded to extract and decode the input neural signal features and output the corresponding action instructions. The output instruction updates the cursor position on the page. At this time, the monkey knows the cursor change through visual feedback and generates new neural signals. The new neural signals are input into the decoder, and the decoder generates new actions, thus forming a closed loop. A closed-loop adaptive brain-computer interface decoding device based on reinforcement learning, characterized by comprising: A preprocessing unit, used for preprocessing the collected neural signals of the test animal; A new neural signal generating unit is used to input the pre-processed neural signal into the decoder and map it into a specific action instruction, and the test animal body generates a new neural signal by providing visual feedback to the specific action instruction executed; The new action command generation unit is used to input new neural signals into the decoder, and the decoder generates new action commands, forming a closed loop that integrates reinforcement learning theory and brain-computer interface decoding paradigm. The closed-loop adaptive brain-computer interface decoding device based on reinforcement learning according to claim 9, characterized in that the device also includes: The signal acquisition unit is used to collect nerve signals from the test animal.