CN118035816A - Electroencephalogram signal classification method, device and storage medium - Google Patents

Abstract

The invention discloses an electroencephalogram signal classification method, a device and a storage medium, wherein the method comprises the following steps: acquiring a target domain to be classified; inputting the target domain to be classified into a target generator to obtain migration data to be classified; inputting migration data to be classified into a main migration neural network model to obtain an electroencephalogram signal classification result; the main body migration neural network model is obtained through the following steps: acquiring a first motor imagery electroencephalogram signal and a second motor imagery electroencephalogram signal; calculating a domain set, wherein the domain set comprises a first source domain and a first target domain; inputting the first source domain into a preset neural network model for training to obtain a main migration neural network model; the target generator is obtained by the following steps: training a preset generator according to the first source domain, the first target domain and the main body migration neural network model to obtain a target generator. The invention improves the classification accuracy and reduces the signal calibration time. The invention can be widely applied to the technical field of machine learning.

Description

A method, device and storage medium for classifying electroencephalogram signals

Technical Field

The present invention relates to the field of machine learning technology, and in particular to an electroencephalogram (EEG) signal classification method, device and storage medium.

Background technique

Brain Computer Interface (BCI) enables users to communicate with computers directly using brain signals. The most common non-invasive BCI method, electroencephalogram (EEG), is sensitive to noise or artifacts, and there are individual differences between subjects and non-stationarity within subjects. Therefore, it is difficult to establish an optimal general pattern recognition model suitable for different subjects, different sessions, different devices and different tasks in an EEG-based brain-computer interface system. Calibration sessions are usually required to collect some training data for new subjects, which is time-consuming and user-unfriendly, and also causes waste of previously collected EEG data. Furthermore, it is easy to cause brain fatigue and long signal calibration time. In addition, large individual differences lead to low classification accuracy in many BCI systems.

In summary, the technical problems existing in the relevant technologies need to be improved.

Summary of the invention

The embodiments of the present invention provide a method, device and storage medium for classifying electroencephalogram (EEG) signals, which effectively improve the classification accuracy and reduce the signal calibration time.

In one aspect, an embodiment of the present invention provides a method for classifying an electroencephalogram signal, comprising the following steps:

Obtain the target domain to be classified;

Inputting the target domain to be classified into a target generator to obtain migration data to be classified;

Inputting the to-be-classified migration data into the subject migration neural network model to obtain an EEG signal classification result;

The subject migration neural network model is obtained by the following steps:

Acquire a first motor imagery EEG signal of a gold subject and a second motor imagery EEG signal of a target subject, wherein the gold subject is used to characterize that the first motor imagery EEG signal has a high classification accuracy in a classification model before migration, and the target subject is used to characterize that the second motor imagery EEG signal has a low classification accuracy in a classification model before migration;

Calculate a domain set according to the first motor imagery EEG signal and the second motor imagery EEG signal, wherein the domain set includes a first source domain and a first target domain;

Inputting the first source domain into a preset neural network model so that the preset neural network model is trained to obtain the subject transfer neural network model;

The target generator is obtained by the following steps:

According to the first source domain, the first target domain and the subject transfer neural network model, the preset generator is trained to obtain the target generator.

In some embodiments, the step of acquiring the first motor imagery EEG signal of the gold subject and the second motor imagery EEG signal of the target subject comprises:

Collecting the first motor imagery EEG signal of the gold subject through brain signal electrodes;

The second motor imagery EEG signal of the target subject is collected through brain signal electrodes.

In some embodiments, the calculating of the domain set according to the first motor imagery EEG signal and the second motor imagery EEG signal comprises:

Calculating a second source domain according to the first motor imagery EEG signal;

calculating a second target domain according to the second motor imagery EEG signal;

The domain set is calculated according to the second source domain and the second target domain.

In some embodiments, the calculating the domain set according to the second source domain and the second target domain comprises:

Calculating a first reference matrix according to the second source domain;

performing a first Euclidean alignment on the second source domain according to the first reference matrix to obtain the first source domain;

Calculating a second reference matrix according to the second target domain;

According to the second reference matrix, a second Euclidean alignment is performed on the second target domain to obtain the first target domain.

In some embodiments, the steps of constructing the subject migration neural network model architecture include:

Constructing an input layer, wherein the output space dimension of the input layer is 1 dimension;

After the input layer, a conventional convolutional layer is constructed;

After the conventional convolution layer, a channel-by-channel convolution layer is constructed;

After the channel-by-channel convolution layer, a point-by-point convolution layer is constructed;

After the point-by-point convolution layer, construct a first pooling layer;

After the first pooling layer, a spatial convolution layer is constructed;

After the spatial convolution layer, construct a second pooling layer;

After the second pooling layer, a compression layer is constructed;

After the compression layer, an attention layer is constructed;

After the attention layer, a temporal convolution layer is constructed;

After the temporal convolution layer, a feature map extraction layer is constructed;

After the feature map extraction layer, a fully connected layer is constructed.

In some embodiments, the training of a preset generator according to the first source domain, the first target domain and the subject transfer neural network model to obtain the target generator includes:

Inputting the first target domain into the preset generator to obtain first migration data;

Inputting the first source domain and the first migration data into the subject migration neural network model to obtain the cross entropy loss and the mean square error corresponding to each key layer output feature in the subject migration neural network model;

Add the mean square errors of the above multiple items to calculate the style loss;

Add the style loss and the cross entropy loss to calculate the total loss;

According to the total loss, the preset generator is trained by a stochastic gradient descent algorithm to obtain the target generator.

In some embodiments, calculating a first reference matrix according to the second source domain includes:

According to the second source domain, a first reference matrix is calculated using a reference matrix calculation formula, where the reference matrix calculation formula is:

In the formula, is the first reference matrix, _xi is the ith EEG signal in the second source domain, and n is the total number of EEG signals in the second source domain.

On the other hand, an embodiment of the present invention provides an electroencephalogram signal classification device, comprising:

The first module is used to obtain the target domain to be classified;

The second module is used to input the target domain to be classified into a target generator to obtain migration data to be classified;

The third module is used to input the migration data to be classified into the subject migration neural network model to obtain the EEG signal classification result;

The subject migration neural network model is obtained by the following steps:

Acquire a first motor imagery EEG signal of a gold subject and a second motor imagery EEG signal of a target subject, wherein the gold subject is used to characterize that the first motor imagery EEG signal has a high classification accuracy in a classification model before migration, and the target subject is used to characterize that the second motor imagery EEG signal has a low classification accuracy in a classification model before migration;

Calculate a domain set according to the first motor imagery EEG signal and the second motor imagery EEG signal, wherein the domain set includes a first source domain and a first target domain;

Inputting the first source domain into a preset neural network model so that the preset neural network model is trained to obtain the subject transfer neural network model;

The target generator is obtained by the following steps:

According to the first source domain, the first target domain and the subject transfer neural network model, a preset generator is trained to obtain the target generator.

On the other hand, an embodiment of the present invention provides an electroencephalogram signal classification device, comprising:

at least one processor;

at least one memory for storing at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

On the other hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

The beneficial effects of the present invention are as follows:

The present invention first obtains the target domain to be classified, inputs the target domain to be classified into the target generator, obtains the migration data to be classified, and then inputs the migration data to be classified into the subject migration neural network model to obtain the EEG signal classification result, thereby realizing brain-computer interface migration learning, improving classification accuracy, and reducing signal calibration time. Among them, the subject migration neural network model calculates the domain set including the first source domain and the first target domain through the first motor imagery EEG signal of the golden subject and the second motor imagery EEG signal of the target subject, and then trains through the first source domain to improve the accuracy of model classification, while the target generator is obtained by training the preset generator according to the first source domain, the first target domain and the subject migration neural network model, thereby improving the accuracy of the generator migration data.

Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a flow chart of a method for classifying EEG signals according to an embodiment of the present invention;

FIG2 is a flow chart of obtaining a subject transfer neural network model according to an embodiment of the present invention;

FIG3 is a schematic diagram of stimulating a subject to generate an electroencephalogram signal according to an embodiment of the present invention;

FIG4 is a schematic diagram of a timing diagram of a brain electrical signal acquisition process according to an embodiment of the present invention;

FIG5 is a schematic diagram of electrode distribution of an electroencephalogram signal acquisition instrument according to an embodiment of the present invention;

FIG6 is a schematic diagram of electrode distribution of an electrooculographic signal acquisition instrument according to an embodiment of the present invention;

FIG7 is a schematic diagram of a subject transfer neural network model architecture according to an embodiment of the present invention;

FIG8 is a schematic diagram of an attention mechanism according to an embodiment of the present invention;

FIG9 is a schematic diagram of a process of processing a feature map by a generator according to an embodiment of the present invention;

FIG10 is a schematic diagram of a test result of a subject transfer neural network model according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are described as follows:

Transfer learning: A method in the field of machine learning that applies knowledge learned from one task to another related task. This learning method uses knowledge from the source field to improve learning performance in the target field. EEG signals have strong individual differences. Simply put, two people imagine the same left hand movement, but their measured EEG signals are very different. Therefore, a model used to classify the EEG signal of subject A may perform poorly in the task of classifying the EEG signal of subject B. Transfer learning methods can help solve such problems.

Brain-computer interface (BCI): Brain-computer interface is a technology that directly connects the human brain and external devices (usually computers or other intelligent systems). Its goal is to achieve direct communication with computers or other devices by monitoring and interpreting brain activity without the need for traditional input devices (such as keyboards or mice).

BCI illiterates: target subjects, some subjects who perform poorly (low accuracy) in classification.

Golden subjects: golden subjects, which are subjects who perform better (with higher accuracy) in classification than BCI illiterates.

Euclidean Alignment (EA): A signal processing method that transforms data in Euclidean space.

Generator: A data processing structure that uses one-dimensional convolutional layers and upsampling layers to transform feature maps, used in conjunction with the subject transfer neural network model.

Electroencephalogram (EEG): It is a graph obtained by amplifying and recording the spontaneous biopotential of the brain from the scalp through a sophisticated electronic instrument. It is the spontaneous and rhythmic electrical activity of brain cell groups recorded by electrodes. Conventional EEG, dynamic EEG monitoring, and video EEG monitoring.

EEG signal: It is the overall reflection of the electrophysiological activity of brain nerve tissue on the surface of the cerebral cortex, and is the synthesis of the postsynaptic potential of brain neurons. It can be divided into spontaneous EEG and induced EEG. Spontaneous EEG refers to the potential changes that occur spontaneously in brain nerve cells without specific external stimulation, and induced EEG refers to the EEG changes caused by artificially applying light, sound, and electrical stimulation to the sensory organs. The activity data of the changes in electrical signals during brain activity can form an electroencephalogram.

The EEG signal classification method provided in the embodiment of the present application can be applied to a terminal, a server, or a software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an EEG signal classification method, etc., but is not limited to the above forms.

The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

The following is a detailed explanation of the embodiments of the present application in conjunction with the accompanying drawings:

As shown in FIG1 , an embodiment of the present invention provides a method for classifying EEG signals. The method of this embodiment may include but is not limited to the following steps:

Step S11, obtaining the target domain to be classified;

Step S12, input the target domain to be classified into the target generator to obtain the migration data to be classified;

Step S13: input the migration data to be classified into the subject migration neural network model to obtain the EEG signal classification result.

In this embodiment, the motor imagery EEG signal to be classified of the target subject can be obtained first, and then multiple motor imagery EEG signals to be classified are combined to obtain a target domain to be classified, and then the target domain to be classified is input into the target generator to obtain the migration data to be classified, and finally the migration data to be classified is input into the subject migration neural network model to obtain the EEG signal classification result. Among them, the EEG signal classification result can include the left hand motor imagery EEG signal, the right hand motor imagery EEG signal, the two feet motor imagery signal, and the tongue motor imagery signal.

In this embodiment, as shown in FIG2 , the specific implementation process of obtaining the subject transfer neural network model includes but is not limited to steps S201 to S203:

Step S201, obtaining a first motor imagery EEG signal of a gold subject and a second motor imagery EEG signal of a target subject, wherein the gold subject is used to characterize that the first motor imagery EEG signal has a high classification accuracy in the classification model before migration, and the target subject is used to characterize that the second motor imagery EEG signal has a low classification accuracy in the classification model before migration.

In this embodiment, obtaining the first motor imagery EEG signal of the gold subject and the second motor imagery EEG signal of the target subject may be by collecting the first motor imagery EEG signal of the gold subject through brain signal electrodes and collecting the second motor imagery EEG signal of the target subject through brain signal electrodes.

In this embodiment, in obtaining the motor imagery EEG signal of the golden subject or the target subject, the subject is required to imagine the preset action according to the instruction of the screen, so that the subject's brain generates EEG signals, and the EEG signals are collected by an EEG signal acquisition instrument containing 25 electrode channels. That is, each time the acquisition process is performed, the subject is instructed to perform an action imagination and collect an EEG signal. After performing multiple acquisition processes, multiple EEG signals can be obtained. In this embodiment, the motor imagery EEG signal can be divided into multiple categories according to actual needs. In this embodiment, the motor imagery EEG signal is divided into 4 categories, including left-hand motor imagery EEG signal, right-hand motor imagery EEG signal, foot motor imagery signal, and tongue motor imagery signal. In the process of acquiring EEG signals, as shown in Figure 3, a display device can be used to show the subject images and sound prompts, including left-hand imagination movement prompts, right-hand imagination movement prompts, foot motor imagery prompts and tongue motor imagery prompts, to stimulate the subject to generate EEG signals by motor imagery. The time and sequence of the test process are shown in Figure 4. In each test, the subject sits in a comfortable armchair with a computer screen in front of him. At the beginning of the test (t=0s), a fixed cross will appear on the black screen, and there will be a short sound prompt (Beep). Two seconds later (t=2s), an arrow pointing to the left, right, down or up (corresponding to the four categories of left hand movement, right hand movement, foot movement and tongue movement) will appear on the screen for about 1.25s as a prompt, prompting the subject to imagine the movement corresponding to the picture. Each subject needs to complete this imagination task until the cross on the screen disappears (t=6s), followed by a short break until the screen turns black again. The average time for each test is about 8 seconds, and the state of the screen during this period is shown in Figure 3. In addition, in the process of performing an 8-second EEG signal acquisition, the EEG signal acquired each time can be intercepted and retained. Exemplarily, only the 1.5-second to 6-second portion of each EEG signal is retained, and the other portions are deleted. From the perspective of noise distribution, the 1.5-second to 6-second portion of each EEG signal has less noise and can maximize the retention of the EEG signal generated during motor imagery. The other portions have more noise and therefore have more outliers. Outliers can be screened out by intercepting and retaining. The intercepted 4.5s is used as the electroencephalogram (EEG) data generated by the final motor imagery, and the final data format is (22, 1, 1125), 22 represents the number of channels, i.e., the number of electrodes, 1125 represents the time point (sampling rate 250hz×4.5s), and 250hz represents 1s to collect 250 data. 1 represents the number of feature maps (dimension of feature space). In this embodiment, among the 25 electrodes, 22 brain signal electrodes as shown in FIG5 can be included for collecting EEG and generating EEG signals, and 3 electrodes as shown in FIG6 for collecting electrooculograms (EOG) and generating electrooculograms. When the equipment is limited, only the 22 brain signal electrodes in Figure 5 can be used to collect EEG signals, and the EOG three electrodes do not participate in the collection of EEG signals. The signal is sampled at 250Hz (250 samples per second), band-pass filtered between 0.5Hz and 100Hz, and an additional 50Hz notch filter is used to suppress line noise.

Step S202: Calculate a domain set according to the first motor imagery EEG signal and the second motor imagery EEG signal, where the domain set includes a first source domain and a first target domain.

In this embodiment, the domain set is calculated based on the first motor imagery EEG signal and the second motor imagery EEG signal. The second source domain is first calculated based on the first motor imagery EEG signal, and then the second target domain is calculated based on the second motor imagery EEG signal. Finally, the domain set is calculated based on the second source domain and the second target domain.

In this embodiment, the subject with the highest classification accuracy can be selected as the "golden subject", i.e., the golden subject, based on the classification accuracy of the EEG signals of all subjects, and its EEG signal and its label are used as the second source domain, while the EEG signal of the target subject and its label are used as the second target domain. Specifically, the EEG signal data of the subject user are labeled {S1, S2, S3, S4, S5, S6, S7, S8, S9, ...}, and are respectively input into the classification model before migration for classification prediction to obtain the accuracy {Acc1, Acc2, Acc3, Acc4, Acc5, Acc6, Acc7, Acc8, Acc9, ...}. The subject with the highest accuracy is selected as the golden subject, i.e., argmax{Acc1, Acc2, Acc3, Acc4, Acc5, Acc6, Acc7, Acc8, Acc9, ...}, and the EEG signal of the golden subject is used as the second source domain {X _S , Y _S }, while the EEG signal of the target subject (i.e., BCI illiterate) is used as the second target domain {X _T , Y _T }. Among them, the size of the EEG signal x obtained in one experiment is (22, 1, 1125) and its label y size is (1), and the label y can be a number in {0, 1, 2, 3}, representing the left hand, right hand, feet, and tongue respectively. The EEG signals and labels obtained from n experiments are superimposed to obtain n×(22, 1, 1125), i.e., X _S , and n×(1), i.e., Y _S .

In this embodiment, the domain set is calculated according to the second source domain and the second target domain. The first reference matrix is first calculated according to the second source domain, and then the first Euclidean alignment is performed on the second source domain according to the first reference matrix to obtain the first source domain. Then, the second reference matrix is calculated according to the second target domain, and finally, the second Euclidean alignment is performed on the second target domain according to the second reference matrix to obtain the first target domain.

In this embodiment, the first reference matrix may be calculated according to the second source domain {X _S , Y _S } by using the reference matrix calculation formula, and then the first Euclidean alignment is performed on the second source domain according to the first reference matrix by using the Euclidean alignment calculation formula to obtain the aligned first source domain Source domain ^EA The reference matrix calculation formula is:

In the formula, is the first reference matrix, _xi is the ith EEG signal in the second source domain, and n is the total number of EEG signals in the second source domain.

The calculation formula for Euclidean alignment is:

In the formula, is the ith EEG signal after alignment, /> is the first reference matrix, and _xi is the i-th EEG signal in the second source domain.

In this embodiment, the second reference matrix is calculated according to the second target domain {X _T , Y _T } by the reference matrix calculation formula, and finally, the second target domain is subjected to second Euclidean alignment according to the second reference matrix by the Euclidean alignment calculation formula to obtain the first target domain Target domain ^EA Furthermore, the mean covariance matrix of all n aligned experiments for the aligned subject is:

Where I is the mean covariance matrix, is the reference matrix, _xi is the ith EEG signal, and n is the total number of EEG signals in the first source domain or the first target domain.

In this embodiment, the average covariance matrix of the first source domain and the first target domain is equal to the unit matrix after alignment, and the distribution of the covariance matrices of the two is more similar, further improving the similarity of the feature space distribution of the source domain and the target domain. To a certain extent, the migration from the golden subject to the target subject is achieved. In this embodiment, the Euclidean alignment converts and calibrates the EEG test in the Euclidean space. It does not require any label information from the subject, has a fast calculation speed, and is a good signal processing method.

Step S203: input the first source domain into the preset neural network model so that the preset neural network model is trained to obtain a subject transfer neural network model.

In this embodiment, the aligned EEG signal data of the first source domain is input into the preset neural network model to obtain the prediction result output by the preset neural network model, and the loss function between the prediction result and the label is calculated. The distance between the prediction result and the label is used to determine the training end condition, and the stochastic gradient descent method is used to reduce the loss function to achieve the purpose of optimizing the network, until the distance between the prediction result and the corresponding label is less than the preset threshold or the number of training times reaches a predetermined value, the training of the preset neural network model is terminated, and the subject transfer neural network model is obtained.

In this embodiment, the steps of constructing the subject migration neural network model architecture include:

Construct the input layer, the output space dimension of the input layer is 1 dimension;

After the input layer, a regular convolutional layer is constructed;

After the regular convolution layer, a channel-by-channel convolution layer is constructed;

After the channel-by-channel convolution layer, a point-by-point convolution layer is constructed;

After the point-by-point convolution layer, the first pooling layer is constructed;

After the first pooling layer, a spatial convolution layer is constructed;

After the spatial convolution layer, the second pooling layer is constructed;

After the second pooling layer, a compression layer is constructed;

After the compression layer, build the attention layer;

After the attention layer, a temporal convolution layer is constructed;

After the temporal convolution layer, a feature map extraction layer is constructed;

After the feature map extraction layer, a fully connected layer is constructed.

In this embodiment, the architecture of the subject transfer neural network model is shown in Figure 7. The subject transfer neural network consists of an input layer, a conventional convolution layer, a depth-separable convolution layer, a first pooling layer, a spatial convolution layer, a second pooling layer, a compression layer, an attention layer, a temporal convolution layer, a feature map interception layer, and a fully connected layer, and is connected in sequence in the described order. In this embodiment, when the subject transfer neural network model is trained using EEG signals, the subject neural network model receives the input EEG signal through the input layer and sends it to the conventional convolution layer. In the conventional convolution layer, the input EEG signal is convolved in a two-dimensional convolution manner, wherein the size of the convolution kernel is 1×64, the convolution step is 1×3, and the output space dimension is 16, that is, after passing through the conventional convolution layer, the number of output feature maps is 16, and the features output by the conventional convolution layer include the temporal and spatial features of the EEG signal.

After obtaining the feature map output by the conventional convolution layer, it is input into the depthwise separable convolution layer, including channel-by-channel convolution and point-by-point convolution. In the channel-by-channel convolution, the convolution kernel size is 22×1, the convolution kernel step size is 1×1, the output spatial dimension is 16, and the output feature map size is 16×1×1125. In the point-by-point convolution, the convolution kernel size is 1×1, the convolution kernel step size is 1×1, the output spatial dimension is 32, and the output feature map size is 32×1×1125. Channel-by-channel convolution extracts features from each layer of the channel. Combined with point-by-point convolution, it can enhance the information interaction between feature channels and further improve network performance.

After obtaining the feature map output by the depthwise separable convolutional layer, it is input into the first pooling layer, which uses the average pooling method to compress the feature map size to 32×1×140, with a pooling step of 1×8 and an output feature space dimension of 32. The first pooling layer achieves dimensionality reduction of features, compresses data and reduces the number of parameters, which can avoid the problem of network overfitting.

After obtaining the feature map output by the first pooling layer, it is input into the spatial convolution layer, the size of the convolution kernel is 1×16, the step size of the convolution kernel is 1×1, the output feature space dimension is 32, and the output feature map size is 32×1×140. Spatial convolution further enhances the information interaction between EEG data channels, and the output feature map realizes the enhancement of spatial features.

After obtaining the feature map output by the spatial convolution layer, it is input to the compression layer. The function of the compression layer is to transform the feature map size from 32×1×20 to 32×20, and the feature space dimension is kept at 1 from this layer. Furthermore, the feature map can be compressed by the sliding window method, where the window length is set to 16. The specific method of the sliding window is: for a feature map of size 32×20, the data of its second dimension is sequentially intercepted with indexes 1-16, 2-17, 3-18, 4-19, and 5-20, and the feature map size obtained by each sliding window is 32×16. The setting of the sliding window can improve the generalization ability of the model and make the trained network effect better.

After obtaining the feature map output by each window, the feature map of each window is input into the attention layer in turn. In deep neural networks, the attention mechanism is a method to mimic the behavior of the human brain, that is, to selectively focus on some important elements and ignore other elements. The attention mechanism can be simulated by three components: value (sensory input), key (non-volitional clue) and query (volitional clue). The interaction between query and key produces attention concentration, which leads to the selection of value. The attention layer consists of query Q, key K and value V, and the interaction between query and key produces attention score. The feature map size output by the attention layer is 32×16. As shown in Figure 8, the attention layer can make the main transfer neural network model selectively focus on a few important elements and ignore other elements, that is, it is more prominent on the elements that are more critical for classification, while the less important parts of classification prediction are ignored to a certain extent, thereby further improving the classification performance of the network.

After obtaining the feature map output by the attention layer, it is input into the temporal convolution layer. The temporal convolution layer consists of two dilated causal convolutions, with convolution kernel size of 1×4, convolution kernel step size of 1×1, and output feature map size of 32. Causal convolution introduces causality in the convolution process, that is, each element of the output depends only on the element that is not later than it in the input sequence. This design enables causal convolution to process time series data, ensuring that the model can only use current and previous information when predicting the future, and will not use future information, thereby ensuring that the subject transfer neural network model conforms to the basic before-after dependency relationship of EEG signal data in the temporal sequence during construction and training. In view of the small coverage of historical information in causal convolution, dilated causal convolution is used to expand the receptive field to improve it. The input feature map size in the temporal convolution layer is 32×16. The processing method of this layer is to split it into 16 time elements, each element is a sequence of vectors with a length of 32. The temporal convolution layer uses dilated causal convolution to process this sequence. Each element of the output sequence is calculated through the convolution operation, and only the part of the input sequence that is not later than the element is used to calculate each element. Therefore, the last element of the output sequence is calculated by considering the convolution operation of the entire input sequence. The final output of the temporal convolution layer can be obtained by the feature map interception method.

After obtaining the feature map output by the temporal convolution layer, the output of the temporal convolution layer of each sliding window is averaged and then input into the fully connected layer. In the fully connected layer, the final prediction result is obtained through the four-classification output of the softmax layer. Exemplarily, the output prediction results may include left hand movement imagination, right hand movement imagination, both feet movement imagination, and tongue movement imagination.

In this embodiment, the specific implementation process of obtaining the target generator includes but is not limited to the following steps:

According to the first source domain, the first target domain and the subject transfer neural network model, the preset generator is trained to obtain the target generator.

In this embodiment, the preset generator is trained according to the first source domain, the first target domain and the main transfer neural network model to obtain the target generator. The first target domain can be first input into the preset generator to obtain the first transfer data, and then the first source domain and the first transfer data are input into the main transfer neural network model to obtain the cross entropy loss and the mean square error corresponding to the output feature of each key layer in the main transfer neural network model. Multiple mean square errors are added to calculate the style loss, and then the style loss and the cross entropy loss are added to calculate the total loss. Finally, according to the total loss, the preset generator is trained by the stochastic gradient descent algorithm to obtain the target generator.

In this embodiment, the target generator processing flow is shown in Figure 9, and the target generator consists of an encoder and a decoder. In the encoding process, the input feature map is first downsampled in the form of convolution, and then the output data is normalized and nonlinearly activated. After the encoder, the size of the feature map is doubled, and the number of feature maps is doubled. In the decoding process, the input feature map is first upsampled in the form of convolution, and then the output feature map is normalized. The encoding and decoding process is actually to downsample and upsample the feature map in the form of convolution. This method can accelerate the training of the generator and increase the size of the effective receptive field. More importantly, after the feature map is input into the generator, the size of the feature map remains unchanged, and the output of the generator is the EEG signal data migrated from the first source domain to the first target domain. In this embodiment, the first target domain can be first input into the preset generator to obtain the first migration data, and then the first migration data and the aligned first source domain are respectively input into the subject migration neural network model to obtain the cross entropy loss and the mean square error corresponding to the output feature of each key layer in the subject migration neural network model. Among them, there are some key network layers in the subject transfer neural network model, such as the depth-separable convolution layer, the attention layer, and the time convolution layer. The first migration data and the aligned first source domain will output corresponding features after passing through these key layers. The mean square error between the output features of the first migration data and the aligned first source domain in these layers can be calculated. It can be understood that after the EEG signals of the first source domain and the first target domain are input into the subject transfer neural network model, the features output by each layer are in the form of a matrix. x and y represent the output matrices of the key layers after the first source domain and the first target domain are respectively sent to the subject transfer neural network. When the amount of data in the output matrix of a key layer is n (that is, a matrix has n data), the data at the corresponding positions of the two matrices x and y are subtracted, squared, and then added up to obtain the mean square error calculated for the layer. The calculation formula for the mean square error is:

Where MSE is the mean square error, _xi is the i-th data in the output matrix obtained from the first source domain, _yi is the i-th data in the output matrix obtained from the first target domain, and n is the amount of data in the output matrix.

More importantly, a mean square error is calculated at the output of each key layer, and the mean square errors calculated at each layer are summed to obtain the style loss.

At the same time, after the first migration data is input into the main migration neural network model, the cross entropy loss between the predicted category of the first target domain EEG signal and the true label of the first target domain EEG signal is obtained, and the distance between the predicted category and the true label can be calculated by the calculation formula of the cross entropy loss. Among them, the calculation formula of the cross entropy loss is:

Where L is the cross entropy loss, y is the predicted category of the EEG signal in the first target domain, is the true label of the EEG signal in the first target domain, and Q is the total number of classification task categories.

In this embodiment, after calculating the style loss and the cross entropy loss, the style loss and the cross entropy loss can be added to calculate the total loss, which measures the quality of the migration effect. Finally, according to the total loss, the preset generator is trained by the stochastic gradient descent algorithm to obtain the target generator. In this way, the migration from the source domain to the target domain is maximized, that is, the EEG signal data is migrated from the golden subject to the BCI illiterate (i.e., the target subject).

In this embodiment, the subject transfer neural network model is tested using the BCI IV-2a data set, and the test results are shown in Figure 10. Among them, this data set is a public data set provided by the brain-computer interface competition and is widely used in the field of brain-computer interface. The data set consists of EEG signal data of 9 subjects represented by S1-S9. During the experiment, the subjects need to perform four different motor imagery tasks, namely left hand (1 category), right hand (2 categories), feet (3 categories) and tongue (4 categories) motor imagery. From the test results, it can be seen that the average classification accuracy of this embodiment (EA-G-ATNN) reaches 75.54%, which is higher than the average classification accuracy of EEGNet, DeepConvNet, and FBCSP methods.

The beneficial effects of implementing the embodiments of the present invention include: the embodiments of the present invention first obtain the target domain to be classified, input the target domain to be classified into the target generator, obtain the migration data to be classified, and then input the migration data to be classified into the subject migration neural network model to obtain the EEG signal classification result, thereby realizing brain-computer interface transfer learning, improving classification accuracy, and reducing signal calibration time. Among them, the subject migration neural network model calculates the domain set including the first source domain and the first target domain through the first motor imagery EEG signal of the golden subject and the second motor imagery EEG signal of the target subject, and then trains through the first source domain to improve the accuracy of model classification, while the target generator is obtained by training the preset generator according to the first source domain, the first target domain and the subject migration neural network model, thereby improving the accuracy of the generator migration data.

In this embodiment, the subject transfer neural network model used includes multiple convolution scales, multiple convolution types, and multiple data processing methods. The network layer type and the size and step size of each convolution kernel are reasonably designed, and it has a high recognition accuracy. The training set used for training the subject transfer neural network model is obtained after the Euclidean space alignment of the collected EEG signals, which can improve the training effect of the subject transfer neural network. Moreover, residual connections, random neuron inactivation, nonlinear activation between layers, normalization and other technologies are added between layers in the subject transfer neural network model to improve network performance. In this embodiment, the subject transfer neural network model based on the Euclidean alignment and generator hybrid transfer learning method does not require EEG signal data of the target domain during the training process. This embodiment only needs to collect a small amount of EEG signals from the target domain to calibrate the generator model, which can save a lot of time compared with the traditional collection of a large number of EEG signals of current brain-computer interface users to calibrate the classifier. At the same time, this embodiment has a greater improvement in the accuracy of EEG signal recognition for BCI illiterates compared with traditional classifiers.

The embodiment of the present invention also provides an electroencephalogram signal classification device, comprising:

The first module is used to obtain the target domain to be classified;

The second module is used to input the target domain to be classified into the target generator to obtain the migration data to be classified;

The third module is used to input the migration data to be classified into the subject migration neural network model to obtain the EEG signal classification result;

Among them, the subject transfer neural network model is obtained through the following steps:

Acquire a first motor imagery EEG signal of a gold subject and a second motor imagery EEG signal of a target subject, wherein the gold subject is used to characterize that the first motor imagery EEG signal has a high classification accuracy in the classification model before migration, and the target subject is used to characterize that the second motor imagery EEG signal has a low classification accuracy in the classification model before migration;

Calculate a domain set according to the first motor imagery EEG signal and the second motor imagery EEG signal, where the domain set includes a first source domain and a first target domain;

Inputting the first source domain into a preset neural network model so that the preset neural network model is trained to obtain a subject transfer neural network model;

The target generator is obtained by the following steps:

According to the first source domain, the first target domain and the subject transfer neural network model, the preset generator is trained to obtain the target generator.

The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

The embodiment of the present invention also provides an electroencephalogram signal classification device, comprising:

at least one processor;

at least one memory for storing at least one program;

When at least one program is executed by at least one processor, the at least one processor implements the method shown in FIG. 1 .

The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method shown in FIG. 1 is implemented.

The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims (10)

1. An electroencephalogram signal classification method is characterized by comprising the following steps of:

Acquiring a target domain to be classified;

inputting the target domain to be classified into a target generator to obtain migration data to be classified;

inputting the migration data to be classified into a main migration neural network model to obtain an electroencephalogram signal classification result;

The main migration neural network model is obtained through the following steps:

Acquiring a first motor imagery electroencephalogram of a golden subject and a second motor imagery electroencephalogram of a target subject, wherein the golden subject is used for representing that the classification accuracy of the first motor imagery electroencephalogram in a classification model before migration is high, and the target subject is used for representing that the classification accuracy of the second motor imagery electroencephalogram in the classification model before migration is low;

calculating a domain set according to the first motor imagery electroencephalogram signal and the second motor imagery electroencephalogram signal, wherein the domain set comprises a first source domain and a first target domain;

Inputting the first source domain into a preset neural network model so as to train the preset neural network model and obtain the main migration neural network model;

The target generator is obtained through the following steps:

Training a preset generator according to the first source domain, the first target domain and the main body migration neural network model to obtain the target generator.

2. The method of claim 1, wherein the acquiring the first motor imagery electroencephalogram of the golden subject and the second motor imagery electroencephalogram of the target subject comprises:

collecting the first motor imagery electroencephalogram signal of the golden subject through a brain signal electrode;

the second motor imagery electroencephalogram of the target subject is acquired through a brain signal electrode.

3. The method of claim 1, wherein the computing a domain set from the first motor imagery electroencephalogram signal and the second motor imagery electroencephalogram signal comprises:

calculating a second source domain according to the first motor imagery electroencephalogram signal;

calculating a second target domain according to the second motor imagery electroencephalogram signals;

And calculating the domain set according to the second source domain and the second target domain.

4. A method according to claim 3, wherein said calculating said set of domains from said second source domain and said second target domain comprises:

Calculating a first reference matrix according to the second source domain;

According to the first reference matrix, performing first European alignment on the second source domain to obtain the first source domain;

Calculating a second reference matrix according to the second target domain;

and according to the second reference matrix, performing second European alignment on the second target domain to obtain the first target domain.

5. The method of claim 1, wherein the step of constructing the subject migrating neural network model architecture comprises:

Constructing an input layer, wherein the dimension of an output space of the input layer is 1 dimension;

after the input layer, constructing a conventional convolution layer;

after the conventional convolution layer, constructing a channel-by-channel convolution layer;

After the channel-by-channel convolution layer, constructing a point-by-point convolution layer;

After the point-by-point convolution layer, constructing a first pooling layer;

after the first pooling layer, constructing a spatial convolution layer;

after the space convolution layer, constructing a second pooling layer;

after the second pooling layer, constructing a compression layer;

After the compression layer, constructing an attention layer;

after the attention layer, constructing a time convolution layer;

After the time convolution layer, a feature map interception layer is constructed;

And after the feature map intercepting layer, constructing a full-connection layer.

6. The method of claim 1, wherein training a preset generator according to the first source domain, the first target domain, and the subject migrating neural network model to obtain the target generator comprises:

Inputting the first target domain into the preset generator to obtain first migration data;

Inputting the first source domain and the first migration data into the main migration neural network model to obtain the cross entropy loss and the mean square error corresponding to each key layer output characteristic in the main migration neural network model;

Adding the mean square errors to calculate the style loss;

Adding the style loss and the cross entropy loss to calculate a total loss;

training the preset generator through a random gradient descent algorithm according to the total loss to obtain the target generator.

7. The method of claim 4, wherein said calculating a first reference matrix from said second source domain comprises:

according to the second source domain, calculating a first reference matrix through a reference matrix calculation formula, wherein the reference matrix calculation formula is as follows:

In the method, in the process of the invention, For the first reference matrix, x _i is the ith electroencephalogram signal in the second source domain, and n is the total number of electroencephalogram signals in the second source domain.

8. An electroencephalogram signal classifying device, characterized by comprising:

The first module is used for acquiring a target domain to be classified;

the second module is used for inputting the target domain to be classified into a target generator to obtain migration data to be classified;

The third module is used for inputting the migration data to be classified into a main migration neural network model to obtain an electroencephalogram signal classification result;

The main migration neural network model is obtained through the following steps:

Acquiring a first motor imagery electroencephalogram of a golden subject and a second motor imagery electroencephalogram of a target subject, wherein the golden subject is used for representing that the classification accuracy of the first motor imagery electroencephalogram in a classification model before migration is high, and the target subject is used for representing that the classification accuracy of the second motor imagery electroencephalogram in the classification model before migration is low;

calculating a domain set according to the first motor imagery electroencephalogram signal and the second motor imagery electroencephalogram signal, wherein the domain set comprises a first source domain and a first target domain;

Inputting the first source domain into a preset neural network model so as to train the preset neural network model and obtain the main migration neural network model;

The target generator is obtained through the following steps:

Training a preset generator according to the first source domain, the first target domain and the main body migration neural network model to obtain the target generator.

9. An electroencephalogram signal classifying device, characterized by comprising:

at least one processor;

At least one memory for storing at least one program;

the at least one program, when executed by the at least one processor, causes the at least one processor to implement the method of any of claims 1-7.

10. A computer readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method of any one of claims 1 to 7.