CN117892175A - SNN multi-mode target identification method, system, equipment and medium - Google Patents

Abstract

The invention provides an SNN multi-mode target identification method, an SNN multi-mode target identification system, SNN multi-mode target identification equipment and an SNN multi-mode target identification medium, and relates to the field of artificial intelligence, wherein the SNN multi-mode target identification method comprises the following steps: extracting key features in the multi-mode information through a feature extractor; adjusting the time scale of key features corresponding to different single-mode information, simulating forward propagation of each time-aligned single-mode key feature and a multi-mode classifier parameter through matrix multiplication to obtain single-mode output and multi-mode output, evaluating the contribution ratio of different single-mode outputs to a target task, and determining a mode adjustment factor; and calculating cross entropy loss corresponding to the single-mode output and the multi-mode output, dynamically adjusting a loss function according to the mode adjustment factor and the cross entropy loss, and determining the final loss of the impulse neural network so as to identify the multi-mode target of the impulse neural network. The method can effectively solve the problems of unbalanced multi-mode convergence and unmatched time scales, and remarkably improves the overall performance of the multi-mode model in the target recognition task.

Description

A SNN multimodal target recognition method, system, device and medium

Technical Field

The present invention relates to the field of artificial intelligence, and in particular to a SNN multimodal target recognition method, system, device and medium.

Background technique

In today's era of information explosion, multimodal learning has become an important branch of artificial intelligence. Multimodal learning aims to build models with better performance and stronger robustness by integrating information from different modalities. Multimodal models use multiple information sources such as text, images, and sounds, and are applied in many complex tasks such as visual question answering, sentiment analysis, medical image processing, and cross-modal retrieval. With the continuous development of spiking neural network (SNN) training algorithms, the applicability of SNN in various application scenarios has gradually increased, and its high biological interpretability has also shown certain advantages in dealing with multimodal problems. On the one hand, the neurons in SNN simulate the dynamic behavior of real neurons, which can not only learn the spatial characteristics of multimodal data, but also efficiently utilize the temporal information of the modality, especially suitable for processing video, speech and other data with time dimension. On the other hand, the event-driven characteristics of SNN make it efficient when processing sparse modal data, which can significantly reduce the energy consumption of multimodal processing. Researchers have used SNN to process event, visual, audio and other data to complete tasks such as lip reading, speech, and object recognition. Reference 1 uses the information fusion of dynamic vision sensor (DVS) and dynamic audio sensor (DAS) to realize lip reading recognition task, and filters noise events by calculating the cross-correlation between different modal event streams to achieve inter-modal alignment and solve the time offset problem in audio-visual event stream recording. In addition, Reference 2 borrows from gating logic and uses excitatory and inhibitory synaptic connections to form a supermodal layer to achieve cross-modal information coupling. There is also research 3 that uses convolutional pulse neural network and recurrent pulse neural network to process DVS and DAS data respectively, and draws on the attention mechanism to weight the output of the unimodal network to achieve selective fusion. The multimodal algorithm based on SNN has significantly improved the overall performance of the model in various tasks.

Compared with unimodal models, the complementary information of different modes in multimodal models can usually help optimize the model effect and improve the robustness of the model to noise and missing data. However, in the existing SNN multimodal algorithms, there are still problems of modality imbalance and time scale mismatch. These challenges limit the application of SNN in multimodal tasks.

The modality imbalance problem refers to the situation that in the multimodal joint learning scenario, the performance of the optimal single-modal model is better than that of the multimodal model, and as the number of input streams increases, the performance of the multimodal model decreases more significantly; even if the effect of the joint training of the multimodal model is better than that of the single-modal model, its single-modal branch is difficult to exceed the performance of the single-modal model trained alone. The multimodal imbalance problem makes it difficult for the single-modal branch to fully converge, thereby weakening the ability of the multimodal model to utilize different modal information. From the perspective of the optimization process, the multimodal imbalance problem is mainly caused by the difference in convergence speed of different modalities. When the multimodal model is jointly trained, if a unified learning rate is used, the modality with a faster convergence speed may dominate the entire optimization process, while other modalities cannot be fully trained. Therefore, most of the existing algorithms for the modality imbalance problem adjust the learning process by monitoring the convergence of different modalities, aiming to slow down the optimization speed of the dominant modality to alleviate the underfitting problem of other modalities, thereby improving the overall performance of the multimodal model. For example, Reference 4 first proposed a gradient hybrid offline adjustment algorithm, which adjusts the loss function of each modality by calculating the ratio of overfitting degree to generalization ability. However, this algorithm can only achieve offline adjustment and cannot be optimized in real time. Reference 5 introduces an adaptive tracking factor to adjust the learning rate of each modality in real time, so that the modality closer to convergence has a lower learning rate. In order to measure the learning dependence of the multimodal model on each modality, Reference 6 proposed the concept of conditional utilization and optimized the conditional utilization to the conditional learning speed to achieve online adjustment. Reference 7 proposed an online gradient adjustment algorithm, which dynamically monitors the difference in contribution of different modalities to the learning objective during the training phase, and then uses the difference to adaptively adjust the gradient, and introduces Gaussian noise to achieve generalization enhancement. However, when the accuracy difference between modalities is large, the online gradient adjustment algorithm cannot adjust the gradient well, which limits the improvement of the multimodal fusion effect.

In the process of applying SNN to handle multimodal tasks, in addition to the modal imbalance problem caused by the difference in convergence speed of different modalities, the characteristics of time scale mismatch and different event sparsity of different modalities in SNN also bring greater challenges to the full utilization of multimodal information. The main reason for the time scale mismatch problem is that the information of different modalities of SNN has different sensitivity to time. The static visual modality itself does not involve the time dimension. Using static visual information repeated in multiple time steps may lead to information redundancy; dynamic visual modal data is inherently high in time resolution and dynamic change characteristics, and requires a certain time step to capture the rapid and fine-grained changes in the event stream; while the auditory modality usually contains relatively rich time dimension information, and requires more time steps to accurately capture the changes in sound, especially when processing complex audio scenes. In existing SNN multimodal algorithms, the same time step is usually uniformly assigned to all modalities, and the time scale mismatch problem between modalities is not effectively handled. This approach will lead to the over-expression or loss of some modal information, further increasing the difficulty of solving the modal imbalance problem.

references:

Reference 1. Li, X., Neil, D., Delbruck, T. & Liu, S.-C. Lip reading deep network exploiting multi-modal spiking visual and auditory sensors. in 2019 IEEE International Symposium on Circuits and Systems (ISCAS) 1-5 (IEEE, 2019).

Reference 2. Wysoski, S.G., Benuskova, L. & Kasabov, N. Brain-like evolving spiking neural networks for multimodal information processing. in Brain-Inspired Information Technology 15-27 (Springer, 2010).

Reference 3. Liu, Q., Xing, D., Feng, L., Tang, H. & Pan, G. Event-based multimodal spiking neural network with attention mechanism. in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 8922-8926 (IEEE, 2022).

Reference 4. Wang, W., Tran, D. & Feiszli, M. What makes training multi-modal classification networks hard? in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 12695-12705(2020).

Reference 5. Sun, Y., Mai, S. & Hu, H. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Process. Lett. 28, 1650-1654 (2021).

Reference 6. Wu, N., Jastrzebski, S., Cho, K. & Geras, K. J. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. in International Conference on Machine Learning 24043-24055 (PMLR, 2022).

Reference 7. Peng, X., Wei, Y., Deng, A., Wang, D. & Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. in Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition 8238-8247 (2022).

Summary of the invention

The purpose of the present invention is to provide a SNN multimodal target recognition method, system, device and medium to solve the problems of multimodal convergence imbalance and time scale mismatch.

To achieve the above object, the present invention provides the following solutions:

A SNN multimodal target recognition method, comprising:

Extracting key features from the multimodal information by a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor;

Using a convolutional time alignment module to adjust the time scale of key features corresponding to different unimodal information, and determine the unimodal key features of time alignment; the convolutional time alignment module includes a visual convolutional time alignment module and an auditory convolutional time alignment module;

Simulating forward propagation by matrix multiplication of each of the time-aligned unimodal key features and the multimodal classifier parameters to obtain unimodal output and multimodal output, and evaluating the contribution ratio of different unimodal outputs to the target task according to the unimodal output and the multimodal output to determine the modality adjustment factor;

The cross entropy loss corresponding to the unimodal output and the multimodal output is determined according to the unimodal output and the multimodal output, and the loss function is dynamically adjusted according to the modality adjustment factor and the cross entropy loss to determine the final loss of the spiking neural network to identify the multimodal target of the spiking neural network; the final loss is the optimization target of the spiking neural network.

Optionally, the key features in the multimodal information are extracted by a feature extractor, which also includes:

Encoding the visual information and the auditory information respectively to generate encoded visual information and encoded auditory information;

A multimodal data set is constructed according to the encoded visual information and the encoded auditory information; the multimodal data set includes multimodal information.

Optionally, key features in the multimodal information are extracted by a feature extractor, specifically including:

After the visual information passes through a visual feature extractor consisting of three pulse fully connected layers, visual features of _Te time steps are generated;

The auditory information is passed through an auditory feature extractor consisting of a three-layer recurrent pulse network to generate auditory features of _Ta time steps.

Optionally, a convolution time alignment module is used to adjust the time scale of key features corresponding to different unimodal information to determine the time-aligned unimodal key features, specifically comprising: the visual convolution time alignment module includes T 1×1×T _e convolution kernels; the auditory convolution time alignment module includes T 1×1×T _a convolution kernels;

Processing the visual features using a visual convolutional time alignment module to generate visual alignment features of T time steps;

The auditory features are processed using an auditory convolutional time alignment module to generate auditory alignment features of T time steps.

Optionally, each of the time-aligned unimodal key features is respectively subjected to matrix multiplication to simulate forward propagation with the multimodal classifier parameters to obtain unimodal output and multimodal output, and the contribution ratio of different unimodal outputs to the target task is evaluated according to the unimodal output and the multimodal output to determine the modality adjustment factor, specifically including:

According to the formula Determine multimodal output; wherein, /> is the multimodal output; W is the fully connected layer parameter; /> It is a visual alignment feature; /> is the auditory alignment feature; b is the bias term;

According to the formula Determine the visual modality output; wherein, /> is the visual modality output; _We is the visual classifier parameter;

According to the formula Determine the auditory modality output; wherein, /> is the auditory modal output; Wa _is the auditory classifier parameter;

Inputting the visual modality output and the auditory modality output into a softmax function to determine a visual modality score and an auditory modality score;

Determining a visual contribution ratio of the visual modality to the target task and an auditory contribution ratio of the auditory modality to the target task according to the visual modality score and the auditory modality score;

A modality adjustment factor is determined according to the visual contribution ratio and the auditory contribution ratio.

Optionally, the cross entropy loss corresponding to the unimodal output includes a cross entropy loss corresponding to a visual modality and a cross entropy loss corresponding to an auditory modality;

The cross entropy loss _Le corresponding to the visual modality is: Where C is the number of sample types, j is the sample type serial number, N is the number of samples; _yi represents the true category label of the i-th sample;

The cross entropy loss _La corresponding to the auditory modality is:

The cross entropy loss _Lf corresponding to the multimodal output is:

Optionally, the final loss L is: Among them, β is a hyperparameter,/> is the auditory regulation factor,/> is the visual adjustment factor.

A SNN multimodal target recognition system, comprising:

A feature extraction module, used to extract key features from multimodal information through a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor;

The convolutional time alignment module is used to adjust the time scale of the key features corresponding to different unimodal information and determine the unimodal key features for time alignment;

A simulated feedforward module is used to simulate forward propagation of each of the time-aligned unimodal key features with the multimodal classifier parameters through matrix multiplication to obtain unimodal output and multimodal output, and to evaluate the contribution ratio of different unimodal outputs to the target task according to the unimodal output and the multimodal output, and determine the modal adjustment factor;

An online loss adjustment module is used to determine the cross entropy loss corresponding to the unimodal output and the multimodal output according to the unimodal output and the multimodal output, and dynamically adjust the loss function according to the modality adjustment factor and the cross entropy loss to determine the final loss of the pulse neural network; the final loss is the optimization target of the pulse neural network.

An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program so that the electronic device executes the above-mentioned SNN multimodal target recognition method.

A computer-readable storage medium stores a computer program, which implements the above-mentioned SNN multimodal target recognition method when executed by a processor.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

The present invention modifies the traditional multimodal model loss function and introduces an additional cross entropy loss corresponding to the unimodal output. The cross entropy loss corresponding to the unimodal output is a supervised loss for unimodal feature extraction, so that the entire invention not only focuses on the effect of multimodal classification, but also focuses on the characterization effect of the unimodal feature extractor, and the supervised model fully learns the information within the unimodal. In addition, in view of the problem that some algorithms can only solve the modal imbalance problem offline, the present invention draws on the method of evaluating the contribution of a unimodal to the target in the online gradient adjustment algorithm, introduces a modal adjustment factor before the unimodal loss, and reduces the learning rate of the dominant mode with faster convergence online, thereby alleviating the modal imbalance problem. The multimodal model involved in the present invention is an SNN multimodal model.

The present invention also proposes a convolutional time alignment module, which adjusts the time scale of key features corresponding to different unimodal information by adding a fixed number of convolution kernels after the feature extractor of each modality, determines the unimodal key features of time alignment, and adapts to the time step differences of different modal data.

The present invention adopts two key technologies, online loss adjustment and convolutional time alignment, to effectively solve the two major problems of multimodal convergence imbalance and time scale mismatch, thereby significantly improving the overall performance of the multimodal model in target recognition tasks.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

FIG1 is a flow chart of the SNN multimodal target recognition method provided by the present invention;

FIG2 is a structural diagram of the SNN multimodal target recognition system provided by the present invention.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The purpose of the present invention is to provide a SNN multimodal target recognition method, system, device and medium, which can effectively solve the problems of multimodal convergence imbalance and time scale mismatch, and significantly improve the overall performance of the multimodal model in target recognition tasks.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

Embodiment 1

As shown in FIG1 , the present invention provides a SNN multimodal target recognition method, comprising:

Step 101: extract key features from multimodal information through a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor.

In practical applications, the step 101 also includes encoding preprocessing of the input multimodal information to meet the input requirements of multiple modal information.

Visual information encoding: In this multimodal target recognition system, the visual information modality selects the N-MNIST dataset as input. This dataset is recorded by an event sensor by displaying the original handwritten images in the MNIST dataset on a slow-moving display. The N-MNIST dataset includes 60,000 training samples and 10,000 test samples, each with a size of 34*34 pixels. When encoding visual event information, the following steps are taken: the event stream is divided into _Te segments according to the total number N, and the event stream in each segment is integrated into a frame. In this way, a visual input of _Te time steps is finally formed, where the input format of each frame is defined as (t, p, h, w), t represents the current time step, p represents the event polarity, and (h, w) represents the size of the frame. Finally, the visual input 2312× _Te is obtained. In this experiment, the time step _Te is set to 5.

Auditory information encoding: In order to maintain the consistency of multimodal categories, the auditory modality in this multimodal target recognition system uses 0-9 digital audio samples from the Google voice command dataset. The dataset contains a total of 38,908 samples, which are divided into training and test sets in a ratio of 9:1. In the process of preprocessing the auditory signal, each auditory sample is first converted into a Mel spectrogram. The number of frames of the spectrogram is equal to the given auditory modality time step _Ta . The third-order difference is calculated based on the spectrogram to form 3-channel input data, and it is normalized across time steps. Finally, the auditory input format is 120× _Ta . In this experiment, the time step _Ta is set to 20, and the time step T of the subsequent multimodal fusion module is also set to 20.

Multimodal dataset construction: First, randomly select the category label y ⁱ of the sample from 0 to 9, and then randomly sample the data corresponding to the category label in the visual modality and auditory modality to obtain visual data and auditory data/> Through this process, the training set D = {x ⁱ , y ⁱ } _{{i = 1, 2, ..., N}} is constructed. The same method is used to construct the test set. In the experimental verification of the present invention, the size of the training set is set to 10,000 samples, while the size of the test set is set to 2,000 samples. This step is intended to ensure that the system can effectively process and identify sample data from different modalities, thereby verifying the practicality and effectiveness of the system.

Perform feature extraction on the encoded information to obtain visual features and auditory features/> The feature extraction operation includes the following steps:

In the present invention, the visual feature extractor _φe is composed of three layers of pulse fully connected layers of 2312-800-128, and its parameters are represented by the learnable network parameters _θe of the visual feature extractor. After the visual information passes through the feature extractor, the visual features of _Te time steps are obtained. The formula is as follows:

The auditory feature extractor _φa is composed of a three-layer recurrent pulse network of 120-240-128, and its parameters are represented by the learnable network parameters _θa of the auditory feature extractor. After the auditory information passes through the feature extractor, the auditory features of T _a time steps are obtained. The formula is as follows:

Step 102: Use a convolutional time alignment module to adjust the time scale of key features corresponding to different unimodal information to determine the time-aligned unimodal key features; the convolutional time alignment module includes a visual convolutional time alignment module and an auditory convolutional time alignment module.

In practical applications, the alignment of modal features with different time scales is realized, and finally the visual alignment feature is obtained. and auditory alignment features/> The convolution time alignment operation includes the following steps:

Temporal alignment of visual features: The visual features are processed by the visual convolutional temporal alignment module to obtain visual alignment features of T time steps. The visual convolutional temporal alignment module consists of T 1×1×T _e convolution kernels, and the formula is as follows:

in, Represents the model parameters of the visual convolutional temporal alignment module.

Time alignment of auditory features: The auditory features are processed through the auditory convolution time alignment module to obtain auditory alignment features of T time steps. The auditory convolution time alignment module consists of T 1×1×T _a convolution kernels, and the formula is as follows:

in, Representation of model parameters for the auditory convolutional time alignment module.

The time alignment operation in this step ensures the consistency of different modal features on the time scale, providing support for the effective operation of the multimodal target recognition system.

Step 103: Simulate forward propagation by matrix multiplication of each of the time-aligned unimodal key features and the multimodal classifier parameters to obtain unimodal output and multimodal output, and evaluate the contribution ratio of different unimodal outputs to the target task based on the unimodal output and the multimodal output to determine the modal adjustment factor.

In practical applications, multi-modal output and single-modal output are obtained by simulating the feedforward process, and the target contribution of different modes is calculated, which includes the following steps:

Calculate multimodal output: The model classifier consists of a 256-10 pulse fully connected layer. W is the fully connected layer parameter, b is the bias term, and the multimodal output is first obtained by feedforward. The calculation formula is:

Simulate single-modal feedforward: In order to estimate the contribution of different modalities to the target, the original classifier is divided into two parts: a 128-10 visual classifier and a 128-10 auditory classifier. W = [ _We , _Wa ]. _We and _Wa are the parameters of the visual and auditory classifiers, respectively. Simulate visual and auditory feedforward classification to obtain visual and auditory modal outputs

Multimodal contribution ratio calculation: Output single mode They are respectively passed into the softmax function to obtain different modal scores s _e , s _a . The calculation formulas are:

Among them, C represents the number of sample categories, j is the sample type serial number; _yi is the true category label of the i-th sample, is an indicator function. When j is equal to the true category _yi of sample i, its value is 1; otherwise, it is 0. It is used to ensure that only the correct category score is counted. From the formula, that is, when j = _yi , /> The jth component of is added to _Le . Then, the visual and auditory proportions of the current batch of samples are calculated based on the modal scores. and/> _Bt represents the sample set of the current batch, and the calculation formulas are:

Modal adjustment factor calculation: If the proportional factor of the current mode is greater than 1, it means that the current mode contributes more to the overall goal and is the dominant mode. The optimization process is slowed down by adjusting the factor; otherwise, it means that the current mode is a weak mode and is not suppressed. This step obtains the modal adjustment factor through the following formula: Where u = {e, a}, indicating a specific mode.

Among them, α is a hyperparameter used to control the degree of regulation. The larger α is, the stronger the suppression of the dominant mode is.

Step 104: Determine the cross entropy loss corresponding to the unimodal output and the multimodal output according to the unimodal output and the multimodal output, and dynamically adjust the loss function according to the modality adjustment factor and the cross entropy loss to determine the final loss of the pulse neural network to identify the multimodal target of the pulse neural network; the final loss is the optimization target of the pulse neural network.

In practical applications, the cross entropy loss corresponding to the unimodal output and the multimodal output is calculated and regulated according to the adjustment factor, including the following steps:

Loss calculation: This step is based on the visual modality output Auditory modality output/> and multimodal output/> Calculate the corresponding cross entropy losses _Le , _La and _Lf respectively. The calculation formulas of each loss are as follows.

Loss adjustment: Calculate the final loss L of the model based on the weighted target contribution.

Among them, β is a manually set hyperparameter, which represents the proportion of single-modal supervision loss in the overall loss function. is the auditory regulation factor,/> is the visual adjustment factor. During training, the loss L is used as the overall optimization target of the model.

The present invention has the following advantages:

(1) Innovation: This paper proposes a solution to the multimodal imbalance problem in pulse neural networks for the first time, effectively overcoming the problem that multimodal joint training may lead to a decrease in the feature extraction ability of a single modal model.

(2) Efficiency: By integrating two key technologies, the online loss adjustment module and the convolutional time alignment module, the present invention achieves comprehensive optimization of feature extractors of different modalities. In the object recognition task on the example dataset, the accuracy is improved by 2.3%, which verifies the efficiency of the present invention in multimodal learning.

(3) Online adjustment: The online loss adjustment module in the present invention relies on a single-mode feedforward process to monitor the contribution of different modes to the target, effectively weakening the dominant role of the strong mode in the overall optimization process, while alleviating the inhibition of optimization of other modes, thus achieving the effect of real-time loss adjustment.

(4) Time synchronization: The convolutional time alignment module in the present invention allows the system to process multiple modalities with different time scales, realize adaptive learning of modal information at different time scales, and achieve effective alignment of modal time scales, with good scalability.

(5) Information complementarity: The target recognition task experiment conducted by the present invention on the example data set shows that the auditory modality accuracy is improved by 40.65% compared with the original system, and exceeds the training accuracy of the auditory unimodal model. This result proves that the adjustment algorithm of the present invention can efficiently utilize cross-modal information and achieve information complementarity.

Embodiment 2

In order to execute the method corresponding to the above-mentioned embodiment 1 to achieve the corresponding functions and technical effects, a SNN multimodal target recognition system is provided below.

As shown in FIG2 , a SNN multimodal target recognition system includes:

The feature extraction module is used to extract key features in multimodal information through a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor.

The convolutional time alignment module is used to adjust the time scale of the key features corresponding to different unimodal information and determine the unimodal key features for time alignment.

The simulated feedforward module is used to simulate the forward propagation of each of the time-aligned unimodal key features with the multimodal classifier parameters through matrix multiplication to obtain unimodal output and multimodal output, and evaluate the contribution ratio of different unimodal outputs to the target task based on the unimodal output and the multimodal output to determine the modal adjustment factor.

An online loss adjustment module is used to determine the cross entropy loss corresponding to the unimodal output and the multimodal output according to the unimodal output and the multimodal output, and dynamically adjust the loss function according to the modality adjustment factor and the cross entropy loss to determine the final loss of the pulse neural network; the final loss is the optimization target of the pulse neural network.

Embodiment 3

An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program so that the electronic device executes the SNN multimodal target recognition method described above.

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned SNN multimodal target recognition method.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims (10)

1. A SNN multimodal target recognition method, characterized by comprising: Extracting key features from the multimodal information by a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor; Using a convolutional time alignment module to adjust the time scale of key features corresponding to different unimodal information, and determine the unimodal key features of time alignment; the convolutional time alignment module includes a visual convolutional time alignment module and an auditory convolutional time alignment module; Simulating forward propagation by matrix multiplication of each of the time-aligned unimodal key features and the multimodal classifier parameters to obtain unimodal output and multimodal output, and evaluating the contribution ratio of different unimodal outputs to the target task according to the unimodal output and the multimodal output to determine the modality adjustment factor; The cross entropy loss corresponding to the unimodal output and the multimodal output is determined according to the unimodal output and the multimodal output, and the loss function is dynamically adjusted according to the modality adjustment factor and the cross entropy loss to determine the final loss of the spiking neural network to identify the multimodal target of the spiking neural network; the final loss is the optimization target of the spiking neural network. 2. The SNN multimodal target recognition method according to claim 1 is characterized in that the key features in the multimodal information are extracted by a feature extractor, and the method also includes: Encoding the visual information and the auditory information respectively to generate encoded visual information and encoded auditory information; A multimodal data set is constructed according to the encoded visual information and the encoded auditory information; the multimodal data set includes multimodal information. 3. The SNN multimodal target recognition method according to claim 1 is characterized in that the key features in the multimodal information are extracted by a feature extractor, specifically comprising: After the visual information passes through a visual feature extractor consisting of three pulse fully connected layers, visual features of _Te time steps are generated; The auditory information is passed through an auditory feature extractor consisting of a three-layer recurrent pulse network to generate auditory features of _Ta time steps. 4. The SNN multimodal target recognition method according to claim 1 is characterized in that the time scale of the key features corresponding to different single-modal information is adjusted by using a convolution time alignment module to determine the single-modal key features of time alignment, specifically comprising: the visual convolution time alignment module includes T 1×1×T _e convolution kernels; the auditory convolution time alignment module includes T 1×1×T _a convolution kernels; Processing the visual features using a visual convolutional time alignment module to generate visual alignment features of T time steps; The auditory features are processed using an auditory convolutional time alignment module to generate auditory alignment features of T time steps. 5. The SNN multimodal target recognition method according to claim 4 is characterized in that each of the time-aligned unimodal key features is respectively simulated with the multimodal classifier parameters by matrix multiplication to simulate forward propagation to obtain unimodal output and multimodal output, and the contribution ratio of different unimodal outputs to the target task is evaluated according to the unimodal output and the multimodal output to determine the modal adjustment factor, which specifically includes: According to the formula Determine multimodal output; wherein, /> is the multimodal output; W is the fully connected layer parameter; /> It is a visual alignment feature; /> is the auditory alignment feature; b is the bias term; According to the formula Determine the visual modality output; wherein, /> is the visual modality output; _We is the visual classifier parameter; According to the formula Determine the auditory modality output; wherein, /> is the auditory modal output; Wa _is the auditory classifier parameter; Inputting the visual modality output and the auditory modality output into a softmax function to determine a visual modality score and an auditory modality score; Determining a visual contribution ratio of the visual modality to the target task and an auditory contribution ratio of the auditory modality to the target task according to the visual modality score and the auditory modality score; A modality adjustment factor is determined according to the visual contribution ratio and the auditory contribution ratio. 6. The SNN multimodal target recognition method according to claim 5, characterized in that the cross entropy loss corresponding to the single modal output includes the cross entropy loss corresponding to the visual modality and the cross entropy loss corresponding to the auditory modality; The cross entropy loss _Le corresponding to the visual modality is: Where C is the number of sample types, j is the sample type serial number, N is the number of samples; _yi represents the true category label of the i-th sample; The cross entropy loss _La corresponding to the auditory modality is: The cross entropy loss _Lf corresponding to the multimodal output is: 7. The SNN multimodal target recognition method according to claim 6, characterized in that the final loss L is: Among them, β is a hyperparameter,/> is the auditory regulation factor,/> is the visual adjustment factor. 8. A SNN multimodal target recognition system, characterized by comprising: A feature extraction module, used to extract key features from multimodal information through a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor; The convolutional time alignment module is used to adjust the time scale of the key features corresponding to different unimodal information and determine the unimodal key features for time alignment; A simulated feedforward module is used to simulate forward propagation of each of the time-aligned unimodal key features with the multimodal classifier parameters through matrix multiplication to obtain unimodal output and multimodal output, and to evaluate the contribution ratio of different unimodal outputs to the target task according to the unimodal output and the multimodal output, and determine the modal adjustment factor; An online loss adjustment module is used to determine the cross entropy loss corresponding to the unimodal output and the multimodal output according to the unimodal output and the multimodal output, and dynamically adjust the loss function according to the modality adjustment factor and the cross entropy loss to determine the final loss of the pulse neural network; the final loss is the optimization target of the pulse neural network. 9. An electronic device, characterized in that it includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the SNN multimodal target recognition method as described in any one of claims 1-7. 10. A computer-readable storage medium, characterized in that it stores a computer program, and when the computer program is executed by a processor, it implements the SNN multimodal target recognition method as described in any one of claims 1-7.