CN111310836B - A defense method and defense device for an integrated model of voiceprint recognition based on a spectrogram - Google Patents

Abstract

The invention discloses a defense method for an integrated model of voiceprint recognition based on a spectrogram, comprising: (1) collecting an audio file, and converting the audio file into a spectrogram, and the spectrogram is used as a benign sample; (2) Use benign samples to train multiple voiceprint recognition models to obtain multiple trained voiceprint recognition models; (3) use a voting mechanism to obtain better multiple voiceprint recognition models from multiple trained voiceprint recognition models Carry out integration to form an integrated voiceprint recognition model, and use benign samples to retrain the integrated voiceprint recognition model; (4) collect cuckoo search algorithms to attack multiple voiceprint recognition models respectively to generate adversarial samples; (5) use adversarial samples and benign The sample retrains the integrated voiceprint recognition model obtained in step (3) to obtain an integrated voiceprint recognition model that can resist attacks; Figure for defensive identification.

Description

A defense method and defense device for an integrated model of voiceprint recognition based on a spectrogram

technical field

The invention belongs to the field of information security research, and in particular relates to a defense method and a defense device of an integrated model of voiceprint recognition based on a spectrogram.

Background technique

Since everyone's vocal organs—tongue, teeth, lungs, etc., are very different in size and shape, everyone's voice is different, and their spectrograms are different. In fact, everyone's voice has a Unique identity information, voiceprint recognition is to use this characteristic of voice to identify the identity of the speaker. Voiceprint recognition is a kind of biometric technology, which is divided into text-related and text-independent voiceprint recognition. Text-independent voiceprint recognition: The voiceprint recognition system does not have any requirements for the voice text content, and the speaker's speech content is relatively free. Text-related voiceprint recognition: Refers to a speaker recognition system that requires users to pronounce according to pre-specified content. The text-related voiceprint recognition model requires the user to pronounce according to the specified text. Once the user's pronunciation is wrong, the identity cannot be recognized, and the application is narrow. The text-independent voiceprint recognition model has no requirements on the content of the user's voice, and is easy to recognize. It has a wide range of applications, but it is difficult to implement.

The deep neural network can make full use of the correlation between speech features, combine the speech features of consecutive frames and train them, so that the recognition rate of the voiceprint recognition system can be greatly improved. While the voiceprint recognition system based on deep neural network improves the recognition accuracy and brings convenience to people, it also brings corresponding risks. Deep neural networks are vulnerable to adversarial attacks in the form of adding subtle perturbations to the input data. After obtaining the characteristics of a target speaker, the attacker can add carefully calculated perturbations to the audio of a speaker, so that the generated adversarial samples are voiceprinted. The recognition model wrongly identifies the target speaker, which brings great security risks to the voiceprint recognition system and personal property safety.

Existing voiceprint recognition attack methods are mainly divided into white-box and black-box attacks. The white-box attack is carried out by the attacker when the internal parameters of the model are known. The gradient of the model with respect to noise is calculated through backpropagation, and the noise to be added is continuously optimized through iteration to achieve the purpose of generating adversarial samples. The black-box attack is carried out by the attacker without knowing the model parameters. Optimization algorithms such as genetic algorithm and particle swarm optimization algorithm can be used to optimize the disturbances that need to be added to generate adversarial samples. Both white-box and black-box attacks can attack the voiceprint recognition system, making the voiceprint recognition system mistakenly identify the adversarial sample as the target speaker.

Contents of the invention

Aiming at the security problems of low accuracy, poor robustness, and vulnerability to adversarial sample attacks in the current voiceprint recognition system, the present invention provides a defense method and defense device based on a voiceprint recognition integrated model based on a spectrogram. The method and defense device can improve the accuracy and robustness of voiceprint recognition, resist the attack of adversarial samples, and improve the security of voiceprint recognition.

Technical scheme of the present invention is:

A defense method for an integrated model of voiceprint recognition based on a spectrogram, comprising the following steps:

(1) collect the audio file, and convert the audio file into a spectrogram, which is used as a benign sample;

(2) Using benign samples to train a plurality of image recognition models, so that the image recognition models can achieve the effect of voiceprint recognition, thereby obtaining a plurality of trained image-based voiceprint recognition models;

(3) Using a voting mechanism to integrate a plurality of image-based voiceprint recognition models trained in step (2) to form a voiceprint recognition integration model, and to retrain the voiceprint recognition integration model with benign samples;

(4) Collect the cuckoo search algorithm to attack multiple voiceprint recognition models respectively, generate adversarial samples, and convert the adversarial samples into spectrograms as malicious samples;

(5) retraining the integrated voiceprint recognition model based on the image obtained in step (3) by using the malignant samples and the benign samples to obtain an integrated voiceprint recognition model capable of resisting attacks;

(6) Use the integrated voiceprint recognition model obtained in step (5) to perform defense recognition on the spectrogram corresponding to the audio file.

Preferably, the specific steps of converting an audio file into a spectrogram are:

Frame the audio, and perform short-time Fourier transform after windowing each frame of speech signal;

Calculate the power spectrum of the short-time Fourier transform result, and normalize the power spectrum to obtain a spectrogram, and form a benign sample with the spectrogram and the corresponding speaker.

Preferably, the image recognition model adopts VGG16 or VGG19.

Preferably, the specific process of using benign samples to train multiple voiceprint recognition models is:

Preprocess the spectrogram, set the spectrogram size to 224×224×3, and obtain the spectrogram sample;

The confidence degree output by the spectrogram sample x _i through the voiceprint recognition model is y _ipre , using cross entropy as the loss function, and using the loss function L(xi ₎ to optimize the parameters of the voiceprint recognition model;

L(x _i )＝-[y _i logy _ipre +(1-y _i )log(1-y _ipre )]

Use the spectrogram in the test set to test the accuracy of the trained voiceprint recognition model. When the recognition accuracy does not meet the requirements, retrain the voiceprint recognition model until the recognition accuracy meets the requirements.

The concrete process of step (3) is:

Using the voting mechanism to integrate multiple image-based voiceprint recognition models to obtain an integrated voiceprint recognition model;

Before voting, the prediction confidence returned by each voiceprint recognition model is converted into a prediction category, that is, the category mark corresponding to the highest confidence level is used as the prediction result of the voiceprint recognition model;

After each voiceprint recognition model obtains the prediction result of the spectrogram sample, if a certain prediction category receives more than half of the voiceprint recognition model votes, the prediction category is the prediction result of the voiceprint recognition integrated model;

Then use benign samples to train the integrated model of voiceprint recognition, and test it with the test set to improve the integrated model of voiceprint recognition.

A defense device based on a voiceprint recognition integrated model of a spectrogram, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, the computer processor When the computer program is executed, the above-mentioned defense method based on the voiceprint recognition integrated model of the spectrogram is realized.

In the present invention, based on the possible defects of the above voiceprint recognition system and the limitations of existing attack methods, a method of converting speech into a spectrogram and using the spectrogram to train an image recognition model is studied to achieve the goal of voiceprint recognition. Purpose. And integrate multiple trained image recognition models together, while improving the accuracy of the model, this special voiceprint recognition model can resist the attack of adversarial samples, and further improve the defense ability of the model through adversarial training, realizing the dialogue box or defense against black-box attacks.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is the flow chart of the defense method of the integrated model of voiceprint recognition based on the spectrogram provided by the embodiment;

Fig. 2 is a schematic structural diagram of obtaining an adversarial example provided by the embodiment;

Fig. 3 is a schematic diagram of retraining the integrated voiceprint recognition model provided by the embodiment.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, and do not limit the protection scope of the present invention.

Referring to Figures 1 to 3, the defense method of the voiceprint recognition integrated model based on the spectrogram provided by the embodiment includes the following steps:

1) Prepare a data set for voiceprint recognition model training, using the train-clean-100 data set in the Librispeech voice data set as the data set. Each file of the train-clean-100 data stores the audio of different speakers, so a folder corresponds to a speaker, and the file name is actually a label;

2) Preprocess the audio files in each folder, convert them into spectrograms, and save them in corresponding folders. The file name is the corresponding class mark of the spectrogram, that is, the identity of the speaker. Divide it into a training set and a test set according to a certain ratio. The specific process is as follows:

Step1: For each audio file x(n) in the train-claen-100 data set, divide it into frames, and the length of each frame is 25ms. During this time period, the speech signal is regarded as a stable state. A window function is added to the audio signal after framing to avoid signal leakage of high frequency parts. After framing and windowing, short-time Fourier transform is performed on the speech signal:

Where k∈{0,1,...N-1}, where N represents the number of sampling points contained in a frame of audio file, and w(n-m) is a window function sliding along the time axis.

Step2: Calculate its power spectrum according to X(n,k) as

P(n,k)＝|X(n,k)| ² (2)

Step3: Since there is a large amount of non-zero noise in the silent segment in the speech, the spectrogram is processed with the maximum-minimum normalization method. After the normalization process, the lightness and darkness corresponding to the spectrogram mean and variance and the lightness distribution are more uniform. The normalization formula is as follows:

In G(a,b), a represents the corresponding time, b represents the frequency at time a, and the size of G(a, b) represents the energy contained in the audio component with frequency b at the corresponding time a. G(a,b) can draw a spectrogram, using the same color but different shades of color to represent the energy contained in different frequency components at each moment.

Step4: Store the generated spectrogram in the corresponding folder according to the corresponding speaker. The file name is the class label, that is, the corresponding speaker. Divide the generated spectrogram data set according to a certain proportion. for training set and test set.

3) Train the voiceprint recognition model based on the spectrogram: use the generated spectrogram to train the VGG16 model, and the file name is the class mark of the spectrogram, so as to achieve the purpose of realizing voiceprint recognition by image recognition. After training, use the test set to test to make the recognition accuracy meet the requirements. If it fails to meet the requirements, continue to train the model until the model accuracy meets the requirements. The specific steps are as follows:

Step1: Preprocess the image, and set the size of the spectrogram to 224×224×3.

Step2: Build the VGG16 model. Build an image recognition model based on the CNN structure, which has 13 convolutional layers and 3 fully connected layers.

Step3: Set relevant parameters and perform training. Let the confidence degree of the spectrogram sample x _i output by the VGG16 model be y _ipre , and use the cross entropy as the loss function:

L(x _i )＝-[y _i logy _ipre +(1-y _i )log(1-y _ipre )] (4)

where _yi denotes the ground truth label.

Step4: Use the test data set to test the accuracy of the recognition model to ensure that the preset recognition accuracy is achieved, otherwise modify the structure and parameters of the model and retrain.

4) Replace the model structure, repeat step 3), and train multiple voiceprint recognition models based on spectrograms with different structures. After training, use the test set to test each image recognition model to make the recognition accuracy meet the requirements. If the requirements are not met, change the model parameters and continue training the model until the accuracy of each model meets the requirements. Thereby, multiple voiceprint recognition models based on spectrograms are obtained.

5) Integrating the multiple voiceprint recognition models based on the spectrogram obtained above. The integrated model has multiple voiceprint recognition models based on spectrograms with different structures, and the output of each model is voted by the voting method. Then train again to further improve the recognition accuracy and robustness of the model. The specific steps are:

Step1: Integrate the multiple spectrogram-based voiceprint recognition models obtained above, and adopt a voting mechanism after integration.

Step2: Before voting, convert the prediction confidence returned by each voiceprint recognition model into a prediction category, that is, the category mark corresponding to the highest confidence level is used as the prediction result of the voiceprint recognition model.

Step3: After each model obtains the final prediction of the input sample x, if a certain prediction category receives more than half of the model votes, that is, if for the spectrogram sample, more than half of the output of the voiceprint recognition integrated model is speaker A, then It is considered that the audio corresponding to the spectrogram sample belongs to speaker A;

Step trains the voiceprint recognition integrated model with the train-clean-100 data set and tests it with the test set to further improve the recognition accuracy of the model and the defense capability of the model.

6) Attacking the voiceprint recognition model based on the spectrogram: using the cuckoo search algorithm to attack the voiceprint recognition model based on the spectrogram. For the multiple spectrogram-based voiceprint recognition models obtained in step 4), the cuckoo search algorithm is used to attack each model, and iteratively optimizes to find the optimal disturbance, which is superimposed on the original audio to generate an adversarial sample. The specific steps are as follows:

Step1: Initialize the fitness function and define the fitness function as follows:

f＝[y _ti logy _ipre +(1-y _ti )log(1-y _advipre )]+c||x _advi -x _i,0 || ₂ (5)(5) where x _advi represents an adversarial example , xi _{, 0} represents the original audio, y _ti represents the label of the target speaker, y _advipre represents the output of the adversarial sample, in which the L2 function is used to measure the difference between the adversarial sample and the original sample, and the difference is controlled by the parameter c the size of.

Step2: Initialize the nest. Set the number of nests to G, initialize random perturbations of the same size as the original audio, and superimpose them on the original audio to form an initial adversarial sample. That is, the initial bird's nest is set to:

X=x ₁ , x ₂ , . . . , x _G (6)

Step3: Obtain a new bird's nest through Levi's flight, that is, obtain a new confrontation sample through Levi's flight. The update of Levi's flight is as follows:

x _i = x _i +α*S*n (7)

where α is the step scaling factor and n is an array of random numbers from the standard normal distribution with the same dimension as _xi . S is the step size:

Among them, u and v are two variables that obey the Gaussian distribution, β is a constant, and ^σ2 is calculated by the following formula:

in is the gamma function.

Step4: Calculate the fitness of each individual, denoted as F=f ₁ , f ₂ , ..., f _G , and find the optimal individual in the population, that is, the individual with the smallest fitness function value, denoted as F _global . If the number of iterations reaches the set maximum number of iterations or the generated adversarial samples can be classified into the target category, the iteration is stopped and the adversarial samples are output. If the above conditions are not met, repeat the steps in Step1-Step3, and continue to iteratively optimize the population. In this way, adversarial examples generated under different models can be obtained.

7) Adversarial training The integrated voiceprint recognition model based on the spectrogram: convert the confrontation sample generated in step 6) into a spectrogram and add it to the training data set, retrain the integrated voiceprint recognition model based on the spectrogram, and improve Integrate the recognition accuracy and defense capabilities of the voiceprint recognition model to improve the security and stability of the voiceprint recognition model.

The embodiment also provides a defense device based on a voiceprint recognition integrated model of a spectrogram, including a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, When the computer processor executes the computer program, the above-mentioned defense method based on the voiceprint recognition integrated model of the spectrogram is realized.

Since the defense device and the computer program stored in the computer memory are mainly used to implement the above-mentioned defense method based on the voiceprint recognition integrated model of the spectrogram, its effect on the above-mentioned defense method is corresponding, and will not be repeated here.

In view of possible attacks on the white box or black box of the voiceprint recognition system, the present invention converts the voice signal into a spectrogram, uses the image recognition model to achieve the purpose of voiceprint recognition, and integrates multiple image recognition models , while improving the accuracy of voiceprint recognition, it also obtains defense capabilities against adversarial samples, and realizes defense against white-box or black-box attacks.

The above-mentioned specific embodiments have described the technical solutions and beneficial effects of the present invention in detail. It should be understood that the above-mentioned are only the most preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, supplements and equivalent replacements made within the scope shall be included in the protection scope of the present invention.

Claims (4)

1. The voiceprint recognition integrated model defending method based on the spectrogram is characterized by comprising the following steps of:

(1) Collecting an audio file, and converting the audio file into a spectrogram, wherein the spectrogram is used as a benign sample;

(2) Training a plurality of image recognition models by utilizing benign samples to enable the image recognition models to achieve the effect of voiceprint recognition, so as to obtain a plurality of image-based voiceprint recognition models;

(3) Integrating the plurality of the trained voiceprint recognition models based on the images in the step (2) by adopting a voting mechanism to form a voiceprint recognition integrated model, and retraining the voiceprint recognition integrated model by utilizing a benign sample, wherein the method specifically comprises the following steps of: integrating a plurality of voiceprint recognition models by utilizing a voting mechanism to obtain a voiceprint recognition integrated model; before voting, converting the prediction confidence coefficient returned by each voiceprint recognition model into a prediction category, namely, using a category label corresponding to the highest confidence coefficient as a prediction result of the voiceprint recognition model; after each voiceprint recognition model obtains a prediction result of a voiceprint sample, if a certain prediction category obtains more than half of voiceprint recognition model votes, the prediction category is the prediction result of the voiceprint recognition integrated model; training the voiceprint recognition integrated model by using benign samples, and testing by using a testing set to improve the voiceprint recognition integrated model;

(4) The cuckoo search algorithm is adopted to attack a plurality of voiceprint recognition models respectively, an countermeasure sample is generated, and the countermeasure sample is converted into a spectrogram which is used as a malignant sample, and the method specifically comprises the following steps:

(4-1) initializing a fitness function, defining the fitness function as follows:

f＝[y _ti logy _ipre +(1-y _ti )log(1-y _advipre )]+c·||x _advi -x _i，0 || ₂

wherein x is _advi Representing challenge samples, x _i,0 Representing the original audio, y _ti Tag representing target speaker, y _advipre Representing the output of the challenge sample, wherein the difference between the challenge sample and the original audio is measured by an L2 function, the magnitude of this difference being controlled by a parameter c, y _ipre Representing confidence of voiceprint recognition model output;

(4-2) initializing bird nests, setting the number of bird nests as G, initializing random disturbance with the same size as the original audio frequency, and superposing the random disturbance on the original audio frequency to form an initial countermeasure sample, namely setting the initial bird nests as follows:

X＝x ₁ ，x ₂ ，…，x _G

(4-3) obtaining a new bird nest by a lewy flight, i.e., obtaining a new challenge sample by a lewy flight, the lewy flight being updated as follows:

x _i ＝x _i +α*S*n

where α is the step size scale factor and n is the sum of x _i The number of dimensions is the same, an array of standard normal distributed random numbers, S is the step size:

where u, v are two variables subject to a gaussian distribution, β is a constant, σ ² Calculated from the formula:

wherein the method comprises the steps ofIs a gamma function;

(4-4) calculating fitness of each individual, denoted as f=f ₁ ,f ₂ ,…,f _G Finding the optimal individual in the population, namely the individual with the smallest fitness function value, and marking the optimal individual as F _global If the iteration number reaches the set maximum iteration number or the generated challenge sample can be classified into the target class, stopping iteration, outputting the challenge sample, and if the conditions are not met, repeating the steps (4-1) - (4-3), and continuing iteration optimizing on the population, so that the challenge sample generated under different voiceprint recognition models can be obtained;

(5) Retraining the voiceprint recognition integrated model based on the image obtained in the step (3) by utilizing a malignant sample and a benign sample to obtain a voiceprint recognition integrated model capable of resisting attack;

(6) And (5) performing defending and identifying on the spectrogram corresponding to the audio file by utilizing the voiceprint identification integrated model obtained in the step (5).

2. The method for defending a voiceprint recognition integrated model based on a spectrogram according to claim 1, wherein the specific steps of converting an audio file into the spectrogram are as follows:

framing the audio, windowing each frame of voice signal, and performing short-time Fourier transform;

calculating a power spectrum of the short-time Fourier transform result, normalizing the power spectrum to obtain a spectrogram, and forming a benign sample by the spectrogram and a corresponding speaker.

3. The method for defending a voiceprint recognition integrated model based on a spectrogram according to claim 1, wherein the image recognition model adopts VGG16 or VGG19.

4. A device for defending a voiceprint recognition integrated model based on a spectrogram, comprising a computer memory, a computer processor and a computer program stored in the computer memory and executable on the computer processor, wherein the computer processor implements the method for defending a voiceprint recognition integrated model based on a spectrogram according to any one of claims 1 to 3 when the computer program is executed.