KR20230103667A - Method and apparatus for determining speaker similarity - Google Patents

Abstract

The present invention is a method for determining speaker similarity, which is performed by a processor of a speaker similarity determination device, by receiving speaker information and test voice data, searching for registered voice data of a speaker registered in advance according to the speaker information, and testing voice data. A first speaker feature vector and a first auxiliary feature vector are derived from the data, a second speaker feature vector and a second auxiliary feature vector are extracted from the registered speech data, and based on the first speaker feature vector and the second speaker feature vector to derive a speaker feature similarity matrix, generate a first composite feature vector based on the first speaker feature vector and the first auxiliary feature vector, and generate a second composite feature vector based on the second speaker feature vector and the second speaker feature vector generating a feature vector, calculating a cross concentration using the first comprehensive feature vector and the second comprehensive feature vector, and calculating a similarity between the test voice and the speaker's voice based on the speaker feature similarity matrix and the cross concentration can include

Description

Method and apparatus for determining speaker similarity {METHOD AND APPARATUS FOR DETERMINING SPEAKER SIMILARITY}

The present invention relates to a method and apparatus for determining the similarity of speakers, and more particularly, to determine speaker similarity to determine whether a speaker of an uttered voice is a previously registered speaker using a cross attention technique. The invention relates to a method and apparatus.

The contents described below are only described for the purpose of providing background information related to an embodiment of the present invention, and the contents described do not naturally constitute prior art.

Speaker recognition is a field in which the identity of a speaker is discriminated by extracting and analyzing features from an input voice. Speaker recognition is a technology that has various applications, such as authentication for device use and service access, speaker identification to provide services appropriate to the user by determining whether the speaker is a preselected type, and filtering procedures to determine whether the speaker is blocked from access. am.

In general, speaker recognition through speech analysis includes a process of extracting frame-unit features from input sentences, a process of extracting a fixed-dimensional feature vector called speaker embedding through frame-unit features, and a speaker obtained from sentences. It consists of a process of determining whether the embeddings are the same through comparison.

Recently, research on the development of performance of various computing devices and the field of machine learning has been actively carried out, and technology has been developed. As large data sets are provided, the performance of deep learning-based speaker recognition technologies has improved considerably, and more effective deep learning technologies have been developed. Various studies on learning-based speaker recognition systems (e.g., deep neural network structure, loss function modeling, pooling technique, etc.) are in progress.

Speech analysis requires an understanding of the linguistic habits or characteristics of a specific person among the linguistic factors of each individual speaker, and these factors vary widely among individuals. In addition, since the voice characteristics may vary depending on the surrounding environment such as the receiving device and noise, it is not easy to improve the accuracy of the technology for recognizing a specific speaker through voice analysis. Advances in technology for recognition are required.

On the other hand, the above-mentioned prior art is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and cannot necessarily be said to be known art disclosed to the general public prior to the filing of the present invention. .

An object of the present invention is to provide a method and apparatus capable of determining how similar a voice provided with speaker information is to a speaker's voice according to speaker information, based on voice data provided together with speaker information.

An object of the present invention is a method and apparatus for determining the similarity of a speaker's voice by comparing input test voice data with pre-registered voice data of a speaker so as to calculate the same speaker verification score in a deep learning-based speaker recognition framework is to provide

The object of the present invention is not limited to the above-mentioned tasks, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the embodiments of the present invention. It will also be seen that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

The method and apparatus according to an embodiment of the present invention calculates the correlation between frame-by-frame auxiliary feature vectors extracted from speech using a cross attention technique and uses it in the verification score step to determine the similarity between information other than speaker information. It is about a technique for calculating the same speaker verification score among the deep learning-based speaker recognition frameworks using this method.

A technique according to an embodiment of the present invention extracts as many sets of auxiliary feature vectors and speaker-related feature vectors as the number of input frames for information capable of representing speaker information from speech, and utilizes the auxiliary feature vectors in registration and testing processes. Thus, it may be configured to determine whether or not they are the same person by calculating the degree of similarity between the registered speaker and the feature vector set of the input speaker.

A method for determining speaker similarity according to an embodiment of the present invention is a method for determining speaker similarity, which is performed by a processor of a speaker similarity determining device, receives speaker information and test voice data, and determines a speaker previously registered according to the speaker information. Retrieves the registration speech data of , derives a first speaker feature vector and a first auxiliary feature vector from the test speech data, extracts a second speaker feature vector and a second auxiliary feature vector from the registration speech data, and A speaker feature similarity matrix is derived based on the feature vector and the second speaker feature vector, a first synthetic feature vector is generated based on the first speaker feature vector and the first auxiliary feature vector, and the second speaker feature vector and the second speaker feature vector are generated. 2 Generates a second comprehensive feature vector based on the speaker feature vector, calculates cross concentration using the first comprehensive feature vector and the second comprehensive feature vector, and calculates the cross concentration based on the speaker feature similarity matrix and the cross concentration, and compares the test voice and The method may include calculating a similarity of the speaker's voice.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to the present embodiment, speaker recognition can be performed more effectively and accurately by improving the accuracy of speaker similarity determination through voice analysis.

According to this embodiment, authentication for access to a device or service can be more accurately performed by determining whether an input voice is the voice of a speaker registered in advance.

In the method according to the present embodiment, in the process of comparing speaker similarity between two voices, the speaker similarity between two voices is calculated in the inference step because the speaker similarity is calculated using the intersection attention technique from the frame-unit feature vector without a separate summary process. Concentrate on the meaningful speaker characteristics that can be obtained from a short time in the unit of hour frame to enable comparison.

The technique according to the present embodiment has an advantage in that the speaker recognition system can compare and analyze speaker information and other similar information between two voices simultaneously.

In addition, the model structure for determining speaker similarity according to this embodiment partially applies a structure widely used in the existing speaker recognition field, but can be easily replaced with an appropriate model structure to suit the purpose and applied, so that it can be used in various fields. There is an advantage to being

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is an exemplary view of an operating environment in which a speaker similarity determination according to an exemplary embodiment is implemented.
2 is a flowchart of a method for determining speaker similarity according to the present embodiment.
3 is a block diagram of an apparatus for determining speaker similarity according to the present embodiment.
4 is a schematic diagram exemplarily showing the overall configuration of a speaker recognition system according to the present embodiment.
5 is a schematic diagram exemplarily illustrating a cross attention extraction unit in the speaker recognition system according to the present embodiment.

Hereinafter, the present invention will be described in more detail with reference to the drawings. The invention may be embodied in many different forms and is not limited to the embodiments set forth herein. In the following embodiments, parts not directly related to the description are omitted in order to clearly describe the present invention, but this does not mean that the omitted configuration is unnecessary in implementing a device or system to which the spirit of the present invention is applied. . In addition, the same reference numbers are used for the same or similar elements throughout the specification.

In the following description, terms such as first and second may be used to describe various components, but the components should not be limited by the terms, and the terms refer to one component from another. Used only for distinguishing purposes. Also, in the following description, singular expressions include plural expressions unless the context clearly indicates otherwise.

In the following description, terms such as "comprise" or "having" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that it does not preclude the possibility of addition or existence of features, numbers, steps, operations, components, parts, or combinations thereof.

The present invention will be described in detail with reference to the drawings below.

1 is an exemplary diagram of an operating environment in which a speaker similarity determination according to an exemplary embodiment is implemented.

In determining the speaker similarity according to the present embodiment, first, when the user 100 to be verified speaks for authentication, service access, etc., the input interface 200 collects it, and based on the collected voice data, the speaker similarity is determined. This can be done by performing similarity determination in the determination device 400 .

Here, the input interface 200 may be any device having a voice receiving device, and may be, for example, various electronic devices such as a smart phone, a laptop computer, and a fixedly installed terminal.

The input interface 200 may include a microphone that converts a user's utterance, which is a similarity determination target, into audio data.

The voice data received by the speaker similarity determination device 400 may be an audio file including at least a part of the voice itself collected through the microphone 200 or a voice spectrum file obtained by converting the audio file, and may be used. A variety of possible transformations can be selected.

In FIG. 1 , the speaker similarity determining device 400 is a kind of remote server, and voice data collected through the Internet 300 is transferred to the speaker similarity determining device 400 to determine the speaker similarity.

Here, the Internet 300 is an example of a network, and the network includes wired and wireless networks, such as a local area network (LAN), a wide area network (WAN), the Internet, an intranet, and an extranet ( extranet), and mobile networks such as cellular, 3G, LTE, 5G, WiFi networks, ad hoc networks, and combinations thereof.

Also, a network may include connections of network elements such as hubs, bridges, routers, switches, and gateways. A network may include one or more connected networks, such as a multi-network environment, including a public network such as the Internet and a private network such as a secure corporate private network. Access to the network may be provided through one or more wired or wireless access networks.

Meanwhile, depending on the device's processor performance, memory capacity, and data capacity to be processed, the speaker similarity determining device 400 may be implemented as an integrated device connected to the microphone 200.

For example, the speaker similarity determining device 400 may be a user terminal including a microphone 200 .

The speaker similarity determining apparatus 400 may include a memory storing a model for determining speaker similarity and a speaker's voice to be determined together with speaker information, and at least one processor operatively connected to the memory.

Meanwhile, the memory may be implemented as a memory of an external server that can be connected through wired/wireless communication, the Internet, or the like, other than being coupled to the speaker similarity determining device 400 .

The memory may include internal memory and/or external memory, and may include volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, NAND flash memory, or non-volatile memory such as NOR flash memory, flash drive such as SSD, compact flash (CF) card, SD card, Micro-SD card, Mini-SD card, Xd card, or memory stick, or HDD The same storage device may be included. The memory may include magnetic storage media or flash storage media, but is not limited thereto.

The processor, as a kind of central processing unit, may control the operation of the speaker similarity determining device 400 by executing one or more commands stored in memory.

A processor may include any kind of device capable of processing data. For example, a processor may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or instructions included in a program.

As an example of such a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but is not limited thereto. A processor may include one or more processors.

The speaker information may include information for identifying the speaker, such as the speaker's name, address, resident registration number, phone number, user account, and service access level. Such speaker information is stored in advance according to the like.

The neural network model stored in the speaker similarity determining apparatus 400 may be a model previously trained through various learning methods such as a supervised learning method.

In the case of the supervised learning method, training may be performed using voice data and training data labeled with accurate information about the speaker of the corresponding voice data, and the speaker similarity judgment model is the voice data of two different speakers and the voice of the same speaker. Training may be performed through training data labeled with data and whether each pair is identical.

2 is a flowchart of a method for determining speaker similarity according to the present embodiment.

First, voice data generated by the user may be input to the speaker similarity determining device (S210).

The voice data is collected through the input interface 200, and may be an audio file including at least a part of the collected voice itself, or may be a voice spectrum file obtained by performing Fourier transform on the audio file. can be variously selected.

The speaker similarity determining apparatus 400 may receive speaker information together with voice data. Here, the received voice data is a user's voice that is the subject of speaker similarity determination, and may be referred to as test voice data.

For example, if the input interface 200 is a microphone of a smart phone, the speaker information may include information for identifying the owner of the smart phone, such as a name, phone number, and resident registration number of a person set as the owner of the smart phone.

If the input interface 200 is a microphone of a laptop computer and a speaker similarity determination is requested after a user logs in to a specific website, speaker information may include user information according to login information of the specific website.

According to the speaker information, the speaker similarity determining apparatus 400 may search for registered voice data of a previously registered speaker.

In a memory connected to the speaker similarity determining device 400, voice data of a registered speaker linked with speaker information is stored in advance, and the speaker similarity determining device 400 searches for and loads the registered voice data based on the received speaker information. can do.

The speaker similarity determining apparatus 400 may extract frame-level speaker feature information and frame-step auxiliary feature information from the loaded registration voice data and the received test voice data (S220 and S230).

In the frame-level feature extraction step, frame-wise feature information output is calculated from the voice feature vector (e.g., MFCC, STFT, Mel Filter Bank, etc.) of the input voice data.

The voice feature vectors are processed by shifting the input wave signal to a certain length by covering a window of a specific length (eg, 250 ms), and the voice feature vectors extracted from the test voice data A and the registered voice data B It can be defined as x _A,t (0≤t≤T _A ) and x _B,t (0≤t≤T _B ). Meanwhile, the specific length may be adjusted according to data characteristics, authentication strength, and processor performance.

The extracted speech feature vectors are converted into speaker feature vectors and auxiliary feature vectors through a frame stage speaker feature information extraction step and an auxiliary feature extraction step. The extracted speaker feature vectors are defined as h _A,t (0≤t≤T _A ) and h _B,t (0≤t≤T _B ), and the auxiliary feature vectors are defined as c _A,t (0≤t≤T _A ), c _B,t (0≤t≤T _B ).

Here, the speaker characteristic information means information showing the characteristics of a speaker that reveals the characteristics of each individual in the voice data, and the auxiliary characteristics are the presence or absence of the audio itself and the size of the volume apart from the characteristics of each individual in the voice data , may mean auxiliary characteristics such as presence or absence of noise.

A time delay neural network (TDNN), a convolutional neural network (CNN), a recurrent neural network (RNN), or the like can be used as a neural network model for processing input speech features.

The processor of the speaker similarity determination apparatus 400 derives a first speaker feature vector and a first auxiliary feature vector from voice data according to a speaker feature extraction step (S220) and an auxiliary feature extraction step (S230), and from the registered voice data. A second speaker feature vector and a second auxiliary feature vector may be extracted.

The speaker similarity determination apparatus 400 may calculate the similarity by using the extracted speaker feature vector and auxiliary feature vector (S230).

The similarity calculation unit calculates similarities between frame-level speaker feature vectors extracted from test speech data A and registration speech data B and similarities between auxiliary feature vectors and outputs them in a matrix. In this process, cosine similarity may be calculated as the degree of similarity. The similarity matrix calculation process can be defined as follows.

[Equation 1]

는 화자 특징 유사도 행렬이며,

는 보조 특징 유사도 행렬이다. 두 행렬 모두

의 차원을 가질 수 있다.in the above expression

is the speaker feature similarity matrix,

is the auxiliary feature similarity matrix. both matrices

can have dimensions of

Meanwhile, when calculating the similarity, a similarity calculation method based on distance-based similarity and L2 distance may be used in addition to cosine similarity.

Also, the speaker similarity determining apparatus 400 may calculate cross attention (S250).

를 만들 수 있다. The cross attentional concentration extraction unit calculates the cross attention through two stages as shown in FIG. 5 . First, the concentration extraction unit receives the frame-level speaker feature vector and the auxiliary feature vector as input and concatenates the dimensions to create a comprehensive feature vector.

can make

Comprehensive feature vectors are extracted by voice concentration over time through a neural network model. As a neural network model for feature processing, two or more fully connected deep neural networks (FCDNNs) are usually used, and the output dimension of the last neural network is 1.

The speaker similarity determining apparatus 400 generates a first comprehensive feature vector based on the first speaker feature vector and the first auxiliary feature vector, and generates a second comprehensive feature vector based on the second speaker feature vector and the second speaker feature vector. A synthetic feature vector can be created.

Calculating the cross concentration may include generating a first voice concentration vector for test speech data and a second voice concentration vector for registration voice data based on the first comprehensive feature vector and the second comprehensive feature vector; Deriving an auxiliary feature similarity matrix based on the first auxiliary feature vector and the second auxiliary feature vector, and calculating cross concentration based on the auxiliary feature similarity matrix, the first voice concentration vector for each time, and the second voice concentration vector for each time. steps may be included.

The calculation of the voice concentration by time shown in FIG. 5 may be performed by a neural network model, and the equation for this may be defined as follows.

[Equation 2]

는 시간별 음성 집중도 벡터이고,

과

는 신경망의 파라미터이며,

는 시그모이드(sigmoid) 함수 등의 활성화 함수이다. 여기서 활성화 함수는 시그모이드 함수 외에 다른 종류의 활성화 함수가 사용될 수 있음은 물론이다.in the above expression

is the hourly speech concentration vector,

class

is a parameter of the neural network,

Is an activation function such as a sigmoid function. It goes without saying that other types of activation functions other than the sigmoid function may be used as the activation function.

와

는 테스트, 등록 음성 집중도 벡터이며, 각각

와

차원의 벡터이다.

and

are test and registration speech concentration vectors, respectively

and

is a vector of dimensions.

과 시간별 음성 집중도 벡터

를 입력 받아 행렬곱을 통해 교차 집중도를 계산할 수 있다. In the cross attention calculation unit, the auxiliary feature similarity matrix

and hourly speech concentration vector

can be input and the cross concentration can be calculated through matrix multiplication.

The calculating of the cross-concentration may include calculating a first cross-concentration for the first voice concentration vector for each time and a second cross-concentration for the second voice concentration vector for each time.

Calculation of the first cross-concentration, which is the cross-concentration for the registration voice data, and the second cross-concentration, which is the cross-concentration for the test voice data, can be expressed as follows.

[Equation 3]

[Equation 4]

와 등록 음성 교차 집중도

는

와

차원의 벡터일 수 있다.Calculated test voice cross-concentration

and registration voice cross-concentration

Is

and

It can be a vector of dimensions.

The speaker similarity determination apparatus 400 may perform speaker similarity calculation using the above result (S260).

와 등록 음성 집중도

그리고 화자 특징 유사도 행렬을 통해 화자 유사도를 계산한다. 계산 과정은 다음과 같이 정의될 수 있다.In the process of calculating the similarity, the test voice concentration

and registration speech concentration

Then, the speaker similarity is calculated through the speaker feature similarity matrix. The calculation process can be defined as:

[Equation 5]

The finally obtained speaker similarity s is a one-dimensional value, and as its magnitude increases, it can be used as an indicator that the speakers of the registration voice and the test voice are similar.

Meanwhile, the method of determining speaker similarity may further include selecting at least one frame from each of the registration voice data and the test voice data after the step of searching for the registered voice data and before the step of extracting the registered voice data.

In addition, the step of extracting the speaker feature vector and the auxiliary feature vector may include deriving a first speaker feature vector and a first auxiliary feature vector from at least one frame of the test speech data, and deriving a first speaker feature vector and a first auxiliary feature vector from at least one frame of the registration speech data. Extracting the 2-speaker feature vector and the second auxiliary feature vector may be included.

Meanwhile, in the technique according to an embodiment of the present invention, the pooling process is a process of extracting a feature vector having one fixed dimension from frame-unit feature vectors obtained from speeches of different lengths. How well information effective for speaker recognition is aggregated from feature vectors in units of frames is directly related to performance.

However, since existing pooling processes generally average frame-level feature vectors using simple arithmetic mean or weighted arithmetic mean, much information of frame-step feature vectors may be lost in this process.

In addition, the fixed-dimensional vectors, called embeddings obtained through the extracted pooling process, are scored using cosine similarity or separate machine learning techniques (e.g., Linear Discriminant Analysis, Probabilistic Linear Discriminant Analysis) to derive speaker recognition results. ) system is required.

Therefore, in the present invention, by using auxiliary feature vectors and cross-attention techniques, a vertical speaker recognition system that can calculate more accurate verification scores by considering short speech information using frame-level feature vectors and does not require separate scoring system learning Suggest.

On the other hand, in the present invention, voice analysis can be performed more effectively by determining which frames should be more focused on the similarity and by focusing more on a portion of the frames having a high similarity of auxiliary feature vectors for comparison.

3 is a block diagram of an apparatus for determining speaker similarity according to the present embodiment.

The speaker similarity determination apparatus 400 includes a voice data receiver 410, a frame-level speaker feature information extractor 420, a frame-step auxiliary feature extractor 430, a voice similarity calculator 440, a cross attentional concentration extractor ( 450), a speaker similarity calculator 460, a memory 470, and a processor 480.

Each unit may be implemented as a hardware or software module and may be programmed to perform the steps described above. Meanwhile, the frame-level speaker feature extraction unit and the frame-level auxiliary feature extraction unit may have the same structure.

4 is a schematic diagram exemplarily showing the overall configuration of a speaker recognition system according to the present embodiment.

A black arrow may indicate a transfer process of the test voice data (A), and a white arrow may indicate a transfer process of the registration voice data (B).

FIG. 5 is a schematic diagram exemplarily illustrating a cross attention extraction unit in the speaker recognition system according to the present embodiment, and is a diagram for explaining the cross attention extraction unit shown in FIG. 4 in more detail.

According to FIG. 5, the test subject feature vector and the registration comprehensive feature vector are output as hourly speech concentration vectors after passing through the concentration extraction unit, and the cross attention extraction unit crosses the registration speech based on these vectors and a pre-calculated auxiliary feature similarity matrix. Concentration and test voice cross-concentration can be output.

Referring back to FIG. 4 , the speaker similarity calculation unit may calculate the similarity of the speakers based on the speaker feature similarity matrix, the cross-concentration of the registered voice, and the cross-concentration of the test voice.

The value finally output through the above process is the probability that the speaker is the same person. The higher the speaker similarity, the higher the possibility that the user of the test voice data and the speaker of the registered voice data are the same person. It may be determined that the speaker of the voice data is less likely to be the same person.

The above-described method for determining speaker similarity according to an embodiment of the present invention can be implemented as computer readable code on a medium on which a program is recorded. The computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. there is

The description of the embodiments of the present invention described above is for illustrative purposes, and those skilled in the art can easily modify them into other specific forms without changing the technical spirit or essential features of the present invention. you will understand that Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof should be construed as being included in the scope of the present invention.

100: user
200: input interface
300: network
400: speaker similarity determination device

Claims (13)

A method for determining speaker similarity performed by a processor of a speaker similarity determination device, the method comprising:
receiving speaker information and test voice data;
searching for registered voice data of a previously registered speaker according to the speaker information;
extracting a first speaker feature vector and a first auxiliary feature vector from the test speech data, and extracting a second speaker feature vector and a second auxiliary feature vector from the registered speech data;
deriving a speaker feature similarity matrix based on the first speaker feature vector and the second speaker feature vector;
generating a first synthetic feature vector based on the first speaker feature vector and the first auxiliary feature vector, and generating a second synthetic feature vector based on the second speaker feature vector and the second speaker feature vector; ;
calculating cross concentration using the first comprehensive feature vector and the second comprehensive feature vector; and
Calculating a similarity between the test voice and the speaker's voice based on the speaker feature similarity matrix and the cross concentration.
How to judge speaker similarity.
According to claim 1,
The step of calculating the cross concentration,
generating a first voice concentration vector for each time of the test speech data and a second voice concentration vector for each time of the registered voice data based on the first comprehensive feature vector and the second comprehensive feature vector;
deriving an auxiliary feature similarity matrix based on the first auxiliary feature vector and the second auxiliary feature vector; and
Calculating a cross concentration based on the auxiliary feature similarity matrix, the first voice concentration vector for each time, and the second voice concentration vector for each time,
How to judge speaker similarity.
According to claim 2,
The step of generating the voice concentration vector is performed through a neural network model according to Equation 1 below,
Equation 1:

here,

Is a voice concentration vector by time, W ₁ and W ₂ are parameters of the neural network model,

is the activation function,

is the overall feature vector,
How to judge speaker similarity.
According to claim 3,
The step of calculating the cross concentration,
Calculating a first cross-concentration degree for the first voice intensity vector over time and a second cross-concentration degree for the second voice intensity vector over time;
The first cross-concentration is calculated according to Equation 2 below, and the second cross-concentration is calculated according to Equation 3 below,
Equation 2:

Equation 3:

Here, R ^C is the auxiliary feature similarity matrix,

Is the voice concentration vector for each first time,

Is the second voice concentration vector for each time,
How to judge speaker similarity.
According to claim 4,
The step of calculating the similarity of the speaker's voice is performed according to Equation 4 below,
Equation 4:

Here, s is the similarity, R ^h is the speaker feature similarity matrix, and the larger the size of s, the higher the similarity.
How to judge speaker similarity.
According to claim 1,
After the step of retrieving the registered voice data and before the step of extracting,
selecting at least one frame from each of the registration voice data and the test voice data;
The extraction step is
Deriving a first speaker feature vector and a first auxiliary feature vector from at least one frame of the test speech data, and extracting a second speaker feature vector and a second auxiliary feature vector from at least one frame of the registered speech data. including,
How to judge speaker similarity.
A computer readable recording medium storing a computer program comprising one or more instructions configured to execute a method according to any one of claims 1 to 6.
As a device for determining speaker similarity,
memory for storing instructions and neural network models; and
at least one processor electrically connected to the memory;
the processor,
receiving speaker information and test voice data;
Searching for registered voice data of a previously registered speaker according to the speaker information;
extracting a first speaker feature vector and a first auxiliary feature vector from the test speech data, and extracting a second speaker feature vector and a second auxiliary feature vector from the registered speech data;
Deriving a speaker feature similarity matrix based on the first speaker feature vector and the second speaker feature vector;
An operation of generating a first synthetic feature vector based on the first speaker feature vector and the first auxiliary feature vector, and generating a second synthetic feature vector based on the second speaker feature vector and the second speaker feature vector. ,
Calculating cross concentration using the first comprehensive feature vector and the second comprehensive feature vector; and
And configured to perform an operation of calculating a similarity between the test voice and the speaker's voice based on the speaker feature similarity matrix and the cross concentration.
Speaker similarity judgment device.
According to claim 8,
The operation of calculating the cross concentration,
generating a first voice concentration vector for each time of the test speech data and a second voice concentration vector for each time of the registered voice data based on the first comprehensive feature vector and the second comprehensive feature vector;
Deriving an auxiliary feature similarity matrix based on the first auxiliary feature vector and the second auxiliary feature vector; and
Calculating a cross concentration based on the auxiliary feature similarity matrix, the first voice concentration vector for each time, and the second voice concentration vector for each time,
Speaker similarity judgment device.
According to claim 9,
The operation of generating the voice concentration vector is performed through a neural network model according to Equation 1 below,
Equation 1:

here,

Is a voice concentration vector by time, W ₁ and W ₂ are parameters of the neural network model,

is the activation function,

is the overall feature vector,
Speaker similarity judgment device.
According to claim 10,
The operation of calculating the cross concentration,
Calculating a first cross-concentration for the first voice concentration vector for each time and a second cross-concentration for the second voice concentration vector for each time;
The first cross-concentration is calculated according to Equation 2 below, and the second cross-concentration is calculated according to Equation 3 below,
Equation 2:

Equation 3:

Here, R ^C is the auxiliary feature similarity matrix,

Is the voice concentration vector for each first time,

Is the second voice concentration vector for each time,
Speaker similarity judgment device.
According to claim 11,
The operation of calculating the similarity of the speaker's voice is performed according to Equation 4 below,
Equation 4:

Here, s is the similarity, R ^h is the speaker feature similarity matrix, and the larger the size of s, the higher the similarity.
Speaker similarity judgment device.
According to claim 8,
the processor,
After the operation of searching for the registered voice data and before the operation of extracting,
configured to further perform an operation of selecting at least one frame from each of the registration voice data and the test voice data;
The extraction operation is
Deriving a first speaker feature vector and a first auxiliary feature vector from at least one frame of the test speech data, and extracting a second speaker feature vector and a second auxiliary feature vector from at least one frame of the registered speech data. including,
How to judge speaker similarity.

Images