KR20250112644A - The method of real-time detection of audio spoofing in phone call and the apparatus thereof - Google Patents

Abstract

The present invention relates to a method and a device for enhancing voice security of a mobile communication device, and more specifically, to storing voice characteristics of a counterparty for each phone number stored in the mobile communication device, and comparing the characteristics of the counterparty's voice when calling the corresponding phone number with the characteristics of a previously stored voice, thereby verifying the identity of the counterparty.
To this end, the present invention comprises a management unit that manages a database for checking the other party's call number, checking whether voice characteristic data for the corresponding number is stored, and additionally storing it if necessary, an input unit that receives a voice input signal, a preprocessing unit that preprocesses the input voice signal or extracts the characteristics of the voice signal, a comparison unit that learns and compares whether the signal received from the preprocessing unit matches the voice characteristics of the existing corresponding number, and a database unit that stores the learned characteristics. If the difference in characteristic signals between the voice during a call and the existing voice exceeds a certain level, a warning is given so that the other party's identity can be suspected. Through this, voice spoofing can be prevented in calls where the voice is stolen or the caller number is stolen.

Description

{The method of real-time detection of audio spoofing in phone call and the apparatus thereof}

The present invention relates to a method for preventing a person from deceiving the listener about the identity of a speaker by using a fake voice on a personal calling device such as a mobile phone, based on artificial intelligence, and is a technology for an operating method that enables real-time spoofing detection and a device using the same.

In social life, identifying the identity of the other party on the phone is a very important issue. Primarily, it is determined using the other party's call number, but methods for deceiving this and detecting deceiving the caller number are becoming increasingly sophisticated. In general, means for preventing such spoofing can be implemented in the system of the person receiving the call or the portable device receiving the call, or conversely, spoofing prevention means can be implemented in the device making the call. For example, Korean Patent Publication No. 10-2022-0133235, 'Powerful spoofing detection system using deep residual neural network', presents a computer implementation method for responding to spoofing. In this method, relatively large-scale deep learning models can be utilized based on powerful computing power, but implementing such deep learning models on personal devices such as mobile phones can cause disruption in natural phone conversations due to increased system complexity and increased delay time due to such data processing.

Therefore, a relatively simple but effective device is required to prevent spoofing in personal mobile communication devices. Methods for synthesizing the voice of a person to be deceived include changing text input into voice, or modulating the voice of someone else to have the characteristics of a specific person's voice, and as shown in Document 1, the synthesized voice retains characteristics that are different from the original voice, and this can be used to distinguish it as a synthesized voice. Document 2 showed the possibility of distinguishing a synthesized voice by extracting the characteristics of the voice and then applying this to machine learning.

KR 1020220133235 A

[Document 1] Frank, John and Schonherr, Lea. Wavefake: a data set to facilitate audio deepfake detection, Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021), 2021. [Document 2] Farkhund Iqbal, Ahmed Abbasi, Abdul Rehman Javed, Zunera Jalil, and Jamal Al-Karaki. Deepfake Audio Detection via Feature Engineering and Machine Learning, Woodstock’22: Symposium on the irreproducible science, June 07-11, 2022, Woodstock, NY

In conversations using personal mobile devices, attempts to deceive the listener by imitating another person's voice when the other person is not visible are becoming increasingly sophisticated. In particular, trust in the other party's phone number, such as a friend or acquaintance, with whom one has previously made calls is relatively high, and if a person A successfully attempts to deceive C by stealing another person B's mobile phone or by having B's phone number appear as the caller ID on C's phone instead of his own, the likelihood of C being deceived increases even more. The purpose of the present invention is to implement a technology for preventing C from mistaking the identity of the other party's phone number by verifying on C's phone whether the voice of the other party's phone number receiving a call from B's phone number is actually B's or a synthetic voice imitating B's voice, and a device equipped with such a technology.

To achieve this task, the present invention includes the following configuration.

That is, a mobile phone system according to one embodiment of the present invention is characterized by including a processor that verifies the other party's phone number, searches, processes, and stores a related database, and takes on a management role including data during a voice call or after the call is completed, a database unit that stores data such as voice data and characteristics for each other's phone number and the weights of an artificial neural network utilizing the same, an input unit that receives the voice of the other party, a preprocessing unit that samples appropriate voice data from the input unit and performs preprocessing on the sampled voice data, a learning unit that learns characteristics of voice data on which preprocessing has been performed in the preprocessing unit, and a comparison unit that verifies whether previously stored voice data is the voice of the same person. The learning unit may include an artificial neural network that learns voice characteristics and provides a numerical value that can determine whether the voice is the voice of the same person.

In the early days of using the mobile phone system, if the voice characteristic data for the other party's phone number is not stored, sampling and storing the other party's voice, extracting voice characteristics, and learning an artificial neural network are performed, and the identity of the other party is not verified. Such learning of an artificial neural network generally requires a lot of time and computing resources, so it can be performed during a call, but during a call, the other party's voice is sampled, and after the call, preprocessing of the sampled voice and artificial neural network learning using the preprocessed data can be performed. After acquiring the characteristics of the voice data of the other party's phone number and learning the artificial neural network are completed, the previously stored data and the voice data input into the input unit during the call are compared to compare the two voice characteristics, and whether or not there is spoofing can be determined in real time. If spoofing is determined, the security of the device is enhanced by using a warning device such as light or sound to warn the other party about their identity.

According to the present invention, in an advanced voice spoofing situation, such as acquiring the personal calling device of the original owner A or imitating or synthesizing the original owner's voice in a forged call with the owner A's telephone number, the voice or voice characteristics of A stored in the personal calling device of the call counterpart B are utilized to make a countermeasure.

By verifying the identity of the other party, it can have the effect of preventing B from being fooled by such spoofing.

The drawings are intended to better understand the present solution and are not intended to limit the present application.
FIG. 1 is a block diagram of a calling device provided according to one embodiment of the present invention.
Figure 2 is a schematic diagram of the flow of processing a voice signal when making a call using a calling device to which the present invention is applied.
FIG. 3 is a schematic diagram showing a signal processing flow during learning of an artificial intelligence model in one embodiment of the present invention.

It should be understood that the material described in this section is not intended to identify key or critical features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will be readily understood by the specification below.

With respect to the embodiments illustrated in the drawings, specific terms used in the specification are used to describe the same, but are not intended to limit the scope of the present invention thereby. Alternatives and additional modifications of the novel features illustrated in the present specification, and additional applications of the principles of the invention illustrated in the present specification, which may occur to those skilled in the art to which the present disclosure pertains, should be considered to be within the scope of the present invention.

In a personal voice call, the identification of the caller can be used for various crimes, from simple confusion to the other party. In particular, if a person A's mobile phone is transferred to another person B, and B attempts to call another person C using A's mobile phone, C has no choice but to rely solely on the voice on the phone to identify the other party. Another example is when A calls C through a caller ID manipulation, but B's caller ID appears on C's phone number, C is likely to believe that the other party is B, and especially if the voice on the call is similar to B's voice, C is likely to believe that the other party is B. To prevent this situation, if C's mobile device verifies that the voice of the other party is the voice of the owner of the phone number displayed on the mobile phone, the security of the mobile phone device is strengthened. Another example is when C installs a type of phishing software on his mobile phone to make him call another number, D, when he tries to call B's phone, and if the voice of the person answering is similar to B, C is likely to believe that the other party is B.

Therefore, the present invention is characterized by storing the voice characteristics of the other party based on the other party's mobile phone number, and comparing the other party's voice during the call with the previously stored number to confirm the other party's identity.

That is, a mobile phone system according to one embodiment of the present invention comprises a processor (10) that checks the other party's phone number, searches, processes, stores, etc. a related database, and takes on a management role including data during a voice call or after the call is completed, a database unit (70) that stores data such as voice data and characteristics by the other party's phone number and the weights of an artificial neural network utilizing the same, an input unit (20) that receives the other party's voice, and samples appropriate voice data from the input unit, and the sampling unit

The present invention is characterized by including a preprocessing unit (30) that performs preprocessing on voice data, a learning unit (40) that learns the characteristics of voice data on which preprocessing has been performed in the preprocessing unit, and a comparison unit (50) that verifies whether previously stored voice data is the voice of the same person. The comparison unit may include an artificial neural network that learns voice characteristics to determine whether the voice is the voice of the same person.

In the early stage of using the mobile phone system, if the voice characteristic data for the other party's phone number is not stored, sampling and storing the other party's voice, extracting voice characteristics, and learning an artificial neural network are performed, and the identity of the other party is not checked. Such learning of an artificial neural network generally requires a lot of time and computing resources, so it can be performed during a call, but the other party's voice can be sampled during the call, and preprocessing of the sampled voice and learning of an artificial neural network using the preprocessed data can be performed after the call. After acquiring the characteristics of the voice data of the other party's phone number and learning the artificial neural network are completed, the previously stored data and the voice data input into the input section during the call are compared to compare the two voice characteristics, and it is possible to determine in real time whether there is spoofing. If spoofing is determined, the security of the device is enhanced by making the other party aware of the identity of the other party through a warning device (60) such as light or sound.

One embodiment of the present invention is shown in Fig. 2. When receiving a call (S110), the phone number is confirmed (S120), the database is checked to see if the voice characteristics of the corresponding number are secured as reference data (S130), and if there is reference data, it is checked whether the voice signal of the other party is the voice of the same person based on the data. Specifically, a part of the voice signal of the other party is sampled (S150), and basic voice data features such as pitch or MFCC can be extracted through preprocessing (S160), and the difference from the previously stored voice characteristics is checked (S170), and if it is not the voice of the same person, a warning is given (S180). The warning is mainly used to make the other party suspect the identity by blinking a light or making a warning sound, but the means of warning are not limited to light and sound. The sampling of the voice signal (S150 or S190) can be performed for a short time of 1 to 2 seconds, or it is also possible to sample the sound of the entire section where the other party's voice continues. For voice preprocessing (S160 or S210), Fourier transform, short time Fourier transform (STFT), Mel spectrogram, etc. can be performed, and audio characteristics such as pitch and centroid can also be extracted. The method for determining whether existing stored voice data matches the voice characteristics sampled during a call (S170) can be very diverse. In one example of the invention, the distribution of characteristics of audio data that has undergone the above-described preprocessing process can be determined based on the location on the statistical distribution of existing stored audio data, and in another embodiment of the invention, it can be confirmed whether the characteristics of the preprocessed data match the stored voice through a machine learning classifier such as a support vector machine (SVP). In another embodiment of the invention, it can be determined whether it is similar to existing voice data by utilizing a deep learning model such as an autoencoder. An autoencoder is an artificial neural network that can learn to change an input signal into a kind of compressed signal in a new latent dimension through an encoder, and to construct a signal similar to the input signal again through a decoder. It has excellent characteristics in reproducing signals learned through artificial neural network learning. However, when a signal of a type that has not been learned is input, it cannot reproduce the original signal well. Using this, the difference between the signal from the autoencoder and the data that has undergone preprocessing is checked, and if the difference between the input and output signals of the autoencoder is above a certain level, a warning is sent so that the identity of the other party can be suspected. The autoencoder used in the present invention is not limited to a simple autoencoder, but is an autoencoder.

Variations of this, such as variational AE, cyclic VAE, and GAN (generative adversarial network), can be used.

In this way, when the other party's voice characteristics are stored, a relatively simple process such as sampling the other party's voice, preprocessing, and comparison with previously stored voice characteristics is performed, so that the other party's identity can be verified in real time as a background process during a voice call with the other party.

If the voice characteristics of the other party are not stored in the database, or some characteristics are stored but are not sufficient for characteristic comparison, the identity of the other party is not verified during the voice call, and sampling is performed (S190). The sampled data can be subjected to preprocessing and feature extraction (S210), and statistical feature analysis, or learning of a machine learning model or an artificial neural network (S220) after the call ends. Whether the learning is complete can be determined by checking the statistical features of the preprocessed voice feature data, or the loss function result of a machine learning or artificial neural network, and if the learning is complete, the result is stored in the database (S240). If the learning is not sufficient, the process (S210 to S230) can be repeated using additional data (S190) sampled during the call. In addition, if the learning is not complete and the sampled data is not sufficient, the intermediate results of S210 to S230 can be stored, and additional learning can be performed using voice data sampled in the next call. The intermediate result storage part is not shown in Figure 2, but it can be stored in the database as an incomplete learning state and can be used to retrieve the incomplete learning state during the next call in the future.

The learning completion process in the case of using an autoencoder, which is an example of the invention, is shown in Fig. 3. Preprocessed voice and feature data (510) are used as inputs to the autoencoder, and learning is repeated until the difference between the output of the autoencoder and the input becomes less than or equal to a threshold value (540), thereby determining the weights inside the autoencoder, and the weights of the artificial neural network for each phone number are stored in a database.

Another example of the invention may utilize an attention-based artificial neural network, in which case embedding may be included in the preprocessing of the voice, and the distance between the voice in the existing database and the embedding of the voice acquired during the call may be used to determine whether the voice is the same person.

All processes shown in S100 of Fig. 2 are carried out during a call with the other party, and processes outside S100 can be carried out during a call or after the call ends.

The use of these voice characteristics to verify the identity of the other party does not require a separate process of having the other party speak a specific pattern or letter that has been designated in advance. In addition, the above description was made for the case where the other party calls, but the same process can be carried out for the owner of the device calling the other party.

In the initial use of the device, training of the autoencoder 120 is required, and the process for this is schematically shown in Fig. 3.

10: Processor 20: Input
30: Preprocessing section 40: Learning section
50 : Comparison section 60 : Warning device
70 : Database 80 : Bus

Claims (8)

A method for preventing spoofing in a personal voice call device, comprising: extracting and storing voice characteristics by the caller's number; comparing the characteristics of the caller's voice data sampled during a call with existing stored voice data to verify the identity of the caller; and a call device utilizing the same.
An input unit for receiving an input voice signal in the configuration of claim 1; and
A preprocessing unit that performs preprocessing of a voice signal or extracts features; and
A learning unit that enables individual allocation of voice characteristics from the output data of the preprocessing unit; and
A comparison unit for comparing voice data calculated from data sampled during a call with previously stored input voice characteristic data; and
An alarm device that notifies when two voice data characteristics are suspected to be the voices of the same person; and
A device consisting of a processor that manages all these processes. A device that uses a method of sampling the voice of the other party without verifying the identity by the voice characteristics of the other party in the case where voice data corresponding to the telephone number of the other party is not secured in the device configuration of claim 1, and performing preprocessing and learning of the sampled voice data during or after the call A method for extracting statistical characteristics of preprocessed voice data in claim 2, storing the same in a database, and confirming that the voice data of the other party during a call is from the same person based on the statistical distribution, and a device using the same An implementation method for learning the voice characteristics of a counterpart through machine learning of preprocessed voice data in claim 2 and a device using the same An implementation method for learning the voice characteristics of a counterpart through learning of an artificial intelligence network using preprocessed voice data in claim 2, and a device using the same Method for using an artificial neural network of the autoencoder type in claim 6 and device for using the same In claim 2, a method of giving a real-time warning when the identity of the other party to a call is suspicious, a method of notifying this through light or sound and a device using the same