WO2014084706A1 - Method for three-dimensional audio localisation in real time using a parametric mixer and pre-decomposition into frequency bands - Google Patents

Abstract

The invention relates to a method for processing at least one monaural audio signal with known parameters, with the aim of generating a stereo signal allowing the perception that sound comes from any spatial position in real time by means of the use of earphones. The method is based on the division of the process into two steps, one step wherein the audio signal is prepared by being dividing into frequency sub-bands and stored in a memory, and the second step that is the spatial processing of the signal by means of the use of empirical parametric tables where the spatial localisation coordinates are inserted either by a person or by a device via an interface. Said embodiment allows this effect to be obtained in real time in devices with little computing power, without being limited thereto.

Description

 THREE-DIMENSIVE REAL-TIME AUDIO LOCATION METHOD USING A PARAMETRIC MIXER AND PREDESCOMPOSITION IN

FREQUENCY BANDS

Field of the Invention

The present invention is located in the field of signal processing and acoustics. Specifically, the present invention relates to a method for processing one or more monaural audio signals, to generate a stereo signal that allows the real-time perception that sound comes from any spatial position through the use of headphones.

Background of the Invention

It is known that the human ear is able to discern the spatial location of sound. This means that a listener with average listening skills can locate the sources of the sounds around him. This 3D characteristic of the sound is based on the fact that each ear receives the acoustic waves with different amplitude and arrival time; information that the brain compares, and based on it, determines the location of the origin of the sound. The effect described above is known as binaural listening.

Different configurations have been used to give the user a binaural listening sensation. Within these solutions, physically positioning the speakers where it is desired that the apparent sound of its origin is the simplest method. Another well known solution in the acoustic environment It consists of using a dummy head that simulates a real head. A microphone is placed in each of the dummy's ears to subsequently make the desired recording. The 3D effect is obtained by reproducing, without any special processing, the sound recorded in each microphone in the corresponding hearing aid (right microphone in the right earphone, and similarly, the left microphone in the left earphone). The main problem with this system is that the spatial location of the sound source is fixed.

To provide flexibility and ability to locate a source of sound to complacency, several methods have been proposed. The most commonly used procedure is the use of the Head-Related-Transfer-Function (HRTF). This transfer function describes mathematically how different frequencies for different locations of sound sources are heard by each of the ears (The anatomical configuration of the human ear is considered in this function). The main problem of this method is the considerable computing capacity that is required and that does not allow its extended use in real-time applications, this being understood as that the auditory perception is congruent with the rest of interactions in a given implementation. This requirement is due to the need to transform the audio signal from the time domain to the frequency domain by means of the Fourier transform; multiply the transfer function by the spectrum of the signal, and then transform the modified signal back to the time domain by means of the inverse Fourier transform. All these operations Mathematics implies processing capacity that in many applications it is not recommended that the required computing capacity be carried out or not, being a clear example the applications in mobile devices. It is true that it is possible to avoid Fourier transforms by using the convolution of the audio signal with the inverse transformation of the HRTF (Head-Related-Impulse-Function), but given the nature in which the HRTF is obtained, it is It is preferable to use the method that uses Fourier transforms.

In relation to the technology described above, US 5105462 mentions a system to create the illusion of sound sources located in different locations in a three-dimensional space, said method employs a conventional system of two loudspeakers, these having to meet certain specific requirements as the distance between them and their location with respect to listening. The original sound sequence used is a monaural signal that is copied into two channels, each representing the right / left channels of a stereo audio system. These two signals are subdivided into frequency intervals. To each frequency range its amplitude is modified and a delay between the right and the left channel is added according to an empirically obtained transfer function. This transfer function is dependent on the desired location and the frequency range to be processed. Using this empirical transfer function avoids the use of the HRTF; however, the amplitude modifications and delays for the different Frequency bands, in a digital implementation of the system, are necessary to do in the frequency domain. This last requirement demands the use of the Fourier transform to transform the monaural audio signal to the frequency domain, apply the transfer function twice to obtain the two stereo audio channels, and finally, use the Fourier reverse transform to get the signals in temporal domain. All these mathematical operations require a large computing capacity that, for devices with limited processing capacity and applications that require this effect in real time, this solution is hardly used. An important point to emphasize on this proposal is that separation in frequency subbands and spatial processing of sound is considered an indivisible process, not to mention that the stages can be separated in order to have independent processing stages, which would grant great flexibility to the method.

In the case of US 5521981, a 3D sound localization system is handled. In this case the main objective is to reduce the need for computation that this effect needs, for this purpose the following considerations are made: First a set of significant locations is defined (in your case example: up, down, right, left, opposite ). Secondly, having a sound bank that is known to be spatially located, they are preprocessed using the HRFT transfer function for each of the locations of the set at a distance from the fixed listener. These Pre-processed audio sequences are stored in memory. This will produce n number of audio sequences for each sound, with n being the number of locations that were previously defined. The third step is to obtain the coordinates in which you want to locate the sound source. As the signal amplitude and delay values are pre-processed and their fixed locations; it is necessary to perform a linear interpolation of the predetermined locations that have already been pre-processed. Although the configuration of this method reduces computing needs, the HRTF function is still used for the spatial location of the sound. Likewise, although the system is divided into two stages, in both modifications are made regarding the location of the sound source

Summary of the Invention

Given the foregoing, an objective of the present invention is to provide a method that allows the treatment of one or several monaural audio signals that when reproduced in two headphones produces the effect of locating the sound source or sources in real time, in three wherein said method comprises a preprocessing stage that prepares the signal by subdividing it into frequency subbands and saving a set of different audio sequences, each representing each of the frequency subbands. Saving is done in a memory device indefinitely; and a real-time positioning stage, which generates the stereo audio signal, taking the audio sequences from the different subbands and modifying them in amplitude and time delay for each audio channel, these parameters being obtained from an empirically generated data table under a trial and error scheme, where the location of the sound source or sources are inserted either by a person or device through an interface. The sequences for each frequency subband are mixed to obtain the final signal of each audio channel and these signals are sent to the respective headphones.

 A further objective of the present invention is to provide a method that allows the treatment of a monaural audio signal, to store it indefinitely and that when reproduced in two headphones produces the effect of locating the sound source three-dimensionally.

Brief Description of the Invention Figures

To complete the description that is being made and in order to help a better understanding of the characteristics of the invention, a series of schemes is attached, where, as an illustration, and not limitation, the following has been represented:

 Figure 1 refers to a block diagram of the audio processing method according to the present invention.

Figure 2 refers to a block diagram of the audio processing method according to the present invention with auxiliary inputs that carry information of the spectral division of the signal in the pre-stage. Processing and its possible format.

 Figure 3 refers to a block diagram of the audio processing method according to the present invention with a number n of audio inputs and n number of spatially located sounds.

Detailed Description of the Representative Modalities of the Invention

The present invention relates to a method that processes a digital monaural signal, but not limited to it, in order to create the real-time illusion that the sound comes from a particular spatial location.

 The method of processing the audio signal is represented in the block diagram of Figure 1 and contains the following steps:

The preprocessing stage (100) can be an implementation in software and / or hardware, of the type one input-several outputs, where said signals, both the input and the outputs, represent a digital audio signal, but not limited to this one. Said stage comprises a module (102) that generates n audio sequences Bl, B2, B3, ..., Bn (103) from the original audio sequence that is input by the input (101), where n is the number of subbands into which the signal spectrum was divided. Each of these audio sequences (103) represents each of the frequency subbands. Said audio sequences are stored indefinitely in a memory device (200) which can be, but not limited to, optical devices, flash memories, discs. hard, etc. The separation into frequency subbands can be done by different techniques. The selected technique is not necessarily restricted in computing capacity, this is due to the fact that the proposed method keeps the processing of the two stages that composes it independent, and can be performed in different equipment. This configuration allows this stage to be carried out with greater precision in equipment with great computing capacity and allows to have higher quality audio sequences.

The second stage corresponds to the spatial processing of the real-time audio signal (300), which can be an implementation in software and / or hardware. This stage accesses the different audio sequences corresponding to the frequency subbands Bl, B2, ..., Bn (103) and sends them to the mixer and phase shifter (302). This module receives the three spatial coordinates (x, y, z) at the entrance (303) that are inserted either by a person or device through an interface, patented or unpatent and without tort of the present invention. These variables can also be given in some other type of polar or spherical coordinates, or some segmentation or representation system that is considered more optimal. From these coordinates, the mixer and phase shifter (302) accesses an empirical parametric table (301), where it obtains the corresponding gain values G (dB) and delay times (t) of the two audio channels for the different frequency bands The delay information (t) may be expressed in seconds, or in some amount that indirectly contains information from a time delay, such as the number of samples or positions to be delayed in a digital signal, known the sampling frequency. The empirical parametric table (301) is obtained with a method based on trial and error. Based on these parameters, the audio sequences Bl, B2, ..., Bn (103) are mixed with their corresponding change in gain and delay time. The result is two signals (304) and (305) that correspond to the left and right audio channel respectively. These signals can be input to a pair of headphones, or they can be stored in a memory device for an indefinite period to be played back at a later time.

It is important to emphasize that the spatial processing stage of the real-time audio signal (300) can be repeatedly implemented in the same application, being able to generate a quantity of 3D audio signals without a known technical limit more than the capacity of computation The scheme corresponding to this configuration is represented in figure 3. In this scheme it is necessary to keep the audio sequences Bl, B2, ..., Bn (103) of each sound that is needed. This operation can be done with a single implementation of the preprocessing stage, only by saving each set of audio sequences (103) in different memory locations; or, have several stages of preprocessing in parallel that save the different sequences, also in different memory locations. With the data already stored in memory (200), the spatial processing stage of the real-time audio signal (300) can be optimized by sharing elements of this stage like the table of values or the mixing system. This multiple implementation would imply the need for a mixer in the total output of the system that in figure 3 is represented as, but not limited to, two adders (306) and (307). The output mixer (306) generates the audio sequence for the left channel (304) from all the left channel output signals of each subsystem and the output mixer (307) generates the audio sequence of the right channel ( 305) from the right channel output signals of each subsystem.

 It is important to add that the decomposition in different frequency ranges is performed by the module (102) using any known and known technique, such as, but not limited to, analog, digital filtering, Fourier analysis, wavelets and convolution . The possibility of converting the signal internally into any of its possible representations to any other possible representation is also contemplated, being able to experience a frequency decomposition process as many times as necessary. This is in order to be able to accept any signal format and deliver any signal format.

Regarding the subdivision of the audio sequence into frequency subbands, the number of these and their spectral width are determined specifically for each application. It may be the case where the bands do not have the same spectral width since certain frequency ranges can be considered equal for auditory perception purposes, and which are not evenly divided into the auditory spectrum; or contemplated that any sub-band or sub-band set is discarded. Some reasons for making this consideration may be, without being limited, for reasons of compression, little audible content in said subbands or optimization of the apparatus.

 Due to the above, it is possible that the preprocessing stage (100) has additional inputs that direct its behavior being possible variables, but not limited to them, the number of output signals, their spectral widths, format and decomposition method . Said implementation is represented in Figure 2, where the spectral division variables of the signal (104) are given externally either by the user or an application which chooses its optimal parameters.

It is important to mention that the mixer and phase shifter module (302) makes modifications to the audio sequence divided into frequency subbands. The first step is that using the spatial coordinates (303), look in the parametric table for the corresponding gain and delay time values for each frequency sub-band and audio channel. The mixer and phase shifter (302) takes the set of audio sequences (103) and modifies this set with the parametric values obtained previously by applying the gain value and delay time corresponding to the left channel to the set (103), and subsequently the values of gain and delay time of the right channel to the same set (103). It is important to mention that the order of profit and delay operations can be reversed. In the end the modified audio sequences of all sub-bands of each channel and the stereo sound signal composed of the output signals (304) and (305) is obtained with the effect of three-dimensional location.

 The present invention requires an interconnectivity of the two stages in which the described method is composed. The connection of the preprocessing stage (100) with the spatial processing stage of the real-time audio signal (300) is given by means of a memory (200). The scheme described in Figure 1 contemplates that the n signals that come from the preprocessing stage (100), and those required by the spatial processing stage of the real-time audio signal have the same format; The invention claimed herein is not limited to this particular case, but leaves the possibility of altering the format of the Bl ... Bn (103) signals of any format to any different format for each signal of the frequency subbands. These modifications are made as many times as deemed necessary, without altering the nature of the exposed invention.

 In accordance with the present invention, the output signals (304) and (305) are connected to headphones which provides the mobility capability method, this property is useful for, but not limited to, applications for mobile devices or consoles. mobile video games Another advantage factor of this lies in the generation of the parametric table, since it is facilitated since it is independent of the acoustics of the place, in addition to this technology can be used in situations where loudspeakers are not possible to use.

It is necessary to add that when the location is changed spatial sound very quickly, this change can generate unpleasant sound effects; To avoid these, you can add a module that softens these transitions to the listener. This "smoother" or softener module receives signals (304) and (305) from an apparatus that implements the present method and provides two audio outputs, each representing the respective left and right channels of a stereo audio system. These signals are smoothed by some special filter or other method generating two signals for the two stereo audio channels without the annoying sounds presented above.

Claims

CLAIMS Having described the invention, the content of the following claims is claimed as property: 1. A method of processing one or more monaural audio signals that allows the simulation of 3D location of sound sources, characterized in that it comprises:  A preprocessing stage (100) that can be implemented in software and / or hardware, of the type one input-several outputs, where the input signal is the original audio sequence that enters through the input (101) and the Output signals are audio sequences (103) that correspond to the frequency subbands into which the original signal is divided. This step stores the different audio sequences (103) in a memory device (200) indefinitely. A stage of spatial processing of the real-time audio signal (300) that can be implemented in software and / or in hardware of several inputs, which consist of n audio sequences (103) (obtained from the memory device (200 )) and spatial coordinates (303). With the use of these, the mixer and phase shifter (302) searches the parametric table (301) for the modification values of the different audio sequences (103) for each of the stereo channels. The mixer and phase shifter (302) adds the audio sequences (103) with their respective modifications to each of the stereo audio channels and sends the signals to the respective audio channel. left audio (304) and right audio channel (305) of headphones.  2. The method of processing one or more monaural audio signals according to claim 1, wherein the preprocessing stage (100) can have several inputs, being the original audio sequence entering through the various inputs (101 ) and spectral decomposition variables (104). The audio output sequences (103) corresponding to the frequency subbands into which the original signal is divided will have spectral characteristics defined by the spectral decomposition variables (104).  3. The method of processing one or more monaural audio signals according to claim 1, wherein in the preprocessing stage (100), the original audio sequence that enters through the input (101) can be broken down into different sub -frequency bands, and that these audio sequences may, by any known and known technique, internally become any of their possible representations to any other possible representation as many times as necessary, this to achieve the desired frequency decomposition.  4. The method of processing one or more monaural audio signals according to claim 1, wherein the step of spatial processing of the real-time audio signal (300) can be implemented several times in the same application, generating an amount of 3D signals at the decision of the system implementer. 5. The method of processing one or more monaural audio signals according to the claims above where the spatial location coordinates are inserted either by a person or device through a real-time interface.  6. The method of processing one or more monaural audio signals according to the preceding claims wherein the spatial location coordinates are taken from a memory device that have been previously entered by a person or device through an interface.  7. The method of processing one or more monaural audio signals according to the preceding claims, wherein the output audio signals are stored in memory for later reproduction.  8. The method of processing one or more monaural audio signals according to the preceding claims wherein the output audio signal passes through a smoothing module.