RU2841604C2 - Reverberation level automated control device and method using perceptual model - Google Patents

Abstract

FIELD: audio encoding means.

SUBSTANCE: reverberation amplification information is determined depending on the input audio signal. Output audio signal is obtained depending on reverberation amplification information by summation of artificial reverberation with input audio signal or with pre-processed audio signal, which depends on input audio signal. Determination of the reverberation amplification information is carried out depending on an estimate of the perceived reverberation intensity in the input audio signal. Determination of reverberation gain information is carried out using a model which returns an estimate of the perceived reverberation intensity in the input audio signal when receiving information on one or more features of the input audio signal.

EFFECT: high efficiency of encoding audio.

15 cl, 7 dwg, 1 tbl

Description

The present invention relates to audio signal processing, in particular to a device and a method for automated control of reverberation level, for example by using a perceptual model.

Reverberation is a very complex element in acoustics and audio signal processing, and its perceived intensity and its control are of particular interest. Audio signals such as musical recordings or radio broadcasts, for example, can be processed through artificial reverberation to emulate the acoustic properties of a particular environment, such as a concert venue or hall. The input signals, for example, can be mixtures of direct signals and spatial ambient signals. Direct signals, for example, can be recordings of singing, musical instruments, and sound events such as gunshots and alarm sirens. The term "direct sounds" indicates that these sounds are directional and can be localized as coming from a single direction. Spatial ambient (or diffuse) sound components, for example, can be reverberation and environmental sounds that are not perceived as coming from specific directions, such as wind and rain. In musical recordings, reverberation is the most noticeable source of ambient sounds.

The perceived level of reverberation depends on the input signal and the impulse response of the reverberation, such as the length of the pre-delay and the reverberation tail (see [1]). Non-stationary input signals with transient parts and fast attacks of the sub-band envelopes, such as drum sounds, produce higher reverberation intensity.

Input signals with fast decaying envelopes and quiet parts are less effective in masking reverberation (see [2]). When reverberation times and pre-delays are short, the input signal coincides with and partially masks the reverberation signal over a greater extent than when reverberation times and pre-delays are long. In addition, the acoustic environment (see [3]), the playback system (see [4]) and other aesthetic aspects related to the genre of music influence the preferred gain setting.

A model for predicting multiple spatial attributes trained using listening-based test data is described in [5], but the proposed model is not applied in real time, and the model itself is also more complex since it uses up to 12 audio features, resulting in performance comparable to the presented algorithm that uses only 3 features. The model for the perceived level of late reverberation in (see [6]) uses the partial loudness level of the direct and reverberant components of the input audio signal. A study of the perceived level of artificial late reverberation in [1] shows that the perceived level depends on the reverberation time and the input signals, rather than on the interchannel correlation of the impulse response and the differences in rating between individuals, and repeated tests on an identical individual are similar. The preferred level is studied in [7], concluding that aesthetic quality suffers more when the applied reverberation is higher than the preferred level than when it is lower.

The objective of the present invention is to create improved concepts for automatic control of reverberation level. The objective of the present invention is achieved by the device according to claim 1, the method according to claim 19 and the computer program according to claim 20.

A device is provided for processing an input audio signal comprising one or more audio channels to obtain an output audio signal according to an embodiment. The device comprises a reverberation gain determination module configured to determine reverberation gain information depending on the input audio signal. In addition, the device comprises a signal processor configured to obtain an output audio signal depending on the reverberation gain information by summing artificial reverberation with the input audio signal or with a pre-processed audio signal that depends on the input audio signal.

In addition, a method for processing an input audio signal in such a way as to obtain an output audio signal according to an embodiment is provided. The method comprises:

- determining the reverberation gain information depending on the input audio signal, and

- obtaining an output audio signal depending on the reverberation gain information by summing artificial reverberation with the input audio signal or with a pre-processed audio signal that depends on the input audio signal.

In addition, a computer program is provided for implementing the above-described method when executed on a computer or in a signal processor.

Embodiments of the present invention are described in more detail below with reference to the drawings, in which:

Fig. 1 illustrates a device for processing an input audio signal comprising one or more audio channels to obtain an output audio signal according to an embodiment.

Fig. 2 illustrates a block diagram of a reverberation control algorithm according to an embodiment.

Fig. 3 illustrates the mean ratings per test item based on listening and the overall mean with 95 confidence intervals according to an embodiment.

Fig. 4 illustrates predicted reverberation intensity compared to observed reverberation intensity according to an embodiment.

Fig. 5 illustrates: in Fig. 5 - top - a spectrogram of an audio signal with a transition from classical music to pop music after 25 s; in Fig. 5 - in the middle - the equivalent reverberation level before and after temporal smoothing; and in Fig. 5 - bottom - the smoothing factor and the overall scaling value according to an embodiment.

Fig. 6 illustrates the reference gain when sending reverberation compared to the equivalent reverberation level with a fitted polynomial according to an embodiment.

Fig. 7 illustrates the equivalent reverberation level compared to the gain error when sending reverberation according to an embodiment.

Fig. 1 illustrates a device for processing an input audio signal comprising one or more audio channels to obtain an output audio signal according to an embodiment.

The device comprises a reverberation gain determination module 110 configured to determine reverberation gain information depending on an input audio signal.

In addition, the device comprises a signal processor 120, configured to obtain an output audio signal depending on the reverberation gain information by summing the artificial reverberation with the input audio signal or with a pre-processed audio signal, which depends on the input audio signal.

According to an embodiment, the reverberation gain determination module 110 may, for example, be configured to determine reverberation gain information depending on an estimated perceived reverberation intensity in the input audio signal.

In an embodiment, the reverberation gain determination module 110, for example, may be configured to determine reverberation gain information by using a model that returns an estimate of the perceived intensity of reverberation in the input audio signal upon receiving information about one or more features of the input audio signal.

According to an embodiment, the model used by the reverberation gain determination module 110 may, for example, be a linear regression model using one or more feature values for one or more features of the input audio signal as input to the linear regression model.

In an embodiment, to obtain the reverberation sending gain, the reverberation gain determining module 110, for example, may be configured to determine the reverberation sending gain as reverberation gain information by transforming an estimate of the perceived reverberation intensity in the input audio signal according to a transform function. An equivalent reverberation intensity (= the perceived intensity after applying the scaling values), for example, may be supplied to the transform function. For example, the estimate of the perceived reverberation intensity is transformed into test data based on listening, and accordingly, a fitted curve, for example, may then be used for transforming into the reverberation sending gain.

According to an embodiment, one or more features of the input audio signal may, for example, depend on an inter-channel correlation of at least one of the one or more frequency subbands of two audio channels for one or more audio channels of the input audio signal.

In an embodiment, one or more features of the input audio signal, for example, may depend on a spectral smoothness index of at least one of the one or more frequency subbands of one or more audio channels of the input audio signal.

According to an embodiment, the reverberation gain determination module, for example, may be configured to determine an estimate of the perceived intensity of reverberation in the input audio signal by using a model. The reverberation gain determination module, for example, may be configured to determine one or more scaling factors depending on one or more features of the input audio signal. Furthermore, the reverberation gain determination module, for example, may be configured to determine reverberation gain information depending on the estimate of the perceived intensity of reverberation and depending on one or more scaling factors.

In an embodiment, one or more scaling factors, for example, may depend on an inter-channel correlation of at least one of the one or more frequency subbands of two audio channels for one or more audio channels of the input audio signal.

According to an embodiment, one or more scaling factors may, for example, depend on the presence of transient signal components in at least one of the one or more audio channels of the input audio signal.

In an embodiment, one or more scaling factors, for example, may depend on an index of spectral transient parts of at least one of the one or more audio channels of the input audio signal. The index of spectral transient parts, for example, may be set depending on the following:

Where:

,

wherein m indicates a time index, wherein k indicates a frequency index, and wherein b _k,i indicates one of one or more frequency bins in a frequency band/subband of k frequencies, wherein Y _d (m, b _k,i ) indicates a coefficient with an absolute value with a time index m for one or more frequency bins b _k,i in a frequency band of k of said at least one of the one or more audio channels or a combination of said at least one of the one or more audio channels, and wherein fs() denotes a recursive averaging in time and implements a smoothing of the subband signal.

According to an embodiment, the signal processor 120, for example, can be configured to generate a pre-processed audio signal by de-reverberating the input audio signal to weaken the original reverberant signal components of the input audio signal. The signal processor 120, for example, can be configured to obtain an output audio signal depending on the reverberation gain information by summing the artificial reverberation with the pre-processed audio signal.

In an embodiment, the signal processor 120, for example, can be configured to generate a pre-processed audio signal by performing time smoothing of the input audio signal. The signal processor 120, for example, can be configured to obtain an output audio signal depending on the reverberation gain information by summing artificial reverberation with the pre-processed audio signal.

According to an embodiment, the signal processor 120, for example, may be configured to adjust the amount of temporal smoothing depending on changes in the input audio signal.

In an embodiment, the signal processor 120, for example, may be configured to adjust the amount of temporal smoothing depending on changes in the input audio signal.

According to an embodiment, the signal processor 120, for example, may be configured to adjust the amount of temporal smoothing depending on a change in the volume level of the input audio signal.

In an embodiment, the signal processor 120, for example, may be configured to adjust the amount of temporal smoothing depending on changes in the variance of one or more features of the input audio signal.

According to an embodiment, the device, for example, may comprise a circular buffer, for example, with a large length and/or strong overlap, for real-time processing, which, for example, may be configured to receive an input audio signal or a pre-processed audio signal. The signal processor 120, for example, may be configured to process the input audio signal or the pre-processed audio signal in the circular buffer to obtain an output audio signal.

Specific embodiments are described below. First, for example, an overview can be presented that presents an algorithm according to an embodiment, after which a listening test and results including a statistical evaluation will be given. In addition, a model for predicting the perceived intensity of reverberation according to an embodiment is presented, post-processing using scaling factors according to an embodiment is presented, real-time application of a trained model according to an embodiment is described, and conversion to gain when sending reverberation is presented.

In some embodiments, the perceived intensity of reverberation in audio signals, for example, can be estimated, and the level of the artificial reverberation signal, for example, can be controlled in such a way that the artificially reverberated output signal, for example, can have similar reverberation properties with the corresponding input signal. The estimation, for example, can use a linear regression model with inter-channel coherence in a frequency subband and a spectral smoothness index as input features, which are trained using test data based on listening. For adaptation (for example, for applying scaling factors), artificial reverberation control signals, for example, can be calculated depending on the properties of the time modulation and/or, for example, depending on the correlation between the input channel signals and, for example, can be used to calculate an equivalent reverberation level. The resulting value, for example, can be post-processed using signal-adaptive integration. The concepts can, for example, be applied to control the reverb send gain for artificial reverb used to reproduce sound in a car (reverb send gain: e.g., (artificial) reverb send gain).

Reverberation is a very complex element in acoustics and audio signal processing, and embodiments focus on its perceived intensity and on its control, for example by adjusting the gain when sending the reverberation. Some embodiments, for example, can be implemented independently of other aspects, such as the times of the frequency-dependent impulse response reverberation or its correlation between channels. Other embodiments, for example, can take these aspects into account.

A brief schematic view of a specific embodiment is shown in Fig. 2. The first stage, for example, may comprise dereverberation and audio feature extraction. In particular, Fig. 2 illustrates a block diagram of a reverberation control algorithm according to an embodiment. Feature values are input to a model for predicting the perceived reverberation intensity in the input signal. The model uses linear regression and is trained using listening-based test data. The model output is post-processed with signal-adaptive scaling factors that are manually adjusted with respect to the properties of the new summed artificial reverberation in order to account for the changed interaction of the excitation signal and the reverberation due to the different characteristics of the artificial and "original" reverberation. It should be noted that data-driven modeling of the perceived reverberation intensity for the artificial reverberation (which is performed for the original reverberation) requires listening-based test data for a variety of parameter settings and cannot adapt when these settings are changed. Therefore, such an approach is not preferred.

The equivalent reverb level is post-processed with real-time signal-adaptive time integration. The result is used to calculate reverb send gains using a function fitted to the preferred send gains adjusted by expert listeners. The calculated reverb send gain controls the level of artificial reverb applied to the dereverberated input signal. Dereverberation is used to attenuate the reverberant signal components of the input, since summing artificial reverb with an input signal with large amounts of "original" reverb results in a loss of transparency, or in the perception of too much reverb, or in an aesthetically undesirable superposition of the two reverb signals.

Below are the results of the listening test.

In the listening test, 27 loudness-normalized audio signals with durations ranging from 4.8 s to 14.7 s are presented. The elements are selected in such a way that only small changes in spatial signal cues (the amount of reverberation and the distribution of energy in the stereo panorama) are evident. The set of elements varies from very dry to very reverberant and includes recordings of various musical genres, as well as recordings of solo instruments and dry speech recordings.

Participants are asked to rate the perceived reverberation intensity and ensemble width by adjusting a slider on a discrete unipolar scale ranging from 1-9. The labels "Very Low", "Low", "Medium", "High", and "Very High" are distributed evenly across the scale. An additional attribute, ensemble width, is provided to help the listener determine the reverberation intensity independently of the stereo panorama. The test begins with a training session in which three stimuli are presented that provide examples of low reverberation magnitude and high ensemble width, and vice versa. Fifteen listeners participate in the test.

Fig. 3 illustrates the average ratings per test item based on listening and the overall average with 95% confidence intervals according to an embodiment. In particular, Fig. 3 shows the average values and 95% confidence intervals of the reverberation rating. The confidence intervals are obtained as follows:

, (1)

Where is the estimated mean, tα _/2,n-1 is the quantile α/2 (here α=0.05) from the t-distribution with n-1 degrees of freedom, is the estimated standard deviation, and n is the sample size [8].

Below in the text, a linear regression model according to an embodiment is presented.

A computational model for predicting the perceived reverberation intensity of an audio signal and training thereof according to specific embodiments is described below.

First, a description of the model according to the embodiment will be given.

To assess perceived intensity reverberations in the form of a linear combination of x _k input features, a linear regression model is applied, which consists of the following:

, (2)

with the coefficients β _k of the model, the bias value β ₀ and the number K of input features.

The extraction of audio features according to an embodiment is described below.

The input to the linear regression model is the inter-channel coherence (ICC) in the sub-band and the spectral smoothness measure (SFM) calculated from the short-time Fourier transform (STFT) coefficients. ICC is calculated in 5 and SFM in 4 frequency bands, as shown in Table 1.

Table 1 illustrates the division into ICC and SFM frequency bands according to an embodiment.

Frequency band ICC SFM 1 70Hz - 164Hz 70Hz - 305Hz 2 164Hz - 445Hz 305Hz - 680Hz 3 445 Hz - 4.9 kHz 680 Hz - 1.4 kHz 4 4.9 kHz - 11 kHz 1.4 kHz - 21 kHz 5 11 kHz - 21 kHz

Table 1

The STFT coefficients are computed from audio signals sampled at 48 kHz with a frame size of 1024 samples, a hop length of 512 samples, and no zero padding. STFT frames with loudness levels below -65 LUFS are removed to ensure that the model training is agnostic to silent periods such as speech pauses. The ICC [9] is computed from both channel signals of a two-channel stereo input signal according to the following:

where Y _1,1 (m,k) and Y _2,2 (m,k) are the automatic power spectral densities (PSDs) of the left and right channels, respectively, and Y _1,2 (m,k) is the cross PSD, where all the PSDs are accumulated over the frequency bins corresponding to the k-th frequency band in the time index m. For example, if b _k (i,1) indicates the i-th complex spectral bin of the k-th frequency band of the first audio channel of the input audio signal, and b _k (i,2) indicates the i-th complex spectral bin of the k-th frequency band of the second audio channel of the input audio signal, then Y _1,1 (m,k), Y _1,2 (m,k), and Y _2,2 (m,k), for example, can be defined as follows:

,

where b _k (m,i,1)* indicates the complex conjugate of b _k (m,i,1), and b _k (m,i,2)* indicates the complex conjugate of b _k (m,i,2), with time index m. N, for example, may indicate the number of downmixed squared absolute value STFT coefficients for the b _k bins in the k frequency band.

For example, in a particular embodiment, the frequency band, for example, may comprise one or more frequency bins per audio channel (For example, in a particular embodiment, the frequency band, for example, may be only one bin). For example, in a particular embodiment, for obtaining the inter-channel correlation for said frequency band, for example, formula (3) may be used.

SFM (see [10]) is calculated as the ratio of the geometric mean to the arithmetic mean according to the following:

Where:

where g() is a compression function, e.g., the logarithm or square root or y=x ^0.25 , and Y _d (m, b _k,i ) ² are the downmixed squared absolute value STFT coefficients of the left and right input channels for the i-th bins b _k,i in a bandwidth of k consisting of N bins with time index m. (Thus, N, for example, may indicate the number of bins b _k,i in a bandwidth of k.) For example, in an embodiment with two audio channels, the coefficients Y _d (m, b _k,i ) ² with the downmixed squared absolute value of the STFT, for example, may be obtained by squaring the coefficients Y _d (m,b _k,i ) with the downmixed absolute value of the STFT, and the coefficients Y _d (m, b _k,i ) with the downmixed absolute value of the STFT, for example, may be obtained by combining (e.g., averaging) the coefficients with the absolute value of the two audio channels of the frequency bin of the k-th frequency band.

In embodiments, features are extracted for each frame (e.g., for each block), and then the mean and standard deviation are calculated to obtain block-wise single values, which are then used in training a regression model.

For example, each block, for example, may contain audio data for playing one second of an audio recording. A block, for example, may contain spectral data for 94 points in time (for example, for 94 frames, provided that a frame, for example, may have, in a particular embodiment, a duration of 21.3 milliseconds, for example, approximately 94 frames / second, when the frames have a 50% overlap). In other embodiments, a block, for example, may contain spectral data for any other number of points in time. In an example, the spectrum, for example, may be divided into five frequency bands and / or, for example, into four frequency bands, for example, as indicated in Table 1.

The ICC value is determined, for example, taking into account the ICC, for example, for each of the five frequency bands, for example, of Table 1, and, for example, for each of the 94 time points. For example, for each of the five frequency bands, the arithmetic mean of, for example, the 94 ICCs of the block is determined, and by virtue of this, for example, five arithmetic mean ICC values for the five frequency bands can be obtained. Alternatively, for example, for each of the five frequency bands, the standard deviation of, for example, the 94 ICCs of the block is determined, and by virtue of this, for example, five ICC values of the standard deviation for the five frequency bands can be obtained. By virtue of this, for example, 10 ICC-related feature values are obtained, for example, for the five frequency bands for the block.

The SFM value is determined, for example, taking into account the SFM, for example, for each of the four frequency bands, for example, of Table 1, and, for example, for each of the 94 points in time. For example, for each of the four frequency bands, the arithmetic mean of, for example, the 94 SFM of the block is determined, and by virtue of this, four arithmetic mean SFM values can be obtained, for example, for the four frequency bands. Alternatively, for example, for each of the four frequency bands, the standard deviation of, for example, the 94 ICC of the block is determined, and by virtue of this, for example, four SFM values of the standard deviation for the four frequency bands can be obtained. By virtue of this, for example, 8 SFM-related feature values are obtained, for example, for the four frequency bands for the block.

For example, regarding the combination of both the ICC example and the SFM example, 18 feature values for a block can be obtained.

The training of the model according to the embodiment is described below.

Processing short blocks of data is vital for real-time processing, particularly for non-stationary signals, and to increase the available training data. Each feature is windowed with a training block length of 6 s with zero overlap. Features that are shorter than 12 s are processed as a single block over the entire length. The performance of training with overlapping blocks is also evaluated without observing any improvement in model accuracy.

The training dataset is expanded by adding a second excerpt with similar spatial characteristics for some of the songs used in the listening test. Additionally, a set of critical items whose predictions are inappropriate are added, with anchor ratings provided by a single expert listener. To ensure that the model predicts small values for dry and clean speech signals, recordings of male and female speakers are added to the dataset with anchor ratings set to -1MOS (mean expert score). Expanding the dataset and using multiple bins for each item yields a total of 100 observations.

To be able to account for nonlinear relationships between the listening test data and the feature values, exponentiation is applied to the means and standard deviations of the feature values. The exponent is determined by manual adjustment based on an evaluation of the residual plot, which reveals nonlinearities that by definition cannot be modeled by linear regression.

The model is trained using an ordinary least squares (OLS) based algorithm that minimizes the mean square error between predictions and the average ratings obtained in the listening test. The evaluation is performed with leave-one-out cross-validation [11], in which the model is trained with all observations except one and evaluated using the left-out item. This procedure is repeated for each item.

The selection of features according to the embodiment is described below.

The training starts with all available values, for example, the arithmetic mean and standard deviation, per subband feature. The subband feature, for example, can be the ICC or SFM for the said subband (band).

To avoid overfitting and reduce computational burden, non-preferential independent variables are removed one by one by evaluating p-values. Low p-values indicate that there is likely no relationship between the corresponding feature value and the listening test result (see [11]).

For example, in the above example with 18 feature values (10 ICC-related feature values and 8 SFM-related feature values), gradually removing all irrelevant feature values, for example, can lead to a model with 4 input variables (e.g., 14 other irrelevant or less relevant input variables/feature values, for example, can be gradually removed, for example, by performing a multivariate regression analysis (see, for example, [11])). The performance of this model is shown in Fig. 4.

In particular, Fig. 4 illustrates the predicted reverberation intensity compared to the observed reverberation intensity according to an embodiment. The predictions of the training blocks of the listening-based test items are shown using dots, and the predictions obtained for the training blocks of the extended data set (items from the extended data set) are shown using triangles. The predictions for the average per item are shown using crosses. The markers representing blocks of the same item have the same listening-based test rating. The graph reveals a good correlation of the predictions and observations with a correlation coefficient of 0.75 and a generalized average error of 1.27 MOS. For listening-based test ratings in the region of 3 MOS, there is a tendency for values to be too high, while high ratings are slightly underestimated.

The following describes the calculation of the reverb sending gains according to the embodiment. In particular, the calculation of the preferred reverb sending gain for the artificial reverb is described, taking into account the perceived intensity of the primary reverb in the input signal. Since the original reverb (of the input signal) and the secondary (artificial) reverb are not the same, an equal reverb level may result in different perceived reverb intensities. Therefore, the reverb sending gain cannot be directly calculated, taking into account the perceived intensity of the reverb.

Data-driven modeling of perceived intensity for artificial reverb is not practical because it requires subjective data for different reverb settings.

In embodiments, manually adjusted scaling coefficients s _t and s _c are introduced to compensate for signal-dependent effects, such as described below. To describe the processing, the intermediate value, the equivalent level reverberation is set as follows:

The equivalent reverb level represents the desired level of artificial reverb for a given input signal that results in similar reverb intensity for the artificial and original reverb. The equivalent reverb level is then converted to a reverb send gain using the conversion function defined in a separate adjustment experiment.

Aspects of the equivalent reverberation level according to embodiments are described below.

Signals with strong transients, such as drum sounds, result in higher perceived reverberation intensity than stationary signals and are less effective at masking reverberation.

Artificial reverberation with large time constants may require a lower reverberation level to produce a similar intensity perception to the original reverberation. Some embodiments provide a new spectral transient measure (STM) to quantify the intensity of transient parts in a signal as follows:

Where:

fs() stands for recursive time averaging and implements subband signal smoothing.

For example, in a particular embodiment, the frequency bands, for example, may be the frequency bands of table 1, left column, or table 1, right column. For example, other frequency bands/frequency band configurations may alternatively be used. Each of the frequency bands, for example, in such a case may contain, for example, all time-frequency bins in said frequency band for time index m.

A single STM value is obtained by averaging the STM values of the frequency bands from 23 to 70 Hz and from 5.2 to 21 kHz. This choice of frequency bands emphasizes the transient parts caused by percussion instruments, while the transient parts of other instruments, such as the piano, are taken into account to a lesser extent.

The scaling factor s _t is calculated from STM as follows:

Where denotes the average STM value of 1 s blocks, then cut off according to:

where m _t , n _t , s _{t, min} and s _{t, max} are adjustable parameters that are adjusted depending on the properties of the artificial reverberation.

If the reverberation time for the "original" reverberation exceeds that of the artificial reverberation, a higher equivalent reverberation level is required. The second scaling factor s _c is designed to adjust the estimated reverberation intensity for input signals containing a significant amount of reverberation with a long reverberation time, such as classical music. It is assumed that such elements are instead decorrelated due to the large number of diffuse sound components. The scaling factor s _c is calculated as follows:

Where is the average ICC value of all frequency bands and frames in 1 s blocks and is then clipped according to the following:

where m _c , n _c , s _{c, min} and s _{c, max} are adjustable parameters.

Signal-adaptive time integration according to embodiments is described below.

To control the gain when sending reverb in real time, it is necessary to respond to changes in the input signal without introducing noticeable modulations in the reverb level. To allow this with low latency and low computational load, predictions are calculated in 8 s blocks, each with an overlap of 7 s, such that new predictions are calculated at a rate of 1 Hz. Signals with low input levels, such as below -65 LUFS, are not fed to the model, which happens during training. Equivalent level reverberation (6) is temporally smoothed using recursive averaging with a single-pole IIR filter:

with smoothing coefficient α.

In order to avoid too high reverberation levels when transitioning between input signals with significantly different characteristics, track changes are detected and a fast adaptation of the equivalent reverberation level is implemented. The fast adaptation is performed by reducing the model block length to 1 s and increasing the smoothing coefficient α. For detection, changes in the STM variance are identified, since a change in the number of transient parts is a good indicator for a transition, for example, from music to speech, requiring a much lower reverberation level.

Fig. 5 illustrates the real-time processing of an explanatory input file that consists of 25 s of classical music and an abrupt transition to pop music, which can be observed in the upper graph showing the spectrogram according to an embodiment. In particular, Fig. 5 - top - illustrates the spectrogram of the audio signal with a transition from classical music to pop music after 25 s. Fig. 5 - middle - illustrates the equivalent reverberation level before (solid line) and after (dotted line) temporal smoothing. Fig. 5 - bottom - shows the smoothing factor (solid line) and the overall scaling value (dotted line).

The middle subgraph illustrates the equivalent reverberation level before (solid line) and after (dashed line) temporal smoothing with a value of around 7 MOS for classical music and 2.2 MOS for the pop element. The lower subgraph shows the product of the scaling values (s _t *s _c ), called the "total scaling value", as a dotted line. Due to the low correlation of the classical element, the total scaling value exceeds 1 before the track change. Increasing the number of transients at t = 25 s triggers the detection of the track change and reduces the total scaling value to approximately 0.7.

The smoothing factor α, illustrated by the solid line in the lower subgraph, decreases after the adjustment phase of 11 s, since the loudness level and the equivalent reverberation level are then fairly stationary. The detection of a track change at t = 25 s temporarily increases α and leads to a rapid adaptation of the equivalent reverberation level.

The conversion from reverb prediction to reverb sending gain according to embodiments is described below.

For example, as a final step, the gain when sending a reverb processor, for example, can be calculated taking into account the equivalent reverb level. For this purpose, a second listening test is performed, in which 5 expert listeners adjust the gains when sending two artificial reverberations simulating the acoustic environment of a concert hall (T60=2.2 s) and a jazz club (T60=1.3 s) in a car according to their preferences.

Fig. 6 illustrates the reference reverberation send gain compared to the equivalent reverberation level with a fitted polynomial according to an embodiment. In particular, Fig. 6 illustrates the obtained average send gain compared to the equivalent reverberation level with 95% confidence intervals according to equation (1) for all 30 test items for two artificial reverberations. Since a longer reverberation tail results in less masking and a higher perceived reverberation intensity with transients requiring a lower reverberation send gain, the adjustable parameters in equation (9) and equation (11) are adjusted such that strong transients and low correlation result in a lower equivalent reverberation level for a concert hall compared to a jazz club. As a result, one separate conversion curve can be used for both reverberations. With a value of 0.81, the Spearman's rank correlation coefficient confirms a monotonic relationship between the two variables, which can be approximated by a degree 3 polynomial with low error.

Gain adjustments are used to fit the polynomial transform:

,

shown in Fig. 6, which transmits a smoothed equivalent level reverb into the predicted g gain when sending the reverb.

Aspects of evaluation are discussed below.

To estimate the polynomial transform, a leave-one-out cross-validation technique is used, meaning that all pairs of values except one are used to fit the polynomial function, and the last pair of data serves to obtain the error for that particular data point. This process is repeated for all listening-based test items, with a mean absolute error (MAE) of 1.74 dB. According to [7], such a deviation from the preferred gain when sending reverberation results in a negligible decrease in aesthetic quality. The 95% confidence interval limits obtained according to equation (1) are 1.27 dB and 2.21 dB. The absolute maximum error is 5.11 dB, and the correlation between the predicted and observed values is 0.89.

Fig. 7 illustrates the equivalent reverberation level versus the gain error when sending reverberation according to an embodiment. The scatter plot illustrated in Fig. 7 shows the gain errors when sending reverberation versus the equivalent reverberation level, revealing a uniform distribution of errors over the entire range of values with only one outlier value of 2.6 MOS.

Concepts for controlling the level of artificial reverberation are presented. Some of the concepts presented, for example, may use a model for predicting the perceived intensity of reverberation based on a linear regression model trained using listening-based test data. The method uses manually tuned adaptations to different settings of artificial reverberation parameters. Fast adaptation to changes in the characteristics of the input signals is realized through signal-adaptive time integration. The algorithm is evaluated using an in-car audio system and allows for prediction of preferred gains when sending reverberation with an MAE of 1.74 dB.

Extensions of the above-described envisaged embodiments are described below.

For example, other application scenarios can be implemented.

Embodiments can be used, for example, to control reverberation in various other application scenarios. In binaural sound reproduction, artificial reverberation is used to support a sense of externalization, such as the listener localizing sound events, objects, and sources outside the head. This is in contrast to standard headphone sound reproduction, in which sounds are typically perceived as coming from inside the head.

Artificial reverberation is used in music production, where it is applied to a mix of multiple sound sources and to individual sound sources. The proposed method can also be extended to predict the perceived reverberation magnitude for individual sound sources or individual components of a mixed signal (e.g., a group of sound sources). The information on the magnitude per individual component enables automated control or artificial reverberation in sound generation (music production, film sound, podcasts, and other user-generated content).

Other embodiments, for example, may implement alternative models.

The linear model, for example, can be replaced by other data-driven models, such as any function:

where is the scalar output is calculated, given the input vector x and the parameters Θ of the model, which are determined by optimizing a criterion (e.g. minimizing the mean square error) between and target outputs y. Another well-known class of such trainable models are artificial neural networks, in particular deep neural networks (DNN). It should be noted that the larger the set of parameters Θ, the more data is required to train the model, so that the performance is good not only for the training data, but also for a new pair data that is not used during training. This property is called "generalization." However, with additional training data, other models such as DNNs can produce better predictions (with greater accuracy).

Such DNNs are implemented by combining layers of processing modules, where the module itself has trainable parameters. Commonly used layer types are convolutional layers, dense layers (also called “fully connected layers”), and recurrent layers. They differ in what types of modules are implemented and how these modules are connected to each other. Additional layer types are used to support the learning process, where learning means the process of optimizing the parameters Θ through numerical optimization.

Additional embodiments are provided below.

A device and/or method is provided for controlling the level of an artificially generated reverberation signal that is to be summed with an audio signal, using a model of the perceived intensity of reverberation in the original input signal.

According to an embodiment, the input signal may, for example, be dereverberated to attenuate the original reverberant signal components.

In an embodiment, a linear regression model, for example, may be used with different audio feature values as input.

According to an embodiment, SFM subbands and/or ICC subbands, for example, may be used as input audio features.

In an embodiment, for example, adjustable scaling values may be used to alter prediction based on models that depend on audio feature values to compensate for the altered interaction of artificial reverberation and the dereverberated input signal compared to the original reverberation and the direct signal.

According to an embodiment, for example, adjustable scaling values may be used that depend on the ICC subband and the STM subband.

In an embodiment, for example, a highly overlapping circular buffer may be used for real-time processing.

According to an embodiment, adaptive temporal smoothing, for example, may be used to adjust the amount of smoothing according to changes in the input signal.

In an embodiment, for example, changes in loudness level and model-based predictions may be estimated to control the smoothing factor of the temporal recursive averaging.

According to an embodiment, track changes, for example, may be detected depending on changes in the variance of audio features in order to temporarily reduce the smoothing factor of the temporal recursive averaging and reset the ring buffer.

In an embodiment, a transfer function fitted to test data based on listening, for example, may be used to convert adjusted model-based predictions into gains when sending reverberation.

The proposed concepts can be applied, for example, to car audio reproduction to emulate large-scale acoustic environments with pleasant-sounding spatial properties. The emulation of an acoustic environment is achieved by processing the input audio signal in such a way that the reverberant signal components of the output signal are perceptually similar to the reproduction of direct signal components in the new environment. The implementation has low latency and low workload.

Although some aspects are described in the context of a device, it is obvious that these aspects also represent a description of the corresponding method, wherein a block or device corresponds to a step of the method or a feature of a step of the method. Similarly, aspects described in the context of a step of the method also represent a description of the corresponding block or element, or a feature of the corresponding device. Some or all of the steps of the method can be performed by (or using) a hardware device, such as, for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, one or more of the most important steps of the method can be performed by this device.

Depending on the specific implementation requirements, embodiments of the invention may be implemented in hardware or in software, or at least partially in hardware or at least partially in software. The implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, having stored electronically readable control signals that interact (or are capable of interacting) with a programmable computer system in such a way that the corresponding method is performed. Accordingly, the digital storage medium may be machine-readable.

Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of interacting with a programmable computer system such that one of the methods described herein is performed.

In general, embodiments of the present invention can be implemented as a computer program product with a program code, wherein the program code is designed to implement one of the methods when the computer program product is executed on a computer. The program code can be stored, for example, on a machine-readable medium.

Other embodiments comprise a computer program for implementing one of the methods described herein stored on a machine-readable medium.

In other words, an embodiment of the method according to the invention is therefore a computer program having a program code for implementing one of the methods described in this document when the computer program is executed on a computer.

Therefore, a further embodiment of the methods according to the invention is a data carrier (digital data carrier or machine-readable carrier) containing a recorded computer program for implementing one of the methods described herein. The data carrier, digital data carrier or carrier with recorded data is usually tangible and/or permanent.

Therefore, a further embodiment of the method according to the invention is a data stream or a sequence of signals representing a computer program for implementing one of the methods described in this document. The data stream or sequence of signals can, for example, be configured to be transmitted via a data connection, such as via the Internet.

An additional embodiment comprises a processing means, such as a computer or a programmable logic device, configured to perform one of the methods described herein.

An additional embodiment comprises a computer having a computer program installed for implementing one of the methods described herein.

An additional embodiment according to the invention comprises a device or system configured to transmit (e.g. by electronic or optical means) a computer program for implementing one of the methods described in this document to a receiving device. The receiving device, for example, may be a computer, a mobile device, a storage device, etc. The device or system, for example, may comprise a file server for transmitting the computer program to the receiving device.

In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to implement some or all of the functionalities of the methods described herein. In some embodiments, the field programmable gate array may cooperate with a microprocessor to implement one of the methods described herein. In general, the methods are preferably implemented by any hardware device.

The apparatus described in this document may be implemented using a hardware device, using a computer, or using a combination of a hardware device and a computer.

The methods described in this document may be implemented using a hardware device, using a computer, or using a combination of a hardware device and a computer.

The above-described embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and changes to the arrangements and details described herein will be obvious to those skilled in the art. Therefore, they are intended to be limited only by the scope of the claims below and not by the specific details presented by way of description and explanation of the embodiments herein.

Bibliographic list

[1] Paulus, J., Uhle, C. and Herre, J. "Perceived Level of Late Reverberation in Speech and Music", in Proc. AES 130th Conv., 2011.

[2] Gardner, W. G. and Griesinger, D. "Reverberation level matching experiments", in Proc. Sabine Centennial Symposium, Acoust. Soc. of Am., 1994.

[3] Leonard, B., King, R. and Sikora, G. "The Effect of Playback System on Reverberation Level Preference", in Audio Engineering Society Convention 134, 2013.

[4] Leonard, B., King, R. and Sikora, G. "The Effect of Acoustic Environment on Reverberation Level Preference", in Audio Engineering Society Convention 133, 2012.

[5] Sarroff, A. M. and Bello, J. P. "Toward the Computational Model of Perceived Spaciousness in Recorded Music", J. Audio Eng. Soc, 59(7/8), pp. 498-513, 2011.

[6] Uhle, C., Paulus, J. and Herre, J. "Predicting the Perceived Level of Late Reverberation Using Computational Models of Loudness", in Proc. 17th Int. Conf. on Digital Signal Process. (DSP), 2011.

[7] Paulus, J., Uhle, C., Herre, J. and Höpfel, M. "A Study on the Preferred Level of Late Reverberation in Speech and Music", in Proc. of the 60th Int. Conf. of Audio Eng. Soc., 2016.

[8] Gilleland, E. “Confidence Intervals for Forecast Verification,” National Center for Atmospheric Research, Boulder, Colorado, 2010.

[9] Allen, J., Berkeley, D. and Blauert, J. "Multimicrophone Signal-Processing Technique to Remove Room Reverberation from Speech Signals", J. Acoust. Soc. Am., 62, 1977.

[10] Gray, A. H. and Markel, J. D. “A Spectral-Flatness Measure for Studying the Autocorrelation Method of Linear Prediction of Speech Analysis,” IEEE Trans. Acoust., Speech and Sig. Process., 22, pp. 207-217, 1974.

[11] James, G., Witten, D., Hastie, T. and Tibshirani R., editors, “An Introduction to Statistical Learning: with Applications in R,” number 103 in Springer Texts in Statistics, Springer, New York, 2013.

Claims (51)

1. A device for processing an input audio signal containing one or more audio channels to obtain an output audio signal, wherein the device comprises: - a reverberation gain determination module (110) configured to determine reverberation gain information depending on an input audio signal, and - a signal processor (120) configured to obtain an output audio signal depending on the reverberation gain information by summing the artificial reverberation with the input audio signal or with a pre-processed audio signal that depends on the input audio signal, wherein the reverberation gain determination module (110) is configured to determine reverberation gain information depending on the estimated perceived reverberation intensity in the input audio signal, wherein the reverberation gain determination module (110) is configured to determine reverberation gain information by using a model that returns an estimate of the perceived intensity of reverberation in the input audio signal upon receiving information about one or more features of the input audio signal. 2. The device according to item 1, wherein the model used by the reverberation gain determination module (110) is a linear regression model using one or more feature values for one or more features of the input audio signal as input to the linear regression model. 3. The device according to item 1 or 2, wherein, in order to obtain the reverberation sending gain, the reverberation gain determining module (110) is configured to determine the reverberation sending gain as reverberation gain information by converting the estimated perceived intensity of the reverberation in the input audio signal according to a conversion function. 4. The device according to one of paragraphs 1-3, wherein one or more features of the input audio signal depend on an inter-channel correlation of at least one of the one or more frequency subbands of two audio channels for one or more audio channels of the input audio signal. 5. The device according to one of paragraphs 1-4, wherein one or more features of the input audio signal depend on a spectral smoothness index of at least one of the one or more frequency subbands of one or more audio channels of the input audio signal. 6. The device according to one of paragraphs 1-5, wherein the reverberation gain determination module is configured to determine an estimate of the perceived reverberation intensity in the input audio signal by using a model, wherein the reverberation gain determination module is configured to determine one or more scaling factors depending on one or more features of the input audio signal, and wherein the reverberation gain determination module is configured to determine reverberation gain information depending on the estimated perceived reverberation intensity and depending on one or more scaling factors. 7. The device according to item 6, in which one or more scaling factors depend on at least one of interchannel correlation of at least one of one or more frequency subbands of two audio channels for one or more audio channels of the input audio signal, the presence of transient signal components in at least one of the one or more audio channels of the input audio signal. 8. The device according to one of paragraphs 6 or 7, wherein one or more scaling factors depend on the index of spectral transient portions of at least one of the one or more audio channels of the input audio signal, in this case, the index of spectral transition parts is set depending on the following: in this case: where m indicates the time index, and k indicates the frequency index, where b _k,i indicates one of one or more frequency resolution elements in the k frequency subband, wherein Y _d (m, b _k,i ) indicates a coefficient with an absolute value with a time index m for one or more frequency elements b _k,i of the resolution in a frequency subband k of said at least one of the one or more audio channels or a combination of said at least one of the one or more audio channels, and where f _s () denotes recursive averaging over time and implements smoothing of the subband signal, where g is a contraction function. 9. A device according to one of the preceding paragraphs, in which the signal processor (120) is configured to generate a pre-processed audio signal by reverberating the input audio signal to attenuate the original reverberant signal components of the input audio signal, and wherein the signal processor (120) is configured to obtain an output audio signal depending on the reverberation gain information by summing the artificial reverberation with the pre-processed audio signal. 10. A device according to one of the preceding paragraphs, in which the signal processor (120) is configured to generate a pre-processed audio signal by performing temporal smoothing of the input audio signal, and wherein the signal processor (120) is configured to obtain an output audio signal depending on the reverberation gain information by summing the artificial reverberation with the pre-processed audio signal. 11. The device according to item 10, in which the signal processor (120) is configured to adjust the amount of time smoothing depending on changes in the input audio signal. 12. The device according to item 11, in which the signal processor (120) is configured to adjust the amount of time smoothing depending on changes in the volume level of the input audio signal, changes in the dispersion of one or more features of the input audio signal. 13. A device according to one of the preceding paragraphs, wherein the device comprises a circular buffer for receiving an input audio signal or a pre-processed audio signal, and wherein the signal processor (120) is configured to process the input audio signal or the pre-processed audio signal in the ring buffer to obtain the output audio signal. 14. A method for processing an input audio signal to obtain an output audio signal, wherein the method comprises the steps of: - determine the reverberation gain information depending on the input audio signal, and - obtain an output audio signal depending on the reverberation gain information by summing the artificial reverberation with the input audio signal or with a pre-processed audio signal that depends on the input audio signal, wherein the determination of the reverberation gain information is performed depending on the assessment of the perceived reverberation intensity in the input audio signal, wherein determining the reverberation gain information is performed by using a model that returns an estimate of the perceived intensity of reverberation in the input audio signal upon receiving information about one or more features of the input audio signal. 15. A non-transitory machine-readable medium on which a computer program is stored for implementing the method according to item 14 when executed on a computer or in a signal processor.