EP3381204B1 - Method and device for estimating sound reverberation - Google Patents
Method and device for estimating sound reverberation Download PDFInfo
- Publication number
- EP3381204B1 EP3381204B1 EP16819332.4A EP16819332A EP3381204B1 EP 3381204 B1 EP3381204 B1 EP 3381204B1 EP 16819332 A EP16819332 A EP 16819332A EP 3381204 B1 EP3381204 B1 EP 3381204B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- characteristic
- acoustic
- reverberation
- determined
- decay rate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R29/00—Monitoring arrangements; Testing arrangements
- H04R29/001—Monitoring arrangements; Testing arrangements for loudspeakers
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
- G10L25/21—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being power information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R29/00—Monitoring arrangements; Testing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
Definitions
- the present invention relates to methods and devices for estimating acoustic reverberation.
- Estimating the acoustic reverberation of a medium is essential for capturing acoustic signals such as speech in a reverberant medium such as, for example, a room in a building.
- the microphone When a sound is emitted and picked up by a microphone in a reverberant environment, the microphone picks up not only the received signal directly, but also reverberant signals in the medium.
- This reverberation is reflected by the impulse response of the medium, from which various known parameters, including the reverberation time, flow.
- the impulse response can be measured directly by emitting an acoustic pulse in the medium, but this method is cumbersome and difficult to envisage for performing repeated measurements while one or more speakers are speaking in the room.
- the reverberation time can be estimated blindly, for example while one or more speakers are speaking.
- the most common parameter used to represent the reverberation time is the 60 dB RT 60 reverberation time.
- the document US2014169575 describes a method for blind estimation of the reverberation time in a room.
- the reverberation time is not representative of the distance between the transmitter and the microphone, which however has a significant impact on the reverb level.
- the aforementioned known methods of the type do not therefore make it possible to process the acoustic signals sensed satisfactorily.
- the present invention therefore aims to provide a method for estimating acoustic reverberation, which avoids this disadvantage.
- the aim of the invention is to estimate the acoustic reverberation of a medium 7, for example a room in a building as schematized on the figure 1 , so as to process the acoustic signals picked up by an electronic device 1 provided with a microphone 2.
- the electronic device 1 can be for example a telephone in the example shown, or a computer or other.
- the electronic device 1 may comprise for example an electronic central unit 8 such as a processor or the like, connected to the microphone 2 and various other elements, including for example a speaker 9, a keyboard 10, a screen 11.
- L electronic central unit 8 can communicate with an external network 12, for example a telephone network.
- These parameters can be used to suppress the effects of echoes or more generally to optimize the sound signals picked up by the microphone 2.
- the parameters in question are estimated in a repetitive way, so that the device 1 is adapted for example to changes in speakers 3, to movements of speakers 3, to movements of the device 1 or other objects in the medium 7.
- RT 60 is the time at the time index n required for EDC (n) to decrease by 60 dB.
- RT 60 reverberation time is the most commonly used, we could estimate another reverberation time characteristic of the medium 7.
- ⁇ The two most commonly used values of ⁇ are 50 ms and 80 ms, in particular 50 ms (indices C 50 and D 50 ), but other durations are possible and more generally other indices reflecting the ratio of the direct sound to the sound. reverberation could be estimated in the method according to the invention, implemented for example by the aforementioned electronic central unit 8.
- the microphone 2 captures "blind" (that is to say, without prior knowledge of the signals transmitted), an acoustic signal broadcast in the medium 7, for example when the speaker 3 speaks. This signal is sampled and stored in processor 8 or an ancillary memory (not shown).
- an acoustic energy decay rate distribution is determined from the acoustic signal measured in step (a).
- ⁇ s and ⁇ h are respectively the rate of energy decay of the emitted anechoic signal and medium 7 (the signal picked up is a convolution of the emitted anechoic signal (speech) with the impulse response of the medium between the speaker 3 and the microphone 2, n being the temporal index defined above).
- the energy decay rate of the reverberated signal ⁇ x can be approximated by: ⁇ x ⁇ max ⁇ h ⁇ s which justifies formula (5) above.
- the calculation of ⁇ (m) can typically be done on a number M of frames of at least 2000, corresponding to at least 1 min of signal according to the selected analysis parameters.
- the frames can have an individual duration of 10 to 100 ms, in particular of the order of 32 ms.
- the frames may overlap each other, for example with a recovery rate of the order of 50% between successive frames.
- the characteristic function can be calculated for angular frequencies f ranging for example from 0 to 0.4, in increments of 0.001.
- p can be for example between 256 and 2048.
- the characteristic function is a complex number, it can be represented by a vector of X belonging to R p constituting the random input vector x of the estimator used.
- the random output vector y of the estimator, belonging to R 2 has as component the two estimated parameters, for example (RT 60 , C 50 ) or (RT 60 , D 50 ).
- the estimator used may advantageously be a kernel function estimator, for example a Nadaraya-Watson estimator.
- a kernel function estimator for example a Nadaraya-Watson estimator.
- Such an estimator has the advantage of simultaneously determining the characteristic reverberation time and the characteristic reverberation level.
- the estimator in question can be determined in advance in an initial calibration phase, where at least one initial step of determination of reference signals (a ') and at least one initial observation step (b') is implemented.
- a plurality of reference acoustic signals and reference characteristic reverberation times and corresponding reference characteristic reverb levels are determined.
- the acoustic energy decay rate distribution and the reference characteristic function for each reference acoustic signal are determined identically or similarly to the observation step (b). ) above.
- the acoustic reference signals are generally vocal signals of the number of N and correspond to N cases of different figures (different speakers, different positions, different media 7). N can be several hundred or even thousands.
- the aforementioned synthetic acoustic signals can be calculated by convolving pre-recorded impulse responses with pre-recorded speech anechoic signals from different speakers.
- the prerecorded impulse responses may for example come from impulse response databases, for example from open access databases such as Aachen Impulse Response databases. (http://www.openairlib.net/auralizationdb), MARDY ( Wen et al., Evaluation of speech dereverberation algorithms using the Mardy database, Sept. IWAENC 2006, By is), QueenMary ( R. Stewart and M. Sandler, Database of omnidirectional and b-format room impulse responses, In Acoustics Speech and Signal Processing (ICASSP).
- IASSP Acoustics Speech and Signal Processing
- Anechoic speech signals may be signals recorded from various speakers, for example of different ages and sexes, with for example recording times for example of a few minutes, for example of the order of 5 minutes.
- the energy decay rate distributions can for example be calculated over frames of 10 to 100 ms, in particular of the order of 32 ms.
- the frames may overlap each other, for example with a recovery rate of the order of 50% between successive frames.
- the characteristic functions can be calculated for angular frequencies ranging for example from 0 to 0.4, in increments of 0.001.
- the kernel function K ⁇ (x, x i ) is a function of x and x i as defined in particular by Scholkopf et al. ( B. Scholkopf and AJ Smola, Learning with Kernels, MIT Press, Cambridge, MA, 2001 ).
- the tests carried out show that the method of the invention is more accurate than the methods of the prior art for determining the reverberation time, and it also makes it possible to determine the reverberation level at the same time as the reverberation time, which is a big gain.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Otolaryngology (AREA)
- Human Computer Interaction (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- Multimedia (AREA)
- Measurement Of Mechanical Vibrations Or Ultrasonic Waves (AREA)
- Reverberation, Karaoke And Other Acoustics (AREA)
Description
La présente invention est relative aux procédés et dispositifs pour estimer la réverbération acoustique.The present invention relates to methods and devices for estimating acoustic reverberation.
L'estimation de la réverbération acoustique d'un milieu est essentielle pour capter des signaux acoustiques tels que la parole dans un milieu réverbérant tel que par exemple une pièce dans un bâtiment.Estimating the acoustic reverberation of a medium is essential for capturing acoustic signals such as speech in a reverberant medium such as, for example, a room in a building.
Lorsqu'un son est émis puis capté par un microphone dans un milieu réverbérant, le microphone capte non seulement le signal reçu directement, mais également des signaux réverbérés dans le milieu.When a sound is emitted and picked up by a microphone in a reverberant environment, the microphone picks up not only the received signal directly, but also reverberant signals in the medium.
Cette réverbération est reflétée par la réponse impulsionnelle du milieu, dont découlent divers paramètres connus, notamment le temps de réverbération. La réponse impulsionnelle est mesurable directement en émettant une impulsion acoustique dans le milieu, mais cette méthode est lourde et difficilement envisageable pour effectuer des mesures répétées pendant qu'un ou des locuteurs parlent dans la pièce.This reverberation is reflected by the impulse response of the medium, from which various known parameters, including the reverberation time, flow. The impulse response can be measured directly by emitting an acoustic pulse in the medium, but this method is cumbersome and difficult to envisage for performing repeated measurements while one or more speakers are speaking in the room.
Le temps de réverbération peut être estimé à l'aveugle, par exemple pendant qu'un ou des locuteurs parlent. Le paramètre le plus couramment utilisé pour représenter le temps de réverbération est le temps de réverbération à 60 dB RT60.The reverberation time can be estimated blindly, for example while one or more speakers are speaking. The most common parameter used to represent the reverberation time is the 60 dB RT 60 reverberation time.
A titre d'exemple, le document
Toutefois, le temps de réverbération n'est pas représentatif de la distance entre l'émetteur et le microphone, qui pourtant a un impact important sur le niveau de réverbération. Les procédés connus du type susmentionnés ne permettent donc pas de traiter les signaux acoustiques captés de façon satisfaisante.However, the reverberation time is not representative of the distance between the transmitter and the microphone, which however has a significant impact on the reverb level. The aforementioned known methods of the type do not therefore make it possible to process the acoustic signals sensed satisfactorily.
Les documents suivants font partie de l'état de l'art lié à la présente invention:
-
.JIMI Y C WEN ET AL: "Blind estimation of reverberation time based on the distribution of signal decay rates", IEEE ICASSP 2008, p. 329-332 -
.FALK T H ET AL: "Temporal Dynamics for Blind Measurement of Room Acoustical Parameters",IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, vol. 59, no. 4, 2010, p. 978-989 -
.EATON JAMES ET AL: "Noise-robust reverberation time estimation using spectral decay distributions with reduced computational cost",2013 IEEE ICASSP, 2013, p. 161-165
-
.JIMI YC WEN AND AL: "Blind estimation of reverberation time based on the distribution of signal decay rates", IEEE ICASSP 2008, p. 329-332 -
.FALK TH ET AL: "Temporal Dynamics for Measurement of Room Acoustical Parameters", IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, vol. 59, no. 4, 2010, p. 978-989 -
.EATON JAMES ET AL: "Noise-robust reverberation time estimation using spectral decay distributions with reduced computational cost," 2013 IEEE ICASSP, 2013, p. 161-165
La présente invention a donc pour objet de proposer un procédé pour estimer la réverbération acoustique, qui permette d'éviter cet inconvénient.The present invention therefore aims to provide a method for estimating acoustic reverberation, which avoids this disadvantage.
A cet effet, l'invention propose un procédé pour estimer la réverbération acoustique dans un milieu, comprenant les étapes suivantes :
- (a) une étape de mesure dans laquelle on capte au moins un signal acoustique dans le milieu,
- (b) une étape d'observation au cours de laquelle on détermine une distribution de taux de décroissance d'énergie acoustique à partir du signal acoustique capté à l'étape (a) et on détermine la fonction caractéristique de la distribution de taux de décroissance d'énergie acoustique,
- (c) une étape d'estimation au cours de laquelle on estime un temps de réverbération caractéristique et un niveau de réverbération caractéristique du son dans le milieu à partir de données représentatives de la distribution de taux de décroissance d'énergie acoustique déterminée à l'étape (b), la régression étant faite en référence à :
- des fonctions caractéristiques de référence représentatives respectivement de plusieurs distributions de taux de décroissance d'énergie acoustique,
- des temps de réverbération caractéristiques de référence correspondant auxdites fonctions caractéristiques de référence,
- et des niveaux de réverbération caractéristiques de référence correspondant auxdites fonctions caractéristiques de référence.
- (a) a measurement step in which at least one acoustic signal is sensed in the medium,
- (b) an observation step during which an acoustic energy decay rate distribution is determined from the acoustic signal picked up in step (a) and the characteristic function of the decay rate distribution is determined acoustic energy,
- (c) an estimation step in which a characteristic reverberation time and a characteristic reverberation level of the sound in the medium are estimated from data representative of the acoustic energy decay rate distribution determined at step (b), the regression being made with reference to:
- reference characteristic functions representative respectively of several distributions of decay rate of acoustic energy,
- reference characteristic reverberation times corresponding to said characteristic functions reference,
- and reference characteristic reverberation levels corresponding to said reference characteristic functions.
Grâce à ces dispositions, et notamment grâce au fait que l'on applique la méthode d'estimation à la distribution de taux de décroissance d'énergie acoustique, on peut déterminer à la fois un temps de réverbération caractéristique et un niveau de réverbération caractéristique du son dans le milieu, de façon fiable. Ces deux paramètres permettent de traiter de façon satisfaisante les signaux sonores captés.Thanks to these provisions, and notably because the method of estimation is applied to the distribution of decay rates of acoustic energy, it is possible to determine both a characteristic reverberation time and a reverberation level characteristic of the sound in the middle, reliably. These two parameters make it possible to satisfactorily process the sound signals picked up.
Dans divers modes de réalisation du procédé selon l'invention, on peut éventuellement avoir recours en outre à l'une et/ou à l'autre des dispositions suivantes :
- au cours de l'étape d'estimation (c), on utilise un estimateur à fonction noyau et on détermine simultanément le temps de réverbération caractéristique et le niveau de réverbération caractéristique ;
- au cours de l'étape d'estimation (c), on utilise un estimateur de Nadaraya-Watson ;
- au cours de l'étape d'estimation (c), le niveau de réverbération caractéristique du son dans le milieu (7) est choisi parmi l'indice de clarté Cτ et l'indice de définition Dτ ;
- au cours de l'étape d'observation (b), on détermine les taux de décroissance d'énergie en calculant l'énergie Em du signal acoustique sur des trames successives m de signal, puis en calculant un rapport logarithmique entre l'énergie de deux trames successives :
- le procédé comporte en outre une phase préliminaire de calibrage comprenant les étapes suivantes :
- (a') au moins une étape initiale de détermination de signaux de référence dans laquelle on détermine une pluralité de signaux acoustiques de référence correspondant auxdits temps de réverbération caractéristiques de référence et auxdits niveaux de réverbération caractéristiques de référence,
- (b') au moins une étape initiale d'observation au cours de laquelle, pour chaque signal acoustique de référence, on détermine une distribution de taux de décroissance d'énergie acoustique et la fonction caractéristique de référence ;
- au cours de ladite étape de détermination de signaux de référence, on détermine par calcul au moins une partie des signaux acoustiques de référence et les temps de réverbération caractéristiques et niveaux de réverbération caractéristiques de référence correspondant auxdits signaux acoustiques de référence, à partir d'un ensemble prédéterminé de réponses impulsionnelles ;
- au cours de ladite étape de détermination de signaux de référence, on détermine par mesure au moins une partie des signaux acoustiques de référence, les temps de réverbération caractéristiques et les niveaux de réverbération caractéristiques de référence correspondant auxdits signaux acoustiques de référence.
- le procédé comporte en outre une phase préliminaire de calibrage comprenant les étapes suivantes :
- during the estimation step (c), a kernel function estimator is used and the characteristic reverberation time and the characteristic reverberation level are simultaneously determined;
- during the estimation step (c), a Nadaraya-Watson estimator is used;
- during the estimation step (c), the characteristic reverberation level of the sound in the medium (7) is chosen from the clarity index C τ and the definition index D τ ;
- during the observation step (b), the energy decay rates are determined by calculating the energy Em of the acoustic signal on successive signal frames m, and then calculating a logarithmic ratio between the energy of the signal two successive frames:
- the method further comprises a phase preliminary calibration procedure comprising the following steps:
- (a ') at least one initial step of determining reference signals in which a plurality of reference acoustic signals corresponding to said reference characteristic reverberation times and said reference characteristic reverb levels are determined,
- (b ') at least one initial observation step during which, for each reference acoustic signal, an acoustic energy decay rate distribution and the reference characteristic function are determined;
- during said step of determining reference signals, at least a portion of the reference acoustic signals and the characteristic reverberation times and reference characteristic reverberation levels corresponding to said reference acoustic signals are determined by calculation from a predetermined set of impulse responses;
- during said step of determining reference signals, at least a portion of the reference acoustic signals, the characteristic reverberation times and the reference characteristic reverberation levels corresponding to said reference acoustic signals are determined by measurement.
- the method further comprises a phase preliminary calibration procedure comprising the following steps:
Par ailleurs, l'invention a également pour objet un dispositif pour estimer la réverbération acoustique dans un milieu, comprenant :
- (a) des moyens de mesure pour capter au moins un signal acoustique émis dans le milieu,
- (b) des moyens pour déterminer une distribution de taux de décroissance d'énergie acoustique à partir du signal acoustique capté par les moyens de mesure, et pour déterminer la fonction caractéristique de la distribution de taux de décroissance d'énergie acoustique,
- (c) des moyens pour estimer un temps de réverbération caractéristique et un niveau de réverbération caractéristique du son dans le milieu à partir de données représentatives de la distribution de taux de décroissance d'énergie acoustique, la régression étant faite en référence à :
- des fonctions caractéristiques de référence représentatives respectivement de plusieurs distributions de taux de décroissance d'énergie acoustique,
- des temps de réverbération caractéristiques de référence correspondant auxdites fonctions caractéristiques de référence,
- et des niveaux de réverbération caractéristiques de référence correspondant auxdites fonctions caractéristiques de référence.
- (a) measuring means for sensing at least one acoustic signal emitted in the medium,
- (b) means for determining an acoustic energy decay rate distribution from the signal acoustics captured by the measuring means, and to determine the characteristic function of the acoustic energy decay rate distribution,
- (c) means for estimating a characteristic reverberation time and a reverberation level characteristic of sound in the medium from data representative of the acoustic energy decay rate distribution, the regression being made with reference to:
- reference characteristic functions representative respectively of several distributions of decay rates of acoustic energy,
- reference characteristic reverberation times corresponding to said reference characteristic functions,
- and reference characteristic reverberation levels corresponding to said reference characteristic functions.
D'autres caractéristiques et avantages de l'invention apparaîtront au cours de la description suivante d'un de ses modes de réalisation, donné à titre d'exemple non limitatif, en regard des dessins joints.Other features and advantages of the invention will become apparent from the following description of one of its embodiments, given by way of non-limiting example, with reference to the accompanying drawings.
Sur les dessins :
- la
figure 1 est une vue schématique illustrant la réverbération du son dans une pièce lorsqu'un sujet parle de façon que ses paroles soient captées par un dispositif selon une forme de réalisation de l'invention, - la
figure 2 est un schéma de principe du dispositif de lafigure 1 .
- the
figure 1 is a schematic view illustrating the reverberation of sound in a room when a subject speaks so that his words are picked up by a device according to one embodiment of the invention, - the
figure 2 is a schematic diagram of the device of thefigure 1 .
Sur les différentes figures, les mêmes références désignent des éléments identiques ou similaires.In the different figures, the same references designate identical or similar elements.
L'invention a pour but d'estimer la réverbération acoustique d'un milieu 7, par exemple une pièce dans un bâtiment telle que schématisée sur la
Lorsqu'un son est émis dans le milieu 7, par exemple par une personne 3, ce son se propage jusqu'au microphone 2 selon divers trajets 4, soit directs, soit après réflexion sur une ou plusieurs parois 5, 6 du milieu 7.When a sound is emitted in the
Comme représenté sur la
L'invention permet au dispositif électronique 1 de mesurer à l'aveugle, deux paramètres caractéristiques de la réverbération du milieu 7 :
- un temps de réverbération caractéristique, par exemple le temps de réverbération à 60 dB RT60,
- et un niveau de réverbération caractéristique (par exemple indice de clarté ou de définition, ou indice de signal direct sur signal réverbéré).
- a characteristic reverberation time, for example the reverberation time at 60 dB RT 60 ,
- and a characteristic reverberation level (eg clarity or definition index, or direct signal index on reverberated signal).
Ces paramètres peuvent être utilisés pour supprimer les effets d'échos ou plus généralement pour optimiser les signaux sonores captés par le microphone 2. Les paramètres en question sont estimés de façon répétitive, de façon que le dispositif 1 s'adapte par exemple à des changements de locuteurs 3, à des déplacements de locuteurs 3, à des déplacements du dispositif 1 ou d'autres objets dans le milieu 7.These parameters can be used to suppress the effects of echoes or more generally to optimize the sound signals picked up by the microphone 2. The parameters in question are estimated in a repetitive way, so that the device 1 is adapted for example to changes in speakers 3, to movements of speakers 3, to movements of the device 1 or other objects in the
Le temps de réverbération à 60dB RT60 peut être défini par la méthode d'intégration inverse de
- h est la réponse impulsionnelle du milieu de longueur Nh ,
- n est un indice temporel, par exemple un nombre d'échantillons obtenus par un échantillonnage de pas temporel constant, n étant compris entre 1 et Nh.
- h is the impulse response of the medium of length N h ,
- n is a time index, for example a number of samples obtained by sampling a constant time step, n being between 1 and N h .
RT60 est le temps à l'indice temporel n requis pour que EDC(n) diminue de 60 dB.RT 60 is the time at the time index n required for EDC (n) to decrease by 60 dB.
Bien que le temps de réverbération RT60 soit le plus couramment utilisé, on pourrait estimer un autre temps de réverbération caractéristique du milieu 7.Although the RT 60 reverberation time is the most commonly used, we could estimate another reverberation time characteristic of the
Le niveau de réverbération est le plus couramment représenté par l'indice de clarté :
- N τ est le nombre d'échantillons à pas temporels constant correspondant au temps τ, généralement compris entre 0.1 ms et 1 s,
- n est un indice temporel compris entre 1 et N τ , représentatif d'un nombre d'échantillons de pas temporel constant,
- h(n) est la réponse impulsionnelle du
milieu 7.
- N τ is the number of samples with a constant time step corresponding to the time τ , generally between 0.1 ms and 1 s,
- n is a time index between 1 and N τ , representative of a number of samples of constant time step,
- h ( n ) is the impulse response of
medium 7.
Ces indices ont été décrits notamment par
Les deux valeurs les plus couramment utilisées de τ sont 50 ms et 80 ms, notamment 50 ms (indices C50 et D50), mais d'autres durées sont possibles et plus généralement d'autres indices reflétant le rapport du son direct au son réverbéré pourraient être estimés dans le procédé selon l'invention, mis en œuvre par exemple par l'unité centrale électronique 8 susmentionnée.The two most commonly used values of τ are 50 ms and 80 ms, in particular 50 ms (indices C 50 and D 50 ), but other durations are possible and more generally other indices reflecting the ratio of the direct sound to the sound. reverberation could be estimated in the method according to the invention, implemented for example by the aforementioned electronic central unit 8.
Ce procédé comporte les étapes suivantes :
- (a) une étape de mesure de signal acoustique,
- (b) une étape d'observation au cours de laquelle on détermine une distribution de taux de décroissance d'énergie acoustique à partir des signaux acoustiques mesurés à l'étape (a),
- (c) une étape d'estimation au cours de laquelle on estime, par régression à partir de la distribution de taux de décroissance d'énergie acoustique déterminée à l'étape (b), un temps de réverbération caractéristique et un niveau de réverbération caractéristique du son dans le
milieu 7.
- (a) an acoustic signal measurement step,
- (b) an observation step during which an acoustic energy decay rate distribution is determined from the acoustic signals measured in step (a),
- (c) an estimation step in which it is estimated, by regression from the sound energy decay rate distribution determined in step (b), a characteristic reverberation time and a characteristic reverberation level sound in the middle 7.
Au cours de cette étape, le microphone 2 capte « à l'aveugle » (c'est-à-dire sans connaissance préalable des signaux émis), un signal acoustique diffusé dans le milieu 7, par exemple lorsque le locuteur 3 parle. Ce signal est échantillonné et stocké dans le processeur 8 ou une mémoire annexe (non représentée).During this step, the microphone 2 captures "blind" (that is to say, without prior knowledge of the signals transmitted), an acoustic signal broadcast in the
Au cours de cette étape, on détermine une distribution de taux de décroissance d'énergie acoustique à partir du signal acoustique mesuré à l'étape (a).During this step, an acoustic energy decay rate distribution is determined from the acoustic signal measured in step (a).
Pour cela, on détermine d'abord l'enveloppe d'énergie du signal réverbéré dx(n) telle que décrit notamment par Wen et al. (
En faisant un calcul sur des trames de Nω , échantillons de signal séparées par des sauts de R échantillons de signal, on peut calculer une énergie totale de la trame m par la formule :
En effet, l'enveloppe d'énergie dx(n) peut être exprimée par la formule :
Comme la somme est dominée par le terme exponentiel correspondant à la plus grande valeur de λ, le taux de décroissance d'énergie du signal réverbéré λ x peut être approximé par :
Le calcul de ρ(m) peut typiquement être fait sur un nombre M de trames d'au moins 2000, correspondant à au moins 1 mn de signal selon les paramètres d'analyse choisis. Les trames peuvent avoir une durée individuelle de 10 à 100 ms, notamment de l'ordre de 32 ms. Les trames peuvent se recouvrir mutuellement, par exemple avec un taux de recouvrement de l'ordre de 50 % entre trames successives.The calculation of ρ (m) can typically be done on a number M of frames of at least 2000, corresponding to at least 1 min of signal according to the selected analysis parameters. The frames can have an individual duration of 10 to 100 ms, in particular of the order of 32 ms. The frames may overlap each other, for example with a recovery rate of the order of 50% between successive frames.
On obtient ainsi différentes valeurs du taux de décroissance d'énergie ρ(m), qui présentent une certaine distribution statistique (nombre de réalisations ou probabilité de réalisation en fonction du taux de décroissance d'énergie ρ(m), comme explicité par exemple dans l'article de Wen et al. ci-dessus).Different values of the energy decay rate ρ (m) are obtained, which have a certain statistical distribution (number of realizations or probability of realization as a function of the rate of energy decay ρ (m), as explained, for example, in the article by Wen et al., above).
On détermine ensuite la fonction caractéristique de la distribution de taux de décroissance d'énergie, par la formule suivante (voir
La fonction caractéristique peut être calculée pour des fréquences angulaires f allant par exemple de 0 à 0,4, par incrément de 0,001.The characteristic function can be calculated for angular frequencies f ranging for example from 0 to 0.4, in increments of 0.001.
On part de la fonction caractéristique Φρ(m)(f), calculée pour p/2 fréquences f (p étant un entier naturel pair), la gamme de fréquences f et leur échantillonnage étant prévus pour que |Φρ(m) (f)| soit compris préférentiellement entre 0.1 et 1.We start from the characteristic function Φ ρ (m) (f), calculated for p / 2 frequencies f (p being an even natural integer), the frequency range f and their sampling being provided so that | Φ ρ (m) ( f) | is preferably comprised between 0.1 and 1.
Typiquement, p peut être compris par exemple entre 256 et 2048.Typically, p can be for example between 256 and 2048.
La fonction caractéristique étant un nombre complexe, elle peut être représentée par un vecteur de X appartenant à Rp, constituant le vecteur d'entrée aléatoire x de l'estimateur utilisé. Le vecteur de sortie aléatoire y de l'estimateur, appartenant à R2, a pour composante les deux paramètres estimés, par exemple (RT60, C50) ou (RT60, D50).Since the characteristic function is a complex number, it can be represented by a vector of X belonging to R p constituting the random input vector x of the estimator used. The random output vector y of the estimator, belonging to R 2 , has as component the two estimated parameters, for example (RT 60 , C 50 ) or (RT 60 , D 50 ).
L'estimateur utilisé peut être avantageusement un estimateur à fonction noyau, par exemple un estimateur de Nadaraya-Watson. Un tel estimateur présente l'avantage de déterminer simultanément le temps de réverbération caractéristique et le niveau de réverbération caractéristique.The estimator used may advantageously be a kernel function estimator, for example a Nadaraya-Watson estimator. Such an estimator has the advantage of simultaneously determining the characteristic reverberation time and the characteristic reverberation level.
L'estimateur en question peut être déterminé préalablement dans une phase initiale de calibrage, où on met en œuvre au moins une étape initiale de détermination de signaux de référence (a') et au moins une étape initiale d'observation (b').The estimator in question can be determined in advance in an initial calibration phase, where at least one initial step of determination of reference signals (a ') and at least one initial observation step (b') is implemented.
Au cours de l'étape initiale de détermination de signaux de référence, on détermine une pluralité de signaux acoustiques de référence et des temps de réverbération caractéristiques de référence et niveaux de réverbération caractéristiques de référence correspondants.During the initial step of determining reference signals, a plurality of reference acoustic signals and reference characteristic reverberation times and corresponding reference characteristic reverb levels are determined.
Au cours de l'étape initiale d'observation, on détermine la distribution de taux de décroissance d'énergie acoustique et la fonction caractéristique de référence pour chaque signal acoustique de référence, de façon identique ou similaire à l'étape d'observation (b) susmentionnée.During the initial observation step, the acoustic energy decay rate distribution and the reference characteristic function for each reference acoustic signal are determined identically or similarly to the observation step (b). ) above.
Les signaux acoustiques de référence sont des signaux généralement vocaux au nombre de N et correspondent à N cas de figures différents (locuteurs différents, positions différentes, milieux 7 différents). N peut être de plusieurs centaines, voire plusieurs milliers.The acoustic reference signals are generally vocal signals of the number of N and correspond to N cases of different figures (different speakers, different positions, different media 7). N can be several hundred or even thousands.
L'étape initiale de détermination de signaux de référence peut être effectuée :
- avec des mesures réelles nouvelles effectuées par exemple avec un dispositif électronique 1 d'un modèle déterminé (dans ce cas, le temps de réverbération caractéristique et le niveau de réverbération caractéristique peuvent être également mesurés) ;
- et / ou avec des signaux acoustiques synthétiques.
- with real new measurements made for example with an electronic device 1 of a given model (in this case, the characteristic reverberation time and the characteristic reverberation level can also be measured);
- and / or with synthetic acoustic signals.
Dans le cas des mesures réelles, celles-ci ne seront généralement pas faites dans le milieu 7 particulier ou le dispositif électronique 1 sera utilisé, bien que ce cas de figure puisse être envisagé.In the case of real measurements, these will generally not be done in the particular medium where the electronic device 1 will be used, although this case may be considered.
Les signaux acoustiques synthétiques susmentionnés peuvent être calculés en convoluant des réponses impulsionnelles préenregistrées avec des signaux anéchoïques de parole également préenregistrés provenant de locuteurs différents. Les réponses impulsionnelles préenregistrées peuvent par exemple provenir de bases de données de réponses impulsionnelles, par exemple provenant de bases de données en accès libre telles que les bases de données Aachen Impulse Response (http://www.openairlib.net/auralizationdb), MARDY (
Les distributions de taux de décroissance d'énergie peuvent par exemple être calculées sur des trames de 10 à 100 ms, notamment de l'ordre de 32 ms. Les trames peuvent se recouvrir mutuellement, par exemple avec un taux de recouvrement de l'ordre de 50 % entre trames successives. Les fonctions caractéristiques peuvent être calculées pour des fréquences angulaires allant par exemple de 0 à 0,4, par incrément de 0,001.The energy decay rate distributions can for example be calculated over frames of 10 to 100 ms, in particular of the order of 32 ms. The frames may overlap each other, for example with a recovery rate of the order of 50% between successive frames. The characteristic functions can be calculated for angular frequencies ranging for example from 0 to 0.4, in increments of 0.001.
On obtient ainsi N réalisations des vecteurs x et y susmentionnés, et l'estimateur de Nadaraya-Watson peut alors être déterminé par la formule :
- xi, yi, i = 1 à N, sont les N réalisations des vecteurs x, y utilisées pour l'étape de calibration,
- Kλ(x, xi) est une fonction noyau (kernel) de fenêtre λ (λ est une constante également appelée paramètre de lissage),
- x est le vecteur d'entrée inconnu (mesure effectuée à l'étape de mesure (a) en vue d'estimer le vecteur y par la formule y = f̂(x)).
- x i , y i , i = 1 to N, are the N realizations of the x, y vectors used for the calibration step,
- K λ (x, x i ) is a kernel function of window λ (λ is a constant also called smoothing parameter),
- x is the unknown input vector (measured at the measurement step (a) to estimate the vector y by the formula y = f ( x )).
La fonction noyau Kλ(x, xi) est une fonction de x et xi telle que définie notamment par Scholkopf et al. (
On peut notamment utiliser un noyau gaussien, par exemple avec une fenêtre de λ = 5.10-4 (exemple non limitatif) :
Les tests effectués montrent que le procédé de l'invention est plus précis que les procédés de l'art antérieur pour la détermination du temps de réverbération, et il permet en outre de déterminer le niveau de réverbération en même temps que le temps de réverbération, ce qui est un gain important.The tests carried out show that the method of the invention is more accurate than the methods of the prior art for determining the reverberation time, and it also makes it possible to determine the reverberation level at the same time as the reverberation time, which is a big gain.
Claims (9)
- A method for estimating the acoustic reverberations in an environment (7) comprising the following steps:(a) a measurement step in which at least one acoustic signal emitted in the environment (7) is captured;(b) an observation step during which an acoustic energy decay rate distribution is determined from the acoustic signal measured in step (a) and the characteristic function of the acoustic energy decay rate distribution is determined;(c) an estimation step during which a characteristic reverberation time and a characteristic reverberation level of the sound in the environment (7) thereof are estimated by regression from said characteristic function determined in step (b), where the regression is done with reference to:- reference characteristic functions representative respectively of several acoustic energy decay rate distributions;- reference characteristic reverberation times corresponding to said reference characteristic functions; and- reference characteristic reverberation levels corresponding to said reference characteristic functions.
- The method according to claim 1 wherein during the estimation step (c), a kernel function estimator is used and the characteristic reverberation time and the characteristic reverberation level are determined simultaneously.
- The method according to claim 2 wherein during the estimation step (c), a Nadaraya-Watson estimator is used.
- The method according to any one of the preceding claims wherein during the estimation step (c), the characteristic reverberation level of the sound in the environment (7) is chosen among the clarity index Cτ and the definition index Dτ.
- The method according to any one of the preceding claims wherein during the observation step (b), the energy decay rates are determined by calculating the energy Em of the acoustic signal on successive signal frames m, and then calculating a logarithmic ratio between the energy of two successive frames:
- The method according to any one of the preceding claims further comprises a preliminary calibration phase comprising the following steps:(a') at least one initial reference signal determination step in which a plurality of reference acoustic signals corresponding to said reference characteristic reverberation times and said reference characteristic reverberation levels are determined;(b') at least one initial observation step during which, an acoustic energy decay rate distribution and the reference characteristic function are determined for each reference acoustic signal.
- The method according to claim 6 wherein during said reference signal determination step, at least one part of the reference acoustic signals and the reference characteristic reverberation times and characteristic reverberation levels corresponding to said reference acoustic signals are determined by calculation from a predetermined set of impulse responses.
- The method according to claim 6 wherein during said reference signal determination step, at least one part of the reference acoustic signals, the characteristic reverberation times and the reference characteristic reverberation levels corresponding to said reference acoustic signals are determined by measurement.
- A device for estimating the acoustic reverberations in an environment (7) comprising:(a) means of measurement (2) for capturing at least one acoustic signal emitted in the environment (7);(b) means of determination (8) of an acoustic energy decay rate distribution from the acoustic signal captured by the means of measurement, and for determining the characteristic function of the acoustic energy decay rate distribution;(c) means of estimation (8) of a characteristic reverberation time and a characteristic reverberation level of the sound in the environment from data representative of the acoustic energy decay rate distribution, where the regression is done with reference to:- reference characteristic functions representative respectively of several acoustic energy decay rate distributions;- reference characteristic reverberation times corresponding to said reference characteristic functions; and- reference characteristic reverberation levels corresponding to said reference characteristic functions.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| FR1561404A FR3044509B1 (en) | 2015-11-26 | 2015-11-26 | METHOD AND DEVICE FOR ESTIMATING ACOUSTIC REVERBERATION |
| PCT/FR2016/053034 WO2017089688A1 (en) | 2015-11-26 | 2016-11-21 | Method and device for estimating sound reverberation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| EP3381204A1 EP3381204A1 (en) | 2018-10-03 |
| EP3381204B1 true EP3381204B1 (en) | 2019-11-13 |
Family
ID=55236682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP16819332.4A Active EP3381204B1 (en) | 2015-11-26 | 2016-11-21 | Method and device for estimating sound reverberation |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US10313809B2 (en) |
| EP (1) | EP3381204B1 (en) |
| FR (1) | FR3044509B1 (en) |
| WO (1) | WO2017089688A1 (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP7449856B2 (en) | 2017-10-17 | 2024-03-14 | マジック リープ, インコーポレイテッド | mixed reality spatial audio |
| CN109754821B (en) * | 2017-11-07 | 2023-05-02 | 北京京东尚科信息技术有限公司 | Information processing method and system, computer system and computer readable medium |
| CA3090390A1 (en) | 2018-02-15 | 2019-08-22 | Magic Leap, Inc. | Mixed reality virtual reverberation |
| WO2019241754A1 (en) | 2018-06-14 | 2019-12-19 | Magic Leap, Inc. | Reverberation gain normalization |
| US11304017B2 (en) | 2019-10-25 | 2022-04-12 | Magic Leap, Inc. | Reverberation fingerprint estimation |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040213415A1 (en) * | 2003-04-28 | 2004-10-28 | Ratnam Rama | Determining reverberation time |
| US7813499B2 (en) | 2005-03-31 | 2010-10-12 | Microsoft Corporation | System and process for regression-based residual acoustic echo suppression |
| EP1885154B1 (en) | 2006-08-01 | 2013-07-03 | Nuance Communications, Inc. | Dereverberation of microphone signals |
| DE102007031677B4 (en) * | 2007-07-06 | 2010-05-20 | Sda Software Design Ahnert Gmbh | Method and apparatus for determining a room acoustic impulse response in the time domain |
| EP2058804B1 (en) | 2007-10-31 | 2016-12-14 | Nuance Communications, Inc. | Method for dereverberation of an acoustic signal and system thereof |
| US9407992B2 (en) | 2012-12-14 | 2016-08-02 | Conexant Systems, Inc. | Estimation of reverberation decay related applications |
| WO2016049403A1 (en) * | 2014-09-26 | 2016-03-31 | Med-El Elektromedizinische Geraete Gmbh | Determination of room reverberation for signal enhancement |
| US9607627B2 (en) * | 2015-02-05 | 2017-03-28 | Adobe Systems Incorporated | Sound enhancement through deverberation |
-
2015
- 2015-11-26 FR FR1561404A patent/FR3044509B1/en not_active Expired - Fee Related
-
2016
- 2016-11-21 EP EP16819332.4A patent/EP3381204B1/en active Active
- 2016-11-21 WO PCT/FR2016/053034 patent/WO2017089688A1/en not_active Ceased
- 2016-11-21 US US15/778,146 patent/US10313809B2/en active Active
Non-Patent Citations (1)
| Title |
|---|
| None * |
Also Published As
| Publication number | Publication date |
|---|---|
| US20180359582A1 (en) | 2018-12-13 |
| US10313809B2 (en) | 2019-06-04 |
| WO2017089688A1 (en) | 2017-06-01 |
| EP3381204A1 (en) | 2018-10-03 |
| FR3044509B1 (en) | 2017-12-15 |
| FR3044509A1 (en) | 2017-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3381204B1 (en) | Method and device for estimating sound reverberation | |
| EP3807669B1 (en) | Location of sound sources in a given acoustic environment | |
| EP2788980B1 (en) | Harmonicity-based single-channel speech quality estimation | |
| EP0589791A1 (en) | Method and apparatus for measuring the resemblance between speech samples | |
| CN107408394A (en) | Determination of noise power level difference and sound power level difference between main channel and reference channel | |
| EP4046390A1 (en) | Improved location of an acoustic source | |
| EP3040989B1 (en) | Improved method of separation and computer program product | |
| Shi et al. | Geoacoustic inversion of the acoustic-pressure vertical phase gradient from a single vector sensor | |
| EP1707970A1 (en) | Method for detecting the position of a wave front in a signal received by a detector | |
| EP3583403B1 (en) | Method for estimating the intensity of a wave emitted by an emitting source | |
| EP4248231A1 (en) | Improved location of an acoustic source | |
| EP1258168B1 (en) | Method and device for comparing signals to control transducers and transducer control system | |
| EP3295212B1 (en) | Method and device for probing by wave propagation | |
| EP3044580B1 (en) | Method for the non-destructive ultrasonic testing of a part by echo analysis | |
| EP3385899A1 (en) | Method and device for real-time detection of a scene | |
| CN115240701A (en) | Training method of noise reduction model, speech noise reduction method, device and electronic device | |
| WO2018189451A1 (en) | Echo sounding method and device for obtaining dispersion curves from a probe medium | |
| FR3122261A1 (en) | Device and process for processing the signal from a set of ultrasonic transducers | |
| Bilbao et al. | Directional reverberation time and the image source method for rectangular parallelepipedal rooms | |
| FR3030721A1 (en) | DEVICE AND METHOD FOR DETECTING FISH IN A CROSSING DEVICE | |
| FR3051959A1 (en) | METHOD AND DEVICE FOR ESTIMATING A DEREVERBERE SIGNAL | |
| FR3112017A1 (en) | Electronic equipment including a distortion simulator | |
| WO2011012789A1 (en) | Source location | |
| WO2024218269A1 (en) | Improved estimation of source-microphone distance and room geometry | |
| EP0015363A1 (en) | Speech detector with a variable threshold level |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
| 17P | Request for examination filed |
Effective date: 20180515 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| AX | Request for extension of the european patent |
Extension state: BA ME |
|
| DAV | Request for validation of the european patent (deleted) | ||
| DAX | Request for extension of the european patent (deleted) | ||
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602016024438 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: H04R0029000000 Ipc: G10L0025210000 |
|
| GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
| RIC1 | Information provided on ipc code assigned before grant |
Ipc: H04R 29/00 20060101ALI20190514BHEP Ipc: G10L 25/21 20130101AFI20190514BHEP Ipc: H04S 7/00 20060101ALI20190514BHEP |
|
| INTG | Intention to grant announced |
Effective date: 20190618 |
|
| GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
| GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
| AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 1202500 Country of ref document: AT Kind code of ref document: T Effective date: 20191115 |
|
| REG | Reference to a national code |
Ref country code: CH Ref legal event code: NV Representative=s name: VALIPAT S.A. C/O BOVARD SA NEUCHATEL, CH |
|
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602016024438 Country of ref document: DE |
|
| REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D Free format text: LANGUAGE OF EP DOCUMENT: FRENCH |
|
| REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20191113 |
|
| REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200213 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200213 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200214 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200313 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200313 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602016024438 Country of ref document: DE |
|
| REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1202500 Country of ref document: AT Kind code of ref document: T Effective date: 20191113 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
| 26N | No opposition filed |
Effective date: 20200814 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20161121 Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191113 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: MC Payment date: 20221020 Year of fee payment: 7 Ref country code: LU Payment date: 20221021 Year of fee payment: 7 Ref country code: IE Payment date: 20221014 Year of fee payment: 7 Ref country code: GB Payment date: 20221121 Year of fee payment: 7 Ref country code: DE Payment date: 20221114 Year of fee payment: 7 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: CH Payment date: 20221129 Year of fee payment: 7 Ref country code: BE Payment date: 20221125 Year of fee payment: 7 |
|
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 602016024438 Country of ref document: DE |
|
| REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231130 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231121 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231130 |
|
| GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20231121 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231130 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231121 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231130 |
|
| PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20240603 Year of fee payment: 8 |
|
| REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20231130 |
|
| REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20240601 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231121 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231121 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231130 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231121 Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231121 Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20240601 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231130 |
|
| PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20241130 |