[go: up one dir, main page]

EP1777698A1 - Réduction de débit dans un codeur audio utilisant un effet de non-harmonique et masquage temporaire - Google Patents

Réduction de débit dans un codeur audio utilisant un effet de non-harmonique et masquage temporaire Download PDF

Info

Publication number
EP1777698A1
EP1777698A1 EP07002179A EP07002179A EP1777698A1 EP 1777698 A1 EP1777698 A1 EP 1777698A1 EP 07002179 A EP07002179 A EP 07002179A EP 07002179 A EP07002179 A EP 07002179A EP 1777698 A1 EP1777698 A1 EP 1777698A1
Authority
EP
European Patent Office
Prior art keywords
determining
masking
temporal
temporal masking
subband
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP07002179A
Other languages
German (de)
English (en)
Other versions
EP1777698B1 (fr
Inventor
Hossein Najaf-Zadeh
Hassan Lahdili
Louis Thibault
William Treurniet
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canada Minister of Industry
Original Assignee
Canada Minister of Industry
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canada Minister of Industry filed Critical Canada Minister of Industry
Priority claimed from EP03405620A external-priority patent/EP1398761B1/fr
Publication of EP1777698A1 publication Critical patent/EP1777698A1/fr
Application granted granted Critical
Publication of EP1777698B1 publication Critical patent/EP1777698B1/fr
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders

Definitions

  • the present invention relates generally to the field of perceptual audio coding and more particularly to a method for determining masking thresholds using a psychoacoustic model.
  • perceptual models based on characteristics of a human ear are typically employed to reduce the number of bits required to code a given input audio signal.
  • the perceptual models are based on the fact that a considerable portion of an acoustic signal provided to the human ear is discarded - masked - due to the characteristics of the human hearing process. For example, if a loud sound is presented to the human ear along with a softer sound, the ear will likely hear only the louder sound. Whether the human ear will hear both, the loud and soft sound, depends on the frequency and intensity of each of the signals.
  • audio coding techniques are able to effectively ignore the softer sound and not assign any bits to its transmission and reproduction under the assumption that a human listener is not capable of hearing the softer sound even if it is faithfully transmitted and reproduced. Therefore, psychoacoustic models for calculating a masking threshold play an essential role in state of the art audio coding. An audio component whose energy is less than the masking threshold is not perceptible and is, therefore, removed by the encoder. For the audible components, the masking threshold determines the acceptable level of quantization noise during the coding process.
  • the MPEG-1 Layer 2 audio encoder is widely used in Digital Audio Broadcasting (DAB) and digital receivers based on this standard have been massively manufactured making it impossible to change the decoder in order to improve sound quality. Therefore, enhancing the psychoacoustic model is an option for improving sound quality without requiring a new standard.
  • DAB Digital Audio Broadcasting
  • an object of the present invention to provide a method for encoding an audio signal employing an improved psychoacoustic model for calculating a masking threshold.
  • a method for encoding an audio signal comprising the steps of:
  • a method for encoding an audio signal comprising the steps of:
  • a method for encoding an audio signal comprising the steps of:
  • a method for encoding an audio signal comprising the steps of:
  • a method for encoding an audio signal comprising the steps of:
  • Fig. 1 is a simplified flow diagram of a first embodiment of a method for encoding an audio signal according to the present invention
  • Fig. 2 is a diagram illustrating reduction in SMR due to temporal masking
  • Figs. 3a and 3b are diagrams illustrating an example of a harmonic and an inharmonic signal, respectively;
  • Fig. 4 is a simplified flow diagram illustrating a process for determining inharmonicity of an audio signal according to the invention
  • Figs. 5a and 5b are diagrams illustrating the outputs of a gammatone filterbank for a harmonic and an inharmonic signal, respectively;
  • Figs. 6a and 6b are diagrams illustrating the envelope autocorrelation for a harmonic and an inharmonic signal, respectively.
  • Fig. 7 is a simplified flow diagram of a second embodiment of a method for encoding an audio signal according to the present invention.
  • Temporal masking occurs when a masker - louder sound - and a maskee - weaker sound - are presented to the hearing system at different time instances. Detailed information about the temporal masking is disclosed in the following references:
  • the temporal masking characteristic of the human hearing system is asymmetric, i.e. "backward masking” is effective approximately 5 msec before occurrence of a masker, whereas “forward masking” lasts up to 200 msec after the end of the masker.
  • Different phenomena contributing to temporal auditory masking effects include temporal overlap of basilar membrane responses to different stimuli, short term neural fatigue at higher neural levels and persistence of the neural activity caused by a masker, disclosed in B. Moore, “An Introduction to the Psychology of Hearing", Academic Press, 1997 ; and A. Harma, "Psychoacoustic Temporal Masking Effects with Artificial and Real Signals", Hearing Seminar, Espoo, Finland, pp. 665-668, 1999 .
  • psychoacoustic models are used for adaptive bit allocation, the accuracy of those models greatly affects the quality of encoded audio signals. Since digital receivers have been massively manufactured and are now readily available, it is not desirable to change the decoder requirements by introducing a new standard. However, enhancing the psychoacoustic model employed within the encoders allows for improved sound quality of an encoded audio signal without modifying the decoder hardware. Incorporating non-linear masking effects such as temporal masking and inharmonicity into the MPEG-1 psychoacoustic model 2 significantly reduces the bit rate for transparent coding or equivalently, improves the sound quality of an encoded audio signal at a same bit rate.
  • a temporal masking index is determined in a non-linear fashion in time domain and implemented into a psychoacoustic model for calculating a masking threshold.
  • a combined masking threshold considering temporal and simultaneous masking is calculated using the MPEG-1 psychoacoustic model 2. Listening tests have been performed with MPEG-1 Layer 2 audio encoder using the combined masking threshold.
  • the temporal masking method according to the invention is implemented in the MPEG-1 Layer 2 encoder, the relation between some of the encoder parameters and the temporal masking method will be discussed in the following.
  • 32 Signal-to-Mask-Ratios (SMR) corresponding to 32 subbands are calculated for each block of 1152 input audio samples. Since the time-to-frequency mapping in the encoder is critically sampled, the filterbank produces a matrix - frame - of 1152 subband samples, i.e. 36 subband samples in each of the 32 subbands.
  • the temporal masking method according to the invention as implemented in the MPEG-1 psychoacoustic model acquires 72 subband samples - 36 samples belonging to a current frame and 36 samples belonging to a previous frame - in each subband and provides 32 temporal masking thresholds.
  • Fig. 1 a simplified flow diagram of the first embodiment of a method for encoding an audio signal is shown.
  • the temporal masking method has been implemented using the following model suggested by W. Jesteadt, S. Bacon, and J. Lehman, "Forward masking as a function of frequency, masker level, and signal delay", J. Acoust. Soc. Am., Vol. 71, No. 4, pp.
  • M a ⁇ b - log 10 ⁇ t ⁇ L m - c
  • M the amount of masking in dB
  • t the time distance between the masker and the maskee in msec
  • L m the masker level in dB
  • a , b, and c are parameters found from psychoacoustic data.
  • the time axis is reversed.
  • the time distance ⁇ between successive subband samples is a function of the sampling frequency. Since the filterbank in the MPEG audio encoder is critically sampled - box 10 - one subband sample in each subband is produced for 32 input time samples. Therefore, the time distance ⁇ between successive subband samples is 32/ f s msec, where f s is the sampling frequency in kHz.
  • the masker level is calculated as the average energy of the 36 subband samples in the corresponding subband in the previous frame and the subband samples in the current frame up to time index i.
  • the above equation gives the backward masker level at any time as the average energy of the current and future subband samples.
  • a combined masking threshold is then calculated considering the effect of both temporal and simultaneous masking.
  • the SMRs due to temporal masking are translated into allowable noise levels within a frequency domain.
  • Parseval's theorem is used to calculate the equivalent noise level in the frequency domain.
  • the noise levels due to temporal and simultaneous masking are combined - box 28.
  • One possibility is to linearly sum the masking energies.
  • the linear combination results in an under-estimation of the net masking threshold.
  • p a value of 0.4 has been found to provide an accurate combined masking threshold.
  • the acoustic signal is encoded using the masking threshold determined above - box 32.
  • Figure 2 shows an amount of reduction in SMR due to temporal masking in a frame of 1152 subband samples - 36 samples in each of 32 subbands.
  • Table 1 shows the average bit rate for a few test files coded with a MPEG-1 Layer 2 encoder using the standard psychoacoustic model 2 and using the modified psychoacoustic model.
  • the test files were 2-channel stereo audio signals sampled at 48 kHz with 16-bit resolution.
  • a sound is harmonic if its energy is concentrated in equally spaced frequency bins, i.e. harmonic partials.
  • the distance between successive harmonic partials is known as the fundamental frequency whose inverse is called pitch.
  • Many natural sounds such as harpsichord or clarinet consist of partials that are harmonically related.
  • inharmonic signals consist of individual sinusoids, which are not equally separated in the frequency domain.
  • a model developed to measure inharmonicity recognizes that an auditory filter output envelope is modulated when the filter passes two or more sinusoids as shown in Appendix A. since a harmonic masker has constant frequency differences between its adjacent partials, most auditory filters will have the same dominant modulation rate. On the other hand, for an inharmonic masker, the envelope modulation rate varies across auditory filters because the frequency differences are not constant.
  • the signal is a complex masker comprising a plurality of partials
  • interaction of neighboring partials causes local variations of the basilar membrane vibration pattern.
  • the output signal from an auditory filter centered at the corresponding frequency has an amplitude modulation corresponding to that location.
  • the modulation rate of a given filter is the difference between the adjacent frequencies processed by that filter. Therefore, the dominant output modulation rate is constant across filters for a harmonic signal because this frequency difference is constant.
  • the modulation rate varies across filters. Consequently, in the case of a harmonic masker the modulation rate for each filter output signal is the fundamental frequency.
  • inharmonicity is introduced by perturbing the frequencies of the partials, a variation of the modulation rate across filters is noticeable. The variation increases with increasing inharmonicity.
  • the harmonicity nature of a complex masker is characterized by the variance calculated from the envelope modulation rates across a plurality of auditory filters.
  • a harmonic signal is characterized by particular relationships among sharp peaks in the spectrum
  • an appropriate starting point for measuring the effect of harmonicity is a masker having a similar distribution of energy across filters, but with small perturbations in the relationships among the spectral peaks.
  • Fig. 3a shows an example of a harmonic signal comprising a fundamental frequency of 88 Hz, and a total of 45 equally spaced partials covering a range from 88 Hz to 3960 Hz.
  • Fig. 3b shows an inharmonic signal generated by slightly perturbing the frequencies and randomizing the phases of the harmonic signal partials.
  • a process for estimating the harmonicity is illustrated in the flow chart of Fig. 4.
  • the signal is analyzed using a "gammatone" filterbank based on the concept of critical bands disclosed in E. Zwicker, and E. Terhardt, "Analytical expressions for critical-band rate and critical bandwidth as a function of frequency", J. Acoust. Soc. Am., 68(5), pp. 1523-1525, 1980 .
  • the output of each filter is processed with a Hilbert transform to extract the envelope.
  • An autocorrelation is then applied to the envelope to estimate its period.
  • the harmonicity measure is related to the variance of the modulation rates, i.e. envelope periods. This variance is negligible for a harmonic masker.
  • Figs. 5a, 5b, 6a, and 6b illustrate the output signals of the gammatone filterbank - channels 7-12 - and the corresponding autocorrelation functions for the harmonic - Figs. 5a and 6a - and inharmonic inputs- Figs. 5b and 6b.
  • Figs. 6a and 6b there is a notable difference between the autocorrelation functions. In the case of the harmonic signal all the peaks related to the dominant modulation rate are coincident.
  • a harmonicity estimation model based on the variability of envelope modulation rates differentiates harmonic from inharmonic maskers.
  • the variance of the modulation rate measures the degree to which an audio signal departs from harmonicity, i.e. a near zero value implies a harmonic signal while a large value - a few hundreds - corresponds to a noise-like signal.
  • the minimum SMRs are computed for 32 subbands as follows.
  • a block of 1056 input samples is taken from the input signal.
  • the first 1024 samples are windowed using a Hanning window and transformed into the frequency domain using a 1024-point FFT.
  • the tonality of each spectral line is determined by predicting its magnitude and phase from the two corresponding values in the previous transforms.
  • the difference of each DFT coefficient and its predicted value is used to calculate the unpredictability measure.
  • the unpredictability measure is converted to the "tonality" factor using an empirical factor with a larger value indicating a tonal signal.
  • NMT j is set to 5.5 dB and TMN j is given in a table provided in the MPEG audio standard.
  • SNR j is determined to be larger than the minimum SNR minval j given in the standard.
  • the SMR is calculated for each of the 32 subbands from the corresponding SNR.
  • the MPEG-1 psychoacoustic model 2 has been modified considering imperfect harmonic structures of complex tonal sounds. It will become apparent to those skilled in the art that the method considering imperfect harmonic structures is not limited to the implementation in the MPEG-1 psychoacoustic model 2 but is also implementable into other psychoacoustic models.
  • the TMN parameter is given in a table.
  • the values for the TMNs are based on psychoacoustic experiments in which a pure tone is used to mask a narrowband noise.
  • the masker is periodic, which is the case with an inharmonic masker.
  • a noise probe is detected at a lower level when the masker is harmonic. This is likely caused by a disruption of the pitch sensation due to the periodic structure of the masker's temporal envelope, as taught in W.C. Treumiet, and D.R. Boucher, "A masking level difference due to harmonicity", J. Acoust. Soc. Am., 109(1), pp. 306-320, 2001 .
  • the TMN parameter is modified in dependence upon the input signal inharmonicity, as shown in the flow diagram of Fig. 7. Since in the MPEG-1 Layer 2 psychoacoustic model 2 a set of 32 SMRs is calculated for each 1152 time samples, the same time samples are analyzed for measuring the level of input signal inharmonicity. After determining the input signal inharmonicity, an inharmonicity index is calculated and subtracted from the TMN values. The inharmonicity index as a function of the periodic structure of the input signal is calculated as follows. The input block of 1632 time samples is decomposed using a gammatone filterbank - box 100.
  • each bandpass auditory filter output is detected using the Hilbert transform - box 102.
  • the pitch of each envelope is calculated based on the autocorrelation of the envelope - box 104.
  • Each pitch value is then compared with the other pitch values and an average error is determined and the variance of the average errors is calculated - box 106.
  • the above equation produces a zero value for a perfect harmonic signal and up to 10 dB for noise-like input signals.
  • the level of inharmonicity is defined as the variance of the periods of the envelopes of auditory filters outputs.
  • the period of each envelope is found using the autocorrelation function.
  • the location of the second peak of the autocorrelation function - ignoring the largest peak at the origin - determines the period. Since the autocorrelation function of a periodic signal has a plurality of peaks, the second largest peak sometimes does not correspond to the correct period.
  • the smaller period is compared to a submultiple of the larger period if the difference becomes smaller.
  • a MATLAB script for calculating the pitch variance is presented in Appendix B. Another problem occurs when there is no peak in the autocorrelation function. This situation implies an aperiodic envelope. In this case the period is set to an arbitrary or random value.
  • the envelope of the output signal is periodic. Therefore, in order to correctly analyze an audio signal the lowest frequency of the gammatone filterbank is chosen such that the auditory filter centered at this frequency passes at least two harmonics. Therefore, the corresponding critical bandwidth centered at this frequency is chosen to be greater than twice the fundamental frequency of the input signal.
  • the fundamental frequency is determined by analyzing the input signal either in the time domain or the frequency domain. However, in order to avoid extra computation for determining the fundamental frequency the median of the calculated pitch values is assumed to be the period of the input signal. The fundamental frequency of the input signal is then simply the inverse of the pitch value. Therefore, the lower bound for the analysis frequency range is set to twice the inverse of the pitch value.
  • the masking threshold is modified based on the local harmonic structure of the input signal based on a local wideband frequency spectrum of the input signal.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
EP07002179A 2002-08-27 2003-08-27 Réduction de débit dans un codeur audio utilisant un effet de masquage temporaire Expired - Lifetime EP1777698B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US40605502P 2002-08-27 2002-08-27
EP03405620A EP1398761B1 (fr) 2002-08-27 2003-08-27 Réduction de débit dans un codeur audio utilisant un effet de non harmonique

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
EP03405620A Division EP1398761B1 (fr) 2002-08-27 2003-08-27 Réduction de débit dans un codeur audio utilisant un effet de non harmonique

Publications (2)

Publication Number Publication Date
EP1777698A1 true EP1777698A1 (fr) 2007-04-25
EP1777698B1 EP1777698B1 (fr) 2008-09-03

Family

ID=37887867

Family Applications (1)

Application Number Title Priority Date Filing Date
EP07002179A Expired - Lifetime EP1777698B1 (fr) 2002-08-27 2003-08-27 Réduction de débit dans un codeur audio utilisant un effet de masquage temporaire

Country Status (1)

Country Link
EP (1) EP1777698B1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103761969A (zh) * 2014-02-20 2014-04-30 武汉大学 基于高斯混合模型的感知域音频编码方法及系统

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
TED PAINTER ET AL: "Perceptual Coding of Digital Audio", PROCEEDINGS OF THE IEEE, IEEE. NEW YORK, US, vol. 88, no. 4, April 2000 (2000-04-01), XP011044355, ISSN: 0018-9219 *
YUAN-HAO HUANG ET AL: "A New Audio Coding Scheme Using a Forward Masking Model and Perceptually WeightedVector Quantization", IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, IEEE SERVICE CENTER, NEW YORK, NY, US, vol. 10, no. 5, July 2002 (2002-07-01), XP011079654, ISSN: 1063-6676 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103761969A (zh) * 2014-02-20 2014-04-30 武汉大学 基于高斯混合模型的感知域音频编码方法及系统
CN103761969B (zh) * 2014-02-20 2016-09-14 武汉大学 基于高斯混合模型的感知域音频编码方法及系统

Also Published As

Publication number Publication date
EP1777698B1 (fr) 2008-09-03

Similar Documents

Publication Publication Date Title
US20080221875A1 (en) Bit rate reduction in audio encoders by exploiting inharmonicity effects and auditory temporal masking
Johnston Transform coding of audio signals using perceptual noise criteria
RU2734781C1 (ru) Устройство для постобработки звукового сигнала с использованием выявления места всплеска
van de Par et al. A perceptual model for sinusoidal audio coding based on spectral integration
JP3418198B2 (ja) オーディオ信号の聴覚に適応した品質評価方法および装置
Thiede et al. A new perceptual quality measure for bit rate reduced audio
US20020120445A1 (en) Coding signals
US7634400B2 (en) Device and process for use in encoding audio data
EP1517300B1 (fr) Codage de données audio
Thiagarajan et al. Analysis of the MPEG-1 Layer III (MP3) algorithm using MATLAB
Huber Objective assessment of audio quality using an auditory processing model
US11830507B2 (en) Coding dense transient events with companding
EP1777698B1 (fr) Réduction de débit dans un codeur audio utilisant un effet de masquage temporaire
Najaf-Zadeh et al. Perceptual matching pursuit for audio coding
Najaf-Zadeh et al. Use of auditory temporal masking in the MPEG psychoacoustic model 2
Taghipour Psychoacoustics of detection of tonality and asymmetry of masking: implementation of tonality estimation methods in a psychoacoustic model for perceptual audio coding
Spanias et al. Analysis of the MPEG-1 Layer III (MP3) Algorithm using MATLAB
Boland et al. Hybrid LPC And discrete wavelet transform audio coding with a novel bit allocation algorithm
Fenton et al. Hybrid Multiresolution Analysis of “Punch” in Musical Signals
van de Par et al. Informed postprocessing for auditory roughness removal for low-bitrate audio coders
Goodwin et al. Predicting and preventing unmasking incurred in coded audio post-processing
Najaf-Zadeh et al. Incorporation of Inharmonicity Effects into Auditory Masking Models
Nemer et al. Perceptual Weighting to Improve Coding of Harmonic Signals
Boland et al. A new hybrid LPC-DWT algorithm for high quality audio coding
Krimi et al. Realization of a psychoacoustic model for MPEG 1 using gammachirp wavelet transform

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AC Divisional application: reference to earlier application

Ref document number: 1398761

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR

17P Request for examination filed

Effective date: 20071024

RIN1 Information on inventor provided before grant (corrected)

Inventor name: THIBAULT, LOUIS

Inventor name: LAHDILI, HASSAN

Inventor name: NAJAF-ZADEH, HOSSEIN

Inventor name: TREURNIET, WILLIAM

17Q First examination report despatched

Effective date: 20071121

AKX Designation fees paid

Designated state(s): DE FR GB

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RTI1 Title (correction)

Free format text: BIT RATE REDUCTION IN AUDIO ENCODERS BY EXPLOITING AUDITORY TEMPORAL MASKING

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AC Divisional application: reference to earlier application

Ref document number: 1398761

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE FR GB

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REF Corresponds to:

Ref document number: 60323412

Country of ref document: DE

Date of ref document: 20081016

Kind code of ref document: P

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20090604

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20120725

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20120823

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20121025

Year of fee payment: 10

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20130827

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20140301

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 60323412

Country of ref document: DE

Effective date: 20140301

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20140430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20130827

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20130902