JP7686439B2 - Audio processing device, control method, and program - Google Patents

Description

The present invention relates to an audio processing device capable of reducing noise contained in audio data.

A digital camera, which is an example of an audio processing device, can record surrounding audio when recording video data. Digital cameras also have an autofocus function that drives an optical lens to focus on a subject while recording video data. Digital cameras also have a function that drives an optical lens to zoom while recording video.

In this way, when an optical lens is driven while a video is being recorded, the sound of the optical lens being driven may be included as noise in the sound recorded along with the video. Conventionally, therefore, when a digital camera picks up noise such as the sliding sound generated when the optical lens is driven as noise, it is possible to reduce that noise and record surrounding sounds. Patent Document 1 discloses a digital camera that reduces noise using the spectral subtraction method.

JP 2011-205527 A

However, in Patent Document 1, the digital camera creates a noise pattern from noise picked up by a microphone that records surrounding sounds, so there is a possibility that an accurate noise pattern cannot be obtained from the sliding sounds generated inside the housing of the optical lens. In this case, there is a risk that the digital camera will not be able to effectively reduce the noise contained in the picked-up sounds.

Therefore, the present invention aims to effectively reduce noise.

The audio processing device of the present invention includes a first microphone for acquiring environmental sound, a second microphone for acquiring sound from a noise source, a first conversion means for Fourier-transforming an audio signal from the first microphone to generate a first audio signal, a second conversion means for Fourier-transforming an audio signal from the second microphone to generate a second audio signal, a generation means for generating first noise data corresponding to the noise from the noise source using the second audio signal and parameters related to the noise of the noise source, a first noise reduction process for reducing the noise of the noise source contained in the first audio signal based on the first noise data, and a second noise reduction process for reducing constant noise contained in the first audio signal, and includes a first noise reduction processing means for performing the first noise reduction process on the first audio signal output from the first conversion means, and a second noise reduction processing means for performing the first noise reduction process on the first audio signal output from the first conversion means, and a second noise reduction processing means for performing the first noise reduction process on the first audio signal output from the first conversion means, and a second noise reduction processing means for performing the first noise reduction process on the first audio signal output from the first conversion means, The noise reduction means includes a second noise reduction processing means for performing the second noise reduction processing on the audio signal output from the processing means, a switching means for outputting the first audio signal output from the first conversion means to the first noise reduction processing means so that the first noise reduction processing is performed when noise from the noise source is occurring, and outputting the first audio signal output from the first conversion means to the second noise reduction processing means so that the first noise reduction processing is not performed when noise from the noise source is not occurring, and a third conversion means for performing an inverse Fourier transform on the audio signal from the noise reduction means, and the second noise reduction processing means performs the second noise reduction processing on the audio signal output from the first noise reduction processing means when noise from the noise source is occurring, and performs the second noise reduction processing on the first audio signal output from the first conversion means when noise from the noise source is not occurring.

The audio processing device of the present invention can effectively reduce noise.

FIG. 1 is a perspective view of an imaging device according to a first embodiment. 1 is a block diagram showing a configuration of an imaging apparatus according to a first embodiment. 2 is a block diagram showing a configuration of a voice input unit of the imaging device according to the first embodiment. FIG. 3 is a diagram showing an arrangement of microphones in a voice input unit of the imaging device according to the first embodiment. FIG. FIG. 4 is a diagram showing noise parameters in the first embodiment. 13A and 13B are diagrams illustrating a frequency spectrum of an audio and a frequency spectrum of a noise parameter in a case where a drive sound occurs in a situation where it is considered that there is no environmental sound in the first embodiment. 11 is a diagram showing a frequency spectrum of audio when a drive sound occurs in a situation where an environmental sound is present in the first embodiment; FIG. FIG. 4 is a block diagram showing a configuration of a noise parameter selection unit in the first embodiment. 4 is a timing chart relating to the audio noise reduction process in the first embodiment. FIG. 11 is a block diagram showing a configuration of a voice input unit of an imaging device according to a second embodiment. FIG. 13 is a block diagram showing a configuration of a voice input unit of an imaging device according to a third embodiment. FIG. 13 is a diagram illustrating noise parameters in the third embodiment. 13 is a timing chart relating to an audio noise reduction process in the third embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings.

[First embodiment]
<External view of imaging device 100>
1A and 1B show an example of an external view of an image capture device 100 as an example of an audio processing device to which the present invention can be applied. Fig. 1A is an example of a front perspective view of the image capture device 100. Fig. 1B is an example of a rear perspective view of the image capture device 100. In Fig. 1, an optical lens (not shown) is attached to a lens mount 301.

The display unit 107 displays image data, text information, and the like. The display unit 107 is provided on the rear surface of the imaging device 100. The outside-finder display unit 43 is a display unit provided on the top surface of the imaging device 100. The outside-finder display unit 43 displays the settings of the imaging device 100, such as the shutter speed and aperture value. The eyepiece viewfinder 16 is a peer-in type viewfinder. The user can check the focus and composition of the optical image of the subject by observing the focusing screen inside the eyepiece viewfinder 16.

The release switch 61 is an operating member that allows the user to give instructions to shoot. The mode changeover switch 60 is an operating member that allows the user to switch between various modes. The main electronic dial 71 is a rotating operating member. By turning the main electronic dial 71, the user can change the settings of the imaging device 100, such as the shutter speed and aperture value. The release switch 61, the mode changeover switch 60, and the main electronic dial 71 are included in the operation unit 112.

The power switch 72 is an operating member that switches the power of the imaging device 100 on and off. The sub electronic dial 73 is a rotating operating member. The user can use the sub electronic dial 73 to move the selection frame displayed on the display unit 107 and to advance images in playback mode. The cross key 74 is a cross key (four-way key) that can be pressed up, down, left, or right. The imaging device 100 executes processing according to the part (direction) of the cross key 74 that is pressed. The power switch 72, the sub electronic dial 73, and the cross key 74 are included in the operation unit 112.

The SET button 75 is a push button. The SET button 75 is mainly used by the user to confirm the selection items displayed on the display unit 107. The LV button 76 is a button used to switch live view (hereinafter referred to as LV) on and off. In the video recording mode, the LV button 76 is used to instruct the start and stop of video shooting (recording). The enlarge button 77 is a push button for turning the enlargement mode on and off in the live view display in the shooting mode, and for changing the magnification ratio in the enlargement mode. The SET button 75, LV button 76, and enlargement button 77 are included in the operation unit 112.

In the playback mode, the enlarge button 77 functions as a button for increasing the magnification of image data displayed on the display unit 107. The reduce button 78 is a button for decreasing the magnification of image data enlarged and displayed on the display unit 107. The play button 79 is an operation button for switching between the shooting mode and the play mode. When the user presses the play button 79 while the imaging device 100 is in the shooting mode, the imaging device 100 transitions to the play mode, and the image data recorded on the recording medium 110 is displayed on the display unit 107. The reduce button 78 and the play button 79 are included in the operation unit 112.

The quick return mirror 12 (hereinafter, mirror 12) is a mirror for switching the light beam incident from the optical lens attached to the imaging device 100 so that it is incident on either the eyepiece finder 16 side or the imaging unit 101 side. The mirror 12 is raised and lowered by the control unit 111 controlling an actuator (not shown) during exposure, live view shooting, and video shooting. The mirror 12 is normally arranged to allow the light beam to be incident on the eyepiece finder 16. When shooting or live view display is performed, the mirror 12 flips up (mirror up) so that the light beam is incident on the imaging unit 101. The center of the mirror 12 is also a half mirror. A part of the light beam that passes through the center of the mirror 12 is incident on a focus detection unit (not shown) for focus detection.

The communication terminal 10 is a communication terminal for communication between the imaging device 100 and the optical lens 300 attached to the imaging device 100. The terminal cover 40 is a cover that protects a connector (not shown) such as a connection cable that connects the imaging device 100 to a connection cable with an external device. The lid 41 is a lid for a slot that stores the recording medium 110. The lens mount 301 is an attachment portion to which the optical lens 300 (not shown) can be attached.

The L microphone 201a and the R microphone 201b are microphones for picking up the user's voice, etc. When viewed from the rear of the imaging device 100, the L microphone 201a is located on the left side and the R microphone 201b is located on the right side.

<Configuration of imaging device 100>
FIG. 2 is a block diagram showing an example of the configuration of the image pickup apparatus 100 according to this embodiment.

The optical lens 300 is a lens unit that can be attached to and detached from the imaging device 100. For example, the optical lens 300 is a zoom lens or a varifocal lens. The optical lens 300 has an optical lens, a motor for driving the optical lens, and a communication unit that communicates with the lens control unit 102 of the imaging device 100 described below. The optical lens 300 can focus and zoom on a subject and correct camera shake by moving the optical lens using the motor based on a control signal received by the communication unit.

The imaging unit 101 has an imaging element for converting an optical image of a subject formed on an imaging surface via an optical lens 300 into an electrical signal, and an image processing unit for generating and outputting image data or video data from the electrical signal generated by the imaging element. The imaging element is, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). In this embodiment, a series of processes for generating image data including still image data and video data in the imaging unit 101 and outputting the image data from the imaging unit 101 is called "shooting". In the imaging device 100, the image data is recorded on a recording medium 110 (described later) in accordance with the DCF (Design rule for Camera File system) standard.

The lens control unit 102 transmits a control signal to the optical lens 300 via the communication terminal 10 based on the data output from the imaging unit 101 and a control signal output from the control unit 111 described later, and controls the optical lens 300.

The information acquisition unit 103 detects the tilt of the imaging device 100 and the temperature inside the housing of the imaging device 100. For example, the information acquisition unit 103 detects the tilt of the imaging device 100 using an acceleration sensor or a gyro sensor. Also, for example, the information acquisition unit 103 detects the temperature inside the housing of the imaging device 100 using a temperature sensor.

The audio input unit 104 generates audio data from audio acquired by a microphone. The audio input unit 104 acquires audio around the imaging device 100 by the microphone, performs analog-to-digital conversion (A/D conversion) and various audio processes on the acquired audio, and generates audio data. In this embodiment, the audio input unit 104 has a microphone. A detailed configuration example of the audio input unit 104 will be described later.

The volatile memory 105 temporarily records image data generated by the imaging unit 101 and audio data generated by the audio input unit 104. The volatile memory 105 is also used as a temporary storage area for image data displayed on the display unit 107, a working area for the control unit 111, etc.

The display control unit 106 controls the display unit 107 to display image data output from the imaging unit 101, characters for interactive operation, menu screens, etc. The display control unit 106 can also control the display unit 107 to sequentially display digital data output from the imaging unit 101 during still image shooting and video shooting, thereby allowing the display unit 107 to function as an electronic viewfinder. For example, the display unit 107 is a liquid crystal display or an organic EL display. The display control unit 106 can also control the image data and video data output from the imaging unit 101, characters for interactive operation, menu screens, etc. to be displayed on an external display via the external output unit 115 described later.

The encoding processing unit 108 can encode the image data and audio data temporarily recorded in the volatile memory 105. For example, the encoding processing unit 108 can generate moving image data by encoding and compressing the image data according to the JPEG standard or RAW image format. For example, the encoding processing unit 108 can generate moving image data by encoding and compressing the video data according to the MPEG2 standard or H.264/MPEG4-AVC standard. Also, for example, the encoding processing unit 108 can generate audio data by encoding and compressing the audio data according to the AC3AAC standard, the ATRAC standard, or the ADPCM method. Also, the encoding processing unit 108 may encode the audio data according to, for example, the linear PCM method so as not to compress the audio data.

The recording control unit 109 can record data to the recording medium 110 and read data from the recording medium 110. For example, the recording control unit 109 can record still image data, video data, and audio data generated by the encoding processing unit 108 to the recording medium 110 and read the data from the recording medium 110. The recording medium 110 is, for example, an SD card, a CF card, an XQD memory card, a HDD (magnetic disk), an optical disk, or a semiconductor memory. The recording medium 110 may be configured to be detachable from the imaging device 100, or may be built into the imaging device 100. In other words, the recording control unit 109 only needs to have a means for accessing at least the recording medium 110.

The control unit 111 controls each component of the imaging device 100 via the data bus 116 in accordance with the input signals and a program described below. The control unit 111 has a CPU, ROM, and RAM for executing various controls. Note that instead of the control unit 111 controlling the entire imaging device 100, multiple pieces of hardware may share the control of the entire imaging device. The ROM of the control unit 111 stores programs for controlling each component. The RAM of the control unit 111 is a volatile memory used for calculation processing, etc.

The operation unit 112 is a user interface for receiving instructions from the user for the imaging device 100. The operation unit 112 has, for example, a power switch 72 for turning the power of the imaging device 100 on or off, a release switch 61 for issuing an instruction to shoot, a playback button for issuing an instruction to play back image data or video data, and a mode change switch 60.

The operation unit 112 outputs a control signal to the control unit 111 in response to a user's operation. The operation unit 112 can also include a touch panel formed on the display unit 107. The release switch 61 has SW1 and SW2. When the release switch 61 is pressed halfway, SW1 is turned on. This accepts a preparation instruction for performing a preparation operation for imaging, such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing. When the release switch 61 is pressed fully, SW2 is turned on. This user operation accepts an imaging instruction for performing an imaging operation. The operation unit 112 also includes an operating member (e.g., a button) that can adjust the volume of audio data reproduced from a speaker 114, which will be described later.

The audio output unit 113 can output audio data to the speaker 114 and the external output unit 115. The audio data input to the audio output unit 113 is audio data read from the recording medium 110 by the recording control unit 109, audio data output from the non-volatile memory 117, and audio data output from the encoding processing unit. The speaker 114 is an electro-acoustic transducer that can play back audio data.

The external output unit 115 can output image data, video data, audio data, and the like to an external device. The external output unit 115 is composed of, for example, a video terminal, a microphone terminal, and a headphone terminal.

Data bus 116 is a data bus for transmitting various data such as audio data, video data, and image data, as well as various control signals, to each block of imaging device 100.

The non-volatile memory 117 is a non-volatile memory that stores programs executed by the control unit 111, which will be described later. The non-volatile memory 117 also stores audio data. This audio data is, for example, audio data of electronic sounds such as a focusing sound that is output when the subject is focused on, an electronic shutter sound that is output when an instruction to shoot is given, and an operation sound that is output when the imaging device 100 is operated.

<Operation of Imaging Apparatus 100>
The operation of the imaging device 100 of this embodiment will now be described.

In the imaging device 100 of this embodiment, when the user turns on the power by operating the power switch 72, power is supplied to each component of the imaging device from a power source (not shown). For example, the power source is a battery such as a lithium ion battery or an alkaline manganese dry battery.

When power is supplied, the control unit 111 determines whether the camera will operate in a shooting mode or a playback mode, for example, based on the state of the mode changeover switch 60. In the video recording mode, the control unit 111 records the video data output from the imaging unit 101 and the audio data output from the audio input unit 104 as one piece of video data with audio. In the playback mode, the control unit 111 reads out the image data or video data recorded on the recording medium 110 using the recording control unit 109, and controls the display unit 107 to display it.

First, the video recording mode will be described. In the video recording mode, the control unit 111 first transmits a control signal to each component of the imaging device 100 to transition the imaging device 100 to a shooting standby state. For example, the control unit 111 controls the imaging unit 101 and the audio input unit 104 to perform the following operations.

The imaging unit 101 converts the optical image of the subject formed on the imaging surface via the optical lens 300 into an electrical signal, and generates video data from the electrical signal generated by the imaging element. The imaging unit 101 then transmits the video data to the display control unit 106, which displays it on the display unit 107. The user can prepare for shooting while viewing the video data displayed on the display unit 107.

The audio input unit 104 A/D converts analog audio signals input from multiple microphones to generate multiple digital audio signals. The audio input unit 104 then generates multiple channels of audio data from the multiple digital audio signals. The audio input unit 104 transmits the generated audio data to the audio output unit 113, which plays the audio data from the speaker 114. While listening to the audio data played from the speaker 114, the user can use the operation unit 112 to adjust the volume of the audio data recorded in the video data with audio.

Next, in response to the user pressing the LV button 76, the control unit 111 transmits an instruction signal to start shooting to each component of the imaging device 100. For example, the control unit 111 controls the imaging unit 101, the audio input unit 104, the encoding processing unit 108, and the recording control unit 109 to perform the following operations.

The imaging unit 101 converts the optical image of the subject formed on the imaging surface via the optical lens 300 into an electrical signal, and generates video data from the electrical signal generated by the imaging element. The imaging unit 101 then transmits the video data to the display control unit 106, which displays the video data on the display unit 107. The imaging unit 101 also transmits the generated video data to the volatile memory 105.

The audio input unit 104 performs A/D conversion on the analog audio signals input from the multiple microphones, and generates multiple digital audio signals. The audio input unit 104 then generates multi-channel audio data from the multiple digital audio signals. The audio input unit 104 then transmits the generated audio data to the volatile memory 105.

The encoding processing unit 108 reads out the video data and audio data temporarily recorded in the volatile memory 105 and encodes each of them. The control unit 111 generates a data stream from the video data and audio data encoded by the encoding processing unit 108 and outputs it to the recording control unit 109. The recording control unit 109 records the input data stream as video data with audio on the recording medium 110 according to a file system such as UDF or FAT.

Each component of the imaging device 100 continues to perform the above operations while shooting video.

Then, in response to the user pressing the LV button 76, the control unit 111 transmits an instruction signal to end shooting to each component of the imaging device 100. For example, the control unit 111 controls the imaging unit 101, the audio input unit 104, the encoding processing unit 108, and the recording control unit 109 to perform the following operations.

The imaging unit 101 stops generating video data. The audio input unit 104 stops generating audio data.

The encoding processing unit 108 reads out and encodes the remaining video data and audio data recorded in the volatile memory 105. The control unit 111 generates a data stream from the video data and audio data encoded by the encoding processing unit 108 and outputs it to the recording control unit 109.

The recording control unit 109 records the data stream as a file of video data with audio on the recording medium 110 according to a file system such as UDF or FAT. Then, when the input of the data stream stops, the recording control unit 109 completes the video data with audio. When the video data with audio is completed, the recording operation of the imaging device 100 stops.

In response to the recording operation being stopped, the control unit 111 transmits a control signal to each component of the imaging device 100 to transition to a shooting standby state. As a result, the control unit 111 controls the imaging device 100 to return to the shooting standby state.

Next, the playback mode will be described. In the playback mode, the control unit 111 transmits a control signal to each component of the imaging device 100 to transition to a playback state. For example, the control unit 111 controls the encoding processing unit 108, the recording control unit 109, the display control unit 106, and the audio output unit 113 to perform the following operations.

The recording control unit 109 reads the video data with audio recorded on the recording medium 110 and transmits the read video data with audio to the encoding processing unit 108.

The encoding processing unit 108 decodes the video data with audio into image data and audio data. The encoding processing unit 108 transmits the decoded video data to the display control unit 106 and the decoded audio data to the audio output unit 113.

The display control unit 106 displays the decoded image data on the display unit 107. The audio output unit 113 plays the decoded audio data through the speaker 114.

As described above, the imaging device 100 of this embodiment can record and play back image data and audio data.

In this embodiment, the audio input unit 104 performs audio processing such as adjusting the level of the audio signal input from the microphone. In this embodiment, the audio input unit 104 performs this audio processing in response to the start of video recording. Note that this audio processing may be performed after the power of the imaging device 100 is turned on. Also, this audio processing may be performed in response to the selection of a shooting mode. Also, this audio processing may be performed in response to the selection of a mode related to audio recording, such as a video recording mode or an audio memo function. Also, this audio processing may be performed in response to the start of recording of the audio signal.

<Configuration of voice input unit 104>
FIG. 3 is a block diagram showing an example of a detailed configuration of the voice input unit 104 in this embodiment.

In this embodiment, the audio input unit 104 has three microphones: an L microphone 201a, an R microphone 201b, and a noise microphone 201c. The L microphone 201a and the R microphone 201b are each an example of a first microphone. In this embodiment, the imaging device 100 picks up environmental sounds using the L microphone 201a and the R microphone 201b, and records the audio signals input from the L microphone 201a and the R microphone 201b in a stereo format. For example, environmental sounds are sounds generated outside the housing of the imaging device 100 and outside the housing of the optical lens 300, such as the user's voice, animal cries, the sound of rain, and music.

The noise microphone 201c is an example of a second microphone. The noise microphone 201c is a microphone for acquiring noise (noise), such as a driving sound from a predetermined noise source (noise source) generated within the housing of the imaging device 100 and the housing of the optical lens 300. The noise source is, for example, a motor such as an ultrasonic motor (USM) and a stepper motor (STM). The noise (noise) is, for example, a vibration sound generated by driving a motor such as a USM and an STM. For example, the motor is driven in an AF process for focusing on a subject. The imaging device 100 acquires noise (noise), such as a driving sound generated within the housing of the imaging device 100 and the housing of the optical lens 300, using the noise microphone 201c, and generates a noise parameter to be described later using the audio data of the acquired noise. In this embodiment, the L microphone 201a, the R microphone 201b, and the noise microphone 201c are non-directional microphones. An example of the arrangement of the L microphone 201a, the R microphone 201b, and the noise microphone 201c in this embodiment will be described later with reference to FIG. 4.

The L microphone 201a, the R microphone 201b, and the noise microphone 201c each generate an analog audio signal from the captured sound and input it to the A/D conversion unit 202. Here, the audio signal input from the L microphone 201a is referred to as Lch, the audio signal input from the R microphone 201b as Rch, and the audio signal input from the noise microphone 201c as Nch.

The A/D conversion unit 202 converts the analog audio signals input from the L microphone 201a, the R microphone 201b, and the noise microphone 201c into digital audio signals. The A/D conversion unit 202 outputs the converted digital audio signals to the FFT unit 203. In this embodiment, the A/D conversion unit 202 converts the analog audio signals into digital audio signals by performing sampling processing with a sampling frequency of 48 kHz and a bit depth of 16 bits.

The FFT unit 203 performs fast Fourier transform processing on the time-domain digital audio signal input from the A/D conversion unit 202, converting it into a frequency-domain digital audio signal. In this embodiment, the frequency-domain digital audio signal has a frequency spectrum of 1024 points in the frequency band from 0 Hz to 48 kHz. Also, the frequency-domain digital audio signal has a frequency spectrum of 513 points in the frequency band from 0 Hz to 24 kHz, which is the Nyquist frequency. In this embodiment, the imaging device 100 performs noise reduction processing using the 513-point frequency spectrum from 0 Hz to 24 kHz of the audio data output from the FFT unit 203.

Here, the frequency spectrum of the Lch after the fast Fourier transform is represented by array data of 513 points from Lch_Before[0] to Lch_Before[512]. When these array data are referred to collectively, they are written as Lch_Before. Moreover, the frequency spectrum of the Rch after the fast Fourier transform is represented by array data of 513 points from Rch_Before[0] to Rch_Before[512]. When these array data are referred to collectively, they are written as Rch_Before. Moreover, Lch_Before and Rch_Before are each an example of the first frequency spectrum data.

The frequency spectrum of Nch after fast Fourier transformation is represented by array data of 513 points, Nch_Before[0] to Nch_Before[512]. When these array data are collectively referred to, they are written as Nch_Before. Note that Nch_Before is an example of second frequency spectrum data.

The noise data generating unit 204 generates data for reducing noise contained in Lch_Before and Rch_Before based on Nch_Before. In this embodiment, the noise data generating unit 204 generates array data NL[0] to NL[512] for reducing noise contained in Lch_Before[0] to Lch_Before[512] using noise parameters. The noise data generating unit 204 also generates array data NR[0] to NR[512] for reducing noise contained in Rch_Before[0] to Rch_Before[512]. The frequency points in the array data NL[0] to NL[512] are the same as the frequency points in the array data Lch_Before[0] to Lch_Before[512]. Additionally, the frequency points in the array data NR[0] to NR[512] are the same as the frequency points in the array data Rch_Before[0] to Rch_Before[512].

Note that the array data NL[0] to NL[512] are collectively referred to as NL. Furthermore, the array data NR[0] to NR[512] are collectively referred to as NR. NL and NR are each an example of the third frequency spectrum data.

The noise parameter recording unit 205 records noise parameters that the noise data generating unit 204 uses to generate NL and NR from Nch_Before. The noise parameter recording unit 205 records multiple types of noise parameters according to the type of noise. The noise parameters for generating NL from Nch_Before are collectively referred to as PLx. The noise parameters for generating NR from Nch_Before are collectively referred to as PRx.

PLx and PRx have the same number of arrays as NL and NR, respectively. For example, PL1 is array data from PL1[0] to PL1[512]. The frequency points of PL1 are the same as the frequency points of Lch_Before. For example, PR1 is array data from PR1[0] to PR1[512]. The frequency points of PR1 are the same as the frequency points of Rch_Before. The noise parameters will be described later with reference to Figure 5.

The noise parameter selection unit 206 determines the noise parameters to be used in the noise data generation unit 204 from the noise parameters recorded in the noise parameter recording unit 205. The noise parameter selection unit 206 determines the noise parameters to be used in the noise data generation unit 204 based on Lch_Before, Rch_Before, Nch_Before, and data received from the lens control unit 102. The operation of the noise parameter selection unit 206 will be described in detail later with reference to FIG. 8.

In this embodiment, the noise parameter recording unit 205 records all the coefficients for each of the 513 frequency spectrum points as noise parameters. However, the noise parameter recording unit 205 only needs to record the coefficients for at least the frequency points necessary to reduce noise, not the coefficients for all of the 513 frequency points. For example, the noise parameter recording unit 205 may record, as noise parameters, coefficients for each frequency spectrum of 20 Hz to 20 kHz, which are considered to be typical audible frequencies, and may not record coefficients for other frequency spectra. Also, for example, coefficients for frequency spectra whose coefficient value is zero may not be recorded in the noise parameter recording unit 205 as noise parameters.

The subtraction processing unit 207 subtracts NL and NR from Lch_Before and Rch_Before, respectively. For example, the subtraction processing unit 207 has an L subtractor 207a that subtracts NL from Lch_Before, and an R subtractor 207b that subtracts NR from Rch_Before. The L subtractor 207a subtracts NL from Lch_Before, and outputs array data of 513 points from Lch_After[0] to Lch_After[512]. The R subtractor 207b subtracts NR from Rch_Before, and outputs array data of 513 points from Rch_After[0] to Rch_After[512]. In this embodiment, the subtraction processing unit 207 performs subtraction processing using the spectral subtraction method.

The iFFT unit 208 performs an inverse fast Fourier transform (inverse Fourier transform) on the frequency domain digital audio signal input from the subtraction processing unit 207 to convert it into a time domain digital audio signal.

The audio processing unit 209 performs audio processing on the time domain digital audio signal, such as an equalizer, an auto level controller, and stereo enhancement processing. The audio processing unit 209 outputs the processed audio data to the volatile memory 105.

In this embodiment, the imaging device 100 has two microphones as the first microphone, but the imaging device 100 may have one microphone or three or more microphones as the first microphone. For example, when the imaging device 100 has one microphone as the first microphone in the audio input unit 104, the imaging device 100 records the audio data picked up by the one microphone in a monaural format. Also, when the imaging device 100 has three or more microphones as the first microphone in the audio input unit 104, the imaging device 100 records the audio data picked up by the three or more microphones in a surround format.

In this embodiment, the L microphone 201a, the R microphone 201b, and the noise microphone 201c are non-directional microphones, but these microphones may also be directional microphones.

<Arrangement of microphones in audio input unit 104>
Here, an example of the arrangement of the microphones in the voice input unit 104 of this embodiment will be described. Fig. 4 shows an example of the arrangement of the L microphone 201a, the R microphone 201b, and the noise microphone 201c.

Figure 4 is an example of a cross-sectional view of a portion of the imaging device 100 to which the L microphone 201a, the R microphone 201b, and the noise microphone 201c are attached. This portion of the imaging device 100 is composed of an exterior part 302, a microphone bushing 303, and a fixed part 304.

The exterior part 302 has a hole (hereinafter referred to as a microphone hole) for inputting environmental sound to the microphone. In this embodiment, the microphone hole is formed above the L microphone 201a and the R microphone 201b. On the other hand, the noise microphone 201c is provided to acquire drive sounds generated within the housing of the imaging device 100 and within the housing of the optical lens 300, and does not need to acquire environmental sound. Therefore, in this embodiment, no microphone hole is formed in the exterior part 302 above the noise microphone 201c.

Drive sounds generated within the housings of the image capture device 100 and the optical lens 300 are picked up by the L microphone 201a and the R microphone 201b through the microphone holes. When drive sounds or the like are generated within the housings of the image capture device 100 and the optical lens 300 while the ambient sound is low, the sound picked up by each microphone is mainly this drive sound. Therefore, the sound level from the noise microphone 201c is higher than the sound levels from the L microphone 201a and the R microphone 201b. In other words, in this case, the relationship between the levels of the sound signals output from each microphone is as follows:
Lch≒Rch<Nch
Furthermore, when the environmental sound becomes louder, the sound level of the environmental sound from the L microphone 201a and the R microphone 201b becomes louder than the sound level of the drive sound from the microphone 201c generated by the image capture device 100 or the optical lens 300. Therefore, in this case, the relationship between the levels of the audio signals output from each microphone is as follows:
Lch≒Rch>Nch
In this embodiment, the shape of the microphone hole formed in the exterior part 302 is elliptical, but it may be other shapes such as circular or rectangular. Furthermore, the shape of the microphone hole on the microphone 201a and the shape of the microphone hole on the microphone 201b may be different from each other.

In this embodiment, the noise microphone 201c is positioned close to the L microphone 201a and the R microphone 201b. Also, in this embodiment, the noise microphone 201c is positioned between the L microphone 201a and the R microphone 201b. As a result, the audio signal generated by the noise microphone 201c from the drive sounds etc. generated within the housing of the imaging device 100 and the housing of the optical lens 300 becomes a signal similar to the audio signal generated by the L microphone 201a and the R microphone 201b from this drive sound etc.

The microphone bushing 303 is a member for fixing the L microphone 201a, the R microphone 201b, and the noise microphone 201c. The fixing part 304 is a member for fixing the microphone bushing 303 to the exterior part 302.

In this embodiment, the exterior part 302 and the fixed part 304 are made of a molded material such as PC material. The exterior part 302 and the fixed part 304 may also be made of a metal material such as aluminum or stainless steel. In this embodiment, the microphone bushing 303 is made of a rubber material such as ethylene propylene diene rubber.

<Noise parameters>
5 shows an example of noise parameters recorded in the noise parameter recording unit 205. The noise parameters are parameters for correcting an audio signal generated by the noise microphone 201c acquiring drive sounds generated in the housing of the image capture device 100 and the housing of the optical lens 300. As shown in FIG. 5, in this embodiment, PLx and PRx are recorded in the noise parameter recording unit 205. In this embodiment, the source of the drive sounds will be described as being inside the housing of the optical lens 300. The drive sounds generated inside the housing of the optical lens 300 are transmitted to the inside of the housing of the image capture device 100 via the lens mount 301, and are acquired by the L microphone 201a, the R microphone 201b, and the noise microphone 201c.

The frequency of the drive sound varies depending on the type of drive sound. Therefore, in this embodiment, the imaging device 100 records multiple noise parameters corresponding to the type of drive sound (noise). Then, noise data is generated using one of these multiple noise parameters. In this embodiment, the imaging device 100 records noise parameters for white noise as constant noise. The imaging device 100 also records noise parameters for short-term noise generated, for example, by the meshing of gears in the optical lens 300. The imaging device 100 also records noise parameters for long-term noise, for example, sliding noise within the housing of the lens 300. In addition, the imaging device 100 may record noise parameters for each type of optical lens 300, and for each temperature within the housing of the imaging device 100 and tilt of the imaging device 100 detected by the information acquisition unit 103.

<Method of generating noise data>
6 and 7, the process of generating noise data in the noise data generator 204 will be described. Here, the process of generating noise data relating to the Lch data will be described, but the method of generating noise data relating to the Rch data is similar.

First, the process of generating noise parameters in a situation where it is considered that there is no environmental sound will be described. Fig. 6(a) is an example of the frequency spectrum of Lch_Before when drive sound occurs within the housing of the optical lens 300 in a situation where it is considered that there is no environmental sound. Fig. 6(b) is an example of the frequency spectrum of Nch_Before when drive sound occurs within the housing of the optical lens 300 in a situation where it is considered that there is no environmental sound. The horizontal axis indicates the frequency from the 0th point to the 512th point, and the vertical axis indicates the amplitude of the frequency spectrum.

Because it is assumed that there is no environmental sound, the amplitude of the frequency spectrum in the same frequency band is large in Lch_Before and Nch_Before. In addition, because drive sound is generated inside the housing of the optical lens 300, the amplitude of each frequency spectrum for the same drive sound tends to be larger in Nch_Before than in Lch_Before.

Figure 6 (c) is an example of PLx in this embodiment. In this embodiment, PLx is the coefficient of each frequency spectrum calculated by dividing the amplitude of each frequency spectrum of Lch_Before by the amplitude of each frequency spectrum of Nch_Before. The result of this division is written as Lch_Before/Nch_Before. In other words, PLx is the ratio of the amplitudes of Lch_Before and Nch_Before. The noise parameter recording unit 205 records the value of Lch_Before/Nch_Before as the noise parameter PLx. As mentioned above, the amplitude of the frequency spectrum for the same driving sound tends to be larger in Nch_Before than in Lch_Before, so the value of each coefficient of the noise parameter PLx tends to be smaller than 1. However, if the value of Nch_Before[n] is smaller than a predetermined threshold, the noise parameter recording unit 205 records the noise parameter PLx as PLx[n] = 0.

Next, the process of applying the generated noise parameters to Nch_Before will be described. FIG. 7(a) is an example of the frequency spectrum of Lch_Before when drive sound occurs within the housing of the optical lens 300 in a situation where environmental sound is present. FIG. 7(b) is an example of the frequency spectrum of Nch_Before when drive sound occurs within the housing of the optical lens 300 in a situation where environmental sound is present. The horizontal axis indicates the frequency from the 0th point to the 512th point, and the vertical axis indicates the amplitude of the frequency spectrum.

Figure 7 (c) is an example of NL when drive sound occurs inside the housing of the optical lens 300 in a situation where environmental sound is present. The noise data generator 204 multiplies each frequency spectrum of Nch_Before by each coefficient of PLx to generate NL. NL is the frequency spectrum generated in this way.

Figure 7 (d) is an example of Lch_After when drive sound occurs inside the housing of the optical lens 300 in a situation where environmental sound is present. The subtraction processing unit 207 subtracts NL from Lch_Before to generate Lch_After. Lch_After is the frequency spectrum generated in this way.

This allows the imaging device 100 to reduce noise caused by the driving sound inside the housing of the optical lens 300 and record environmental sounds with less noise.

<Description of Noise Parameter Selection Unit 206>
FIG. 8 is a block diagram showing an example of a detailed configuration of the noise parameter selection unit 206. As shown in FIG.

Lch_Before, Rch_Before, Nch_Before, and the lens control signal are input to the noise parameter selection unit 206.

The Nch noise detection unit 2061 detects noise due to drive sound generated within the housing of the optical lens 300 from Nch_Before. Based on the noise detection result, the Nch noise detection unit 2061 outputs data on the noise detection result to the noise determination unit 2063. Note that in this embodiment, the Nch noise detection unit 2061 detects noise using data from 513 points in Nch_Before.

The environmental sound detection unit 2062 detects the level of the environmental sound from Lch_Before and Rch_Before. Based on the detection result of the level of the environmental sound, the environmental sound detection unit 2062 outputs data regarding the detection result of the level of the environmental sound to the noise determination unit 2063.

The noise determination unit 2063 determines the noise parameters to be used by the noise data generation unit 204 based on the lens control signal input from the lens control unit 102, the data input from the Nch noise detection unit 2061, and the data input from the environmental sound detection unit 2062. The noise determination unit 2063 outputs data indicating the type of the determined noise parameter to the noise data generation unit 204.

The Nch differentiation unit 2064 performs differentiation processing on Nch_Before. The Nch differentiation unit 2064 outputs data indicating the result of differentiation processing on Nch_Before to the short-term noise detection unit 2065. The short-term noise detection unit 2065 detects whether or not short-term noise is occurring based on the data input from the Nch differentiation unit 2064. The short-term noise detection unit 2065 outputs data indicating whether or not short-term noise is occurring to the noise determination unit 2063. Note that the Nch differentiation unit 2064 and the short-term noise detection unit 2065 are included in the Nch noise detection unit 2061.

The Nch integrator 2066 performs integration processing on Nch_Before. The Nch integrator 2066 outputs data indicating the result of differentiation processing of Nch_Before to the long-term noise detector 2067. The long-term noise detector 2067 detects whether or not long-term noise is occurring based on the data input from the Nch integrator 2066. The long-term noise detector 2067 outputs data indicating whether or not long-term noise is occurring to the noise determination unit 2063. Note that the Nch integrator 2066 and the long-term noise detector 2067 are included in the Nch noise detector 2061.

The environmental sound extraction unit 2068 extracts environmental sound. In this embodiment, the environmental sound extraction unit 2068 extracts data of frequencies that are less affected by noise based on a noise parameter. For example, the environmental sound extraction unit 2068 extracts data of frequencies where the noise parameter is equal to or less than a predetermined value. Then, the environmental sound extraction unit 2068 outputs data indicating the volume of the environmental sound based on the extracted frequency data. Note that the environmental sound extraction unit 2068 is included in the environmental sound detection unit 2062.

The environmental sound determination unit 2069 determines the volume of the environmental sound. The environmental sound determination unit 2069 inputs data indicating the determined volume of the environmental sound to the Nch noise detection unit 2061 and the noise determination unit 2063. The Nch noise detection unit 2061 changes a first threshold value and a second threshold value, which will be described later, based on the data indicating the volume of the environmental sound input from the environmental sound determination unit 2069. Note that the environmental sound determination unit 2069 is included in the environmental sound detection unit 2062.

<Timing chart of noise reduction processing>
The noise reduction process in this embodiment will be described with reference to FIG.

Figures 9(a) to (i) are example timing charts of audio processing in the noise data generation unit 204, the noise parameter selection unit 206, and the subtraction processing unit 207. In this embodiment, for simplicity, audio processing of the Lch will be described, but audio processing of the Rch is similar. The horizontal axis of the graphs in Figures 9(a) to (i) is the time axis.

Figure 9 (a) shows an example of a lens control signal. The lens control signal is a signal that the lens control unit 102 uses to instruct the optical lens 300 to drive. In this embodiment, the level of the lens control signal is expressed as two values, High and Low. When the level of the lens control signal is High, the lens control unit 102 is instructing the optical lens 300 to drive. When the level of the lens control signal is Low, the lens control unit 102 is not instructing the optical lens 300 to drive.

Figure 9 (b) is a graph showing an example of the value of Lch_Before[n]. The vertical axis indicates the value of Lch_Before[n]. In this embodiment, Lch_Before[n] is the signal at the nth frequency point, among the Lch_Before output from the FFT unit 203, at which a signal indicating the driving sound of the optical lens 300 appears characteristically. Note that in this embodiment, the signal at the nth frequency point is described, but audio processing is performed similarly for other frequencies. Furthermore, the signals indicated by signal X and signal Y are signals containing noise. In this embodiment, signal X indicates a signal containing short-term noise. Signal Y indicates a noise signal containing long-term noise.

Figure 9 (c) is a graph showing an example of the volume of the environmental sound extracted by the environmental sound extraction unit 2068. The vertical axis indicates the level of the audio signal generated from the acquired environmental sound. Threshold value Th1 and threshold value Th2 are two threshold values used in the environmental sound determination unit 2069.

Figure 9 (d) is a graph showing an example of the value of Nch_Before[n]. Nch_Before[n] is the signal at the nth frequency point, among the Nch_Before output from the FFT unit 203, at which the signal indicating the driving sound of the optical lens 300 appears characteristically. The vertical axis indicates the value of Nch_Before[n]. In Nch_Before[n], the noise signals indicated by signals X and Y in Figure 9 (b) appear more characteristically than in Lch_Before.

9(e) is a graph showing an example of the value of Ndiff[n]. Ndiff[n] indicates the value of the signal at the nth frequency point of Ndiff output from the Nch differentiation unit 2064. The vertical axis indicates the value of Ndiff[n]. If the amount of change in the value of Nch_Before[n] per given time is large, the value of Ndiff[n] becomes large. The short-term noise detection unit 2065 has a first threshold value Th_Ndiff[n] to detect short-term noise. The threshold value Th_Ndiff[n] changes between levels 1 to 3 based on the data indicating the volume of the environmental sound input from the environmental sound determination unit 2069 and the lens control signal. The initial value of the threshold value Th_Ndiff[n] is set to level 2. The level of the threshold value Th_Ndiff[n] is indicated by a horizontal dashed line.

Figure 9(f) is a graph showing an example of the value of Nint[n]. In this embodiment, Nint[n] indicates the value of the signal at the nth frequency point of Nint output from the Nch integration unit 2066. The vertical axis indicates the value of Nint[n]. If Nch_Before[n] is continuously large, the value of Nint[n] increases. The long-term noise detection unit 2067 has a second threshold, Th_Nint[n], to detect long-term noise. The threshold Th_Nint[n] changes between levels 1 to 3 based on the data indicating the volume of the environmental sound input from the environmental sound determination unit 2069 and the lens control signal. The initial value of the threshold Th_Nint[n] is level 2. The level of the threshold Th_Nint[n] is indicated by a horizontal dashed line.

Figure 9(g) shows an example of noise parameters selected by the noise parameter selection unit 206. In this embodiment, the solid part indicates that only the noise parameter of PL1 has been selected. The diagonal line part indicates that the noise parameters of PL1 and PL2 have been selected. The checkered part indicates that the noise parameters of PL1 and PL3 have been selected.

Figure 9 (h) is a graph showing an example of the value of NL[n]. In this embodiment, NL[n] indicates the signal value of the nth frequency point of NL generated by the noise data generating unit 204. The vertical axis indicates the value of NL[n].

Figure 9(i) is a graph showing an example of the value of Lch_After[n]. In this embodiment, Lch_After[n] indicates the value of the signal at the nth frequency point of Lch_After output from the subtraction processing unit 207. The vertical axis indicates the value of Lch_After[n].

Next, the timing of each operation will be explained using times t701 to t709.

At time t701, the lens control unit 102 outputs a High signal as a lens control signal to the optical lens 300 and the noise parameter selection unit 206 (FIG. 9(a)). At time t701, since there is a high possibility that drive noise will occur within the housing of the optical lens 300, the short-term noise detection unit 2065 lowers the threshold Th_Ndiff[n] to level 1 (FIG. 9(e)). Also at time t701, since there is a high possibility that drive noise will occur within the housing of the optical lens 300, the long-term noise detection unit 2067 lowers the threshold Th_Nint[n] to level 1 (FIG. 9(f)).

At time t702, the optical lens 300 is driven, generating short-term drive noise such as the sound of gears meshing. The noise microphone 201c picks up the short-term drive noise, causing the value of Ndiff[n] to exceed the threshold value Th_Ndiff[n] (FIG. 9(e)). In response to this, the noise parameter selection unit 206 selects the noise parameters PL1 and PL2 (FIG. 9(g)). The noise data generation unit 204 generates NL[n] based on Nch_Before[n] and the noise parameters PL1 and PL2 (FIG. 9(h)). The subtraction processing unit 207 subtracts NL[n] from Lch_Before[n] and outputs Lch_After[n] (FIG. 9(i)). In this case, Lch_After[n] becomes an audio signal in which constant noise and short-term noise have been reduced.

At time t703, the optical lens 300 starts to drive continuously, and long-term drive noise such as sliding noise occurs within the housing of the optical lens 300. The noise microphone 201c picks up the long-term drive noise, and the value of Nint[n] exceeds the threshold value Th_Nint[n] (FIG. 9(f)). In response to this, the noise parameter selection unit 206 selects the noise parameters PL1 and PL3 (FIG. 9(g)). The noise data generation unit 204 generates NL[n] based on Nch_Before[n] and the noise parameters PL1 and PL3 (FIG. 9(h)). The subtraction processing unit 207 subtracts NL[n] from Lch_Before[n] and outputs Lch_After[n] (FIG. 9(i)). In this case, Lch_After[n] becomes an audio signal in which constant noise and long-term noise have been reduced.

At time t704, the optical lens 300 stops continuous driving. Because the noise microphone 201c no longer picks up the long-term driving sound, the value of Nint[n] becomes equal to or less than the threshold value Th_Nint[n] (FIG. 9(f)). In response to this, the noise parameter selection unit 206 selects the noise parameter PL1 (FIG. 9(g)). The noise data generation unit 204 generates NL[n] based on Nch_Before[n] and the noise parameter PL1 (FIG. 9(h)). The subtraction processing unit 207 subtracts NL[n] from Lch_Before[n] and outputs Lch_After[n] (FIG. 9(i)). In this case, Lch_After[n] becomes an audio signal with constant noise reduced.

At time t705, the lens control unit 102 outputs a Low signal as a lens control signal to the optical lens 300 and the noise parameter selection unit 206 (FIG. 9(a)). In this case, the possibility of drive noise occurring within the housing of the optical lens 300 is low, so the short-term noise detection unit 2065 increases the threshold Th_Ndiff[n] to level 2 (FIG. 9(e)). In addition, in this case, the possibility of drive noise occurring within the housing of the optical lens 300 is low, so the long-term noise detection unit 2067 increases the threshold Th_Nint[n] to level 2 (FIG. 9(f)).

At time t706, the volume of the environmental sound extracted by the environmental sound extraction unit 2068 exceeds the threshold Th1. When the environmental sound is loud, the user is less likely to perceive the noise contained in the audio signal, so the short-term noise detection unit 2065 raises the threshold Th_Ndiff[n] to level 3 (Figure 9(e)). Also, when the environmental sound is loud, the user is less likely to perceive the noise contained in the audio signal, so the long-term noise detection unit 2067 raises the threshold Th_Nint[n] to level 3 (Figure 9(f)).

At time t707, the lens control unit 102 outputs a High signal as a lens control signal to the optical lens 300 and the noise parameter selection unit 206 (FIG. 9(a)). In this case, since there is a high possibility that drive noise will occur within the housing of the optical lens 300, the short-term noise detection unit 2065 lowers the threshold Th_Ndiff[n] to level 2 (FIG. 9(e)). In addition, since there is a high possibility that drive noise will occur within the housing of the optical lens 300, the long-term noise detection unit 2067 lowers the threshold Th_Nint[n] to level 2 (FIG. 9(f)).

At time t708, the volume of the environmental sound extracted by the environmental sound extraction unit 2068 exceeds the threshold value Th2. Here, the noise parameter selection unit 206 selects only the noise parameter PL1 regardless of the data input from the Nch noise detection unit 2061. In this way, when the environmental sound is very loud, the noise contained in the audio signal is difficult for the user to perceive, so by reducing only the constant noise, the imaging device 100 records environmental sound that is more natural than when processing is performed to further reduce short-term noise and long-term noise.

As described above, the imaging device 100 can record environmental sounds with reduced noise by performing noise reduction processing using the second microphone, the noise microphone 201c.

The imaging device 100 also detects the occurrence of noise using the output signal of the noise microphone 201c, and sets the noise parameters in accordance with the timing at which the occurrence of the noise is detected. Therefore, the imaging device 100 can appropriately set the noise parameters in synchronization with the occurrence of noise, and appropriately reduce the noise.

In addition, when the volume of the environmental sound is equal to or less than the threshold Th2, the imaging device 100 executes noise reduction processing according to the noise detected by the Nch noise detection unit 2061, and when the environmental sound is greater than the threshold Th2, it reduces only constant noise. This enables the imaging device 100 to record environmental sound with reduced noise according to the volume of the environmental sound so as to minimize discomfort for the user.

In this embodiment, the imaging device 100 reduces the drive noise generated within the housing of the optical lens 300, but the drive noise generated within the imaging device 100 may also be reduced. The drive noise generated within the imaging device 100 is, for example, the noise of the board and radio wave noise. The noise of the board is, for example, the sound generated by the creaking of the board when a voltage is applied to a capacitor on the board.

The thresholds Th1 and Th2 of the environmental sound determination unit 2069, the threshold Th_Ndiff[n] of the short-term noise detection unit 2065, and the threshold Th_Nint[n] of the long-term noise detection unit 2067 are determined based on the generated drive sound and environmental sound. Therefore, the imaging device 100 may change these thresholds depending on the type of the optical lens 300, the tilt of the imaging device 100, etc.

[Second embodiment]
Fig. 10 is a block diagram showing an example of the configuration of the voice input unit 104 in the second embodiment. The parts different from the configuration of the voice input unit 104 shown in Fig. 3 are a subtraction processing unit 207 and an iFFT unit 208. Here, a description of the processing units similar to those in Fig. 3 will be omitted.

The iFFT unit 208a performs an inverse fast Fourier transform on Lch_Before and Rch_Before input from the FFT unit 203, respectively, to convert the frequency domain digital audio signals into time domain digital audio signals. The iFFT unit 208b also performs an inverse fast Fourier transform on NL and NR, respectively, to convert the frequency domain digital audio signals into time domain digital audio signals.

The subtraction processing unit 207 subtracts the digital audio signal input from the iFFT unit 208b from the digital audio signal input from the iFFT unit 208a. The calculation process in the subtraction processing unit 207 is a waveform subtraction method that subtracts the digital audio signal in the time domain.

When performing waveform subtraction, the imaging device 100 may also record parameters related to the phase of the digital audio signal as noise parameters.

The rest of the configuration and operation of the imaging device 100 is the same as in the first embodiment.

[Third embodiment]
In the third embodiment, a configuration in which the image pickup apparatus 100 has two subtraction processing units will be described.

Figure 11 is a block diagram showing an example of the configuration of the voice input unit 104 in the third embodiment.

Here, the microphone 201, A/D conversion unit 202, FFT unit 203, iFFT unit 208, and audio processing unit 209 shown in FIG. 11 are the same as those shown in FIG. 3, so their explanations are omitted.

The switching unit 210 switches paths based on control information from the lens control unit 102. In this embodiment, when the optical lens 300 is driven, the switching unit 210 switches paths so that noise reduction processing is performed by the calculation processing unit A217, which will be described later. Also, when the optical lens 300 is not driven, the switching unit 210 switches paths so that noise reduction processing is not performed by the calculation processing unit A217.

The noise data generating unit A214 generates data for reducing noise related to lens drive contained in Lch_Before and Rch_Before based on Nch_Before. The noise related to lens drive contained in the audio signal input from the L microphone and the R microphone is an example of a first noise. In this embodiment, the noise data generating unit A214 uses noise parameters to generate array data of NLA[0] to NLA[512] for reducing noise contained in Lch_Before[0] to Lch_Before[512], respectively. The noise data generating unit A214 also generates array data of NRA[0] to NRA[512] for reducing noise contained in Rch_Before[0] to Rch_Before[512], respectively.

The frequency points in the array data of NLA[0] to NLA[512] are the same as the frequency points in the array data of Lch_Before[0] to Lch_Before[512]. Also, the frequency points in the array data of NRA[0] to NRA[512] are the same as the frequency points in the array data of Rch_Before[0] to Rch_Before[512].

Note that the array data NLA[0] to NLA[512] are collectively referred to as NLA. Furthermore, NRA[0] to NRA[512] are collectively referred to as NRA. NLA and NRA are each an example of the third frequency spectrum data.

The noise parameter recording unit 205 records noise parameters for the noise data generating unit A214 to generate NLA and NRA from Nch_Before. Note that in this embodiment, the noise parameter recording unit 205 records noise parameters related to lens drive for each lens type, which are noise parameters used by the noise data generating unit A214. Note that in this embodiment, the noise data generating unit A214 does not switch noise parameters while recording audio data.

Here, the noise parameters for generating NLA from Nch_Before are collectively referred to as PLxA. The noise parameters for generating NRA from Nch_Before are collectively referred to as PRxA.

PLxA and PRxA have the same number of arrays as NLA and NRA, respectively. For example, PL1A is array data from PL1A[0] to PL1A[512]. The frequency points of PL1A are the same as the frequency points of Lch_Before. For example, PR1A is array data from PR1A[0] to PR1A[512]. The frequency points of PR1A are the same as the frequency points of Rch_Before. The noise parameters will be described later with reference to FIG. 12.

In this embodiment, the noise parameter recording unit 205 records all the coefficients for each of the 513 frequency spectrum points as noise parameters. However, the noise parameter recording unit 205 only needs to record the coefficients for at least the frequency points necessary to reduce noise, not the coefficients for all of the 513 frequency points. For example, the noise parameter recording unit 205 may record, as noise parameters, coefficients for each frequency spectrum of 20 Hz to 20 kHz, which are considered to be typical audible frequencies, and may not record coefficients for other frequency spectra. Also, for example, coefficients for frequency spectra whose coefficient value is zero may not be recorded in the noise parameter recording unit 205 as noise parameters.

The subtraction processing unit A217 subtracts the NLA and the NRA from Lch_Before and Rch_Before, respectively. For example, the subtraction processing unit A217 has an L subtractor A217a that subtracts the NLA from Lch_Before, and an R subtractor A217b that subtracts the NRA from Rch_Before. The L subtractor A217a subtracts the NLA from Lch_Before, and outputs array data of 513 points from Lch_A_After[0] to Lch_A_After[512]. The R subtractor A217b subtracts the NRA from Rch_Before, and outputs array data of 513 points from Rch_A_After[0] to Rch_A_After[512]. In this embodiment, the subtraction processing unit A217 performs subtraction processing using the spectral subtraction method, and subtracts noise related to lens driving, which is particularly noisy.

The noise data generating unit B224 generates data for reducing the second noise related to lens driving contained in Lch_A_After and Rch_A_After based on Nch_Before.

In this embodiment, the noise data generation unit B224 uses noise parameters to generate array data for NLB[0] to NLB[512] for reducing the noise contained in Lch_A_After[0] to Lch_A_After[512], respectively. The noise data generation unit B224 also uses noise parameters to generate array data for NRB[0] to NRB[512] for reducing the noise contained in Rch_A_After[0] to Rch_A_After[512], respectively.

The frequency points in the array data of NLB[0] to NLB[512] are the same as the frequency points in the array data of Lch_A_After[0] to Lch_A_After[512]. Also, the frequency points in the array data of NRB[0] to NRB[512] are the same as the frequency points in the array data of Rch_A_After[0] to Rch_A_After[512].

Note that the array data NLB[0] to NLB[512] are collectively referred to as NLB. Furthermore, the array data NRB[0] to NRB[512] are collectively referred to as NRB. NLB and NRB are each an example of (fourth frequency spectrum data).

The noise parameter recording unit 205 records noise parameters that the noise data generating unit B224 uses to generate NLB and NRB from Nch_Before.

In this embodiment, the noise parameter recording unit 205 records noise parameters used in the noise data generating unit B224, such as noise parameters for microphone floor noise and electrical noise. In this embodiment, the noise data generating unit A214 does not switch noise parameters while recording audio data.

Here, the noise parameters for generating NLB from Nch_Before are collectively referred to as PLxB. The noise parameters for generating NRB from Nch_Before are collectively referred to as PRxB.

PLxB and PRxB have the same number of arrays as NLB and NRB, respectively. For example, PL1B is array data from PL1B[0] to PL1B[512]. The frequency points of PL1B are the same as the frequency points of Lch_Before. For example, PR1B is array data from PR1B[0] to PR1B[512]. The frequency points of PR1B are the same as the frequency points of Rch_Before. The noise parameters will be described later with reference to FIG. 12.

In this embodiment, the noise parameter recording unit 205 records all the coefficients for each of the 513 frequency spectrum points as noise parameters. However, rather than coefficients for all of the 513 frequency points, it is sufficient that the noise parameter recording unit 205 records at least the coefficients for the frequency points necessary to reduce noise. For example, the noise parameter recording unit 205 may record, as noise parameters, coefficients for each frequency spectrum of 20 Hz to 20 kHz, which are considered to be typical audible frequencies, but may not record coefficients for other frequency spectra. Also, for example, coefficients for frequency spectra whose coefficient value is zero may not be recorded in the noise parameter recording unit 205 as noise parameters.

The subtraction processing unit B227 subtracts NLB and NRB from Lch_A_After and Rch_A_After, respectively. For example, the subtraction processing unit B227 has an L subtractor B227a that subtracts NLB from Lch_A_AFTER, and an R subtractor B227b that subtracts NRB from Rch_Before. The L subtractor B227a subtracts NLB from Lch_Before, and outputs array data of 513 points from Lch_After[0] to Lch_After[512]. The R subtractor B227b subtracts NRA from Rch_Before, and outputs array data of 513 points from Rch_After[0] to Rch_After[512]. In this embodiment, the subtraction processing unit B227 performs subtraction processing using the spectral subtraction method, and subtracts noise related to lens driving, which is particularly noisy.

In this embodiment, the subtraction processing unit B227 subtracts constantly occurring noise such as microphone floor noise and electrical noise other than noise generated by lens driving. Note that in this embodiment, the noise data generating unit B224 generates NLB and NRB based on Nch_Before, but other methods are also possible. For example, NLB and NRB may be recorded in the noise parameter recording unit 205, and the subtraction processing unit B227 may read NLB and NRB directly from the noise parameter recording unit 205 without going through the noise data generating unit B224. This is because there is little need to refer to the noise included in Nch_Before, since microphone floor noise, electrical noise, and other noise occur constantly.

In this embodiment, the noise reduction process is described as being performed in the order of subtraction processing unit A217 to subtraction processing unit B227, but the noise reduction process may be performed in the reverse order of subtraction processing unit B227 to subtraction processing unit A217.

The rest of the configuration and operation of the imaging device 100 is the same as in the first embodiment.

<Noise parameters in the third embodiment>
Fig. 12 is an example of noise parameters recorded in the noise parameter recording unit 205 in the third embodiment. The noise parameters are parameters for correcting an audio signal generated by the noise microphone 201c acquiring drive sounds generated in the housing of the image pickup device 100 and the housing of the optical lens 300. As shown in Fig. 12, in this embodiment, PLxA, PRxA, PLxB, and PRxB are recorded in the noise parameter recording unit 205. In this embodiment, the source of the drive sounds as PLxA and PRxA is described as being inside the housing of the optical lens 300. The drive sounds generated inside the housing of the optical lens 300 are transmitted to the inside of the housing of the image pickup device 100 via the lens mount 301, and are acquired by the L microphone 201a, the R microphone 201b, and the noise microphone 201c.

In this embodiment, multiple noise parameters corresponding to the type of optical lens 300 are recorded in the noise parameter recording unit 205. This is because the frequency of the drive sound differs depending on the type of optical lens 300. The imaging device 100 generates noise data using the noise parameter corresponding to the type of optical lens 300 out of the multiple noise parameters.

In addition, in this embodiment, the imaging device 100 records noise parameters for constant noise as PLxB and PRxB for each video mode. Constant noise is, for example, white noise, microphone floor noise, and electrical noise. In addition, the imaging device 100 may record noise parameters for each type of optical lens 300, and for each temperature inside the housing of the imaging device 100 and tilt of the imaging device 100 detected by the information acquisition unit 103.

Note that the average value of the coefficient values of PLxA and PRxA is greater than the average value of the coefficient values of PLxB and PRxB. This is because the noise reduced by PLxA and PRxA is louder and more harsh than the noise reduced by PLxB and PRxB.

<Timing chart of noise reduction processing in the third embodiment>
The noise reduction process in this embodiment will be described with reference to FIG.

Figures 13(a), (b), (d), (g), (hA), (hB), and (i) are examples of timing charts of audio processing in the noise data generating unit 204, the noise parameter selecting unit 206, and the subtraction processing unit 207. In this embodiment, for the sake of simplicity, audio processing of the Lch will be described, but audio processing of the Rch is similar. The horizontal axis of the graphs in Figures 9(a), (b), (d), (g), (hA), (hB), and (i) is the time axis.

Figure 13 (a) shows an example of a lens control signal. The lens control signal is a signal that the lens control unit 102 uses to instruct the optical lens 300 to drive. In this embodiment, the level of the lens control signal is expressed as two values, High and Low. When the level of the lens control signal is High, the lens control unit 102 is instructing the optical lens 300 to drive. In other words, when the level of the lens control signal from the control unit 111 is High, it can be determined that noise is being generated from the optical lens 300. When the level of the lens control signal is Low, the lens control unit 102 is not instructing the optical lens 300 to drive.

Figure 13 (b) is a graph showing an example of the value of Lch_Before[n]. The vertical axis indicates the value of Lch_Before[n]. In this embodiment, Lch_Before[n] is the signal at the nth frequency point, among the Lch_Before output from the FFT unit 203, at which a signal indicating the driving sound of the optical lens 300 appears characteristically. Note that in this embodiment, the signal at the nth frequency point is described, but audio processing is performed similarly for other frequencies. Furthermore, the signals indicated by signal V and signal W are signals containing noise. In this embodiment, signal V indicates a signal containing noise associated with lens driving. Signal W indicates a noise signal containing constant noise such as microphone floor noise and electrical noise.

Figure 13(d) is a graph showing an example of the value of Nch_Before[n]. Nch_Before[n] is the signal at the nth frequency point, among the Nch_Before output from the FFT unit 203, at which the signal indicating the driving sound of the optical lens 300 appears characteristically. The vertical axis indicates the value of Nch_Before[n]. In Nch_Before[n], the noise signals indicated by the signals V and W in Figure 9(b) appear more characteristically than in Lch_Before.

Figure 13 (g) shows an example of the operating state of subtraction processing unit A217 and subtraction processing unit B227 selected by the switching unit 210. In this embodiment, the solid part indicates that noise reduction processing is being performed only by subtraction processing unit B227. The checkered part indicates that noise reduction processing is being performed by subtraction processing unit A217 and subtraction processing unit B227.

Figure 13 (hA) is a graph showing an example of the value of NLA[n]. In this embodiment, NLA[n] indicates the signal value of the nth frequency point of the NLA generated by the noise data generating unit A214. The vertical axis indicates the value of NLA[n].

Figure 13 (hB) is a graph showing an example of the value of NLB[n]. In this embodiment, NLB[n] indicates the signal value of the nth frequency point among the NLBs generated by the noise data generating unit B224. The vertical axis indicates the value of NLB[n].

Figure 13(i) is a graph showing an example of the value of Lch_After[n]. In this embodiment, Lch_After[n] indicates the value of the signal at the nth frequency point of Lch_After output from the subtraction processing unit 207. The vertical axis indicates the value of Lch_After[n].

Next, the timing of each operation will be explained using times t1301 to t1302.

At time t1301, the lens control unit 102 outputs a High signal as a lens control signal to the optical lens 300 and the noise parameter selection unit 206 (Figure 13 (a)).

At time t1301, the switching unit 210 switches from noise reduction processing performed only by the subtraction processing unit B 227 to noise reduction processing performed by both the subtraction processing unit A 217 and the subtraction processing unit B 227 ( FIG. 13( g )).
From time t1301, the noise data generating unit A214 generates NLA[n] based on Nch_Before[n] and the noise parameter PLxA[n] (FIG. 13(hA)). Alternatively, the noise data generating unit A214 may constantly generate NLA[n], and the subtraction processing unit A217 may start subtraction in response to the lens control signal becoming High. The noise data generating unit B224 generates NLB[n] based on Nch_Before[n] and the noise parameter PLxB[n] (FIG. 13(hB)).

From time t1301, subtraction processing unit A217 and subtraction processing unit B227 subtract NLA[n] and NLB[n] from Lch_Before[n] and output Lch_After[n] (Figure 13(i)).

At time t1302, the lens control unit 102 determines that driving of the optical lens 300 has ended, and outputs a low signal as a lens control signal to the optical lens 300 and the noise parameter selection unit 206 (Figure 13 (a)).

At time t1302, the switching unit 210 switches from noise reduction processing by the subtraction processing unit A217 and the subtraction processing unit B227 to noise reduction processing performed only by the subtraction processing unit B227 (FIG. 13(g)). Note that from time t1302, the subtraction processing unit A217 is not used, and therefore NLA[n] is not used (shaded area in FIG. 13(hA)). Meanwhile, NLB[n] generated by the noise data generating unit B224 continues to be used by the subtraction processing unit B227 (FIG. 13(hB)).

At time t1302, the subtraction processing unit B227 subtracts NLB[n] from Lch_Before[n] and outputs Lch_After[n] (Figure 13 (i)).

After that, the audio input unit 104 performs noise reduction processing as described above based on the signal output from the lens control unit 102.

In this way, the imaging device 100 can reduce power consumption by switching the noise reduction process, which is performed only while the optical lens 300 is in operation, based on the lens control signal.

[Other Examples]
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The present invention is not limited to the above-mentioned examples as they are, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned examples. For example, some components may be deleted from all the components shown in the examples. Furthermore, components from different examples may be appropriately combined.

Claims (11)

a first microphone for acquiring environmental sounds;
a second microphone for acquiring sound from the noise source;
a first conversion means for performing a Fourier transform on the audio signal from the first microphone to generate a first audio signal;
a second conversion means for performing a Fourier transform on the audio signal from the second microphone to generate a second audio signal;
a generating means for generating first noise data corresponding to the noise from the noise source by using the second audio signal and a parameter related to the noise from the noise source;
a noise reduction means for performing a first noise reduction process for reducing noise of the noise source contained in the first audio signal based on the first noise data, and a second noise reduction process for reducing constant noise contained in the first audio signal, the noise reduction means including: a first noise reduction processing means for performing the first noise reduction processing on the first audio signal output from the first conversion means; and a second noise reduction processing means for performing the second noise reduction processing on the first audio signal output from the first conversion means or the audio signal output from the first noise reduction processing means;
a switching means for outputting the first audio signal output from the first conversion means to the first noise reduction processing means when noise from the noise source is occurring so that the first noise reduction processing is performed, and for outputting the first audio signal output from the first conversion means to the second noise reduction processing means when noise from the noise source is not occurring so that the first noise reduction processing is not performed;
a third transform means for performing an inverse Fourier transform on the audio signal from the noise reduction means;
having
The audio processing device is characterized in that the second noise reduction processing means performs the second noise reduction processing on the audio signal output from the first noise reduction processing means when noise from the noise source is occurring, and performs the second noise reduction processing on the first audio signal output from the first conversion means when noise from the noise source is not occurring. The generating means generates second noise data corresponding to the constant noise,
2. The audio processing apparatus according to claim 1, wherein the second noise reduction processing means performs the second noise reduction processing based on the second noise data. the first noise reduction processing means performs a process of subtracting the first noise data from the first audio signal output from the first conversion means;
3. The audio processing device according to claim 2, characterized in that the second noise reduction processing means performs a process of subtracting the second noise data from the first audio signal output from the first conversion means or the audio signal output from the first noise reduction processing means. The switching means, based on a control signal for controlling a drive circuit which is the noise source,
4. The audio processing device according to claim 1, wherein the first audio signal output from the first conversion means is output to either the first noise reduction processing means or the second noise reduction processing means. the drive circuit is a circuit for driving a lens,
5. The audio processing device according to claim 4, wherein the control signal is a signal output to the driving circuit for moving the lens. The audio processing device according to any one of claims 1 to 5, characterized in that the constant noise includes at least one of white noise, floor noise of the first microphone, and electrical noise of the first microphone. the first microphone includes a left channel microphone and a right channel microphone;
7. The audio processing device according to claim 1, wherein the generating means generates the first noise data based on the parameters corresponding to the left channel microphone and the right channel microphone. 8. The audio processing device according to claim 1, wherein the first microphone has a hole for inputting environmental sound, and the second microphone does not have a hole for inputting environmental sound. The audio processing device according to any one of claims 1 to 8, characterized in that the parameter is a ratio of the amplitudes of the first audio signal and the second audio signal acquired when it is assumed that there is no environmental sound. a first microphone for acquiring environmental sounds;
and a second microphone for acquiring a sound from a noise source,
a first transformation step of performing a Fourier transform on the audio signal from the first microphone to generate a first audio signal;
a second transformation step of performing a Fourier transform on the audio signal from the second microphone to generate a second audio signal;
a generating step of generating first noise data corresponding to the noise from the noise source by using the second audio signal and a parameter related to the noise from the noise source;
a noise reduction step of performing a first noise reduction process for reducing noise of the noise source contained in the first audio signal based on the first noise data, and a second noise reduction process for reducing constant noise contained in the first audio signal, the noise reduction step including a first noise reduction process step for performing the first noise reduction process on the first audio signal output from the first conversion step, and a second noise reduction process step for performing the second noise reduction process on the first audio signal output from the first conversion step or the audio signal output from the first noise reduction process step;
a switching step of outputting the first audio signal output from the first conversion step to the first noise reduction processing step so that the first noise reduction processing is performed when noise from the noise source is occurring, and outputting the first audio signal output from the first conversion step to the second noise reduction processing step so that the first noise reduction processing is not performed when noise from the noise source is not occurring;
performing an inverse Fourier transform on the audio signal obtained by the noise reduction step;
having
A control method characterized in that the second noise reduction processing step performs the second noise reduction processing on the audio signal output from the first noise reduction processing step when noise from the noise source is occurring, and performs the second noise reduction processing on the first audio signal output from the first conversion step when noise from the noise source is not occurring. A computer-readable program for causing a computer to function as each of the means of the voice processing device according to any one of claims 1 to 9 .

Images