JP7774997B2 - Imaging device, control method, and program - Google Patents

Description

The present invention relates to a voice processing device that performs voice processing on a person's voice.

When shooting video using an imaging device, it is important to capture the shooting situation exactly as the photographer imagined it, and this applies not only to video but also to audio.

Patent document 1 discloses a method for extracting the sound of a subject and individually adjusting the extracted audio signal according to the subject's position to create an acoustic space with a sense of realism and stereophony.

JP 2012-138930 A

However, when humans listen to a conversation, the accurately reproduced acoustic space does not necessarily match what humans imagine. For example, even when many people are chatting away, humans can naturally hear the conversation of people they are interested in, as well as their own name. It is also said that humans use visual information in addition to audio information, and by visually confirming the speaker, they supplement their hearing with information obtained from the person's mouth movements and gestures. In other words, it is important that the audio recorded on video is recorded so that it matches the conversational audio that is remembered (imaged) by humans.

However, since the aim of Patent Document 1 is to accurately reproduce the acoustic space of a voice based on the relative positions of people (sound sources), there is a risk that the resulting video may differ from the photographer's image.

The present invention aims to record video and audio that matches the photographer's imagination.

The imaging device of the present invention comprises a detection means for detecting a subject from a moving image, a determination means for determining a sound of the subject from the moving image, a selection means for selecting a main subject from the subjects detected from the moving image, and a determination means for determining a subject related to the main subject selected by the selection means,
When a subject related to the main subject moves out of the field of view of the video, the determination means determines whether the subject that has moved out of the field of view of the video continues to be related to the main subject based on the sound of the subject determined by the determination means.

This invention makes it possible to record video and audio in line with the photographer's imagination.

FIG. 1 is a block diagram of an imaging apparatus according to a first embodiment. FIG. 2 is a block diagram of an imaging processing unit and an audio processing unit according to the first embodiment (during recording); FIG. 2 is a block diagram of an imaging processing unit and an audio processing unit (during post-processing) according to the first embodiment. FIG. 10 is a diagram illustrating a main target selection method according to the first embodiment. FIG. 4 is a diagram showing an operation flow of a moving image recording sequence according to the first embodiment. FIG. 1 is a diagram illustrating an assumed scene according to a first embodiment. FIG. 2 is a diagram illustrating the content of audio processing according to the first embodiment. FIG. 10 is a block diagram of an imaging processing unit and an audio processing unit according to a second embodiment. FIG. 10 is a diagram showing an operation flow of a video recording sequence according to the second embodiment. FIG. 10 is a diagram for explaining a problem of the second embodiment. FIG. 10 is a diagram illustrating a problem scene in the second embodiment.

A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[First embodiment]
In this embodiment, an audio processing device included in an imaging device will be described with reference to FIGS. 1 to 3. FIG.

Figure 1 is a block diagram showing the configuration of the imaging device 100 of the first embodiment.

The imaging unit 101 converts the optical image of the subject captured by the photographing optical lens into an image signal using an imaging element, and the image processing unit 102 performs analog-to-digital conversion, image adjustment processing, etc. to generate image data. The photographing optical lens may be a built-in optical lens or a detachable optical lens. The imaging element may be a photoelectric conversion element such as a CCD or CMOS. The audio input unit 103 uses a microphone built in or connected via an audio terminal to collect sound from the surroundings of the imaging device 100, and the analog-to-digital converted sound is subjected to various audio processes by the audio processing unit 104 to generate audio data. The microphone may be directional or omnidirectional. The memory 105 temporarily stores image data obtained by the imaging unit 101 and image processing unit 102, and audio data obtained by the audio input unit 103 and audio processing unit 104. The display control unit 106 displays images related to the image data obtained by the image processing unit 102, the operation screen of the imaging device 100, menu screens, etc. on the display unit 107 or on an external display via a video terminal (not shown). The display unit 107 has a touch panel function, and can be operated by the photographer to select menus, subjects, etc.

The encoding processing unit 108 reads the image data and audio data temporarily stored in the memory 105 and performs a predetermined encoding process to generate compressed image data, compressed audio data, etc. The audio data may not be compressed. The compressed image data may be compressed using any compression method, such as MPEG2 or H.264/MPEG4-AVC. The compressed audio data may also be compressed using a compression method such as AC3(A)AC, ATRAC, or ADPCM. The recording/playback unit 109 records the compressed image data, compressed audio data, or audio data generated by the encoding processing unit 108, and various data, to and reads them from the recording medium 110. The recording medium 110 includes any type of recording medium capable of recording image data, audio data, etc., such as a magnetic disk, optical disk, or semiconductor memory.

The control unit 111 controls each block of the imaging device 100 by transmitting control signals to each block of the imaging unit 101 and the imaging device 100. It includes a CPU and memory for executing various controls. The memory 105 used by the control unit 111 includes ROM for storing various control programs, RAM for arithmetic processing, and other memory external to the control unit 111. The operation unit 112 includes buttons and dials and transmits instruction signals to the control unit 111 in response to user operations. In this embodiment of the imaging device, it includes a shooting button for starting and stopping video recording, a zoom lever for optically or electronically zooming the image, a cross key for various adjustments, and a confirmation key. The audio output unit 113 outputs audio data and compressed audio data reproduced by the recording/playback unit 109, or audio data output by the control unit 111, to a speaker 114, an audio terminal, or the like. The external output unit 115 outputs compressed video data, compressed audio data, audio data, and the like reproduced by the recording/playback unit 109 to an external device. The data bus 116 supplies various data such as audio data and image data, as well as various control signals, to each block of the imaging device 100.

Now, we will explain the normal operation of the imaging device 100 of this embodiment.

In this embodiment, the imaging device 100 supplies power to each block of the imaging device from a power supply unit (not shown) in response to a user operating the operation unit 112 to issue an instruction to turn on the power.

When power is supplied, the control unit 111 checks the mode selected by the mode selector switch of the operation unit 112, for example, shooting mode, playback mode, etc., based on an instruction signal from the operation unit 112. In video recording mode, the image data (video data) obtained by the imaging unit 101 and image processing unit 102 and the audio data obtained by the audio input unit 103 and audio processing unit 104 are saved as a video file. In playback mode, the compressed image data recorded on the recording medium 110 is played back by the recording/playback unit 109 and displayed on the display unit 107.

In video recording mode, the control unit 111 first sends control signals to each block of the imaging device 100 to transition to a shooting standby state, causing the following operations to occur. The imaging unit 101 converts the optical image of the subject captured by the shooting optical lens into an image signal using the imaging element, and the image processing unit 102 performs image adjustment processing and other processes to generate image data. The obtained image data is then sent to the display control unit 106, which displays it on the display unit 107. The user prepares for shooting while looking at the screen displayed in this way.

The audio input unit 103 converts analog audio signals obtained by multiple microphones into digital form and processes the resulting digital audio signals to generate multi-channel audio data. The resulting audio data is then sent to the audio output unit 113, which outputs the audio from a connected speaker 114 or earphones (not shown). The user can adjust the manual volume to determine the recording volume while listening to the audio output in this manner.

Next, when the user operates the recording button on the operation unit 112 to send an instruction signal to start shooting to the control unit 111, the control unit 111 sends an instruction signal to start shooting to each block of the imaging device 100, causing it to perform the following operations.

The imaging unit 101 converts the optical image of the subject captured by the photographing optical lens into an image signal using an imaging element, and the image processing unit 102 performs image adjustment processing and other processes to generate image data. The obtained image data is then sent to the display control unit 106, which displays it on the display unit 107. The obtained image data is also sent to the memory 105.

The audio input unit 103 digitally converts analog audio signals obtained by multiple microphones, and processes the multiple digital audio signals obtained by the audio processing unit 104 to generate multi-channel audio data. The obtained audio data is then sent to memory 105. If there is only one microphone, the obtained analog audio signal is digitally converted to generate audio data, and the audio data is sent to memory 105.

The encoding processing unit 108 reads the image data and audio data temporarily stored in the memory 105 and performs the specified encoding to generate compressed image data, compressed audio data, etc.

The control unit 111 then combines the compressed image data and compressed audio data to form a data stream, which it outputs to the recording and playback unit 109. If the audio data is not compressed, the control unit 111 combines the audio data stored in memory 105 with the compressed image data to form a data stream, which it outputs to the recording and playback unit 109. The recording and playback unit 109 writes the data stream to the recording medium 110 as a single video file under file system management such as UDF or FAT. The above operations continue while shooting is in progress.

Then, when the user operates the recording button on the operation unit 112 to send an instruction signal to end shooting to the control unit 111, the control unit 111 sends an instruction signal to end shooting to each block of the imaging device 100, causing it to perform the following operations.

The imaging unit 101, image processing unit 102, audio input unit 103, and audio processing unit 104 each stop generating image data and audio data. The encoding processing unit 108 reads the remaining image data and audio data stored in memory and performs the specified encoding, stopping operation once it has finished generating compressed image data, compressed audio data, etc. If the audio data is not compressed, it will naturally stop operation once it has finished generating the compressed image data.

The control unit 111 then combines these final compressed image data with the compressed audio data or voice data to form a data stream, which is output to the recording/playback unit 109. The recording/playback unit 109 writes the data stream to the recording medium 110 as a single video file under file system management such as UDF or FAT. When the supply of the data stream stops, the video file is completed and the recording operation is stopped. When the recording operation stops, the control unit 111 sends a control signal to each block of the imaging device 100 to transition to a shooting standby state, and the imaging device returns to the shooting standby state.

Next, in playback mode, the control unit 111 sends control signals to each block of the imaging device 100 to transition to playback mode, causing the following operations to occur: The recording and playback unit 109 reads a video file consisting of compressed image data and compressed audio data recorded on the recording medium 110, and sends the read compressed image data and compressed audio data to the encoding processing unit 108.

The encoding processing unit 108 decodes the compressed image data and compressed audio data and transmits them to the display control unit 106 and audio output unit 113, respectively. The display control unit 106 displays the decoded image data on the display unit 107. The audio output unit 113 outputs the decoded audio data from a built-in or attached external speaker.

As described above, the imaging device 100 of this embodiment is capable of recording and playing back images and audio.

In this embodiment, when obtaining an audio signal, the audio input unit 103 and audio processing unit 104 perform processing such as level adjustment of the audio signal obtained by the microphone. This processing may be performed continuously after the device is started, or may be performed after a shooting mode is selected, or after a mode related to audio recording is selected. Furthermore, in a mode related to audio recording, the above processing may be performed in response to the start of audio recording. In this embodiment, the above processing is performed when video recording begins.

Figure 2 is a block diagram showing an example of the detailed configuration of the imaging unit 101, image processing unit 102, audio input unit 103, and audio processing unit 104 of the imaging device 100 of this embodiment.

The imaging unit 101 has an optical system, such as an optical lens 201, that captures an optical image of a subject, and an imaging element 202 that converts the optical image of the subject captured by the optical lens 201 into an electrical signal (image signal). It also has an optical lens control unit 203 that has a position sensor for moving the optical lens 201 and a known driving mechanism, such as a motor. In this embodiment, the imaging unit 101 is described as having the optical lens 201 and optical lens control unit 203 built in, but these may also be detachable, interchangeable optical lenses. For example, when a user inputs an instruction for zooming, focus adjustment, or the like by operating the operation unit 112, the control unit 111 sends a control signal (drive signal) to the optical lens control unit 203 to move the optical lens. In response to this control signal, the optical lens control unit 203 confirms the position of the optical lens 201 using a position sensor and moves the optical lens 201 using a motor or the like.

The image processing unit 102 performs various image quality adjustment processes on the image signal converted by the image sensor 202 in the image adjustment unit 221 to form image data, which is then sent to the memory 105 via the data bus 116. Based on the image data formed here, the control unit 111 performs various adjustments such as focus adjustment and light intensity adjustment.

Furthermore, in this embodiment, the image processing unit 102 has various detection functions. The person detection unit 222 extracts facial features such as the eyes, nose, and mouth from the image data formed by the image adjustment unit 221, and detects the position and face size of the person in the image data. By storing information on these feature points in memory 105, it is possible to individually recognize the subject person based on this information. The person detection unit 222 also has a person movement detection unit 223 that detects lip and head movement, and a person speech detection unit 224 that uses this information to determine whether the person is speaking. The image processing unit 102 also has a main subject selection unit 225 that selects which person, from the people detected by the person detection unit 222, will be the main subject of audio processing (hereinafter also referred to as the main subject or main target). The main subject selection unit 225 selects the main target based on conditions set by the control unit 111. The conditions for selecting the main target by the main target selection unit 225 will be described later.

The image processing unit 102 further includes a conversation group detection unit 226. The conversation group detection unit 226 detects, from among the people detected by the person detection unit 222, people who are conversing with the person selected by the main subject selection unit 225. This detection is determined based on the relative positions of the people, the direction of their faces, their movements, and so on. For example, the conversation group detection unit 226 determines that the subject closest to the main subject is the person conversing with the main subject (a related person). Also, for example, the conversation group detection unit 226 determines that a subject whose body, face, line of sight, etc. faces the main subject is the person conversing with the main subject. Also, if the main subject is moving, the conversation group detection unit 226 determines that a subject in the direction of the movement is the person conversing with the main subject. This is because such a subject is likely to converse with the main subject in the near future.

Note that if the conversation group detection unit 226 determines that a person who is conversing with the main target has not conversed with the main target for longer than a predetermined time, that person is considered to be a person who is not conversing with the main target (not related to the main target). In other words, if a person who is conversing with the main target has not conversed with the main target for a predetermined time, that person is considered to be a person who is conversing with the main target, even if it is determined that they are not conversing with the main target.

Next, we will explain the audio input unit 103 and audio processing unit 104. The audio input unit 103 is a microphone 211 that converts audio vibrations into an electrical signal and outputs it as an audio signal. In this embodiment, the microphone 211 is a stereo system consisting of two channels, left and right channels (Lch/Rch), but it may also be a mono system with one channel, or a system that includes multiple microphones with two or more channels. The A/D conversion unit 212 is a means for converting the analog audio signal obtained by the microphone 211 into a digital audio signal.

The audio processing unit 104 is a block that performs various audio processes on the audio signal converted by the audio input unit 103. In this embodiment, the audio processing unit 104 has an audio extraction unit 213, an audio adjustment unit 215, and an audio synthesis unit 217. The audio extraction unit 213 is capable of extracting (determining) audio into human voices and other audio (hereinafter referred to as "non-human voices"). Furthermore, the human voice extraction unit 214 is capable of extracting individual audio from human voices based on information from the person detection unit 222. For example, the human voice extraction unit 214 extracts individual audio based on the frequency, volume, and intonation of the audio. Furthermore, in the first embodiment, the control unit 111 can associate audio with a subject based on the audio extracted by the human voice extraction unit 214 and the subject's movements detected by the image processing unit 102. For example, the subject's movements include the frequency of speech, the timing of speech, and mouth movements.

The audio adjustment unit 215 can also individually perform level adjustments, equalizers, and other audio processing for each frequency band on the audio extracted by the audio extraction unit 213. In particular, the conversation audio adjustment unit 216 performs adjustments based on information from the conversation group detection unit 226, emphasizing the extracted audio to make it easier to hear, or reducing its volume to make it harder to hear. The details of these adjustments will be described later. Furthermore, the audio synthesis unit 217 synthesizes the audio that has been individually adjusted by the audio adjustment unit 215, and returns it to a single audio signal. The amplitude of the synthesized audio signal is then adjusted to a predetermined level by an auto level controller (hereinafter, ALC 219). With the above configuration, the audio processing unit 104 performs predetermined processing on the audio signal, forms audio data, and sends it to memory 105.

Figure 3 is a block diagram showing another example of the configuration of the image processing unit 102 and audio processing unit 104 of the imaging device 100 of this embodiment. The difference between Figure 3 and Figure 2 is the input sources of image data and audio data. In Figure 2, the image signal comes from the imaging unit 101, and the audio signal comes from the audio input unit 103. On the other hand, in Figure 3, the image and audio input sources are data stored in memory 105. By using data temporarily stored (held) in memory 105 in this way, it is possible to use the proposed method not only for processing during shooting, but also for post-processing after recording. Furthermore, the main subject selection unit 225 can select a person to be the target of audio processing from a series of video data.

Here, an example of a method for selecting a main subject by the main subject selection unit 225 will be described using Figure 4. In this embodiment, the main subject will be described as a person who is thought to be the photographer's focus. For example, in Figure 4(a), the focus mark 402 is a mark indicating the subject on which the imaging device 100 is focusing. In Figure 4(a), the main subject 401 and the focus mark 402 match, so the imaging device 100 recognizes the main subject 401 as the main subject and is focusing on the main subject 401. The main subject selection unit 225 determines this main subject 401 to be the main subject. In this way, a person recognized as the main subject can be selected as the main subject.

Figure 4(b) also shows a method using a registered face image. The registered face image 403 is an image of a subject that has been registered in advance in the memory 105. The main subject selection unit 225 selects a person whose face is determined to match the face in the image as the main subject.

Figure 4(c) also shows a method for determining the main subject at the photographer's discretion. The photographer selects the main subject by touching the touch panel of the display unit 107 from among the people displayed on the display unit 107. The main subject selection unit 225 determines that the subject selected by the photographer is the main subject.

Also, Figure 4(d) shows a method using pre-recorded video data. For example, if pre-recorded video data 404 is stored in memory 105, the main subject selection unit 225 determines the person 405 who appears most frequently in the video data 404 as the main subject. Alternatively, for example, the main subject selection unit 225 may select a person who is frequently in focus.

Note that, for example, when the main object selection unit 225 selects a focused subject as the main object, even if the focus on that main object is lost, the main object selection unit 225 will maintain that subject as the main object as long as the focus returns to that subject within a predetermined time. In other words, if the focus is lost from the main object for longer than a predetermined time, the main object selection unit 225 will select a new main object.

Next, the operation of the imaging device 100 of this embodiment will be explained using Figures 5 to 7.

Figure 5 is a flowchart showing an example of a video recording sequence for the imaging device 100. The processing of this imaging device 100 is realized by software stored in ROM (not shown) being loaded into memory 105 and executed by the CPU. The processing of this flowchart is triggered when the imaging device 100 is powered on.

In step S501, the control unit 111 accepts an instruction to start video recording via a user's operation of the operation unit 112.

In step S502, the control unit 111 connects an audio path for recording audio.

In step S503, after the audio path is established, the control unit 111 performs initial signal processing settings, including the control described in this embodiment, and starts signal processing for video recording. The following description focuses on the recording sequence. Until signal processing for video recording is completed, the control unit 111 records the video to be recorded in the video.

In step S504, the person detection unit 222 of the image processing unit 102 detects the subject.

In step S505, the main subject selection unit 225 of the image processing unit 102 selects (determines) a main subject from the subjects detected in step S504.

In step S506, the conversation group detection unit 226 of the image processing unit 102 determines the person (subject) who is conversing with the main subject selected in step S505.

In step S507, the audio extraction unit 213 of the audio processing unit 104 extracts the person's audio.

The audio adjustment unit 215 of the audio processing unit 104 performs adjustment processing on the audio extracted in step S507. The content of the audio adjustment processing differs depending on whether the subject (person) of the audio extracted in step S507 is a subject (person) belonging to the main conversation group. Details of the audio adjustment processing will be described later using Figures 6 and 7, but this flowchart will provide a simplified explanation.

In step S508, the audio adjustment unit 215 of the audio processing unit 104 determines whether the person whose voice was extracted in step S507 is a subject belonging to the main conversation group. If the person whose voice was extracted is a subject belonging to the main conversation group, the process of step S509 is executed. If the person whose voice was extracted is not a subject belonging to the main conversation group, the process of step S510 is executed.

In step S509, the audio adjustment unit 215 of the audio processing unit 104 adjusts the level of the extracted audio so that the volume is increased.

In step S510, the audio adjustment unit 215 of the audio processing unit 104 adjusts the level of the extracted audio to reduce its volume. In step S511, the audio adjustment unit 215 of the audio processing unit 104 performs adjustment processing other than volume on the extracted audio.

In step S512, the voice synthesis unit 217 of the voice processing unit 104 synthesizes the extracted voices that have been individually adjusted to generate a single voice data.

In step S513, the control unit 111 determines whether to end video recording. For example, the control unit 111 determines to end video recording when the user operates the operation unit 112 to instruct the end of video recording, or when it is determined that the remaining capacity of the recording medium 110 is low. If it is determined not to end video recording, the process returns to step S504, and the recording sequence process continues. If it is determined to end video recording, the process of step S514 is executed.

If it is determined here that video recording should not be ended, the process returns to step S504. In other words, while video recording is in progress, the main subject and the people who are conversing with the main subject are repeatedly determined. This allows the control unit 111 to determine a different subject as the main subject, even if, for example, the main subject disappears outside the angle of view or goes out of focus. Also, even if the number of people conversing with the main subject increases or decreases, the control unit 111 can determine the people who are conversing with the main subject accordingly.

In step S514, the control unit 111 disconnects the audio path and terminates signal processing.

Here, we will explain the audio adjustment process using Figures 6 and 7.

Figure 6 shows a hypothetical scene for audio adjustment processing. Assume that four subjects (people), person 602 to person 605, are present within angle of view 601, and person 602 is conversing (speaking) with person 603, and person 604 is conversing (speaking) with person 605. In this case, if person 602 is selected as the main target for audio processing by main target selection unit 225, people 602 and 603 are detected as conversation group 610 from the image data by conversation group detection unit 226. In this case, the audio of people 602 and 603 is adjusted to emphasize the audio that deserves attention, and the audio of people 604 and 605 is adjusted to emphasize the audio that is not required for emphasis.

Figures 7(a) to (c) show the audio adjustment process. In Figure 7, person 602, person 603, and person 604 in Figure 6 are represented as person A, person B, and person C, respectively (person 605 is not shown).

Figure 7(a) shows the voice signals of persons A to C extracted by the person voice extraction unit 214. That is, signal 701 shows the extracted voice signal of person A, signal 702 shows the extracted voice signal of person B, and signal 703 shows the extracted voice signal of person C. In each signal, sections with large amplitude indicate periods when the respective person is speaking (uttering voice) (voiced timing), and sections with small amplitude indicate periods when the person is not speaking (silent timing). For example, comparing signals 704 and 705, since persons A and B are conversing, voiced timing and silent timing appear almost alternately. On the other hand, since person C is not a conversation partner of persons A and B, signal 706 does not alternate between voiced timing and silent timing as much as signals 704 and 705.

Figure 7 (b) shows the audio correction coefficient for each person. In this embodiment, a correction coefficient of 1.0 indicates that no level adjustment (gain adjustment) is performed. Furthermore, when the correction coefficient is greater than 1.0, the processing is an audio adjustment process that emphasizes the audio to make it easier to hear (increase the volume), and when the coefficient is less than 1.0, the processing is a process that makes the audio harder to hear (decrease the volume).

For example, let us consider a case where the conversation group detection unit 226 determines that person A and person B are conversing during period 710. In this case, since person A is the main target, the conversation audio adjustment unit 216 recognizes the voices of person A and person B as targets for emphasis and sets large correction coefficients for each voice (coefficients 714 and 715). In this embodiment, the correction coefficients for person A and person B are set to the same value. This is because it is assumed that the user (photographer) is listening to both voices equally. On the other hand, the conversation audio adjustment unit 216 sets a small correction coefficient for the voice of person C, who is determined not to be conversing with person A, so that person C's voice is relatively difficult to hear (coefficient 716). In this way, the conversation audio adjustment unit 216 emphasizes the voices of the main target person A and his/her conversation partner person B and reduces the volume of other voices. For example, the conversation audio adjustment unit 216 increases the gain and level for the voices of the main target person A and his/her conversation partner person B compared to those for other voices. This results in video data with video and audio that matches the image of the user who filmed it.

Figure 7(c) shows an audio signal that has been adjusted based on the correction coefficients of Figure 7(b) described above. For example, if the audio adjustment by the conversation audio adjustment unit 216 is achieved by gain adjustment, during period 710, the audio of persons A and B who have been determined to be conversing (signals 724 and 725) will have a correction coefficient greater than 1.0, making them louder and easier for the user to hear. Furthermore, the audio of person C who has not been determined to be conversing (signal 726) will have a correction coefficient less than 1.0, making them quieter and harder to hear. The extracted audio that has been individually adjusted in this way is synthesized by the audio synthesis unit 217, and as a result, only the conversation determined to be the target of attention is generated as audio data that is easy to hear.

Note that in this embodiment, the sound related to the main subject is emphasized (corrected to be louder) and the sound unrelated to the main subject is made less audible (corrected to be quieter), but adjustments can be applied to only one of them. In other words, it is sufficient if the correction coefficients for the main subject (person) and the subjects (people) with whom they are talking are larger than the correction coefficients for the other subjects.

Furthermore, the emphasis method used by the conversational voice adjustment unit 216 is not limited to adjusting the overall gain as described above, but may also be adjusted by frequency within the frequency band of human voices using an equalizer or the like.

[Second embodiment]
In the first embodiment, after selecting a main target, conversation groups of people who are conversing with the main target are detected based on their relative positions relative to the main target and their movements, and the audio of the conversation group is emphasized or other unnecessary audio is suppressed, thereby obtaining audio data that makes the conversation of interest easier to hear.

In the first embodiment, the method for detecting conversation groups is to determine the positional relationships, facial orientations, and movements of people who are conversing with the person selected by the main subject selection unit 225 from among the people detected by the person detection unit 222. In this way, in the first embodiment, the conversation group detection unit 226 detects people who are present within the angle of view 601 of the imaging device 100.

Now, suppose that person A, who is the main subject, and persons B and D (603, 606) within the angle of view 601 are detected as a conversation group, as shown in Figure 10(a). If person B moves out of the angle of view due to a zoom or panning operation by the photographer, even if the conversation between persons A, B, and D continues, person B will be removed from the conversation group when the next conversation group is detected, as shown in Figure 10(b). As a result, even if person B is participating in the conversation, the conversation group detection unit 226 does not determine that person B is part of the conversation group, which could result in the voice of person B not being emphasized and making the conversation difficult to hear.

The second embodiment aims to maintain the conversation group in the state it was in before the person left the field of view, even if at least one person in the conversation group that was within the field of view moves out of the field of view, if it is determined that the conversation of the person who moved out of the field of view is continuing, and continue to capture audio that is easy to hear.

The second embodiment will now be described in detail with reference to the accompanying drawings. Note that the configuration of the imaging device 100 in Figure 1 is the same as that of the first embodiment, so a description thereof will be omitted.

Figure 8 is a block diagram showing the detailed configuration of the imaging unit 101, image processing unit 102, audio input unit 103, and audio processing unit 104 of the imaging device 100 of this embodiment. Note that blocks having the same functions as those in Figure 2 are assigned the same numbers, and their explanations will be omitted.

The feature extraction unit 801 associates the sound extracted by the person voice extraction unit 214 with the person corresponding to that sound. For example, the feature extraction unit 801 associates the extracted sound with the corresponding person based on the sound features and the movement of the subject within the field of view. For example, the sound features are frequency, volume, and intonation. For example, the movement of the subject is the frequency of speech, the timing of speech, and mouth movement. This association can improve the accuracy of identifying the speaker. This allows the control unit 111 to identify the speaker from the sound even if the person in the conversation group is out of the field of view.

The conversation group correction unit 802 determines whether a person who is out of the field of view is continuing a conversation based on the voice characteristics associated with the person acquired by the feature extraction unit 801. The control unit 111 uses this result and the detection result of the conversation group detection unit 226 to correct the conversation group so that it takes into account the person who is out of the field of view.

In the second embodiment, a feature extraction unit 801 and a conversation group correction unit 802 were added to the block diagram shown in Figure 2, but the operation is the same even when the conversation group correction unit 802 is added to the block diagram shown in Figure 3.

Next, the operation of the imaging device 100 of the second embodiment will be described using Figures 9 and 11.

Figure 9 is a flowchart explaining a series of recording operations of the imaging device 100. In Figure 9, blocks that perform the same operations as in Figure 5 are assigned the same step numbers as in Figure 5. Here, we will first explain an example of an expected scene for the operation in Figure 9 using Figure 11.

Figure 11(a) shows the scene at the time the photographer presses the record button. In the scene shown in Figure 11(a) (hereinafter referred to as the initial shooting scene), Person A, Person B, and Person D (602, 603, 606) are present within the field of view. Person A is the main subject, and a conversation group containing the main subject is detected, consisting of three people: Person A, Person B, and Person D. We will now explain the scene when at least one person in the conversation group is outside the field of view.

Figures 11(b) to (e) show examples of scenes in which a person in a conversation group moves out of the field of view. Figures 11(b) to (d) show scenes in which person B (603) moves out of the field of view 601. Figure 11(e) shows a scene in which all members of the conversation group move out of the field of view due to the cameraman panning the imaging device 100. The horizontal "V" characters near the mouths of the people in each figure represent the vocalizations of each person, with the thickness of the lines representing the volume of their voices and the frequency of their participation in the conversation. Each scene in Figure 11 will be referred to as Figure 11(a) as the initial shooting scene, Figure 11(b) as scene b, Figure 11(c) as scene c, Figure 11(d) as scene d, and Figure 11(e) as scene e. Person 602, person 603, and person 606 appearing in each figure will be referred to as person A, person 603, and person D, respectively. The main subject in each scene is person A. In each diagram, the angle of view 601 indicates the imaging angle of the image capture device 100, and the conversation group 610 indicates the conversation group.

The assumptions for each scene in Figures 11(a) to (e) are as follows:

In scene b, person B is out of the frame of view compared to the initial shot, but continues to converse as if he were in the frame of view.

Scene c shows a scene in which person B is out of the field of view and is not engaged in conversation, as opposed to the initial shooting scene. Note that in scene c, neither person A nor person D are looking towards person B.

In scene d, compared to scene b, person B has moved into the distance, but the conversation continues. Note that in scene d, person B's voice is being input to the imaging device 100. Also, person A, who is within the field of view, is facing in the direction of person B, and is speaking louder.

Scene e shows a scene in which Person A, Person B, and Person D are out of the field of view compared to the initial shot scene. Note that in scene e, Person A, Person B, and Person D are continuing to have a conversation.

An example of an assumed scene for the operation in FIG. 9 has been described above using FIG. 11. The operation of the imaging device 100 will now be described using the flowchart in FIG. 9. In this embodiment, steps S901 to S904 will be mainly described.

First, steps S501 to S507 are performed to detect people within the field of view, identify the main subject, detect people conversing with the main subject, and extract audio.

In step S901, control unit 111 determines whether the number of people conversing with the main target detected in step S506 matches the number of people in the conversation group associated by feature extraction unit 801 and conversation group correction unit 802. For example, control unit 111 makes this determination by calculating the difference between the number of people in the conversation group detected in step S506 and the number of people in the current conversation group. If it is determined that the number of people has decreased, it means that there is a person in the conversation group associated by feature extraction unit 801 and conversation group correction unit 802 who is out of the field of view. If it is determined that the number of people matches, processing of step S904 is executed. If it is determined that the number of people does not match, processing of step S902 is executed.

In step S902, the conversation group correction unit 802 determines whether a conversation is ongoing between a person outside the field of view and a person within the field of view. If it is determined that a conversation is not ongoing between a person outside the field of view and a person within the field of view, the control unit 111 performs the processing from step S904 onwards, treating the current conversation group as the detection result of step S506. If it is determined that a conversation is ongoing, the processing of step S903 is executed.

In step S903, the control unit 111 corrects the person (subject) who is conversing with the main subject detected in step S506 so that the person outside the field of view is included in the conversation group within the field of view.

In step S904, the feature extraction unit 801 associates the audio with the person corresponding to that audio based on the audio extracted for each subject (person) extracted by the person audio extraction unit 214.

Here, using the above scene, we will explain an example of determining in step S902 whether person B is continuing a conversation with person A and person D.

In scene b, in steps S505 and S506 of FIG. 9, person D is identified as the person conversing with person A, the main subject. However, in step S901 of FIG. 9, it is determined that person B, who belonged to the conversation group in the initial shooting scene, has moved out of the field of view. Then, in step S902 of FIG. 9, the conversation group correction unit 802 determines that the person section is continuing to converse with person A and person D. Therefore, in step 903 of FIG. 9, the control unit 111 adds person B to the subjects (people) conversing with person A, the main subject. In other words, in scene b, the same conversation group as in the initial shooting scene is maintained.

Here, an example of determining whether person B is continuing a conversation with persons A and D will be described. Based on the information from feature extraction unit 801, conversation group correction unit 802 determines that the conversation is continuing if there is no change in the volume or intonation of person B's voice and the timing of his/her speech when conversing with persons A and D is synchronized. A case in which the timing of his/her speech is synchronized is, for example, when persons A and D and person B are alternating conversation (the conversation is continuing). In this case, control unit 111 adds person B to the people (subjects) conversing with person A, the main target. Furthermore, when image processing unit 102 can determine the direction outside the field of view of the subject and the direction of the subject's face, conversation group correction unit 802 further determines whether the conversation is continuing based on the direction of the face of person A or person D within the field of view and the direction outside the field of view of person B. In other words, even if it is determined that a conversation is ongoing based on the volume of the voices, bathing, and timing of speech described above, if the direction of the face of person A or person D within the angle of view does not match the direction in which person B has left the angle of view, conversation group correction unit 802 will determine that the conversation is not ongoing.

Scene c is a case where the voice of person B is not detected in scene b. In such a case, it is determined that person B is not participating in the conversation between person A and person D, and the control unit 111 leaves person D as the subject conversing with person A, the main subject, without making any modifications.

In scene d, person B moves away from person A and person D from the situation in scene b, but the conversation continues. In this scene, person B's voice has become quieter, but the timing of his speech when he is conversing with person A and person D is correct. Furthermore, person B's voice has become quieter, but person A's voice, in contrast, has become louder. From this information, conversation group correction unit 802 determines that person A and person B are conversing. In response, control unit 111 adds person B as a subject conversing with person A, the main subject.

In scene e, the photographer changes the imaging direction from the direction of persons A, B, and D toward the fireworks. In other words, persons A, B, and D are continuing their conversation, but person A, who is the main subject, has disappeared from the field of view. However, information from feature extraction unit 801 indicates that the acquired audio includes the voice of person A, so in this case, control unit 111 determines that person A is the main subject. Additionally, information from feature extraction unit 801 indicates that the voices of persons B and D are also continuing to be detected, so control unit 111 adds persons B and D as subjects who are conversing with person A, who is the main subject. In this way, in a scene like scene e, no persons are within the field of view, but the voices of the conversation group are recorded with emphasis. Note that control unit 111 may convert the content of the conversation into text and display it in a form such as a speech bubble, as shown in FIG. 11(f).

The above describes the operation of the imaging device 100 in the second embodiment.

Note that the number of people conversing with the main subject detected in step S506 is based on the person detection within the angle of view in step S504, so in the initial shooting scene (Figure 11(a)), there are two people, person B and person D, and in Figure 11(b) there is one person, person D.

In the second embodiment, audio extraction is performed based on the detection results of the person detection unit 222 until a predetermined time has elapsed since the start of video recording, and thereafter based on the results of the person movement detection unit 223 and information from the feature extraction unit 801.

As described above, according to the second embodiment, even if a person belonging to a conversation group moves out of the field of view, the conversation group can be corrected to an appropriate one as long as the conversation is ongoing.

In the second embodiment, the determination of conversation continuation in Figures 11(b), (c), and (d) was explained assuming that the factors that caused a person belonging to a conversation group to move out of the field of view were not taken into consideration, but this may also be taken into consideration. For example, if the photographer causes a person belonging to a conversation group to move out of the field of view by zooming the lens 201, the control unit 111 may determine that the conversation is continuing without using information from the feature extraction unit 801, since the person moved out of the field of view regardless of the photographer's intention.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the invention.

[Other embodiments]
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The present invention is not limited to the above-described embodiments, and can be embodied by modifying the components within the scope of the spirit of the invention when implemented. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all of the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

Claims (18)

a detection means for detecting a subject from a video;
a determining means for determining the sound of a subject from the video;
a selection means for selecting a main subject from the subjects detected from the video;
a determining means for determining a subject related to the main subject selected by the selecting means,
The audio processing device is characterized in that, when a subject related to the main subject moves out of the angle of view of the video, the determination means determines, based on the audio of the subject determined by the determination means, whether the subject that has moved out of the angle of view of the video continues to be related to the main subject. 2. The audio processing device according to claim 1, wherein the determination means determines that the subject outside the field of view of the video is related to the main subject when it is determined that the voice of the main subject and the subject outside the field of view of the video are having an ongoing conversation, and determines that the subject outside the field of view of the video is not related to the main subject when it is determined that the voice of the main subject and the subject outside the field of view of the video are not having an ongoing conversation. 3. The audio processing device according to claim 1, wherein the determination means determines that the subject outside the field of view of the video is not continuously associated with the main subject, even if it can determine that the voice of the main subject and the subject outside the field of view of the video are continuing to have a conversation based on the voice of the subject determined by the determination means, if the direction of the face of the main subject does not match the direction in which the subject outside the field of view of the video is out of the field of view. 4. The audio processing device according to claim 1, wherein the determination means determines that the main subject and a subject related to the main subject continue to be related to each other when it determines that a conversation between the main subject and a subject that is out of the field of view of the video is continuing even when the main subject is out of the field of view of the video. An audio processing device according to any one of claims 1 to 4, characterized in that the selection means selects a subject that is in focus in the video as the main subject. An audio processing device according to any one of claims 1 to 4, characterized in that the selection means selects a main subject from the video based on an image recorded as the main subject. An audio processing device according to any one of claims 1 to 4, characterized in that the selection means selects the subject that appears most frequently among the subjects captured in the video as the main subject. An audio processing device according to any one of claims 1 to 7, characterized in that the determination means determines subjects related to the main subject based on the distance from the main subject. An audio processing device according to any one of claims 1 to 8, characterized in that the determination means determines that the subject closest to the main subject is the subject related to the main subject. An audio processing device according to any one of claims 1 to 7, characterized in that the determination means determines that a subject facing the main subject is a subject related to the main subject. An audio processing device according to any one of claims 1 to 7, characterized in that the determination means determines subjects related to the main subject based on the movement of the main subject. an associating means for associating the subject detected by the detecting means with the sound extracted by the determining means;
and image processing means,
12. The audio processing device according to claim 1, wherein the associating means associates the subject detected by the detecting means with the audio extracted by the determining means based on the audio extracted by the determining means and the movement of the subject detected by the image processing means. The audio processing device described in claim 12, characterized in that the image processing means detects the frequency of speech, timing of speech, or mouth movements of the subject. An audio processing device according to any one of claims 1 to 13, characterized in that the determination means extracts the subject's audio based on the audio frequency, volume, and intonation. An audio processing device according to any one of claims 1 to 14, further comprising an imaging means for capturing the video. An audio processing device according to any one of claims 1 to 15, characterized in that the determination means determines that a subject that has been detected by the detection means has moved out of the angle of view of the video when the subject is no longer detected by the detection means. A method for controlling an audio processing device, comprising:
a detection step of detecting an object from a video;
a determining step of determining a subject's voice from the video;
a selection step of selecting a main subject from the subjects detected from the video;
a determining step of determining an object related to the main object selected in the selecting step,
In the determination step, when a subject related to the main subject moves out of the angle of view of the video, it is determined whether the subject that moves out of the angle of view of the video continues to be related to the main subject, based on the audio of the subject determined in the determination step . A computer-readable program for causing a computer to function as each of the means of the voice processing device described in any one of claims 1 to 15.