JP2014236276A - Image processing device, imaging device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of specifying a main subject with high accuracy.
An image acquisition unit that acquires a plurality of images that are continuously captured in time series, a sound acquisition unit that acquires sound recorded at the time of capturing a plurality of images and associated with the plurality of images, and a sound Voice information is extracted from the subject, and a subject estimation unit that estimates the main subject of the image from the voice information, and a subject specification unit that specifies the estimated main subject from the image.
[Selection] Figure 1

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Various techniques for detecting a main subject from captured images have been developed.

  For example, a saliency map is obtained by integrating a plurality of feature amounts from an image, and an image area having a saliency equal to or higher than a predetermined threshold that is likely to attract human visual attention in the saliency map is selected as a main subject. There is a technique for extracting a region of a main subject from an image by setting it as a seed of a core region and performing region division processing (see Patent Document 1, etc.)

JP 2011-35636 A

  However, in the prior art, since the main subject is detected based on a feature amount such as saliency, it may be different from the main subject intended by the user who captured the image.

  In view of the above-described problems of the conventional technology, an object of the present invention is to provide a technology capable of specifying a main subject with high accuracy.

  In order to solve the above-described problem, an aspect of an image processing apparatus illustrating the present invention includes an image acquisition unit that acquires a plurality of images that are continuously captured in time series, and a plurality of images that are recorded when a plurality of images are captured. An audio acquisition unit that acquires audio associated with the image, a subject estimation unit that extracts audio information from the audio and estimates a main subject of the image from the audio information, and a subject that identifies the estimated main subject from the image A specific unit.

  In addition, the subject specifying unit may specify the main subject from an image captured after the time when the audio information is extracted.

  Moreover, you may provide the speaker specific part which specifies the person who utters an audio | voice from an image.

  Further, an information adding unit that adds information on the main subject to the image in which the main subject is specified by the subject specifying unit may be provided.

  In addition, a storage unit that stores information in which the audio information and the main subject are associated in advance may be provided.

  An aspect of an imaging apparatus illustrating the present invention is an imaging unit that captures an image of a scene continuously in time series to generate a plurality of images, a focus adjustment unit that adjusts a focus state of the scene, The image processing apparatus of the present invention, and a control unit that controls the focus adjustment unit so as to focus on the main subject estimated by the subject estimation unit.

  One aspect of the image processing program illustrating the present invention is an image acquisition procedure for acquiring a plurality of images that are continuously captured in time series, and voices that are recorded when a plurality of images are captured and are associated with the plurality of images. The computer executes an audio acquisition procedure to be acquired, audio information is extracted from the audio, a subject estimation procedure for estimating the main subject of the image from the audio information, and a subject specifying procedure for specifying the estimated main subject from the image.

  According to the present invention, a main subject can be specified with high accuracy.

The figure which shows the structure of the digital camera which concerns on one Embodiment of this invention. 6 is a flowchart showing imaging processing by a digital camera according to an embodiment of the present invention. A figure showing an example of the composition of the object scene at the start of imaging Figure showing an example of the composition of the object scene after estimation of the main subject (part 1) A diagram showing an example of the composition of the object scene after estimation of the main subject (part 2) A diagram showing an example of the composition of the object scene after estimation of the main subject (part 3) FIG. 4 shows an example of the composition of the object scene after estimation of the main subject (Part 4) FIG. 5 shows an example of the composition of the object scene after estimation of the main subject (No. 5) A figure showing another example of the composition of the object scene at the start of imaging and after estimation of the main subject

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram showing a configuration of a digital camera 100 according to an embodiment of the present invention.

  The digital camera 100 of the present embodiment includes an imaging lens 11, an imaging device 12, an AFE 13, an image processing unit 14, microphones 15a and 15b, a display control unit 16a, a monitor 16b, an SDRAM 17, a lens driving unit 18, an imaging device driving circuit 19, The CPU 20, the operation member 24, the nonvolatile memory 25, the motion sensor 26, the recording I / F 27, and the bus 29 are included. The image processing unit 14, the microphones 15 a and 15 b, the display control unit 16 a, the SDRAM 17, the CPU 20, and the recording I / F 27 are connected via a bus 29 so as to be able to transmit information. The lens driving unit 18, the image sensor driving circuit 19, the operation member 24, the nonvolatile memory 25, and the motion sensor 26 are connected to the CPU 20.

  The imaging lens 11 includes a plurality of lens groups including a zoom lens and a focusing lens. The lens position of the imaging lens 11 is adjusted in the optical axis direction by the lens driving unit 18 based on a control instruction from the CPU 20. For simplicity, the imaging lens 11 is shown as a single lens in FIG.

  The image pickup device 12 is a device that is driven by the image pickup device drive circuit 19 and picks up an image of an object scene formed by the light flux that has passed through the image pickup lens 11 based on a control signal from the CPU 20. A plurality of light receiving elements are arranged in a matrix on the light receiving surface of the imaging element 12. In addition, red (R), green (G), and blue (B) color filters are arranged in each light receiving element of the imaging element 12 according to a known Bayer array. Therefore, each light receiving element of the imaging element 12 outputs an analog image signal corresponding to each color by color separation in the color filter. The output of the image sensor 12 is input to the AFE 13. Note that the image sensor 12 of the present embodiment may be a progressive scan type solid-state image sensor (CCD or the like) or an XY address type solid-state image sensor (CMOS or the like).

  The AFE 13 is an analog front end circuit that performs analog signal processing on the output of the image sensor 12. The AFE 13 performs correlated double sampling, image signal gain adjustment, and image signal A / D conversion. The output of the AFE 13 is sent to the image processing unit 14. In the present embodiment, the imaging device 12 and the AFE 13 constitute an imaging unit.

  The image processing unit 14 includes a white balance processing circuit, a pixel interpolation (demosaicing) circuit, a matrix processing circuit, a non-linear transformation (γ correction) processing circuit, an edge enhancement processing circuit, and the like. Processing such as balance, pixel interpolation, matrix, nonlinear transformation (γ correction), and contour enhancement is performed. Note that the pixel interpolation circuit converts a Bayer array signal of one color per pixel into a normal color image signal composed of three colors per pixel.

  The three color image signals output from the image processing unit 14 are stored in the SDRAM 17 through the bus 29. The image data stored in the SDRAM 17 is read out under the control of the CPU 20 and sent to the display control unit 16a. The display control unit 16a converts the input image data into a predetermined display signal (for example, an NTSC color composite video signal) and outputs it as a (through) image to be displayed on the monitor 16b.

  Also, image data acquired in response to a release button operation of the operation member 24 described later is read from the SDRAM 17 and sent to a compression / decompression processing unit (not shown). A compression / decompression processing unit (not shown) performs compression processing on the image data to generate an image file, and records the file on a memory card 28 as a recording medium via the recording I / F 27.

  The microphones 15a and 15b receive sound emitted from a user, a person in the scene, or the surroundings and convert the sound into an analog sound signal of an electrical signal. The analog audio signal is converted into a digital signal by an A / D conversion circuit (not shown), and then stored in the SDRAM 17 as audio data in a stereo format. A general microphone can be used as the microphones 15a and 15b. Further, at the time of reproducing a moving image or the like, the recorded audio signal can be reproduced by a speaker (not shown).

  The CPU 20 is a processor that comprehensively controls each unit of the digital camera 100 in accordance with a control program stored in the nonvolatile memory 25. For example, the CPU 20 executes automatic focus (AF) control by known contrast detection or phase difference detection, known automatic exposure (AE) calculation, and the like based on image data captured by the image sensor 12. Further, the CPU 20 of the present embodiment operates as the speaker specifying unit 21, the subject estimating unit 22, and the subject specifying unit 23 according to the image processing program stored in the nonvolatile memory 25.

  The speaker specifying unit 21 detects and specifies a speaker who speaks audio data used by the subject estimating unit 22 (to be described later) to estimate the main subject, using the image data captured by the image sensor 12. The speaker specifying unit 21 of the present embodiment specifies a speaker by applying a known subject detection processing algorithm to image data. For example, the speaker specifying unit 21 reads a template indicating the shape of a person stored in advance in the non-volatile memory 25 and performs pattern matching processing on the image data, thereby detecting a person's face area and serving as a speaker. Identify. For example, it is preferable that the CPU 20 sets the microphones 15a and 15b so that the voice from the specified speaker can be clearly acquired based on the result. When the face areas of a plurality of persons are detected, the speaker specifying unit 21 may specify all of the plurality of persons as speakers, or specify the person selected by the CPU 20 or the user as a speaker. May be. Further, when no person is detected from the image data, the speaker specifying unit 21 preferably specifies the user or a person outside the angle of view of the image data as the speaker. The speaker specifying unit 21 may be configured to specify (estimate) the speaker by performing voiceprint analysis on the audio signals acquired by the microphones 15a and 15b.

  The subject estimation unit 22 reads the voice data stored in the SDRAM 17 and extracts the utterance content of the speaker specified by the speaker specifying unit 21 from the voice data as voice information based on a known voice recognition algorithm. get. The subject estimation unit 22 estimates the main subject to be imaged next by the image sensor 12 from the acquired audio information based on the dictionary data stored in the nonvolatile memory 25. The dictionary data is a list of data in which the audio information and the main subject are associated with each other. That is, the dictionary data of the present embodiment corresponds to a main subject such as a flower or a rainbow together with a list of noun audio information representing each main subject such as a person, an animal, a flower, a vehicle, a building, and a mountain. It consists of a list of audio information such as adjectives such as “beautiful”.

  The subject specifying unit 23 performs, for example, subject detection processing on the image data captured by the image sensor 12 after the estimation by the subject estimation unit 22 and detects the image area of the main subject estimated by the subject estimation unit 22. Then identify. For example, the subject specifying unit 23 reads a template indicating the shape of the main subject estimated by the subject estimation unit 22 from the non-volatile memory 25 and performs pattern matching processing on the image data to detect the image area of the main subject. Identify. The CPU 20 performs the focus adjustment by driving the lens driving unit 18 to advance and retract the photographing lens 11 in the optical axis direction with the identified image area of the main subject as the focusing area. It is assumed that the nonvolatile memory 25 stores a template of the main subject registered in the dictionary data.

  The operation member 24 includes, for example, a release button, a power button, a command dial, a cross-shaped cursor key, a determination button, and the like. The operation member 24 receives various inputs of the digital camera 100 from the user. In the present embodiment, a touch panel that is configured by a transparent panel having the same shape as the monitor 16b and is stacked on the entire surface of the monitor 16b may be used as the operation member 24. That is, the touch panel may detect the position of a stylus (or fingertip or the like) that is in contact with the panel surface, and output the detected position information to the CPU 20 to accept an instruction input from the user.

  The motion sensor 26 detects the posture and movement of the digital camera 100 such as an acceleration sensor or an electronic gyro, and outputs a detection signal to the CPU 20.

  Next, the imaging operation by the digital camera 100 of the present embodiment will be described with reference to the flowchart of FIG. In this embodiment, the case where the moving image mode is set as the shooting mode will be described. However, the same applies to the case where a still image is captured in the continuous shooting mode or the case where a through image is captured during standby. is there.

  The CPU 20 receives a power-on instruction by turning on the power button on the operation member 23 by the user, and turns on the digital camera 100. The CPU 20 reads and executes the control program and the image processing program stored in the nonvolatile memory 25 and initializes the digital camera 100. When the CPU 20 accepts an imaging instruction for a release button full-press operation of the operation member 24 by the user, the CPU 20 causes the imaging device 12 to start imaging a moving image. At the same time, the CPU 20 causes the microphones 15a and 15b to receive sound and start acquiring sound data. Note that it is assumed that an object scene that is captured as a moving image at the start of imaging is an object scene 30 having a composition of only a person 31 as shown in FIG.

  In step S101, the speaker specifying unit 21 performs subject detection processing on the frame of the moving image captured by the image sensor 12, detects the face area of the person 31, and the audio signal acquired by the microphones 15a and 15b. In synchronization, it is determined that a part (mouth) of the detected face area is moving, and the person 31 is identified as the speaker.

  In step S102, the subject estimation unit 22 reads voice data recorded in the SDRAM 17, performs voice recognition processing, and extracts and acquires the utterance content of the person 31 as voice information. The subject estimation unit 22 estimates a main subject to be imaged next by the image sensor 12 based on the sound information and dictionary data stored in the nonvolatile memory 25.

Here, specific operations of the subject estimation unit 22 and the subject specification unit 23 will be described by taking the cases of the following five cases as examples.
[Case 1]
As shown in FIG. 4A, when the person 31 in the scene 30 speaks, for example, as “beautiful flower”, the subject estimation unit 22 performs the next imaging by the imaging device 12 in step S102. The subject is estimated to be “flower”.

  In step S <b> 103, the subject specifying unit 23 reads a “flower” template from the SDRAM 17 based on the estimation result of the subject estimation unit 22. The subject specifying unit 23 starts to detect “flower” as the main subject from the next frame imaged by the image sensor 12. Then, as a result of panning or zooming of the digital camera 100 by the user, when the imaging device 12 images the object scene 40 shown in FIG. 4B, the subject specifying unit 23 is not in front of the mountain behind the frame. The image area 42 of the flower 41 is detected and specified as the main subject.

In step S <b> 104, the CPU 20 performs focus adjustment by driving the lens driving unit 18 to advance and retract the photographing lens 11 in the optical axis direction with the identified image area 42 of the flower 41 as the focusing area. At the same time, the CPU 20 displays a through image of the frame on the monitor 16b and superimposes and displays an AF frame indicating that the image area 42 of the flower 41 is an in-focus area.
[Case 2]
As shown in FIG. 5A, in a meeting or the like, when a person 31 of the object scene 30 speaks, for example, “please see the next figure”, in step S102, the subject estimation unit 22 12, the main subject to be imaged next is estimated to be “Figure”.

  In step S <b> 103, the subject specifying unit 23 reads a template such as a graph from the SDRAM 17 based on the estimation result of the subject estimation unit 22. The subject specifying unit 23 starts to detect the “figure” that is the main subject from the next frame imaged by the image sensor 12. Then, when the imaging device 12 images the scene 50 including the graph 51 projected on a projector or the like as shown in FIG. 5B, the subject specifying unit 23 mainly uses the image area 52 of the graph 51 from the frame. Identify as a subject.

  In step S <b> 104, the CPU 20 uses the identified image area 52 of the graph 51 as the in-focus area, drives the lens driving unit 18, and advances and retracts the photographing lens 11 in the optical axis direction to perform focus adjustment. At the same time, the CPU 20 displays a through image of the frame on the monitor 16b and superimposes and displays an AF frame indicating that the image area 52 of the graph 51 is an in-focus area.

Note that, in specifying the “figure”, the subject specifying unit 23 may specify the image region of the “figure” using, for example, a known character recognition or graph recognition technique instead of a template such as a graph. .
[Case 3]
As illustrated in FIG. 6A, when the person 31 in the object scene 30 speaks, for example, “Wow, beautiful”, the subject estimation unit 22 is next imaged by the imaging element 12 in step S102. The main subject is presumed to be “clear”.

  In step S <b> 103, the subject specifying unit 23 reads templates such as “flowers” and “rainbows” associated as “clear” from the SDRAM 17 based on the estimation result of the subject estimation unit 22. The subject specifying unit 23 starts to detect “beautiful” as the main subject from the next frame imaged by the image sensor 12. Then, as a result of panning or zooming of the digital camera 100 by the user, when the image sensor 12 images the flower 41 shown in FIG. 4B or the rainbow 61 shown in FIG. The image area 42 of 41 and the image area 62 of the rainbow 61 are specified as main subjects.

  In step S104, the CPU 20 drives the lens driving unit 18 with the identified image area 42 of the flower 41 and the image area 62 of the rainbow 61 as the in-focus area, and advances and retracts the photographing lens 11 in the optical axis direction to adjust the focus. I do. At the same time, the CPU 20 displays a through image of the frame on the monitor 16b, and superimposes an AF frame indicating that the image area 42 of the flower 41 and the image area 62 of the rainbow 61 are in-focus areas.

Note that in specifying “beautiful”, the subject specifying unit 23 uses, for example, a high-saturation region and a bright region based on a saliency map instead of using a template such as “flower” or “rainbow”. May be specified as an image area. At this time, in the saliency map, it is preferable to increase the weight of color and brightness and decrease the weight of directional edge. Further, the CPU 20 sets the bright area in the specified high saturation area as the focusing area, but if there are a plurality of bright areas in the high saturation area, the CPU 20 sets the closest area or the largest image area as the focusing area. Is preferred. According to the present embodiment, even if the utterance content does not directly indicate the main subject of the next scene and is expressed in a form such as an adjective or an adjective verb, for example, Can be identified.
[Case 4]
As shown in FIG. 7A, when the person 31 in the object scene 30 speaks, for example, “Wow, early”, the subject estimation unit 22 captures the next image by the image sensor 12 in step S102. Estimate that the main subject is “fast”.

  In step S <b> 103, the subject specifying unit 23 reads a template such as “car”, “train”, or “airplane” associated as “fast” from the SDRAM 17 based on the estimation result of the subject estimation unit 22. The subject specifying unit 23 starts detecting the “early subject” that is the main subject from the next frame imaged by the image sensor 12. Then, as a result of panning or zooming of the digital camera 100 by the user, when the image pickup device 12 images the train 71 shown in FIG. 7B, the subject specifying unit 23 does not use the mountain behind but the train 71 running on the fence. Is specified as the main subject.

  In step S <b> 104, the CPU 20 performs focus adjustment by driving the lens driving unit 18 to advance and retract the photographing lens 11 in the optical axis direction with the identified image area 72 of the train 71 as the focusing area. At the same time, the CPU 20 displays a through image of the frame on the monitor 16b and superimposes and displays an AF frame indicating that the image area 72 of the train 71 is an in-focus area.

Note that in specifying “early thing”, the subject specifying unit 23 performs a correlation process between a frame and a frame one frame before, for example, instead of using a template such as “train”, and performs interframe difference. Alternatively, a motion vector may be calculated, and an “early” image region may be specified based on the interframe difference or the motion vector. However, when the user pans the digital camera 100 and captures the “early thing”, the relationship between the “early thing” and the inter-frame difference or motion vector in the background is reversed. In this case, for example, it is preferable that the subject specifying unit 23 determines the motion of the digital camera 100 itself based on the detection signal of the motion sensor 26 and specifies the “fast” image area. Further, it is preferable that the CPU 20 performs the continuous AF on the “fast” main subject.
[Case 5]
As illustrated in FIG. 8A, when the person 31 in the object scene 30 speaks, for example, “Wow, wide”, the subject estimation unit 22 is next imaged by the imaging element 12 in step S102. The main subject is estimated to be “wide”.

  In step S <b> 103, the subject specifying unit 23 reads a template such as “sky” or “sea” associated with “wide” from the SDRAM 17 based on the estimation result of the subject estimation unit 22. The subject specifying unit 23 starts to detect a “wide subject” that is the main subject from the next frame imaged by the image sensor 12. Then, as a result of panning of the digital camera 100 by the user or the like, when the imaging device 12 images the object scene 80 shown in FIG. 8B, the subject specifying unit 23 displays “sky” instead of “flowers” in the foreground. Identify as the main subject.

  In step S <b> 104, the CPU 20 performs single AF control at infinity so as to focus on “sky”, drives the drive unit 18, and advances and retracts the photographing lens 11 in the optical axis direction to perform focus adjustment.

  In step S <b> 105, the CPU 20 determines whether an instruction to end imaging is received from the operation of the operation member 24 by the user. When the CPU 20 receives an instruction to end imaging, the CPU 20 proceeds to step S106 (YES side). On the other hand, if the CPU 20 has not received an instruction to end imaging, the CPU 20 proceeds to step S101 (NO side) and performs the processing of steps S101 to S104 until an instruction to end imaging is received.

  In step S106, the CPU 20 reads the moving image data stored in the SDRAM 17 and subjected to image processing by the image processing unit 14, and sends the moving image data to a compression / decompression processing unit (not shown). A compression / decompression processing unit (not shown) performs a compression process on the moving image data to generate a moving image file, and records it on the memory card 28 as a recording medium via the recording I / F 27. The CPU 20 ends a series of processes.

  As described above, in the present embodiment, an image captured by the image sensor 12 is estimated by estimating the next main subject in advance based on the acquired audio data even though the next main subject is not in the image. Thus, the main subject can be identified with high accuracy and speed.

In addition, by preliminarily estimating and specifying the next main subject, the digital camera 100 can reliably focus on the main subject and can take an image in an optimal state.
<< Additional items of embodiment >>
(1) Although the reproduction processing apparatus of the present invention is realized by causing the CPU 20 of the digital camera 100 to execute an image processing program, the present invention is not limited to this. For example, the present invention can also be applied to a processing program for realizing the processing in the reproduction processing apparatus according to the present invention with an electronic device such as a smartphone having a computer or an imaging unit, and a medium on which the program is recorded.

  When the computer is operated as the image processing apparatus of the present invention, when the computer reproduces a moving image file read from the digital camera 100 or the like, for example, the subject estimation unit 22 receives the audio data added to the moving image file or the user. The subject identification unit 23 identifies the main subject estimated from the frame reproduced after the estimation by the subject estimation unit 22. Then, the CPU of the computer, as an information adding unit, associates information such as the identified main subject and the size and position of the image area with the frame and adds them to the header area of the moving image file. Thereby, it is possible to easily confirm what is captured as a main subject in each frame of the moving image, and editing of the moving image file can be facilitated.

  (2) In the above embodiment, the example in which the CPU 20 realizes the processing of the speaker specifying unit 21, the subject estimating unit 22, and the subject specifying unit 23 by software has been described. However, by hardware using an ASIC or the like. It may be realized.

  (3) In the above embodiment, the dictionary data stored in the non-volatile memory 25 is data in which speech information such as nouns and adjectives is associated with the main subject, but the present invention is not limited to this. For example, the dictionary data may be associated with a main subject such as a car or train using voice data itself from a car or train as voice information.

  For example, consider a case where the digital camera 100 is initially focused on a mountain of the object scene 90 shown in FIG. During the imaging, the microphones 15a and 15b of the digital camera 100 receive the sound of a train approaching the object scene 90, and the signal level is high, or the frequency component is high (due to the Doppler effect). If it has changed, the subject estimation unit 22 estimates that the main subject to be imaged next by the image sensor 12 is a train based on the received train audio information. The subject specifying unit 23 reads a train template from the SDRAM 17 and starts detection of the train from the next frame imaged by the image sensor 12. Then, when a train enters the object scene 90 shown in FIG. 9B without panning or zooming the digital camera 100, the subject specifying unit 23 uses the image area of the train 91 running in front as the main subject. The CPU 20 uses the image area 92 of the train 91 from the mountain as the in-focus area, drives the lens driving unit 18 to advance and retract the photographing lens 11 in the optical axis direction, and performs focus adjustment.

  Note that the subject specifying unit 23 may detect a main subject that enters the object scene from the direction in which the sound approaches, based on stereo audio data from the microphones 15a and 15b.

  (4) In the above-described embodiment, the microphones 15a and 15b may be arranged with microphones that receive the user's voice in addition to the voice inside and outside the object scene, in addition to the microphones 15a and 15b.

  (5) In the above embodiment, the example in which the main subject of the screen to be photographed next is specified based on the audio information acquired by the microphones 15a and 15b has been described. For example, the screen shown in FIG. If the person 31 recognizes the utterance of “please see this figure” when the person is photographed, the main subject may be changed from the person 31 to the graph 52 in the same photographing screen. Good.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Imaging lens, 12 ... Image sensor, 13 ... AFE, 14 ... Image processing part, 15a, 15b ... Microphone, 20 ... CPU, 21 ... Speaker specific part, 22 ... Subject estimation part, 23 ... Subject specific part, 100 …Digital camera

Claims (7)

An image acquisition unit that acquires a plurality of images that are continuously captured in time series;
An audio acquisition unit that acquires audio recorded during imaging of the plurality of images and associated with the plurality of images;
A subject estimation unit that extracts speech information from the speech and estimates a main subject of the image from the speech information;
A subject identifying unit for identifying the estimated main subject from the image;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The image processing apparatus according to claim 1, wherein the subject specifying unit specifies the main subject from an image captured after the time when the audio information is extracted. The image processing apparatus according to claim 1 or 2,
An image processing apparatus comprising: a speaker specifying unit that specifies a person who speaks the sound from the image. The image processing device according to any one of claims 1 to 3,
An image processing apparatus comprising: an information adding unit that adds information on the main subject to the image in which the main subject is specified by the subject specifying unit. The image processing apparatus according to claim 1, wherein:
An image processing apparatus comprising: a storage unit that stores information in which the audio information and the main subject are associated in advance. An imaging unit that continuously images in time series and generates a plurality of images;
A focus adjustment unit for adjusting a focus state of the object scene;
An image processing apparatus according to claim 1;
A control unit that controls the focus adjustment unit so as to focus on the main subject estimated by the subject estimation unit;
An imaging apparatus comprising: An image acquisition procedure for acquiring a plurality of images that are continuously captured in time series,
An audio acquisition procedure for acquiring audio recorded at the time of capturing the plurality of images and associated with the plurality of images;
Subject estimation procedure for extracting speech information from the speech and estimating a main subject of the image from the speech information;
A subject identification procedure for identifying the estimated main subject from the image;
An image processing program for causing a computer to execute.

Images