JP6808424B2 - Image recording device and its control method - Google Patents

Description

The present invention relates to an image recording device having a function of recording an image and magnifying and displaying a part of the image, and a control method thereof.

In an image pickup apparatus, it is very important to accurately focus on the subject in order to capture a desired image. In particular, when shooting a 4K or 8K image, it is difficult to accurately perform manual focus while visually observing a relatively small viewfinder or monitor provided in the image pickup apparatus. Therefore, a function to assist this is required. Note that 4K refers to 2160p (4096 x 2160) or a pixel count close to it, and 8K refers to 4320p (8192 x 4320) or a pixel count close to it.

Patent Document 1 proposes that it is easy to check the focus state of the details of a subject by displaying a moving image of the subject magnified at a predetermined magnification at the time of manual focus.

Japanese Unexamined Patent Publication No. 11-341331

As the number of pixels of image pickup sensors and video formats increases, the processing performance required for image recording devices such as image pickup devices, especially the read / write performance of the storage means (RAM) used for data processing in the device, increases. ing.

For example, in order to perform enlarged display as in Patent Document 1 with high quality, it is necessary to cut out a partial region of the RAW signal obtained from the image sensor, develop it, and pass the data to the display device. At this time, the RAW image Use storage means for temporary storage and reading and writing VRAM data. Further, in order to record moving image data conforming to a predetermined video format on a recording medium, read / write processing to a storage means is separately performed by development processing or encoding processing. Further, additional functions such as face detection and image transmission to an external device also use a storage means for each control.

If all of these processes are operated at the same time, the read / write performance of the storage means may be insufficient depending on the system. In that case, the development process and the encoding process cannot be completed within the specified time, resulting in dropped frames and damage to the recorded data.

The possibility of facing such a problem is higher in an image recording device that handles multi-pixel video data such as 4K and 8K. Further, in the case of high frame rate (slow motion) shooting, which has been widely used in recent years, it is necessary to perform signal processing for one frame in a shorter unit time. Assuming that high frame rate shooting and multi-pixel recording are combined for shooting, it can be said that this problem cannot be easily solved by improving the system performance in the near future.

Therefore, an object of the present invention is to provide an image recording device capable of performing high-definition enlarged display and a control method thereof by using the limited read / write performance of the storage means.

The present invention
It is not reduced by the acquisition means for acquiring an image, the storage means, the reduced image generation means for reducing the image acquired by the acquisition means to generate a reduced image and storing it in the storage means, and the reduced image generation means. A first cut-out image generation means that cuts out a part of the image to generate a first cut-out image and stores it in the storage means, and storage of data in the storage means or reading of data from the storage means. A processing means for performing a specific process to be accompanied, a process for image recording accompanied by storage in the storage means for the reduced image stored in the storage means by the reduced image generation means, and the processed and described. A recording processing means that performs recording processing including a process of recording an image stored in the storage means on a recording medium, and an image based on the reduced image stored in the storage means by the reduced image generating means are output to the display means. By doing so, it is controlled to perform the first display, and the image based on the first cutout image stored in the storage means by the first cutout image generation means is output to the display means. A display control means that controls to perform a second display that is larger than the display of 1, and at least, when the second display is performed during the recording process, the specific process is not executed. When the second display is not performed even during the recording process, it is characterized by having a control means capable of executing the specific process.

According to the present invention, high-definition enlarged display can be performed by using the limited read / write performance of the storage means.

It is a hardware block diagram and a functional block diagram of an image recording apparatus 100. It is a flowchart which shows the whole operation of the image recording apparatus 100. It is a flowchart of a setting change process. It is a flowchart of face detection and tracking processing. It is a flowchart of browser communication processing. It is a flowchart of a recording state change and a Magnify state change process. It is an output image from HDMI (registered trademark) 116 and a display example on a panel 118. This is an output image from HDMI (registered trademark) 116 and a display example on the panel 118 when the Magnify key is operated. This is a display example on the Web browser 126. This is an example of the internal configuration of the external output unit 115 and the panel output unit 117 in the modified example.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, as an image recording device 100 as an embodiment to which the present invention is applied, an image pickup device capable of capturing and recording a moving image will be given as an example.

FIG. 1A is a hardware block diagram showing an example of the internal configuration of the image recording device 100 according to the present embodiment. FIG. 1B is a functional block diagram showing a data path of image processing performed by the image recording device 100. In blocks 150, 151, and 173 of FIG. 1B, the number of pixels in each process when an image captured in 4K is recorded in 2K as an example is added. Needless to say, this does not apply when at least one of the number of pixels of the captured image and the number of pixels to be recorded are set to have different pixel numbers.

In FIG. 1A, the lens unit 101 has a function of correcting an imaging position moved by the movement of a fixed lens group, a variable magnification lens group, an aperture, and a variable magnification lens group for focusing, and a function of performing focus adjustment. It is composed of a correction lens group that combines and. The lens unit 101 finally forms a subject image on the image plane of the image sensor 102, which will be described later.

The image sensor 102 converts light into electric charges to generate an imaging signal. The generated imaging signal is output to the preprocessing unit 103. The image sensor 102 is an image sensor such as a CCD image sensor or a CMOS image sensor.

The preprocessing unit 103 converts the image pickup signal input and acquired from the image sensor 102 into RAW data (RAW image) that can be received by the RAW reduction unit 104 and the RAW cutout unit 105, and the RAW reduction unit 104 and the RAW cutout unit 105. Output to (block 150).

The RAW reduction unit 104 generates reduced RAW data (reduced RAW image) obtained by reducing the RAW data generated by the preprocessing unit 103 at a predetermined ratio, and outputs the reduced RAW data (reduced RAW image) to the image generation unit 106 (reduced image generation in block 151). ). For example, the RAW data of 4096x2160 pixels input from the preprocessing unit 103 is reduced to 1/2 both horizontally and vertically to generate 2048x1080 reduced RAW data, which is output to the image generation unit 106. If the data input from the preprocessing unit 103 is not reduced, the RAW data is output to the image generation unit 106 without the reduction processing.

The RAW cutout unit 105 cuts out a predetermined range of RAW data generated by the preprocessing unit 103 to a predetermined size, generates cutout RAW data (block 173), and stores it in the DRAM 113 (block 174). For example, the RAW data of 4096 × 2160 pixels input from the preprocessing unit 103 is cut out from the predetermined position of the RAW data of 2048 × 1080 pixels to generate the cut-out RAW data. The range cut out by the RAW cutout unit 105 can be changed by a user operation for changing the enlargement range (position / magnification) by the operation unit 110.

The image generation unit 106 performs RAW development processing such as interpolation processing and image quality adjustment processing on the reduced RAW data input from the RAW reduction unit 104 to generate YUV format full angle of view image data (block 152). It is stored in the DRAM 113 (block 153). The full angle of view image data is the developed data of the entire range of the captured image that has not been cut out.

Further, the cut-out RAW data input from the RAW cut-out unit 105 is read from the DRAM 113 and subjected to RAW development processing to generate cut-out image data (block 175), which is passed to the resolution conversion unit 2 (114). Since the image size that can be processed by the image generation unit 106 within a unit time is limited, the reduced RAW data and the cut out RAW data cannot be processed at the same time. Therefore, the cut-out RAW data is temporarily stored in the DRAM 113 (block 174) before being input to the image generation unit 106, so that the processing timing is shifted.

The resolution conversion unit 1 (123) performs resolution conversion of YUV data. The resolution conversion unit 1 (123) performs reduction / resizing processing on the full angle of view image data stored in the DRAM 113 to generate reduced image data for face detection (block 154), and stores the reduced image data in the DRAM 113 (block 155). ). Further, the full angle of view image data stored in the DRAM 113 is reduced and resized to generate JPEG compression reduced data (block 154), and the data is stored in the DRAM 113 (block 157).

The still image compression unit 107 encodes the reduced data for JPEG compression stored in the DRAM 113 in the JPEG format to generate the JPEG data (block 158), and stores the data in the DRAM 113 (block 159).

The network unit 108 transmits a Web application to the communication destination Web browser 126 using the HTTP protocol on wireless communication such as WiFi (registered trademark) or Bluetooth (registered trademark). The Web application is composed of HTML, JavaScript (registered trademark), and CSS data stored in ROM 125, which will be described later. In addition, the network unit 108 also functions to receive the processing request from the Web browser 126 and transmit it to the control unit 111. With this Web application, the Web browser 126 can function as a remote controller for the image recording device 100. Further, the network unit 108 can display the live view image in real time on the Web browser 126 by sequentially transmitting the JPEG data arranged in the DRAM 113 in response to the request from the Web browser 126 (block 160). .. These series of communication functions are collectively called browser remote.

The face detection unit 109 detects the position of the face from the reduced image data for face detection arranged in the DRAM 113 (block 156), and stores the position information of the face in the DRAM 113 (subject detection process). It also has a function to track a specific face among the detected faces. The face position information is used as position information when the face frame OSD data generated by the OSD unit 124 described later is combined with the external output image and the panel output image described later.

The focus / exposure control unit 127 automatically controls focus and exposure by driving a lens group and an aperture in the lens unit 101 based on the face position information detected by the face detection unit 109. In addition, the image generation unit 106 is instructed to control the luminance gain.

The operation unit 110 is for the user to input an operation, and is, for example, a touch panel or an operation key (button, dial, lever, etc.). When the user operates the operation unit 110, the operation information is notified to the control unit 111.

The control unit 111 is a processing unit composed of a CPU (Central Processing Unit) or the like, and controls each block included in the image recording device 100.

The RAM (Random Access Memory) 112 is a volatile memory used by the control unit 111 as a work area.

The DRAM (Dynamic Random Access Memory) 113 is a volatile memory used by each block of the image recording apparatus 100 as a work area. The DRAM 113 has an upper limit on the amount of data (read / write speed) that can be read / written (read / write) in a predetermined period, and data cannot be read / written beyond the upper limit. The DRAM 113 can be replaced with a high-speed volatile memory or non-volatile memory by a mechanism different from that of the DRAM. Further, the RAM 112 and the DRAM 113 may be configured to coexist on the same storage device.

The ROM (Read Only Memory) 125 is a non-volatile recording medium in which a program or the like executed by the control unit 111 is stored, and is composed of, for example, a Flash ROM. Alternatively, the program data stored in the recording medium 121 described later may be loaded into the RAM 112 and used as a ROM.

The on-screen display (OSD) unit 124 generates OSD data (display items) such as various setting menus and face frames, and arranges them in the DRAM 113.

The resolution conversion unit 2 (114) resizes the full angle-of-view image data stored in the DRAM 113 and the cut-out image data input from the image generation unit 106 to a resolution output to HDMI (registered trademark) 116 for external output. Is generated (block 161). Both the full angle of view image data and the cutout image data are developed YUV data. The generated external output image is stored in the DRAM 113 (block 162). Similarly, a panel output image resized to a resolution output to the panel 118 is generated (block 161) and stored in the DRAM 113 (block 165). Similarly, the moving image compression unit 119 generates a moving image recording image resized to a resolution for moving image compression (block 161) and stores it in the DRAM 113 (block 168).

The external output unit 115 synthesizes the external output image and OSD data stored in the DRAM 113 and outputs the external output signal to HDMI (registered trademark) 116 (block 163). The external output unit 115 cuts out and enlarges a part of the image for external output and outputs it as an external output signal to HDMI (registered trademark) 116. The cutout and enlargement function for external output (processes a modification described later). Function).

The HDMI (registered trademark) 116 changes the external output signal input from the external output unit 115 into the HDMI (registered trademark) format and outputs the signal to the outside (block 164).

The panel output unit 117 synthesizes the panel output image and the OSD data stored in the DRAM 113 and outputs the panel output signal to the panel 118 (block 166). The panel output unit 117 has a panel output cutout / enlargement function (a function for processing a modification described later) that cuts out and enlarges a part of the panel output image and outputs it to the panel 118 as a panel output signal. doing.

The panel 118 is a display panel such as a liquid crystal panel or an organic EL panel, and displays a panel output signal input from the panel output unit 117 (block 167).

The moving image compression unit 119 compresses the moving image recording image stored in the DRAM 113 in MPEG format and stores it in the DRAM 113 as moving image data (compression processing of block 169, block 170).

The media control unit 120 records the moving image data generated by the moving image compression unit 119 and stored in the DRAM 113 on the recording medium 121 according to a format compatible with a computer such as a FAT file system (blocks 171 and 172).

Examples of the recording medium 121 include a memory card and the like. This recording medium is a removable recording medium that can be removed from the image recording device 100, and can be attached to a PC or the like in addition to the image recording device 100.

The bus 122 is a data bus for each block of the image recording device 100 to exchange data, and each block of the image recording device 100 exchanges data via this bus.

The Web browser 126 is a Web browser provided in an external device different from the image recording device 100, executes a Web application received from the network unit 108, and makes a processing request to the image recording device 100 based on the received input. Can be issued. In addition, JPEG data can be sequentially received and a live view image can be displayed in real time. As the external device, for example, a smartphone, a desktop type / notebook type / tablet type PC (personal computer) that can be connected to the image recording device 100 wirelessly or by wire can be used. Further, as long as it has a Web browser function, it is not limited to these. Subsequently, the operation of the image recording apparatus 100 according to the present invention will be described based on the flowcharts of FIGS. 2 to 6. These flows are realized by the control unit 111 controlling each unit of the image recording device 100 based on the program stored in the ROM 125. 7 (A) to 7 (D) are display examples that are output to HDMI (registered trademark) 116 and displayed on the panel 118 during the operation of the flowcharts of FIGS. 2 to 4. 8 (A) to 8 (C) are display examples that are output to HDMI (registered trademark) 116 and displayed on the panel 118 during the operation of the flowchart of FIG. 9 (A) to 9 (C) are display examples displayed on the Web browser 126 during the operation of the flowchart of FIG. Each will be described together with this flow.

FIG. 2 is a flowchart showing the entire operation according to the present invention.

In S201, the control unit 111 changes various settings related to the image recording device 100 based on the user's operation. The details of this setting change process will be described later with reference to FIG.

In S202, the control unit 111 determines whether or not the operation of the face detection and tracking function is permitted and the face detection and tracking function is ON. If true, the process proceeds to S203, and if false, the process proceeds to S204. Whether or not the operation of the face detection and tracking functions is permitted / disallowed is determined based on whether or not the Magnify priority is set during recording, which will be described later in FIG. Further, ON / OFF of the face detection and tracking function is based on the state set by the user on the selection screen displayed by selecting the menu item related to the setting of the face detection and tracking function on the menu screen (details are shown in FIG. 3). Will be described later in S313).

In S203, the control unit 111 performs face detection and tracking processing. Details of face detection and tracking processing will be described later with reference to FIG.

In S204, the control unit 111 determines whether or not the browser remote setting is in the ON state. If true, proceed to S205, and if false, proceed to S206. The ON / OFF of the browser remote setting is based on the state set by the user on the displayed selection screen displayed by selecting the menu item related to the face detection and tracking function setting on the menu screen.

In S205, the control unit 111 performs browser communication processing. The details of the browser communication process will be described later with reference to FIG.

In S206, the control unit 111 performs a recording state change and a Magnify (enlarged display function for manual focus adjustment) state change process. Details of this process will be described later with reference to FIG.

In S207, the control unit 111 determines whether or not the recording control (recording process) of the moving image on the recording medium 121 is in operation (recording process). If S207 is true, the process proceeds to S208, and if S207 is false, the process proceeds to S209.

FIG. 7A shows an example of displaying an image output to HDMI (registered trademark) 116 when the determination of S207 is true (during recording control), and an example of displaying a shooting screen displayed on the panel 118. .. FIG. 7B shows an example of displaying an image output to HDMI (registered trademark) 116 and an example of displaying a shooting screen displayed on the panel 118 when the determination of S207 is false (waiting for recording). ..

The display item 701a (REC display) of FIG. 7A is a display item indicating that the state of the image recording device 100 is in the recording operation.

The display item 701b (STAND-BY display) of FIG. 7B is a state display indicating that the state of the image recording device 100 is in recording standby (stopped).

The display items 702 of FIGS. 7A and 7B are display items indicating the currently set imaging frame rate. The recorded data path and the Magnify data path, which will be described later, operate periodically in synchronization with the frame rate displayed here. Further, in FIGS. 7A and 7B, the entire LV image 700 is displayed in the background of the display item. The entire LV image 700 is a live view image in a non-enlarged state. The entire LV image 700 output to HDMI (registered trademark) 116 is an image for external output stored in the DRAM 113 in block 162, and the entire LV image 700 displayed on the panel 118 is for panel output stored in block 165. It is an image. The entire range of the captured image is displayed for each.

In S208, the control unit 111 operates the recorded data path to perform the following series of processing.
1. 1. Processing to store the moving image recording image resized by the resolution conversion unit 2 (114) in the DRAM 113 (processing of blocks 161 and 168).
2. 2. A process of reading a moving image recording image from the DRAM 113, compressing the moving image compression unit 119, and storing the moving image data in the DRAM 113 again (processing of blocks 168 to 170).
3. 3. Processing to read moving image data from DRAM 113 and record it on the recording medium 121 by the media control unit 120 (processing from block 170 to block 172).

Read / write to the DRAM 113 is generated for these series of processes.

In S209, the control unit 111 determines whether or not the image recording device 100 is in the Magnify operation (the Magnify operation will be described later with reference to FIG. 6). If true, the process proceeds to S210, and if false, the process returns to S201 and the entire step of this flow is repeated.

In S210, the control unit 111 operates the Magnify data path by cutting out. The Magnify data path by cutting out refers to the following series of processes.
1. 1. Processing to generate cut-out RAW data by the RAW cutting-out unit 105 and store it in the DRAM 113 (processing of blocks 150, 173, 174)
2. 2. Processing to read the cut-out RAW data from the DRAM 113 and convert (develop) it into the cut-out image data by the image generation unit 106 (process of blocks 174 and 175).
3. 3. Processing to generate an external output image and a panel output image from the cut-out image data by the resolution conversion unit 2 (114) (processing from blocks 161 to 162, processing from blocks 161 to 165).

Read / write to the DRAM 113 is generated for these series of processes.

FIG. 3 is a flowchart showing the details of the setting change process of S201.

In S301, the control unit 111 determines whether or not an operation for requesting the operation unit 110 to display the menu screen has been performed. If true, the process proceeds to S302, and if false, the flow ends.

In S302, the control unit 111 displays the menu screen on the panel 118 and outputs it to HDMI (registered trademark) 116 by the OSD unit 124.

In S303, the control unit 111 determines whether or not a menu item for setting whether to prioritize the Magnify operation is selected during recording by the user input from the operation unit 110 when the menu screen is displayed. To do. If true, the process proceeds to S304, and if false, the process proceeds to S310.

In S304, the control unit 111 outputs a selection screen for setting whether to prioritize the Magnify operation during recording to HDMI (registered trademark) 116 and displays it on the panel 118. FIG. 7C shows a display example of a selection screen for setting whether to prioritize the Magnify operation during recording.

The screen title 703 is a title display indicating that the currently displayed screen is a selection screen for setting to prioritize the Magnify operation during recording.

Message 704 is a message display indicating that the face detection and tracking function and the live view function of the browser remote cannot be used when this function is enabled. This display allows the user to know in advance what features will be restricted in exchange for this feature.

The button 705 is a button for turning off this setting, and the button 706 is a button for turning on this setting. The operation unit 110 can select either the button 705 or the button 706 to determine the operation. The setting state determined here is stored in the RAM 112. The entire LV image 700 is displayed in the background. It should be noted that the entire LV image 700 may not be displayed on the selection screen of the setting that prioritizes the Magnify operation during recording.

In S305, the control unit 111 determines whether or not the setting that prioritizes the Magnify operation is enabled during recording by the user's selection operation for the operation unit 110. If true, proceed to S306, if false, proceed to S308. When the button 706 is selected and the determination operation is performed, it is determined to be true, and when the button 705 is selected and the determination operation is performed, it is determined to be false.

In S306, the control unit 111 prohibits the use of the live view function of the browser remote.

In S307, the control unit 111 prohibits the use of the face detection and tracking functions. If the use of the face detection and tracking function is prohibited by this step, the menu item for setting the face detection and tracking function cannot be selected or hidden on the menu screen, and the setting cannot be performed.

In S308, the control unit 111 permits the use of the live view function of the browser remote (makes it executable).

In S309, the control unit 111 permits the use of the face detection and tracking functions (makes them executable).

The prohibited or permitted states of the functions determined in S306 to S309 are all stored in the RAM 112.

In S310, the control unit 111 reads out the state stored in the RAM 112 and determines whether or not the face detection and tracking functions are permitted. If true, proceed to S311. If false, proceed to S316.

In S311 the control unit 111 determines whether or not the menu item for setting the face detection and tracking function is selected by the input from the operation unit 110 to the menu screen. If true, the process proceeds to S312, and if false, the process proceeds to S316.

In S312, the control unit 111 displays the face detection and tracking function selection screen on the panel 118 and outputs it to HDMI (registered trademark) 116. On this screen, options for enabling the face detection and tracking functions and options for disabling the face detection and tracking functions are displayed, and one of them can be selected by the operation unit 110.

In S313, the control unit 111 determines whether or not the face detection and tracking functions are turned on by the operation on the operation unit 110. If true, the process proceeds to S314, and if false, the process proceeds to S315.

In S314, the control unit 111 turns on the face detection and tracking functions.

In S315, the control unit 111 turns off the face detection and tracking functions.

The operating states (ON / OFF) of the functions determined in S314 and S315 are both stored in the RAM 112.

In S316, the control unit 111 determines whether or not the menu item for setting the browser remote function is selected by the input from the operation unit 110 to the menu screen. If true, the process proceeds to S317, and if false, the process proceeds to S321.

In S317, the control unit 111 displays the selection screen of the browser remote function. On this screen, options for enabling the browser remote function and options for disabling the browser remote function are displayed, and one of them can be selected by the operation unit 110.

In S318, the control unit 111 determines whether or not the browser remote function is turned on by the operation on the operation unit 110. If true, the process proceeds to S319, and if false, the process proceeds to S320.

In S319, the control unit 111 turns on the browser remote function.

In S320, the control unit 111 turns off the browser remote function.

The operating states (ON / OFF) of the functions determined in S319 and S320 are both stored in the RAM 112.

In S321, the control unit 111 determines whether or not the menu screen end is requested by the operation on the operation unit 110. If true, the process proceeds to S322, and if false, the process returns to S303.

In S322, the control unit 111 exits the menu screen and transitions to the shooting screen.

FIG. 4 is a flowchart showing details of the face detection and tracking process of S203.

In S401, the control unit 111 generates face detection reduced image data by the resolution conversion unit 1 (123) and stores it in the DRAM 113. This corresponds to the processing of blocks 154 to 155 described above.

In S402, the control unit 111 is a process of detecting the position of the face from the reduced image data for face detection stored in the DRAM 113 and a process of detecting the moving destination of the face to be tracked. This process corresponds to the process of block 156 described above.

Read / write to DRAM 113 occurs due to the processing of S401 and S402.

In S403, the control unit 111 performs focus control by the focus / exposure control unit 127 based on the position of the face detected in S402.

In S404, the control unit 111 performs exposure control by the focus / exposure control unit 127 based on the position of the face detected in S402.

In S405, the control unit 111 updates the display position of the face frame displayed on the HDMI (registered trademark) 116 and the panel 118 by the OSD unit 124 based on the position of the face detected in S402. A display example of the face frame is shown in FIG. 7 (D).

The face frame 707 and the face frame 708 are displayed on the detected face so as to be superimposed on the entire LV image 700. The face frame 707 represents the main face, and the face frame 708 represents a face other than the main face. The main face can be changed by input from the operation unit 110, and the focus control by the focus / exposure control unit 127 is performed on the main face.

FIG. 5 is a flowchart showing details of the browser communication process of S205.

In S501, the control unit 111 determines whether or not the connection between the network unit 108 and the Web browser 126 is established. If true, the process proceeds to S504, and if false, the process proceeds to S502.

In S502, the control unit 111 determines whether or not the network unit 108 has received the connection request from the Web browser 126. If true, the process proceeds to S503, and if false, the flow ends.

In S503, the control unit 111 performs a connection process between the network unit 108 and the Web browser 126 to establish the connection. The connection process also includes the process of transmitting HTML and JavaScript® and CSS data for executing a browser remote Web application on the Web browser 126.

In S504, the control unit 111 determines whether or not a request for acquiring the state of each unit of the image recording device 100 has been received from the Web browser 126. The state of each part includes, for example, an operating state such as whether or not a recording operation is being performed, various parameters related to focus / exposure control, and the remaining free space of the recording medium 121. It also includes the permission status of the browser remote live view function set by S306 or S308. All of these states are stored in the RAM 112. If this step is true, the process proceeds to S505, and if this step is false, the process proceeds to S506.

In S505, the control unit 111 reads the state information requested in S504 from the RAM 112 and transmits it via the network unit 108.

In S506, the control unit 111 determines whether or not the update request of the live view image of the browser remote is received from the Web browser 126. If the live view image is being displayed on the web browser 126, the update request for the live view image is periodically transmitted from the web browser 126. If this step is true, the process proceeds to S507, and if this step is false, the process proceeds to S510.

In S507, the control unit 111 determines whether or not the live view function of the browser remote is permitted based on the state set in S306 or S308 and stored in the RAM 112. If true, proceed to S508, if false, proceed to S510.

In S508, the control unit 111 generates reduced data for JPEG compression by the resolution conversion unit 1 (123) and stores it in the DRAM 113, and further performs a process of JPEG-compressing the data by the still image compression unit 107 and storing it in the DRAM 113. Do. This process corresponds to the process of blocks 154, 157, 158, 159 described above.

In S509, the control unit 111 reads the JPEG data generated in S508 from the DRAM 113 and transmits it to the Web browser 126. This process corresponds to the process of blocks 159 and 160 described above.

Due to the processing of S508 and S509, read / write to the DRAM 113 is generated.

In S510, the control unit 111 determines whether or not a processing request has been received from the Web browser 126. The processing request includes, for example, a request for starting and stopping recording, a request for changing various parameters related to focus / exposure control, a request for specifying the main face and the tracking target in the face detection and tracking function, and the like. If this step is true, the process proceeds to S511, and if it is false, this flow ends.

In S511, the control unit 111 controls each unit of the image recording device 100 to perform processing in response to the processing request received in S510.

9 (A) to 9 (C) show an example of displaying the screen displayed on the Web browser 126 while the communication with the Web browser 126 is established.

FIG. 9A is a display example when the image recording device 100 is in the live view permitted state and is in the live view display state on the Web browser 126 side.

In the area 901, a live view image in which the JPEG data transmitted from the image recording device 100 in S509 is expanded is displayed.

The button 902 is a button for switching whether or not to display the live view. When this button is pressed (touched or clicked), the state shown in FIG. 9B is obtained.

FIG. 9B is a display example when the image recording device 100 is in the live view permitted state and the live view is not displayed on the Web browser 126 side.

The area 901 is an area in which the live view image is displayed, but is hidden in this state.

The button 902 is a button for switching whether or not to display the live view. When this button is pressed, the state shown in FIG. 9A is reached.

FIG. 9C is a display example when the image recording device 100 is in the live view prohibited state.

The area 901 is an area in which the live view image is displayed, but in this state, the JPEG data is not transmitted from the image recording device 100, so that the area 901 is hidden.

The button 902 is a button for switching whether or not to display the live view, but since the image recording device 100 has received the fact that the live view is prohibited, the button 902 is in an inoperable state (grayed out display).

Since the face selection and tracking functions on the remote browser issue a processing request to the image recording device 100 based on the touch operation on the live view image, the face selection and tracking functions can be operated only when the live view image is displayed. When the live view image is displayed on the Web browser 126 side, the face can be selected as the tracking target by touching or clicking the face of the subject on the displayed live view image. Information indicating the coordinates of the touched or clicked position is transmitted to the image recording device 100, and the face based on the coordinate information received by the image recording device 100 is targeted for tracking. Since processing requests related to other functions can be issued regardless of the display state of the live view image, the operation on the Web browser 126 is not restricted.

FIG. 6 is a flowchart showing the details of the recording state change and the Magnify state change process of S206.

In S601, the control unit 111 determines whether or not the REC key included in the operation unit 110 has been pressed. Alternatively, if the browser remote function is being executed, whether or not a processing request equivalent to pressing the REC key is received from the Web browser 126 in response to the reception of the recording start / stop instruction operation on the Web browser 126 side. judge. If this step is true, the process proceeds to S602, and if this step is false, the process proceeds to S608.

In S602, the control unit 111 determines whether or not the image recording device 100 is in the video recording operation. If true, proceed to S603, if false, proceed to S604.

In S603, the control unit 111 stops the moving image recording operation of the image recording device 100.

In S604, the control unit 111 determines whether or not the Magnify function is being executed. If true, proceed to S605, if false, proceed to S607.

In S605, the control unit 111 determines whether or not the Magnify operation is prioritized during recording based on the information selected and set in S304 of FIG. 3 (state information stored in the RAM 112). .. If true, proceed to S607, if false, proceed to S606.

In S606, the control unit 111 cancels the Magnify operation. When the Magnify operation is canceled, the entire LV image 700 is displayed, and the display is not enlarged (non-enlarged display in which the entire image fits). In addition, the display item 801a (described later in FIG. 8A) or 801c displayed on the screen of the panel 118 indicating that Magnify is being executed is deleted.

Even if S605 is true, it may be a flow for executing S606. That is, even during the execution of Magnify (enlarged display), the enlarged display is temporarily released when the moving image recording is started. By doing so, it is possible to make the user recognize that the recorded moving image is not only the portion enlarged and displayed by Magnify but also the range of the full angle of view image. After the start of recording the moving image, if there is an operation for instructing the execution of Magnify during recording, Magnify is performed again.

Alternatively, if S605 is true, the process proceeds to S607 to start recording while being enlarged, a warning is displayed that Magnify is being executed, and display items 801a and 801c described later are blinked for a certain period of time. You may. That is, the display may be performed to emphasize that the enlarged display is in progress. By doing so, it is possible to prevent the user from misunderstanding that recording is performed at a magnified display angle of view (a part of the entire angle of view image, not the entire range).

In S607, the control unit 111 starts the recording operation (recording control) of the image recording device 100. As a result, the same control as in S208 described above is started.

In S608, the control unit 111 determines whether or not the Magnify key included in the operation unit 110 has been pressed. If true, the process proceeds to S609, and if false, the flow ends.

In S609, the control unit 111 determines whether or not the image recording device 100 is executing the Magnify function. If true, proceed to S610, if false, proceed to S612.

In S610, the control unit 111 releases the Magnify operation in the same manner as in S606.

In S611, the control unit 111 erases the display item 801a (described later in FIG. 8A) or 801c displayed on the screen of the panel 118 indicating that the Magnify is being executed.

In S612, the control unit 111 determines whether or not the image recording device 100 is in the recording operation, similarly to S602. If true, the process proceeds to S613, and if false, the process proceeds to S614.

In S613, the control unit 111 determines whether or not the Magnify operation is prioritized during recording, as in S605. If true, the process proceeds to S614, and if false, the process proceeds to S616.

In S614, the control unit 111 starts the Magnify operation and controls the display so as to perform an enlarged display as shown in FIG. 8A described later.

In S615, the control unit 111 outputs a display item 801a (described later in FIG. 8A) indicating that Magnify is being executed to HDMI (registered trademark) 116 and displays it on the screen of the panel 118.

FIG. 8A shows a display example when Magniffy is executed. The enlarged LV image 804 is a live view image in which a part of the entire range (the range of the total angle of view image data) that has been captured is enlarged and displayed. As described above, this image is an image obtained by reducing an image cut out from RAW data according to an enlarged range according to the number of pixels of the panel 118 or the number of pixels of HDMI (registered trademark). Therefore, as long as the number of pixels of the cut-out RAW data is sufficient for the number of pixels of the panel 118, the enlargement (the process of stretching to increase the number of pixels from the original number of pixels) is not performed during the processing. Since the image quality is less deteriorated and the display is high-definition, it is easy to check the focus condition during manual focus. The display item 801a is an icon indicating that Magnify is being executed by cutting out, and is displayed superimposed on the enlarged LV image 804 by the processing of S615. This icon is displayed in yellow to emphasize that the enlarged LV image 804 displayed on the screen is a part of the whole and is different from the angle of view (range) of the recorded image.

The rectangle 802 and the rectangle 803 are radar displays for indicating a cutout range with respect to the entire image, that is, the position of the currently displayed image area. The rectangle 802 shows the entire image (the range of the entire LV image 700). The rectangle 803 indicates a cutout range (range of the enlarged LV image 804) in the entire image.

The position and magnification of the enlargement range in the Magnify can be changed by the user's operation on the operation unit 110. If there is an operation to change the position of the enlargement range or an operation to change the enlargement magnification during the execution of Magnify, the display of the rectangle 803 is updated so as to indicate the position of the changed enlargement range and the enlargement magnification. Further, the RAW cutout unit 105 updates the magnified LV image 804 by cutting out a range corresponding to the changed magnified range. Further, while the Magnify function is being executed (during enlarged display), it is possible to accept a manual focus operation (MF operation) from the user on the operation unit 110. Based on the received MF operation, the focus / exposure control unit 127 drives the lens unit 101. In some cases, the focus lens of the lens unit 101 cannot be driven by the focus / exposure control unit 127. In this case, or even when the lens can be driven by the focus / exposure control unit 127, the position of the focus lens is directly adjusted by directly operating the lens operation unit such as the focus ring provided in the lens unit 101.

On the other hand, in S616, the control unit 111 outputs the display item 801b indicating that the Magnify cannot be executed to HDMI (registered trademark) 116 and displays it on the screen of the panel 118. A display example in this case is shown in FIG. 8 (B). The display item 801b is an icon indicating that the Magnify could not be executed, and is added to and displayed on the entire LV image 700 displayed before the Magnify key is pressed in S608. This icon is displayed in red to emphasize that the function could not be executed in response to the key operation in S608. The display item 801b may be automatically hidden when a predetermined time has elapsed from the start of display. Further, the display item 801b may be displayed at all times when the Magnify cannot be executed regardless of whether or not the Magnify key is pressed.

According to the present embodiment, when the Magnify operation is set to be prioritized during recording, it is possible to control not to execute the face detection and tracking function and the live view function of the browser remote. Thereby, it is possible to avoid (reduce) the read / write processing of the DRAM 113 consumed by the operation of the face detection and tracking function and the live view function of the browser remote. Therefore, even if the Magnify function and the moving image recording process are operated at the same time, the read / write performance of the DRAM 113 can be prevented from being insufficient. As a result, it is possible to record video in a predetermined format including high load conditions such as multiple pixels and a high frame rate while accurately performing manual focus using the Mafnyfy function.

Whether or not to prioritize the Magnify operation during recording can be selected from the menu, so if you want to use the face detection and tracking function or the live view function of the browser remote, change the priority function by this menu setting. Can be done.

Further, according to the present embodiment, when the Magnify operation is set to be prioritized during recording, the face detection and tracking function and the browser remote can be used regardless of whether or not the Magnify function or the moving image recording process is executed. The use of the live view function is restricted. By doing so, the content of the function restriction can be clearly shown to the user as shown in FIG. 7C, and the user can be prevented from being confused by complicated restriction conditions. On the other hand, the use of the face detection and tracking function and the live view function of the browser remote may be restricted only during the period when the Magnify function or the moving image recording process is being executed. That is, when the magnified display by the Magnify function is not performed and / or the moving image recording process is not executed, the use of the face detection and tracking function and the live view function of the browser remote may be enabled without restriction. In this way, the use of the face detection and tracking function and the live view function of the browser remote can be restricted for a minimum period of time.

In the present embodiment, the functions (limited functions) that are exclusive to the Magnify operation during recording are the face detection and tracking function and the live view function of the browser remote, but the present invention is not limited to this. If there is another function of the DRAM 113 that involves read / write processing, that function may be excluded. As a result, it is possible to make the face detection and tracking function and the live view function of the browser remote compatible with the Magnify operation during recording, depending on the performance of the system.

<Modification example>
As a modification, when the Magnify operation is not set to be prioritized during recording (when No is determined in S613), instead of displaying the display item 801b in S616, a simple enlargement that does not use the cutout image data is used. It may be displayed. A display example in this case is shown in FIG. 8 (C). The simple enlarged LV image 805 is a live view image in which a part of the entire range (the range of the full angle of view image data) that has been captured is enlarged and displayed. The simple enlarged LV image 805 is generated as follows.

-Image to be output to HDMI (registered trademark) 116: An image cut out from the external output image (data stored in block 162) stored in the DRAM 113 according to the enlarged range is output from HDMI (registered trademark) 116. The magnification is changed according to the number of pixels to be used.

-Image to be displayed on the panel 118: An image cut out from the panel output image (data stored in the block 165) stored in the DRAM 113 according to the enlargement range is scaled according to the number of pixels of the panel 118.

The external output image of the block 162 or the panel output image of the block 165 is an image that has already been reduced in the block 151 according to the setting of the number of recorded pixels and further reduced in the block 162. From here, the number of pixels of the image cut out according to the enlarged range is smaller than the number of recorded pixels. Since it is scaled according to the number of pixels of the panel 118 or the number of pixels of HDMI (registered trademark) 116, the number of pixels of the image cut out according to the enlarged range is the number of pixels of the panel 118 or the pixels of HDMI (registered trademark). If it is less than the number, it will be expanded (stretching process). Since the image quality is deteriorated by that amount, the image becomes coarser than the above-mentioned enlarged LV image 804. The enlarged LV image 804 generates an external output image and a panel output image from the RAW data before the reduction process (processing of the block 151) is performed by the RAW reduction unit 104.

On the other hand, in the Magnify operation by simple enlargement, an external output image and a panel output image are generated from the reduced RAW data after the reduction process is performed by the RAW reduction unit 104. Therefore, when the image range enlarged and displayed on the HDMI (registered trademark) 116 or the panel 118 is the same, the enlarged LV image 804 has higher definition than the simple enlarged LV image 805.

However, the display of the simple enlarged LV image 805 can be realized at least without performing the process of storing the cut-out RAW data having a relatively large amount of data in the DRAM 113 (the process of the block 174). The amount of read / write data to the DRAM 113 can be reduced accordingly. Therefore, the simplified magnified LV image 805 displays the data for face detection and / or the data for JPEG compression for the browser remote LV image without restriction in the read / write (block 155/157) processing to the DRAM 113. There is a merit that it can be done. The display item 801c is an icon indicating that Magnify is being executed by simple enlargement. The icon is different from the display item 801a to indicate that it is a simple enlargement. Further, in order to emphasize that the image displayed on the screen (simple enlarged LV image 805) is a part of the whole and is different from the angle of view (range) of the recorded image as in the display item 801a. , This icon is displayed in yellow.

As described above, according to the modified example, when the face detection and tracking function and other functions such as the live view function of the browser remote are prioritized, it is possible to prevent the Magnify function from becoming unusable. In addition, the Magnify function can be used by the simple enlarged LV image 805 in combination with other functions such as the live view function of the browser remote during recording, and the enlarged LV image 804 is displayed during the recording standby. It is possible to perform a fine enlarged display.

Since the purpose of the simple enlarged display is to reduce the amount of read / write data to the DRAM 113 during recording, when the recording stop operation is performed, the enlarged display using the cut-out image data can be performed. become. Therefore, when the recording stop operation of S603 is performed in the state where the simple enlarged display is performed, the simple enlarged display is canceled. Following the cancellation of the simple enlarged display accompanying the recording stop operation, the enlargement may be canceled and the entire LV image 700 may be displayed, or the Magnify operation itself may be maintained and the simple enlarged display may be displayed on the enlarged LV image 804. You may switch directly.

Further, when the REC key is operated while the enlarged LV image 804 is displayed, if the Magnify operation is not set to be prioritized during recording, the processing of S606 is not performed (that is, the Magnify is canceled). You may switch to the simple enlarged LV image 805.

Further, in the modified example, it is possible to configure the configuration so that the setting itself of whether or not to give priority to the Magnify operation during recording is not performed. That is, other functions such as the face detection and tracking function and the live view function of the browser remote can be executed regardless of the Magnify operation. In the case where the setting itself of whether or not to prioritize the Magnify operation is not performed during recording, when it is determined in S612 described above that recording is in progress (S612Yes), the simple expansion LV805 is performed without performing the processing of S613. indicate. Further, in the case of S604Yes, the simple enlarged LV805 is displayed without performing the processing of S605. In the case of S604Yes, if the enlargement LV804 is directly switched to the simple enlargement LV805, the change in image quality may be noticeable, and the timing shift at the time of switching the path of the enlargement processing may occur. In order to prevent these, even if the configuration itself does not set whether or not to prioritize the Magnify operation during recording, the Magnify operation may be temporarily canceled and the LV image 700 may be displayed in the case of S604Yes.

In addition, in order to improve the image quality of the above-mentioned simple enlarged display, super-resolution processing may be performed. The scaling process for simple enlarged display in this case will be described. FIG. 10 shows an example of the internal configuration of the external output unit 115 or the panel output unit 117. The DRAM_I / F1001 reads the image data from the DRAM 113. The resolution conversion unit 1002 performs scaling processing to increase the number of pixels by referring to peripheral pixels and interpolating the image data read by DRAM_I / F1001. The super-resolution processing unit 1003 performs scaling processing to increase the number of pixels by interpolating pixels with reference to a plurality of frames (frames before and after) of the image data read by DRAM_I / F. The selector 1004 outputs the data read by the DRAM_I / F 1001 as it is to the subsequent stage (processing after the peaking processing unit 1005), outputs the data processed by the resolution conversion unit 1002, or the super-resolution processing unit 1003. Select whether to output the processed data. The peaking processing unit 1005 performs peaking processing on the image acquired via the selector 1004. The peaking process is a process of coloring the outline portion of a subject in an image with colored lines (red, blue, yellow, etc.). The outline in the image is highlighted in different colors depending on the focus state. The highlight color is for display and does not affect the recorded video. The selector 1006 selects whether to output the data that has been peaked by the peaking processing unit 1005 or the data that has not been peaked received from the selector 1004 to the output signal generation 1007. The output signal generation unit 1007 converts the image received via the selector 1006 into a data format that can be received by HDMI (registered trademark) 116 or panel 118. As described above, the scaling process (enlargement processing) for the simple enlarged display is performed by either the scaling process by the resolution conversion or the scaling process by the super-resolution processing. When pixels are interpolated by referring to the previous and next frames by super-resolution processing, it is possible to display a higher-definition image than the normal electronic zoom enlargement. Therefore, it is easy to visually check whether or not the subject is in focus, so that manual focus can be performed more accurately. In addition, it is possible to make it easier to focus by performing the peaking process during the simple enlarged display.

The various controls described above as those performed by the control unit 111 may be performed by one hardware, or the entire device may be controlled by sharing the processing among a plurality of hardware.

Further, although the present invention has been described in detail based on the preferred embodiment thereof, the present invention is not limited to these specific embodiments, and various embodiments within the scope of the gist of the present invention are also included in the present invention. included. Furthermore, each of the above-described embodiments is merely an embodiment of the present invention, and each embodiment can be combined as appropriate.

Further, in the above-described embodiment, the case where the present invention is applied to the image recording device 100 having the image sensor 102 has been described as an example, but this is not limited to this example, and the input image is enlarged and displayed. It can be applied to any electronic device that is controlled in this way. For example, it can be applied to a moving image recorder in which a moving image input from an external input terminal is enlarged and output to a display device and recorded on an external or built-in recording medium. Similarly, the present invention is applicable to personal computers, PDAs, mobile phone terminals, portable image viewers, printer devices equipped with displays, digital photo frames, and the like. Further, it can be applied to music players, game machines, electronic book readers, tablet terminals, smartphones, projection devices, home appliances equipped with displays, in-vehicle devices, and the like.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (21)

The acquisition method to acquire the image and
Memories and
A reduced image generation means that reduces the image acquired by the acquisition means to generate a reduced image and stores it in the storage means.
Generating a first cut-out image by cutting out a portion of the image which is not reduced by the reduced image generation unit, a first cut-out image generating means for storing in said memory means,
A processing means that performs a specific process that involves storing data in the storage means or reading data from the storage means, and
For the stored the reduced image in the storage unit by the reduced image generation unit, and the processing of the image recording with the storage in the storage means, the processed image stored in the storage means the recording medium Recording processing means for performing recording processing including processing for recording in
The reduced image generation means controls to perform the first display by outputting an image based on the reduced image stored in the storage means to the display means, and the first cutout image generation means controls the storage means. A display control means for controlling the display so as to perform a second display larger than the first display by outputting an image based on the first cut-out image stored in the display means to the display means.
At least , when the second display is performed during the recording process , control is performed so that the specific process is not executed, and when the second display is not performed even during the recording process, the second display is not performed. An image recording device including a control means capable of executing the specific process . Further having a setting means for setting whether or not to enable the second display by the image based on the first cutout image during the recording process based on the user's selection operation.
When the control means is not effectively set by the setting means, the control means controls so as not to execute the second display during the recording process, and executes the specific process during the recording process. The image recording apparatus according to claim 1, wherein the image recording device is controlled so as to enable it. When the second display during the recording process is effectively set by the setting means, the control means controls not to execute the specific process regardless of whether or not the recording process is in progress. The image recording apparatus according to claim 2. When the second display during the recording process is effectively set by the setting means, the control means controls so that the specific process is not executed during the recording process, and the recording The image recording apparatus according to claim 2, wherein the image recording device is controlled so that the specific process can be executed during the standby period during which the process is not being performed. When the second display during the recording process is effectively set by the setting means, the control means does not execute the specific process regardless of whether the second display is performed. The image recording apparatus according to claim 2 or 3, wherein the image recording device is controlled. When the second display during the recording process is effectively set by the setting means, the control means controls so as not to execute the specific process when the second display is performed. The image recording apparatus according to claim 2 or 4, wherein when the second display is not performed, control is performed so that the specific process can be executed. In response to there is an operation for instructing the execution of the second display when the second display is not enabled during Hence the recording process in the setting unit, wherein the display control unit, the without displaying the 2, image recording apparatus according to any one of claims 2 to 6, wherein the controller controls so as to perform a display indicating that the second display is disabled .. Further having a second cutout image generation means for cutting out a part of the reduced image to generate a second cutout image.
When the second display in the result in the recording process in the setting means is not enabled, the display control means, an image based on the second cut-out image in the recording process on the display means The image recording apparatus according to any one of claims 2 to 6, wherein the image is controlled to be simply enlarged and displayed by outputting the image. The image recording device according to claim 8, wherein the display control means controls to end the simple enlarged display when the recording process is completed during the simple enlarged display. The display control means controls the simple enlarged display so as to display an image that has undergone super-resolution processing to increase the number of images by interpolation processing that refers to the frame images before and after the second cropped image. The image recording apparatus according to claim 8 or 9. The display control means is characterized in that, in the simple enlarged display, the second cropped image is controlled to display an image that has been subjected to a peaking process for coloring an outline portion in the image. Item 6. The image recording apparatus according to any one of Items 8 to 10. The display control means controls the simple enlarged display so as to display an enlarged display different from the second display, which displays an image based on the first cropped image. The image recording apparatus according to any one of claims 8 to 11. Any one of claims 1 to 12, wherein the specific process is at least one of a detection process for detecting a specific subject from the image and a transmission process for transmitting the image to an external device. The image recording apparatus described in 1. The image recording apparatus according to any one of claims 1 to 13, further comprising an imaging means, wherein the acquiring means acquires an image captured by the imaging means. The image recording device according to any one of claims 1 to 14, wherein the image is a moving image. The image recording apparatus according to any one of claims 1 to 15, wherein the image recording process includes a compression process. The image recording apparatus according to any one of claims 1 to 16, wherein the reduced image is a reduced RAW image obtained by reducing the RAW image acquired by the acquisition means. The image recording apparatus according to claim 17, wherein the first cut-out image is a cut-out RAW image obtained by cutting out a part of the RAW image acquired by the acquisition means. The acquisition step to acquire the image and
A reduced image generation step of reducing the image acquired in the acquisition step to generate a reduced image and storing it in a storage means .
Generating a first cut-out image by cutting out a portion of the image which is not reduced by the reduced image generating step, a first cut-out image generating step makes stored in the storage means,
A processing step of performing a specific process involving storing data in the storage means or reading data from the storage means, and
For the stored the reduced image in the storage unit, the processing of the image recording with the storage in the storage means, and a process of recording the processed image stored in the storage means on a recording medium A recording process step that performs recording processing including
A first display control step for controlling to perform the first display by outputting an image based on the reduced image stored in the storage means to the display means .
A second display control step of controlling the display means to output an image based on the first cut-out image stored in the storage means so as to perform a second display enlarged from the first display. When,
At least , when the second display is performed during the recording process , control is performed so that the specific process is not executed, and when the second display is not performed even during the recording process, the second display is not performed. A control method for an image recording device, which comprises a control step that enables execution of the specific process. A program for causing a computer to function as each means of the image recording apparatus according to any one of claims 1 to 18 . A computer-readable storage medium containing a program for causing the computer to function as each means of the image recording apparatus according to any one of claims 1 to 18 .

Images