KR102861769B1 - Image processing method and appratus - Google Patents

Abstract

An electronic device according to various embodiments of the present disclosure includes a camera module, a sensor module, a memory, and a processor operatively connected to the camera module, the sensor module, and the memory, wherein the processor is configured to convert motion-related data of the electronic device obtained from the camera module and the sensor module into blur detection data, and detect a frame in which blur occurs in an imaging operation according to a first frame rate based on the converted blur detection data, and when the detected frame is a base frame, remove the first base frame in which blur is detected, and generate a second base frame to replace the removed first base frame, thereby processing an image according to the second frame rate.
Many other embodiments may be possible.

Description

Image processing method and apparatus {IMAGE PROCESSING METHOD AND APPRATUS}

Various embodiments of the present disclosure relate to image processing methods and devices.

As the number of users taking photos or videos using electronic devices increases, the performance of the cameras included in these devices is also improving.

Recently, considering the quality of the results taken by cameras embedded in electronic devices, various image stabilizers can be applied to the cameras of electronic devices.

Conventional electronic devices utilize image stabilizers to prevent hand shake when using a camera, thereby minimizing the resulting blur in photos or videos. Electronic devices can perform image stabilization through hardware, including image stabilizers, and/or software.

When a user uses a camera in a dark space or in an area with insufficient light, the processor of the electronic device may adjust the shutter speed to compensate for the lack of light entering the camera module. For example, if the shutter speed is adjusted slowly, the amount of movement of the electronic device, such as due to the user's hand shaking, increases, and the amount of image shake (e.g., photo or video) increases, which may lead to frequent image stabilization operations or processing. If the image stabilization operation or processing of the electronic device increases, the resulting photograph may become blurry or distorted.

Various embodiments may provide a method and device for processing shake correction of an image (e.g., a photograph or video) in an electronic device. Furthermore, various embodiments may provide a method and device for image processing in an electronic device.

An electronic device according to various embodiments of the present disclosure includes a camera module, a sensor module, a memory, and a processor operatively connected to the camera module, the sensor module, and the memory, wherein the processor is configured to convert motion-related data of the electronic device obtained from the camera module and the sensor module into blur detection data, and detect a frame in which blur occurs in an imaging operation according to a first frame rate based on the converted blur detection data, and when the detected frame is a base frame, remove the first base frame in which blur is detected, and generate a second base frame to replace the removed first base frame, thereby processing an image according to the second frame rate.

An image processing method of an electronic device according to various embodiments of the present disclosure may include an operation of converting movement-related data of the electronic device obtained from a camera module and a sensor module into blur detection data, an operation of detecting a frame in which blur occurs in an imaging operation according to a first frame rate based on the converted blur detection data, an operation of removing a first base frame in which blur is detected when the detected frame is a base frame, and an operation of generating a second base frame to replace the removed first base frame and synthesizing the second base frame according to the second frame rate.

A storage medium storing commands according to various embodiments of the present disclosure may be configured to cause at least one processor to perform at least one operation when the commands are executed by at least one processor, wherein the operation may include: converting motion-related data of an electronic device obtained from a camera module and a sensor module into blur detection data; detecting a frame in which blur occurs in an imaging operation according to a first frame rate based on the converted blur detection data; removing a first base frame in which blur is detected when the detected frame is a base frame; and generating a second base frame to replace the removed first base frame and synthesizing the second base frame according to the second frame rate.

According to various embodiments of the present disclosure, the frame rate of a photograph may be changed to provide additional frames to compensate for blurring that may occur due to image stabilization performed during photographing using an electronic device. For example, the electronic device may perform photographing at a high frame rate.

In addition, when shooting with an electronic device, frames that are likely to cause image degradation (e.g., blur) due to shake correction are identified and removed based on blur detection data containing information on camera motion and/or gyro motion, and a smooth image can be displayed by replacing the removed frames with frames that are temporally continuous with the removed frames.

FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.
FIG. 2 is a block diagram illustrating a camera module according to various embodiments.
FIG. 3 is a block diagram of an electronic device according to various embodiments.
FIG. 4 is a flowchart illustrating an operation method of an electronic device having an image processing function according to various embodiments.
FIG. 5A is an exemplary diagram of base frame extraction when an electronic device executes a capturing operation according to various embodiments.
FIG. 5b is an exemplary diagram of blur detection of a base frame during an imaging operation of an electronic device according to various embodiments.
FIG. 6A is an exemplary diagram of second data acquisition of a base frame when an electronic device executes a capturing operation according to various embodiments.
FIG. 6b is an example diagram of the use of second data of a base frame that is replaced when an imaging operation of an electronic device is performed according to various embodiments.
FIG. 7A is an exemplary diagram of a motion change amount according to a first frame rate when an electronic device performs a shooting operation according to various embodiments.
FIG. 7b is an example diagram of a motion change amount after peak frame removal due to a motion change amount according to a first frame rate when an electronic device executes a shooting operation according to various embodiments.
FIGS. 8A and 8B are exemplary diagrams of a method for image processing by removing and/or generating a base frame when executing an imaging operation of an electronic device according to various embodiments.

FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with the electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input device (150), an audio output device (155), a display device (160), an audio module (170), a sensor module (176), an interface (177), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the display device (160) or the camera module (180)), or may have one or more other components added. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, a sensor module (176) (e.g., a fingerprint sensor, an iris sensor, or an ambient light sensor) may be implemented embedded in a display device (160) (e.g., a display).

The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may load a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) into a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor), and a secondary processor (123) (e.g., a graphics processing unit, an image signal processor, a sensor hub processor, or a communication processor) that may operate independently or together therewith. Additionally or alternatively, the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part of it.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display device (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)).

The memory (130) can store various data used by at least one component (e.g., a processor (120) or a sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., a program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a non-volatile memory (134).

The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input device (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input device (150) can include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

The audio output device (155) can output audio signals to the outside of the electronic device (101). The audio output device (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback, and the receiver may be used to receive incoming calls. In one embodiment, the receiver may be implemented separately from the speaker or as part of the speaker.

A display device (160) can visually provide information to an external party (e.g., a user) of an electronic device (101). The display device (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display device (160) may include touch circuitry configured to detect a touch, or a sensor circuit (e.g., a pressure sensor) configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through an input device (150), or output sound through an audio output device (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or headphone)) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) to an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (388) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device via a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to identify and authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module may include one antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., an RFIC) may be additionally formed as a part of the antenna module (197).

At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the electronic devices (102, 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that have received the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

Electronic devices according to various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the aforementioned devices.

The various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish the corresponding component from other corresponding components, and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" as used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more commands stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store ^™ ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. According to one embodiment, the image stabilizer (240) can detect the movement of the camera module (180) or the electronic device (101) by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display device (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

The image signal processor (260) can perform one or more image processing operations on an image acquired through the image sensor (230) or an image stored in the memory (250). The one or more image processing operations can include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) can perform control (e.g., exposure time control, readout timing control, etc.) on at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., the memory (130), the display device (160), the electronic device (102), the electronic device (104), or the server (108)). According to one embodiment, the image signal processor (260) may be configured as at least a part of the processor (120) or may be configured as a separate processor that operates independently of the processor (120). When the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display device (160) by the processor (120) as is or after undergoing additional image processing.

According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

FIG. 3 is a block diagram of an electronic device according to various embodiments.

Referring to FIG. 3, an electronic device (e.g., an electronic device (101) of FIG. 1) according to various embodiments may include a processor (310) (e.g., a processor (120) of FIG. 1), a camera module (320) (e.g., a camera module (180) of FIG. 1), a memory (330) (e.g., a memory (130) of FIG. 1), and a sensor module (340) (e.g., a sensor module (176) of FIG. 1), and may include at least a part of the structure and/or function of the electronic device (101) of FIG. 1.

According to various embodiments, the processor (310) may include an image signal processor (321) (e.g., the image signal processor (260) of FIG. 2), and may include at least some of the structure and/or function of the processor (120) of FIG. 1 and/or the image signal processor (260) of FIG. 2. The image signal processor (321) may be included in the form of an auxiliary processor, for example, within a camera module (320) (e.g., the camera module (180) of FIG. 1) or may be provided in the form included in the main processor.

According to one embodiment, the processor (310) may perform a function of controlling the overall operations of the electronic device. For example, when performing an imaging operation such as capturing an image based on the operation of a camera application, the processor (310) may detect the movement of the electronic device (e.g., shaking caused by the user's hand shaking, shaking caused by a shooting method, etc.) through the sensor module (340) (e.g., the sensor module (176) of FIG. 1). In addition, the processor (310) and/or the image signal processor (321) (e.g., the image signal processor (260) of FIG. 2) may perform control to correct the detection of the movement of the electronic device. In addition, according to various embodiments, the processor (310) and/or the image signal processor (321) of the electronic device may perform control to correct an image obstruction phenomenon or deterioration phenomenon (e.g., blur phenomenon, jitter phenomenon).

For example, control to reduce image artifacts may involve performing image processing. Image processing can refer to the process of applying techniques to manipulate photographs or images. Image processing techniques can utilize a variety of methods, including analog, optical, or digital processing.

There are several methods that can be used to process images (e.g., prevent image shake) due to the movement of such electronic devices. For example, optical image stabilization (OIS) may include a method of physically compensating for shake using a hardware element (e.g., an image stabilizer (240) of FIG. 2), and video digital image stabilization (VDIS or DIS) or electrical image stabilization (EIS) may include a method of compensating for image shake by performing software processing on the output value of an image sensor (e.g., an image sensor (230) of FIG. 2).

For example, in the case of optical image stabilization (OIS), when shake is detected in the camera module (320), a hardware element for compensating for it may operate. For example, a camera lens of an electronic device (e.g., a lens assembly (210) of FIG. 2) can follow the movement (e.g., motion) of the electronic device (e.g., a camera) to maintain the focus of the subject and keep the amount of light entering the image sensor constant. In this way, OIS, which is shake compensation through a hardware element, can physically or mechanically move the lens (e.g., the lens assembly (210) of FIG. 2) to maintain the horizontal level according to the movement of the electronic device and/or the camera module (320) detected using a sensor module (340) (e.g., a gyro sensor, an acceleration sensor, and/or a gesture sensor, etc.). In addition, OIS can compensate for shake by having a lens (e.g., a lens assembly (210) of FIG. 2) and an image sensor (e.g., an image sensor (230) of FIG. 2) move together.

For example, VDIS (or DIS), a digital image stabilization system, extracts motion vectors based on the differences between frames in a video (different images) and can enhance clarity through image processing. Furthermore, by extracting motion vectors based on the video, VDIS can recognize subject movement itself as shake, in addition to shake itself.

For example, electronic image stabilization (EIS) uses a gyro sensor to extract the amount of shake, and subsequent shake compensation can be done in the same way as VDIS.

Referring to FIG. 3, according to various embodiments, the camera module (320) may include the components illustrated in FIG. 2 and may include at least some of the structures and/or functions of the camera module (180) of FIG. 1 and the camera module of FIG. 2.

According to various embodiments, the image signal processor (321) may perform control to correct image blurring or degradation. According to one embodiment, the image signal processor (321) may use optical image stabilization (OIS), which is easy to correct high frequency motion, and video digital image stabilization (VDIS), which is easy to correct low frequency motion. OIS may provide an effect of reducing motion of an input image by moving in the opposite direction of the movement of an electronic device (e.g., a camera) to make the motion less within the correction range, and VDIS may perform shake correction by combining individual motions obtained through the input image and gyro data, so that the shake of the input image can be used as is. Optical shake correction and/or digital shake correction may be performed by adjusting the field of view (FOV) of the input/output.

The VDIS related to image shake reduction of the present disclosure can detect the amount of shake by digital signal processing of the image, identify the characteristics of the image, and control the system to compensate for the shake. For example, the shake compensation algorithm can be a method of estimating a frame motion vector (FMV) from a local motion vector (LMV) between two frames continuously as a shake compensation parameter. Motion estimation and compensation are basic technologies of video compression, and motion estimation can use a block matching algorithm (BMA), a gradient matching algorithm (GMA), phase correlation, etc. For example, when a user of an electronic device performs a shooting operation, the VDIS can detect the amount of shake of the electronic device, identify the characteristics of the captured image, and ultimately compensate for the shake of the image.

Image jitter may occur when performing image stabilization.

Blur can refer to the phenomenon where the edges of objects within an image are blurred and appear blurred, resulting in a lack of focus. For example, blur can be referred to as image blur, motion blur, or blur.

Blur can occur for a variety of reasons. For example, it can occur when the subject's movement is faster than the shutter speed. Another example is when the camera is shooting in a dark or low-light environment (e.g., indoors), and the electronic device (camera) slows down the shutter speed to compensate for the lack of light.

Image jitter is a phenomenon that can occur when performing shake correction operations or processing such as OIS and/or VDIS to correct images with repeated blur.

For example, jitter can occur in indoor environments with insufficient lighting or in dark areas. Solutions to improve jitter include using a high-performance image sensor to increase shutter speeds even in areas with limited lighting, using optical image stabilization (OIS) in conjunction with VDIS to eliminate shaking in the incoming image, or setting VDIS compensation to a low level to minimize jitter itself.

According to various embodiments, the image signal processor (321) may receive movement-related data of the electronic device from a sensor module (340) that detects an image and/or movement of the electronic device during a shooting operation based on the operation of a camera application, and collects movement-related data of the electronic device by detecting an image from an image sensor that may be included in a camera module when performing a shooting operation. In addition, the image signal processor (321) may convert the received information (e.g., image and/or movement-related data) into blur detection data and, based on the blur detection data, detect a frame in which blur occurs in the shooting operation. According to one embodiment, if the frame in which blur occurs is a base frame, the image signal processor (321) may remove the frame and generate a base frame to replace it.

The image signal processor (321) according to various embodiments may control shake correction through software based on movement-related data of the electronic device received from the camera module (320) and/or the sensor module (340). The image signal processor (321) may control the movement-related data of the electronic device to be converted into blur detection data including information on camera motion and/or gyro motion for each frame in accordance with a first frame rate (e.g., 120 fps). Hereinafter, the first frame rate may be used interchangeably with, for example, a shooting frame rate.

The image signal processor (321) according to various embodiments may collect blur detection data including information about camera motion and/or gyro motion for each frame while performing a shooting operation according to a first frame rate, and control the detection of a frame in which blur occurs. The image signal processor (321) may control the detection of a frame in which blur occurs if the range of change in camera motion and/or gyro motion of an individual frame shot according to the first frame rate exceeds a preset range of change based on the blur detection data. For example, the range of change in the preset camera motion and/or gyro motion used as a reference by the image signal processor (321) may be set in a table as a range of change that may cause jitter to occur during shake correction due to repeated blur, and may be changeable by a user's setting.

The image signal processor (321) according to various embodiments can, if a frame in which blur is detected exists, determine whether the detected image (frame) corresponds to a base frame. The base frame may refer to frames displayed at a second frame rate (e.g., 30 fps). Hereinafter, the second frame rate may be used interchangeably with the output frame rate, for example. For example, the image signal processor (321) may set the first frame rate to 30 fps or higher so that the second frame rate is displayed at 30 fps. In order to generate an image output at 30 fps, the example of this document sets the shooting to be performed through a basic operation in which the first frame rate is 120 fps during the image shooting operation of the electronic device. The image signal processor (321) can identify the first frame among four consecutive frames shot at 120 fps as a base frame, and can extract each base frame to ultimately display an image according to a second frame rate of 30 fps. For example, if the image signal processor (321) generates an image with frames obtained through the basic operation of 120 fps, the amount of blur may be reduced due to the short exposure time between frames, but the image may become darker and noise may be generated compared to 30 fps due to the short exposure time.

When an electronic device's image capture operation is performed at a higher frame rate, the shutter speed can be faster, thereby reducing jitter caused by blur. According to various embodiments, jitter caused by image stabilization in an electronic device can be detected based on the image itself or a gyro sensor.

Detection of jitter based on the image itself can be done by checking, for example, through filtering between input images, and since the image sharpness is reduced when there is a lot of blur, jitter can be detected by measuring the degree of sharpness of the entire image.

Detection of jitter based on a gyro sensor can be performed by distinguishing the conditions and occurrence sections in which jitter occurs during shake correction using gyro motion, for example. For example, a static condition does not generate jitter, and a panning condition does not have a significant impact on the entire image due to jitter because it moves consistently in one direction. Detection of jitter can be performed based on data for frame-by-frame blur detection (e.g., frame-by-frame camera motion and/or gyro motion information) that is matched to the first frame rate, for example, in a walking or running situation excluding a static or horizontal rotation condition, since jitter occurs when VDIS is performed in a part where the slope of the motion changes abruptly, when extracting frame-by-frame camera motion and/or gyro motion according to the first frame rate. In the present disclosure, jitter can be prevented by detecting blur, which can be used as a jitter detection index, and when blur is detected based on frame-by-frame gyro data, at least three or more gyro data can be extracted per frame to calculate gyro motion and detect blur.

An image signal processor (321) according to various embodiments may detect whether a base frame is blurred according to a second frame rate. The image signal processor (321) may remove a blur detection base frame that may cause jitter and generate a base frame to replace the removed base frame. The base frame generated by the image signal processor (321) may be generated using gyro data of the removed base frame and image information (e.g., video information) of a frame having time-series continuity. For example, the process of the image signal processor (321) generating a replacement base frame may be to replace the removed base frame by calculating and synthesizing an average value from a frame having time-series continuity, or to directly replace the frame having time-series continuity, or to use both methods.

An image signal processor (321) according to various embodiments may be responsible for controlling the imaging operation of an electronic device. For example, the image signal processor (321) may control an image sensor (e.g., image sensor (230) of FIG. 2) and/or an image stabilizer (e.g., image stabilizer (240) of FIG. 2) used for compensation (e.g., OIS, VDIS, etc.) to prevent image shaking due to movement of the electronic device.

The image storage unit (322) (e.g., memory (250) of FIG. 2) can store, for example, images obtained through a photographing operation of an electronic device, and can store frames obtained through image sensors (e.g., image sensors (230) of FIG. 2) having a first frame rate (e.g., shooting frame rate) set to 120 fps as an example in the present disclosure. Using the frames stored in the image storage unit (322), the image signal processor (321) can generate, for example, a base frame to be replaced.

According to various embodiments, the memory (330) may include at least some of the structure and/or function of the memory (130) of FIG. 1.

According to various embodiments, the sensor module (340) may include at least a part of the structure and/or function of the sensor module (176) of FIG. 1. For example, the sensor module (340) may include a gyro sensor and may be controlled to detect movement of the electronic device (e.g., shaking caused by a user's hand shaking, shaking caused by a photographing method, etc.). According to another embodiment, the sensor module (340) may be controlled to transmit movement-related data of the electronic device (e.g., second data) generated as a result of detecting the movement of the electronic device to the image signal processor (321) (e.g., transmission of the detection result).

FIG. 4 is a flowchart illustrating an operation method of an electronic device having an image processing function according to various embodiments. For example, it is a flowchart illustrating an image shake prevention function in an electronic device (e.g., the electronic device (101) of FIG. 1). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two or more operations may be performed in parallel. Hereinafter, the operation of the electronic device according to various embodiments of the present disclosure may be performed by a processor of the electronic device (e.g., the processor (120) of FIG. 1, the processor (310) of FIG. 3, or the image signal processor (321) of FIG. 3).

Referring to FIG. 4, the processor may control a camera module (e.g., the camera module (180) of FIG. 1, the camera module (320) of FIG. 3) to perform shooting at a first frame rate in operation 401. For example, the first frame rate of the electronic device may be set to 120 fps. When shooting is performed at the first frame rate of 120 fps, if only the first frame (e.g., the base frame) of every four consecutive frames is listed, the result of the video shooting may be displayed at 30 fps.

The processor may control the sensor module (e.g., the sensor module (176) of FIG. 1, the sensor module (340) of FIG. 3) to detect movement-related data of the electronic device in operation 402. The sensor module may include, for example, a gyro sensor, an acceleration sensor, and/or a gesture sensor. The movement-related data of the electronic device may include, for example, first data based on an image being captured (e.g., data detected through an image sensor) and/or second data based on the movement of the electronic device (e.g., data detected through the sensor module). Hereinafter, the first data may include image edge data, and the second data may include gyro data. The processor may control shooting at regular time intervals (e.g., same time interval, same distance interval) in accordance with the first frame rate of operation 401.

Referring to FIG. 4, the processor may convert motion-related data (e.g., first data or second data) of an electronic device into blur detection data at a first frame rate in operation 403. The processor may convert motion-related data of the electronic device into blur detection data that may include information about camera motion and/or gyro motion for each frame at the first frame rate.

Among the blur detection data of operation 403, information regarding camera motion may correspond to, for example, an indicator of camera motion for each frame collected according to the first frame rate. Furthermore, the processor may calculate camera motion. For example, camera motion may refer to the movement of an electronic device that occurs during the electronic device's imaging operation, and may be expressed as a camera trajectory or movement path.

Information about gyro motion among the blur detection data of operation 403 may be, for example, collecting frame motion. Frame motion may be a concept including image motion and gyro motion, and frame motion may be utilized in VDIS for shake correction, and may have 30 fps or 60 fps as a basic operation. An image stabilizer (e.g., image stabilizer (240) of FIG. 2) that may be included in a camera module of an electronic device may perform VDIS, and a processor (e.g., processor (120) of FIG. 1, image signal processor of FIG. 2, image signal processor (321) of FIG. 3) may control the image stabilizer. Frame motion may be understood as motion between base frames in a display image of base frames that are arranged according to a second frame rate. This may be acquired by utilizing a vector through image motion and gyro motion.

Referring to FIG. 4, the processor may detect blur for each frame based on blur detection data at operation 404 according to a second frame rate. For example, in order to display an image at a second frame rate of 30 fps, four frames may be grouped together when shooting at a shooting frame rate of 120 fps. The base frame may refer to the first frame of a grouped unit, which may vary depending on how the starting point of the grouped unit is set, and may be easily changed through customization.

Referring to FIG. 4, the processor can detect whether a blur exists in the base frame at operation 405. The reason for detecting whether a blur exists in the base frame may be that if a blur occurs in the base frame, there is a high probability that jitter will occur in the image due to shake correction.

Referring to FIG. 4, if the processor determines that blur exists in the base frame in operation 405, the processor can remove the base frame in which blur exists in operation 406. If blur exists in the base frame, there is a high probability that jitter will occur in the image, and therefore, the processor can remove the base frame in which blur exists through operation 406.

Referring to FIG. 4, if the processor determines that no blur exists in the base frame at operation 405, the processor may terminate without performing any additional operations.

Referring to FIG. 4, the processor may generate a base frame to replace the base frame removed through operation 406 in operation 407. For example, the processor may generate and replace a new base frame to be displayed in an image in place of the removed base frame. According to one embodiment, the processor may generate the base frame to be replaced based on a frame that has temporal continuity with the removed base frame, and the gyro data of the removed base frame may be used as is in the generated base frame. In addition, the image information (e.g., image information) of the base frame to be replaced may be obtained by synthesizing an average value of frames that have temporal continuity, may be replaced by directly using the frame that has temporal continuity, or both methods may be used.

Although not shown in FIG. 4, the processor can perform digital gain interpolation for frames darkened by shake compensation. The processor can calculate a rolling shutter compensation (RSC) value. The processor can use gyro data to check whether an image displayed at a second frame rate (e.g., 30 fps) is in sync with the base frame through RSC. The gyro data utilized by the processor can utilize the gyro data of the removed base frame as is, even if the removed base frame exists.

FIGS. 5A and 5B are exemplary diagrams for a method of extracting a base frame for displaying an image according to a capturing operation performed by an electronic device. According to various embodiments, a processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the image signal processor (321) of FIG. 3) of an electronic device (e.g., the electronic device (101) of FIG. 1) may control capturing using a camera module (e.g., the camera module (180) of FIG. 1, the camera module (320) of FIG. 3). According to various embodiments, when the electronic device performs capturing of an image, the processor may detect the presence or absence of blur for each individual frame according to a first frame rate (e.g., 120 fps).

For example, the processor can obtain gyro motion through a sensor module (e.g., sensor module (176) of FIG. 1, sensor module (340) of FIG. 3). The processor can obtain a camera trajectory for each frame using the gyro motion. According to one embodiment, the processor can determine that blur exists in a frame when there is a section of rapid change (e.g., a section where the slope of the motion changes significantly) in the camera trajectory according to the frame number (e.g., a number sequentially numbered according to the passage of time). For example, the section of rapid change in the camera trajectory described above may correspond to a range of change in camera motion and/or gyro motion that may be included in the blur detection data as described in the description of FIG. 3. In addition, the processor can determine the presence or absence of blur for each frame according to a first frame rate to detect a blur frame in which jitter may occur, and can determine whether there is blur in a base frame according to a second frame rate (e.g., 30 fps).

FIG. 5A is an example diagram of base frame extraction when an electronic device executes a capturing operation according to various embodiments.

Referring to FIG. 5A, an electronic device according to various embodiments may perform capturing at a first frame rate when capturing an image (e.g., during a capturing operation) based on the execution of a camera application. For example, in order to set the frame rate of an image to be output through the electronic device to 30 fps, the actual capturing may be performed at a frame rate of 120 fps, and the image may be output at 30 fps through image synthesis and/or generation via the processor of the electronic device.

Referring to FIG. 5A, the imaging operation of an electronic device according to various embodiments may be performed at a second frame rate of 120 fps. Each of the frames (510 to 580) illustrated in FIG. 5A may represent one frame. In order for an image output through an electronic device to be reflected at 30 fps, a method may be used in which the image being captured at 120 fps is divided into one group of four frames each, and a base frame (e.g., frame 1 (510), frame 5 (550)) for each group is extracted and listed. The base frame may mean the first frame of each group.

Referring to FIG. 5A, when capturing a video, a starting point (e.g., the first base frame) for outputting a 30fps video may be determined by the default settings of the electronic device or may be changed through customization. For example, the moment at which extraction begins by capturing frame 1 (510) of FIG. 5A may be set as the starting point. In another embodiment, frame 5 (550) may be used as the starting point, and in order to obtain a video at a second frame rate (e.g., 30fps) by capturing at a first frame rate (e.g., 120fps), 4 frames may be required per bundle.

The frames (510 to 580) illustrated in FIG. 5A are arranged in chronological order with time on the horizontal axis, and the time intervals between the frames can be adjusted to a constant value. In addition, the second data (e.g., gyro data) (511 to 583) acquired for each frame (e.g., illustrated by three vertical lines in frames 510 to 580) can be acquired based on an arbitrary time interval. According to one embodiment, the time intervals between the illustrated frames (510 to 580) and/or the time intervals of the second data are not limited to the illustrated embodiment, and the time intervals may be changed according to various embodiments. For example, the time intervals at which the gyro data is extracted may not be the same or similar, but may be different from each other.

Referring to FIG. 5a, the second data for each frame acquired during the imaging operation of the electronic device is illustrated in three rows of cells, but this is only an example and there is no limitation on the number, and more than three may be included.

Referring to FIG. 5A, the image motion from the first frame (510) to the second frame (520) may be represented as MV12, and the gyro motion corresponding to MV12 may be represented as MVg12. Since the image motion and/or gyro motion between the frames (510 to 580) are the same as or similar to MV12 and/or MVg12, their description will be omitted below. The base frame may have a significant influence on the processor finding the gyro motion. However, the movement direction of the camera can be known in advance through frames other than the base frame. The gyro motion corresponding to the image motion (MV15 (motion vector)) from the first frame to the fifth frame of FIG. 5A is MVg15 (gyro motion vector), but the movement direction of the camera can be known through MVg56, MVg67, and MVg78. In this way, when motion smoothing is performed using camera trajectories, it is possible to additionally predict future motion information, making it easier to respond to motion. For example, assuming that the previous motion up to MVg15 was a panning motion, it is possible to analyze in advance whether panning will continue in the future and at what speed through MVg56, MVg67, and MVg78. Therefore, the processor can flexibly respond to continuous rapid movements.

FIG. 5b is an exemplary diagram of blur detection of a base frame during an imaging operation of an electronic device according to various embodiments.

A processor according to various embodiments (e.g., a processor (120) of FIG. 1, an image signal processor (260) of FIG. 2, an image signal processor (321) of FIG. 3) can control an electronic device (e.g., an electronic device (101) of FIG. 1) to extract a base frame (base) during a capturing operation and output an image at a second frame rate.

Referring to FIGS. 5A and 5B , a processor may control the operation of shake correction during an imaging operation of an electronic device. The processor may control an image stabilizer (e.g., an image stabilizer (240) of FIG. 2) of a camera module (e.g., a camera module (180) of FIG. 1 , a camera module (320) of FIG. 3 ) to perform a compensation operation for preventing shake. For example, the shake correction may be in the form of an optical image stabilizer (OIS) and/or a video digital image stabilizer (VDIS).

Referring to FIGS. 5A and 5B, in FIG. 5A, the base frames may be frame 1 (510) and frame 5 (550). For example, the processor may control compensation for shake prevention through an image stabilizer using automatic exposure (AE) or the like to prevent shake. As a result of the shake correction of the processor, blur, which is an image deterioration, may occur in frame 5 (550), which is one of the base frames (e.g., the first base frame).

When blurring occurs in the base frame, jitter may occur when displaying an image by synthesizing the base frames (510, 550) according to a second frame rate (e.g., 30 fps). The processor can detect a blur frame with a high probability of jitter occurring.

A processor according to various embodiments may remove a base frame (e.g., a first base frame) in which blurring occurs. Referring to FIG. 5B, frame 5 (550) is a base frame, but since blurring occurs and jitter is likely to occur during shake correction (e.g., OIS, VDIS, etc.), the processor may control an operation to remove frame 5 (550) because it is unsuitable as a base frame. According to one embodiment, the processor may control a base frame (e.g., a second base frame) to replace the removed frame 5 (550). At this time, the processor may use time-dependent gyro data extracted from the existing frame 5 (550) and images of frames (e.g., frame 4 (540), frame 6 (560)) having temporal continuity with frame 5 (550) to generate a new base frame (e.g., the second base frame). The newly generated base frame may be one that directly utilizes the image (e.g., first data) of the 4th frame (540) or the 6th frame (560) (e.g., image replacement of the frame), or may utilize an image (not shown) synthesized with the average values of the 4th frame (540) and the 6th frame (560), but is not limited thereto.

According to various embodiments, the process of replacing with a new base frame (e.g., a second base frame) may utilize image information (e.g., image information) of surrounding frames (e.g., frame 4 (540), frame 6 (560)) of the removed base frame (550) (e.g., the first base frame), and gyro data applied to the generated base frame (e.g., the second base frame) may utilize second data of the removed base frame (550) (e.g., the first base frame).

A blur detection process according to various embodiments may include a process of detecting and collecting first data (e.g., based on a captured image detected through an image sensor) and/or second data (e.g., based on a movement of an electronic device detected through a sensor module) from image information detected through a camera module (e.g., camera module (180) of FIG. 1, camera module (320) of FIG. 3) that may include an image sensor (e.g., image sensor (230) of FIG. 2) and/or a sensor module (e.g., sensor module (176) of FIG. 1, sensor module (340) of FIG. 3) that may include a gyro sensor.

A process for acquiring motion-related data according to various embodiments may include a process in which a processor acquires electronic data of first data and/or second data of images captured by a camera (e.g., a capturing operation) of an electronic device. The first data and/or second data may be viewed as time stamps extracted at regular intervals from individual frames, which may be utilized by the processor. For example, the second data may include information on angular velocity and time stamps, and acquiring the time stamps may be accomplished using a processor (e.g., the main processor (121)).

FIG. 6A is an example diagram of obtaining second data (e.g., gyro data) of a base frame when an electronic device executes a capturing operation according to various embodiments.

Referring to FIGS. 5A and 6A, the vertical lines (511 to 583) of individual frames (510 to 580) of FIGS. 5A and 5B may be second data of the individual frames. In addition, the horizontal lines of the first frame (510) and the fifth frame (550) of FIG. 6A may be second data of the individual frames and may correspond to the vertical lines of each frame illustrated in FIGS. 5A and 5B.

Referring to FIGS. 5A to 6A, a processor (e.g., a processor (120) of FIG. 1, an image signal processor (260) of FIG. 2, an image signal processor (321) of FIG. 3) may be controlled to detect movement-related data (e.g., first data, second data) of an electronic device. For example, the processor may obtain first data (e.g., image edge data) confirmed from image information (e.g., image information) detected through an image sensor (e.g., an image sensor (230) of FIG. 2), and/or second data (e.g., gyro data) confirmed from movement information of the electronic device detected through a sensor module (e.g., a sensor module (176) of FIG. 1, a sensor module (340) of FIG. 3).

Referring to FIGS. 5A to 6A, the processor can collect second data (511 to 513, 551 to 553) at arbitrary time intervals for each individual frame. For example, the processor can control the sensor module to detect the second data (511 to 513) of the first frame (510) three times. In this case, the detected second data 1_1 (gyro data 1_1) (511), second data 1_2 (gyro data 1_2) (512), and second data 1_3 (gyro data 1_3) (513) can be detected at arbitrary time intervals (e.g., distance intervals). The arbitrary time intervals at which the processor acquires the second data can vary depending on the time at which the individual frames are captured, and can be changed depending on the first frame rate (e.g., 120 fps). Additionally, the same time interval can be used to compute distance using angular velocity, so the concept of the same interval can be applied to each individual frame.

Referring to FIGS. 5A to 6A, a processor according to various embodiments may acquire second data (551 to 553) of a fifth frame (550) detected at an arbitrary time interval. For example, the second data (551 to 553) acquired by the processor from the fifth frame (550), which is a base frame, are illustrated as second data 5_1 (gyro data 5_1) (551), second data 5_2 (gyro data 5_2) (552), and second data 5_3 (gyro data 5_3) (553), but this is merely an example, and three or more sets of second data may be collected.

According to one embodiment, when blur occurs in the base frame 5 (550) and is removed, the processor may utilize the second data (e.g., 551 to 553) of the base frame (e.g., 550) to be removed (e.g., the first base frame). Referring to FIG. 5b, when blur occurs in the base frame 5 (550) and the frame 5 (550) is removed, the processor may apply the second data (551 to 553) of the existing frame 5 (550) to a base frame (e.g., the second base frame) to be replaced. The processor may utilize the second data to provide continuity in displaying the image and express smooth movement based on the data acquired at the corresponding time.

FIG. 6b is an example diagram of the use of second data (e.g., gyro data) of a base frame that is replaced when an imaging operation of an electronic device is performed according to various embodiments.

Referring to FIGS. 5A to 6B, the vertical lines of individual frames (510 to 580) of FIGS. 5A and 5B may be second data of the individual frames. In addition, the horizontal lines of the first frame (510) and the replacement frame (new) (610) (e.g., the second base frame) of FIGS. 6A and 6B may be second data of the individual frames and may correspond to the vertical lines of each frame illustrated in FIGS. 5A and 5B.

Referring to FIGS. 5A to 6B, a processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, and the image signal processor (321) of FIG. 3) may be controlled to generate a new base frame (new) (610) (e.g., a second base frame) to replace the removed base frame (550) (e.g., a first base frame). The new base frame (610) may be generated in one or more ways. According to one embodiment, the processor may utilize image information of surrounding frames (e.g., the 4th frame (540), the 6th frame (560)) having temporal continuity with the removed base frame (550) as image information of the new base frame (610). In addition, the surrounding frames utilized by the processor may be frames in which no blur occurs, and may be frames in which relatively less blur occurs than the removed frame (550).

According to another embodiment, the processor may be controlled to calculate and synthesize an average value of image information of surrounding frames (e.g., frame 4 (540), frame 6 (560)) having temporal continuity with the removed base frame (550) (e.g., first base frame). In addition, the surrounding frames utilized by the processor may be frames in which no blur occurs, and may be frames in which relatively less blur occurs than the removed frame (550) (e.g., first base frame).

In another embodiment, the processor may also generate a new base frame (610) (e.g., a second base frame) by utilizing both of the above methods together.

Referring to FIGS. 5A to 6B , the processor may apply second data (551 to 553) obtained from a removed base frame (550) (e.g., a first base frame) to a new base frame (610) (e.g., a second base frame). For example, the processor may obtain second data among the motion-related data of the generated base frame (610) from the removed frame (550) and control the smooth display of motion of an image output at the second frame rate. According to various embodiments, the processor may control the synthesis and/or modification (e.g., replacement with one of the surrounding frames of the removed frame) of the first data of a new frame (610) to replace the removed frame (550), and the acquisition and application of the second data from the removed frame (550).

FIG. 7A is an exemplary diagram of a motion change amount according to a first frame rate when an electronic device performs a shooting operation according to various embodiments.

Referring to FIG. 7A, a processor according to various embodiments (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the image signal processor (321) of FIG. 3) may detect a frame (e.g., frame 702) which is a peak of pixels by checking first data collected from image information sensed by an image sensor (e.g., the image sensor (230) of FIG. 2) that may be included in a camera module (e.g., the camera module (180) of FIG. 1, the camera module (320) of FIG. 3). The diagram of FIG. 7A may represent the number of pixels (vertical axis) according to an individual frame number (horizontal axis). The processor may find a peak where a blur occurs and an increase in pixels and may utilize it to detect a frame with a possibility of jitter.

The frames illustrated in Fig. 7a are frames 701, 702, and 703, and the processor can control the calculation of the motion change amount for each frame at any time interval. For example, the processor can control the calculation of the motion change amount between frames 701 and 702 and between frames 702 and 703 at any time interval. The time intervals can be the same or similar, and can be changed by the user.

Referring to FIG. 7A, the processor may obtain a value of 704 by calculating the amount of motion change between frames 701 and 702. Additionally, the processor may obtain a value of 705 by calculating the amount of motion change between frames 702 and 703. According to various embodiments, the processor may perform image processing according to the movement of the electronic device based on the calculated amount of motion change.

FIG. 7b is an example diagram of a motion change amount after peak frame removal due to a motion change amount according to a first frame rate when an electronic device executes a shooting operation according to various embodiments.

Referring to Fig. 7b, the calculated motion change amount is shown by removing frame 702, which is the peak frame of the pixel. Referring to Fig. 7a, in frames captured at arbitrary time intervals, the processor removes frame 702 located in the middle, and can calculate the motion change amount (706) according to a time interval twice that of the existing one (e.g., the motion change amount (704, 705) of Fig. 7a). This means that the motion change amount (706) that changes from frame 701 to frame 703 is calculated, rather than the motion change amounts (704, 705) for frames 701, 702, and 703 of Fig. 7a, and can be calculated as a value (706) greater than the motion change amounts (704, 705) of Fig. 7a. That is, if the processor removes a frame with a lot of image blur (e.g., frame 702 in FIG. 7b) as a peak frame of pixels, the value of the motion change amount may increase, but a problem may not occur because the shake between frames is compensated for through VDIS. In one embodiment, since the removal of frame 702 by the processor may cause a frame drop, the processor may be configured to control shake compensation by creating an interpolation image using an image that has temporal continuity with the removed frame (e.g., frame 702) and replace it with an image (e.g., frame 701, frame 703).

FIGS. 8A and 8B illustrate exemplary methods for image processing by removing and/or generating a base frame when an electronic device executes a capturing operation, according to various embodiments. For example, the drawings illustrate exemplary methods for preventing image shake by removing and/or generating a base frame in an electronic device.

Referring to FIGS. 5A to 8B, a processor of an electronic device (e.g., the electronic device (101) of FIG. 1) may control a camera module (e.g., the camera module (180) of FIG. 1, the camera module (320) of FIG. 3) to perform an imaging operation at a first frame rate (e.g., 120 fps). Referring to FIG. 8A, a processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the image signal processor (321) of FIG. 3) may extract any frame among a plurality of frames (811 to 814, 821 to 824, 831 to 834) as a base frame. According to one embodiment, if there is a blur in the base frame (e.g., 811, 821, 831), the processor may remove the base frame (e.g., frame 821).

Referring to FIG. 8A, frames captured at the first frame rate may be referred to as the nth frame in accordance with the time sequence. For example, base frames for each section (810, 820, 830) (e.g., each group) at the first frame rate may be the 1st frame (811), the 5th frame (821), and the 9th frame (831).

Referring to FIGS. 8A and 8B , the processor may detect and remove blur of frame 5 (821), which should be the second base frame among base frames (811, 821, 831). In one embodiment, the processor may be configured to generate a base frame (801) (e.g., a second base frame) to replace the removed frame 5 (821) (e.g., a first base frame). The processor may be configured to display a video result at a second frame rate of 30 fps based on frame 1 (811), the new base frame (801), and frame 9 (831).

Referring to FIGS. 5A to 8B, the processor may perform shake correction after removing peak frames due to the amount of motion change in FIG. 7B. For example, the processor may remove a frame (821) with blur among a plurality of frames (811 to 814, 821 to 824, 831 to 834). In addition, the processor may generate an interpolated image using frames without blur (e.g., utilizing the first data of the 6th frame (822) and the second data of the 5th frame (821)) and perform shake correction. For example, the processor may remove the blurred base frame (821) based on the base frames (811, 821, 831) and generate a replacement base frame (801), and display the blur-free base frames (811, 801, 831) to generate an image according to the second frame rate.

According to another embodiment, the processor may generate a base frame by synthesizing frames 5 (821) to 8 (824) (e.g., an average value of motion-related data of frames 5 to 8 (821 to 824) of FIG. 8A) when there is no blur based on frame 5 (821). For example, the processor may also synthesize second data among the motion-related data of the electronic device (e.g., first data, second data) while generating a new base frame. In addition, the processor may control the display of the video result at a second frame rate of 30 fps based on the new base frames.

According to various embodiments, the processor may be controlled to generate a new base frame (e.g., 801 of FIG. 8b) and/or to display the video result by generating a new base frame using a synthesis of frames without blur (e.g., an average value of motion-related data of frames 1 to 4 (811 to 814) of FIG. 8a). For example, in FIG. 8A, the processor may set frame 1 as the base frame in the first bundle (810) (e.g., section 1), generate a replacement base frame (801) by removing the blurred base frame (821) in the second bundle (820) (e.g., section 2) (e.g., utilizing the first data of frame 6 (822) and the second data of frame 5 (821)), and generate a new base frame (not shown) by synthesizing frames 9 (831) to 12 (834) in the third bundle (830) (e.g., section 3) (average value of motion-related data of frames 9 to 12 (831 to 834)), and generate an image according to the second frame rate based on the base frames. That is, the processor can be controlled to display an image result according to the second frame rate based on the first frame (811) of the first bundle (810), the replacement base frame (801) of the second bundle (820), and the new base frame (not shown) of the third bundle (830).

According to various embodiments of the present disclosure, an electronic device (e.g., an electronic device (101) of FIG. 1) includes a camera module (e.g., a camera module (180) of FIG. 1, a camera module (320) of FIG. 3), a sensor module (e.g., a sensor module (176) of FIG. 1, a sensor module (340) of FIG. 3), a memory (e.g., a memory (130) of FIG. 1), and a processor (e.g., a processor (120) of FIG. 1, an image signal processor (260) of FIG. 2, an image signal processor (321) of FIG. 3) operatively connected to the camera module, the sensor module, and the memory, wherein the processor converts motion-related data of the electronic device acquired from the camera module and the sensor module into blur detection data, and detects a frame in which blur occurs in an imaging operation according to a first frame rate based on the converted blur detection data, and when the detected frame is a base frame, the first frame in which blur is detected The base frame may be removed, a second base frame may be generated to replace the removed first base frame, and the image may be processed at a second frame rate.

According to various embodiments of the present disclosure, a sensor module (e.g., sensor module (176) of FIG. 1, sensor module (340) of FIG. 3) of an electronic device (e.g., electronic device (101) of FIG. 1) may include a gyro sensor. According to various embodiments of the present disclosure, a camera module of the electronic device may include an image sensor, and movement-related data of the electronic device may include at least one of first data acquired from the image sensor of the camera module or second data acquired from the sensor module.

According to various embodiments of the present disclosure, a processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the image signal processor (321) of FIG. 3) of an electronic device (e.g., the electronic device (101) of FIG. 1) may be configured to convert motion-related data into blur detection data including data on camera motion or gyro motion for each frame according to a first frame rate. According to various embodiments of the present disclosure, the processor of the electronic device may be configured to detect a frame in which blur occurs when a range of change in camera motion or gyro motion of an individual frame according to the first frame rate exceeds a predetermined range of change based on the blur detection data.

According to various embodiments of the present disclosure, a processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the image signal processor (321) of FIG. 3) of an electronic device (e.g., the electronic device (101) of FIG. 1) may be configured to use first data including image edge data of a frame having time-series continuity with a removed first base frame when generating a second base frame to be replaced. According to various embodiments of the present disclosure, the processor of the electronic device may be configured to use second data including gyro data of the removed first base frame when generating the second base frame to be replaced.

According to various embodiments of the present disclosure, a processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the image signal processor (321) of FIG. 3) of an electronic device (e.g., the electronic device (101) of FIG. 1) may be configured to, when processing an image, synthesize base frames that are captured and not removed according to a first frame rate with a second base frame, and process the image according to the second frame rate. According to various embodiments of the present disclosure, the processor of the electronic device may be configured to use an average value of motion-related data of frames that have time-series continuity with the removed first base frame when generating a second base frame to be replaced.

According to various embodiments of the present disclosure, an image processing method of an electronic device (e.g., an electronic device (101) of FIG. 1) may include an operation of converting movement-related data of the electronic device, obtained from a camera module (e.g., a camera module (180) of FIG. 1, a camera module (320) of FIG. 3) and a sensor module (e.g., a sensor module (176) of FIG. 1, a sensor module (340) of FIG. 3), into blur detection data, an operation of detecting a frame in which blur occurs in an imaging operation according to a first frame rate based on the converted blur detection data, an operation of removing a first base frame in which blur is detected when the detected frame is a base frame, and an operation of generating a second base frame to replace the removed first base frame and synthesizing the second base frame according to the second frame rate.

According to various embodiments of the present disclosure, the converting operation of the image processing method may include at least one of an operation of acquiring first data among motion-related data from an image sensor of a camera module (e.g., the camera module (180) of FIG. 1, the camera module (320) of FIG. 3) or an operation of acquiring second data among the motion-related data from a sensor module. According to various embodiments of the present disclosure, the converting operation of the image processing method may include an operation of converting the motion-related data so that data regarding camera motion or gyro motion is included for each frame according to the first frame rate.

According to various embodiments of the present disclosure, the detecting operation of the image processing method may include an operation of detecting a frame in which blur occurs when a change in camera motion or gyro motion of an individual frame exceeds a predetermined change range according to a first frame rate, based on data for blur detection.

According to various embodiments of the present disclosure, the synthesizing operation of the image processing method may include an operation of using first data including image edge data of a frame having time-series continuity with the removed first base frame when generating a second base frame to be replaced. According to various embodiments of the present disclosure, the synthesizing operation of the image processing method may include an operation of using second data including gyro data of the removed first base frame when generating the second base frame to be replaced. According to various embodiments of the present disclosure, the synthesizing operation of the image processing method may include an operation of synthesizing base frames that have been captured at a first frame rate and not removed with the second base frame, and an operation of image processing the synthesized base frames at the second frame rate. According to various embodiments of the present disclosure, the synthesizing operation of the image processing method may include an operation of using an average value of motion-related data of frames having time-series continuity with the removed first base frame when generating the second base frame to be replaced.

According to various embodiments of the present disclosure, the commands of a storage medium storing the commands are set to cause the processor to perform at least one operation when executed by at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the image signal processor (321) of FIG. 3), wherein the operation comprises: an operation of converting motion-related data of an electronic device obtained from a camera module (e.g., the camera module (180) of FIG. 1, the camera module (320) of FIG. 3) and a sensor module (e.g., the sensor module (176) of FIG. 1, the sensor module (340) of FIG. 3) into blur detection data; an operation of detecting a frame in which blur occurs in an imaging operation according to a first frame rate based on the converted blur detection data; an operation of removing the first base frame in which blur is detected when the detected frame is a base frame; and an operation of generating a second base frame to replace the removed first base frame, thereby generating the second frame It may include an operation of synthesizing according to a rate. In a storage medium according to various embodiments of the present disclosure, the motion-related data may include first data or second data, the first data may include image edge data, and the second data may include gyro data. In a storage medium according to various embodiments of the present disclosure, the operation of synthesizing may include an operation of using first data of a frame having time-series continuity with the removed first base frame and using second data of the first base frame when generating the second base frame to be replaced.

Claims (20)

In electronic devices,
camera module;
sensor module;
Memory that stores instructions; and
At least one processor operatively connected to the camera module, the sensor module, and the memory;
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
Converting the movement-related data of the electronic device obtained from the camera module and the sensor module into blur detection data,
Based on the above-mentioned converted blur detection data, a frame in which blur occurs is detected in an imaging operation according to a first frame rate,
If the above detected frame is a base frame, the first base frame in which blur is detected is removed,
An electronic device configured to generate a second base frame to replace the removed first base frame based on at least one frame among frames having time-series continuity with the first base frame, and to process images according to a second frame rate. delete In the first paragraph,
The above camera module
Includes an image sensor,
The movement-related data of the above electronic device
An electronic device comprising at least one of first data acquired from an image sensor of the camera module or second data acquired from the sensor module. In the first paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
An electronic device configured to convert the motion-related data into blur detection data including data regarding camera motion or gyro motion for each frame according to the first frame rate. In paragraph 4,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
An electronic device configured to detect a frame in which blur occurs when the range of change in camera motion or gyro motion of an individual frame exceeds a predetermined range of change based on the data for detecting blur according to the first frame rate. In the third paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
An electronic device configured to use first data including image edge data of a frame having time-series continuity with the removed first base frame when generating the second base frame to be replaced. In the third paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
An electronic device configured to use second data including gyro data of the removed first base frame when generating the second base frame to be replaced. In the third paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
When processing the above image,
An electronic device configured to synthesize base frames that have been photographed and not removed according to the first frame rate and the second base frame, and to process the image according to the second frame rate. In the third paragraph,
The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to:
An electronic device configured to use the average value of motion-related data of frames having time-series continuity with the removed first base frame when generating the second base frame to be replaced. In a method for image processing of an electronic device,
An operation of converting movement-related data of the electronic device obtained from a camera module and a sensor module into data for blur detection;
An operation of detecting a frame in which blur occurs in a shooting operation according to a first frame rate based on the above-mentioned converted blur detection data;
If the detected frame is a base frame, an operation of removing the first base frame in which blur is detected; and
A method comprising: generating a second base frame to replace the removed first base frame based on at least one frame among frames having time-series continuity with the first base frame, and synthesizing the second base frame according to a second frame rate; In Article 10,
The above conversion operation is
A method comprising at least one of an operation of acquiring first data among the movement-related data from the image sensor of the camera module or an operation of acquiring second data among the movement-related data from the sensor module. delete delete delete delete delete delete delete delete delete