WO2025150740A1 - Electronic device comprising image sensor, and operating method thereof - Google Patents

Abstract

An electronic device according to one embodiment comprises a camera module including: a lens assembly including at least one lens and a sensor that outputs a signal for acquiring motion data corresponding to motion of the electronic device; an image sensor; and an optical image stabilizer (OIS) for moving the lens assembly and/or the image sensor in order to perform an optical image stabilization operation on the basis of the motion data. The electronic device can perform an autofocus function on the basis of autofocus data corrected on the basis of parameters for the optical image stabilization operation.

Description

Electronic device including image sensor and method of operating same

The present disclosure relates to a technology for performing an AF (auto focus) function in an electronic device including an image sensor.

As high-resolution mode is required, methods for improving AF performance using image sensors are being proposed. Recently, image sensors are being developed in the direction of reducing pixel size and increasing the number of pixels due to constraints in camera layout structure. In particular, as image sensors with high AF performance, not just high-pixel mode, are being required, pixel structures capable of full-pixel phase detection are being proposed.

An electronic device employing the AF method includes an image sensor in which two PDs (photodiodes) having a 2x1 array are arranged under one micro lens, and the AF function is performed by classifying the AF data output from each PD into left data and right data and focusing using the phase difference between the two. However, as the size of the PD decreases and the demand for a structure capable of high resolution increases, a structure in which four PDs having a 2x2 array instead of a 2x1 array share one micro lens is proposed. In this case, in addition to the advantage of high resolution, there is the advantage of further improving the AF performance because the phase differences between the left, right, up, and down can be known.

Meanwhile, the output of the PD pixel has a very large difference in sensitivity depending on the chief ray angle (CRA) of the incident light, and if the value of the PD pixel is used as is to generate actual image data, deviation in the pixel value may occur depending on the physical location of the lens and pixel.

The above information may be provided as related art for the purpose of assisting in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

An electronic device according to one embodiment may include a sensor, a camera module, a processor, and a memory. The sensor may output a signal for obtaining motion data regarding a movement of the electronic device. The camera module may include a lens assembly, an image sensor, and an optical image stabilizer (OIS). The lens assembly may include at least one lens. The optical image stabilizer may be configured to move at least one of the lens assembly or the image sensor to perform an optical image stabilization operation based on the motion data. The memory may store instructions executable by the processor. The instructions may be executed by the processor to cause the electronic device to obtain first AF data from the image sensor. The instructions may be executed by the processor to cause the electronic device to correct the first AF data based on parameters for the OIS operation to generate second AF data. The instructions may be executed by the processor to cause the electronic device to perform an AF (auto focus) function based on the second AF data.

An operating method of an electronic device according to one embodiment may include an operation of outputting a signal for obtaining motion data corresponding to a movement of the electronic device. The operating method may include an operation of controlling the lens assembly or the image sensor to perform an optical image stabilization (OIS) operation based on the motion data. The operating method may include an operation of obtaining first AF data from the image sensor and an operation of generating second AF data by correcting the first AF data based on a parameter for the OIS operation. The operating method may include an operation of performing an AF (auto focus) function based on the second AF data.

A computer-readable non-transitory recording medium according to one embodiment of the present disclosure may store at least one command and/or instruction that, when executed, causes an electronic device to perform the method described above or an operation of the electronic device.

FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

FIG. 3 is a diagram showing the main hardware configuration of an electronic device according to one embodiment.

FIG. 4 is a cross-sectional view of a pixel array of an image sensor according to one embodiment.

FIG. 5 is a drawing for explaining the effect of shading on an image sensor according to one embodiment.

Figure 6 illustrates an example of a shadow pattern detected according to the operation of an optical image stabilizer.

FIG. 7 is a flowchart illustrating a method of operating an electronic device according to one embodiment.

FIGS. 8 and 9 illustrate the OIS operation of an electronic device according to one embodiment.

Figure 10 illustrates examples of determining whether to perform shading compensation depending on the OIS operation direction when detecting a horizontal phase difference.

FIG. 11 illustrates an example of weights determined based on OIS positions according to one embodiment.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. The advantages and features, and the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the attached drawings. The embodiments disclosed in this document are not limited to the embodiments disclosed below, but may be implemented in various different forms, and are only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification can be used in the meaning commonly understood by a person of ordinary skill in the art. In addition, terms defined in commonly used dictionaries are not to be ideally or excessively interpreted unless explicitly specifically defined. The terminology used in this specification is for the purpose of describing embodiments and is not intended to limit the embodiments. In this specification, the singular also includes the plural unless specifically stated in the phrase.

The terms "comprises" and/or "comprising", as used in the specification, do not exclude the presence or addition of one or more other components, operations, operations and/or elements.

In one embodiment, an electronic device and a method of operating the same can be provided that can correct a pattern that occurs due to a mismatch between an angle of incidence of light incident on a lens (lens angle of incidence) and an angle of incidence of light incident on a sensor (sensor angle of incidence).

In one embodiment, an electronic device and a method of operating the same can be provided to improve accuracy when operating an autofocus function by correcting a pattern that occurs due to a mismatch between a lens incident angle and a sensor incident angle.

The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field included in the description of the present disclosure.

FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

The processor (120) may control at least one other component (e.g., a hardware or software component) of an electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store result data in a nonvolatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one of the components of the electronic device (101) (e.g., the display module (160), the sensor module (176), or the communication module (190)), for example, while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing device) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. Such learning may be performed, for example, in the electronic device (101) itself on which the artificial intelligence model is executed, or may be performed through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may additionally or alternatively include a software structure.

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or nonvolatile memory (134).

The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The audio output module (155) can output an audio signal to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented separately from the speaker or as a part thereof.

The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can obtain sound through an input module (150), or output sound through an audio output module (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or a headphone) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect an operating state (e.g., power or temperature) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electric signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user can perceive through a tactile or kinesthetic sense. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery (189) can power at least one component of the electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108)), and performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., the application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The wireless communication module (192) can support a 5G network and next-generation communication technology after a 4G network, for example, NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), terminal power minimization and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) may support various requirements specified in an electronic device (101), an external electronic device (e.g., an electronic device (104)), or a network system (e.g., a second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL) each, or 1 ms or less for round trip) for URLLC realization.

The antenna module (197) can transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module (197) can include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) can include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), can be selected from the plurality of antennas by, for example, the communication module (190). A signal or power can be transmitted or received between the communication module (190) and the external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) can be additionally formed as a part of the antenna module (197).

According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC positioned on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) positioned on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

At least some of the above components may be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

In one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). In one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform at least a part of the function or service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result, as is or additionally processed, as at least a part of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an IoT (Internet of Things) device. The server (108) may be an intelligent server using machine learning and/or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199). The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

The electronic devices according to various embodiments disclosed in this document may be devices of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliance devices. The electronic devices according to embodiments of this document are not limited to the above-described devices.

It should be understood that the various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly dictates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first) is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the called at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store ^TM ) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities, and some of the multiple entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, the multiple components (e.g., a module or a program) may be integrated into one component. In such a case, the integrated component may perform one or more functions of each of the multiple components identically or similarly to those performed by the corresponding component of the multiple components before the integration. According to various embodiments, the operations performed by the module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of an image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as, for example, an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of an image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to a shutter, or when a plurality of images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130), or as a separate memory that operates independently therefrom.

The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, or read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., the image sensor (230)). The image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or may be provided to an external component of the camera module (180) (e.g., the memory (130), the display module (160), the electronic device (102), the electronic device (104), or the server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120). It may be configured as a part of the image signal processor (260) or as a separate processor that operates independently of the processor (120). When the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.

Fig. 3 illustrates a main hardware configuration of an electronic device according to one embodiment. In the description of Fig. 3, the configurations described in Figs. 1 and 2 may be briefly described or omitted.

Referring to FIG. 3, in one embodiment, an electronic device (101) (e.g., the electronic device (101) of FIG. 1) may include a lens assembly (210) (e.g., the lens assembly (210) of FIG. 2), an image sensor (230) (e.g., the image sensor (230) of FIG. 2), an image signal processor (260) (e.g., the image signal processor (260) of FIG. 2), a processor (120) (e.g., the processor (120) of FIG. 1), a display (160) (e.g., the display module (160) of FIG. 1), and a memory (250) (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2).

According to one embodiment, the lens assembly (210) may have different numbers, arrangements, types, etc. of lenses depending on the front camera and the rear camera. Depending on the type of the lens assembly (210), the front camera and the rear camera may have different characteristics (e.g., focal length, maximum magnification, etc.).

In one embodiment, when the image signal processor (260) and the image sensor (230) are physically separate, there may be a compliant sensor interface.

According to one embodiment, the image signal processor (260) can perform image processing on electrically converted image data. The process in the image signal processor (260) can be divided into pre-ISP (hereinafter, pre-processing) and ISP chain (hereinafter, post-processing). The pre-processing process can include 3A processing, lens shading correction, edge enhancement, dead pixel correction, and knee correction. The 3A can include at least one of AWB (auto white balance), AE (auto exposure), and AF (auto focusing). The post-processing process can include at least one of changing a sensor index value, changing a tuning parameter, and adjusting an aspect ratio. The post-processing process can include a process of processing image data output from the image sensor (230) or image data output from a scaler. The image signal processor (260) can adjust the contrast, sharpness, saturation, dithering, etc. of the image through a post-processing process. Here, the contrast, sharpness, and saturation adjustment procedures can be performed in a YUV color space, and the dithering procedure can be performed in an RGB (Red Green Blue) color space. The image signal processor (260) can transmit the image data obtained after processing the post-processing process to a memory (e.g., a display buffer) (250). The display (160) can display the image data stored in the memory (e.g., a display buffer) (220) on a display screen under the control of the processor (120).

According to one embodiment, the processor (120) can execute/control various functions supported by the electronic device (101). For example, the processor (120) can execute an application and control various hardware by executing a code written in a programming language stored in the memory (250). For example, the processor (120) can execute an application supporting a shooting function stored in the memory (250). In addition, the processor (120) can execute a camera module (e.g., the camera module (180) of FIG. 1) and set and support an appropriate shooting mode so that the camera module (180) can perform an operation intended by the user.

According to one embodiment, the memory (250) may store instructions executable by the processor (120). The memory (250) may be understood as a concept including a component such as a RAM (random access memory) in which data is temporarily stored and/or a component such as an SSD (solid state drive) in which data is permanently stored. For example, the processor (120) may call instructions stored in the SSD to implement a software module in the RAM space. In various embodiments, the memory (250) may include various types, and an appropriate type may be adopted according to the purpose of the device. According to one embodiment, the memory (250) may include one or more storage media (or one or more storage devices). For example, the memory (250) may include a memory aggregate including one or more storage media. For example, the one or more storage media may include a hard drive, flash memory, permanent memory such as read-only memory (ROM), semi-permanent memory such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof. The memory (250) may include a cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (101). As a non-limiting example, the cache memory may be included within the processor (120). The memory (250) may be fixedly embedded within the electronic device (101) or incorporated into one or more suitably types of components (e.g., a subscriber identity module (SIM) card and/or a secure digital (SD) card) that can be repeatedly inserted into and removed from the electronic device (101).

According to one embodiment, the memory (250) may store an application associated with the camera module (180). For example, the memory (250) may store a camera application. The camera application may support various shooting functions such as taking pictures, shooting videos, taking panoramas, and taking slow motion pictures.

According to one embodiment, the memory (250) may store one or more software applications, such as an operating system (or system software application), a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and/or applet) software application, and/or any other suitable software applications. For example, the one or more software applications may include instructions executable by at least one processor (120). For example, the memory (250) may store instructions callable by an application programming interface (API). For example, the memory (250) may store instructions within a library.

According to one embodiment, the processor (120) may display an execution screen of an application executed by the processor (120) or contents such as images and/or videos stored in the memory (250) on the display (160). In addition, the processor (120) may display image data acquired through the camera module (180) on the display (160).

According to one embodiment, the electronic device (101) may include a gyro sensor (330) and a hall sensor (320). For example, the gyro sensor (330) may measure a rotation angle or inclination of the electronic device (101) about three axes (e.g., X-axis, Y-axis, or Z-axis). However, the gyro sensor is exemplary, and the electronic device (101) may include various sensors (e.g., motion sensors) such as an acceleration sensor or a hall sensor (e.g., sensor module (176) of FIG. 1).

According to one embodiment, the OIS module (310) can obtain motion data from the gyro sensor (330). The OIS module (310) can obtain information on the degree of shaking of the electronic device (101) by performing an integral operation on the motion data. The OIS module (310) can control the OIS function of the camera module (180) based on the information on the motion data. In FIG. 3, the OIS module (310) is illustrated as controlling the lens assembly (210), but is not limited thereto. For example, the OIS module (310) may be configured to control the position of the image sensor (230).

In one embodiment, the AF (auto focus) module (315) may control the AF function of the camera module (180) based on information acquired from the processor. For example, the AF module (315) may control the position of the lens assembly (210) (e.g., to adjust the distance between the lens assembly (210) and the image sensor (230)) so that the image sensor (230) is focused. In FIG. 3, the AF module (315) is illustrated as controlling the lens assembly (210), but is not limited thereto. For example, the AF module (315) may also be configured to control the position of the image sensor (230).

According to one embodiment, when the camera module (180) is of a lens shift type that moves the lens assembly (210), the processor (120) can detect the position to which the lens assembly (210) has moved through the hall sensor (320). According to one embodiment, when the camera module (180) is of a sensor shift type that moves the image sensor (230), the processor (120) can detect the position to which the image sensor (230) has moved through the hall sensor (320).

FIG. 4 is a cross-sectional view of a pixel array of an image sensor (e.g., the image sensor (230) of FIGS. 2 and 3) according to one embodiment.

Referring to FIG. 4, the image sensor (230) may include a plurality of unit pixels (410). According to one embodiment, each unit pixel (410) may include four or more photo diodes (PD) (413). According to one embodiment, the plurality of unit pixels (410) may be positioned on a plane substantially perpendicular to a Z-axis corresponding to a direction in which light is incident. According to one embodiment, a first direction (e.g., a direction of the X-axis) of the plurality of unit pixels (410) may be substantially perpendicular to a second direction (e.g., a direction of the Y-axis) of the unit pixels (410). According to one embodiment, the first direction (e.g., a direction of the X-axis) and the second direction (e.g., a direction of the Y-axis) may be substantially perpendicular to the direction of the Z-axis.

According to one embodiment, each of the unit pixels (410) may include a micro lens (411), a color filter (412), and a plurality of photo diodes (413), or a combination thereof. According to one embodiment, each of the plurality of photo diodes (413) may also be referred to as a light-receiving element. According to one embodiment, the plurality of photo diodes (413) may also be referred to as a multi-photo diode. The color filter (412) of FIG. 4 may be arranged with color filters of the same color channel, but color filters including different color channels may also be arranged.

According to one embodiment, the micro lens (411) can focus light incident on the micro lens (411). According to one embodiment, the micro lens (411) can adjust the path of light incident on the micro lens (411) so that the light reaches each of the plurality of photodiodes (413).

According to one embodiment, the color filter (412) can transmit light of a pre-designated color (or color channel). According to one embodiment, the color filter (412) of each of the plurality of photodiodes (413) can transmit light of one color (e.g., red) among the pre-designated colors (e.g., red, blue, or green) according to a pre-designated pattern (e.g., Bayer pattern). According to one embodiment, the color filter (412) can block light of a color other than the pre-designated color (or color channel).

According to one embodiment, the number of photodiodes (413) arranged at a position corresponding to one micro lens (411) may be four or more. However, the present invention is not limited thereto. For example, two photodiodes may be arranged at a position corresponding to one micro lens (411). In the present disclosure, the phrase “a photodiode is arranged at a position corresponding to a micro lens” may mean that the photodiode is arranged at a position to receive light passing through the micro lens. The plurality of photodiodes (413) may be arranged symmetrically, but the present invention is not limited thereto. In the present disclosure, an image sensor having a plurality of photodiodes (413) corresponding to one micro lens (411) may be referred to as a multi-photodiode image sensor. According to one embodiment, each of the plurality of photodiodes (413) may output a value corresponding to incident light. According to one embodiment, each of the plurality of photodiodes (413) may output a value corresponding to incident light based on the photoelectric effect. According to one embodiment, each of the plurality of photodiodes (413) can output a value corresponding to the intensity (or illuminance) of incident light based on the photoelectric effect.

According to one embodiment, each of the plurality of photodiodes (413) can generate a charge according to the intensity (or illuminance) of the incident light based on the photoelectric effect. According to one embodiment, each of the plurality of photodiodes (413) can output a current according to the amount of the generated charge.

FIG. 5 is a drawing for explaining the effect of shading on an image sensor according to one embodiment.

The unit pixel (410) illustrated in FIGS. 4 and 5 may have a different amount of light reaching the photodiode (413) included in the unit pixel (410) depending on the angle at which light is incident. Accordingly, the unit pixel (410) may detect light that is incident within a certain range of angles with respect to the micro lens (411) or the photodiode (413). The light incident on the image sensor may include a chief ray and a marginal ray (or, marginal ray). The chief ray may refer to the main ray among the light rays incident on the image sensor. The marginal ray may refer to the light rays surrounding the chief ray. Among the light rays that reach the photodiode (413) of the image sensor, the chief ray is the strongest, and the intensity of the light becomes weaker as it approaches the marginal ray. Outside of the marginal ray, light other than light generated by flare or internal reflection may not be incident on the image sensor. The range of angles at which the principal light for detecting light is incident on the image sensor (e.g., micro lens (411) or photo diode (413)) of the unit pixel (410) may be referred to as an incidence angle range. The principal light may be incident on only some of the photo diodes (413) (e.g., two photo diodes, four photo diodes) that receive light passing through the micro lens (411) depending on the incidence angle of the principal light. Bright light may be detected on the photo diode to which the principal light is incident, and dark light may be detected on the photo diode to which the principal light is not incident. This may cause a shading phenomenon.

In addition, the camera module (180) equipped with the OIS function may have the principal ray angle of the optical system shifted according to the OIS operation. Shading may occur in the image sensor having a plurality of photodiodes arranged to correspond to one micro lens according to the OIS operation.

FIG. 5 (a), (b), and (c) illustrate the same unit pixel included in the image sensor. Referring to FIG. 5 (b), when the operating position of the optical image stabilizer (e.g., the OIS module (310) of FIG. 3) is at a basic position (e.g., the center of the OIS operating range), the chief ray angle (512) of the image sensor may substantially coincide with the chief ray angle (522) of the lens. The chief ray angle of the image sensor may mean an angle at which light is incident so as to detect light without a shading effect within the unit pixel. The chief ray angle of the lens may mean an angle at which light passing through the lens assembly (e.g., the lens assembly (210) of FIG. 2, the lens assembly (210) of FIG. 3) is incident on the unit pixel (or the light-receiving surface of the image sensor). The operation of the optical image stabilizer may cause a lens assembly (e.g., lens assembly (210) of FIG. 2, lens assembly (210) of FIG. 3) or an image sensor (e.g., image sensor (230) of FIG. 2, image sensor (230) of FIG. 3) of a camera module (e.g., camera module (180) of FIG. 1, camera module (180) of FIG. 2) to move or tilt. The movement of the lens assembly or the image sensor may cause a shift in the chief ray. The tilting of the lens assembly or the image sensor may cause an angle at which the lens assembly and the image sensor are aligned with respect to each other to change. Accordingly, the chief ray angle (521, 522, 523) of the lens with respect to a unit pixel (e.g., unit pixel (410) of FIG. 4)) and the chief ray angle (511, 512, 513) of the image sensor viewed by the unit pixel may change. This may cause a mismatch in the chief ray angles of the image sensor and the lens due to the operation of the optical image stabilizer. Fig. 5 (a) illustrates the chief ray angle (511) of the image sensor and the chief ray angle (521) of the lens when the optical image stabilizer operates in a first direction. Fig. 5 (c) illustrates the chief ray angle (513) of the image sensor and the chief ray angle (523) of the lens when the optical image stabilizer operates in a second direction different from the first direction. Referring to Figs. 5 (a) and (c), among the photodiodes (413-1, 413-2), the chief ray of the lens is incident at a different angle from the chief ray of the image sensor, and one photodiode absorbs stronger light than the other photodiode, so that shading may occur in the light detected through the photodiodes (413-1, 413-2). Depending on the angle at which the principal light is incident, the amount of light detected by the photodiodes (413-1, 413-2) may vary. When detecting a phase difference in a state where shadows caused by OIS operation are included in the values detected through the photodiodes (413-1, 413-2), it may not be possible to distinguish whether the phase difference determined through a correlation operation for the detected values is caused by the shadows or caused by the phase difference for the actual subject. As a result, when an auto-focus function based on the phase difference is performed while the optical image stabilizer is operating, the accuracy of the auto-focus function may be reduced. Alternatively, when an image is composed by using each of the data detected from the first photodiode (413-1) and the second photodiode (413-2) as a single pixel value (e.g., when an image is composed without a process that merges data (e.g., a binning operation)), the image quality of the acquired image may be low.

Figure 6 illustrates an example of a shadow pattern detected according to the operation of an optical image stabilizer.

FIG. 6 illustrates a shading pattern detected in a plurality of unit pixels (311, 312, 313, 314), each of which includes one micro lens and four photodiodes corresponding to the micro lens.

FIG. 6 illustrates shadow patterns detected in four operating states (601, 602, 603, 604) of the optical image stabilizer (e.g., states in which the optical image stabilizer is controlled to different positions). In FIG. 6, XOIS may mean an X-axis coordinate value indicating an operating state (or operating position) of the image stabilizer. Alternatively, XOIS may mean a first displacement value in the X-axis direction of the center of the lens assembly defined with the center of the image sensor as a reference point. In FIG. 6, YOIS may mean a Y-axis coordinate value indicating an operating state (or operating position) of the image stabilizer. Alternatively, YOIS may mean a second displacement value in the Y-axis direction of the center of the lens assembly defined with the center of the image sensor as a reference point. The first state (601) illustrates a case in which the OIS position in a first direction is about -3 degrees (XOIS = 8) and the OIS position in a second direction different from the first direction is about -3 degrees (YOIS = 8). The second state (602) illustrates a case where the OIS position in the first direction is about 3 degrees (XOIS = 248) and the OIS position in the second direction is about -3 degrees (YOIS = 8). The third state (603) illustrates a case where the OIS position in the first direction is about -3 degrees (XOIS = 8) and the OIS position in the second direction is about 3 degrees (YOIS 248). The fourth state (604) illustrates a case where the OIS position in the first direction is about 3 degrees (XOIS = 248) and the OIS position in the second direction is about 3 degrees (YOIS = 248).

In the first state (601), among the detection values (611, 612, 613, 614) of the photodiodes included in the first unit pixel (311), the gain value (612) detected from the photodiode in the (+x, +y) direction can be detected as the highest. Conversely, the gain value (613) detected from the photodiode in the (-x, -y) direction can be detected as the lowest.

In the second state (602), among the detection values (621, 622, 623, 624) of the photodiodes included in the first unit pixel (311), the gain value (621) detected from the photodiode in the (-x, +y) direction can be detected as the highest. Conversely, the gain value (624) detected from the photodiode in the (+x, -y) direction can be detected as the lowest.

In the third state (603), among the detection values (631, 632, 633, 634) of the photodiodes included in the first unit pixel (311), the gain value (634) detected from the photodiode in the (+x, -y) direction can be detected as the highest. Conversely, the gain value (631) detected from the photodiode in the (-x, +y) direction can be detected as the highest.

In the fourth state (604), among the detection values (631, 632, 633, 634) of the photodiodes included in the first unit pixel (6311), the gain value (643) detected from the photodiode in the (-x, -y) direction can be detected as the highest. Conversely, the gain value (642) detected from the photodiode in the (+x, +y) direction can be detected as the lowest.

Due to the shading pattern, the gain value of each photodiode can vary from approximately -80% to approximately +80%.

When the image of 4PD is binned, the luminance difference due to shading may not be a problem by using the average value, but when performing AF or a remosaic process, each gain value becomes significant data, and the luminance difference unrelated to the subject may become a problem. The remosaic process may mean an operation of converting the color order of data detected by the image sensor so that a demosaic operation that converts the image data into a color image can be performed. In one embodiment, the electronic device may correct the luminance difference due to shading before performing AF or a remosaic process.

The present disclosure can provide an electronic device and an operating method thereof capable of efficiently correcting shading caused by a difference between the chief ray angle of a lens and the chief ray angle of an image sensor having unit pixels such as 2PD or 4PD.

FIG. 7 is a flowchart (700) illustrating an operating method of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) according to one embodiment.

In the present disclosure, it can be understood that the operation of the electronic device is performed by a processor of the electronic device (e.g., the processor (120) of FIG. 1, the processor (120) of FIG. 3) executing instructions stored in a memory (e.g., the memory (130) of FIG. 1) to perform operations or control components of the electronic device. In one embodiment, FIG. 7 may be performed while the electronic device activates a camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2) to capture an image. For example, the operations illustrated in FIG. 7 may be performed while the electronic device executes a camera application and displays a preview screen including an image acquired through the camera.

According to one embodiment, in operation 701, the electronic device may obtain first AF data from an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (230) of FIG. 3). For example, a unit pixel according to FIG. 4 (e.g., the unit pixel (410) of FIG. 4) may include a multi-photodiode structure in which the phases of light incident on PDs (e.g., the photodiodes (413) of FIG. 4) corresponding to one micro lens (e.g., the micro lens (411) of FIG. 4) are separated. The electronic device may obtain first AF data including phase difference information for at least one phase difference in at least one direction.

According to one embodiment, in operation 703, the electronic device may obtain at least one of motion data and OIS position information. For example, the electronic device may obtain motion data that detects movement of the electronic device through a sensor (e.g., the sensor module (176) of FIG. 1). The sensor may include, for example, a gyro sensor, but the type of the sensor is not limited thereto. For example, the electronic device may control the OIS position of the optical image stabilizer to compensate for the movement of the electronic device based on the motion data. The electronic device may obtain OIS position information indicating the OIS position of the optical image stabilizer. For example, the electronic device may obtain OIS position information from information detected by a Hall sensor (e.g., the Hall sensor (320) of FIG. 3)) disposed in the optical image stabilizer.

According to one embodiment, in operation 705, the electronic device may determine whether to perform shading compensation on the first AF data. The electronic device may be configured to determine whether to perform shading compensation based on whether at least one specified condition is satisfied. For example, the electronic device may determine whether a first condition indicating whether a direction corresponding to at least one of the motion data or the OIS position information (e.g., a direction of the motion data or a direction in which the OIS is moved) corresponds to a phase difference direction of the first AF data is satisfied. The electronic device may determine whether to perform shading compensation based on the satisfaction of the first condition. A more specific description of the first condition will be described in more detail later with reference to FIG. 10. For example, the electronic device may determine whether to perform shading compensation based on the satisfaction of a second condition indicating whether an amount of movement of the electronic device during a specified time interval is greater than or equal to a threshold. For example, the electronic device may determine that the second condition is satisfied based on the integration of the motion data detected during the specified time interval is greater than or equal to a threshold. For example, the electronic device may be configured to determine that shading compensation is to be performed based on the satisfaction of both the first condition and the second condition. The at least one specified condition may be configured as a condition for determining whether shading compensation is required, and may include at least one condition other than the conditions described above.

According to one embodiment, based on the determination that shading compensation is not performed in operation 705, the electronic device may perform an AF operation based on the first AF data acquired in operation 701. The AF operation may include an operation of controlling an AF actuator included in the camera (e.g., an actuator of the AF module (315) of FIG. 3) according to an AF position determined based on the first AF data.

According to one embodiment, based on the determination that shading correction is performed in operation 705, the electronic device may perform shading correction on the first AF data in operation 707. In one embodiment, the electronic device may perform shading correction on the first AF data based on a look-up table (LUT) stored in a memory. The look-up table may include a gain value for correcting the first AF data according to OIS location information based on a shading pattern (e.g., the shading pattern illustrated in FIG. 6). The gain value may include a value for correcting a luminance value of a pixel included in the first AF data. The electronic device may obtain a gain value corresponding to the OIS location information through the look-up table, and obtain corrected second AF data by applying the obtained gain value to the first AF data. In one embodiment, the electronic device may also adjust the gain value based on the OIS location. For example, the electronic device may determine a weight based on the OIS location information, and adjust the gain value based on the determined weight. In operation 709, the electronic device can perform an AF operation based on the second AF data corrected in operation 707. The AF operation can include an operation of controlling an AF actuator included in the camera according to an AF position determined based on the second AF data.

According to one embodiment, in operation 711, the electronic device may determine whether to terminate the AF operation. For example, the electronic device may determine to terminate the AF operation if the focus is on a region of interest (ROI) based on the AF operation performed in operation 709. For example, the electronic device may determine to terminate the AF operation if the operation of the camera is determined to be deactivated or terminated (e.g., if the execution of the camera application is terminated). If the AF operation is determined not to be terminated, the electronic device may execute the operations illustrated in the flowchart (700) from operation 701.

In one embodiment, the electronic device may be configured to omit operation 705 and perform operation 707, which performs shading compensation.

In one embodiment, the electronic device can change the direction in which the phase detection is performed based on the direction of the motion data or the direction in which the OIS position is moved. For example, a 4PD image sensor in which PDs are arranged in a 2 x 2 array to correspond to one micro lens can detect phase difference data in a first direction (e.g., horizontal direction) or phase difference data in a second direction (e.g., vertical direction). In a state in which the horizontal phase detection is performed, if the value in the horizontal direction among the motion data continues to have a positive value or a negative value (or if the integral value for a specified section is a positive value or a negative value), it means that a horizontal movement is occurring, and therefore the electronic device can change the direction in which the phase detection is performed to a vertical direction. In a state in which the vertical phase detection is performed, if the value in the vertical direction among the motion data continues to have a positive value or a negative value (or if the integral value for a specified section is a positive value or a negative value), it means that a vertical movement is occurring, and therefore the electronic device can change the direction in which the phase detection is performed to a horizontal direction. In one embodiment, the electronic device may be configured to omit operation 707 of compensating for shading if the direction in which the phase detection is performed is changed and there is little movement of the electronic device corresponding to that direction.

FIGS. 8 and 9 illustrate the OIS operation of an electronic device according to one embodiment.

A gyro sensor (330) according to one embodiment can detect acceleration acting on three axes (e.g., X-axis, Y-axis, or Z-axis) of an electronic device (101), and the OIS operation can move on two axes (e.g., X-axis or Y-axis). FIGS. 8 and 9 show a position where a lens assembly (210) is placed by performing an OIS operation based on the X-axis component and the Y-axis component of the gyro sensor (330).

According to one embodiment, when the value of the motion data acquired from the gyro sensor is 0, the electronic device (101) may perform an OIS operation to control the OIS position (e.g., the x-axis/y-axis direction position of the lens assembly) so that the center of the lens is located at a first position (e.g., 800 of FIG. 8 and 900 of FIG. 9) that is the center of the aperture of the lens.

According to one embodiment, when motion data acquired from the gyro sensor includes an x-axis acceleration value (Gx) of a first magnitude, the electronic device (101) can perform an OIS operation to control an OIS position (e.g., an x-axis direction position of the lens assembly) so that the center of the lens is located at a second position (801) moved from the center of the aperture of the lens.

According to one embodiment, when motion data acquired from the gyro sensor includes an x-axis acceleration value (2Gx) of a second magnitude greater than the first magnitude, the electronic device (101) may perform an OIS operation to control an OIS position (e.g., an x-axis direction position of the lens assembly) such that the center of the lens is located at a third position (802) that is moved by a greater amount than the second position (801) from the center of the aperture of the lens.

According to one embodiment, when motion data acquired from a gyro sensor includes an acceleration value (Gy) in the y-axis direction, the electronic device (101) can perform an OIS operation to control the OIS position (e.g., the y-axis direction position of the lens assembly) so that the center of the lens is located at a fourth position (901) moved in the y-axis direction from the center of the aperture of the lens.

According to one embodiment, when motion data acquired from a gyro sensor includes an acceleration value (-Gy) in the -y-axis direction, the electronic device (101) may perform an OIS operation to control the OIS position (e.g., the y-axis direction position of the lens assembly) so that the center of the lens is located at a fifth position (902) moved in the -y-axis direction from the center of the aperture of the lens.

In one embodiment, since the electronic device performs the OIS operation based on the motion data, the electronic device may be configured to perform an operation (e.g., operation 705 of FIG. 7) of determining whether to perform shading compensation based on the motion data. In one embodiment, the electronic device may also be configured to perform an operation (e.g., operation 705 of FIG. 7) of determining whether to perform shading compensation based on a parameter (e.g., a parameter indicating an OIS position) obtained through a Hall sensor (e.g., the Hall sensor (320) of FIG. 3) disposed in the optical image stabilizer.

An electronic device according to one embodiment may determine whether shading compensation is required by identifying an OIS position based on at least one of an OIS movement direction or a parameter. The electronic device may compare a direction of the OIS with a direction of AF detection (phase difference direction). If the phase difference direction does not match the OIS eccentricity direction and is perpendicular, the electronic device may not perform shading compensation even if the lens assembly is in a specific direction. This is because shading exists in the vertical direction of the phase difference, but shading in the horizontal direction may be small or non-existent.

Figure 10 illustrates examples of determining whether to perform shading correction depending on the OIS operation direction when detecting a phase difference in the horizontal direction (x-axis direction).

In one embodiment, data acquired from an image sensor may include a repeating pattern as illustrated in FIG. 10. For example, the data illustrated in FIG. 10 may represent data acquired from an image sensor having a designated pattern. The designated pattern may include a pattern in which a 2 X 2 array of photodiodes arranged at positions corresponding to one micro lens are repeated. The designated pattern may include a pattern in which a 2 X 2 array of microlenses arranged to correspond to the same color pattern are repeated. Accordingly, the designated pattern may be a 4 X 4 array of photodiodes arranged to correspond to the same color pattern. Referring to FIG. 10, a pattern may be formed in units of 4 X 4 arrays within the data. The pattern illustrated in FIG. 10 may represent a pattern of pixel values included in the acquired data.

In the first data (910) configured based on information acquired from photodiodes of the image sensor while the OIS operation is performed in the Y-axis direction, shading included in values acquired from photodiodes arranged in the horizontal direction (911) may be small or non-existent (i.e., shading in the horizontal direction may be small or non-existent). Accordingly, since the influence of shading on phase difference detection in the horizontal direction (x-axis direction) is low, the electronic device can omit an operation for performing shading compensation and perform an AF operation.

In a state where an OIS operation is performed in the X-axis direction and the Y-axis direction (for example, in a state where the camera module moves diagonally with respect to the X-axis and the Y-axis), second data (920) configured based on information acquired from photodiodes of the image sensor may include shading that occurs in values acquired from photodiodes arranged in the vertical and horizontal directions (921) (i.e., shading in the horizontal/vertical directions may occur). Accordingly, since shading may affect phase difference detection in the horizontal direction (x-axis direction), the electronic device may perform an AF operation based on the AF data on which shading compensation has been performed.

Since no shading occurs due to the operation of the OIS in the third data (930) configured based on information acquired from the photodiodes of the image sensor in a state where the OIS position is at the center (e.g., the first position (800, 900) of FIG. 8 or FIG. 9), the electronic device can omit an operation for performing shading compensation and perform an AF operation.

In the fourth data (940) configured based on information acquired from photodiodes of the image sensor while the OIS operation is performed in the X-axis direction, shading may be included in values acquired from photodiodes arranged in the horizontal direction (941) (i.e., shading in the horizontal direction may occur). Accordingly, since shading may affect phase difference detection in the horizontal direction (x-axis direction), the electronic device may perform the AF operation based on the AF data on which shading compensation has been performed.

In one embodiment, when the phase difference direction and the OIS direction are identical, it is possible to adjust whether or not to perform shading correction and the amount of correction, but the electronic device including the 4PD sensor can also adjust the phase difference operation direction according to the direction requiring correction only when the phase difference direction is not important. Accordingly, shading correction can be omitted by changing the operation direction.

FIG. 11 illustrates an example of weights determined based on OIS positions according to one embodiment.

The M x N pixel-by-pixel luminance correction values for performing shading correction may be different depending on the OIS location. The M x N pixel-by-pixel luminance correction values are assigned to a look up table (LUT), and each correction value can be determined by considering the CRA characteristics of the micro lens provided for each unit pixel.

The coordinate values shown in Fig. 11 may indicate the OIS location. A higher brightness within the OIS range shown in Fig. 11 may indicate a higher weight value.

Fig. 11 illustrates a weight distribution applied when the operating direction of the OIS is the X-axis. For example, when the eccentricity in the X-axis direction is severe due to the OIS operation (section C), the correction value assigned to the LUT can be increased. Conversely, when the eccentricity in the X-axis direction is weak due to the OIS operation (section B), the correction value assigned to the LUT can be decreased. In addition, when the eccentricity in the X-axis direction due to the OIS operation is intermediate (section B), the correction values used in sections A and C can be considered and an average value can be applied.

By determining a correction value for performing shading correction using the weights illustrated in Fig. 11, it is possible to enable an electronic device to perform shading correction without storing a LUT for all OIS positions. Accordingly, the amount of storage capacity of the memory can be saved.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure belongs.

An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) according to one embodiment may include a sensor that outputs a signal for obtaining motion data corresponding to a movement of the electronic device. An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) according to one embodiment may include a lens assembly (e.g., the lens assembly (210) of FIG. 2 and the lens assembly (210) of FIG. 3) including at least one lens, an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (230) of FIG. 3) and a camera module (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2) including an optical image stabilizer (OIS) configured to move at least one of the lens assembly or the image sensor to perform an optical image stabilization operation based on motion data. An electronic device (e.g., the electronic device 101 of FIG. 1, the electronic device 101 of FIG. 3) according to one embodiment may include a processor (e.g., the processor 120 of FIG. 1, the processor 120 of FIG. 3) and a memory storing instructions executable by the processor. The instructions according to one embodiment may be configured to cause the electronic device to obtain first AF data from an image sensor. The instructions according to one embodiment may be configured to cause the electronic device to correct the first AF data based on a parameter for an OIS operation to generate second AF data. The instructions according to one embodiment may be configured to cause the electronic device to perform an AF (auto focus) function based on the second AF data.

According to one embodiment, the parameters may include first displacement values and second displacement values, which are coordinate values of the center of a lens assembly (e.g., 210 of FIG. 2 and 210 of FIG. 3) defined with the center of the image sensor (e.g., 230 of FIG. 2 and 230 of FIG. 3) as a reference point.

Instructions according to one embodiment may obtain a look up table (LUT) corresponding to at least one of a parameter or a Chief Ray Angle (CRA) of at least one lens, and correct first AF data based on the LUT.

Instructions according to one embodiment can identify a phase difference direction of first AF data and determine whether to perform correction based on a relationship between the phase difference direction and the parameter.

An instruction according to one embodiment may omit correction if the relationship between the phase difference direction and the parameter corresponds to a vertical relationship.

Instructions according to one embodiment can adjust the weights of a LUT for compensation when the relationship between the phase difference direction and the parameter corresponds to a horizontal relationship.

Instructions according to one embodiment can change the direction of phase detection based on first AF data when the direction of movement of at least one lens due to OIS operation is maintained for a defined time period or longer.

An instruction according to one embodiment may perform an AF (auto focus) function based on a first phase difference and a second phase difference included in the second AF data having a changed luminance value. The first phase difference may indicate a phase difference in a first direction, and the second phase difference may indicate a phase difference in a second direction different from the first direction.

An image sensor according to one embodiment (e.g., 230 of FIG. 2, 230 of FIG. 3) includes a plurality of unit pixels, each unit pixel including four or more PDs (photo diodes), the PDs being arranged adjacent to each other in a first direction and a second direction different from the first direction, and each of the unit pixels may include at least one color filter formed over the four or more PDs (photo diodes) included in the unit pixel, and at least one micro lens formed over the at least one color filter.

In one embodiment, the first direction can be perpendicular to the second direction.

An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to an embodiment may include an operation of outputting a signal for obtaining motion data corresponding to a movement of the electronic device. An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to an embodiment may include an operation of controlling a lens assembly (e.g., 210 of FIG. 2 and 210 of FIG. 3) or an image sensor (e.g., 230 of FIG. 2, 230 of FIG. 3) to perform an optical image stabilization (OIS) operation based on the motion data. An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to an embodiment may include an operation of obtaining first AF data from an image sensor (e.g., 230 of FIG. 2, 230 of FIG. 3). An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to one embodiment may include an operation of generating second AF data by correcting first AF data based on parameters for OIS operation. An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to one embodiment may include an operation of performing an AF (auto focus) function based on the second AF data.

According to one embodiment, the parameters may include a first displacement value and a second displacement value, which are coordinate values of the center of the lens assembly defined with the center of the image sensor as a reference point.

An operating method of an electronic device according to one embodiment may further include an operation of obtaining a look up table (LUT) corresponding to at least one of the parameters or a Chief Ray Angle (CRA) of the at least one lens; and an operation of correcting the first AF data based on the LUT.

An operating method of an electronic device according to one embodiment may further include an operation of checking a phase difference direction of the first AF data; and an operation of determining whether to perform the correction based on a relationship between the phase difference direction and the parameter.

The operating method of the electronic device according to one embodiment may omit the operation of generating the second AF data when the relationship between the phase difference direction and the parameter corresponds to a vertical relationship.

The operating method of the electronic device according to one embodiment may further include an operation of adjusting a weight of an LUT for the correction when the relationship between the phase difference direction and the parameter corresponds to a horizontal relationship.

An operating method of an electronic device according to one embodiment may further include an operation of changing a direction of phase difference detection based on the first AF data when a direction of movement of the at least one lens by the OIS operation is maintained for a defined time or longer.

The operating method of the electronic device according to one embodiment may further include an operation of performing an AF (auto focus) function based on a first phase difference and a second phase difference included in the second AF data having a changed luminance value. The first phase difference may indicate a phase difference for a first direction, and the second phase difference may indicate a phase difference for a second direction different from the first direction.

An image sensor according to one embodiment includes a plurality of unit pixels, each unit pixel including four or more PDs (photo diodes) and the PDs are arranged adjacent to each other in a first direction and a second direction different from the first direction, and each of the unit pixels may include at least one color filter formed over the four or more PDs (photo diodes) included in the unit pixel and at least one micro lens formed over the at least one color filter.

In one embodiment, the first direction can be perpendicular to the second direction.

An electronic device and an operating method thereof according to embodiments of the present disclosure can improve the accuracy of a focus adjustment function performed by the electronic device to capture an image.

An electronic device and an operating method thereof according to embodiments of the present disclosure can improve the accuracy of an auto-focus function by correcting auto-focus data based on a situation in which the electronic device captures an image. For example, the accuracy of an auto-focus function can be improved in a situation in which an image is captured by tracking a subject, a situation in which an image is captured while panning a camera, a situation in which a super-resolution image is captured using an optical image stabilizer, or a situation in which a panoramic image is captured.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art to which the present disclosure belongs.

Claims (15)

In electronic devices, A sensor that outputs a signal for obtaining motion data corresponding to the movement of the electronic device; A camera module comprising a lens assembly including at least one lens, an image sensor, and an optical image stabilizer (OIS) configured to move at least one of the lens assembly or the image sensor to perform an optical image stabilization (OIS) operation based on the motion data; processor; a memory storing instructions executable by the processor; The above instructions cause the electronic device to: Acquire first AF data from the image sensor, Generate second AF data by correcting the first AF data based on the parameters for the above OIS operation, An electronic device that performs an AF (auto focus) function based on the second AF data. In paragraph 1, The above parameters are, An electronic device including a first displacement value and a second displacement value, which are coordinate values of the center of the lens assembly defined with the center of the image sensor as a reference point. In the second paragraph, At least one of the above instructions causes the electronic device to: Obtain a look up table (LUT) corresponding to at least one of the above parameters or the CRA (Chief Ray Angle) of the at least one lens, An electronic device for correcting the first AF data based on the LUT. In the second paragraph, At least one of the above instructions causes the electronic device to: Check the phase difference direction of the above first AF data, An electronic device that determines whether to perform the correction based on the relationship between the phase difference direction and the parameters. In paragraph 4, At least one of the above instructions causes the electronic device to: An electronic device that allows the correction to be omitted when the relationship between the phase difference direction and the parameter corresponds to a vertical relationship. In paragraph 4, At least one of the above instructions causes the electronic device to: An electronic device that adjusts the weight of the LUT for the correction when the relationship between the phase difference direction and the parameter corresponds to a horizontal relationship. In paragraph 1, At least one of the above instructions causes the electronic device to: An electronic device that changes the direction of phase difference detection based on the first AF data when the direction of movement of at least one lens by the OIS operation is maintained for a defined period of time or longer. In paragraph 1, At least one of the above instructions causes the electronic device to: The AF (auto focus) function is performed based on the first phase difference and the second phase difference included in the second AF data whose brightness value has been changed. An electronic device wherein the first phase difference indicates a phase difference in a first direction, and the second phase difference indicates a phase difference in a second direction different from the first direction. In paragraph 1, The above image sensor, A plurality of unit pixels, each unit pixel including four or more PDs (photo diodes), the PDs being arranged adjacent to each other in a first direction and a second direction different from the first direction, An electronic device wherein each of the unit pixels includes at least one color filter formed on four or more photo diodes (PDs) included in the unit pixel and at least one micro lens formed on the at least one color filter. In Article 9, An electronic device, wherein the first direction is perpendicular to the second direction. In a method of operating an electronic device, An action of outputting a signal for obtaining motion data corresponding to the movement of the electronic device; An operation of controlling the lens assembly or the image sensor to perform an optical image stabilization (OIS) operation based on the motion data; An operation of acquiring first AF data from the image sensor; An operation of generating second AF data by correcting the first AF data based on the parameters for the OIS operation; and An operating method of an electronic device including an operation of performing an AF (auto focus) function based on the second AF data. In Article 11, The above parameters are, An operating method of an electronic device including a first displacement value and a second displacement value, which are coordinate values of the center of the lens assembly defined with the center of the image sensor as a reference point. In Article 12, An operation of obtaining a look up table (LUT) corresponding to at least one of the above parameters or the CRA (Chief Ray Angle) of the at least one lens; and An operating method of an electronic device further comprising an operation of correcting the first AF data based on the LUT. In Article 12, An operation for checking the phase difference direction of the above first AF data; and A method of operating an electronic device further comprising an operation of determining whether to perform the correction based on the relationship between the phase difference direction and the parameters. In Article 14, An operating method of an electronic device, wherein an operation of generating the second AF data is omitted when the relationship between the phase difference direction and the parameter corresponds to a vertical relationship.