WO2025089769A1 - Electronic device and image editing method - Google Patents

Abstract

The electronic device of the present disclosure comprises: a display; a memory for storing instructions; and a processor, wherein, when executed by the processor, the instructions allow the electronic device to: select an image on the basis of a user input; execute an image editing application including a user interface related to position movement of an object; check depth information of the image; recognize and classify the object in the image; change the position of the selected object on the image on the basis of a user input through the user interface related to position movement of the object; and perform an out-painting operation and/or an in-painting operation on one of the image and the selected object on the basis of the position of the selected object.

Description

Electronic devices and image editing methods

The present disclosure relates to an electronic device and an image editing method.

Various electronic devices, such as smart phones, tablet PCs, portable multimedia players (PMPs), personal digital assistants (PDAs), laptop personal computers (Laptop PCs), and wearable devices such as wrist watches and head-mounted displays (HMDs), contain cameras and can capture images using cameras.

There is a problem that when an object moves on an image, the amount of information that the electronic device must calculate is large, and the spatial information is ignored, resulting in an unnatural appearance of the image containing the moved object.

The electronic device and image editing method of the present disclosure aim to move an object on an image based on depth information on the image.

An electronic device of the present disclosure may include a display; a memory storing instructions; and a processor.

The instructions of the present disclosure, when executed by the processor, may cause the electronic device to: execute an image editing application including a user interface for selecting an image based on a user input, the image editing application including a user interface for moving a position of an object, determine depth information of the image, recognize and classify an object in the image, change a position of the selected object on the image based on the user input through the user interface for moving the position of the object, and perform an outpainting and/or inpainting operation on one of the image or the selected object based on the position of the selected object.

The image editing method of the present disclosure may include: an operation of selecting an image based on a user input; an operation of executing an image editing application including a user interface relating to movement of a position of an object; an operation of checking depth information of the image; an operation of recognizing and classifying an object in the image; an operation of changing a position of the selected object on the image based on a user input through the user interface relating to movement of the position of the object; and an operation of performing an outpainting and/or inpainting operation on one of the image or the selected object based on the position of the selected object.

The electronic device and image editing method of the present disclosure can obtain an image in which the position of an object is naturally edited by moving the object based on depth information on the image.

In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

FIG. 2 is a block diagram of an electronic device according to one embodiment of the present disclosure.

FIG. 3 is a drawing illustrating an image editing method of an electronic device according to one embodiment of the present disclosure.

FIG. 4 is a drawing illustrating an image editing method of an electronic device according to one embodiment of the present disclosure.

FIG. 5 is a drawing showing an image editing method according to one embodiment of the present disclosure.

FIG. 6 is a drawing showing an image editing method according to one embodiment of the present disclosure.

FIG. 7 is a drawing illustrating an image editing method according to one embodiment of the present disclosure.

FIG. 8A is a diagram illustrating a user interface for object position editing based on an indicator according to one embodiment of the present disclosure.

FIG. 8b is a diagram illustrating a user interface for object position editing based on an indicator according to one embodiment of the present disclosure.

FIG. 8c is a diagram illustrating a user interface for object position editing based on an indicator according to one embodiment of the present disclosure.

FIG. 9 is a drawing showing an image editing method according to one embodiment of the present disclosure.

FIG. 10a is a diagram showing an image when a selected object is moved in one embodiment of the present disclosure.

FIG. 10b is a diagram showing an image when a selected object is moved in one embodiment of the present disclosure.

FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store result data in a nonvolatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or processor) or a secondary processor (123) (e.g., a graphic processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a given function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one of the components of the electronic device (101) (e.g., the display module (160), the sensor module (176), or the communication module (190)), for example, while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing device) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. Such learning may be performed, for example, in the electronic device (101) itself on which the artificial intelligence model is executed, or may be performed through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may additionally or alternatively include a software structure.

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or nonvolatile memory (134).

The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The audio output module (155) can output an audio signal to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented separately from the speaker or as a part thereof.

The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can obtain sound through an input module (150), or output sound through an audio output module (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or a headphone) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect an operating state (e.g., power or temperature) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electric signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user can perceive through a tactile or kinesthetic sense. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery (189) can power at least one component of the electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108)), and performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., the application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The wireless communication module (192) can support a 5G network and next-generation communication technology after a 4G network, for example, NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), terminal power minimization and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) may support various requirements specified in an electronic device (101), an external electronic device (e.g., an electronic device (104)), or a network system (e.g., a second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL) each, or 1 ms or less for round trip) for URLLC realization.

The antenna module (197) can transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module (197) can include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) can include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), can be selected from the plurality of antennas by, for example, the communication module (190). A signal or power can be transmitted or received between the communication module (190) and the external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) can be additionally formed as a part of the antenna module (197).

According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC positioned on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) positioned on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

At least some of the above components may be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

In one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). In one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of executing the function or service itself or in addition, request one or more external electronic devices to perform at least a part of the function or service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide an ultra-low latency service by using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an IoT (Internet of Things) device. The server (108) may be an intelligent server using machine learning and/or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199). The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

FIG. 2 is a block diagram of an electronic device (101) according to one embodiment of the present disclosure.

In one embodiment, the electronic device (101) may include a camera (280), a display (260), a processor (120), and memory (130).

In one embodiment, the camera (280) of FIG. 2 may include the camera module (180) of FIG. 1. The display (260) of FIG. 2 may include the display module (160) of FIG. 1. The processor (120) of FIG. 2 may include the processor (120) of FIG. 1. The memory (130) of FIG. 2 may include the memory (130) of FIG. 1.

In one embodiment, the memory (130) may include a depth information generation module (131), an object recognition/classification module (132), an out painting/in painting module (133), and/or a re-lighting module (134).

In one embodiment, the depth information generation module (131), the object recognition/classification module (132), the outpainting/inpainting module (133), and/or the rewriting module (134) may be programs and/or applications stored in the memory (130).

In one embodiment, each of the depth information generation module (131), the object recognition/classification module (132), the outpainting/inpainting module (133), and/or the rewriting module (134) may be a generative artificial intelligence (AI).

In one embodiment, the depth information generation module (131), the object recognition/classification module (132), the outpainting/inpainting module (133), and/or the rewriting module (134) may be a single generative AI.

In one embodiment, the generative AI may perform operations of a depth information generation module (131), an object recognition/classification module (132), an out painting/in painting module (133), and/or a rewriting module (134) based on input defined by a prompt under the control of the processor (120).

In one embodiment, the depth information generation module (131) may generate a depth map for an image to be edited under the control of the processor (120). The depth information generation module (131) may include not only a generative AI but also a general AI. The depth map may include information related to a surface distance of an object included in an image from a view point. The depth information generation module (131) may be an artificial intelligence learned to calculate depth information from a single image.

In one embodiment, the depth information may include information acquired using a sensor that acquires depth information when capturing an image (e.g., lidar or time of flight (ToF)). The depth information may be stored in metadata within the image.

In one embodiment, the depth information generation module (131) can update the depth information of the image based on the result of the object moving within the image. Until the user finishes editing the object movement within the image, the electronic device (101) can provide image editing that maintains perspective when the object is moved using the updated depth information.

In one embodiment, the object recognition/classification module (132) can perform segmentation in an image under the control of the processor (120). The object recognition/classification module (132) can detect and classify objects through segmentation in an image under the control of the processor (120). The object recognition/classification module (132) can identify information about a movable object among at least one object detected in the image under the control of the processor (120). The movable object can be, for example, an object whose position within the image can be changed. For example, the movable object can be moved within the image based on an input via a user interface for editing the position of the object.

In one embodiment, the object recognition/classification module (132) can, under the control of the processor (120), identify objects recognized or recognized in an image whose size is greater than a specified size as movable objects.

In one embodiment, the object recognition/classification module (132) may, under the control of the processor (120), identify an object corresponding to a specified object classification (e.g., a person, a flower pot, a piece of furniture, etc.) as a movable object.

In one embodiment, the object recognition/classification module (132) may, under the control of the processor (120), perform a segmentation operation only on objects corresponding to a classification of a specified object.

In one embodiment, the object recognition/classification model (132) may, under the control of the processor (120), perform an operation of recognizing and classifying another object corresponding to a movable location of a movable object within an image. For example, if an image including a living room includes a person, a sofa, and a table as objects, the object recognition/classification model (132) may, under the control of the processor (120), recognize the person within the image as a movable object. If the person is moved behind the sofa based on an input via a user interface for editing an object location, the electronic device (101) may provide a resulting image in which a portion of the person area is covered by the sofa area.

In one embodiment, the object recognition/classification model (132) may perform an operation (e.g., matting) to separate a user-selected object in an image from a background under the control of the processor (120). The object recognition/classification model (132) may perform an operation to smoothly process a boundary area of a separated object by removing background residue under the control of the processor (120).

In one embodiment, the out painting/in painting module (133) can, under the control of the processor (120), identify a first region on the image in which a movable object identified in the image is to be emptied as it moves in position, and/or a second region of an object in which a movable object identified in the image is to be generated as it moves in position. The out painting/in painting module (133) can, under the control of the processor (120), perform a filling operation based on objects and/or pixels recognized or detected in the image for the first region and the second region. The filling operation can include an operation of adding objects and/or pixels recognized or detected in the region to be filled in.

In one embodiment, the out painting/in painting module (133) may determine, under the control of the processor (120), that a part of the movable object has moved outside the image boundary. The out painting/in painting module (133) may perform an operation of displaying the entire shape of the movable object when, under the control of the processor (120), a part of the movable object has moved outside the image boundary. The out painting/in painting module (133) may analyze an area of the original image that requires expansion so that the entire shape of the moved object can be displayed based on the original image size (e.g., resolution) under the control of the processor (120), and may perform out painting on the determined area requiring expansion.

In one embodiment, under the control of the re-lighting module (134), the processor (120), when an object (e.g., a movable object) moves its position on the image by a user input, light and/or shadow corresponding to the moved object can be adjusted. Under the control of the re-lighting module (134), the processor (120), the light and/or shadow can be repeatedly adjusted corresponding to the moved object. Under the control of the re-lighting module (134), the processor (120), an image with adjusted light and/or shadow corresponding to the moved object can be displayed as a preview.

In one embodiment, a re-lighting module (134), under the control of the processor (120), may perform operations to adjust light and/or shadows on a background image based on the moved object position and updated depth information as the object changes.

In one embodiment, the memory (130) may further include an image editing management module. The image editing management module may display a user interface related to editing controls when editing an image.

FIG. 3 is a drawing showing an image editing method of an electronic device (101) according to one embodiment of the present disclosure. In one embodiment, the electronic device (101) may execute an image editing application in operation 301 under the control of the processor (120).

In one embodiment, the memory (130) may include instructions including an image editing method of the electronic device (101) of FIG. 3. The instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to execute an image editing application.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 301, may select one image from among a plurality of images displayed on a gallery application by user input.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to select one image from among a plurality of images displayed on a gallery application based on a user input.

In one embodiment, the electronic device (101), under the control of the processor (120), may execute an image editing application by selecting one image from among a plurality of images displayed on a gallery application in operation 301.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to execute an image editing application when selecting one image from among a plurality of images displayed on the gallery application.

In one embodiment, the electronic device (101), under the control of the processor (120), may, in operation 301, execute an image editing application to display at least one selected image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to display at least one selected image when executing an image editing application.

In one embodiment, the electronic device (101), under the control of the processor (120), may determine depth information of an image in operation 303. The image may be displayed in a running image editing application. The image may be an image selected for image editing.

However, it is not limited thereto, and the electronic device (101) can first perform operation 305 regarding recognition and classification of an object under the control of the processor (120) and then check depth information from the image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to determine depth information of an image. In one embodiment, the electronic device (101), under the control of the processor (120), may generate depth information of an image using the depth information generation module (131) in operation 303.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause electronic device (101) to generate depth information of an image using depth information generation module (131).

In one embodiment, the electronic device (101), under control of the processor (120), may generate a depth map of an image in operation 303.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to generate a depth map of an image.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 305, may recognize and/or detect and classify an object in an image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to recognize and/or detect and classify objects in an image.

In one embodiment, the electronic device (101), under the control of the processor (120), may perform segmentation on an image in operation 305.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to perform segmentation on an image.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 305, may use the object recognition/classification module (132) to recognize and/or detect and classify an object in an image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to recognize and/or detect and classify objects in an image using the object recognition/classification module (132).

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 305, may recognize and/or detect and classify an object through segmentation in an image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to recognize and/or detect and classify objects through segmentation in an image.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 305, may identify information about a movable object among at least one object recognized and/or detected in the image. The movable object may be, for example, an object whose position within the image may change.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to identify information about a movable object among at least one object recognized and/or detected in the image.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 305, may identify an object recognized and/or detected in an image whose size is greater than a specified size as a movable object.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to identify objects recognized and/or detected in an image whose size is greater than a specified size as movable objects.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 305, may identify an object corresponding to a classification of a specified object (e.g., a person, a flower pot, a piece of furniture, etc.) as a movable object.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to identify an object corresponding to a specified object classification (e.g., a person, a flower pot, a piece of furniture, etc.) as a movable object.

In one embodiment, the electronic device (101), under the control of the processor (120), may perform a segmentation operation only for objects corresponding to a classification of a specified object in operation 305.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform a segmentation operation only for objects corresponding to a classification of a specified object.

In one embodiment, the electronic device (101), under the control of the processor (120), may perform outpainting/inpainting on an image based on object recognition information or depth information in operation 307.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform outpainting/inpainting on an image based on object recognition information or depth information.

In one embodiment, the electronic device (101) may, under the control of the processor (120), in operation 307, perform outpainting/inpainting on an image based on object recognition information or depth information through the outpainting/inpainting module (133).

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform outpainting/inpainting on an image through the outpainting/inpainting module (133) based on object recognition information or depth information.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 307, can identify a first area on the image where the movable object identified in the image is to be emptied as the position moves, and/or a second area of the object where the movable object identified in the image is to be created as the position moves.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to identify a first region of the image in which a movable object identified in the image is to be emptied as it moves in position, and/or a second region of an object in which a movable object identified in the image is to be created as it moves in position.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 307, may perform a filling operation based on objects and/or pixels recognized and/or detected on an image for the first area and the second area, under the control of the processor (120). The filling operation may include adding recognized or detected objects and/or pixels to the area to be filled.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform a filling operation based on objects and/or pixels recognized and/or detected on the image for the first area and the second area.

However, it is not limited thereto, and the electronic device (101) may perform out-painting/in-painting when the position of the object on the image changes based on the object recognition information or the depth information under the control of the processor (120). For example, in operation 311 instead of operation 307, the electronic device (101) may perform out-painting/in-painting based on the object recognition information or the depth information under the control of the processor (120). The instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform out-painting/in-painting when the position of the object on the image changes based on the object recognition information or the depth information.

In one embodiment, the electronic device (101), under control of the processor (120), in operation 308, may display a user interface for editing an object location.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to display a user interface for editing object locations.

In one embodiment, the electronic device (101), under control of the processor (120), in operation 308, may display a user interface for editing an object position on the display (260).

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to display a user interface for editing object positions on the display (260).

In one embodiment, the electronic device (101), under control of the processor (120), in operation 308, may display a user interface for editing object positions on an image editing application.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to display a user interface for object position editing on an image editing application.

In one embodiment, a user interface for editing an object location may include a user interface that displays an image highlighting an object selected on an image. The object selected on the image may be selected by user input for positioning movement within the image.

In one embodiment, a user interface for editing object positions can indicate positions between objects with text and/or indicators (or images) based on positions of objects included on a selected image. A user interface for editing object positions can indicate positions between objects with text and/or indicators (or images) in proportion to positions of objects included on a selected image.

For example, a selected image may contain a table and a wall as objects, and the objects may be arranged in the order of table and wall from the observation point of view. A user interface for editing object positions may display the positions between objects in the order of table and wall.

In one embodiment, the user interface for editing object positions may include a sphere-shaped interface and/or a slide-shaped interface. However, the present invention is not limited thereto, and in one embodiment, the user interface for editing object positions may include moving an object in response to a user's drag input on a selected image.

In one embodiment, a user interface for editing an object position may indicate an object (e.g., a person or a person object) that a user wishes to move in a selected image as a first indicator. The first indicator may include a user interface (520) that highlights an outline of the selected object as shown in FIG. 5.

In one embodiment, a user interface for editing object positions may display a second indicator indicating an object(s) corresponding to a movable position in the image image. The second indicator may include a user interface (530) indicating an object(s) corresponding to a movable position in the image of FIG. 5.

In one embodiment, when an object (e.g., a table or a table object) corresponding to a desired location is selected through a user input input into a user interface for editing an object location, the electronic device (101) may, under the control of the processor (120), move the object (e.g., a person or a person object) corresponding to the desired location to be moved behind the object (e.g., the table) corresponding to the desired location so that the object (e.g., the table) corresponding to the desired location is located at the top layer (e.g., the closest distance from the user, or the top layer among overlapping objects) based on depth information (e.g., z-depth). As the object (e.g., the person or the person object) to be moved moves behind the object (e.g., the table) corresponding to the desired location, the electronic device (101) may, under the control of the processor (120), apply perspective based on the depth information to reduce the size of the object (e.g., the person or the person object) to be moved and display it on an image.

In one embodiment, the electronic device (101), under the control of the processor (120), may determine, in operation 309, whether there is object movement on the image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to determine whether there is object movement in the image.

In one embodiment, the electronic device (101), under the control of the processor (120), may determine, in operation 309, whether there is an object movement based on user input.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to determine whether there is object movement based on user input.

In one embodiment, the user input may include an input for moving an indicator representing an object on a user interface for editing the object's position.

In one embodiment, the user input may include an input for moving an object on a selected image.

In one embodiment, if there is an object movement, the electronic device (101) may branch from operation 309 to operation 311.

In one embodiment, if there is no object movement, the electronic device (101) may branch from operation 309 to operation 301.

In one embodiment, the electronic device (101) can, under the control of the processor (120), change the position of an object on an image in operation 311. When the position of an object on an image is changed, the object whose position has been moved (or the object that is moving) can be positioned with its size changed based on depth information. When the size of the object that has been moved (or the object that is moving) is changed based on depth information, the object on the image can have a sense of perspective, thereby reducing the sense of incongruity.

In one embodiment, instructions stored in memory (130), when executed by the processor (120), may cause the electronic device (101) to change the position of an object on an image.

In one embodiment, when an object moved (or a moving object) on an image becomes larger, the electronic device (101), under the control of the processor (120), in operation 311, can improve the image quality of the object moved (or a moving object) and display it on the image.

In one embodiment, when an object moved (or an object in motion) on an image becomes larger, instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to improve the image quality (e.g., perform super resolution processing) of the object moved (or an object in motion) and display it on the image.

In one embodiment, the electronic device (101), under the control of the processor (120), may perform outpainting/inpainting when the position of an object on an image changes in operation 311.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 311, when the position of an object on an image is changed, may perform an outpainting/inpainting operation of the object and/or image whose position has been changed based on object recognition information or depth information.

However, this is not limited to this, and even if the position of an object on an image is changed, if the object or image whose position has been changed is not obscured or there is no obscured area, the electronic device (101) may not perform an out-painting/in-painting operation under the control of the processor (120).

In one embodiment, the electronic device (101) may, under the control of the processor (120), determine an area to be covered or filled in on an object or an image whose position has changed due to a change in the position of the object on the image, and perform an outpainting/inpainting operation. The area to be covered may include an area that should be removed by perspective or depth information according to the movement of the object. The area to be filled in may include an area that should be added by perspective or depth information according to the movement of the object.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform outpainting/inpainting when the position of an object on an image changes.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to determine an object whose position has changed due to a change in the position of the object on the image or an area to be covered or filled in the image, and perform an out-painting/in-painting operation.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 313, may perform a re-lighting operation on an image including an object whose position has changed. In one embodiment, the electronic device (101), under the control of the processor (120), may display light and/or shadow corresponding to the object whose position has changed on the image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform a re-lighting operation on an image that includes an object whose position has changed.

In one embodiment, the electronic device (101), under the control of the processor (120), may, in operation 313, display a re-lighted image as a preview on the display (260).

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to display a re-lighted image as a preview on the display (260).

In one embodiment, the electronic device (101), under the control of the processor (120), may determine, at operation 315, whether the image editing application has been terminated.

In one embodiment, instructions stored in memory (130), when executed by the processor (120), may cause the electronic device (101) to determine whether the image editing application has been terminated.

In one embodiment, when the image editing application is terminated, the electronic device (101) may branch from operation 315 to operation 317.

In one embodiment, if the image editing application has not been terminated, the electronic device (101) may branch from operation 315 to operation 301.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 317, may provide an image of a changed object location. In one embodiment, the electronic device (101), under the control of the processor (120), in operation 317, may cause the image of a changed object location to be displayed on the display (160). In one embodiment, the electronic device (101), under the control of the processor (120), in operation 317, may cause the image of a changed object location to be stored in the memory (130).

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to provide an image of an object having a changed position. In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to display an image of an object having a changed position on the display (160). In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to store an image of an object having a changed position in the memory (130).

FIG. 4 is a drawing showing an image editing method of an electronic device (101) according to one embodiment of the present disclosure.

In one embodiment, the electronic device (101), under control of the processor (120), may, in operation 401, select an object from a first image.

In one embodiment, the memory (130) may include instructions including an image editing method of the electronic device (101) of FIG. 3. The instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to execute an image editing application.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to select an object in a first image.

In one embodiment, the object selected in the first image may be an object selected by a user input. When the electronic device (101) receives a user input for an object included in the first image, the electronic device (101) may, under the control of the processor (120), select the object in the first image. The user input for the object included in the first image may be, for example, a touch input of long pressing the object included in the first image.

In one embodiment, the electronic device (101), under the control of the processor (120), may, in operation 401, select an object included in a first image from among a plurality of images displayed on a gallery application. The object selected through the first image may be a first object.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to select an object included in a first image from among a plurality of images displayed on a gallery application.

In one embodiment, the electronic device (101), under the control of the processor (120), may, in operation 403, determine depth information of the first image and/or the second image.

However, it is not limited thereto, and the electronic device (101) may first perform operation 405 regarding recognition and classification of an object in the second image under the control of the processor (120), and then check depth information in the first image and/or the second image.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to determine depth information of the first image and/or the second image.

In one embodiment, the second image may be an image acquired through the camera (280). The second image may be displayed on the display (260) as a preview image. The second image may be an image being acquired in real time through the camera (280).

In one embodiment, the second image may be an image being displayed via the display (260).

In one embodiment, the electronic device (101), under the control of the processor (120), may generate depth information of the first image and/or the second image using the depth information generation module (131) in operation 403.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to generate depth information of the first image and/or the second image.

In one embodiment, the electronic device (101), under control of the processor (120), may generate a depth map of the second image in operation 403.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to generate a depth map of the second image.

In one embodiment, the electronic device (101), under control of the processor (120), in operation 405, may recognize and/or detect and classify an object in the second image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to recognize and/or detect and classify an object in the second image.

In one embodiment, the electronic device (101), under the control of the processor (120), may perform segmentation on the second image in operation 405.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to perform segmentation on the second image.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 405, may use the object recognition/classification module (132) to recognize and/or detect and classify an object in the second image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to recognize and/or detect and classify an object in the second image using the object recognition/classification module (132).

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 405, may recognize and/or detect and classify an object through segmentation in the second image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to recognize and/or detect and classify an object through segmentation in the second image.

In one embodiment, the electronic device (101), under the control of the processor (120), may perform outpainting/inpainting on the second image based on object recognition information or depth information in operation 407.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform outpainting/inpainting on the second image based on object recognition information or depth information.

In one embodiment, the electronic device (101) may, under the control of the processor (120), perform outpainting/inpainting on the second image through the outpainting/inpainting module (133) in operation 407. The object selected through the first image may be the first object.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform out-painting/in-painting on the second image through the out-painting/in-painting module (133).

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 407, can identify a first region on an image in which a first object is emptied as it moves in position in the second image, and/or a second region of an object in which the first object is to be generated as it moves in position in the second image.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to identify a first region of an image in which a first object is emptied as it moves in position in the second image, and/or a second region of an object in which the first object is to be generated as it moves in position in the second image.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 407, may perform a filling operation based on objects and/or pixels recognized and/or detected on a second image for the first area and the second area based on object recognition information or depth information, under the control of the processor (120).

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform a filling operation based on objects and/or pixels recognized and/or detected on a second image for the first area and the second area based on object recognition information or depth information.

However, the present invention is not limited thereto, and the electronic device (101) may perform out-painting/in-painting when the position of the first object on the second image is changed under the control of the processor (120). For example, in operation 411 instead of operation 407, the electronic device (101) may perform out-painting/in-painting based on object recognition information or depth information under the control of the processor (120). The instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform out-painting/in-painting when the position of the first object on the second image is changed.

In one embodiment, the electronic device (101), under control of the processor (120), in operation 408, may display a user interface for editing an object location.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to display a user interface for editing object locations.

In one embodiment, the electronic device (101), under control of the processor (120), in operation 408, may display a user interface for editing an object position on the display (260).

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to display a user interface for editing object positions on the display (260).

In one embodiment, the electronic device (101), under control of the processor (120), in operation 408, may display a user interface for editing object positions on the image editing application.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to display a user interface for object position editing on an image editing application.

In one embodiment, a user interface for editing an object location may include a user interface that displays an image highlighting a first object on a second image.

In one embodiment, the user interface for editing object locations can indicate locations between objects with text and/or indicators (or images) based on locations of objects included in the second image. The user interface for editing object locations can indicate locations between objects with text and/or indicators (or images) in proportion to locations of objects included in the second image.

In one embodiment, the user interface for editing an object position may include a sphere-shaped interface and/or a slide-shaped interface. However, the present invention is not limited thereto, and in one embodiment, the user interface for editing an object position may include moving a first object in response to a user's drag input on a second image.

In one embodiment, the electronic device (101), under the control of the processor (120), may determine, in operation 409, whether there is movement of a selected object. The selected object may be a first object.

In one embodiment, instructions stored in memory (130), when executed by processor (120), may cause the electronic device (101) to determine whether there is movement of a selected object.

In one embodiment, the electronic device (101), under the control of the processor (120), may determine, in operation 409, whether there is movement of a selected object based on user input.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to determine whether there is movement of a selected object based on user input.

In one embodiment, if there is a selected object movement, the electronic device (101) may branch from operation 409 to operation 411.

In one embodiment, if there is no object movement, the electronic device (101) may branch from operation 409 to operation 401.

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 411, can change the position of the selected object (or the first object) on the second image. When changing the position of the selected object (or the first object) on the second image, the selected object (or the first object) can be positioned with its size changed based on the depth information. When the size of the selected object (or the first object) is changed based on the depth information, the selected object (or the first object) on the second image can have a sense of perspective, thereby reducing the sense of incongruity.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to change the position of the selected object (or, the first object) on the second image. When changing the position of the selected object (or, the first object) on the second image, the selected object (or, the first object) may be arranged with its size changed based on the depth information.

In one embodiment, when an object moved (or an object in motion) on an image becomes larger, the electronic device (101) may, under the control of the processor (120), improve the image quality of the object moved (or an object in motion) in operation 411 (e.g., perform super resolution processing) and display the object in the image.

In one embodiment, when an object moved (or an object in motion) on an image becomes larger, instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to improve the image quality (e.g., perform super resolution processing) of the object moved (or an object in motion) and display it on the image.

In one embodiment, the electronic device (101) may perform out-painting/in-painting when the position of the selected object (or the first object) on the second image changes under the control of the processor (120).

In one embodiment, the electronic device (101), under the control of the processor (120), in operation 411, may perform an outpainting/inpainting operation of the first object and/or the second image based on object recognition information or depth information when the position of the first object on the image changes.

However, this is not limited thereto, and if the first object or the second image is not obscured or there is no obscured area even if the position of the object on the image changes, the electronic device (101) may not perform the out-painting/in-painting operation under the control of the processor (120).

In one embodiment, the electronic device (101) may, under the control of the processor (120), determine an area to be covered or filled in in the first object or the second image according to a change in the position of the object on the image, and perform an out-painting/in-painting operation.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform out-painting/in-painting when the position of the selected object (or the first object) on the second image changes.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to determine an area to be covered or filled in in the first object or second image according to a change in the position of the object, and perform an out-painting/in-painting operation.

In one embodiment, the electronic device (101), under the control of the processor (120), may perform a re-lighting operation on a second image including the selected object (or the first object) whose position has been changed, in operation 413.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to perform a re-lighting operation on a second image that includes a selected object (or, the first object) whose position has changed.

In one embodiment, the electronic device (101) may, under the control of the processor (120), display light and/or shadow corresponding to the selected object (or first object) whose position has changed on the second image.

In one embodiment, the electronic device (101), under the control of the processor (120), may, in operation 413, display the re-lighted second image as a preview on the display (260).

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to display a re-lighted second image as a preview on the display (260).

In one embodiment, the electronic device (101), under the control of the processor (120), may, in operation 416, provide a second image in which the position of the selected object (or the first object) has been changed.

In one embodiment, the electronic device (101), under the control of the processor (120), may, in operation 416, display a second image with a changed position of the selected object (or the first object) on the display (160).

In one embodiment, the electronic device (101), under the control of the processor (120), may store a second image in which the position of the selected object (or the first object) is changed in the memory (130) in operation 416.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to provide a second image in which the position of the selected object (or, the first object) has been changed.

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to display a second image with a changed position of the selected object (or, the first object) on the display (160).

In one embodiment, the instructions stored in the memory (130), when executed by the processor (120), may cause the electronic device (101) to store a second image in which the position of the selected object (or, the first object) has been changed in the memory (130).

FIG. 5 is a drawing showing an image editing method according to one embodiment of the present disclosure.

Screens 501, 503, and 505 are diagrams showing image editing operations on an image editing application.

In screen 501, the image may include at least one object (511, 513, 515). For example, a first object (511) may include a person, a second object (513) may include a table, and a third object (515) may include a wall.

In the 501 screen, the electronic device (101) can recognize and/or detect and classify an object in an image under the control of the processor (120). The electronic device (101) can identify an object corresponding to a classification of a specified object (e.g., a person, a flower pot, furniture, etc.) as a movable object under the control of the processor (120). The electronic device (101) can perform a segmentation operation only for an object corresponding to a classification of a specified object under the control of the processor (120).

In the screen 501, when an image selects one object from among at least one object (511, 513, 515), a user interface (520) for highlighting an outline of the selected object may be displayed. By displaying a user interface (520) for highlighting an outline of the selected object, the visibility of the object selected by the user may be improved. The first indicator may include a user interface (520) for highlighting an outline of the selected object.

In the 503 screen, the selected object can move its position on the image by user input. For example, the first object (511) can move its position on the image by user input (e.g., drag input). When the position of the first object (511) on the image is changed, the first object (511) whose position has been changed can be placed with its size changed based on depth information.

On the screen 503, the electronic device (101) can display a user interface (530) indicating object(s) corresponding to a movable position in the image on the display (160). The second indicator can include the user interface (530) indicating object(s) corresponding to a movable position in the image. However, the present invention is not limited thereto, and the electronic device (101) may not display the user interface (530) or the second indicator indicating object(s) corresponding to the movable position. On the screen 505, the electronic device (101) can perform out-painting/in-painting on the image under the control of the processor (120). The electronic device (101) can perform the out-painting/in-painting operation on the image before the movement of the first object (511) under the control of the processor (120). However, it is not limited thereto, and the electronic device (101), under the control of the processor (120), may perform an out-painting/in-painting operation on the image after the movement of the first object (511). Comparing the screens 501 and 505, the first object (511) was placed in front of the second object (513) and then moved to be placed between the second object (513) and the third object (515). The space left empty due to the movement of the first object (511) may be filled with a table image corresponding to the second object (513), and at least a part of the first object (511) may be obscured.

On the screen 505, the electronic device (101) can perform a rewriting operation on the image under the control of the processor (120). The electronic device (101) can display light and/or shadow corresponding to the first object (511) whose position has changed on the image under the control of the processor (120).

FIG. 6 is a drawing showing an image editing method according to one embodiment of the present disclosure.

Screens 601, 603, and 605 are diagrams showing image editing operations on an image editing application.

In screen 601, the image may include at least one object (511, 513, 515). For example, a first object (511) may include a person, a second object (513) may include a table, and a third object (515) may include a wall.

In the 601 screen, the electronic device (101) can recognize and/or detect and classify an object in an image under the control of the processor (120). The electronic device (101) can identify an object corresponding to a classification of a specified object (e.g., a person, a flower pot, a piece of furniture, etc.) as a movable object under the control of the processor (120). The electronic device (101) can perform a segmentation operation only for an object corresponding to a classification of a specified object under the control of the processor (120).

In the 601 screen, when an image selects one object from among at least one object (511, 513, 515), a user interface (520) for highlighting an outline of the selected object may be displayed. By displaying a user interface (520) for highlighting an outline of the selected object, the visibility of the object selected by the user may be improved. The first indicator may include a user interface (520) for highlighting an outline of the selected object.

In the 601 screen, when an image selects one object from among at least one object (511, 513, 515), a user interface (610) for editing object positions can be displayed. The user interface (610) for editing object positions can display positions between objects with indicators (or images) (612, 613) based on positions of objects included in the selected image.

On the 601 screen, the images are arranged in the order of a first object (511), a second object (513), and a third object (515) when viewed from the observation point. A user interface (610) for editing object positions can display the relative positions among the first object (511), the second object (513), and the third object (515) using a second indicator (612) and a third indicator (613). The second indicator (612) can be an image that represents a wall, which is the third object (515). The third indicator (613) can be an image that represents a table, which is the second object (513). The first indicator (611) can be an icon or representation indicating the selected first object (511).

In the 603 screen, the first indicator (611) corresponding to the first object (511) can move its position on the user interface (610) for editing object positions by user input. For example, the first indicator (611) corresponding to the first object (511) can move its position on the user interface (610) for editing object positions by user input (e.g., drag input or slide input). When the position of the first indicator (611) on the user interface (610) for editing object positions is changed, the position of the first object (511) on the image can also be moved accordingly.

In the 605 screen, the electronic device (101) can perform out painting/in painting on the image under the control of the processor (120). The electronic device (101) can perform a rewriting operation on the image under the control of the processor (120). In the 605 screen, the operation of the electronic device (101) can be the same as the operation of the electronic device (101) in the 505 screen.

FIG. 7 is a drawing illustrating an image editing method according to one embodiment of the present disclosure.

Screens 701, 703, and 605 are diagrams showing image editing operations on an image editing application.

In screen 701, the image may include at least one object (511, 513, 515). For example, a first object (511) may include a person, a second object (513) may include a table, and a third object (515) may include a wall.

In the 701 screen, the electronic device (101) can recognize and/or detect and classify an object in an image under the control of the processor (120). The electronic device (101) can identify an object corresponding to a classification of a specified object (e.g., a person, a flower pot, a piece of furniture, etc.) as a movable object under the control of the processor (120). The electronic device (101) can perform a segmentation operation only for an object corresponding to a classification of a specified object under the control of the processor (120).

In the 701 screen, when an image selects one object from among at least one object (511, 513, 515), a user interface (520) may be displayed that emphasizes the outline of the selected object. By displaying the user interface (520) that emphasizes the outline of the selected object, the visibility of the object selected by the user may be improved. The first indicator may include the user interface (520) that emphasizes the outline of the selected object.

In the 701 screen, when an image selects one object from among at least one object (511, 513, 515), a user interface (610) for editing object positions can be displayed. The user interface (610) for editing object positions can display positions between objects with indicators (or images) (612, 613) based on positions of objects included in the selected image.

On the 701 screen, the images are arranged in the order of a first object (511), a second object (513), and a third object (515) when viewed from the observation point. The user interface (610) for editing object positions can display the relative positions among the first object (511), the second object (513), and the third object (515) as a first text (712) and a second text (713). The first text (712) can represent a wall, which is the third object (515). The second text (713) can represent a table, which is the second object (513). The indicator (711) can be an icon or representation representing the selected first object (511).

In the 703 screen, the indicator (711) corresponding to the first object (511) can move its position on the user interface (610) for editing object positions by user input. For example, the indicator (711) corresponding to the first object (511) can move its position on the user interface (710) for editing object positions by user input (e.g., drag input or slide input). When the position of the indicator (711) on the user interface (710) for editing object positions is changed, the position of the first object (511) on the image can also move accordingly.

In screen 705, the electronic device (101) can perform out painting/in painting on the image under the control of the processor (120). The electronic device (101) can perform a rewriting operation on the image under the control of the processor (120). In screen 705, the operation of the electronic device (101) can be the same as the operation of the electronic device (101) in screen 505.

FIG. 8A is a diagram illustrating a user interface (810) for object position editing based on an indicator according to one embodiment of the present disclosure.

The user interface (810) for editing object locations in FIG. 8a may be identical to the user interface (610) for editing object locations in FIG. 6.

A user interface (810) for editing object positions can display relative positions between objects as indicators.

The fourth indicator (811) may be an icon or representation representing the selected object.

The fifth indicator (812) and the sixth indicator (813) can indicate objects included in the image. The distance between the fifth indicator (812) and the sixth indicator (813) can correspond to a relative distance between the objects corresponding to the fifth indicator (812) and the sixth indicator (813). The distance between the fifth indicator (812) and the sixth indicator (813) can correspond to an arrangement between the objects corresponding to the fifth indicator (812) and the sixth indicator (813). The fourth indicator (811) corresponding to the selected object can move its position on the user interface (810) for editing the object position by a user input (e.g., a drag input or a slide input). When changing the position of the fourth indicator (811) on the user interface (810) for editing the object position, the position of the object selected on the image may also move accordingly.

FIG. 8b is a diagram illustrating a user interface (820) for object position editing based on an indicator according to one embodiment of the present disclosure.

The user interface (820) for editing object positions in FIG. 8b may be identical to the user interface (710) for editing object positions in FIG. 7. The user interface (820) for editing object positions may display relative positions between objects as text.

In one embodiment, the seventh indicator (821) may be an indicator or representation representing a selected object. The third text (823) and the fourth indicator (824) may represent objects included in the image. The distance between the third text (823) and the fourth indicator (824) may correspond to a relative distance between the objects corresponding to the third text (823) and the fourth indicator (824). The distance between the third text (823) and the fourth indicator (824) may correspond to an arrangement between the objects corresponding to the third text (823) and the fourth indicator (824). The seventh indicator (821) corresponding to the selected object may move a position on the user interface (810) for editing the object position by a user input (e.g., a drag input or a slide input). When changing the position of the seventh indicator (821) on the user interface (810) for editing the object position, the position of the object selected on the image may also move accordingly.

FIG. 8c is a diagram illustrating a user interface (830) for object position editing based on an indicator according to one embodiment of the present disclosure.

The user interface (830) for editing object positions in FIG. 8c may be in the shape of a sphere. In one embodiment, the eighth indicator (831) may be an icon or representation indicating a selected object. When the eighth indicator (831) is moved in the user interface (840) for editing object positions, the position of the object selected on the image may also be moved accordingly.

Referring to FIGS. 6 and 8A, the user interface (810) for editing object positions in FIG. 8A allows a selected object to be moved along the x-axis and/or y-axis on the image.

Referring to FIGS. 7 and 8b, the user interface (820) for editing object positions in FIG. 8b allows a selected object to be moved along the x-axis and/or y-axis on the image.

Referring to FIG. 5 and FIG. 8c, the user interface (830) for editing object positions in FIG. 8c allows a selected object to be moved along the z-axis as well as the x-axis and/or y-axis on the image.

FIG. 9 is a drawing showing an image editing method according to one embodiment of the present disclosure.

Screen 912, screen 913 and screen 913 are drawings showing image editing operations on the second image of FIG. 4.

The electronic device (101) can select a first object (911) included in a first image from among a plurality of images (910) displayed on a gallery application by user input. The first object (911) included in a first image from among a plurality of images (910) displayed on a gallery application can be selected by user input.

On the 912 screen, the electronic device (101) can recognize and/or detect and classify an object in the second image under the control of the processor (120). The second image may be an image acquired through the camera (280). The second image may be displayed on the display (260) as a preview image. The second image may be an image being displayed through the display (260).

In one embodiment, the electronic device (101) may generate a depth map of the second image under the control of the processor (120).

In one embodiment, the electronic device (101) may, under control of the processor (120), recognize and/or detect and classify an object in the second image.

In screens 913 and 914, when the first object (911) is placed on the second image, the electronic device (101) may include a user interface (920) for editing the object location. The user interface (920) for editing the object location of FIG. 9 may be identical to the user interface (810) for editing the object location of FIG. 8 a.

The fourth indicator (811) may be an icon or representation representing the selected object.

The tenth indicator (921) and the eleventh indicator (922) can indicate objects included in the second image. The distance between the tenth indicator (921) and the eleventh indicator (922) can correspond to the arrangement between the objects corresponding to the tenth indicator (921) and the eleventh indicator (922). The ninth indicator (921) corresponding to the first object (911) can move its position on the user interface (810) for editing object positions by a user input (e.g., a drag input or a slide input). When the position of the ninth indicator (921) is changed on the user interface (920) for editing object positions, the position of the first object (911) on the second image can also be moved accordingly. At this time, the electronic device (101) can change the size of the first object (911) according to the movement based on the depth map of the second image under the control of the processor (120). In addition, the electronic device (101) can rewrite the second image corresponding to the position of the moved first object (911).

FIG. 10a is a diagram showing an image when a selected object (1003) is moved in one embodiment of the present disclosure.

FIG. 10b is a diagram showing an image when a selected object (1003) is moved in one embodiment of the present disclosure.

When the selected object (1003) in FIG. 10a is moved, a temporarily blank object (1001) corresponding to the outline or size of the selected object (1003) may be displayed. However, the present invention is not limited thereto, and as shown in FIGS. 5, 6, and 7, the electronic device (101) may perform an out-painting/in-painting operation in advance to fill the image without a blank object according to the object movement.

When the selected object (1003) in FIG. 10b is moved, the same object (1005) corresponding to the outline or size of the selected object (1003) can be displayed. However, it is not limited thereto, and as in FIG. 5, FIG. 6, and FIG. 7, the electronic device (101) can perform an out-painting/in-painting operation in advance to fill the image without the same object according to the object movement.

In one embodiment, the image editing method of the electronic device (101) of FIGS. 3 and 4 may be stored in a computer-readable recording medium or storage medium. The image editing method of the electronic device (101) of FIGS. 3 and 4, stored as instructions in a computer-readable recording medium or storage medium, when executed by a processor (120), may cause the electronic device (101) to perform the instructions.

In one embodiment, an electronic device (101) includes a display (160); a memory (130) storing instructions; and a processor (120), wherein the instructions, when executed by the processor (120), cause the electronic device (101) to: execute an image editing application including a user interface for selecting an image based on a user input, and for moving a position of an object; determine depth information of the image; recognize and classify an object in the image; change a position of the selected object on the image based on a user input through the user interface for moving the position of the object; and perform an outpainting and/or inpainting operation on one of the image or the selected object based on the position of the selected object.

In one embodiment, the instructions, when executed by the processor (120), may cause the electronic device (101) to provide an image with a changed position of an object when the image editing application is terminated.

In one embodiment, when executed by the processor (120), the electronic device (101) may be caused to generate depth information using a depth information generation model that generates depth information using a single image.

In one embodiment, when executed by the processor (120), the electronic device (101) may be enabled to recognize and classify objects in an image using an object recognition and classification model that segments an image to recognize and classify objects.

In one embodiment, when executed by the processor (120), it may cause the electronic device (101) to perform re-lighting in response to an object whose position has changed.

In one embodiment, when executed by the processor (120), it may cause the electronic device (101) to improve the image quality of an object whose position has been moved based on the position of the selected object.

In one embodiment, the user interface may display indicators corresponding to recognized objects according to the distance between the recognized objects.

In one embodiment, when executed by the processor (120), the electronic device (101) may be caused to perform an out-painting/in-painting operation by determining an area to be covered or filled in in an object or image whose position has changed due to a change in the position of the object.

In one embodiment, when executed by the processor (120), the electronic device (101) may be configured to select a first object from a first image, verify depth information of the first image, recognize and classify an object from a second image, perform out-painting and/or in-painting of the image based on at least one movable object in the second image, and, if there is a user input for movement of a position of the first object on the second image, change the position of the first object on the second image, and perform re-lighting corresponding to the first object whose position has been changed.

In one embodiment, when executed by the processor (120), it may cause the electronic device (101) to provide the second image in which the position of the first object is changed.

In one embodiment, an image editing method may include: selecting an image based on a user input; executing an image editing application including a user interface relating to movement of a position of an object; checking depth information of the image; recognizing and classifying an object in the image; changing a position of the selected object on the image based on the user input through the user interface relating to movement of the position of the object; and performing an outpainting and/or inpainting operation on one of the image or the selected object based on the position of the selected object.

In one embodiment, the image editing method may include an action of providing an image with a changed position of an object when the image editing application is terminated.

In one embodiment, an image editing method may include generating depth information using a depth information generation model that generates depth information using a single image.

In one embodiment, the image editing method may include recognizing and classifying objects in the image using an object recognition and classification model that segments the image to recognize and classify objects.

In one embodiment, the image editing method may include performing re-lighting in response to a changed position of an object.

In one embodiment, the image editing method may include an operation of improving the image quality of a moved object based on a location of the selected object.

In one embodiment, the image editing method may include an action of displaying indicators corresponding to recognized objects according to a distance between the recognized objects.

In one embodiment, the image editing method may include an operation of performing an outpainting/inpainting operation by determining an area to be covered or filled in of an object or image whose position has changed due to a change in the position of the object.

In one embodiment, an image editing method may include: selecting a first object from a first image; checking depth information of the first image; recognizing and classifying an object from the second image; performing outpainting and/or inpainting of the image based on at least one movable object in the second image; changing a position of the first object on the second image if there is a user input for movement of a position of the first object on the second image; and performing re-lighting corresponding to the first object whose position has been changed.

In one embodiment, the image editing method may include providing a second image in which the position of the first object is changed.

The electronic devices according to various embodiments disclosed in this document may be devices of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliance devices. The electronic devices according to embodiments of this document are not limited to the above-described devices.

It should be understood that the various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly dictates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first) is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the called at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store ^TM ) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities, and some of the multiple entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, the multiple components (e.g., a module or a program) may be integrated into one component. In such a case, the integrated component may perform one or more functions of each of the multiple components identically or similarly to those performed by the corresponding component of the multiple components before the integration. According to various embodiments, the operations performed by the module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims (15)

In electronic devices, display; Memory for storing instructions; and Includes a processor, The above instructions, when executed by the processor, cause the electronic device to: Images are selected based on user input, Run an image editing application that includes a user interface for moving the position of objects, Check the depth information of the image above, Recognize and classify objects in the above image, Changing the position of the selected object on the image based on user input via the user interface regarding movement of the position of the object; An electronic device that performs an out-painting and/or in-painting operation on the image or one of the selected objects based on the location of the selected object. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: An electronic device that provides an image with the object's position changed when the image editing application is terminated. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: An electronic device that generates depth information using a depth information generation model that generates depth information using a single image. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: By using an object recognition and classification model that recognizes and classifies objects by segmenting images, objects are recognized and classified in the image. An electronic device that performs re-lighting in response to an object's changed position. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: Based on the location of the selected object, the image quality of the moved object is improved. An electronic device that displays indicators corresponding to recognized objects according to the distance between the recognized objects. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: An electronic device that determines the area to be covered or filled in of an object or image whose position has changed according to a change in the position of the object, and performs an outpainting/inpainting operation. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: Let's select the first object in the first image, Let's check the depth information of the first image, Recognize and classify objects in the second image above, Perform outpainting and/or inpainting of the image based on at least one movable object in the second image, If there is a user input for a positional movement of the first object on the second image, change the position of the first object on the second image, Let re-lighting be performed in response to the first object whose position has changed, An electronic device providing a second image in which the position of the first object has been changed. In terms of image editing methods, Action where an image is selected based on user input; An action that runs an image editing application that includes a user interface for moving the position of an object; An action to check the depth information of the above image; An action to recognize and classify objects in the above image; An action of changing the position of the selected object on the image based on user input via a user interface regarding movement of the position of the object; and A method comprising: performing an out-painting and/or in-painting operation on the image or one of the selected objects, based on a location of the selected object. In Article 8, A method comprising the action of providing an image with the changed position of an object when said image editing application is terminated. In Article 8 A method comprising the operation of generating depth information using a depth information generation model that generates depth information using a single image. In Article 8, A method comprising an operation of recognizing and classifying an object in an image using an object recognition and classification model that recognizes and classifies objects by segmenting the image. In Article 8, A method comprising an action to perform re-lighting in response to an object whose position has changed. In Article 8, An operation for improving the image quality of an object whose position has been moved based on the position of the selected object; and A method comprising an action of displaying indicators corresponding to recognized objects according to a distance between the recognized objects. In Article 8, A method including an operation of performing an outpainting/inpainting operation by determining an area to be covered or filled in an object or image whose position has changed according to a change in the position of the object. In Article 8 The action of selecting the first object in the first image; An action to check the depth information of the first image; An action to recognize and classify an object in the second image; An operation of performing out-painting and/or in-painting of the image based on at least one movable object in the second image; An action of changing the position of the first object on the second image when there is a user input for a movement of the position of the first object on the second image; An action to perform re-lighting in response to a first object whose position has changed; and A method comprising providing a second image in which the position of the first object is changed.

Images