[go: up one dir, main page]

WO2022145675A1 - Dispositif électronique et son procédé de commande - Google Patents

Dispositif électronique et son procédé de commande Download PDF

Info

Publication number
WO2022145675A1
WO2022145675A1 PCT/KR2021/015572 KR2021015572W WO2022145675A1 WO 2022145675 A1 WO2022145675 A1 WO 2022145675A1 KR 2021015572 W KR2021015572 W KR 2021015572W WO 2022145675 A1 WO2022145675 A1 WO 2022145675A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
information
flow
flow information
electronic device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2021/015572
Other languages
English (en)
Korean (ko)
Inventor
곽재희
박철성
최나래
최웅일
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of WO2022145675A1 publication Critical patent/WO2022145675A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/18—Image warping, e.g. rearranging pixels individually
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present disclosure relates to an electronic device and a control method of the electronic device, and more particularly, to an electronic device capable of obtaining a target image having a view different from that of the original image by performing warping on an original image and a control method of the electronic device will be.
  • the present disclosure is in accordance with the necessity as described above, and an object of the present disclosure is to perform bidirectional warping on an original image based on an optical flow between a plurality of inputted original images, thereby generating a new virtual view of a target image.
  • An object of the present invention is to provide an electronic device capable of improving image quality and a method for controlling the electronic device.
  • an electronic device includes a memory storing at least one instruction and a processor executing the at least one instruction, wherein the processor comprises a first Acquire a first image and a second image, obtain first flow information about an optical flow between the first image and the second image, and based on the first flow information,
  • the second flow information is obtained by performing forward warping to obtain a view and a third image corresponding to a view different from the view of the second image, and based on the first flow information
  • the Third flow information is obtained by performing backward warping to obtain a third image, and an optical corresponding to a position of a hole included in the second flow information based on the third flow information
  • the processor identifies the location of the hole using a flow mask, and when information on an optical flow corresponding to the location of the hole is obtained based on the third flow information, the The fourth flow information may be acquired by inserting optical flow information into the hole.
  • the first flow information includes a flow vector for an optical flow in a direction from the first image to the second image and a flow vector for an optical flow in a direction from the second image to the first image
  • the processor combines flow vectors obtained according to each of forward warping in a direction from the first image to the third image and forward warping in a direction from the second image toward the third image to obtain the second flow information obtains, and combines the flow vectors obtained according to each of the reverse warping in the direction from the third image to the first image and the reverse warping in the direction from the third image to the second image to combine the third flow information can be obtained.
  • the processor obtains a score related to image quality for each of a plurality of pixels included in the third image, and when a pixel having a score less than a preset threshold is identified, reverse warping is further performed on the identified pixel, , a pixel value of the identified pixel may be corrected based on a result of the additionally performed backward warping.
  • the processor may provide the third image when it is determined that there is no pixel having the score less than the threshold value.
  • the score related to the picture quality is information about the peak signal-to-noise ratio (PSNR) for each of the plurality of pixels or a neural network model trained to output a score related to the picture quality of the input image. It may include information on a loss value obtained by using the .
  • PSNR peak signal-to-noise ratio
  • the processor obtains first characteristic information and second characteristic information for each of the first image and the second image, and each of the first characteristic information and the second characteristic information is determined based on the fourth flow information.
  • a method of controlling an electronic device includes: obtaining a first image and a second image; an optical flow between the first image and the second image (obtaining first flow information for optical flow, obtaining a third image corresponding to a view different from a view of the first image and a view of the second image based on the first flow information obtaining second flow information by performing forward warping for obtaining information on an optical flow corresponding to a position of a hole included in the second flow information based on the third flow information, and obtaining fourth flow information for obtaining the third image acquiring and acquiring the third image based on the fourth flow information, the pixel values of the first image, and the pixel values of the second image.
  • the step of obtaining the fourth flow information includes identifying the location of the hole using a flow mask and obtaining information on an optical flow corresponding to the location of the hole based on the third flow information.
  • the method may include inserting information on an optical flow corresponding to the location of the hole into the location of the hole to obtain the fourth flow information.
  • the first flow information includes a flow vector for an optical flow in a direction from the first image to the second image and a flow vector for an optical flow in a direction from the second image to the first image
  • the step of obtaining the second flow information includes a flow vector obtained according to forward warping in a direction from the first image to the third image and forward warping in a direction from the second image to the third image.
  • Combining to obtain the second flow information, and obtaining the third flow information includes: reverse warping in a direction from the third image toward the first image and a direction from the third image toward the second image.
  • the third flow information may be obtained by combining the flow vectors obtained according to each of the backward warping.
  • control method of the electronic device may include: obtaining a score related to image quality for each of a plurality of pixels included in the third image, and when a pixel having a score less than a preset threshold is identified, reverse warping for the identified pixel
  • the method may further include performing additionally , and correcting a pixel value for the identified pixel based on a result of the additionally performed backward warping.
  • control method of the electronic device may provide the third image when it is determined that there is no pixel having the score less than the threshold value.
  • the score related to the picture quality is information about the peak signal-to-noise ratio (PSNR) for each of the plurality of pixels or a neural network model trained to output a score related to the picture quality of the input image. It may include information on a loss value obtained by using the .
  • PSNR peak signal-to-noise ratio
  • the method of controlling the electronic device includes: obtaining first characteristic information and second characteristic information for each of the first image and the second image; and the first characteristic information and the second characteristic information based on the fourth flow information 2 obtaining third and fourth feature information each of which is warped, and the third feature information, the fourth feature information, the fourth flow information, the pixel value of the first image, and the second
  • the method may include acquiring the third image based on pixel values of the image.
  • the method for controlling the electronic device includes a first image and acquiring a second image, acquiring first flow information about an optical flow between the first image and the second image, based on the first flow information, Obtaining second flow information by performing forward warping to obtain a view and a third image corresponding to a view different from the view of the second image, based on the first flow information obtaining third flow information by performing backward warping to obtain the third image; corresponding to the position of a hole included in the second flow information based on the third flow information obtaining information on an optical flow to be used, obtaining fourth flow information for obtaining the third image, and based on the fourth flow information, a pixel value of the first image, and a pixel value of the second image and acquiring the third image.
  • FIG. 1 is a diagram schematically illustrating a configuration of an electronic device according to an embodiment of the present disclosure
  • FIG. 2 is a diagram schematically illustrating a plurality of modules according to an embodiment of the present disclosure
  • FIG. 3 is a view for explaining in more detail a bidirectional warping process according to an embodiment of the present disclosure
  • FIG. 4 is a view showing in detail a plurality of modules according to an embodiment of the present disclosure.
  • FIG. 5 is a view showing in detail the configuration of an electronic device according to an embodiment of the present disclosure.
  • FIG. 6 is a view illustrating a control method of an electronic device according to an embodiment of the present disclosure.
  • FIG. 7 is a diagram illustrating a control method of an electronic device according to another embodiment of the present disclosure.
  • expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.
  • expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together.
  • “A or B,” “at least one of A and B,” or “at least one of A or B” means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.
  • a component eg, a first component
  • another component eg, a second component
  • the certain element may be directly connected to the other element or may be connected through another element (eg, a third element).
  • a component eg, a first component
  • another component eg, a second component
  • the expression “a device configured to” may mean that the device is “capable of” with other devices or parts.
  • a processor configured (or configured to perform) A, B, and C refers to a dedicated processor (eg, an embedded processor) for performing the corresponding operations, or by executing one or more software programs stored in a memory device.
  • a generic-purpose processor eg, a CPU or an application processor
  • a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software.
  • a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented with at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.
  • FIG. 1 is a diagram schematically illustrating a configuration of an electronic device according to an embodiment of the present disclosure
  • FIG. 2 is a diagram schematically illustrating a plurality of modules according to an embodiment of the present disclosure
  • FIG. 3 is a diagram of the present disclosure It is a diagram for explaining in more detail a bidirectional warping process according to an embodiment.
  • the 'electronic device 100' refers to a device capable of acquiring a target image having a different view from the original image by performing warping on the original image.
  • the electronic device 100 may be implemented as various types of devices, such as a server, a cloud server, and an edge computing device, as well as a user terminal device such as a smart phone or a tablet PC. That is, there is no particular limitation on the type of the electronic device 100 according to the present disclosure.
  • 'warping' refers to a technique of obtaining a target image by geometrically transforming an original image and estimating the positions of each pixel included in the target image.
  • the warping process according to the present disclosure may include a forward warping process and a backward warping process.
  • Forward warping' refers to a warping technique in a direction from an original image to a target image. Specifically, the forward warping process transforms the coordinate values indicating the positions of each pixel of the original image based on a matrix predefined according to the view of the target image, thereby transforming the position of each pixel included in the target image. It may include the process of estimating
  • the reverse warping refers to a warping technique in a direction from a target image to an original image.
  • the reverse warping may include a process of estimating positions of pixels of an original image corresponding to respective pixels of a target image based on a predefined reverse matrix.
  • the backward warping process assumes an arbitrary position as a position corresponding to the target image, estimates pixels of the original image corresponding to each pixel of the target image, and iteratively according to the result of the estimation. It may include a process of correcting an inverse matrix for backward warping.
  • the electronic device 100 performs both forward warping in the direction of the target image from the original image and backward warping in the direction of the original image in the target image to obtain one target image.
  • the electronic device 100 performs both forward warping in the direction of the target image from the original image and backward warping in the direction of the original image in the target image to obtain one target image.
  • the electronic device 100 may include a memory 110 and a processor 120 . And, as shown in FIG. 2 , the electronic device 100 includes a flow estimation module 10 (flow estimation module), a bidirectional warping module 20 (bidirectional warping module), a blending module 30 (blending module), and It may include a view synthesis module 40 (view synthesis module).
  • the plurality of modules according to the present disclosure may be implemented as a software module or a hardware module, and when the plurality of modules are implemented as a software module, the processor 120 accesses the software module by loading the software module stored in the memory 110 . can do.
  • At least one instruction related to the electronic device 100 may be stored in the memory 110 .
  • an operating system (O/S) for driving the electronic device 100 may be stored in the memory 110 .
  • various software programs or applications for operating the electronic device 100 according to various embodiments of the present disclosure may be stored in the memory 110 .
  • the memory 110 may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.
  • various software modules for operating the electronic device 100 may be stored in the memory 110 according to various embodiments of the present disclosure, and the processor 120 executes various software modules stored in the memory 110 .
  • the operation of the electronic device 100 may be controlled. That is, the memory 110 is accessed by the processor 120 , and reading/writing/modification/deletion/update of data by the processor 120 may be performed.
  • the term memory 110 refers to the memory 110 , a ROM (not shown) in the processor 120 , a RAM (not shown), or a memory card (not shown) mounted in the electronic device 100 (eg, For example, it may be used in the meaning of including a micro SD card, a memory stick).
  • data corresponding to the first image, the second image, and the third image according to the present disclosure may be stored in the memory 110 .
  • the first flow information, the second flow information, the third flow information, and the fourth flow information according to the present disclosure may be stored in the memory 110 .
  • various information necessary within the scope for achieving the object of the present disclosure may be stored in the memory 110, and the information stored in the memory 110 may be updated as received from an external device or input by a user. .
  • the processor 120 controls the overall operation of the electronic device 100 . Specifically, the processor 120 is connected to the configuration of the electronic device 100 including the memory 110 , and executes at least one instruction stored in the memory 110 as described above. You have total control over your movements.
  • the processor 120 may be implemented in various ways.
  • the processor 120 may include an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (Digital Signal).
  • ASIC application specific integrated circuit
  • DSP digital signal processor
  • the term processor 120 may be used to include a central processing unit (CPU), a graphic processing unit (GPU), a main processing unit (MPU), and the like.
  • the processor 120 performs forward and backward warping on the first image and the second image to obtain a third image having a different view from the first image and the second image. can be obtained
  • the processor 120 may acquire a first image and a second image.
  • the terms 'first image' and 'second image' are terms for specifying a plurality of input images that are different from each other, and specifically, the first image and the second image refer to a plurality of images having different views. .
  • the first image and the second image which are two images, are the original images.
  • first image and the second image may not only be different image frames included in a video sequence, but may also be image frames acquired through different cameras.
  • first image and the second image may be directly acquired through the camera included in the electronic device 100 , or may be transmitted to the electronic device 100 after being acquired through an external device.
  • the 'third image' is an image corresponding to a view different from the view of the first image and the view of the second image, and refers to a target image generated based on the first image and the second image, which are original images.
  • the view of the third image that is the target image may be referred to as a novel view.
  • the view of the target image may be changed according to a user's setting.
  • the processor 120 may obtain first flow information about an optical flow between the first image and the second image. Specifically, the processor 120 may acquire first flow information through a 'flow estimation module (10)'.
  • the flow measurement module 10 refers to a module capable of acquiring flow information for a plurality of input images.
  • 'flow information' is used as a general term for information such as a flow vector or a flow map indicating an optical flow of an image.
  • 'first flow information' is a term for specifying flow information for a first image and a second image that are input original images.
  • the first flow information may include information on an optical flow in a direction from the first image toward the second image and information on an optical flow in a direction from the second image toward the first image.
  • the first flow information may include only one of information about an optical flow in a direction from the first image toward the second image and information about an optical flow in a direction from the second image toward the first image. It goes without saying that the present disclosure may be implemented even in a case.
  • the flow measurement module 10 may include a neural network such as a convolutional neural network (CNN), and for example, it is possible to output flow information for a plurality of input images such as PWC-Net, FlowNet, etc. Any module may be included in the flow measurement module 10 according to the present disclosure.
  • CNN convolutional neural network
  • the processor 120 may obtain the second flow information by performing forward warping to obtain a third image based on the first flow information.
  • the third flow information may be obtained by performing reverse warping to obtain the third image based on the first flow information.
  • the processor may acquire the second flow information and the third flow information through a 'bidirectional warping module 20' module.
  • the 'bidirectional warping module 20' refers to a module capable of acquiring flow information on a target image by performing forward warping and backward warping based on input flow information.
  • the bidirectional warping module 20 may include a forward warping module 21 and a backward warping module 22 .
  • the forward warping module 21 may perform forward warping based on the first flow information to obtain second flow information according to the result
  • the reverse warping module 22 may obtain 'second flow information' Backward warping may be performed based on the 'third flow information' according to the result.
  • the forward warping process converts the coordinate values included in the first flow information (referred to as source view in FIG. 3) through a predefined matrix T. , obtaining second flow information (referred to as forward warping in FIG. 3 ) including information on the transformed coordinate values.
  • the reverse warping process is first flow information corresponding to flow information (referred to as backward warping in FIG. 3 ) corresponding to an arbitrary position through an inverse matrix T ⁇ 1 . It may include a process of estimating (referred to as a source view in FIG. 3 ).
  • the processor 120 may perform backward warping to obtain a third image based on the first flow information while performing forward warping to obtain the third image based on the first flow information.
  • the forward warping process and the backward warping process according to the present disclosure do not have to be sequentially performed, and at least a portion of the time during which the forward warping process is performed may overlap with at least a portion of the time during which the backward warping process is performed.
  • the first flow information may include information on an optical flow in a direction from the first image toward the second image and information on an optical flow in a direction from the second image toward the first image.
  • the flow measurement module 10 may output two different flow vectors and input them to the bidirectional warping module 20 .
  • the processor 120 combines the flow vectors obtained according to each of the forward warping in the direction from the first image to the third image and the forward warping in the direction from the second image to the third image to obtain the second flow information. can be obtained.
  • the processor 120 obtains third flow information by combining the flow vectors obtained according to each of the reverse warping in the direction from the third image to the first image and the reverse warping in the direction from the third image to the second image. can do.
  • the process of combining different flow vectors may be performed through the blending module 30 as will be described later.
  • the processor 120 obtains information on the optical flow corresponding to the position of the hole included in the second flow information based on the third flow information, Fourth flow information for acquiring the third image may be acquired.
  • the hole refers to an area corresponding to the position of the pixel that is not calculated according to the forward warping
  • the 'fourth flow information' refers to a result of forward warping and the result of backward warping to obtain a third image. It refers to the generated flow information.
  • the processor 120 may obtain the fourth flow information through the blending module 30 (blending module).
  • the 'blending module 30' refers to a module capable of acquiring one flow information by combining different information. Specifically, when the location of the hole is identified using a flow mask, the blending module 30 obtains information on the optical flow corresponding to the location of the hole based on the third flow information, and The fourth flow information may be obtained by inserting the optical flow information into the hole position.
  • the processor 120 identifies the location of the hole (shaded in FIG. 3 ), and uses the third flow information through the blending module 30 .
  • the fourth flow information corresponding to the novel view (referred to as a novel view in FIG. 3 ) as shown on the right side of FIG. 3 may be obtained.
  • the blending module 30 may perform a splatting process for calculating a pixel value at an integer position from the result of forward warping.
  • the splatting process may include performing a sum operation on overlapping pixel values or calculating an average value, and a softmax for converting pixel values into weights through a probabilistic operation. It may include a calculation process.
  • the blending module 30 additionally performs various interpolation processes such as nearest neighbor interpolation, bilinear interpolation, and high-order interpolation for interpolating the result of backward warping. can also be done It has been described above that the process of combining different flow vectors can be performed through the blending module 30 .
  • the processor 120 may obtain a third image based on the fourth flow information, the pixel value of the first image, and the pixel value of the second image. Specifically, the processor 120 may acquire the third image through the 'view synthesis module 40 (view synthesis module)'.
  • the view synthesis module 40 refers to a module capable of acquiring a target image based on flow information for acquiring the target image and pixel values of the input image. Specifically, the view synthesizing module 40 provides a view different from the first image and the second image by assigning at least some of the pixel values of the first image and the pixel values of the second image to positions for each pixel according to the fourth flow information.
  • the processor 120 may provide the third image obtained through the view synthesis module 40 . Specifically, the processor 120 may control the display included in the electronic device 100 to display the obtained third image, and transmit data on the obtained third image to an external device such as a user terminal.
  • the communication unit included in the device 100 may be controlled.
  • the electronic device 100 generates a new virtual view by performing bidirectional warping on the original image based on the optical flow between the plurality of input original images. It is possible to improve the quality of the target image.
  • FIG. 4 is a diagram illustrating in detail a plurality of modules according to an embodiment of the present disclosure.
  • a plurality of modules include a flow measurement module 10 , a forward warping module 21 , a backward warping module 22 , a blending module 30 and a view synthesis as shown in FIG. 2 .
  • the module 40 may further include a feature extraction module 15 (a feature extraction module), a feature information warping module 25 (feature warping module), and an image quality estimation module 45 (image quality estimation module).
  • a feature extraction module 15 a feature extraction module
  • feature information warping module 25 feature warping module
  • image quality estimation module 45 image quality estimation module
  • the processor 120 obtains first characteristic information and second characteristic information for each of the first image and the second image, and warps the first characteristic information and the second characteristic information to obtain a second 3 Available for image acquisition.
  • the processor may acquire first feature information about the first image and second feature information about the second image through a 'feature extraction module 15'.
  • the term 'feature information' is used as a generic term for all information capable of quantifying features of an input image, such as a feature vector or a feature map.
  • the terms 'first characteristic information' and 'second characteristic information' are used as terms for specifying the characteristic information of the first image and the characteristic information of the second image, respectively.
  • the feature extraction module 15 refers to a module capable of extracting features of an input image, digitizing the features, and outputting feature information.
  • the feature extraction module 15 may include a convolutional layer capable of obtaining a feature map by performing a convolution operation for each channel of data corresponding to the input image.
  • the feature information output by the feature extraction module 15 may be expressed as a matrix/vector corresponding to a matrix/vector of input data.
  • the processor 120 obtains the third characteristic information and the fourth characteristic information in which the first characteristic information and the second characteristic information are respectively warped based on the fourth flow information.
  • the terms 'third characteristic information' and 'fourth characteristic information' are used as terms to refer to characteristic information in which the first characteristic information and the second characteristic information are warped, respectively.
  • the processor 120 may warp the first feature information and the second feature information through a 'feature warping module 25'.
  • the feature information warping module 25 refers to a module that warps input feature information to correspond to flow information. Specifically, when the fourth flow information is received from the blending module and the first characteristic information and the second characteristic information are received from the feature extraction module 15, the characteristic information warping module 25 is configured to correspond to the fourth flow information.
  • the third characteristic information and the fourth characteristic information may be obtained by respectively warping the input first characteristic information and the second characteristic information.
  • the feature information warping module 25 is a module that warps the features extracted from the original image to convert it into a shape of a location corresponding to the determined flow information. have.
  • the processor 120 performs the third characteristic information, the fourth characteristic information, the fourth flow information, the pixel value of the first image and the second through the view synthesis module 40 .
  • a third image may be acquired based on pixel values of the image.
  • the third characteristic information and the fourth characteristic information are used in the fourth flow in acquiring the third image, such as how to process overlapping portions between the pixel values of the first image and the pixel values of the second image.
  • Information, the pixel values of the first image, and the pixel values of the second image alone may be used as additional information for a process that is difficult to process.
  • the processor 120 evaluates the quality of the third image, which is the target image, and whether to perform additional reverse warping according to the evaluation result, or provides the obtained third image to the user You can also decide whether to provide it or not.
  • the process of evaluating the quality of the third image may be performed through an 'image quality estimation module 45'.
  • the image quality evaluation module 45 refers to a module for obtaining a score related to the image quality of the target image. Specifically, the image quality evaluation module 45 may obtain a score related to image quality for each of a plurality of pixels included in the third image that is the target image.
  • the score related to image quality may include information on a peak signal-to-noise ratio (PSNR) for each of a plurality of pixels.
  • the score related to the picture quality may include information on a loss value obtained by using a neural network model trained to output a score related to the picture quality of the input image.
  • PSNR peak signal-to-noise ratio
  • the processor 120 further performs backward warping on the identified pixel, and based on the result of the additionally performed backward warping, the identified pixel You can correct pixel values for pixels. Correction of pixel values using additional backward warping may be performed until there is no pixel having a score related to image quality less than a preset threshold value. That is, if it is determined that there is no pixel having a score related to image quality less than the threshold value, the processor 120 may provide the third image.
  • FIG. 5 is a diagram illustrating in detail the configuration of an electronic device according to an embodiment of the present disclosure.
  • the electronic device 100 may further include a communication unit 130 , an input unit 140 , and an output unit 150 as well as a memory 110 and a processor 120 .
  • a communication unit 130 may further include a communication unit 130 , an input unit 140 , and an output unit 150 as well as a memory 110 and a processor 120 .
  • the configurations shown in FIG. 5 are merely exemplary, and in implementing the present disclosure, a new configuration may be added or some configurations may be omitted in addition to the configuration shown in FIG. 5 .
  • the communication unit 130 includes a circuit and may communicate with an external device. Specifically, the processor 120 may receive various data or information from an external device connected through the communication unit 130 , and may transmit various data or information to the external device.
  • the communication unit 130 may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, and an NFC module.
  • each of the WiFi module and the Bluetooth module may perform communication using a WiFi method and a Bluetooth method.
  • various types of connection information such as an SSID may be first transmitted and received, and various types of information may be transmitted and received after communication connection using this.
  • the wireless communication module may perform communication according to various communication standards such as IEEE, Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), 5th Generation (5G), and the like.
  • the NFC module may perform communication using a Near Field Communication (NFC) method using a 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860 to 960 MHz, and 2.45 GHz.
  • NFC Near Field Communication
  • the processor 120 may receive an original image, such as a first image and a second image, from an external device through the communication unit 130 .
  • the processor 120 may control the communication unit 130 to transmit an output image such as a third image to an external device.
  • the processor 120 may receive various data/information including data related to a plurality of modules according to the present disclosure from an external device.
  • the input unit 140 includes a circuit, and the processor 120 may receive a user command for controlling the operation of the electronic device 100 through the input unit 140 .
  • the input unit 140 may include a camera 141, a microphone (not shown), and a remote control signal receiver (not shown).
  • the input unit 140 may be implemented as a touch screen and included in the display 151 .
  • the camera 141 may acquire an image of at least one object.
  • the camera 141 includes an image sensor, and the image sensor may convert light entering through a lens into an electrical image signal.
  • the processor 120 may acquire the first image and the second image through the camera 141 . Specifically, when a user input for obtaining a live view image is received, the processor 120 may obtain a live view image through the camera 141 . In addition, when a user input for capturing an image at a specific point in time or a user input for recording a video of a specific section is received, the processor 120 may acquire a photo or a video according to the user input. And, any of a photo and a moving picture may be the first image and the second image according to the present disclosure.
  • the output unit 150 includes a circuit, and the processor 120 may output various functions that the electronic device 100 can perform through the output unit 150 .
  • the output unit 150 may include at least one of a display 151 , a speaker 152 , and an indicator.
  • the display 151 may output image data under the control of the processor 120 . Specifically, the display 151 may output an image pre-stored in the memory 110 under the control of the processor 120 . Also, the display 151 may display a user interface (UI) stored in the memory 110 .
  • the display 151 may be implemented as a liquid crystal display panel (LCD), organic light emitting diodes (OLED), etc., and the display 151 may be implemented as a flexible display 151, a transparent display 151, etc. in some cases. It is also possible to be However, the display 151 according to the present disclosure is not limited to a specific type.
  • the speaker 152 may output audio data under the control of the processor 120 , and the indicator may be lit under the control of the processor 120 .
  • the processor 120 may control the display 151 to display at least one of the first image and the second image that are the original image, and the third image that is the target image.
  • the processor 120 may control the display 151 to display a user interface.
  • the processor 120 provides a user interface including a first image and a second image, and a user input for obtaining a third image having a different view than the first image and the second image through the user interface. can receive
  • the processor 120 displays at least one of the first image, the second image, and the third image. While being displayed on 151 , a voice signal corresponding to at least one of the first image, the second image, and the third image may be output through the speaker 152 .
  • FIG. 6 is a diagram illustrating a control method of an electronic device according to an embodiment of the present disclosure
  • FIG. 7 is a diagram illustrating a control method of an electronic device according to another embodiment of the present disclosure.
  • the electronic device 100 may acquire a first image and a second image that are original images ( S610 ).
  • the electronic device 100 may obtain first flow information about an optical flow between the first image and the second image ( S620 ).
  • the first flow information may include information on an optical flow in a direction from the first image toward the second image and information on an optical flow in a direction from the second image toward the first image.
  • the second flow information and the third flow information may be obtained by performing forward warping and backward warping to obtain a third image based on the first flow information (S630).
  • the electronic device 100 may perform backward warping for obtaining a third image based on the first flow information while performing forward warping for obtaining the third image based on the first flow information.
  • the electronic device 100 combines the flow vectors obtained according to each of the forward warping in the direction from the first image to the third image and the forward warping in the direction from the second image to the third image. 2 It is possible to obtain flow information. Then, the electronic device 100 combines the flow vectors obtained according to each of the reverse warping in the direction from the third image to the first image and the reverse warping in the direction from the third image to the second image to obtain the third flow information. can be obtained
  • the electronic device 100 acquires information on the optical flow corresponding to the location of the hole included in the second flow information based on the third flow information, and obtains the third image It is possible to obtain the fourth flow information for obtaining (S640). Specifically, the electronic device 100 may identify the location of the hole included in the second flow information by using a flow mask, and thereafter, based on the third flow information, the electronic device 100 may perform an optical flow corresponding to the location of the hole based on the third flow information. information on the optical flow corresponding to the location of the hole is obtained, and information on the optical flow corresponding to the location of the hole is inserted into the location of the hole to obtain fourth flow information.
  • the electronic device 100 may acquire a third image based on the fourth flow information, the pixel value of the first image, and the pixel value of the second image ( S650 ). Specifically, the electronic device 100 assigns at least some of the pixel values of the first image and the pixel values of the second image to positions for each pixel according to the fourth flow information, thereby having a view different from that of the first image and the second image. A third image may be generated.
  • the electronic device 100 may obtain a score related to image quality for each of a plurality of pixels included in the third image ( S710 ).
  • the score related to image quality may be information on a peak signal-to-noise ratio (PSNR) for each of a plurality of pixels.
  • the score related to the picture quality may include information on a loss value obtained by using a neural network model trained to output a score related to the picture quality of the input image.
  • the electronic device 100 may identify whether a pixel having a score less than a threshold value among a plurality of pixels included in the third image exists ( S720 ).
  • the electronic device 100 may additionally perform backward warping on the pixel having a score less than the threshold value (S730). ). Then, the electronic device 100 may correct the pixel value of the identified pixel based on the result of the additionally performed backward warping ( S740 ). When the pixel value of the identified pixel is corrected, the electronic device 100 again acquires a score related to image quality for each of a plurality of pixels included in the third image (S710), and a score among the plurality of pixels included in the third image It may be identified whether there is a pixel having a value less than a threshold value ( S720 ).
  • the electronic device 100 may provide the obtained third image (S750).
  • the third image may include a third image corrected based on the result of the additionally performed backward warping as described above.
  • control method of the electronic device 100 may be implemented as a program and provided to the electronic device 100 .
  • a program including a control method of the electronic device 100 may be stored and provided in a non-transitory computer readable medium.
  • the control method of the electronic device 100 includes an optical flow between the first image and the second image. flow), forward warping to obtain a third image corresponding to a view different from the view of the first image and the view of the second image based on the first flow information ( performing forward warping to obtain second flow information, performing backward warping to obtain a third image based on the first flow information to obtain third flow information, a third flow acquiring information on an optical flow corresponding to a position of a hole included in the second flow information based on the information, and acquiring fourth flow information for acquiring a third image; and fourth flow information; and obtaining a third image based on the pixel values of the first image and the pixel values of the second image.
  • the non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, a cache, a memory, and the like, and can be read by a device.
  • the various applications or programs described above may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.
  • control method of the electronic device 100 and the computer-readable recording medium including a program for executing the control method of the electronic device 100 have been briefly described, but this is only for omitting redundant description, and It goes without saying that various embodiments of the device 100 may be applied to a non-transitory computer-readable recording medium including a control method of the electronic device 100 and a program executing the control method of the electronic device 100 . .
  • the electronic device 100 performs bidirectional warping on an original image based on an optical flow between a plurality of inputted original images, thereby enabling a target having a new virtual view.
  • the image quality can be improved.
  • the functions related to the neural network model described above may be performed through the memory 110 and the processor 120 .
  • the processor 120 may include one or a plurality of processors.
  • one or a plurality of processors are general-purpose processors such as CPUs and APs, GPUs. It may be a graphics-only processor, such as a VPU, or an artificial intelligence-only processor, such as an NPU.
  • One or more processors control to process input data according to a predefined operation rule or artificial intelligence model stored in the non-volatile memory and the volatile memory.
  • the predefined action rule or artificial intelligence model is characterized in that it is created through learning.
  • a predefined operation rule or artificial intelligence model of a desired characteristic is created by applying a learning algorithm to a plurality of learning data.
  • Such learning may be performed in the device itself on which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server/system.
  • the artificial intelligence model may be composed of a plurality of neural network layers. Each layer has a plurality of weight values, and the layer operation is performed through the operation of the previous layer and the operation of the plurality of weights.
  • Examples of neural networks include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), GAN.
  • CNN Convolutional Neural Network
  • DNN Deep Neural Network
  • RNN Restricted Boltzmann Machine
  • DBN Deep Belief Network
  • BBN Bidirectional Recurrent Deep Neural Network
  • GAN GAN
  • the learning algorithm is a method of training a predetermined target device (eg, a robot) using a plurality of learning data so that the predetermined target device can make a decision or make a prediction by itself.
  • Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in the present disclosure is specified when It is not limited to the above-mentioned example except for.
  • the device-readable storage medium may be provided in the form of a non-transitory storage medium.
  • 'non-transitory storage medium' is a tangible device and only means that it does not contain a signal (eg, electromagnetic wave). It does not distinguish the case where it is stored as
  • the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.
  • the method according to various embodiments disclosed in this document may be provided as included in a computer program product.
  • Computer program products may be traded between sellers and buyers as commodities.
  • the computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play StoreTM) or on two user devices (eg, It can be distributed (eg downloaded or uploaded) directly or online between smartphones (eg: smartphones).
  • a portion of the computer program product eg, a downloadable app
  • a machine-readable storage medium such as a memory of a manufacturer's server, a server of an application store, or a relay server. It may be temporarily stored or temporarily created.
  • each of the components may be composed of a singular or a plurality of entities, and some of the above-described corresponding sub-components are omitted. Alternatively, other sub-components may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration.
  • operations performed by a module, program, or other component may be sequentially, parallel, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added.
  • unit or “module” used in the present disclosure includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, part, or circuit.
  • a “unit” or “module” may be an integrally formed part or a minimum unit or a part that performs one or more functions.
  • the module may be configured as an application-specific integrated circuit (ASIC).
  • ASIC application-specific integrated circuit
  • Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium readable by a machine (eg, a computer).
  • the device calls the stored instructions from the storage medium. and, as a device capable of operating according to the called command, the electronic device (eg, the electronic device 100) according to the disclosed embodiments may be included.
  • the processor may perform a function corresponding to the instruction by using other components directly or under the control of the processor.
  • Instructions may include code generated or executed by a compiler or interpreter.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

Sont divulgués un dispositif électronique et son procédé de commande. Un procédé de commande d'un dispositif électronique selon la présente invention comprend les étapes consistant à : obtenir des premières informations de flux concernant un flux optique entre une première image et une deuxième image ; obtenir des deuxièmes informations de flux en effectuant une déformation directe afin d'obtenir une troisième image correspondant à une vue différente d'une vue de la première image et d'une vue de la deuxième image sur la base des premières informations de flux ; obtenir des troisièmes informations de flux en effectuant une déformation indirecte afin d'obtenir la troisième image sur la base des premières informations de flux ; obtenir des informations concernant un flux optique correspondant à la position d'un trou inclus dans les deuxièmes informations de flux sur la base des troisièmes informations de flux, de façon à obtenir des quatrièmes informations de flux afin d'obtenir la troisième image ; et obtenir la troisième image sur la base des quatrièmes informations de flux, d'une valeur de pixel de la première image et d'une valeur de pixel de la deuxième image.
PCT/KR2021/015572 2020-12-28 2021-11-01 Dispositif électronique et son procédé de commande Ceased WO2022145675A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020200185244A KR102892290B1 (ko) 2020-12-28 2020-12-28 전자 장치 및 전자 장치의 제어 방법
KR10-2020-0185244 2020-12-28

Publications (1)

Publication Number Publication Date
WO2022145675A1 true WO2022145675A1 (fr) 2022-07-07

Family

ID=82259348

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2021/015572 Ceased WO2022145675A1 (fr) 2020-12-28 2021-11-01 Dispositif électronique et son procédé de commande

Country Status (2)

Country Link
KR (1) KR102892290B1 (fr)
WO (1) WO2022145675A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150055322A (ko) * 2013-11-13 2015-05-21 삼성전자주식회사 다시점 영상 디스플레이 장치 및 그 다시점 영상 디스플레이 방법
KR20170140187A (ko) * 2015-04-23 2017-12-20 오스텐도 테크놀로지스 인코포레이티드 깊이 정보를 이용한 완전 시차 압축 광 필드 합성을 위한 방법
US20190320186A1 (en) * 2018-04-12 2019-10-17 Ostendo Technologies, Inc. Methods for MR-DIBR Disparity Map Merging and Disparity Threshold Determination
WO2020150264A1 (fr) * 2019-01-15 2020-07-23 Portland State University Déformation de pyramide de caractéristiques pour une interpolation de trame vidéo

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150055322A (ko) * 2013-11-13 2015-05-21 삼성전자주식회사 다시점 영상 디스플레이 장치 및 그 다시점 영상 디스플레이 방법
KR20170140187A (ko) * 2015-04-23 2017-12-20 오스텐도 테크놀로지스 인코포레이티드 깊이 정보를 이용한 완전 시차 압축 광 필드 합성을 위한 방법
US20190320186A1 (en) * 2018-04-12 2019-10-17 Ostendo Technologies, Inc. Methods for MR-DIBR Disparity Map Merging and Disparity Threshold Determination
WO2020150264A1 (fr) * 2019-01-15 2020-07-23 Portland State University Déformation de pyramide de caractéristiques pour une interpolation de trame vidéo

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
TINGHUI ZHOU; RICHARD TUCKER; JOHN FLYNN; GRAHAM FYFFE; NOAH SNAVELY: "Stereo Magnification: Learning View Synthesis using Multiplane Images", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 24 May 2018 (2018-05-24), 201 Olin Library Cornell University Ithaca, NY 14853 , XP080882127 *

Also Published As

Publication number Publication date
KR102892290B1 (ko) 2025-11-27
KR20220094003A (ko) 2022-07-05

Similar Documents

Publication Publication Date Title
WO2020171583A1 (fr) Dispositif électronique pour stabiliser une image et son procédé de fonctionnement
WO2020162673A1 (fr) Dispositif électronique pour générer une animation d'avatar et procédé associé
JP7359521B2 (ja) 画像処理方法および装置
WO2016032292A1 (fr) Procédé de photographie et dispositif électronique
WO2020235860A1 (fr) Appareil de traitement d'image et procédé de traitement d'image associé
WO2015102317A1 (fr) Appareil et procédé de traitement d'image
WO2019164185A1 (fr) Dispositif électronique et procédé de correction d'une image corrigée selon un premier programme de traitement d'image, selon un second programme de traitement d'image dans un dispositif électronique externe
US20170269187A1 (en) Image pickup apparatus having function of generating simulation image,control method therefor, and storage medium
WO2019164288A1 (fr) Procédé de fourniture de données de gestion de traduction de texte associées à une application, et dispositif électronique associé
CN110889809B9 (zh) 图像处理方法及装置、电子设备、存储介质
WO2023075508A1 (fr) Dispositif électronique et procédé de commande associé
WO2021162263A1 (fr) Procédé de génération d'image et dispositif électronique associé
WO2022145675A1 (fr) Dispositif électronique et son procédé de commande
WO2024147417A1 (fr) Procédé, dispositif électronique et support lisible par machine pour l'ordonnancement du débruitage avec adaption à la scène et un déparasitage efficace
US20200184693A1 (en) Computer program, image processing method, and image processing apparatus
WO2021015500A1 (fr) Dispositif d'affichage et procédé de commande de celui-ci
WO2022098164A1 (fr) Dispositif électronique et son procédé de commande
WO2025063741A1 (fr) Procédé et dispositif électronique de traitement de signal d'image à l'aide d'une intelligence artificielle à domaine compressé
WO2025127330A1 (fr) Restauration d'images temporellement cohérente à l'aide d'un modèle de diffusion
WO2024035223A1 (fr) Système et procédé pour améliorer la qualité d'une vidéo
WO2025206566A1 (fr) Dispositif électronique pour délivrer en sortie une qualité d'image en tant que score et son procédé de commande
WO2025042130A1 (fr) Dispositif électronique pour fournir un contenu d'image hdr et procédé de fonctionnement associé
WO2024053849A1 (fr) Dispositif électronique et son procédé de traitement d'image
WO2021025445A1 (fr) Mise en correspondance d'histogramme local avec régularisation globale et exclusion de mouvement pour fusion d'image à expositions multiples
WO2024122913A1 (fr) Dispositif électronique pour acquérir une image à l'aide d'un modèle d'apprentissage automatique, et son procédé de fonctionnement

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21915473

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21915473

Country of ref document: EP

Kind code of ref document: A1