[go: up one dir, main page]

WO2021040345A1 - Dispositif électronique et procédé de commande de dispositif électronique - Google Patents

Dispositif électronique et procédé de commande de dispositif électronique Download PDF

Info

Publication number
WO2021040345A1
WO2021040345A1 PCT/KR2020/011227 KR2020011227W WO2021040345A1 WO 2021040345 A1 WO2021040345 A1 WO 2021040345A1 KR 2020011227 W KR2020011227 W KR 2020011227W WO 2021040345 A1 WO2021040345 A1 WO 2021040345A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
information
neural network
network model
angle
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2020/011227
Other languages
English (en)
Korean (ko)
Inventor
황진영
한명희
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of WO2021040345A1 publication Critical patent/WO2021040345A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Definitions

  • the present disclosure relates to an electronic device and a method of controlling the electronic device, and more particularly, to an electronic device and a control method of the electronic device for obtaining a corrected image through a learned neural network model to correct a distorted image. .
  • a deep learning neural network model is being used to determine the density and movement of people.
  • the deep learning neural network model performs training using a lot of data, and the trained neural network model is based on the training data when new data is input. You can print out the best results.
  • the present disclosure was devised in accordance with the above-described necessity, and an object of the present disclosure is to input information on a distorted image and a distorted image into a trained neural network model to correct a distorted image, thereby correcting the distorted image.
  • An electronic device capable of acquiring an image and a method of controlling the electronic device are provided.
  • a method of controlling an electronic device may include: acquiring a first image captured through a camera inclined at a first angle with respect to a ground surface and information on the first angle; And inputting information on the first image and the first angle into a learned first neural network model to correct a distorted image to obtain a second image that is a corrected image for the first image.
  • the first neural network model is characterized in that learning based on a plurality of third images photographed at a plurality of angles with respect to the ground and second angle information for the plurality of angles.
  • an electronic device includes a memory including at least one instruction, a processor connected to the memory and controlling the electronic device, wherein the processor executes the at least one instruction,
  • the first image and the first image captured through a camera inclined at a first angle relative to the ground, and information on the first angle are obtained, and the first image and the first image are added to the learned first neural network model to correct a distorted photo.
  • a second image which is a corrected image for the first image, is obtained, and the first neural network model includes a plurality of third images photographed at a plurality of angles with respect to the ground, and the It may be learned based on second angle information for a plurality of angles.
  • the electronic device acquires information on the object through the first neural network model and the second neural network model, thereby accurately obtaining information on the object included in the image even through an image photographed by distorting the object. can do.
  • FIG. 1 is a diagram illustrating an example of using an electronic device according to an embodiment of the present disclosure
  • FIG. 2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure
  • FIG. 3 is a flowchart illustrating a method for obtaining a corrected image according to an embodiment of the present disclosure
  • 4A is a diagram illustrating a method for obtaining a training image for training a neural network model according to an embodiment of the present disclosure
  • 4B is a diagram illustrating a method of training a neural network model through a training image according to an embodiment of the present disclosure
  • 5A is a diagram illustrating a method of learning a neural network model further using distance information according to an embodiment of the present disclosure
  • 5B is a diagram illustrating a method of learning a neural network model further using distance information according to an embodiment of the present disclosure
  • 6A is a diagram illustrating a method of obtaining a distorted image according to an embodiment of the present disclosure
  • 6B is a diagram illustrating a method of obtaining a corrected image using a distorted image and angle information of a camera according to an embodiment of the present disclosure
  • FIG. 7A is a diagram illustrating a method of obtaining a distorted image according to an embodiment of the present disclosure
  • FIG. 7B is a diagram illustrating a method of obtaining a corrected image using a distorted image, angle information of a camera, and distance information between an object and a camera according to an embodiment of the present disclosure
  • FIG. 8 is a block diagram showing a configuration of an external server for learning a first neural network model according to an embodiment of the present disclosure
  • FIG. 9 is a sequence diagram for obtaining a corrected image using a neural network model learned in a server according to an embodiment of the present disclosure.
  • FIG. 10 is a block diagram illustrating a detailed configuration of the electronic device disclosed in FIG. 2.
  • FIG. 1 is a diagram illustrating an example of using an electronic device according to an embodiment of the present disclosure.
  • the electronic device 100 may acquire a first image 10 photographed through a camera inclined at a first angle relative to the ground.
  • the first image 10 according to FIG. 1 is an image photographed by a camera at a higher position than the object, and is an image photographed by distorting the object to look shorter than the image photographed from the front.
  • the first image 10 of FIG. 1 is illustrated as a distorted image of one object, but the present invention is not limited thereto, and the first image 10 may be an entire image including a plurality of objects. Specifically, the first image may be an entire image captured by a camera installed at a high position in a store or an exhibition. Further, the present invention is not limited thereto, and an image including an area in which at least one object is located in the entire image may be the first image. In addition, the first image 10 of FIG. 1 is an image captured by a camera at a higher position than the object, but is not limited thereto, and may be an image captured by a camera at a lower position than the object, such as a robot cleaner. .
  • the electronic device 100 may obtain information on a first angle indicating a degree of inclination with respect to the camera in which the first image 10 is captured.
  • the information on the first angle means an angle indicating the degree of inclination of the camera with respect to the ground, and according to an embodiment, information on the angle may be obtained by installing a gyro sensor in the camera. A method of obtaining information on the first image and the first angle will be described later with reference to FIG. 5A.
  • the electronic device 100 inputs the acquired first image 10 and information about the first angle into the learned first neural network model 110 to correct the distorted picture, and the first neural network model 110 As an output, a second image 20 that is a corrected image for the first image may be obtained.
  • the first neural network model 110 is a neural network model for correcting a distorted picture, and may be learned based on a plurality of images photographed at a plurality of angles with respect to the ground and information about a plurality of angles.
  • the first neural network model 110 inputs a plurality of distortion learning images photographed at a plurality of angles with respect to the ground, among a plurality of training images, and learns to output a horizontal learning image photographed horizontally with respect to the ground. A specific method of learning the first neural network will be described later with reference to FIG. 4B.
  • the second image 20 output through the first neural network model 110 is a corrected image for the first image 10, which is captured by a camera positioned higher than the object and distorted to make the object look shorter. It is a corrected image as if the object was photographed from the front by correcting.
  • the present invention is not limited thereto, and the second image may be an image obtained by correcting a distorted image to make the object look longer by being photographed by a camera positioned lower than the object.
  • the electronic device 100 inputs the acquired second image 20 to the second neural network model 120 for acquiring information about the image, and uses the second image 20 as an output of the second neural network model 120.
  • the second neural network model 120 is a neural network model for acquiring information on an image, and may be learned based on an image photographed horizontally with respect to the object and information on the object. That is, information on the second image may be obtained by inputting an image photographed horizontally with respect to the ground in the learned second neural network model 120.
  • the information on the second image may be a neural network model for obtaining information on the type of the captured object, coordinate information, and information on the time when the object was captured.
  • the information on the type of the object may include information on gender and age group, etc.
  • the information on the coordinates of the object is information on the coordinates at which the object is located at the time when the object is photographed, and is used together with information on the time when the object is photographed, so that information on the movement of the object according to time may be obtained. That is, by using the information on the second image, information on the density of the object and the movement of the object with respect to the location where the first image 10 corresponding to the second image 20 is captured may be obtained.
  • the first neural network model is learned based on the plurality of images and angle information corresponding to each of the plurality of images, but this is only an embodiment, and each of the plurality of images and the plurality of images It may be learned by not only angle information corresponding to, but also distance information corresponding to each of a plurality of images.
  • the distance information may be information about the distance between the object and the camera that photographed the object, but this is only an example and may be information about the relative distance between objects.
  • the electronic device 100 may obtain a corrected image for the first image by inputting the first image and the first angle information as well as the first distance information to the first neural network model together. A detailed description of this will be described later with reference to FIGS. 7A and 7B.
  • FIG. 2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.
  • the electronic device 100 may include a memory 110 and a processor 120.
  • the memory 110 may store various programs and data necessary for the operation of the electronic device 100. Specifically, at least one command may be stored in the memory 110.
  • the processor 120 may perform an operation of the electronic device 100 by executing a command stored in the memory 110.
  • the memory 110 may be implemented as a nonvolatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or the like.
  • the memory 110 is accessed by the processor 120, and data read/write/edit/delete/update by the processor 120 may be performed.
  • the term memory refers to a memory 110, a ROM (not shown) in the processor 120, a RAM (not shown), or a memory card (not shown) mounted in the electronic device 100 (eg, micro SD Card, memory stick).
  • the memory 110 may store programs and data for configuring various screens to be displayed in the display area of the display.
  • the memory 110 may store the learned first neural network model to correct the distorted picture.
  • the first neural network model may be learned based on information on a plurality of images and a plurality of angles taken of an object at a plurality of angles with respect to the ground.
  • the learned first neural network model may be learned from an external server and transmitted from an external server.
  • the memory 110 may store a second neural network model for obtaining information on an image.
  • the second neural network model may be learned based on an image photographed horizontally with respect to the ground.
  • the learned second neural network model may be learned from an external server and transmitted from an external server.
  • the memory 110 may store a first image captured by a camera inclined at a first angle relative to the ground and information on the first angle.
  • the tilted angle of the camera is measured through a gyro sensor installed in the camera, and information on the measured angle may be stored.
  • the gyro sensor is a sensor for measuring an angle using an inertial input applied from the outside.
  • a gyro sensor installed in the camera may detect the movement of the camera and measure the angle at which the camera is inclined based on a signal corresponding to the movement.
  • a first image may be acquired by an external camera, and the acquired first image may be stored in the memory 110.
  • the first image may be photographed in real time by an external camera and transmitted to the electronic device 100, and may be stored in the memory 110 together with information on a time when the first image was captured.
  • Functions related to artificial intelligence are operated through the processor 120 and the memory 110.
  • the processor 120 may be composed of one or a plurality of processors.
  • one or more processors are general-purpose processors such as CPUs and APs, GPUs. It may be a graphics dedicated processor such as a VPU or an artificial intelligence dedicated processor such as an NPU.
  • One or more processors control to process input data according to a predefined operation rule or an artificial intelligence model stored in the memory.
  • a predefined motion rule or artificial intelligence model is characterized by being created through learning.
  • being made through learning means that a predefined operation rule or an artificial intelligence model of a desired characteristic is created by applying a learning algorithm to a plurality of training data.
  • Such learning may be performed in a device on which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server/system.
  • the artificial intelligence model may be composed of a plurality of neural network layers. Each layer has a plurality of weight values, and a layer operation is performed through the operation result of a previous layer and a plurality of weights.
  • Examples of neural networks include CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network) and deep There are Q-Networks (Deep Q-Networks), and the neural network in the present disclosure is not limited to the above-described example except for the case where it is specified.
  • the learning algorithm is a method in which a predetermined target device (eg, a robot) is trained using a plurality of pieces of learning data so that a predetermined target device can make a decision or make a prediction by itself.
  • Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in this disclosure is specified. It is not limited to the above-described example except for.
  • the processor 120 may be electrically connected to the memory 110 to control overall operation of the electronic device 100. Specifically, the processor 120 may control the electronic device 100 by executing at least one command stored in the memory 110.
  • the processor 120 of the present disclosure may acquire a first image captured through a camera inclined at a first angle with respect to the ground, and may acquire information about the first angle.
  • the first image is an image captured by a camera installed at a high position or a camera installed at a lower position, and may be an image captured by distorting the object to look longer or shorter than an image captured from the front.
  • the processor 120 may input information about the first image and the first angle to the first neural network model, and obtain a second image that is a corrected image for the first image as an output of the first neural network model.
  • the first neural network model is a neural network model that has been trained to correct a distorted picture, and is trained by an external server to transmit the first neural network model from the external server.
  • the first neural network model may be learned based on a plurality of third images photographed at a plurality of angles with respect to the ground and second angle information for the plurality of angles.
  • the first neural network model may be a VAE (Variational Auto-encoder) model or a GAN (Generative Adversarial Networks) model, but this is only an embodiment and may be implemented as another artificial intelligence model.
  • the VAE model is a neural network model including an encoder network and a decoder network, and the VAE model according to the present disclosure uses data corresponding to a distorted image as input data to obtain data corresponding to a corrected image for a distorted image. It may include at least one learned encoder network and at least one decoder network. In this case, the VAE model may acquire information (eg, average, variance) on a latent variable for converting a distorted image into a corrected image through training.
  • information eg, average, variance
  • the GAN model is a neural network model including an identifier (Discriminator) and a generator (Generator), and the generator of the GAN model according to the present disclosure generates a corrected image corresponding to an image in which an object is distorted, and the identifier is a correction generated by the creator. It is possible to determine whether the generated image is an image generated by a creator or an image actually captured. That is, the creator is trained to elaborately generate a corrected image so that it is difficult to determine whether the identification target image is an actual photographed image or an image generated by the creator. It is trained to precisely determine whether it is an image, so that the performance of the GAN model can be improved.
  • the second image is a corrected image for the first image, and may be an image corrected such that an object that is distorted to look short or long is photographed from the front.
  • the processor 120 may identify a region in which an object according to time is located in the first image, and obtain a first object image corresponding to the identified region. That is, the processor 120 may input the first image, which is the entire image captured by the object through the camera, into the learned first neural network model, but is not limited thereto and inputs the acquired first object image into the first neural network model. Thus, a second object image, which is a corrected image for the first object, may be obtained.
  • the processor 120 may obtain information on the second image as an output of the second neural network model by inputting the second image to the second neural network model.
  • the second neural network model is a neural network model for obtaining information on an image
  • the electronic device 100 may receive the learned second neural network model from an external server.
  • the second neural network model may be trained based on an image photographed horizontally with respect to the ground.
  • the second neural network model according to the present disclosure may be implemented as a convolutional neural network (CNN) model.
  • the information on the second image may include information on the type of object included in the second image, coordinate information, and information on a time when the object was detected.
  • the processor 120 may acquire information on a density of a place where an object included in the second image is photographed, and a movement line of at least one object to the place, based on the obtained information on the second image. have.
  • FIG. 3 is a flowchart illustrating a method for obtaining a corrected image according to an embodiment of the present disclosure.
  • the electronic device may obtain a first image captured through a camera inclined at a first angle and information on the first angle (S310 ).
  • the first image may be an image captured by an external camera, and information on the first angle, which is the tilt information of the camera, may be obtained through a gyro sensor installed in the external camera.
  • the external camera is inclined at the first angle relative to the ground, and in a camera at a higher position than the object, the object may be distorted to make it look longer than the one photographed from the front, and a camera at a lower position than the object may be photographed from the front.
  • the object may be distorted to look shorter and photographed. That is, the first image is an image captured by distorting the object to look long or short.
  • the electronic device may obtain a second image by inputting the first image and first angle information to the first neural network model (S320).
  • the first neural network model is a trained neural network model to correct the distorted first image.
  • a first neural network model is learned based on a plurality of third images photographed at a plurality of angles with respect to the surface of an object from an external server and the second angle information for the plurality of angles, and the learned first neural network model is transferred to the electronic device. Can be transmitted.
  • the second image is a corrected image for the first image, and may be an image corrected as if an object included in the first image was photographed from the front.
  • the electronic device 100 may obtain information on the second image by inputting the second image into the second neural network model.
  • the second neural network model is a neural network model for acquiring image information.
  • a second neural network model may be trained based on an image photographed horizontally with respect to the ground by an external server, and the learned second neural network model may be transmitted to the electronic device.
  • the information on the second image may include information on the type of object included in the second image, coordinate information, and information on a time when the object was detected.
  • 4A is a diagram illustrating a method for obtaining a training image for training a neural network model according to an embodiment of the present disclosure.
  • training of the first neural network model may be performed by an external server.
  • a horizontal training image 40-1 and a plurality of Distortion learning images 40-2, 40-3, and 40-4 may be obtained.
  • the horizontal learning image 40-1 is an image photographed by the camera 400-1 installed horizontally with respect to the ground, and is an image photographed without distorting the object.
  • the third distortion learning image 40-4 photographed by the third camera 400-4 at a lower position may be an image photographed by distorting the object to look longer than the one photographed from the front.
  • ⁇ 400-2, ⁇ 400-3, ⁇ 400-4 information about the angles in which the plurality of cameras 400-1 to 400-4 are inclined at a plurality of angles with respect to the ground may be obtained.
  • ⁇ 400-2 is angle information on the degree of inclination of the first camera 400-2, and the first distortion in which the first camera 400-2 is inclined by ⁇ 400-2 with respect to the ground to photograph the object.
  • the training image 40-2 may be acquired.
  • ⁇ 400-3 is angle information on the degree of inclination of the second camera 400-3, and the second distortion learning image ( 40-3) can be obtained.
  • information on a plurality of angles (( ⁇ 400-2, ⁇ 400-3, ⁇ 400-4) for a plurality of cameras 400-1 to 400-4 is a gyro sensor installed in the camera.
  • Fig. 4A an image is obtained through four cameras, but the number of cameras is not limited thereto, and in Fig. 4A, it is illustrated that only one object 40 is photographed, but is not limited thereto.
  • Learning about the first neural network model may be performed based on a horizontal training image photographing a plurality of objects 40 and a plurality of distortion training images.
  • 4B is a diagram illustrating a method of training a neural network model through a training image according to an embodiment of the present disclosure.
  • the first neural network model may be trained through ⁇ 400-4).
  • the first neural network model is a neural network model for correcting a distorted picture.
  • a first distortion learning image 40-2, a second distortion learning image 40-3, a third distortion learning image 40-4, and a plurality of The first neural network model can be trained so that the horizontal training image 40-1 is output by inputting angle information ( ⁇ 400-2, ⁇ 400-3, ⁇ 400-4) corresponding to the distortion learning image of have.
  • the first neural network model may be trained in an external server and the learned first neural network model may be transmitted to the electronic device 100.
  • FIG. 4B shows that the first neural network model is trained through a horizontal training image 40-1 for one object 40 and a plurality of distortion training images 40-2, 40-3, and 40-4.
  • the present invention is not limited thereto, and the first neural network model may be trained through a horizontal learning image and a distortion learning image for a plurality of objects.
  • a corrected image for a distorted image may be obtained through the first neural network model 410 learned based on FIGS. 4A and 4B.
  • 5A is a diagram illustrating a method of learning a neural network model further using distance information according to an embodiment of the present disclosure.
  • a horizontal training image in order to train the first neural network model, a horizontal training image, a plurality of distortion training images 50-2, 50-3, 60-2, 60-3, and a plurality of cameras 500-1 to 500-1 It is possible to obtain information (( ⁇ 500-2, ⁇ 500-3) about the angle 500-3) is inclined at a plurality of angles with respect to the ground. Since the horizontal learning image and the plurality of distortion images have been described in FIG. 4A, Detailed information will be omitted.
  • first distance information (d50-1 to d50-3, d60-1 to d60-3) indicating the distance between the plurality of objects 50 and 60 and the plurality of cameras 500-1 to 500-3 is obtained.
  • the degree of distortion of the distorted image may vary depending on the distance between the object and the camera. For example, an image photographed of an object close to the camera may have a greater degree of distortion than an image photographed of an object far from the camera. That is, in a camera at a higher position than the object, an image of an object close to the camera may be distorted so that the object looks shorter than an image of an object far from the camera may be photographed. Referring to FIG.
  • information d50-1 indicating a distance between the first object 50 and the first camera 500-1 may be obtained through a depth sensor of the first camera 500-1.
  • the depth sensor is a sensor for measuring a distance to an object, and according to an embodiment, a distance to an object may be measured in a Time of Flight (TOF) method.
  • the TOF method is a method of measuring the flight time until the light reflected from the object is received by the depth sensor after light is irradiated onto the object.
  • a horizontal learning image and a plurality of distortion learning images are acquired through two objects 50 and 60 and three cameras 500-1 to 500-3, but are not limited to two objects.
  • a horizontal learning image and a plurality of distortion learning images may be obtained through more than one object.
  • 5B is a diagram illustrating a method of learning a neural network model further using distance information according to an embodiment of the present disclosure.
  • the first neural network model 510 may be trained through information about the angle of and the distance between the camera and the object.
  • the first neural network model 510 is a neural network model for correcting a distorted picture.
  • the first horizontal learning image 50-1 for the first object 50 is set to be output, and the third distortion learning image 60-2 and the fourth distortion learning image for the second object 60 (60-3), information on the angles of the plurality of cameras 500-1 to 500-3 ( ⁇ 500-2, ⁇ 500-3), and the second object 60 and a plurality of cameras 500-1 to 500- 3) Set to output the second horizontal training image 60-1 for the second object 60 by inputting information on the distance between d60-1 to d60-3 into the first neural network model 510
  • the first neural network model can be trained.
  • the first neural network model 510 may be trained in an external server and the learned first neural network model 510 may be transmitted to the electronic device 100.
  • the first neural network model is trained through a horizontal learning image and a distortion learning image for the first object 50 and the second object 60, but the present invention is not limited thereto.
  • the first neural network model may be trained through the horizontal training image and the distortion training image acquired through the training.
  • 6A is a diagram illustrating a method of obtaining a distorted image according to an exemplary embodiment of the present disclosure.
  • the electronic device 100 may obtain a first image captured by distorting the object 60 through the camera 610 inclined at a first angle ⁇ 610 with respect to the ground.
  • the first image is obtained through the camera 610 at a position higher than the object 60, but the present invention is not limited thereto, and the first image may also be obtained through a camera at a position lower than the object 60.
  • the acquired first image may be an image that is distorted to look shorter than the image captured from the height of the object 60.
  • the object of the acquired first image may be an image distorted so that the object looks longer than the image photographed from the front.
  • the electronic device 100 may obtain information on a first angle ⁇ 610 indicating a degree of inclination of the camera 610.
  • information on the first angle ⁇ 610 of the camera 610 may be obtained through a gyro sensor installed in the camera 610.
  • 6B is a diagram illustrating a method of obtaining a corrected image using a distorted image and angle information of a camera according to an exemplary embodiment of the present disclosure.
  • the electronic device 100 converts a first image, which is an image captured by distorting an object, and first angle information, which is angle information about a camera that captures the object, to the learned first neural network model 610 to correct the distorted image. By inputting, a second image that is a corrected image for the first image may be obtained.
  • the first neural network model 610 may be a neural network model learned based on the horizontal training image and the distortion training image described above in FIGS. 4A and 4B. That is, the first neural network model 610 is learned based on the horizontal training image and the distortion training image in the external server, and the electronic device 100 may receive the learned first neural network model 610 from the external server.
  • the second image may be an image obtained by correcting an image of an object that is distorted and photographed as a corrected image of the first image, such that the image is captured from the height of the object. That is, the first neural network model 610 is trained based on the horizontal training image, the angle information of the camera, and the distortion training image, and when information about the distortion image and the angle is input to the learned first neural network model 610, the distortion A corrected image for the image may be output.
  • the electronic device 100 may obtain information on the image by inputting the second image acquired through the first neural network model 610 into a second neural network model (not shown). Details of the second neural network model have been described above with reference to FIG. 1, and thus will be omitted.
  • FIG. 7A is a diagram illustrating a method of obtaining a distorted image according to an embodiment of the present disclosure.
  • the electronic device 100 may obtain a first image captured by distorting the objects 70 and 80 through the camera 710 inclined at a first angle ⁇ 710 with respect to the ground.
  • the camera 710 may be installed at a position higher than the object or may be installed at a position lower than the object.
  • the electronic device 100 may obtain information on the first angle ⁇ 710 indicating the degree of inclination of the camera 710. Also, the electronic device 100 may obtain second distance information indicating a distance between the camera 710 and the objects 70 and 80. As an example, a distance between the camera 710 and the objects 70 and 80 may be calculated using a depth sensor installed in the camera 710 to obtain second distance information. Specifically, the second distance information may include distance information d70 between the first object 70 and the camera 710 and distance information d80 between the second object 80 and the camera 710. However, it is not limited to the first object 70 and the second object 70 as shown in FIG. 7A, and distance information between the plurality of objects photographed by the camera 710 and the camera 710 may be further included.
  • FIG. 7B is a diagram illustrating a method of obtaining a corrected image using a distorted image, angle information of a camera, and distance information between an object and a camera according to an embodiment of the present disclosure.
  • the electronic device 100 is a distorted image of a first image, which is an image captured by distorting an object, first angle information indicating angle information about a camera in which the object was photographed, and second distance information indicating distance information between the object and the camera.
  • a second image which is a corrected image for the first image, may be obtained by inputting into the learned first neural network model 710 to correct for.
  • the first neural network model 710 may be a neural network model trained based on the horizontal training image and the distortion training image described above in FIGS. 5A and 5B. That is, the first neural network model 710 is learned based on the horizontal training image and the distortion training image in the external server, and the electronic device 100 may receive the learned first neural network model 710 from the external server.
  • the second image is a corrected image for the first image, and may be an image obtained by correcting an image of an object that has been distorted and photographed as if it was photographed from the height of the object. That is, the first neural network model 710 is learned based on the horizontal training image, the angle information of the camera, and the distance information between the camera and the object, and when the distortion image, the angle information, and the distance information are input to the first neural network model 710, , A corrected image for the distorted image may be output.
  • the electronic device 100 may obtain information on the image by inputting the second image acquired through the first neural network model 710 into a second neural network model (not shown),
  • the content of the second neural network model has been described above in FIG. 1, and thus will be omitted.
  • FIG. 8 is a block diagram illustrating a configuration of an external server for learning a first neural network model according to an embodiment of the present disclosure.
  • the server 800 may include a memory 810, a communication unit 820, and a processor 830.
  • the memory 810 may store various programs and data required for the operation of the server 800. Specifically, at least one command may be stored in the memory 810.
  • the processor 830 may perform an operation of the server 800 by executing a command stored in the memory 810.
  • the memory 810 may store a distortion training image and a horizontal training image for training the first neural network model.
  • the distortion learning image and the horizontal training image are training images for learning the first neural network model for correcting the distorted image
  • the horizontal training image is an image taken horizontally with respect to the ground
  • the distortion learning image is It may be an image taken at a plurality of angles as a reference.
  • the memory 810 may store second angle information for a plurality of angles corresponding to a plurality of distortion learning images.
  • the memory 810 may store a distortion learning image and a first neural network model learned based on the horizontal learning image.
  • the first neural network model is a neural network model for correcting a distorted image and may be learned by the server 800 and transmitted to the electronic device 100.
  • the communication unit 820 is a component that communicates with various types of external devices according to various types of communication methods.
  • the communication unit 820 may be implemented as a Wi-Fi module. That is, the WiFi module of the communication unit 820 may transmit the learned first neural network model to the electronic device 100. Also, the communication unit 820 may receive the distortion learning image and the horizontal learning image from an electronic device installed with a camera or an external server.
  • the processor 830 may be electrically connected to the memory 810 to control overall operation of the server 800. Specifically, the processor 830 may control the server 800 by executing at least one command stored in the memory 810.
  • the processor 830 of the present disclosure may train the first neural network model based on a plurality of third images photographed at a plurality of angles with respect to the ground and second angle information for the plurality of angles.
  • the third image is a training image for training the first neural network model, and may include a horizontal training image photographing an object horizontally with respect to the ground and a plurality of distortion learning images photographing the object at a plurality of angles with respect to the ground. .
  • the processor 830 inputs second angle information about a plurality of angles corresponding to a plurality of distortion learning images and a plurality of distortion learning images into the first neural network model, and a horizontal learning image corresponding to the plurality of distortion learning images.
  • the first neural network model may be trained to output this.
  • the processor 830 collects a plurality of distortion learning images, angle information corresponding to a plurality of distortion learning images, and distance information between the object and the camera taking the distortion learning image.
  • the first neural network model may be trained to output horizontal training images corresponding to a plurality of distortion training images by inputting the first neural network model.
  • FIG. 9 is a sequence diagram for obtaining a corrected image using a neural network model learned in a server according to an embodiment of the present disclosure.
  • the server 800 may acquire a third image and second angle information (S910).
  • the third image may include a horizontal learning image obtained by photographing the object horizontally with respect to the ground and a plurality of distortion learning images photographing the object at a plurality of angles with respect to the ground.
  • the server 800 may train the first neural network model based on the acquired third image and second angle information (S920). Specifically, the server 800 inputs a plurality of distortion learning images and second angle information about a plurality of angles corresponding to the plurality of distortion learning images into the first neural network model, and outputs a horizontal learning image corresponding to the distortion learning image. If possible, the first neural network model can be trained.
  • the server 800 may transmit the learned first neural network model to the electronic device 100 (S930). That is, the electronic device 100 may receive the first neural network model on which the training has been completed from the server 800 to obtain a corrected image for the distorted image.
  • the electronic device 100 may obtain information about the first image and the first angle (S940).
  • the first image is a distorted image obtained by photographing an object through a camera positioned higher or lower than the object, and may be an image distorted such that the object looks shorter or longer than an image photographed from the front.
  • the information on the first angle means information on an angle indicating the degree to which the camera photographing the object is inclined with respect to the ground.
  • the electronic device 100 may acquire a second image by inputting the acquired first image and information about the first angle into the first neural network model (S950).
  • the second image is an image corrected as if the object was photographed from the front by correcting the first image distorted to make the object look short or long.
  • the first image captured by the camera inclined at the first angle and the information on the first angle are input to the learned first neural network model to correct the distorted image, and the corrected image for the first image.
  • a second image may be acquired.
  • FIG. 10 is a block diagram illustrating a detailed configuration of the electronic device disclosed in FIG. 2.
  • the electronic device 1000 may include a memory 1010, a processor 1020, a communication unit 1030, a camera 1040, and a display 1050.
  • the communication unit 1030 is a component for performing communication with an external server. Meanwhile, the communication connection between the communication unit 130 and an external server may include communicating through a third device (eg, a repeater, a hub, an access point, a server, or a gateway).
  • a third device eg, a repeater, a hub, an access point, a server, or a gateway.
  • the communication unit 1030 may receive a first neural network model and a second neural network model from the server by communicating with the server through wired communication or wireless communication.
  • the electronic device 100 may receive a first neural network model and a second neural network model learned through a wireless connection such as a server and a Wi-Fi, Bluetooth, or NFC tag.
  • the first neural network model and the second neural network model may be learned in one server and transmitted to the electronic device 100, but are not limited thereto, and the first neural network model and the second neural network model are trained in separate servers. It may be received by the device 100.
  • the processor 1020 may obtain the corrected image through the first neural network model received through the communication unit 1030 and may obtain information on the corrected image through the second neural network model.
  • the camera 1040 is disposed on one side of the electronic device 1000 to capture an object.
  • the electronic device 1000 may acquire a first image that is a distorted image by capturing the object through the camera 1040 of the electronic device 1000 at a position lower or higher than the object.
  • the camera 1040 may take a still image or a video, and the processor 1020 may identify a region in which an object according to time is located in the still image or video, and obtain a first object image corresponding to the identified region. I can.
  • the processor 1020 may input the first image captured by the object through the camera to the learned first neural network model, but is not limited thereto, and according to an embodiment of the present disclosure, the acquired first object image is The corrected image may be obtained by inputting it into the first neural network model.
  • the display 1050 may display various information according to the control of the processor 1020.
  • the display 1050 may display a first image that is a distorted image and a second image obtained by correcting the first image.
  • the display 1050 may display information on the second image.
  • the display 1050 may be implemented as a touch screen together with a touch panel.
  • embodiments described above may be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof.
  • embodiments described in the present disclosure include Application Specific Integrated Circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs). ), processors, controllers, micro-controllers, microprocessors, and electric units for performing other functions.
  • ASICs Application Specific Integrated Circuits
  • DSPs digital signal processors
  • DSPDs digital signal processing devices
  • PLDs programmable logic devices
  • FPGAs field programmable gate arrays
  • processors controllers, micro-controllers, microprocessors, and electric units for performing other functions.
  • the embodiments described herein may be implemented by the processor itself.
  • a non-transitory readable medium may be mounted and used in various devices.
  • the non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short moment, such as a register, cache, and memory.
  • programs for performing the above-described various methods may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, or the like.
  • a method according to various embodiments disclosed in this document may be provided by being included in a computer program product.
  • Computer program products can be traded between sellers and buyers as commodities.
  • the computer program product may be distributed online in the form of a device-readable storage medium (eg, compact disc read only memory (CD-ROM)) or through an application store (eg, Play StoreTM).
  • an application store eg, Play StoreTM
  • at least some of the computer program products may be temporarily stored or temporarily generated in a storage medium such as a server of a manufacturer, a server of an application store, or a memory of a relay server.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne un dispositif électronique. Un dispositif électronique de la présente invention comprend : une mémoire comprenant au moins une instruction ; et un processeur qui est connecté à la mémoire et qui commande le dispositif électronique, le processeur exécutant la ou les instructions de façon à obtenir une première image, capturée par l'intermédiaire d'une caméra inclinée selon un premier angle par rapport au sol, et des informations concernant le premier angle, et entre la première image et les informations concernant le premier angle en un premier modèle de réseau neuronal entraîné afin de corriger une photographie déformée, ce qui permet d'obtenir une deuxième image, qui est l'image corrigée de la première image, et le premier modèle de réseau neuronal est entraîné sur la base : d'une pluralité de troisièmes images dans lesquelles un objet est capturé à une pluralité d'angles par rapport au sol ; et des secondes informations d'angle concernant la pluralité d'angles.
PCT/KR2020/011227 2019-08-27 2020-08-24 Dispositif électronique et procédé de commande de dispositif électronique Ceased WO2021040345A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020190104786A KR20210025175A (ko) 2019-08-27 2019-08-27 전자 장치 및 전자 장치의 제어 방법
KR10-2019-0104786 2019-08-27

Publications (1)

Publication Number Publication Date
WO2021040345A1 true WO2021040345A1 (fr) 2021-03-04

Family

ID=74683830

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2020/011227 Ceased WO2021040345A1 (fr) 2019-08-27 2020-08-24 Dispositif électronique et procédé de commande de dispositif électronique

Country Status (2)

Country Link
KR (1) KR20210025175A (fr)
WO (1) WO2021040345A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115223028A (zh) * 2022-06-02 2022-10-21 支付宝(杭州)信息技术有限公司 场景重建及模型训练方法、装置、设备、介质及程序产品

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102677439B1 (ko) * 2023-11-20 2024-06-21 포스텍네트웍스(주) Ai 혼합모델링 기반 지능형 화소보정 led 전광판 시스템

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20110118495A (ko) * 2010-04-23 2011-10-31 광주과학기술원 객체 학습 방법, 객체 학습 방법을 이용한 객체 추적 방법, 객체 학습 및 추적 시스템
KR20170050465A (ko) * 2015-10-30 2017-05-11 에스케이텔레콤 주식회사 얼굴 인식 장치 및 방법
KR101793510B1 (ko) * 2017-03-27 2017-11-06 한밭대학교 산학협력단 얼굴 학습 및 인식 시스템과 그 방법
KR101817440B1 (ko) * 2016-09-23 2018-01-10 한국해양대학교 산학협력단 다중 카메라를 통한 3차원 모델 기반 객체 인식 기법과 시스템
US20180182083A1 (en) * 2016-12-27 2018-06-28 Intel IP Corporation Convolutional neural network for wide-angle camera images

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20110118495A (ko) * 2010-04-23 2011-10-31 광주과학기술원 객체 학습 방법, 객체 학습 방법을 이용한 객체 추적 방법, 객체 학습 및 추적 시스템
KR20170050465A (ko) * 2015-10-30 2017-05-11 에스케이텔레콤 주식회사 얼굴 인식 장치 및 방법
KR101817440B1 (ko) * 2016-09-23 2018-01-10 한국해양대학교 산학협력단 다중 카메라를 통한 3차원 모델 기반 객체 인식 기법과 시스템
US20180182083A1 (en) * 2016-12-27 2018-06-28 Intel IP Corporation Convolutional neural network for wide-angle camera images
KR101793510B1 (ko) * 2017-03-27 2017-11-06 한밭대학교 산학협력단 얼굴 학습 및 인식 시스템과 그 방법

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115223028A (zh) * 2022-06-02 2022-10-21 支付宝(杭州)信息技术有限公司 场景重建及模型训练方法、装置、设备、介质及程序产品
CN115223028B (zh) * 2022-06-02 2024-03-29 支付宝(杭州)信息技术有限公司 场景重建及模型训练方法、装置、设备、介质及程序产品

Also Published As

Publication number Publication date
KR20210025175A (ko) 2021-03-09

Similar Documents

Publication Publication Date Title
WO2021112406A1 (fr) Appareil électronique et procédé de commande associé
WO2020171553A1 (fr) Dispositif électronique appliquant un effet bokeh à une image, et procédé de commande associé
WO2021107610A1 (fr) Procédé et système de production d'une carte triple pour un matage d'image
WO2020027607A1 (fr) Dispositif de détection d'objets et procédé de commande
WO2019039757A1 (fr) Dispositif et procédé de génération de données d'apprentissage et programme informatique stocké dans un support d'enregistrement lisible par ordinateur
WO2021075772A1 (fr) Procédé et dispositif de détection d'objet au moyen d'une détection de plusieurs zones
WO2021040345A1 (fr) Dispositif électronique et procédé de commande de dispositif électronique
WO2021095991A1 (fr) Dispositif et procédé de génération d'une image de défaut
WO2019168264A1 (fr) Dispositif électronique et son procédé de commande
WO2019156543A2 (fr) Procédé de détermination d'une image représentative d'une vidéo, et dispositif électronique pour la mise en œuvre du procédé
WO2023075508A1 (fr) Dispositif électronique et procédé de commande associé
WO2020189953A1 (fr) Caméra analysant des images sur la base d'une intelligence artificielle, et son procédé de fonctionnement
CN115278097B (zh) 图像生成方法、装置、电子设备及介质
WO2023172031A1 (fr) Génération d'images de surveillance panoramique
WO2018097384A1 (fr) Appareil et procédé de notification de fréquentation
WO2023080667A1 (fr) Traitement d'image wdr de caméra de surveillance par reconnaissance d'objets basée sur l'ia
WO2021261727A1 (fr) Système et procédé de lecture d'images d'endoscopie par capsule
WO2021125550A1 (fr) Dispositif électronique et procédé de commande du dispositif électronique
WO2022108249A1 (fr) Procédé, appareil et programme de génération de données d'apprentissage, et procédé de détection de substances étrangères l'utilisant
WO2023224377A1 (fr) Procédé de gestion d'informations d'un objet et appareil appliquant ledit procédé
WO2022191424A1 (fr) Dispositif électronique et son procédé de commande
WO2023149603A1 (fr) Système de surveillance par images thermiques utilisant une pluralité de caméras
WO2023113347A1 (fr) Procédé d'analyse d'une image médicale
WO2022098164A1 (fr) Dispositif électronique et son procédé de commande
CN109547671A (zh) 图像传感器、相机模块以及成像设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20856253

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20856253

Country of ref document: EP

Kind code of ref document: A1