[go: up one dir, main page]

WO2018002677A1 - Procédé de reconstruction 3d à l'aide d'un dispositif mobile - Google Patents

Procédé de reconstruction 3d à l'aide d'un dispositif mobile Download PDF

Info

Publication number
WO2018002677A1
WO2018002677A1 PCT/HU2016/050028 HU2016050028W WO2018002677A1 WO 2018002677 A1 WO2018002677 A1 WO 2018002677A1 HU 2016050028 W HU2016050028 W HU 2016050028W WO 2018002677 A1 WO2018002677 A1 WO 2018002677A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
feature
modules
mobile device
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/HU2016/050028
Other languages
English (en)
Inventor
Ferenc István BALÁZS
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to PCT/HU2016/050028 priority Critical patent/WO2018002677A1/fr
Publication of WO2018002677A1 publication Critical patent/WO2018002677A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/579—Depth or shape recovery from multiple images from motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing

Definitions

  • the subject of the invention is a method and apparatus for three-dimensional reconstruction utilizing at least one mobile device equipped with a camera.
  • solutions are known to utilize one or more camera devices to image an object from multiple angles, and reconstruct the three- dimensional representation of the object.
  • These solutions utilize well-known methods in computer vision, such as methods for feature detection, and matching, of which the latter one is a complicated task, given the change in vantage point or lighting conditions.
  • the goal of the invention is to provide a method and apparatus free from the limitations and issues found in the state of the art - of which some are described above - and especially to provide a method and apparatus enabling three-dimension reconstruction on mobile devices.
  • the goal of the invention is to provide a three-dimensional reconstruction method providing comparable of better quality results given less resources compared to known methods.
  • a mobile device is understood to mean any hand-held programmable computer capable of running applications, having a screen and input peripheries (typically touchscreens or miniature keyboards), typically under one kilogram.
  • Examples of devices having at least one camera device are smartphones, tablets, some personal digital assistants, and some digital (video) cameras, etc.
  • the task of reconstruction is performed according to the method presented in claim 1 . on mobile devices as described in claim 5.
  • Additional subject of the invention is the mobile device of claim 15., utilized for the execution of the method of invention.
  • Figure 1 of the drawing presents the schematic block diagram of the main components of a mobile device of the invention participating in the method of the invention.
  • FIG. 1 The main units participating in the method of invention, and their main components are shown on figure 1 . Units or modules in the context of the invention are understood to be either physical or logical in nature.
  • Figure 1 . of the drawing presents the components of a mobile device 10 of the invention participating in the execution of the method of the invention.
  • Mobile device 10 can be, depending on the embodiment, be a smartphone, a tablet, a PDA, digital camera or video recorder, etc..
  • the mobile device 10 contains at least one camera 1 1.
  • the mobile device 10 contains at least one more 29 sensor, and in preferred embodiments, at least one gyroscope and/or accelerometer.
  • sensor 29 is understood to an apparatus distinct from camera 1 1 , used to, or capable of assisting with the determination of the position of the mobile device 10, such as gyroscope, accelerometer, magnetometer, GPS receiver, or other (such as Bluetooth signal based) localization apparatus, etc.
  • the mobile device 10 may contain multiple 29 sensors, in which case may have multiple sensor data integrator components, meant to fuse the data of the individual sensors.
  • mobile device 10 we define a logical image processing pipeline 12 beginning with the 1 1 camera, consisting of subsequently described, interconnected, data exchanging, functional components.
  • the camera 1 1 is followed the image stream source module, which constructs an image stream from the images recorded by the camera 1 1 .
  • the image stream source module 13 may contain the software components of the mostly hardware defined camera 1 1.
  • the image stream source 13 module is followed by one or more preprocessor 14 modules in the image processing pipeline 12. Multiple preprocessor modules 14 are preferably connected in series. In some embodiments, the preprocessor modules 14 may be connected parallel, in order to divide the image processing pipeline 12 into multiple branches 12a. In some embodiments, one or more preprocessor 14 modules may be elided, or integrated into other modules. Following these, the image processing pipeline 12 contains one - or in the case of multiple branches 12a - multiple feature tracker control units.
  • the individual feature tracker control units 16 contain at least two feature tracker 18 modules, utilizing differing feature description models on the image stream, and quality estimator modules 20, which estimate the expected efficiency of the individual feature tracker 18 modules. At least one classifier module 22 belongs to each quality estimator module 20. The classifier modules 22 can be connected to more than one quality estimator modules 20, as shown on figure 1 .
  • the individual feature tracker control units in preferred embodiments, contain at least as many feature tracker modules 18 as many environments are distinguishable (indoor room, forest, shore, etc.) by the classifier modules 22, as it shall be described later on.
  • the image processing pipeline 12 contains, after the feature tracker control units each, a reconstructor 24 module, which, utilizing the tracked points supplied by the enabled feature tracker 18 modules, computes the projective space coordinates of the tracked feature points, and reconstructs the projective space camera angles for the images containing the tracked feature points.
  • the image processing pipeline 12 contains at least one inertial measurement module 26, meant to estimate the temporal real-space trajectory of at least one camera 1 1 , utilizing data from at least one sensor 29 of the mobile device 10.
  • one or more camera trajectories may be more accurately estimated with the inertial measurement module 26 utilizing the projective space camera angle data, as produced by and transferred to the inertial measurement module 26 by the reconstructor 24 module, and the reconstructor 24 module may utilize the estimate of the trajectory as produced and provided by the inertial measurement module 26 to assist with reconstruction.
  • the sensor data integrator 28 module provides the fused sensor data to the inertial measurement module 26.
  • the image processing pipeline 12 contains a registrator 30, which correlates the projective space coordinates and the real space coordinates of the camera devices 1 1 , which it utilizes compute the real-space coordinates corresponding to the projective space coordinates of the feature points.
  • the registrator 30 module is directly connected to each reconstructor 24 module.
  • step 101 the images captured by at least one camera 1 1 of the mobile device 10 are processed into an image stream by image stream source 13 module connected to the mobile device 10.
  • the mobile device 10 produces images continuously and in real-time, with arbitrary delay and frequency.
  • step 102 the image stream produced by module 13 is forwarded to the preprocessor module 14.
  • the one or more preprocessing components 14 perform a variety of preprocessing operations on the output stream of an image stream source 14, for the purpose of later presented processes of the method of the invention.
  • we classify these preprocessing operations two categories. Examples of operations belonging to the first category are as follows: alteration of the sharpness of the image
  • the preprocessor module 14 subdivides the original image into regions as deemed relevant from an image processing standpoint of subsequent components, and forwards the relevant subdivided image sections to the next appropriate connected component in the pipeline.
  • the last preprocessing module 14 in the image processing pipeline 12 (or in one of its branches 12a) forwards the preprocessed image stream to the feature tracker control unit 16, which, in accordance with earlier statements, contains at least two feature tracker 18 modules.
  • the feature tracker module 18 searches the images of the input image stream for characteristic areas (pixels or pixel regions), that is, feature points, which are well separable from other parts of the image, and their detection is not too sensitive to viewpoint or lighting condition changes.
  • the selection of the feature points can be performed with scale and rotation invariant local feature detectors, tolerant to lighting and viewpoint changes (such as MSER, SIFT, SURF, ORB, etc. as obvious to a person skilled in the art).
  • the feature points can be determined in other, known ways, such as via convolutional image filters detecting the characteristic points (edges, corners, etc.), segmentation of locally similar structures on the image, or by performing local waveform (wavelet) transformations, etc.
  • the feature tracker 18 module associates the features points with well-known feature descriptors, based on the information of the nearby pixels or pixel areas.
  • the feature tracker 18 module subsequently utilizes one or more metric functions to compare the feature descriptors associated with the feature points on subsequences of images of the image stream with each other, thereby tracking a feature point associated with a given feature descriptor on the image stream.
  • a feature point is tracked, on not necessarily consecutive images, of the image stream if the feature point has been identified in at least three images (but naturally, larger repetition counts may be specified).
  • the feature tracker 18 module will keep searching for the feature point based on the corresponding feature descriptor on subsequent images.
  • the output of the feature tracker 18 module is a two-dimensional feature point data stream of tracked points, which isn't part of the image stream processed by module 14.
  • at least two types of feature tracker 18 modules are provided.
  • the difference between the utilizable feature tracker modules lies in their differing efficiency in differing environments. As an example, one 18 module excellent at recognizing real-world corners and edges, would be primarily efficient in indoor areas, such as rooms, offices, buildings.
  • databases consisting of image sequences containing the fundamental properties of differing environments (rooms, cities, forests), are utilized to evaluate feature tracker 18 modules using optionally automated classifiers.
  • the evaluation can be done with human interaction, such as by checking whether the feature tracker module 18 has tracked the feature points sufficiently well on each image of the annotated image database.
  • the ratio of potential errors is used to determine the efficiency of the feature tracker 18 module in the given environment.
  • well-known machine learning algorithms and tools such as Trainable Weka Segmentation
  • the performance data gathered above is used to configure the classifier module 22 so as to deem image sequences upon which a connected feature tracker 18 module performs well to belonging to the given feature tracker 18 module, while the ones on which it does not are deemed neutral, or belonging to other feature tracker 18 modules (in the case multiple feature tracker 18 modules 18 are utilized during the training of the classifier module 22).
  • the feature tracker control unit 16 determines which of the feature tracker modules 18 should be online at a given time.
  • the feature tracker control unit 16 forwards the input image stream towards the individual quality estimator 20 modules, corresponding to each feature tracker 18 module, which, with the aid of the data provided by one or more connected classifier modules 22, estimate the expected efficiency of the feature tracker 18 module on the given image stream, without actually executing said feature tracker 18 module on the image stream.
  • the efficiency ( and performance ) of a feature tracker module 18 is determined by the amount of feature points it is capable of reliably tracking, and through how many images.
  • the classifier module 22 is capable swiftly estimating the efficiency of one or more feature tracker 18 module's for which it has been previously trained, with little resource utilization, sparing the device of the execution of lower quality, high resource intensity feature tracker modules 18.
  • the quality estimator 20 module processes output of one or more corresponding classifiers 22, storing the result of the their performed mathematical operations, to be used as the basis of a meaningful estimation to be sent, with regards to the expected efficiency of its corresponding feature tracker 18 module's performance, to the feature tracker control unit 16.
  • the quality estimator 20 provides an output in the range of 0 to 1 , where 0 stands for least efficient, and 1 stands for maximally efficient feature tracker 18 module, but other representations and schemes are obviously possible for expected efficiency.
  • the quality estimator 20 module passes the determined values to the feature tracker control unit 16, which, based on the expected efficiency and available resources, instructs the individual quality estimator 20 modules, and through it, to the corresponding feature tracker 18 module.
  • the instruction can be of three types as follows: starting of a feature tracker module 18, that is, enabling the execution thereof stopping of a feature tracker module 18, that is, disabling the execution thereof preserving the state of a feature tracker module 18, that is, leaving previously disabled ones offline, and executing ones online.
  • the feature tracker control unit 16 limits the amount of images reaching the quality estimator 20 modules based on the overall device load, and the performance of the quality estimator 20 modules.
  • the resource requirements of the method of the invention can be significantly reduced by disabling the feature tracker 18 module or modules measured to be presently under performing - being inefficient - on a particular image stream, while enabling only the most efficient ones can still ensure good tracking quality; which can be significant in cases when the method is applied on a mobile device 10 of limited resources, such as a smartphone.
  • the ill-performing feature tracker modules 18 can be enabled, which brings potential improvements to the three-dimensional reconstruction, while in the case of resource scarcity, it is advisable to only enable the feature tracker modules 18 with the greatest potential (in some cases, only one feature tracker module 18 might be enabled), in order to ensure the best three-dimensional reconstruction achievable given the available resources.
  • the feature tracking control unit 16 sends the data provided by the enabled one or more feature tracker 18 modules to one or more reconstructor 24 module.
  • the reconstructor 24 module utilizes known SFM (Structure From Motion) and similar algorithms to, based on the arriving data describing the tracked feature points, reconstruct the projective space coordinates of the tracked feature points, and the projective camera angles of the images containing said feature points.
  • SFM Structure From Motion
  • the degrees of freedom can be limited, resulting in faster and more precise camera angles, affecting the feature points three-dimensional position as well.
  • This limitation is achieved by utilization of appropriate real-space position estimates, produced by 26, provided they are available, and deemed sufficiently accurate.
  • Camera angles are understood to be the spatial orientation and position of cameras 1 1 in time points where images bearing tracked feature points are captured. As a result, a data stream consisting of three-dimensional points in projective space is produced.
  • a reconstructor module passes at least data regarding the projective camera angles to the inertial measurement module 26.
  • multiple sensors 29 are connected to the inertial measurement module 26 through the sensor data integrator 28 module.
  • the sensor data integrator 28 module produces a data stream consisting of the fused streams of individual sensors 29 (such as accelerometer, gyroscope, magnetometer, etc.), which it creates accounting for the differences in quality and speed of these streams, and the temporal resolution of the sensors 29.
  • individual sensors 29 such as accelerometer, gyroscope, magnetometer, etc.
  • the data stream of that sensor 29 is interpolated in accordance with the resolution and sampling of the fused data stream, taking the interpolated value closest in time.
  • the sampling rates of the individual sensors are reconfigured to arrive to a minimized summed interpolation error.
  • step 107 the fused sensor data of module 28 is sent to the inertial measurement module 26.
  • the inertial measurement module 26 calculates, from the fused sensor data, and the projective space description of the camera angles as received from the reconstructor 24 module, the real-space (metric) trajectory of at least one camera 1 1 .
  • the real-space metric trajectory In order to calculate the real-space metric trajectory the projective space camera angle descriptions are not strictly necessary, but this information can be utilized to refine it.
  • the registrator module 30 compares projective camera angles received from or more reconstructor modules 24 with the real-space trajectories received from the inertial measurement module 26, and correlates them with each other, that is, computes the real-space camera angles corresponding to the projective camera angle for at least one camera 1 1 , and it subsequently utilizes this to compute, by using projective coordinates of the feature points, the real-space coordinates of the feature points, and by this constructing the three-dimensional metric representation of the area real-space area recorded by the camera 1 1 .
  • the advantage of utilizing the sensors 29 is that the tears in the image stream (occlusions, fast camera movement, processing slowdowns or delays) can be bypassed or compensated for, that is, the feature points lost because of these effects can be recovered, and thus remain trackable.
  • the utilization of the sensors 29 allows the computation of real-space three-dimensional coordinates of the points extracted from the image stream, which can be utilized for multiple goals, such as determining the floor plan of a room, or determining the dimensions, or volume of objects.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne une reconstruction 3D à l'aide d'un dispositif mobile. Un procédé de reconstruction tridimensionnelle sur un dispositif mobile (10) comprenant au moins une caméra (10), caractérisé par la production d'un flux d'images à partir d'images capturées par au moins une caméra (11) à l'aide d'un module de diffusion en continu d'images (13),-fournir au moins deux modules de suivi de caractéristiques (18) qui utilisent différents descripteurs d'images pour le flux d'images,-estimer l'efficacité attendue des modules de suivi de caractéristiques (18) pour le flux d'images à l'aide de modules d'estimation de qualité (20) correspondant aux modules de poursuite de caractéristiques (18)-selon l'efficacité attendue des modules de poursuite de caractéristiques (18), et les ressources disponibles, permettant ou désactivant les modules individuels de suivi de caractéristiques (18) à l'aide d'une unité de commande (16). L'invention concerne en outre un dispositif mobile (10) comprenant au moins une caméra (11), un affichage, une unité de traitement et au moins une mémoire contenant un programme informatique pour l'exécution du procédé.
PCT/HU2016/050028 2016-06-27 2016-06-27 Procédé de reconstruction 3d à l'aide d'un dispositif mobile Ceased WO2018002677A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/HU2016/050028 WO2018002677A1 (fr) 2016-06-27 2016-06-27 Procédé de reconstruction 3d à l'aide d'un dispositif mobile

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/HU2016/050028 WO2018002677A1 (fr) 2016-06-27 2016-06-27 Procédé de reconstruction 3d à l'aide d'un dispositif mobile

Publications (1)

Publication Number Publication Date
WO2018002677A1 true WO2018002677A1 (fr) 2018-01-04

Family

ID=60787204

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/HU2016/050028 Ceased WO2018002677A1 (fr) 2016-06-27 2016-06-27 Procédé de reconstruction 3d à l'aide d'un dispositif mobile

Country Status (1)

Country Link
WO (1) WO2018002677A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109008909A (zh) * 2018-07-13 2018-12-18 宜宾学院 一种低功耗胶囊内窥镜图像采集及三维重建系统
CN113887319A (zh) * 2021-09-08 2022-01-04 北京达佳互联信息技术有限公司 三维姿态的确定方法、装置、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080151101A1 (en) * 2006-04-04 2008-06-26 Qualcomm Incorporated Preprocessor method and apparatus
EP1988717A1 (fr) * 2007-04-30 2008-11-05 ViXS Systems Inc. Système pour combiner une pluralité de flux vidéo et son procédé d'utilisation
US20140037189A1 (en) * 2012-08-02 2014-02-06 Qualcomm Incorporated Fast 3-D point cloud generation on mobile devices
WO2015125025A2 (fr) * 2014-02-10 2015-08-27 Geenee Ug Systèmes et procédés de reconnaissance basée sur des caractéristiques d'image
US20150279083A1 (en) * 2014-03-26 2015-10-01 Microsoft Corporation Real-time three-dimensional reconstruction of a scene from a single camera

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080151101A1 (en) * 2006-04-04 2008-06-26 Qualcomm Incorporated Preprocessor method and apparatus
EP1988717A1 (fr) * 2007-04-30 2008-11-05 ViXS Systems Inc. Système pour combiner une pluralité de flux vidéo et son procédé d'utilisation
US20140037189A1 (en) * 2012-08-02 2014-02-06 Qualcomm Incorporated Fast 3-D point cloud generation on mobile devices
WO2015125025A2 (fr) * 2014-02-10 2015-08-27 Geenee Ug Systèmes et procédés de reconnaissance basée sur des caractéristiques d'image
US20150279083A1 (en) * 2014-03-26 2015-10-01 Microsoft Corporation Real-time three-dimensional reconstruction of a scene from a single camera

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109008909A (zh) * 2018-07-13 2018-12-18 宜宾学院 一种低功耗胶囊内窥镜图像采集及三维重建系统
CN109008909B (zh) * 2018-07-13 2024-01-26 宜宾学院 一种低功耗胶囊内窥镜图像采集及三维重建系统
CN113887319A (zh) * 2021-09-08 2022-01-04 北京达佳互联信息技术有限公司 三维姿态的确定方法、装置、电子设备及存储介质

Similar Documents

Publication Publication Date Title
CN110706259B (zh) 一种基于空间约束的可疑人员跨镜头追踪方法及装置
US10948297B2 (en) Simultaneous location and mapping (SLAM) using dual event cameras
US9098740B2 (en) Apparatus, method, and medium detecting object pose
Rambach et al. Learning to fuse: A deep learning approach to visual-inertial camera pose estimation
JP5722502B2 (ja) モバイルデバイスのための平面マッピングおよびトラッキング
EP3001384B1 (fr) Appareil de calcul de coordonnées tridimensionnelles, procédé de calcul de coordonnées tridimensionnelles et programme de calcul de coordonnées tridimensionnelles
CN109544615B (zh) 基于图像的重定位方法、装置、终端及存储介质
Li et al. RD-VIO: Robust visual-inertial odometry for mobile augmented reality in dynamic environments
US20210274358A1 (en) Method, apparatus and computer program for performing three dimensional radio model construction
CN112861808B (zh) 动态手势识别方法、装置、计算机设备及可读存储介质
CN106352877A (zh) 一种移动装置及其定位方法
CN109559330A (zh) 运动目标的视觉跟踪方法、装置、电子设备及存储介质
CN113256683B (zh) 目标跟踪方法及相关设备
US9990857B2 (en) Method and system for visual pedometry
JP2021060868A5 (fr)
US20220300774A1 (en) Methods, apparatuses, devices and storage media for detecting correlated objects involved in image
JP6662382B2 (ja) 情報処理装置および方法、並びにプログラム
CN109712188A (zh) 一种目标跟踪方法及装置
CN118675106A (zh) 一种基于机器视觉的落石实时监测方法、系统、装置及存储介质
KR20250087618A (ko) 이미지 기반 속도 결정 방법, 디바이스, 기기 및 저장 매체
US11373318B1 (en) Impact detection
WO2018002677A1 (fr) Procédé de reconstruction 3d à l'aide d'un dispositif mobile
CN110969138A (zh) 人体姿态估计方法及设备
CN115752489B (zh) 可移动设备的定位方法、装置和电子设备
CN110706202B (zh) 一种非典型目标探测方法、装置和计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16907187

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 12/04/2019)

122 Ep: pct application non-entry in european phase

Ref document number: 16907187

Country of ref document: EP

Kind code of ref document: A1