[go: up one dir, main page]

WO2017209886A2 - Méthode hybride efficace relative à un mouvement propre à partir de vidéos capturées à l'aide d'une caméra aérienne - Google Patents

Méthode hybride efficace relative à un mouvement propre à partir de vidéos capturées à l'aide d'une caméra aérienne Download PDF

Info

Publication number
WO2017209886A2
WO2017209886A2 PCT/US2017/030673 US2017030673W WO2017209886A2 WO 2017209886 A2 WO2017209886 A2 WO 2017209886A2 US 2017030673 W US2017030673 W US 2017030673W WO 2017209886 A2 WO2017209886 A2 WO 2017209886A2
Authority
WO
WIPO (PCT)
Prior art keywords
denotes
image frame
ego
camera
projection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2017/030673
Other languages
English (en)
Other versions
WO2017209886A3 (fr
Inventor
Yongqiang CAO
Narayan Srinivasa
Shankar R. RAO
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
HRL Laboratories LLC
Original Assignee
HRL Laboratories LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/250,665 external-priority patent/US10078902B1/en
Application filed by HRL Laboratories LLC filed Critical HRL Laboratories LLC
Priority to EP17807179.1A priority Critical patent/EP3453168B1/fr
Priority to CN201780009925.4A priority patent/CN108605113B/zh
Publication of WO2017209886A2 publication Critical patent/WO2017209886A2/fr
Publication of WO2017209886A3 publication Critical patent/WO2017209886A3/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/527—Global motion vector estimation

Definitions

  • the present invention relates to a system for ego-motion compensation
  • the present invention relates to a system ego-motion compensation during video processing and, more particularly, to a system for ego-motion
  • the system comprises one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such mat when executed, the one or more processors perform multiple operations.
  • the system generates
  • An optimal estimation of camera ego-motion is generated using the initial estimate as an input to one of a valley search method and an alternate line search method. Finally, independent moving objects are detected in the scene.
  • the detected independent moving objects are identified and tracked.
  • an estimate (h1, ⁇ 1) of a Last frame pair in the video is generated, and the optimal estimation (h*,v*) of camera ego-motion is generated using bom the initial estimate (h ⁇ , ⁇ ) and (h l,vl) as input to the valley search method.
  • an X projection and a Y projection are determined for each of consecutive image frame pairs 71 and 12.
  • An ego-translation (h,v) of the camera is estimated by determining an optimal projection correlation of the consecutive image frame pairs 11 and 12 with image frame 12 shifted by (h, ⁇ ) according to the following:
  • min denotes a minimization function
  • h denotes horizontal
  • v denotes vertical
  • m denotes a width of an image frame
  • i and j are index variables
  • denotes an X projection for image frame denotes an projection for image frame I2 shifted by A, denotes a
  • y projection for image frame denotes a Y projection for image frame 12
  • the moving platform is selected from a group consisting of an airplane, a helicopter, a satellite, and an unmanned aerial vehicle (UAV).
  • UAV unmanned aerial vehicle
  • die present invention also includes a computer program product and a computer implemented method
  • the computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors, such that upon execution of the instructions, the one or more processors perform the operations listed herein.
  • the computer implemented method includes an act of causing a computer to execute such instructions and perform the resulting operations.
  • FIG. 1 is a block diagram depicting the components of a system for ego- motion compensation during video processing according to some embodiments of the present disclosure
  • FIG.2 is an illustration of a computer program product according to some embodiments of the present disclosure
  • FIG.3 is an illustration of X and Y projections of a projected correlation method according to prior art
  • FIG.4 is a flow diagram illustrating the hybrid valley search method
  • FIG.5 is a flow diagram illustrating the hybrid alternate line search method according to some embodiments of the present disclosure.
  • FIG.6 is a table illustrating a comparison of ego-motion estimation on helicopter sequences according to some embodiments of the present disclosure.
  • the present invention relates to a system for ego-motion compensation during video processing and, more particularly, to a system for ego-motion compensation during video processing which is accurate and computationally efficient
  • the following description is presented to enable one of ordinary skill in the art to make and use me invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications will be readily apparent to those skilled in die art, and the general principles defined herein may be applied to a wide range of aspects. Thus, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
  • any element in a claim that does not explicitly state "means for” performing a specified function, or “step for” performing a specific function, is not to be interpreted as a "means” or “step” clause as specified in 35 U.S.C. Section 112, Paragraph 6.
  • the use of "step of or “act of in the claims herein is not intended to invoke the provisions of 35 U.S.C. 112, Paragraph 6.
  • Neovision2 annotated video datasets. Taken on April 12, 2017 from htt p : // ilab.usc.edu/neo2/dataset/. 5.
  • B.D. Lucas and T. Kanade "An iterative image registration technique with an application to stereo vision," in Proc. 7* International Joint Conference on Artificial Intelligence (LICAI), vol. 81, pp.674-679, 1981.
  • Various embodiments of die invention include three "principal" aspects.
  • the first is a system ego-motion compensation during video processing.
  • the system is typically in die form of a computer system operating software or in the form of a "hard-coded" instruction set. This system may be incorporated into a wide variety of devices mat provide different functionalities.
  • the second principal aspect is a method, typically in the form of software, operated using a data processing system (computer).
  • the third principal aspect is a computer program producL
  • the computer program product generally represents computer-readable instructions stored on a non-transitory computer-readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape.
  • FIG. 1 A block diagram depicting an example of a system (i.e., computer system 100) of die present invention is provided in FIG. 1.
  • the computer system 100 is configured to perform calculations, processes, operations, and/or functions associated with a program or algorithm.
  • certain processes and steps discussed herein are realized as a series of instructions (e.g., software program) mat reside within computer readable memory units and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behavior, such as described herein.
  • the computer system 100 may include an address/data bus 102 mat is
  • processor 104 configured to communicate information. Additionally, one or more data processing units, such as a processor 104 (or processors), are coupled with the address/data bus 102.
  • the processor 104 is configured to process information and instructions.
  • the processor 104 is a microprocessor.
  • the processor 104 may be a different type of processor such as a parallel processor, application-specific integrated circuit (ASIC), programmable logic array (PLA), complex programmable logic device (CPLD), or a field programmable gate array (FPGA).
  • ASIC application-specific integrated circuit
  • PLA programmable logic array
  • CPLD complex programmable logic device
  • FPGA field programmable gate array
  • the computer system 100 is configured to utilize one or more data storage units.
  • the computer system 100 may include a volatile memory unit 106 (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled with the address/data bus 102, wherein a volatile memory unit 106 is configured to store information and instructions for the processor 104.
  • RAM random access memory
  • static RAM static RAM
  • dynamic RAM dynamic RAM
  • the computer system 100 further may include a non-volatile memory unit 108 (e.g., read-only memory (“ROM”), programmable ROM (“PROM”), erasable programmable ROM (“EPROM”), electrically erasable prograrnmable ROM “EEPROM”), flash memory, etc.) coupled with the address/data bus 102, wherein the non- volatile memory unit 108 is configured to store static information and instructions for the processor 104.
  • the computer system 100 may execute instructions retrieved from an online data storage unit such as in "Cloud” computing.
  • the computer system 100 also may include one or more interfaces, such as an interface 110, coupled with the address/data bus 102.
  • the one or more interfaces are configured to enable the computer system 100 to interface with other electronic devices and computer systems.
  • the communication interfaces implemented by the one or more interfaces may include wireline (e.g., serial cables, modems, network adaptors, etc.) and/or wireless (e.g., wireless modems, wireless network adaptors, etc.) communication technology.
  • the computer system 100 may include an input device 112 coupled with the address/data bus 102, wherein the input device 112 is configured to communicate information and command selections to the processor 100.
  • the input device 112 is an alphanumeric input device, such as a keyboard, mat may include alphanumeric and/or function keys.
  • the input device 112 may be an input device other than an alphanumeric input device.
  • the computer system 100 may include a cursor control device 114 coupled with the address/data bus 102, wherein the cursor control device 114 is configured to communicate user input information and/or command selections to the processor 100. m an aspect, the cursor control device 114 is imp
  • the cursor control device 114 is directed and/or activated via input from the input device 112, such as in response to the use of special keys and key sequence commands associated with the input device 112.
  • the cursor control device 114 is configured to be directed or guided by voice commands.
  • the computer system 100 further may include one or more optional computer usable data storage devices, such as a storage device 116, coupled with the address/data bus 102.
  • the storage device 116 is configured to store information and/or computer executable instructions.
  • the storage device 116 is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive (“HDD”), floppy diskette, compact disk read only memory (“CD-ROM”), digital versatile disk (“DVD”)).
  • a display device 118 is coupled with me address/data bus 102, wherein the display device 118 is configured to display video and/or graphics.
  • the display device 118 may include a cathode ray tube (“CRT”), liquid crystal display (“LCD”), field emission display (“FED”), plasma display, or any other display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user.
  • CTR cathode ray tube
  • LCD liquid crystal display
  • FED field emission display
  • plasma display or any other display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user.
  • the computer system 100 presented herein is an example computing
  • the non-limiting example of the computer system 100 is not strictly limited to being a computer system.
  • the computer system 100 represents a type of data processing analysis that may be used in accordance with various aspects described herein.
  • other computing systems may also be
  • one or more operations of various aspects of the present technology are controlled or implemented using computer-executable instructions, such as program modules, being executed by a computer.
  • program modules include routines, programs, objects, components and/or data structures that are configured to perform particular tasks or implement particular abstract data types.
  • an aspect provides that one or more aspects of the present technology are implemented by utilizing one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer-storage media including memory-storage devices.
  • FIG.2 An illustrative diagram of a computer program product (Le., storage device) embodying the present invention is depicted in FIG.2.
  • the computer program product is depicted as floppy disk 200 or an optical disk 202 such as a CD or DVD.
  • the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium.
  • the term "instructions” as used with respect to this invention generally indicates a set of operations to be performed on a computer, and may represent pieces of a whole program or individual, separable, software modules.
  • Non-limiting examples of "instruction” include computer program code (source or object code) and "hard-coded" electronics (i.e. computer operations coded into a computer chip).
  • the "instruction" is stored on any non-transitory computer-readable medium, such as in the memory of a computer or on a floppy disk, a CD-ROM, and a flash drive. In either event, the instructions are encoded on a non-transitory computer-readable medium.
  • the hybrid method first uses a projected correlation method to get an initial rough estimation of the camera ego-motion, then starts using a valley search method to get a final accurate estimation of the ego-motion. As described below, the hybrid method has superior performance over existing technologies while saving computational cost. [00048] (3.1) Projected Correlation Method
  • FIG.3 illustrates an example image 300, a plot conesponding to an X projection 302 of the image, and a plot conesponding to a Y projection 304.
  • 11 and 12 be two consecutive frames in a video taken by a moving camera.
  • the projected correlation method estimates die camera ego-translation (h, v) by finding the best projection correlation of11 and 12, with image 12 shifted by (h, v).
  • the method described herein can be defined by finding the optimal (h, v) that minimizes the differences of the projections between 11 and 12, with image 12 shifted by (h, v):
  • min denotes a minimisation function
  • A denotes horizontal
  • v denotes vertical
  • n denotes a height of an image frame
  • m denotes a width of an image frame
  • j are index variables
  • denotes an projection for image frame denotes an X projection for image frame 12 shifted by h
  • Y projection for image frame denotes a Y projection for image frame 12
  • search method performs a line search along a valley direction where local minima are often distributed. This can be implemented by first locating two local minima, and then search along the direction defined by the two local minima. The two local minima can be found using an optimization method (e.g., the alternate line search method) by starting from two initial points that are sufficiently far from each other. More precisely, me algorithm is described as follows.
  • Application No.62/330,462 starts its search from two initial guesses.
  • the first initial guess is die estimate of die last image frame pair in a video ((0,0) for the first frame pair).
  • the second initial guess is an offset of the first initial guess.
  • one of die initial guesses is the estimate from die projected correlation 400 of two consecutive frames in a video taken by a moving camera (elements 402 and 404), while the other initial guess is still the estimate of die last frame pair in a video ((0,0) for the first frame pair).
  • (h ⁇ , ⁇ ) 406 is the shift estimate from the projected correlation method (element 400).
  • (hl.vl) 408 is the ego-motion estimate of the last frame pair in a video ((0,0) for the first frame pair). For the first frame of the video (h1, ⁇ 1) is initialized to (0,0).
  • (h*,v*) 410 is the final optimal estimate.
  • the alternate line search method minimizes a cost function that is based on the difference between consecutive image frames. As a result, no expensive feature matching and optical flow computations are needed.
  • the method is so fast and efficient that normally it takes no more than two search steps to reach the global minimum, In total, it takes around only 8 frame- diflerence operations to find the correct ego-translation in a video frame.
  • the function f(h, v) is usually a convex function with a global minimum or, in worse cases, a quasi-convex function with a deep global minimum and some shallow local minima. While a gradient descent method can be used here to find the minimum of a convex function, it is susceptible to a zig- zagging problem during search, thus, requiring many steps to reach the minimum To address mis, a method was developed to find the minimum of function/(fc, v). On average, it takes no more than three steps to find the global minimum from an arbitrary initial point [00061]
  • the alternate line search algorithm consists of alternative one directional search (horizontal or vertical).
  • V 0 : v 1 .
  • the hybrid alternate line search method first uses the projected correlation method (element 400) to get an initial estimate of camera ego-translation for two consecutive image frames (elements 402 and 404), then uses the alternate line search method (element 500) to get the final optimal estimate (h*,v*) 410.
  • (h ⁇ , ⁇ ) 406 is the shift estimate from the projected correlation method.
  • the methods according to embodiments of the present disclosure were also compared with other ego-motion methods, such as the Lucas-Kanade method (see Literature References Nos. 6 and 7) and the Lucas-Kanade method with Gaussian Pyramid (see Literature Reference Nos. 8 and 9).
  • the table in FIG.6 depicts a detailed comparison of the various methods, where the error rate in number of shifts is the number of estimate errors in horizontal or vertical shifts (translations) divided by the total number of horizontal and vertical shifts for all frames. Note that there are two shifts (one horizontal and one vertical) for each frame pair. If the estimate is deviated from ground truth over one pixel, it is considered mat an estimate error occurred.
  • the error in average distance to ground truth is the sum of deviates in distance from ground truth for all frames divided by the total number of shifts.
  • the table in FIG.6 shows that the hybrid valley search method according to the embodiments of this disclosure has the best accuracy with the hybrid alternate line search method next to it

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)
  • Studio Devices (AREA)

Abstract

L'invention concerne un système permettant de compenser un mouvement propre au cours d'un traitement vidéo. Le système génère une estimation initiale du mouvement propre d'une caméra mobile pour des paires d'images consécutives d'une vidéo d'une scène par une méthode de corrélation projetée, la caméra étant conçue pour capturer la vidéo à partir d'une plateforme mobile. Une estimation optimale du mouvement propre de la caméra est générée en utilisant l'estimation initiale en tant que donnée d'entrée d'une méthode de recherche de vallées ou d'une méthode de recherche linéaire alternative. Tous les objets mobiles indépendants sont détectés dans la scène à l'aide de la méthode hybride décrite avec de meilleurs résultats par rapport aux méthodes existantes et pour un coût de calcul réduit.
PCT/US2017/030673 2016-05-02 2017-05-02 Méthode hybride efficace relative à un mouvement propre à partir de vidéos capturées à l'aide d'une caméra aérienne Ceased WO2017209886A2 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP17807179.1A EP3453168B1 (fr) 2016-05-02 2017-05-02 Méthode hybride efficace relative à un mouvement propre à partir de vidéos capturées à l'aide d'une caméra aérienne
CN201780009925.4A CN108605113B (zh) 2016-05-02 2017-05-02 用于自运动补偿的方法、系统和非暂时性计算机可读介质

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201662330518P 2016-05-02 2016-05-02
US201662330462P 2016-05-02 2016-05-02
US62/330,462 2016-05-02
US62/330,518 2016-05-02
US15/250,665 US10078902B1 (en) 2015-08-27 2016-08-29 Fast robust method for compensating ego-translations and detecting independent moving objects in video captured with a moving camera
US15/250,665 2016-08-29

Publications (2)

Publication Number Publication Date
WO2017209886A2 true WO2017209886A2 (fr) 2017-12-07
WO2017209886A3 WO2017209886A3 (fr) 2018-02-22

Family

ID=60478873

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2017/030673 Ceased WO2017209886A2 (fr) 2016-05-02 2017-05-02 Méthode hybride efficace relative à un mouvement propre à partir de vidéos capturées à l'aide d'une caméra aérienne

Country Status (1)

Country Link
WO (1) WO2017209886A2 (fr)

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6307959B1 (en) * 1999-07-14 2001-10-23 Sarnoff Corporation Method and apparatus for estimating scene structure and ego-motion from multiple images of a scene using correlation
EP1257971A4 (fr) * 1999-11-26 2005-07-06 Mobileye Inc Systeme et procede d'estimation de l'auto-deplacement d'un vehicule en mouvement au moyen d'images successives enregistrees le long de la trajectoire de deplacement du vehicule
TWI425445B (zh) * 2009-09-29 2014-02-01 Nat Univ Tsing Hua 用於判斷一移動平台之自我運動量的方法以及偵測系統
US8903127B2 (en) * 2011-09-16 2014-12-02 Harman International (China) Holdings Co., Ltd. Egomotion estimation system and method
EP2730888A1 (fr) * 2012-11-07 2014-05-14 Ecole Polytechnique Federale de Lausanne EPFL-SRI Procédé pour déterminer une direction et amplitude d'une estimation de vitesse de courant d'un dispositif mobile

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
C.R. DEL BLANCE, F. JAUREGUIZAR, L. SALGADO, AND N. GARCIA: "estimation through efficient matching of a reduced number of reliable singular points", PROC. SPIE ELECTRONIC IMAGING, vol. 6811, 2008, pages 1 - 12
H. BADINOT. KANADE: "A Head-Wearable Short-Baseline Stereo System for the Simultaneous Estimation of Structure and Motion", PROC. 12TH IAPR CONFERENCE ON MACHINE VISION APPLICATIONS (MVA, June 2011 (2011-06-01)
L ITTI, NEOVISION2 ANNOTATED VIDEO DATASETS, 12 April 2017 (2017-04-12), Retrieved from the Internet <URL:http://ilab.use.edu/rneo2/dataset>
See also references of EP3453168A4
Y, CHENGM, MAMONEL. MATTHIES: "Visual odometry on the Mars exploration rovers", IEEE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN AND CYBERNETICS, vol. 1, 2005, pages 903 - 910

Also Published As

Publication number Publication date
WO2017209886A3 (fr) 2018-02-22

Similar Documents

Publication Publication Date Title
EP3698323B1 (fr) Profondeur à partir d&#39;un mouvement pour une réalité augmentée pour des dispositifs utilisateurs portatifs
US9933264B2 (en) System and method for achieving fast and reliable time-to-contact estimation using vision and range sensor data for autonomous navigation
US11049270B2 (en) Method and apparatus for calculating depth map based on reliability
US10089549B1 (en) Valley search method for estimating ego-motion of a camera from videos
US9008366B1 (en) Bio-inspired method of ground object cueing in airborne motion imagery
CN112215773B (zh) 基于视觉显著性的局部运动去模糊方法、装置及存储介质
EP3414641B1 (fr) Système et procédé permettant d&#39;obtenir une estimation de temps de contact rapide et fiable à l&#39;aide de données de capteur de vision et de distance pour une navigation autonome
Molloy et al. Detection of aircraft below the horizon for vision‐based detect and avoid in unmanned aircraft systems
Warren et al. Large scale monocular vision-only mapping from a fixed-wing sUAS
US20170262992A1 (en) Image analysis system and method
BR102018076778A2 (pt) Remoção de fundo desordenado de conjunto de imagens para detecção de objeto
Kim et al. Planar structures from line correspondences in a manhattan world
US10163220B2 (en) Efficient hybrid method for ego-motion from videos captured using an aerial camera
US20250118068A1 (en) Method and system for detecting changes in areas
US10078902B1 (en) Fast robust method for compensating ego-translations and detecting independent moving objects in video captured with a moving camera
Shan et al. A brief survey of visual odometry for micro aerial vehicles
US11361548B2 (en) Method and system for multi instance visual tracking based on observer motion modelling
EP3453168B1 (fr) Méthode hybride efficace relative à un mouvement propre à partir de vidéos capturées à l&#39;aide d&#39;une caméra aérienne
Ramakrishnan et al. Adaptive window strategy for high-speed and robust KLT feature tracker
US20170316555A1 (en) System and method for ghost removal in video footage using object bounding boxes
Cai et al. Disparity and optical flow partitioning using extended Potts priors
Wan et al. Automatic moving object segmentation for freely moving cameras
DeFranco Detecting and tracking moving objects from a small unmanned air vehicle
WO2017209886A2 (fr) Méthode hybride efficace relative à un mouvement propre à partir de vidéos capturées à l&#39;aide d&#39;une caméra aérienne
Helala et al. Accelerating cost volume filtering using salient subvolumes and robust occlusion handling

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17807179

Country of ref document: EP

Kind code of ref document: A2

ENP Entry into the national phase

Ref document number: 2017807179

Country of ref document: EP

Effective date: 20181203

WWG Wipo information: grant in national office

Ref document number: 2017807179

Country of ref document: EP