[go: up one dir, main page]

WO2019152177A3 - System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering - Google Patents

System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering Download PDF

Info

Publication number
WO2019152177A3
WO2019152177A3 PCT/US2019/013513 US2019013513W WO2019152177A3 WO 2019152177 A3 WO2019152177 A3 WO 2019152177A3 US 2019013513 W US2019013513 W US 2019013513W WO 2019152177 A3 WO2019152177 A3 WO 2019152177A3
Authority
WO
WIPO (PCT)
Prior art keywords
foveated
activity
detection
classification
activity classification
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2019/013513
Other languages
French (fr)
Other versions
WO2019152177A2 (en
Inventor
Deepak Khosla
Ryan M. UHLENBROCK
Yang Chen
Huapeng SU
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
HRL Laboratories LLC
Original Assignee
HRL Laboratories LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/883,822 external-priority patent/US11055872B1/en
Priority claimed from US15/947,032 external-priority patent/US10997421B2/en
Application filed by HRL Laboratories LLC filed Critical HRL Laboratories LLC
Priority to EP19748018.9A priority Critical patent/EP3746938A4/en
Priority to CN201980006835.9A priority patent/CN111566661B/en
Publication of WO2019152177A2 publication Critical patent/WO2019152177A2/en
Publication of WO2019152177A3 publication Critical patent/WO2019152177A3/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/255—Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/08—Detecting or categorising vehicles
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)

Abstract

Described is a system for visual activity recognition. In operation, the system detects a set of objects of interest (OI) in video data and determines an object classification for each object in the set of OI, the set including at least one OI. A corresponding activity track is formed for each object in the set of OI by tracking each object across frames. Using a feature extractor, the system determines a corresponding feature in the video data for each OI, which is then used to determine a corresponding initial activity classification for each OI. One or more OI are then detected in each activity track via foveation, with the initial object detection and foveated object detection thereafter being appended into a new detected-objects list. Finally, a final classification is provided for each activity track using the new detected-objects list and filtering the initial activity classification results using contextual logic.
PCT/US2019/013513 2018-01-30 2019-01-14 System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering Ceased WO2019152177A2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19748018.9A EP3746938A4 (en) 2018-01-30 2019-01-14 SYSTEM AND METHOD FOR CLASSIFYING A NEUROMORPHIC VISUAL ACTIVITY BASED ON FOVEA DETECTION AND CONTEXT FILTERING
CN201980006835.9A CN111566661B (en) 2018-01-30 2019-01-14 Systems, methods, computer-readable media for visual activity classification

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US15/883,822 US11055872B1 (en) 2017-03-30 2018-01-30 Real-time object recognition using cascaded features, deep learning and multi-target tracking
US15/883,822 2018-01-30
US201862642959P 2018-03-14 2018-03-14
US62/642,959 2018-03-14
US15/947,032 2018-04-06
US15/947,032 US10997421B2 (en) 2017-03-30 2018-04-06 Neuromorphic system for real-time visual activity recognition

Publications (2)

Publication Number Publication Date
WO2019152177A2 WO2019152177A2 (en) 2019-08-08
WO2019152177A3 true WO2019152177A3 (en) 2019-10-10

Family

ID=67479383

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/013513 Ceased WO2019152177A2 (en) 2018-01-30 2019-01-14 System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering

Country Status (3)

Country Link
EP (1) EP3746938A4 (en)
CN (1) CN111566661B (en)
WO (1) WO2019152177A2 (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111652128B (en) * 2020-06-02 2023-09-01 浙江大华技术股份有限公司 A safety monitoring method, system and storage device for high-altitude electric power work
US12327345B2 (en) * 2021-08-13 2025-06-10 Baidu Usa Llc Vision-based safety monitoring and/or activity analysis
US12165399B2 (en) 2021-12-29 2024-12-10 Halliburton Energy Services, Inc. Systems and methods to determine an activity associated with an object of interest
US11776247B2 (en) * 2022-01-07 2023-10-03 Tomahawk Robotics Classification parallelization architecture
KR102829715B1 (en) * 2022-03-15 2025-07-04 연세대학교 산학협력단 Multiscale object detection device and method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110081043A1 (en) * 2009-10-07 2011-04-07 Sabol Bruce M Using video-based imagery for automated detection, tracking, and counting of moving objects, in particular those objects having image characteristics similar to background
US20140218468A1 (en) * 2012-04-05 2014-08-07 Augmented Vision Inc. Wide-field of view (fov) imaging devices with active foveation capability
US9230302B1 (en) * 2013-03-13 2016-01-05 Hrl Laboratories, Llc Foveated compressive sensing system
US20170132468A1 (en) * 2015-11-06 2017-05-11 The Boeing Company Systems and methods for object tracking and classification

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2008213586B2 (en) * 2007-02-08 2013-07-11 Motorola Solutions, Inc. Behavioral recognition system
US9008366B1 (en) 2012-01-23 2015-04-14 Hrl Laboratories, Llc Bio-inspired method of ground object cueing in airborne motion imagery
US9147255B1 (en) 2013-03-14 2015-09-29 Hrl Laboratories, Llc Rapid object detection by combining structural information from image segmentation with bio-inspired attentional mechanisms

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110081043A1 (en) * 2009-10-07 2011-04-07 Sabol Bruce M Using video-based imagery for automated detection, tracking, and counting of moving objects, in particular those objects having image characteristics similar to background
US20140218468A1 (en) * 2012-04-05 2014-08-07 Augmented Vision Inc. Wide-field of view (fov) imaging devices with active foveation capability
US9230302B1 (en) * 2013-03-13 2016-01-05 Hrl Laboratories, Llc Foveated compressive sensing system
US20170132468A1 (en) * 2015-11-06 2017-05-11 The Boeing Company Systems and methods for object tracking and classification

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
JEFF DONAHUE ET AL.: "Long-term Recurrent Convolutional Networks for Visual Recognition and Description", THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR, 2015, pages 2625 - 2634, XP032793708, DOI: 10.1109/CVPR.2015.7298878 *

Also Published As

Publication number Publication date
CN111566661B (en) 2023-11-17
WO2019152177A2 (en) 2019-08-08
EP3746938A2 (en) 2020-12-09
EP3746938A4 (en) 2021-10-06
CN111566661A (en) 2020-08-21

Similar Documents

Publication Publication Date Title
WO2019152177A3 (en) System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering
US10943096B2 (en) High-quality training data preparation for high-performance face recognition systems
EP3382643B1 (en) Automated object tracking in a video feed using machine learning
Deng et al. Amae: Adaptive motion-agnostic encoder for event-based object classification
WO2018053536A3 (en) Time-series fault detection, fault classification, and transition analysis using a k-nearest-neighbor and logistic regression approach
US9619753B2 (en) Data analysis system and method
US20140241574A1 (en) Tracking and recognition of faces using selected region classification
CN104966053A (en) Face recognition method and recognition system
WO2016107103A1 (en) Method and device for recognizing main region of image
CN105160318A (en) Facial expression based lie detection method and system
US9589351B2 (en) System and method for pet face detection
CN103824055A (en) Cascaded neural network-based face recognition method
US20170193373A1 (en) Disk capacity predicting method, apparatus, equipment and non-volatile computer storage medium
US20220172378A1 (en) Image processing apparatus, image processing method and non-transitory computer readable medium
US20160125268A1 (en) Image processing system, image processing method, and recording medium
US11062455B2 (en) Data filtering of image stacks and video streams
US20190340452A1 (en) Image processing apparatus, image processing method, and computer-readable recording medium recording image processing program
CN105528587A (en) Target detecting method and device
US20170006281A1 (en) Video Quality Detection Method and Apparatus
EP3772703A3 (en) Image processing apparatus and image processing method
US12073602B2 (en) Automated key frame selection
EP4152180A3 (en) Method of detecting action, electronic device, and storage medium
CN113723431B (en) Image recognition method, apparatus and computer readable storage medium
EP4618565A3 (en) Methods and systems for sport data extraction
US20220076005A1 (en) Person authentication apparatus, control method, and non-transitory storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19748018

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019748018

Country of ref document: EP

Effective date: 20200831