[go: up one dir, main page]

WO2012093381A1 - Ensemble caméra à analyseur de contenu intégré - Google Patents

Ensemble caméra à analyseur de contenu intégré Download PDF

Info

Publication number
WO2012093381A1
WO2012093381A1 PCT/IL2011/000003 IL2011000003W WO2012093381A1 WO 2012093381 A1 WO2012093381 A1 WO 2012093381A1 IL 2011000003 W IL2011000003 W IL 2011000003W WO 2012093381 A1 WO2012093381 A1 WO 2012093381A1
Authority
WO
WIPO (PCT)
Prior art keywords
imagery data
integrated circuit
camera assembly
data
image sensor
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/IL2011/000003
Other languages
English (en)
Inventor
Vitaly Sheraizin
Felix Tsipis
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to PCT/IL2011/000003 priority Critical patent/WO2012093381A1/fr
Publication of WO2012093381A1 publication Critical patent/WO2012093381A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G06V20/47—Detecting features for summarising video content
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/115—Selection of the code volume for a coding unit prior to coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124—Quantisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/167—Position within a video image, e.g. region of interest [ROI]

Definitions

  • the invention relates to a camera assembly integrally including an image sensor and a content analyzer.
  • Remote visual monitoring systems are widely used by the millions in many countries, and provide remote supervision of areas where additional observation is needed. Except from stationary surveillance systems, visual monitoring is utilized in applications such as UAVs (Unmanned Aerial Vehicles), AUVs (Autonomous Underwater Vehicles), land robots and more.
  • UAVs Unmanned Aerial Vehicles
  • AUVs Autonomous Underwater Vehicles
  • land robots and more.
  • Such remote monitoring systems include a set of dispersed cameras which deliver captured imagery data to a central monitoring station through a dedicated or a public network, wired or wireless.
  • the central monitoring station often includes a storage device for storing the incoming imagery data, as well as a CPU (central processing unit) usually capable of performing a certain level of content analysis on the imagery data.
  • CPU central processing unit
  • the CPU may employ advanced imagery analytic techniques, including pattern detection and/or facial recognition, which enable identifying and tracking individuals, vehicles and optionally other objects as they appear, move, or conduct any suspicious activity.
  • advanced imagery analytic techniques may enable autonomous operation of robots, such as finding objects, avoiding obstacles and the like.
  • the invention pertains to a camera assembly integrally including an image sensor and a content analysis chip (FPGA, ASIC or the like), connected therebetween with a fast data connection.
  • the data connection optionally enables a transfer speed sufficient to support the entire data output from the image sensor.
  • the camera assembly has the content analysis features incorporated onto the same die as the image sensor.
  • the camera assembly offers a significant reduction of the communication bandwidth needs when transmitting the captured data from the camera. This is achieved by real-time pattern detection done already at the content analysis chip level, which filters out non-significant data. In addition, detecting patterns at the camera assembly itself may prevent the need to do so later, using central processing resources external to the camera assembly.
  • the camera assembly offers low power consumption.
  • the power consumption of the pattern detector core alone may consume approximately 0.75W (Xilinx Spartan-3ADSP FPGA chip).
  • the power consumption of the whole camera assembly may be approximately 2.3 W.
  • running a similar Adaboost-based pattern detection algorithm on regular hardware Intel E8600- 3.3GHz CPU providing 1.8fps 5Mpix, 8bpp using Intel OpenCV pattern detector library routine
  • performing the pattern detection at the pattern detection level is superior in terms of power consumption to doing so using a general-purpose computer.
  • the content analysis integrated circuit incorporates a pattern detection algorithm specially adapted to leverage the content analysis integrated circuit's hardware capabilities.
  • the pattern detection algorithm is optimized for efficient pipeline parallelism, so that all different chip hardware units are efficiently utilized.
  • the camera assembly advantageously, may be suitable for both portable and fixed platforms, since the pattern detection algorithm analyzes each frame received from the image sensor separately, so that global motion of the camera does not affect the analysis.
  • the camera assembly may include an interface module enabling re-programming of the content analysis integrated circuit to detect different object types using the same basic pattern detection algorithm, by an update of its detection rules.
  • Detection rules may be prepared by an external training utility, which is fed a series of images which serve as a basis for learning, and concludes the detection rules.
  • a programmable integrated circuit for imagery data content analysis comprising: a programmable detection rule storage unit configured to store a set of detection rules; a communication interface enabling programming of said detection rule storage unit using an external device; and a content analyzer having circuitry for detecting, based on the set of detection rules, a pattern in imagery data, and for outputting a size- reduced version of the imagery data.
  • the outputting of the size-reduced version of the imagery data comprises omitting a part of the imagery data which is not part of the detected pattern.
  • the outputting of the size-reduced version of the imagery data comprises transmitting a part of the imagery data which is not part of the detected pattern in a quality lower than a quality of imagery data being part of the detected pattern.
  • the quality comprises resolution, color depth, bit rate, frame rate, compression, or any combination thereof.
  • an image-sensing integrated circuit configured to reduce its data output
  • the integrated circuit comprising: an image sensor configured to capture an image and produce imagery data; and a content analyzer having circuitry for detecting a region of interest in the imagery data and for outputting, based on the detection, a size-reduced version of the imagery data.
  • the integrated circuit is a monolithic integrated circuit.
  • said image sensor and said content analyzer are each incorporated in a separate body, and are both stacked into said integrated circuit.
  • the circuitry of said content analyzer is configured to detect the region of interest based on detection of a pre-determined pattern in the imagery data.
  • a method for reducing the size of imagery data output from a camera assembly comprising: transmitting imagery data from an image sensor of the camera assembly to a content analysis integrated circuit of the camera assembly; dete cting, using the content analysis integrated circuit, a pattern in the imagery data; and producing, based on the detected pattern and using the content analysis integrated circuit, a size-reduced version of the imagery data.
  • the method further comprises transmitting the size- reduced version of the imagery data, over an output channel, to a location external to the camera assembly.
  • the method further comprises transmitting the imagery data from the image sensor to the content analysis integrated circuit over a high-speed data bus.
  • the high-speed data bus is of a transfer speed sufficient to accommodate an entirety of the imagery data in real time.
  • a camera assembly configured to internally reduce its data output
  • the camera assembly comprising: an image sensor configured to capture imagery data; a content analysis integrated circuit connected to said image sensor and being configured to detect a region of interest in the imagery data and to output, based on the detection, a size-reduced version of the imagery data; and a communication interface connected to said content analysis integrated circuit and being configured to transmit the size-reduced version of the imagery data to a location external to the camera assembly.
  • said content analysis integrated circuit is configured to detect the region of interest based on detection of a pre-determined pattern in the imagery data.
  • said content analysis integrated circuit is connected to said image sensor using a high-speed data bus.
  • said high-speed data bus is of a transfer speed sufficient to accommodate an entirety of the imagery data in real time.
  • the imagery data comprises real-time imagery data.
  • the camera assembly further comprises a circuit board having thereon at least one component selected from the group consisting of: said image sensor, said content analysis integrated circuit and said communication interface.
  • said content analysis integrated circuit is a field- programmable gate array (FPGA).
  • said content analysis integrated circuit is an application-specific integrated circuit (ASIC).
  • ASIC application-specific integrated circuit
  • said camera assembly further comprises a second image sensor connected to said content analysis integrated circuit.
  • Fig. 1 shows a block diagram of a camera assembly
  • Fig. 2 shows a semi-pictorial view of an image sensing IC
  • Fig. 3 shows a schematic view of a content analyzer
  • Fig. 4 shows a schematic view of a pattern detector
  • Fig. 5 shows a schematic view of a method for pattern detection
  • Fig. 6 shows a schematic view of an in-line accumulative adder
  • Fig. 7 shows a schematic view of a line-to-line accumulative adder
  • Fig. 8 shows a schematic view of two rectangular regions inside an image
  • Fig. 9 shows a schematic view of a decision making algorithm.
  • An aspect of some embodiments relates to a camera assembly including an image sensor and a content analyzer formed as an integrated circuit (FPGA, ASIC or the like), which advantageously reduces the size of imagery data output from the camera assembly.
  • FPGA field-programmable gate array
  • ASIC application-specific integrated circuit
  • high-resolution surveillance cameras require a fast communication channel as well as vast computational capabilities at the receiving server due to wide bit stream and real time processing of high resolution imagery data.
  • the present camera assembly advantageously, offers significant reduction of the communication bandwidth needs when transmitting the captured data from the camera. This is achieved by real-time pattern detection done internally at the content analyzer level, which determines the regions which include significant data.
  • the term "significant data”, as referred to herein, relates to imagery data of a region which includes a detected object, while the term “non-significant data” relates to imagery data of a region which does not include a detected object, and therefore may also be referred to as "background”.
  • the camera assembly transmits the significant data and optionally a reduced version of the non-significant data to the server, thereby requiring less communication bandwidth.
  • the reduction of communication bandwidth may also eliminate the need for video compressors and decompressors, which may increase the cost and complexity of the whole system and may reduce the imagery quality.
  • the server receives only relevant video data and does not need to perform additional pattern detection by itself. Therefore, the camera assembly may substantially reduce the data processing overhead at the server side and may enable a simple CPU to handle a large number of camera assemblies simultaneously.
  • the camera assembly is configured to handle real-time pattern detection and transmission of predefined patterns to central server with minimum overloading of the communication channels or the central server processing resources.
  • the content analyzer processes the entire imagery data captured by the image sensor and extracts only significant data for further processing. Alternatively, non- significant data may also be transmitted, but with relatively reduced settings. Therefore, the camera assembly enables the use of low capability systems, which include low bandwidth and weak CPU based systems, in order to process a substantially higher imagery resolution and enables these systems to reach high- resolution real-time performances.
  • the camera assembly may be suitable for both portable and fixed platforms, since the pattern detection algorithm analyzes, separately, each frame received from the image sensor and therefore data analysis remains unaffected by the global motion of camera, and renders any motion analysis tools redundant . However, some additional processing and/or analysis may still be performed at the server level.
  • the content analyzer may be incorporated within an integrated circuit (FPGA,
  • the content analyzer may be connected to the image sensor using a high speed data connection, which may enable a transfer speed sufficient to support the entire real-time data output from the image sensor.
  • the camera assembly may have the content analyzer features incorporated in the same integrated circuit as the image sensor, thus allowing higher performance, lower power consumption and lower manufacture cost.
  • the content analyzer uses pattern detection concurrent algorithm designed to exploit the parallelism capabilities of the integrated circuit (FPGA, ASIC or the like).
  • the content analyzer may incorporate a pattern detection pre- processing stage for sub sampling high resolution images in order to expedite the pattern detection.
  • the content analysis integrated circuit incorporates a pattern detection algorithm specially adapted to leverage the content analyzer hardware capabilities.
  • the pattern detection algorithm is optimized for efficient pipeline parallelism, so that all different integrated circuit hardware units are efficiently utilized.
  • the content analyzer may be configured to support multiple pattern detection processes simultaneously.
  • the pattern detection algorithm may be based on Adaboost, a supervised machine learning meta-algorithm (Yoav Freund and Robert E. Schapire, "A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting". In Computational Learning Theory: Eurocolt 1995, pages 23-37) and Haar- like feature pattern characterization methodology (P. Viola, M. Jones, “Robust Realtime Object Detection", I JVC 2001 ; Papageorgiou, M. Oren and T. Poggio, "A General Framework for Object Detection”. International Conference on Computer Vision, 1998.).
  • Haar-like features are an alternate digital feature set based on Haar wavelets instead of the usual image luminance data.
  • the pattern detection algorithm may use rectangular-based features based on a Haar-like function.
  • a rectangular-based feature value is determined by two rectangular regions inside an image and calculated as the difference between the sums of pixels in these rectangular regions.
  • An "integral image”, in this regard, may be defined as a two-dimensional matrix having the same size as the original image (see Viola, id).
  • Each element of the integral image contains the sum of all pixel values located on the upper left region of the original image (in relation to the element's position). This allows simple computing of a sum of pixel values inside a rectangular area in the original image, at any position and/or rectangular dimensions, using only four simple arithmetic calculations:
  • points pt belong to the integral image, such that pti corresponds to the top left vertex point of the rectangle, pt 2 to top right vertex point, pt 3 to the bottom left vertex point and p to the bottom right vertex point.
  • integral image II(X) is defined as follows:
  • iKx. y ⁇ x' ⁇ x ⁇ y x x'. y' (2)
  • X(x,y) is a grayscale image and x,y refer to image column and image row, respectively.
  • the weak classifier has a binary output for determining whether a pattern is found in within a given image.
  • a threshold may be used in order to generate a weak classifier from a given feature.
  • a threshold may be determined using image weights, which are being recalculated during every round of content analyzer training
  • pattern detection algorithm may incorporate image normalization to overcome any inhomogeneous illumination conditions.
  • the camera assembly may include an interface module used for configuration of the content analyzer using a procedure referred to as "training".
  • the training procedure may be performed using an external computing device, for re-configuring the content analyzer to detect various object types using the same algorithm by an update of its detection rules.
  • the detection rules may be prepared by a training utility application.
  • Training utility application may use a set of "positive” images having pattern examples to be detected and a set of "negative” images, which are missing pattern examples (background images) to calculate predefined features, based on differences between pattern and background images.
  • Predefined features may include a set of unique image properties, existing in patterns, and missing in background images.
  • Training utility application automatically may select the most discriminative features from a set of Haar-like rectangular features and output a set of pattern detection rules, required for specific pattern detection.
  • Fig. 1 shows a camera assembly 100 in a schematic view, according to an embodiment.
  • Camera assembly 100 advantageously includes a content analysis electronic device, also referred to as "integrated circuit" 110.
  • camera assembly 100 may include a lens 102 configured to focus an image frame 114 onto an image sensor 104.
  • Image sensor 104 is configured to capture image frame 114 and produce an imagery data 122.
  • Image sensor 104 may be either black and white or color multi-megapixel sensor, such as CCD (charge-coupled device) sensor, CMOS(complementary metal-oxide- semiconductor) sensor and the like.
  • Image sensor 104 contains a sensing surface having an array of photosensitive elements and may optionally be equipped with an optical system such as lens 102 for optically focusing a light flow onto the image sensor's 104 sensing surface.
  • Image sensor 104 may be configured to deliver imagery data 122 including pixel data in various spatial resolutions in accordance with the image sensor 104 mode and tuning parameters.
  • Image sensor 104 may be configured to operate in binning mode and in window mode.
  • image sensor 104 When operated in binning mode, image sensor 104 may be configured to capture the optical scene with greatest possible field of view limited only by the optical parameters and reduce the imagery resolution by combining adjacent imager pixels to produce one output pixel. Furthermore, when operated in binning mode, image sensor 104 may be configured to transform (scale-down) the whole imagery pixels array to a smaller pixels array having a lower resolution imagery data. Therefore, when operating imager sensor 104 in binning mode, the whole image is a subject to pattern detection scanning.
  • image sensor 104 When operated in window mode, image sensor 104 may be configured to transfer only a sub-array of pixels from the imagery data 122 pixels.
  • Image sensor 104 may transfer imagery data 122 via a high speed bus 128 to a content analyzer 110.
  • a content analyzer 110 may be configured to detect a pattern in imagery data 122 transferred from image sensor 104.
  • content analyzer 110 may be configured to calculate the spatial coordinates of the detected object 116 and determine ROI 120 (region of interest) based upon the detected pattern, and acquire the imagery data 122 of the detected pattern from the image sensor 104 by tuning the image sensor 104 with the noted above coordinates.
  • content analyzer 110 may be configured to create an imagery data size-reduced version based on coordinates of detected object 116
  • content analyzer 110 may use the high-speed bus 128 to send control commands and tuning parameters to the image sensor 104.
  • Content analyzer 110 may be configured to operate in pattern detection mode and pattern acquisition mode.
  • the content analyzer 110 configures the image sensor 104 to operate in binning mode receiving the whole imagery data 122 pixels array.
  • content analyzer 110 may configure the image sensor 104 to operate in window mode receiving only a sub-image of the image frame 114 according to the coordination of the detected pattern. In case of more than one detected pattern in the same image frame 114, additional sub-images may be captured.
  • a communication interface 112 is connected content analyzer 110 and configured to transmit imagery data size-reduced version to a location external to the camera assembly 100, such as central server 128.
  • camera assembly 100 include one or more lenses in addition to lens
  • lens 106 which may be similar to lens 102, configured to focus an image frame 114 onto a second image sensor 108, which may be similar to Image sensor 104, and configured to capture image frame 114 and produce an imagery data 122.
  • Communication interface 112 may be further configured to transfer commands from a location external to camera assembly 100, via a communication channel 130, to content analyzer 110.
  • communication channel 130 may be configured for inbound transmission, not only for output.
  • the external location may be, for example, a central server 128.
  • Communication interface 112 may transfer digital data containing, for instance, detected object 116 coordinates and imagery data 122 from content analyzer 110 to an external location, for example a central server 128.
  • FIGs. 2 shows an image-sensing chip (also
  • Image-sensing chip 200 advantageously includes a content analyzer 210 incorporated into the same integrated circuit as an image sensor 204, and, optionally, a communication interface 212; this allows reaching even better performance.
  • each of content analyzer 210 and image sensor 204 may be incorporated in separate bodies of the integrated circuit (namely, in physically-separated integrated circuits inside the main integrated circuit), which are stacked together inside image-sensing chip 200, such as in horizontal layers or differently.
  • Content analyzer 210, image sensor 204 and/or communication interface 212 are optionally similar in function to content analyzer 110, image sensor 104 and/or communication interface 112, respectively, of Fig. 1.
  • Other elements shown in this figures such as a lens 202, an image frame 214, a detected object 216, a region of interest 220, a high-speed data bus 228, a data channel 226 and/or an output channel 230, may be similar to the respective elements shown in Fig. 1, which bear corresponding numbers in the 100's range, and are discussed above.
  • One or more of this elements are optionally adjusted, in form and/or in size, to conform to the requirements of being incorporated in a single integrated circuit chip, such as image- sensing chip 200.
  • FIG. 3 shows a block diagram of content analyzer 110 of Fig. 1 or content analyzer 210 of Fig. 2, according to an embodiment.
  • the content analyzer is referenced with the number 310.
  • a manager 334 may be configured to provide synchronization, general control and/or parameter distribution within and between different elements of content analyzer 310, in accordance with the desired operation scenario.
  • manager 334 may be configured to receive commands from a command parser 346, which may in turn be configured to interpret commands received from an external source, such as central server 128 appearing in Fig. 1, over an output channel 330.
  • a shared memory 332 may be configured to store binary data including imagery data received from imagery data pre-processor 336 and downscaled imagery data received from spatial scaler 338.
  • shared memory 332 may be configured to supply downscaled imagery data to pattern detector 340 and a size-reduced version of the imagery data to pattern transmitter 342.
  • Shared memory 332 may incorporate one or more direct memory access (DMA) controllers configured to expedite data transfer.
  • DMA controllers allow effective concurrent read and write operations for each channel simultaneously.
  • An addresses space of shared memory 332 may be shared with all DMA channels in order to provide data sharing.
  • manager 334 may supply general tuning parameters to shared memory 332 including initialization parameters.
  • a Pattern acquisitioner 344 may be configured to transfer tuning parameters from manager 334 to an image sensor 304, in order to tune image sensor 304 according to desired sensing mode.
  • manager 334 may be configured to deliver tuning parameters to pattern acquisitioner 344, including image sensing resolution parameters such as window height and windows width, shutter speed, desired image sensing mode such as binning mode or window mode, a start/stop command and/or the like.
  • image sensing resolution parameters such as window height and windows width, shutter speed, desired image sensing mode such as binning mode or window mode, a start/stop command and/or the like.
  • Imagery data pre-processor 336 may be configured to receive imagery data from image sensor 304.
  • Image sensor 304 may transfer color imagery data, a combined representation of color pixel values where color and luminance information may be combined using a color format such as Bayer RGB format.
  • Imagery data processing may be done using only luminance information, therefore color imagery data may require additional color and luminance segregation.
  • manager 334 may deliver tuning parameters to imagery data pre- processor 336, including imagery resolution specified by width and height (in number of pixels) and shared memory 332 address for separated luminance and chrominance imagery data storage location.
  • Imagery data pre-processor 336 may be configured to convert imagery data into separated luminance and chrominance imagery data format in order to support consecutive imagery data processing.
  • Imagery data pre-processor 336 may transfer separated luminance and chrominance imagery data to a shared memory 332 which may be configured to store binary data and may include one or more DMA channels.
  • imagery data pre-processor 336 may include the image sensor "glue-logic" components (not shown) providing low-level functions like clocking and/or synchronization of image sensor 304.
  • a spatial scaler 338 may be configured to provide tunable spatial resolution scaling of imagery data.
  • manager 334 may deliver tuning parameters to spatial scaler 338, including a scaling factor.
  • the scaling factor may optionally have a range of 20% to 100% of the input imagery data resolution.
  • the resulting downscaled imagery data may be stored in a shared memory 332.
  • a pattern transmitter 342 may be configured to transfer the detected patterns coordinates and imagery data size-reduced version to a communication interface 312, optionally as a composite data stream.
  • the detected pattern's coordinates may be supplied by manager 334 and an imagery data size-reduced version may be supplied by shared memory 332.
  • manager 334 may be configured to deliver tuning parameters to pattern transmitter 342 including start/stop data transfer command, graphical data segment address in shared memory 332 and/or the detected pattern's coordinates.
  • Pattern detector 340 which is further discussed below, is a major component of content analyzer 310. Pattern detector 340 may be configured to detect a pattern in imagery data and calculate the detected object's spatial coordinates.
  • manager 334 may deliver tuning parameters to pattern detector 340 including CCT (classifier constants table), PC SUM (sum of positive constants), CCT LENGTH (CCT length) and/or start stop commands.
  • CCT classifier constants table
  • PC SUM sum of positive constants
  • CCT LENGTH CCT length
  • Pattern detector 340 may be configured to transfer acknowledge signals to manager 334, including current state flag (CCF) carries busy-ready status of pattern detector 340 , pattern found flag (PFF), and/or found pattern's coordinates listing (FPCA).
  • CCF current state flag
  • PFF pattern found flag
  • FPCA found pattern's coordinates listing
  • Pattern detector 440 may incorporate a pattern detection algorithm based on the Adaboost algorithm.
  • the algorithm is presently adapted to support effective parallelism and pipelined operation over ASIC/FPGA based platform.
  • the algorithm adaptation increases performance effectiveness and results a compact footprint and low power consumption.
  • a shared memory 432 which is similar to shared memory 332 in Fig. 3, may be configured to deliver the luminance part of the imagery data to an imagery data storage RAM (Random Access Memory) 474 which may be configured to store a segment of imagery data.
  • imagery data storage RAM Random Access Memory
  • Imagery data storage RAM 474 may be configured to re-arrange imagery data into sub image segments having a certain pixel size, such as 32 pixels wide and 32 pixels high, which contains pixel values equal to the source imagery data, and ordered from left to right and from top to bottom of the sub image.
  • a scheduler 486 may be configured to enable or disable the data writing process by sending data write enable signal.
  • a data sequencer 450 may be configured to receive, convert and re-arrange the imagery data as a set of sub images segments into a sequence of data words.
  • Each data word contains a multi-pixel luminance value providing a wide data representation required to achieve a high level of parallelization.
  • Two exact copies of the set of sub images segments may be passed to two parallel paths of the calculation flow.
  • the first calculation path is configured to calculate an integral image and basic points extraction, while the second calculation path may be configured to extract and calculate normalization parameters. Both paths may be configured to work simultaneously, and therefore exploit the ASIC/FPGA hardware capabilities.
  • an in-line accumulative adder 462 may be configured to calculate a plurality of cumulative sums of each multi-pixel word, which result (PPi) a sum of each one multi-pixel (Pi) to its neighbor (excluding ppl).
  • a delay equalizer 464 may be used to create data delay equalization and latch result of previous iteration of in-line accumulative adder 462.
  • the equalized signal (PP ) may be transferred to a line-to- line accumulative adder 466, which may be configured to calculate a plurality of cumulative sums of neighborhood multi-pixel words.
  • Fig. 7 shows a schematic view of line-to-line accumulative adder 466 appearing in Fig. 4.
  • a line-to-line accumulative adder 466 may be configured to sum both current result from in-line accumulative adder 462 and previous result latched by delay equalizer 464.
  • PN ! PP ⁇ (7)
  • PN 2 PP + PN !
  • N3 PP 2 e + PN 2 (9)
  • the result (PNi) of line-to-line accumulative adder 466 may be passed to a multi-port RAM 468 that may include an input port and multiple output ports.
  • Multi -port RAM 468 may y be configured to provide simultaneous access to stored integral image.
  • a detection rule storage 484 may be configured to store a variety of binary data including, CCT values, wherein CCT may include a plurality of detection rules data including PC (positive constants), NC , THD f , BBR, BBL, BTR, BTL, ABR, ABL, ATR, ATL.
  • a basic points extractor 470 appearing in Fig. 4 may be configured to acquire the basic points of the integral image from a detection rule storage 484 appearing in Fig. 4.
  • the basic points may include ATL 822, ATR 824, ABL 826 and ABR 828 - coordinates of rectangle A 820; and BTL 842, BTR 844, BBL 846 and BBR 848 - coordinates of rectangle B 840.
  • an exponentiator 452 may be configured to raise a pixel value to the second power.
  • the resulting value may pass to a calculation path, which may be partially similar to the calculation path of the first path and may be executed simultaneously with the first path.
  • An in-line accumulative adder 454 which is similar to in-line accumulative adder 462, may be configured to calculate a plurality of cumulative sums of each multi-pixel word, which result a sum of each one multi-pixel to its neighbor.
  • the delayed signal may be transferred to a line-to-line accumulative adder 458, which may be configured to calculate a plurality of cumulative sums of neighborhood multi-pixels words.
  • line-to-line accumulative adder 458 may be configured to sum both current result from in-line accumulative adder 454 and previous result latched by delay equalizer 456.
  • the result of the line-to-line accumulative adder 458 may be delivered to a normalization parameters extractor 460 which may be configured to calculate normalization parameters, which are mean value ( ⁇ ) and standard deviation value ( ⁇ r).
  • a normalizer 472 may be configured to normalize basic points delivered by basic points extractor 470, using the normalization parameters passed by normalization parameters extractor 460 in accordance to the following equation:
  • ⁇ ( ⁇ - ⁇ )/ ⁇ (1 1 )
  • P is: PN A TL , PN AT R , PN A BL , PN AB R , PN BTL , PN BT R , PN BBL , PN BBR and NP is: PN N ATL , , PN N BTL , PN"BTR , PN N BBL . PN N BBR .
  • the resulting normalized basic point values are applied to the differentiator 482.
  • the differentiator 482 which may be configured to calculate classifier rectangles differences in accordance to the following equation: BBL ) (12) where D corresponds to a feature raw value 5, described in the following equations (13,14):
  • Comparator 480 may be additionally configured to receive values from detection rule storage 484 and output an appropriate value (referred as RES in equation 15) to accumulator 478 in accordance with the comparison result
  • the comparator 480 output value is delivered to the accumulator 478, where the values are cyclically accumulated (referred as ACC in equation 15) and at the same time the sum of remaining positive constants (referred as PS in equation 16) is calculated. Wherein sum of remaining positive constants starting value (referred as PC SUM in equation 16) is received from detection rule storage 484.
  • ACC ACC + RES (15) wherein ACC starting value is 0.
  • PS PS - PC (16) wherein PS starting value is PC SUM.
  • Accumulator 478 may be additionally configured to receive a reset signal for cyclically accumulation value and for sum of remaining positive constants from scheduler 486
  • Fig. 9 shows a schematic view of decision making algorithm.
  • a decision maker 476 appearing in Fig. 4 may be configured to receive cyclically accumulated value and sum of remaining positive constants and output a PDDR (pattern detection decision result) which may include "FOUND", "CONTINUE” or "STOP" values, to scheduler 486 appearing in Fig. 4.
  • PDDR pattern detection decision result
  • Scheduler 486 may include a counter which may be configured to reset by a start command and to increment by "FOUND” or "STOP" values of PDDR.
  • Scheduler 486 which may be configured to output result signals including a CCF (Current State Flag) which is active while the PDDR equals to CONTINUE, and inactive otherwise, a PPF (Pattern Found Flag) which is active while the PDDR equals to "FOUND", and inactive otherwise, a FPCA (Found Pattern Coordinates) which equals the latched value of scheduler 486 counter, a scheduler 486 counter value, a reset signal for cyclically accumulation value and for sum of remaining positive constants
  • CCF Current State Flag
  • PPF Power Found Flag
  • FPCA Fluund Pattern Coordinates
  • detection rule storage 484 may be configured to receive scheduler 486 counter value.
  • FIG. 5 shows a flow chart of a method 500 for reducing the size of imagery data output from a camera assembly, according to an embodiment.
  • the manager tunes detection parameters, providing basic tuning and required preliminary condition setup for different parts of content analyzer.
  • the command parser checks if external commands have arrived from the central server; if external commands have arrived, pattern detection parameters update accordingly.
  • the pattern acquisitioner configures the image sensor to operate in binning mode
  • the imagery data pre-processor receives imagery data from the image sensor, and produces and saves downscaled imagery data to the shared memory.
  • the manager configures the spatial scaler to a desired scaling factor, and the pattern detector performs pattern detection over downscaled imagery data.
  • the pattern acquisitioner configures the image sensor to operate in window mode with coordinates and resolution according to the detected pattern coordinates and size.
  • the imagery data pre-processor is configured to store a sub image into shared memory, while the pattern transmitter transmits imagery data size- reduced version and detected pattern's coordinates to the central server.
  • a decision block 510 it is decided whether multiple patterns have been detected. If so, method 500 returns back to block 508. If, on the other hand, multiple patterns have not been detected, method 500 proceeds to a decision block 512.
  • decision block 512 it is decided whether an external update is available from the central server through the command parser. If an external update is available, method 500 proceeds to decision block 514. If not, method 500 returns to block 506.
  • a decision block 514 it is decided whether an external update includes a command to stop pattern detection. If the external update includes such a command, then method 500 stops. If the external update does not include a command to stop pattern detection, then method 500 returns to block 502.
  • each of the words “comprise” “include” and “have”, and forms thereof, are not necessarily limited to members in a list with which the words may be associated.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

L'invention porte sur un circuit intégré programmable pour analyse de contenu de données d'imagerie, le circuit intégré comprenant : une unité de stockage de règles de détection programmable configurée pour stocker un ensemble de règles de détection ; une interface de communication permettant une programmation de ladite unité de stockage de règles de détection à l'aide d'un dispositif externe ; et un analyseur de contenu comprenant une circuiterie servant à détecter, sur la base de l'ensemble de règles de détection, un motif dans des données d'imagerie, et à délivrer une version de taille réduite des données d'imagerie.
PCT/IL2011/000003 2011-01-03 2011-01-03 Ensemble caméra à analyseur de contenu intégré Ceased WO2012093381A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/IL2011/000003 WO2012093381A1 (fr) 2011-01-03 2011-01-03 Ensemble caméra à analyseur de contenu intégré

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/IL2011/000003 WO2012093381A1 (fr) 2011-01-03 2011-01-03 Ensemble caméra à analyseur de contenu intégré

Publications (1)

Publication Number Publication Date
WO2012093381A1 true WO2012093381A1 (fr) 2012-07-12

Family

ID=46457281

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IL2011/000003 Ceased WO2012093381A1 (fr) 2011-01-03 2011-01-03 Ensemble caméra à analyseur de contenu intégré

Country Status (1)

Country Link
WO (1) WO2012093381A1 (fr)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9471840B2 (en) 2014-09-30 2016-10-18 Qualcomm Incorporated Apparatus and method for low-power object-detection in images using hardware scanning window
US9554100B2 (en) 2014-09-30 2017-01-24 Qualcomm Incorporated Low-power always-on face detection, tracking, recognition and/or analysis using events-based vision sensor
US9704056B2 (en) 2015-04-02 2017-07-11 Qualcomm Incorporated Computing hierarchical computations for computer vision calculations
US9762834B2 (en) 2014-09-30 2017-09-12 Qualcomm Incorporated Configurable hardware for computing computer vision features
US9838635B2 (en) 2014-09-30 2017-12-05 Qualcomm Incorporated Feature computation in a sensor element array
US9923004B2 (en) 2014-09-30 2018-03-20 Qualcomm Incorporated Hardware acceleration of computer vision feature detection
US9986179B2 (en) 2014-09-30 2018-05-29 Qualcomm Incorporated Sensor architecture using frame-based and event-based hybrid scheme
US10515284B2 (en) 2014-09-30 2019-12-24 Qualcomm Incorporated Single-processor computer vision hardware control and application execution
US10614332B2 (en) 2016-12-16 2020-04-07 Qualcomm Incorportaed Light source modulation for iris size adjustment
US10728450B2 (en) 2014-09-30 2020-07-28 Qualcomm Incorporated Event based computer vision computation
US10984235B2 (en) 2016-12-16 2021-04-20 Qualcomm Incorporated Low power data generation for iris-related detection and authentication
US11068712B2 (en) 2014-09-30 2021-07-20 Qualcomm Incorporated Low-power iris scan initialization
JPWO2021152877A1 (fr) * 2020-01-30 2021-08-05

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080101700A1 (en) * 2006-10-27 2008-05-01 Fuji Xerox Co., Ltd. Image generation apparatus, computer readable medium, computer data signal, information generation method, information reading system, electronic pen and recording medium
US20100183199A1 (en) * 2007-09-28 2010-07-22 Eye Controls, Llc Systems and methods for biometric identification

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080101700A1 (en) * 2006-10-27 2008-05-01 Fuji Xerox Co., Ltd. Image generation apparatus, computer readable medium, computer data signal, information generation method, information reading system, electronic pen and recording medium
US20100183199A1 (en) * 2007-09-28 2010-07-22 Eye Controls, Llc Systems and methods for biometric identification

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9986179B2 (en) 2014-09-30 2018-05-29 Qualcomm Incorporated Sensor architecture using frame-based and event-based hybrid scheme
US9986211B2 (en) 2014-09-30 2018-05-29 Qualcomm Incorporated Low-power always-on face detection, tracking, recognition and/or analysis using events-based vision sensor
US9582725B2 (en) 2014-09-30 2017-02-28 Qualcomm Incorporated Apparatus and method for low-power object-detection in images using image integration hardware
US11068712B2 (en) 2014-09-30 2021-07-20 Qualcomm Incorporated Low-power iris scan initialization
US9762834B2 (en) 2014-09-30 2017-09-12 Qualcomm Incorporated Configurable hardware for computing computer vision features
US9838635B2 (en) 2014-09-30 2017-12-05 Qualcomm Incorporated Feature computation in a sensor element array
US9870506B2 (en) 2014-09-30 2018-01-16 Qualcomm Incorporated Low-power always-on face detection, tracking, recognition and/or analysis using events-based vision sensor
US9923004B2 (en) 2014-09-30 2018-03-20 Qualcomm Incorporated Hardware acceleration of computer vision feature detection
US9940533B2 (en) 2014-09-30 2018-04-10 Qualcomm Incorporated Scanning window for isolating pixel values in hardware for computer vision operations
US10515284B2 (en) 2014-09-30 2019-12-24 Qualcomm Incorporated Single-processor computer vision hardware control and application execution
US9554100B2 (en) 2014-09-30 2017-01-24 Qualcomm Incorporated Low-power always-on face detection, tracking, recognition and/or analysis using events-based vision sensor
US9471840B2 (en) 2014-09-30 2016-10-18 Qualcomm Incorporated Apparatus and method for low-power object-detection in images using hardware scanning window
US9977977B2 (en) 2014-09-30 2018-05-22 Qualcomm Incorporated Apparatus and method for low-power object-detection in images using computer vision feature computation hardware
US10728450B2 (en) 2014-09-30 2020-07-28 Qualcomm Incorporated Event based computer vision computation
US9704056B2 (en) 2015-04-02 2017-07-11 Qualcomm Incorporated Computing hierarchical computations for computer vision calculations
US10614332B2 (en) 2016-12-16 2020-04-07 Qualcomm Incorportaed Light source modulation for iris size adjustment
US10984235B2 (en) 2016-12-16 2021-04-20 Qualcomm Incorporated Low power data generation for iris-related detection and authentication
JPWO2021152877A1 (fr) * 2020-01-30 2021-08-05
WO2021152877A1 (fr) * 2020-01-30 2021-08-05 ソニーセミコンダクタソリューションズ株式会社 Dispositif d'imagerie à semi-conducteurs, dispositif électronique et système d'imagerie
CN114902648A (zh) * 2020-01-30 2022-08-12 索尼半导体解决方案公司 固态成像装置、电子设备和成像系统
JP7680969B2 (ja) 2020-01-30 2025-05-21 ソニーセミコンダクタソリューションズ株式会社 固体撮像装置、電子機器及び撮像システム
US12450892B2 (en) 2020-01-30 2025-10-21 Sony Semiconductor Solutions Corporation Solid-state imaging device, electronic apparatus, and imaging system

Similar Documents

Publication Publication Date Title
WO2012093381A1 (fr) Ensemble caméra à analyseur de contenu intégré
US9268993B2 (en) Real-time face detection using combinations of local and global features
US9549120B2 (en) Adaptive data path for computer-vision applications
US12073581B2 (en) Adaptive face depth image generation
CN107993209B (zh) 图像处理方法、装置、计算机可读存储介质和电子设备
US11935257B2 (en) Adding an adaptive offset term using convolution techniques to a local adaptive binarization expression
US20240312036A1 (en) Accelerating speckle image block matching using convolution techniques
US20230127009A1 (en) Joint objects image signal processing in temporal domain
US11696039B2 (en) Smart IP camera with color night mode
US20240320971A1 (en) Pre-processing image frames based on camera statistics
US12015835B2 (en) Multi-sensor imaging color correction
US12094137B2 (en) Accelerated alignment of high-resolution image and depth map for low-bit-width floating-point representation
US20180260929A1 (en) Digital camera methods and devices optimized for computer vision applications
US11823430B2 (en) Video data processing
US12069235B2 (en) Quick RGB-IR calibration verification for a mass production process
US12488425B2 (en) Low-light image enhancement using key frame and dependent frame neural networks
US20230328386A1 (en) Intelligent auto-exposure control for rgb-ir sensor
US20230132230A1 (en) Efficient Video Execution Method and System
EP4542992A1 (fr) Système et procédé de codage d'image

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 11855054

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 11855054

Country of ref document: EP

Kind code of ref document: A1