[go: up one dir, main page]

WO2022033665A1 - Dispositif et procédé de détection d'événement - Google Patents

Dispositif et procédé de détection d'événement Download PDF

Info

Publication number
WO2022033665A1
WO2022033665A1 PCT/EP2020/072535 EP2020072535W WO2022033665A1 WO 2022033665 A1 WO2022033665 A1 WO 2022033665A1 EP 2020072535 W EP2020072535 W EP 2020072535W WO 2022033665 A1 WO2022033665 A1 WO 2022033665A1
Authority
WO
WIPO (PCT)
Prior art keywords
event
pixel
intensity
sensing device
thresholds
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2020/072535
Other languages
English (en)
Inventor
Radu Ciprian Bilcu
Martti Ilmoniemi
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN202080104220.2A priority Critical patent/CN116114262B/zh
Priority to PCT/EP2020/072535 priority patent/WO2022033665A1/fr
Publication of WO2022033665A1 publication Critical patent/WO2022033665A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/703—SSIS architectures incorporating pixels for producing signals other than image signals
    • H04N25/707—Pixels for event detection

Definitions

  • the present disclosure relates generally to data processing, and more particularly to image processing for event-based cameras.
  • the disclosure proposes a device and method for sensing an event, wherein the device and method can be used for image processing in eventbased cameras.
  • Event-based cameras use types of sensors, which responds to changes in the incoming light intensity.
  • event-based cameras are asynchronous and pixels of the event-based cameras are activated only when a change in the intensity of the incoming light is sensed. Consequently, the output data rate of such a camera is variable.
  • there is no change of the incoming light intensity there are also no activated pixels and thus no data is generated by the camera.
  • the pixels that capture the intensity of the objects will generate the so-called events (triggered by changes in the incoming light sensed by the pixels).
  • the pixels of an event-based camera sensor usually capture the logarithm of the incoming light intensity, which is then further processed in additional sensor circuitry.
  • the events are generated in the pixels based on a sensitivity threshold.
  • a sensitivity threshold For a noise-free situation (i.e., a clean log intensity signal), a small sensitivity threshold is the optimum choice. However, for real world noisy cases, it is not obvious how to select a proper sensitivity threshold.
  • embodiments of the present disclosure aim to introduce a method to obtain high-quality (i.e., particularly high signal -to-noise ratio) reconstructed log intensity signals from events.
  • an objective is to use a more accurate sensitivity threshold.
  • One aim is to enable a larger number of time samples of the reconstructed intensity signal while keeping a lower noise level.
  • a first aspect of the disclosure provides an event sensing device, wherein the event sensing device comprises one or more pixel sensors, wherein one or more thresholds are associated with each of the one or more pixel sensors, and each pixel sensor is configured to: detect a time-dependent change in intensity of incoming light at the pixel sensor; generate an event if the time-dependent change in intensity exceeds any one of the one or more thresholds associated with the pixel sensor, wherein each event is associated with a time stamp and a threshold; and generate one or more event streams, each event stream including a plurality of the events associated with the same threshold; wherein the event sensing device is further configured to: reconstruct a plurality of independent intensity signals from a plurality of event streams generated based on a plurality of thresholds by the one or more pixel sensors.
  • the device of the first aspect enables the reconstruction of high-quality intensity signals from event streams generated based on a plurality of different sensitivity thresholds.
  • the sensitivity thresholds can be more accurate.
  • the device of the first aspect enables a larger number of time samples for the reconstructed intensity frame, while keeping a low noise level.
  • the time-dependent change in intensity comprises a change in the intensity from one point in time to a consecutive point in time.
  • each pixel sensor comprises one or more pairs of differentiator and comparator, each differentiator being configured to calculate the time-dependent change in the intensity of the incoming light, and each comparator being configured to generate an event if the time-dependent change in intensity exceeds a predetermined threshold and to generate an event stream.
  • a pixel sensor comprises one differentiator and one comparator. The differentiator calculates a difference between the current value of the log intensity and the previous value. When the difference is greater than a positive threshold Th (also named as sensitivity), a “+1” event is generated by the comparator; and a “-1” event is generated when the difference is smaller than the negative threshold -Th.
  • Th also named as sensitivity
  • the pixel sensor may comprise more than one pair of differentiator and comparator. That means, when the pixel sensor comprises multiple pairs of differentiator and comparator, multiple event streams will be generated by the pixel sensor. Notably, each one of the event streams corresponds to a predetermined threshold.
  • each pixel sensor comprises at least two pairs of differentiator and comparator, and each comparator is set with a respective predetermined threshold and is configured to generate an event stream based on the respective predetermined threshold, wherein the predetermined thresholds are different from each other.
  • different thresholds may be set or predetermined.
  • two similar event streams may be generated.
  • the event sensing device is further configured to: reconstruct an independent intensity signal of each comparator from each event stream generated by the comparator; and generate a final intensity signal for each pixel sensor by fusing reconstructed independent intensity signals of the at least two comparators of the pixel sensor.
  • an independent intensity signal can be calculated.
  • Conventional algorithms may be used here for calculating the independent intensity signal.
  • each comparator may generate an event stream, thus multiple independent intensity signals may be reconstructed by the pixel sensor.
  • a fusion result of the multiple independent intensity signals may lead to a final intensity signal of the pixel sensor. Since different thresholds may be considered when each pixel sensor generates event streams, a larger number of time samples of the reconstructed intensity signal can be ensured while a low noise level can be kept as well.
  • the event sensing device is further configured to: reconstruct the intensity image from a plurality of final intensity signals of the one or more pixel sensors.
  • a respective final intensity signal may be generated for each pixel sensor in the event sensing device. Accordingly, the intensity image can be reconstructed from the plurality of final intensity signals, which are generated for the plurality of pixel sensors comprised in the event sensing device.
  • the event sensing device further comprises a plurality of pixel sensors, wherein the plurality of pixel sensors is grouped into a plurality of super-pixels, wherein each super-pixel comprises at least two pixel sensors, and each of the at least two pixel sensors is set with a respective predetermined threshold, wherein the predetermined thresholds are different from each other.
  • a super-pixel comprising multiple pixel sensors can be designed.
  • the pixel sensors in the same super-pixel may be set with different thresholds.
  • this embodiment can be implemented using an existing event sensor (no change in hardware is required). It may be considered as a software implementation of generating event streams with multiple thresholds for one “pixel” (particularly superpixel).
  • the event sensing device is further configured to: reconstruct an independent intensity signal of each pixel sensor from the event stream generated by the pixel sensor; and generate a final intensity signal for each super-pixel by fusing reconstructed independent intensity signals of the at least two pixel sensors in the super-pixel.
  • an independent intensity signal may be reconstructed for each pixel sensor (based on an event stream generated with a single threshold).
  • multiple reconstructed independent intensity signals may be available for generating a final intensity signal.
  • pixel sensors in the same super-pixel may be set with different thresholds.
  • different thresholds may also be considered. In this way, a larger number of time samples of the reconstructed intensity signal is enabled while a low noise level can be kept as well.
  • the event sensing device is further configured to: reconstruct the intensity image from a plurality of final intensity signals of the plurality of super-pixels.
  • the intensity image can be reconstructed from the plurality of final intensity signals, which are generated for the plurality of super-pixels comprised in the event sensing device.
  • the at least two pixel sensors are adjacent to another.
  • the pixel sensors which belong to the same super-pixel, are not actually collecting light from the exact same point of the scene. It is thus preferred to have superpixels comprising pixel sensors close to each other.
  • each super-pixel comprises four pixel sensors.
  • each super-pixel may comprise a pixel array of four pixel sensors (2x2 array).
  • each pixel sensor is configured to detect a time-dependent change in log intensity of incoming light at the pixel sensor.
  • the pixel sensor may actually capture the logarithm intensity of the incoming light.
  • a second aspect of the disclosure provides a method for sensing event using an event sensing device, wherein the event sensing device comprises one or more pixel sensors, wherein one or more thresholds are associated with each of the one or more pixel sensors, and the method comprises: detecting a time-dependent change in intensity of incoming light at the pixel sensor; generating an event if the time-dependent change in intensity exceeds any one of the one or more thresholds associated with the pixel sensor, wherein each event is associated with a time stamp and a threshold; and generating one or more event streams, each event stream including a plurality of the events associated with the same threshold; wherein the method further comprises: reconstructing a plurality of independent intensity signals from a plurality of event streams generated based on a plurality of thresholds by the one or more pixel sensors.
  • Implementation forms of the method of the second aspect may correspond to the implementation forms of the event sensing device of the first aspect described above.
  • the method of the second aspect and its implementation forms achieve the same advantages and effects as described above for the event sensing device of the first aspect and its implementation forms.
  • a third aspect of the disclosure provides a computer program product comprising a program code for carrying out, when implemented on a processor, the method according to the second aspect and any implementation forms of the second aspect.
  • FIG. 1 shows a block diagram of a pixel sensor according to an embodiment of the disclosure.
  • FIG. 2 shows a noise-free log intensity signal and two event streams generated using different thresholds.
  • FIG. 3 shows the noise-free log intensity signal and two reconstructed log intensity signals (from the two event streams shown in FIG. 2).
  • FIG. 4 shows a noisy log intensity signal, two event streams generated using two different sensitivity thresholds and the corresponding reconstructed log intensity signals from events.
  • FIG. 5 shows an event sensing device according to an embodiment of the disclosure.
  • FIG. 6 shows a block diagram of an event sensing device comprising a pixel sensor according to an embodiment of the disclosure.
  • FIG. 7 shows multiple pixel sensors (and their sensitivity thresholds) of an event sensing device according to an embodiment of the disclosure.
  • FIG. 8 shows a block diagram of an event sensing device comprising a super-pixel according to an embodiment of the disclosure.
  • FIG. 9 shows an input noisy log intensity and reconstructed log intensity signals according to an embodiment of the disclosure.
  • FIG. 10 shows a method according to an embodiment of the disclosure.
  • an embodiment/example may refer to other embodiments/examples.
  • any description including but not limited to terminology, element, process, explanation and/or technical advantage mentioned in one embodiment/example is applicative to the other embodiments/examples.
  • FIG. 1 shows a block diagram of a pixel sensor 100 according to an embodiment of the disclosure.
  • a photoreceptor 101 may capture the incoming light and provide the logarithm of it to the next circuitry, i.e., a differentiator 102 as shown in FIG. 1, which may calculate the difference between the current value of the log intensity and the previous value 1021.
  • a positive threshold Th 1031 also named as sensitivity
  • a “+1” event may be generated by a comparator 103 as shown in FIG. 1.
  • a “-1” event may be generated.
  • An event stream may consist of: +ls or -Is that signal the cases that the incoming light intensity has increased or decreased, respectively, pixel coordinates, and a time stamp for each event.
  • Some event sensors may be capable to capture also intensity values, which may be also available at the output of the camera module.
  • FIG. 2 shows examples of simulated events, which are generated from a noise-free log intensity signal, for instance by the comparator 103 of the pixel sensor 100 as shown in FIG. 1.
  • one stream of events is generated for the Th value 1.75
  • the other stream of events is generated for the Th value 3.5. It can be clearly seen that when Th 1031 is smaller, more events are generated, while for a larger Th 1031, the number of generated events decreases.
  • the events, which are generated by each pixel 100, can be seen as the sign of the time derivatives of the incident log intensity.
  • the threshold Th 1031 can be seen as the quantization step of an analog-to-digital converter, although the output of the pixel 100 is not the digital value of the intensity.
  • an event sensor may not output the light intensity (except in the special case of a DAVIS sensor) but just some estimates of the derivative (of the events).
  • each pixel e.g., the pixel sensor 100 as shown in FIG. 1 do:
  • this algorithm is merely an example that can be used to calculate the log intensity. This disclosure is not limited to any specific algorithm.
  • FIG. 3 An example of a reconstructed log intensity signal is shown in FIG. 3.
  • the two log intensity signals shown in FIG. 3 is reconstructed from the generated events shown in FIG. 2.
  • using a smaller sensitivity threshold not only generates more events, but also provides a better reconstruction of the log intensity. This is actually expected, since finer quantization steps are providing a smoother output signal also in the case of digital-to-analog conversion (which is a somehow similar task to the present one).
  • this observation is only valid for a clean log intensity signal.
  • FIG. 4 illustrates this case of noisy log intensity signal, showing the generated events and the reconstructed log intensity for two values of the sensitivity threshold.
  • a small sensitivity threshold may be the optimum choice; for noisy cases the selection of the sensitivity threshold is not so obvious.
  • noise can have several different sources, such as additive noise due to the additional electronics (e.g., the differentiator 102, the comparator 103 as shown in FIG. 1, etc.), and photon shot noise.
  • Using a single sensitivity threshold leads to a tradeoff between the number of time samples (time instances where the intensity frame is available) and the noise level.
  • the number of time samples can be increased, however, this also increases the noise (variance of the reconstructed log intensity); on the other hand, by increasing the sensitivity threshold the impact of the noise is reduced, however, fewer samples for intensity reconstruction are available.
  • the threshold will in this case have a variable value that will adapt to the noise characteristics.
  • the reconstructed log intensity may have time periods where the quality of reconstruction is of low quality (e.g., due to noisy and/or lack of samples).
  • Embodiments of this disclosure propose methods to obtain a high-quality (high signal-to- noise ratio) reconstructed log intensity components from events only.
  • This disclosure thus enables reconstruction of high-quality intensity frames from event sensors using a plurality of different sensitivity thresholds.
  • the high quality means low noise and more time samples.
  • FIG. 5 shows an event sensing device 10 according to an embodiment of the disclosure.
  • the event sensing device 10 may comprise processing circuitry (not shown) configured to perform, conduct or initiate the various operations of the event sensing device 10 described herein.
  • the processing circuitry may comprise hardware and software.
  • the hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry.
  • the digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multipurpose processors.
  • the event sensing device 10 may further comprise memory circuitry, which stores one or more instruction(s) that can be executed by the processor or by the processing circuitry, in particular under control of the software.
  • the memory circuitry may comprise a non-transitory storage medium storing executable software code which, when executed by the processor or the processing circuitry, causes the various operations of the event sensing device 10 to be performed.
  • the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors.
  • the non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the event sensing device 10 to perform, conduct or initiate the operations or methods described herein.
  • the event sensing device 10 comprises one or more pixel sensors 100, 100’.
  • the event sensing device 10 may comprise a plurality of pixel sensors 100, 100’.
  • the event sensing device 10 may comprise a pixel array, and the plurality of pixel sensors 100, 100’ may be pixels of the pixel array.
  • one or more thresholds are associated with each of the one or more pixel sensors 100, 100’. That is, events will be generated based on any one of these one or more thresholds.
  • Each pixel sensor 100, 100’ is configured to detect a time-dependent change in intensity of incoming light at the pixel sensor 100, 100’.
  • each pixel sensor 100 may comprise a respective detection unit 102, 102’ for detecting the time-dependent change.
  • the detection units 102, 102’ may each be implemented as the differentiator 102 as shown in FIG. 1.
  • the time-dependent change in intensity may comprise a change in the intensity from one point in time to a consecutive point in time.
  • Each pixel sensor 100, 100’ is further configured to generate an event if the time-dependent change in intensity exceeds any one of the one or more thresholds associated with that pixel sensor 100, 100’.
  • each event may be associated with a time stamp and a threshold.
  • each pixel sensor 100, 100’ may further comprise a respective event generation unit 103, 103’ for generating the event based on a particular threshold at a specific time (when the change in intensity of incoming light occurs).
  • the event generation unit 103, 103’ may be the comparator 103 as shown in FIG. 1.
  • each pixel sensor 100, 100’ is configured to generate one or more event streams, each event stream including a plurality of the events associated with the same threshold. For instance, for each change in intensity that exceeds a threshold Thl, an event event Thl is generated.
  • An event stream generated with Thl comprises all events event_Thl generated in a certain period of time.
  • the event sensing device 10 is further configured to reconstruct a plurality of independent intensity signals 11, 11 ’ from a plurality of event streams generated based on a plurality of thresholds by the plurality of pixel sensors 100, 100’.
  • each pixel sensor 100, 100’ comprises one or more pairs of a respective differentiator 102, 102’ and a respective comparator 103, 103’, each differentiator 102, 102’ being configured to calculate the time-dependent change in the intensity of the incoming light, and each comparator 103, 103’ being configured to generate an event if the time-dependent change in intensity exceeds a predetermined threshold and to generate an event stream.
  • each of the two different pixel sensors 100 and 100’ comprises only one pair of a differentiator 102, 102’ and a comparator 103, 103’
  • different thresholds 1031, 1031’ may be set for these two pixel sensors 100 and 100’.
  • One idea of this disclosure is to generate, for each pixel, several event streams, each of the event streams being generated with a different (sensitivity) threshold. For each event stream, an independent log intensity signal can be reconstructed and the results can then fused together to generate a final log intensity signal, for that particular pixel.
  • threshold values can of course be combined with an adaptive threshold method, such that the values of one or more of the thresholds can be made adaptive.
  • the first embodiment proposes an event sensing device 10 comprising one or more pixel sensors 100, 100’, wherein each pixel sensor 100, 100’ comprises multiple differentiators 102, 102’ and comparators 103, 103’.
  • each pixel sensor 100, 100’ comprises multiple differentiators 102, 102’ and comparators 103, 103’.
  • FIG. 6 there are three pairs of differentiator (102, 104, 106) and comparator (103, 105, 107) in each pixel sensor 100.
  • each difference-comparator pair will generate its own event stream, which is independent from the other event streams. Therefore, each pixel may generate several independent event streams.
  • the comparator 103 compares a result from the differentiator 102 and a threshold Thl 1031, and generates an output event stream 1 accordingly.
  • an output event stream 2 and an output event stream 3 are generated by the comparator 105 and the comparator 107, respectively.
  • each comparator (103, 105, 107) may be set with a respective predetermined threshold and is configured to generate an event stream based on the respective predetermined threshold. As shown in the example depicted in FIG.
  • the thresholds Thl 1031, Th2 1051 and Th3 1071 may be set for the comparator 103, the comparator 105 and the comparator 107, respectively. It should also be noted that the predetermined thresholds (e.g., the threshold Thl 1031, Th2 1051 and Th3 1071) may be different from each other. That is, the thresholds Thl 1031, Th2 1051 and Th3 1071 may have different values.
  • each differentiator (102, 104, 106) calculates the change in the log intensity from the moment of the last event. Since each sensitivity threshold generates an independent event stream, there are preferably as many differentiators (102, 104, 106) as comparators (103, 105, 107).
  • one version of the log intensity may be calculated from each event stream (for every pixel).
  • the event sensing device 10 may be further configured to reconstruct an independent intensity signal (108, 109, 1010) of each comparator (103, 105, 107) from each event stream generated by the comparator (103, 105, 107).
  • an independent intensity signal (108, 109, 1010) is reconstructed from each of the output event stream 1, output event stream 2 and output event stream 3.
  • the final log intensity signal can then be obtained by fusing the signals reconstructed from the different event streams of the same pixel.
  • the event sensing device 10 may be configured to generate a final intensity signal 11 for each pixel sensor 100 by fusing reconstructed independent intensity signals (108, 109, 1010) of the at least two comparators (103, 105, 107) of the pixel sensor 100.
  • the final log intensity signal 11 can be calculated by fusing three independent intensity signals (108, 109, 1010) reconstructed from the output event stream 1, output event stream 2 and output event stream 3.
  • the method that can be used to reconstruct estimated log intensity components from the independent event streams is not limited in this disclosure. Further, the fusion method is also not limited in this disclosure. Any suitable conventional methods can be utilized.
  • the event sensing device 10 may be configured to reconstruct an intensity image from a plurality of final intensity signals 11 , 11’ of the one or more pixel sensors 100, 100’.
  • the second embodiment of this disclosure enables to implement the solution in an existing event sensor.
  • An implementation of the second embodiment is illustrated in FIG. 7.
  • the previous implementation of the first embodiment proposes a new hardware structure (i.e., each pixel sensor 100, 100’ comprises multiple differentiators and comparators), the second embodiment could be implemented with an existing event sensor.
  • the plurality of pixel sensors 100, 100’ may be grouped into a plurality of super-pixels 110.
  • Each super-pixel 110 may comprise at least two pixel sensors 100, 100’.
  • each of the at least two pixel sensors 100, 100’ may be set with a respective predetermined threshold, wherein the predetermined thresholds may be different from each other.
  • the pixel array is grouped into the plurality of superpixels (or can be named as multi-pixels) 110.
  • Each super-pixel 110 may contain four adjacent pixel sensors 100, 100’. Further, each of these adjacent pixel sensors 100, 100’ may have a different sensitivity threshold, wherein the thresholds are denoted as Thl, Th2, Th3 and Th4 in FIG. 7.
  • FIG. 8 shows a block diagram of the super-pixel 110 according to the embodiment shown in FIG. 7.
  • the super-pixel 110 comprises four adjacent pixel sensors 100-1, 100-2, 100-3, and 100-4.
  • each pixel sensor 100-1, 100-2, 100-3, and 100-4 may comprise only one differentiator-comparator pair.
  • the thresholds Thl 1031-1, Th2 1031-2, Th3 1031-3 and Th4 1031-4 may be set for the comparator 103-1, the comparator 103-2, the comparator 103-3, and the comparator 103- 4, respectively.
  • the predetermined thresholds may be different from each other. That is, the thresholds Thl 1031-1, Th2 1031-2, Th3 1031-3 and Th4 1031-4 may have different values.
  • one event stream may be generated by each of the pixel sensor 100-1, 100-2, 100-3, and 100-4.
  • the event sensing device 10 may be further configured to reconstruct an independent intensity signal 11-1, 11-2, 11-3, 11-4 of each pixel sensor 100-1, 100-2, 100-3, and 100-4 from the event stream generated by the pixel sensor 100-1, 100-2, 100-3, and 100-4.
  • the event sensing device 10 may be further configured to generate a final intensity signal 12 for each super-pixel 110 by fusing reconstructed independent intensity signals 11-1, 11-
  • the number of pixel sensors 100, 100’ in a super-pixel 110 does not have to be four. In principle, any number larger than one can be applied to the second embodiment of the disclosure.
  • the four event streams which belong to the same super-pixel 110, may not actually collect light from the exactly same point of the scene. Therefore, combining the event streams from these four sensors 100-1, 100-2, 100-
  • each super-pixel 110 comprising pixel sensors 100, 100’ close to each other.
  • each super-pixel 110 it is preferred to design each super-pixel 110 as comprising a 2x2 array of pixel sensors 100-1, 100-2, 100-3, and 100-
  • the event sensing device 10 may be further configured to reconstruct an intensity image from a plurality of final intensity signals 12 of the plurality of super-pixels 110.
  • an existing event sensor can be used, since no change in hardware (of the pixel sensor) is required. It may be considered as a software implementation of generating event streams with multiple thresholds for one “pixel”, i.e., super-pixel.
  • Embodiments of this disclosure enable a larger number of time samples of the reconstructed intensity frame, while a lower noise level can be kept.
  • the increased number of time samples may be important in the synchronization with other cameras, and also in order to be able to carry out de-noising (in time domain) of the reconstructed log intensity.
  • Low noise in the frame is also important in some applications like HDR.
  • FIG. 9 illustrates an input noisy log intensity, four reconstructed log intensity signals from four event streams, and a fusion result of four reconstructed log intensity signals.
  • each pixel sensor 100, 100’ may comprise four pairs of differentiator and comparator, each comparator is set with one of the thresholds (i.e., 2.75, 3, 3.25 or 3.5) and will generate an event stream based on that threshold.
  • a simple method that calculates the average of the samples of the four reconstructed log intensity signals can be used, if they were collected at exact same time stamps. Otherwise, a low-pass filtered version may be used. It is worth mentioning that any other fusion method could be used as well.
  • de-noising methods could be used as well to smooth the fused log intensity. It can be seen in FIG. 9 that for smaller sensitivity thresholds (such as 2.75 or 3), the reconstructed signal is noisier and has more samples compared with the signals obtained from event streams with larger thresholds. Notably, the result of the solution proposed in this embodiment (the fused result) has more samples compared with the other four reconstructed independent signals and it is also smoother.
  • each pixel sensor 100, 100’ of the event sensing device 10 comprises four pairs of differentiator and comparator.
  • the second embodiment i.e., different thresholds are set for a super-pixel 110
  • the second embodiment may cause a loss of the resolution in the reconstructed intensity image.
  • FIG. 10 shows a method 1000 for sensing event according to an embodiment of the disclosure.
  • the method 1000 is performed by an event sensing device 10 as shown in FIG. 5, FIG. 6 or FIG. 8.
  • the event sensing device 10 comprises one or more pixel sensors 100, 100’, wherein one or more thresholds are associated with each of the plurality of pixel sensors 100, 100’.
  • the method 1000 comprises: a step 1001 of detecting a time-dependent change in intensity of incoming light at the pixel sensor; a step 1002 of generating an event if the time-dependent change in intensity exceeds any one of the one or more thresholds associated with the pixel sensor 100, 100’, wherein each event is associated with a time stamp and a threshold; and a step 1003 of generating one or more event streams, each event stream including a plurality of the events associated with the same threshold.
  • the method further comprises a step 1004 of reconstructing a plurality of independent intensity signals 11, 11’ from a plurality of event streams generated based on a plurality of thresholds by the one or more pixel sensors 100, 100’.
  • each of the one or more pixel sensors 100, 100’ in the event sensing device 10 may be the pixel sensor as shown in FIG. 6, or the pixel sensor as shown in FIG. 8.
  • any method according to embodiments of the disclosure may be implemented in a computer program, having code means, which when run by processing means causes the processing means to execute the steps of the method.
  • the computer program is included in a computer readable medium of a computer program product.
  • the computer readable medium may comprise essentially any memory, such as a ROM (Read-Only Memory), a PROM (Programmable Read-Only Memory), an EPROM (Erasable PROM), a Flash memory, an EEPROM (Electrically Erasable PROM), or a hard disk drive.
  • embodiments of the event sensing device 10 comprises the necessary communication capabilities in the form of e.g., functions, means, units, elements, etc., for performing the solution.
  • means, units, elements and functions are: processors, memory, buffers, control logic, encoders, decoders, rate matchers, de-rate matchers, mapping units, multipliers, decision units, selecting units, switches, interleavers, de-interleavers, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiver units, transmitter units, DSPs, trellis-coded modulation (TCM) encoder, TCM decoder, power supply units, power feeders, communication interfaces, communication protocols, etc. which are suitably arranged together for performing the solution.
  • TCM trellis-coded modulation
  • the processor(s) of the event sensing device 10 may comprise, e.g., one or more instances of a Central Processing Unit (CPU), a processing unit, a processing circuit, a processor, an Application Specific Integrated Circuit (ASIC), a microprocessor, or other processing logic that may interpret and execute instructions.
  • the expression “processor” may thus represent a processing circuitry comprising a plurality of processing circuits, such as, e.g., any, some or all of the ones mentioned above.
  • the processing circuitry may further perform data processing functions for inputting, outputting, and processing of data comprising data buffering and device control functions, such as call processing control, user interface control, or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Photometry And Measurement Of Optical Pulse Characteristics (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)

Abstract

La présente divulgation se rapporte à un détecteur de caméra d'événement. À cet effet, la divulgation concerne un dispositif de détection d'événement comprenant une pluralité de détecteurs de pixels, un ou plusieurs seuils étant associés à chaque détecteur de la pluralité de détecteurs de pixels. Chaque détecteur de pixels est configuré : pour détecter un changement en intensité dépendant du temps de la lumière entrante au niveau du détecteur de pixels; pour générer un événement si le changement en intensité dépendant du temps dépasse l'un quelconque du ou des seuils associés au détecteur de pixels, chaque événement étant associé à une estampille temporelle et à un seuil; et pour générer un ou plusieurs flux d'événements, chaque flux d'événements comportant une pluralité d'événements associés au même seuil. Le dispositif de détection d'événement est en outre configuré pour reconstruire une pluralité de signaux d'intensité indépendants à partir d'une pluralité de flux d'événements générés sur la base d'une pluralité de seuils par la pluralité de détecteurs de pixels.
PCT/EP2020/072535 2020-08-11 2020-08-11 Dispositif et procédé de détection d'événement Ceased WO2022033665A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202080104220.2A CN116114262B (zh) 2020-08-11 2020-08-11 事件传感设备及方法
PCT/EP2020/072535 WO2022033665A1 (fr) 2020-08-11 2020-08-11 Dispositif et procédé de détection d'événement

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2020/072535 WO2022033665A1 (fr) 2020-08-11 2020-08-11 Dispositif et procédé de détection d'événement

Publications (1)

Publication Number Publication Date
WO2022033665A1 true WO2022033665A1 (fr) 2022-02-17

Family

ID=72046913

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2020/072535 Ceased WO2022033665A1 (fr) 2020-08-11 2020-08-11 Dispositif et procédé de détection d'événement

Country Status (2)

Country Link
CN (1) CN116114262B (fr)
WO (1) WO2022033665A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115412687A (zh) * 2022-10-31 2022-11-29 深圳时识科技有限公司 主被动结合的降噪装置、方法、视觉传感器和芯片
CN116883304A (zh) * 2023-07-14 2023-10-13 广东博华超高清创新中心有限公司 一种基于超像素的事件数据重建和质量评估方法

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019067054A1 (fr) * 2017-09-28 2019-04-04 Apple Inc. Génération d'images statiques à l'aide d'une caméra événementielle
WO2019099337A1 (fr) * 2017-11-14 2019-05-23 Kaban Technologies Llc Suivi d'objets déformables sur la base d'une caméra d'événements

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2677500B1 (fr) * 2012-06-19 2021-06-23 Samsung Electronics Co., Ltd. Appareil et procédé de traitement d'image basé sur des événements
CN108574793B (zh) * 2017-03-08 2022-05-10 三星电子株式会社 被配置为重新生成时间戳的图像处理设备及包括其在内的电子设备
US10237481B2 (en) * 2017-04-18 2019-03-19 Facebook Technologies, Llc Event camera for generation of event-based images

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019067054A1 (fr) * 2017-09-28 2019-04-04 Apple Inc. Génération d'images statiques à l'aide d'une caméra événementielle
WO2019099337A1 (fr) * 2017-11-14 2019-05-23 Kaban Technologies Llc Suivi d'objets déformables sur la base d'une caméra d'événements

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115412687A (zh) * 2022-10-31 2022-11-29 深圳时识科技有限公司 主被动结合的降噪装置、方法、视觉传感器和芯片
CN116883304A (zh) * 2023-07-14 2023-10-13 广东博华超高清创新中心有限公司 一种基于超像素的事件数据重建和质量评估方法
CN116883304B (zh) * 2023-07-14 2025-02-14 广东博华超高清创新中心有限公司 一种基于超像素的事件数据重建和质量评估方法

Also Published As

Publication number Publication date
CN116114262A (zh) 2023-05-12
CN116114262B (zh) 2025-10-24
CN116114262A8 (zh) 2024-05-24

Similar Documents

Publication Publication Date Title
CN112311964B (zh) 一种像素采集电路、动态视觉传感器以及图像采集设备
CN103888689B (zh) 图像采集方法及图像采集装置
Brandli et al. Real-time, high-speed video decompression using a frame-and event-based DAVIS sensor
US8830360B1 (en) Method and apparatus for optimizing image quality based on scene content
CN111727602A (zh) 单芯片rgb-d相机
WO2022033665A1 (fr) Dispositif et procédé de détection d'événement
JP2008263522A (ja) 画像処理装置及び方法
CN114286024B (zh) 基于动态视觉传感器的光学偏振信息模型构建方法及装置
FR3010603A1 (fr) Dispositif d'acquisition compressive d'une image
JP2015204488A (ja) 動き検出装置および動き検出方法
CN109816603A (zh) 单光子计数成像的图像传感器图像还原方法
CN109788216A (zh) 用于tof的抗干扰方法、装置及tof传感器芯片
WO2011032082A1 (fr) Système et procédé permettant d'acquérir une image fixe à partir d'une image en mouvement
JP2011078084A (ja) 信号の時間分解能の増加方法およびシステム
Muglikar et al. Event cameras meet spads for high-speed, low-bandwidth imaging
CN120752929A (zh) 基于事件处理来自单光子相机传感器的光子流数据的系统和方法
CN109309784B (zh) 移动终端
Chang et al. Towards HDR and HFR video from rolling-mixed-bit spikings
CN118214487B (zh) 脉冲信号的编码处理方法和装置、脉冲相机、设备和介质
WO2023143708A1 (fr) Reconstruction hdr à partir d'expositions bracketées et d'événements
US20250035750A1 (en) Systems, methods, and media for single photon depth imaging with improved efficiency using learned compressive representations
JPWO2014192223A1 (ja) 撮像装置および撮像方法
WO2017158587A1 (fr) Procédés de production d'images vidéo indépendantes de l'éclairage d'arrière-plan
EP4211644B1 (fr) Dispositif et procédé d'interpolation vidéo
JP2018042139A (ja) 撮像素子および撮像素子の動作方法、撮像装置、および電子機器

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20754255

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202080104220.2

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20754255

Country of ref document: EP

Kind code of ref document: A1