KR102826195B1 - Apparatus and method for motion and motion path detection and vision sensor including the same - Google Patents

Abstract

A motion and motion path detection device and method and a vision sensor having the same are disclosed. The motion and motion path detection device according to one embodiment of the present application may include a frame information acquisition unit that has a pair of odd pixels and even pixels that overlap such that an integration time for input light corresponds to a preset overlap ratio, and acquires frame information for the input light using the pair of odd pixels and even pixels, and a frame difference calculation unit that performs an operation that takes into account the overlap ratio based on first frame information acquired by the odd pixels among the frame information and second frame information acquired by the even pixels among the frame information, thereby deriving a frame difference (FD) due to the input light.

Description

{APPARATUS AND METHOD FOR MOTION AND MOTION PATH DETECTION AND VISION SENSOR INCLUDING THE SAME}

The present invention relates to a device and method for detecting movement and movement paths, and a vision sensor having the same.

In recent years, more and more computer vision applications require lower latency and higher energy efficiency, and various on-sensor processing technologies are being introduced to reduce the amount of image data transmitted from vision sensors to digital backend processors to address these issues.

In this regard, frame memory is essential for temporal image processing, and large-capacity memory is required to store digitized pixel information. Therefore, both frame difference (FD)-based vision sensors and event-based vision sensors adopt in-pixel analog storage capacitors and processing circuits for in-sensor event detection, which inevitably increases the pixel size.

Additionally, advanced image processing such as optical flow generation requires large-area SRAM (Static Random Access Memory)-based frame memory.

The background technology of this application is disclosed in Korean Patent Publication No. 10-2209014.

The present invention is intended to solve the problems of the above-mentioned prior art, and to provide a motion and motion path detection device and method, and a vision sensor having the same, to which a frame difference processing technique that does not require an explicit storage capacitor inside a pixel is applied.

The present invention is intended to solve the problems of the above-mentioned prior art, and to provide a motion and motion path detection device and method having an in-pixel temporal memory (TM) circuit for area-efficient mixed-signal TD (Temporal Derivative) processing, and a vision sensor having the same.

However, the technical tasks to be achieved by the embodiments of the present invention are not limited to the technical tasks described above, and other technical tasks may exist.

As a technical means for achieving the above technical task, a motion and motion path detection device according to one embodiment of the present invention may include a frame information acquisition unit that has a pair of odd pixels and even pixels that overlap such that an integration time for input light corresponds to a preset overlap ratio, and acquires frame information for the input light using the pair of odd pixels and even pixels, and a frame difference calculation unit that derives a frame difference (FD) by the input light by performing an operation that takes into account the overlap ratio based on first frame information acquired by the odd pixels among the frame information and second frame information acquired by the even pixels among the frame information.

In addition, the frame difference calculation unit can calculate the frame difference by performing an operation of subtracting twice the second frame information from the first frame information or by performing an operation of subtracting twice the first frame information from the second frame information.

In addition, the motion and motion path detection device according to one embodiment of the present invention may include an event detection unit that compares the frame difference with a preset threshold value to determine whether an event has occurred and generates a reference voltage according to the occurrence of the event.

Additionally, the magnitude of the reference voltage may decrease due to leakage current over time from the time the event occurred.

Additionally, if the event is detected again while the reference voltage is decreasing due to the leakage current, the size of the reference voltage can be updated to a preset initial value.

Additionally, the motion and motion path detection device according to one embodiment of the present invention may include a time memory processing unit including a TM pixel that stores information about the reference voltage.

Additionally, each of the above pair of odd pixels and even pixels may be arranged to surround the TM pixel, and the odd pixels and even pixels may be arranged alternately.

Meanwhile, a vision sensor having a motion and motion path detection device according to one embodiment of the present invention may include a patch-based pixel circuit having a plurality of odd pixels and even pixels each overlapping with each other such that an integration time for input light corresponds to a preset overlap ratio, and a frame difference processing circuit which performs an operation considering the overlap ratio based on first frame information acquired by the odd pixels and second frame information acquired by the even pixels to derive a frame difference (FD) by the input light, and generates a reference voltage according to a result of comparing the frame difference with a preset threshold value.

Additionally, the patch-based pixel circuit may have a TM pixel that stores information about the reference voltage.

In addition, a vision sensor having a motion and motion path detection device according to one embodiment of the present invention may include a converter circuit that converts an output of the TM pixel to output a time series image in which information about the frame difference is reflected in response to a preset first driving mode.

Additionally, the converter circuit can convert the output of the odd pixels, the output of the even pixels, and the output of the TM pixels to output a normal image according to a predetermined reference point in time corresponding to a preset second driving mode.

Additionally, a vision sensor having a motion and motion path detection device according to one embodiment of the present invention may include a switch circuit for switching between the first driving mode and the second driving mode.

Meanwhile, a method for detecting a motion and a motion path according to one embodiment of the present invention may include a step of obtaining frame information on an input light by using a pair of odd pixels and even pixels that overlap such that an integration time for the input light corresponds to a preset overlap ratio, and a step of deriving a frame difference (Frame Difference (FD)) by the input light by performing an operation that takes into account the overlap ratio based on first frame information obtained by the odd pixels among the frame information and second frame information obtained by the even pixels among the frame information.

In addition, the step of deriving the frame difference may calculate the frame difference by performing an operation of subtracting twice the second frame information from the first frame information or by performing an operation of subtracting twice the first frame information from the second frame information.

In addition, a method for detecting movement and movement path according to one embodiment of the present invention may include a step of comparing the frame difference with a preset threshold value to determine whether an event has occurred, and generating a reference voltage according to the occurrence of the event.

In addition, a method for detecting movement and movement path according to one embodiment of the present invention may include a step of storing information about the reference voltage in a TM pixel and a step of converting an output of the TM pixel to output a time series image in which information about the frame difference is reflected.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described means for solving the problem of the present invention, a motion and motion path detection device and method, to which a frame difference processing technique that does not require an explicit storage capacitor inside a pixel is applied, and a vision sensor having the same can be provided.

According to the above-mentioned means for solving the problem of the present invention, the purpose is to provide a motion and motion path detection device and method having an in-pixel temporal memory (TM) circuit for area-efficient mixed-signal TD (Temporal Derivative) processing and a vision sensor having the same.

According to the above-described means for solving the problem of the present invention, a novel multi-mode vision sensor can be provided that integrates an area/power-efficient pixel circuit design and column-level processing technology.

According to the solution to the aforementioned problem of the present invention, frame difference (FD)-based event detection and temporal derivative (TD) generation operations can be performed without separate additional storage.

However, the effects that can be obtained from this invention are not limited to the effects described above, and other effects may exist.

FIG. 1 is a schematic diagram of a vision sensor having a motion and motion path detection device according to one embodiment of the present invention.
FIG. 2 is a conceptual diagram for explaining a frame difference (FD) detection process using a motion and motion path detection device according to one embodiment of the present invention.
FIG. 3a and FIG. 3b are conceptual diagrams for explaining a temporal derivative derivation process using a motion and motion path detection device according to one embodiment of the present invention.
Figure 4 is a drawing for explaining the operation of odd pixels, even pixels, and TM pixels in a normal image sensor mode.
Figures 5a and 5b are drawings for explaining the operation of odd pixels and even pixels in patch mode.
Figure 5c is a drawing for explaining the operation of the TM pixel in patch mode.
Figure 6 is a circuit diagram and timing diagram for explaining the operation of the frame information acquisition unit and the frame difference calculation unit.
Figure 7 is a circuit diagram and timing diagram for explaining the operation of the TM Calibrator.
FIG. 8 is a drawing exemplarily showing a micrograph of a vision sensor having a motion and motion path detection device according to one embodiment of the present invention.
FIG. 9 is a conceptual diagram for explaining the output type of a vision sensor having a motion and motion path detection device according to one embodiment of the present invention.
FIGS. 10A to 10C are diagrams exemplarily showing output images derived through a vision sensor having a motion and motion path detection device according to one embodiment of the present invention.
FIG. 11 is a diagram showing an experimental example linked to a motion and motion path detection device according to one embodiment of the present invention, comparing the performance of a conventional vision sensor and a vision sensor equipped with a motion and motion path detection device according to one embodiment of the present invention.
Figure 12 is a schematic diagram of a motion and motion path detection device according to one embodiment of the present invention.
Fig. 13 is a flowchart of an operation of a method for detecting motion and motion path according to one embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those with ordinary skill in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are assigned similar drawing reference numerals throughout the specification.

Throughout this specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" or "indirectly connected" with another element in between.

Throughout this specification, when it is said that an element is located “on,” “above,” “below,” “under,” or “below” another element, this includes not only cases where an element is in contact with another element, but also cases where another element exists between the two elements.

Throughout this specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

The present invention relates to a device and method for detecting movement and movement paths, and a vision sensor having the same.

Fig. 1 is a schematic configuration diagram of a vision sensor having a motion and motion path detection device according to one embodiment of the present invention. Specifically, Fig. 1 shows a top diagram of the column-level architecture of the vision sensor (10).

Referring to FIG. 1, a vision sensor (10) (hereinafter referred to as “vision sensor (10)”) having a motion and motion path detection device according to one embodiment of the present invention may include a patch-based pixel circuit (11), a frame difference processing circuit (12), a converter circuit (13), a switch circuit (14), a TM correction circuit (15), and a controller/row decoder circuit (16).

Specifically, the patch-based pixel circuit (11) of the vision sensor (10) may be provided with a plurality of odd pixels (1) and even pixels (2) that overlap each other so that the integration time for input light corresponds to a preset overlap ratio.

In addition, the frame difference processing circuit (12) of the vision sensor (10) performs an operation considering the overlap ratio based on the first frame information acquired by the odd pixels (1) and the second frame information acquired by the even pixels (2), thereby deriving the frame difference (Frame Difference, FD) by the input light, and can generate a reference voltage based on the result of comparing the derived frame difference with a preset threshold value.

Additionally, in this regard, the patch-based pixel circuit (11) may be equipped with a TM pixel (3) that stores information about a reference voltage generated by a frame difference processing circuit (12).

In addition, referring to FIG. 1, the patch-based pixel circuit (11) may be designed to have a structure in which each of an odd pixel (1) and an even pixel (2) surrounds a TM pixel (3), and the odd pixel (1) and the even pixel (2) are alternately arranged on the outside of the TM pixel (3). For example, referring to FIG. 1, the patch-based pixel circuit (11) may be designed to have a plurality of patch structures arranged in a plurality of ways, and may be formed as a 3 X 3 structure including one TM pixel (3) arranged in a central region inside each patch structure and four pairs of odd pixels (1) and even pixels (2) surrounding the TM pixel (3). In this case, the TM pixel (3) may be a pixel that stores a reference voltage that reflects an event occurrence detection result according to a frame difference (FD) acquired for four pairs of odd pixels (1) and even pixels (2) of the corresponding patch structure, corresponding to the corresponding patch structure.

Additionally, the converter circuit (13) of the vision sensor (10) can convert the output of the TM pixel (3) to output a time series image in which information on the frame difference (FD) is reflected in response to a preset first driving mode.

In addition, the converter circuit (13) can convert the output of the odd pixel (1), the output of the even pixel (2), and the output of the TM pixel (3) to output a normal image according to a predetermined reference point in time in response to a preset second driving mode.

Meanwhile, with respect to the driving mode of the vision sensor (10) described above, the vision sensor (10) may be provided with a switch circuit (14) for switching between the first driving mode and the second driving mode. For reference, in the description of the embodiment of the present invention, the first driving mode may be referred to as a 'patch mode', and the second driving mode may be referred to as a 'general image sensor mode', and the specific operation of the vision sensor (10) according to each driving mode and the output image generated in each driving mode will be described in detail below.

In addition, according to one embodiment of the present invention, the vision sensor (10) may include a motion and motion path detection device (100) (hereinafter, referred to as 'detection device (100)') according to one embodiment of the present invention, and according to an implementation example of the present invention, the detection device (100) may be a sub-module mounted inside the vision sensor (10) to include at least some circuits among the above-described patch-based pixel circuit (11), frame difference processing circuit (12), converter circuit (13), and switch circuit (14).

For example, the detection device (100) described in detail below may refer to a processing/operation device mounted (built-in) in the vision sensor (10) to include each module (circuit) included in the circuit diagram (enlarged drawing) shown in the form of a dotted box in FIG. 1.

In summary, the pixels forming the vision sensor (10) disclosed in the present invention are divided into two types: odd pixels (Odd; 1) and even pixels (Even; 2), and the integration time (Integration Time) of each type of pixel can be determined according to a preset overlap ratio.

Hereinafter, the driving mode of the vision sensor (10) will be described. Referring to Fig. 1, the frame difference (FD) calculation by the vision sensor (10) is performed for each patch unit including 3 X 3 pixels in which odd pixels (1) and even pixels (2) are arranged in a checkerboard pattern, and a TM pixel (3) used for TM operation is arranged at the center of each patch structure.

In addition, referring to ① of FIG. 1, the outputs of four odd pixels (1) and even pixels (2) arranged in each patch are transferred to a frame difference operation unit (120) including a column-sharing FD processor (Col. Shared FD Processor) through a patch-level column-sharing COLPE/PO line, and referring to ② of FIG. 1, an FD event signal EVT that updates the TM pixel value of the corresponding patch can be generated through the frame difference operation unit (120) including the FD processor and the event detection unit (130) including the FDTH comparator.

In addition, the vision sensor (10) disclosed in the present invention can comprehensively support a first driving mode corresponding to a patch mode and a second driving mode corresponding to a general image sensor mode, and a converter circuit (13) having a plurality of analog-to-digital converters (Column-parallel single-slope ADCs) can convert the pixel output of Col _PIX in the image sensor mode (second driving mode), and, referring to ③ of FIG. 1, can operate to convert the TM output of Col _TMO in the patch mode (first driving mode). Meanwhile, the analog-to-digital converter element of the converter circuit (13) can be designed to have a variable resolution in the range of 1b to 8b according to an implementation example of the present invention, but is not limited thereto.

FIG. 2 is a conceptual diagram for explaining a frame difference (FD) detection process using a motion and motion path detection device according to one embodiment of the present invention.

Referring to FIG. 2, when the overlap ratio for the integration times of odd pixels (1) and even pixels (2) is set to 50%, for example, at the nth frame time point, the F[n] frame value can be stored in the even pixel (2), and the (F[n-1]+F[n]) value can be stored in the odd pixel (1). Therefore, when an operation (O-2×E) of subtracting twice the pixel value of the even pixel (2) from the pixel value of the odd pixel (1) is performed, the frame difference (F[n-1] - F[n]) between the (n-1)th frame and the nth frame can be calculated. In this way, since the frame information at each frame time point can be stored in the photodiode (PD) detection node of each pixel, the vision sensor (10) can implement frame difference (FD) calculation through the frame difference processing circuit (12) without an additional storage capacitor.

FIG. 3a and FIG. 3b are conceptual diagrams for explaining a temporal derivative derivation process using a motion and motion path detection device according to one embodiment of the present invention.

Referring to FIG. 3a, the event detection unit (130) of the frame difference processing circuit (12) compares the generated frame difference value with a preset threshold value (FD _TH ) for FD event processing when a frame difference (FD) is generated, and if it is determined that an event has occurred based on the comparison result (FD>FD _TH ), it controls the switching element so that the reference voltage (V _TM ) is set to high, and as time passes based on that point in time (i.e., the point in time when the event occurred), the size of the reference voltage (V _TM ) decreases to a low state due to leakage current.

In other words, the size of the reference voltage (V _TM ) decreases due to the leakage current over time from the time when the event occurs, and if the event is detected again through frame difference (FD) processing while the reference voltage is decreasing due to the leakage current, the size of the reference voltage (V _TM ) can be updated to a preset initial value, so that the reference voltage (V _TM ) can increase.

Accordingly, the value (magnitude) of the reference voltage (V _TM ) can be utilized as a parameter indicating how frequently a frame difference (FD)-based event occurs. For example, when an object moves quickly, frame difference (FD)-based events occur relatively frequently, so the reference voltage (V _TM ) maintains a high value, whereas when the object is stationary, the reference voltage (V _TM ) value becomes 0.

In this regard, (a) of FIG. 3b shows the change in the size of the reference voltage (V _TM ) when the object measured by the vision sensor (10) moves relatively fast (Fast Moving Image), (b) of FIG. 3b shows the change in the size of the reference voltage (V _TM ) when the object measured by the vision sensor (10) moves relatively slow (Slow Moving Image), and (c) of FIG. 3b shows the change in the size of the reference voltage (V _TM ) when the object measured by the vision sensor (10) is stationary (Static Image).

Hereinafter, the specific structure and operation method of each sub-circuit part constituting the vision sensor (10) will be described in detail with reference to FIGS. 4 to 7.

Figure 4 is a drawing for explaining the operation of odd pixels, even pixels, and TM pixels in a normal image sensor mode.

Referring to FIG. 4, in the normal image sensor mode (the second driving mode), all pixels (in other words, the odd pixel (O; 1), the even pixel (E; 2), and the TM pixel (TM; 3) can operate as basic 3T pixels. Specifically, the PS node, the PS _TM node, the EVT node, and the Col _PE/PO node can be connected to the first power voltage node (VDD), and the RS _TMO node and the Col _TMO node can be connected to the second power voltage node (VSS). In addition, the output values (V _{PD_E/O} and V _{PD_TM} ) of each sensing node are transmitted to the heat sharing output Col _PIX side through the M _SF switch and the M _RS switch, respectively.

Figures 5a and 5b are drawings for explaining the operation of odd pixels and even pixels in patch mode.

Referring to FIGS. 5a and 5b, Col _PIX is connected to a second power supply voltage node (VSS), and RS _E and RS _E for odd pixels (1) and even pixels (2) can be set to a first power supply voltage (VDD), so that the M _RS switches of the odd pixels (1) and even pixels (2) operate as degenerate resistors for the M _SF switches, and the pixel values V _{PD_E} and V _{PD_O} can generate pixel current outputs i _E and i _O , respectively. In addition, i _E and i _O are transferred to the Col _PE and Col _PO sides through the M _PT switches in each pixel for patching operation. In addition, referring to FIG. 5b, a short pulse signal PS can be applied to gate the M _PT switches to reduce power consumption. Additionally, the current output (i _E and i _O ) at each pixel is fed to Col _PE/PO to be summed to the patch-level current output (i _PE/PO ) of the corresponding patch structure according to the type of each pixel.

Additionally, the timing diagram illustrated in Fig. 5b shows that the image integration times between the odd pixels (1) and the even pixels (2) overlap in each frame by alternately resetting the PD (RST _E/O ) of the odd pixels (1) and the even pixels (2).

Figure 5c is a drawing for explaining the operation of the TM pixel in patch mode.

Referring to FIG. 5, in the case of the TM pixel (3), it can be confirmed that, compared to the odd pixel (1) and the even pixel (2), there are additional switch elements (transistors) corresponding to M _SFTM and M _RSTM , and in the first driving mode (patch mode), the reference voltage node (V _TM ) can be set to a high state when both the FD event signal EVT and the PS _TM are high.

In this regard, the M _EVT can be implemented with basic NMOS devices for the logical AND operation of the EVT and PS _TM . In addition, when the RST _TM corresponds to the first reference voltage (VDD) and the RS _TM corresponds to V _BTM , the leakage current of i _RS is generated to decrease the reference voltage (V _TM ) over time. At this time, V _BTM is a negative voltage generated internally to minimize the device size of the M _RS and exhibit a long time constant even with only the parasitic capacitance at the reference voltage (V _TM ) node. The voltage of the reference voltage node (V _TM ) is delivered to the patch-level column sharing COL _TMO through the M _SFTM and the M _RSTM , and the RS _TMO is a row select signal for the TM output (TMO).

In addition, in the timing diagram of the TM operation of (b) of Fig. 5c, GCLK is a global clock signal, and referring to (b) of Fig. 5c, for the [i, k]th patch operation, it can be confirmed that the operation of generating i _PE/PO through PS[i], the operation of generating FD event signal EVT[k] after frame difference (FD) processing (operation), and the operation of reflecting the EVT result to the reference voltage (V _TM ) through PS _TM [i] and storing it in the TM pixel (3) are sequentially performed.

Fig. 6 is a circuit diagram and timing diagram for explaining the operation of the frame information acquisition unit and the frame difference calculation unit. Specifically, Fig. 6 shows a detailed circuit diagram of a frame difference processing circuit (12) having a frame difference calculation unit (120) including an FD processor and an event detection unit (130) including an FD _TH comparator.

First, assuming a static image where the detected object is stationary, one of i _PE or i _PO is generated in half the image integration time of the other, so that the current pixel value of one pixel in a specific patch structure is twice that of the other pixel. For example, if an odd pixel (1) is exposed to a static image for one frame time and an even pixel (2) is also exposed to a static image so that they are equally exposed to the static image for two frame periods, V _{PD_O} is set to twice the value of V _{PD_E} , and assuming that the voltage-to-current conversion using M _SF and M _RS is linear, i _PO also becomes 2×i _PE . Since i _PE/PO can be inversely related to the integration time of the image, I _OFS can be introduced at the input of the FD processor to generate I _OFS -i _PO/PE . In addition, the PSCOL switch is a gating signal of I _OFS , and the pulse widths of PS and PS _TM are identical.

Here, I _OFS is set to the i _PO/PE value when V _{PD_E/O} is equal to VDD. Therefore, (I _OFS -i _PO/PE ) becomes proportional to the integration time, and then an operation to integrate (I _OFS -i PE _{/PO ) into the capacitor C PE/PO} for V _{PE /PO} is performed, and the entire process up to this point occurs during PΦ1 in Fig. 6, and C _PEO (=C _PE/PO1 +C _PE/PO2 ) and PEO signal can be used for the double multiplication operation for the frame difference (FD) operation.

For example, if an odd pixel (1) integrates two frames and an even pixel (2) integrates one frame, then C _PEO is added to V _PO with PEO = 0, doubling the capacitance at V _PO . That is, this means that the integrated value V _PE , (I _OFS -i _PE ), is multiplied by 2.

That is, for a static image, there is no difference between V _PE and V _PO , and in contrast, for a dynamic image, there is a difference between V _PE and V _PO , which is compared with FD _TH in the next step. In PΦ2 illustrated in Fig. 6, the capacitive DAC with C _PE2 increases the lower value of V _{PE / PO} by FD _TH according to the comparison result PΦ2C. In addition, if it is not toggled according to the comparison result in step PΦ3 of Fig. 6 (in other words, if FD > FD _TH is satisfied), the EVT signal is generated.

Figure 7 is a circuit diagram and timing diagram for explaining the operation of the TM Calibrator.

Specifically, Fig. 7 (a) shows a circuit diagram of a TM correction circuit (V _BTM generation circuit; 15) consisting of a column-level dummy TM pixel, a comparator, an accumulator, a 1b output delta-sigma modulator, and a negative charge pump. Referring to Fig. 7 (a), an internally generated correction signal EVT _CAL can be transmitted to the dummy TM pixels according to a target decay time (TDT) for a reference voltage (V _TM ).

Also, Fig. 7(b) shows a conceptual timing diagram when one dummy TM pixel and TDT=8 frames are used for compensation, where EVT _CAL =high is applied once per 8 frames and the control loop is stabilized when COMP _OUT is high for 4 frames and low for the remaining 4 frames. Furthermore, according to one embodiment of the present disclosure, 84 full column level dummy TM pixels may be used to further mitigate internal die process variations, but is not limited thereto.

FIG. 8 is a drawing exemplarily showing a micrograph of a vision sensor having a motion and motion path detection device according to one embodiment of the present invention.

Referring to FIG. 8, a chip of a vision sensor (10) according to one embodiment of the present invention may be implemented with a 0.18 μm standard CMOS process, and a core area including an output buffer used for chip testing may be designed to be at the level of 3.7 × 3.8 mm ² . In addition, a pixel array may be configured with 324 × 252 pixels, and a pixel pitch may be designed to be 10 μm, a fill factor may be designed to be 21%, and a frame speed may be designed to be 12.5 fps.

FIG. 9 is a conceptual diagram for explaining the output type of a vision sensor having a motion and motion path detection device according to one embodiment of the present invention.

Specifically, (a) of FIG. 9 is an image acquired corresponding to a specific frame, (b) of FIG. 9 is an image showing the occurrence of an event between frames derived through a frame information acquisition and calculation method using a pair of odd pixels (1) and even pixels (2) disclosed in the present invention, and (c) of FIG. 9 is a temporal derivative image generated using a TM pixel (3) that stores information on a reference voltage (V _TM ) according to the occurrence of an event.

In this regard, the vision sensor (10) disclosed in the present invention can operate to output a time series image (time derivative image) such as (c) of FIG. 9 in response to the first driving mode (patch mode), and in contrast, can operate to output a general image (frame image) such as (a) of FIG. 9 in response to the second driving mode (general image sensor mode).

FIGS. 10A to 10C are diagrams exemplarily showing output images derived through a vision sensor having a motion and motion path detection device according to one embodiment of the present invention.

Fig. 10a is a drawing showing the result of capturing a white circle moving from the lower left to the upper right on a black background on the screen using a vision sensor (10). The first row of Fig. 10a is an output image according to the general image sensor mode, and represents the object position in the nth frame (F[n]), respectively. The second and third row images are patch mode outputs (8b TD output; TDO) with TDT = 8 frames and TDT = 16 frames, respectively, and it can be confirmed that the moving direction and trajectory of the object are clearly captured with a single frame TDO. In addition, it can be confirmed that the larger the setting value for TDT, the longer the tracking result (trajectory, trace) for the object is reflected in the image.

In addition, the last row image represents 1b TDO, and for 1b processing, both upper and lower thresholds are applied to TMO, and only the middle value of TMO generates TDO=1, and through this double thresholding, only the tracking information of the corresponding object can be extracted.

Fig. 10b shows the results of capturing two vehicles moving at different speeds using a vision sensor (10) and displaying them on the screen. The vehicle shown on the upper side moves relatively fast at twice the speed of the vehicle shown on the lower side. Referring to Fig. 10b, in the general image sensor mode (left image), the image for the vehicle on the upper side appears relatively blurry, so that the speed difference between the two vehicles is indirectly expressed, whereas in the time series image (TDO) according to the patch mode (right image), the speed of the object can be directly extracted through the direction, length, and gradient of the trajectory. In this case, it can be confirmed that the relatively fast vehicle (the vehicle shown on the upper side) exhibits a trace length (Long Trace) that is twice as long as that of the relatively slow vehicle and a smooth gradient (Gentle Gradient).

FIG. 10c shows an image captured by a vision sensor (10) of a fan with three rotating blades and two fixed objects (dummies), and it can be seen that the movement of the blades is captured by the frame difference (FD) processing method disclosed herein, while the fixed objects and the support parts of the fan are filtered out (see the EVT image at the upper right). In addition, the time series image (TDO) generated after TM processing clearly shows the traces of the blades, and although in this demo the frame rate of the sensor is not fast enough compared to the blade speed, so that the EVT of each frame leaves an individual trace, each trace shows a different value depending on the time (8b-TDO, lower right) at which the EVT occurred, so that the fan rotation direction can still be identified.

Additionally, based on the 12 traces generated between the two blades, the fan speed can be estimated as 20.8 rpm (= 1 rotation / (3 blades × 12 traces) × 12.5 fps × 60 seconds), which corresponds to the actual fan speed.

FIG. 11 is a diagram showing an experimental example linked to a motion and motion path detection device according to one embodiment of the present invention, comparing the performance of a conventional vision sensor and a vision sensor equipped with a motion and motion path detection device according to one embodiment of the present invention.

Referring to FIG. 11, the power consumption is scalable according to the TDO resolution, and compared to the multimode vision sensor in the conventional study, [1] T.-H. Hsu et al., "A 0.8 V multimode vision sensor for motion and saliency detection with ping-pong PWM pixel," IEEE JSSC, 2021. and [2] X. Zhong et al., "A fully dynamic multi-mode CMOS vision sensor with mixed-signal cooperative motion sensing and object segmentation for adaptive edge computing", IEEE JSSC, 2020. The conventional vision sensor according to the study shows slightly better performance in terms of FoM, but can only perform simple image processing such as extracting frame differences by additionally arranging capacitors in the pixels or removing the background using an external SRAM, whereas the vision sensor (10) disclosed in the present invention has an advantage of being able to perform temporal derivative derivation without a separate storage element.

In addition, the optical flow sensor disclosed in [5] S. Park et al., "243.3pJ/pixel bio-inspired time-stamp-based 2D optic flow sensor for artificial compound eyes," IEEE ISSCC, 2014. performs time derivative operations but uses a separate SRAM for storing time stamps in addition to a digital post-processing core, whereas the vision sensor (10) disclosed in this application showed similar time processing results with pixels that are 9.6 times smaller without additional storage elements.

Figure 12 is a schematic diagram of a motion and motion path detection device according to one embodiment of the present invention.

Referring to FIG. 12, the detection device (100) may include a frame information acquisition unit (110), a frame difference calculation unit (120), an event detection unit (130), and a time memory processing unit (140).

The frame information acquisition unit (110) can acquire frame information on input light by using a pair of odd pixels (1) and even pixels (2) that overlap so that the integration time for the input light corresponds to a preset overlap ratio.

The frame difference calculation unit (120) can derive the frame difference (Frame Difference, FD) due to input light by performing a calculation that takes into account the overlap ratio based on the first frame information obtained by the odd pixel (1) and the second frame information obtained by the even pixel (2) among the frame information obtained by the frame information acquisition unit (110).

Specifically, the frame difference calculation unit (120) can calculate the frame difference by performing an operation to subtract twice the second frame information from the first frame information or by performing an operation to subtract twice the first frame information from the second frame information.

The event detection unit (130) can determine whether an event has occurred by comparing the frame difference derived by the frame difference calculation unit (120) with a preset threshold value. In addition, the event detection unit (130) can generate a reference voltage according to the occurrence of the determined event.

The time memory processing unit (140) can store information about the reference voltage in the TM pixel (3). In addition, the time memory processing unit (140) can convert the output of the TM pixel (3) to output a time series image in which information about the frame difference is reflected.

Below, we will briefly review the operating flow of this system based on the detailed explanation above.

Fig. 13 is a flowchart of an operation of a method for detecting motion and motion path according to one embodiment of the present invention.

The motion and motion path detection method illustrated in Fig. 13 can be performed by the detection device (100) described above. Therefore, even if the content is omitted below, the content described for the detection device (100) can be equally applied to the description of the motion and motion path detection method.

Referring to FIG. 13, in step S11, the frame information acquisition unit (110) can acquire frame information on input light by using a pair of odd pixels (1) and even pixels (2) that overlap so that the integration time for the input light corresponds to a preset overlap ratio.

Next, in step S12, the frame difference calculation unit (120) performs a calculation that takes into account the overlap ratio based on the first frame information obtained by the odd pixels (1) and the second frame information obtained by the even pixels (2) among the frame information obtained through step S11, thereby deriving the frame difference (Frame Difference, FD) due to the input light.

Specifically, in step S12, the frame difference calculation unit (120) can calculate the frame difference by performing an operation to subtract twice the second frame information from the first frame information or by performing an operation to subtract twice the first frame information from the second frame information.

Next, in step S13, the event detection unit (130) can compare the frame difference derived through step S12 with a preset threshold value to determine whether an event has occurred.

Next, in step S14, the event detection unit (130) can generate a reference voltage according to the occurrence of the event determined in step S13.

Next, in step S15, the time memory processing unit (140) can store information about the reference voltage in the TM pixel (3).

Next, in step S16, the time memory processing unit (140) can convert the output of the TM pixel (3) to output a time series image reflecting information about the frame difference.

In the above description, steps S11 to S16 may be further divided into additional steps or combined into fewer steps, depending on the implementation example of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be changed.

The method for detecting movement and movement path according to one embodiment of the present invention may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention or may be those known to those skilled in the art of computer software and available for use. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, flash memories, etc. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Additionally, the aforementioned motion and motion path detection method can also be implemented in the form of a computer program or application executed by a computer and stored in a recording medium.

The above description of the present invention is for illustrative purposes only, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and likewise, components described as distributed may be implemented in a combined manner.

The scope of the present application is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present application.

10: Vision sensor having motion and motion path detection device
100: Motion and motion path detection device
11: Patch-based pixel circuit
110: Frame information acquisition section
12: Frame difference processing circuit
120: Frame difference calculation unit
130: Event detection unit
140: Time memory processing unit
13: Converter circuit
14: Switch circuit
1: Odd pixels
2: Even pixels
3: TM Pixel

Claims (15)

In a motion and motion path detection device,
A frame information acquisition unit that acquires frame information for the input light by using the pair of odd pixels and even pixels, which comprises a pair of odd pixels and even pixels, wherein the integration time for input light of the odd pixels and the integration time for input light of the even pixels overlap at least partially, and the degree of overlapping of the integration times corresponds to a preset overlap ratio; and
A frame difference calculation unit that derives the frame difference due to the input light by performing an operation to calculate the frame difference (Frame Difference, FD) between the first frame information and the second frame information based on the first frame information acquired by the odd pixels among the frame information and the second frame information acquired by the even pixels among the frame information according to the overlapping ratio.
A sensing device comprising: In the first paragraph,
If the above overlap ratio is preset to 50%,
The above frame difference calculation unit,
A detection device that calculates the frame difference by performing an operation of subtracting twice the second frame information from the first frame information or by performing an operation of subtracting twice the first frame information from the second frame information. In the first paragraph,
An event detection unit that compares the above frame difference with a preset threshold value to determine whether an event has occurred and generates a reference voltage according to the occurrence of the event;
A sensing device further comprising: In the third paragraph,
A sensing device, characterized in that the magnitude of the reference voltage decreases due to leakage current over time based on the time at which the event occurs. In paragraph 4,
A detection device, wherein if the event is detected again while the reference voltage is decreasing due to the leakage current, the size of the reference voltage is updated to a preset initial value. In the third paragraph,
A time memory processing unit including a TM pixel storing information about the above reference voltage,
A sensing device further comprising: In Article 6,
A sensing device, characterized in that each of the above pair of odd pixels and even pixels is arranged to surround the TM pixel, and the odd pixels and the even pixels are arranged alternately. In a vision sensor having a motion and motion path detection device,
A patch-based pixel circuit having a plurality of odd pixels and even pixels, wherein the integration time for input light of the odd pixels and the integration time for input light of the even pixels overlap at least partially, and the degree of overlap of the integration times corresponds to a preset overlap ratio; and
A frame difference processing circuit that derives the frame difference by the input light by performing an operation to calculate the frame difference (Frame Difference, FD) between the first frame information and the second frame information based on the first frame information acquired by the odd pixels and the second frame information acquired by the even pixels according to the overlapping ratio, and generates a reference voltage according to the result of comparing the frame difference with a preset threshold value.
Including, but not limited to,
The above patch-based pixel circuit,
A vision sensor further comprising a TM pixel storing information about the above reference voltage. In Article 8,
A converter circuit that converts the output of the TM pixel to output a time series image reflecting information about the frame difference in response to a preset first driving mode;
A vision sensor further comprising: In Article 9,
The above converter circuit,
Converting the output of the odd pixels, the output of the even pixels, and the output of the TM pixels to output a normal image according to a predetermined reference point in time in response to a preset second driving mode,
A switch circuit for switching between the first driving mode and the second driving mode,
A vision sensor further comprising: In a method for detecting movement and movement path,
A step of obtaining frame information for the input light by using a pair of odd pixels and even pixels, wherein the integration time for the input light of odd pixels and the integration time for the input light of even pixels overlap at least partially, and the degree of overlapping of the integration times corresponds to a preset overlap ratio; and
A step of deriving the frame difference due to the input light by performing an operation to calculate the frame difference (Frame Difference, FD) between the first frame information and the second frame information based on the first frame information acquired by the odd pixels among the frame information and the second frame information acquired by the even pixels among the frame information according to the overlapping ratio.
A detection method comprising: In Article 11,
If the above overlap ratio is preset to 50%,
The steps for deriving the above frame difference are:
A detection method for calculating the frame difference by performing an operation of subtracting twice the second frame information from the first frame information or by performing an operation of subtracting twice the first frame information from the second frame information. In Article 11,
A step of comparing the above frame difference with a preset threshold value to determine whether an event has occurred and generating a reference voltage according to the occurrence of the event;
A detection method further comprising: In Article 13,
The magnitude of the above reference voltage is reduced by the leakage current over time from the time the above event occurred.
A detection method, wherein if the event is detected again while the reference voltage is decreasing due to the leakage current, the size of the reference voltage is updated to a preset initial value. In Article 13,
A step of storing information about the above reference voltage in a TM pixel; and
A step of converting the output of the above TM pixel to output a time series image reflecting information about the frame difference,
A detection method further comprising:

Images