WO2024247481A1 - Imaging device, imaging system, and imaging device control method - Google Patents

Abstract

The present invention suppresses performance deterioration when sparsity is improved in an imaging device that uses a neural network.　The imaging device includes a predetermined number of output-side neurons and a statistic calculation unit. In this imaging device, each of the output-side neurons generates an output signal each time a predetermined unit time elapses on the basis of an input signal from an intermediate layer. The statistic calculation unit calculates a statistic of an output signal generated within a predetermined frame period longer than the unit time, for each output-side neuron.

Description

Image capture device, image capture system, and method for controlling image capture device

This technology relates to an imaging device. More specifically, it relates to an imaging device with a neural network structure, an imaging system, and a method for controlling an imaging device.

In the past, imaging devices that dynamically switch modes have been used in various fields such as crime prevention and traffic. For example, an imaging device has been proposed that captures a low-resolution image in a feature detection mode to detect the features of a subject, and when a specific feature is detected, switches to imaging mode to capture a high-resolution image (see, for example, Patent Document 1).

JP 2021-150814 A

In the conventional technology described above, a low-resolution image is captured until a specific feature is detected, thereby reducing power consumption compared to constantly capturing high-resolution images. However, when using neural networks such as SNN (Spiking Neural Networks) to detect features, there is a problem in that the more the sparsity of the neurons is improved, the more the performance deteriorates.

This technology was developed in light of these circumstances, and aims to suppress the performance degradation that occurs when sparsity is improved in imaging devices that use neural networks.

The present technology has been made to solve the above-mentioned problems, and its first aspect is an imaging device equipped with a simplified detector that includes a predetermined number of output side neurons that generate an output signal every time a predetermined unit time elapses based on an input signal from an intermediate layer, and a statistics calculation unit that calculates, for each of the output side neurons, the statistics of the output signals generated within a predetermined frame period that is longer than the unit time, and a control method for the imaging device. This has the effect of suppressing performance degradation when sparsity is improved.

In addition, in this first aspect, a function calculation unit may be further provided that inputs each of the above statistical quantities into a predetermined function and outputs a calculated value obtained. This provides the effect of outputting the calculated value as the detection result.

In addition, in this first aspect, a control unit may be further provided that controls the length of the frame period in accordance with the detection result of the simple detector. This provides the effect of setting an appropriate frame period.

In addition, in this first aspect, a control unit may be further provided that controls the threshold of the neuron in the simple detector in accordance with the detection result of the simple detector. This provides the effect of controlling the firing rate of the neuron.

In addition, in this first aspect, an image sensor that generates image data may be further provided, and the simple detector may detect a predetermined detection target from the image data by analysis. This provides the effect of detecting the detection target.

In addition, in this first aspect, the system may further include a detailed detector that detects a predetermined detection target from the image data by a more detailed analysis than the simple detector, an evaluator that generates an evaluation value from the image data by an analysis different from that of the detailed detector, and a rule selection unit that selects a rule that controls the operation of the detailed detector and the evaluator based on the detection result of the simple detector. This provides the effect of selecting a rule based on the detection result.

Furthermore, in this first aspect, the simple detector and the detailed detector may be disposed within the neural network circuit. This has the effect of reducing the circuitry outside the neural network circuit.

Furthermore, in this first aspect, the simple detector, the detailed detector, and the evaluator may be arranged within a neural network circuit. This has the effect of reducing circuits outside the neural network circuit.

In addition, in this first aspect, an input control unit may be further provided that controls the operation of the image sensor based on the detection result of the simple detector. This has the effect of reducing the amount of processing by the simple detector.

In addition, in this first aspect, an environmental sensor that outputs a sensor signal different from the image data may be further provided, and the rule selection unit may select the rule based on the sensor signal and the detection result of the simple detector. This provides the effect of improving robustness.

In addition, in this first aspect, the simplified detector may be disposed within an image sensor that generates image data. This has the effect of reducing the amount of circuitry outside the image sensor.

In addition, in this first aspect, the neurons in the simplified detector may be spiking neurons. This has the effect of making it possible to make information sparsifying in both time and space.

The second aspect of this technology is an imaging device in which a difference calculator that calculates the difference between the sum of a predetermined number of input signals and a feedback signal, an integrator that integrates the difference and outputs the integral value, a limiting unit that outputs the integral value as an output signal if the integral value is outside a predetermined range, a comparator that compares the output signal with a predetermined value and supplies the comparison result, and a feedback unit that converts the comparison result into the feedback signal and feeds it back, are arranged in at least one of multiple neurons in a neural network. This has the effect of suppressing performance degradation when sparsity is improved.

The third aspect of the present technology is an imaging system including a simple detector having a predetermined number of output side neurons that generate an output signal every time a predetermined unit time elapses based on an input signal from an intermediate layer, and a statistics calculation unit that calculates, for each of the output side neurons, the statistics of the output signals generated within a predetermined frame period that is longer than the unit time, and an analysis device that analyzes an image based on the detection results of the simple detector. This has the effect of suppressing the degradation of system performance when sparsity is improved.

1 is a block diagram showing a configuration example of an imaging device according to a first embodiment of the present technology; 1 is a block diagram of an imaging device in which a plurality of imaging units are arranged according to a first embodiment of the present technology. 1 is a block diagram of an imaging device in a case where circuits other than an image sensor are arranged in a processor according to a first embodiment of the present technology; 1 is a block diagram showing a configuration example of an SNN circuit and a router according to a first embodiment of the present technology; 2 is a block diagram showing a configuration example of a core according to the first embodiment of the present technology; FIG. 1 is a diagram showing an example of a neural network structure of a simple detector in a first embodiment of the present technology; FIG. 1 is a block diagram showing a configuration example of a ΔΣ neuron according to a first embodiment of the present technology; 1 is a diagram illustrating an example of an input signal and an output signal of a ΔΣ neuron in the first embodiment of the present technology. FIG. FIG. 2 is a block diagram showing an example of the configuration of a ΔΣ neuron in a first comparative example. 11 is a graph showing an example of an output signal and a waveform of a comparison result when a sine wave is input in a first comparative example and the first embodiment of the present technology; 11 is a graph showing an example of an output signal and a waveform of a comparison result when a sparse signal is input in a first comparative example and the first embodiment of the present technology; 11 is a graph showing an example of an output signal and a waveform of a comparison result when a stationary signal is input in a first comparative example and the first embodiment of the present technology; 1 is a block diagram showing a configuration example of a simplified detector in a first embodiment of the present technology and a second comparative example; 4 is a diagram illustrating an example of an output value of an output side neuron in the first embodiment of the present technology. FIG. FIG. 11 is a diagram showing an example of an output format in a second comparative example. FIG. 2 is a diagram showing an example of an output format according to the first embodiment of the present technology. 4 is a flowchart showing an example of an operation of the analysis device according to the first embodiment of the present technology. 11 is a block diagram showing a configuration example of an imaging unit in a first modified example of the first embodiment of the present technology. FIG. 13 is a block diagram showing a configuration example of an imaging device according to a second embodiment of the present technology. FIG. FIG. 13 is a block diagram showing an example configuration of an imaging device according to a third embodiment of the present technology. 13 is a flowchart showing an example of an operation of an analysis device according to a third embodiment of the present technology. FIG. 13 is a block diagram showing an example configuration of an imaging device according to a fourth embodiment of the present technology. FIG. 13 is a block diagram showing an example configuration of an imaging device according to a fifth embodiment of the present technology. 13 is a diagram illustrating an example of a layered structure of an image sensor according to a fifth embodiment of the present technology. FIG. FIG. 23 is a block diagram showing an example configuration of an imaging device according to a sixth embodiment of the present technology. 1 is a block diagram showing a schematic configuration example of a vehicle control system; FIG. 4 is an explanatory diagram showing an example of an installation position of an imaging unit.

Hereinafter, modes for carrying out the present technology (hereinafter, referred to as embodiments) will be described in the following order.
1. First embodiment (example of calculating statistics within a frame period)
2. Second embodiment (an example in which a simple detector, a detailed detector, and an evaluator for calculating statistics within a frame period are provided in an SNN circuit)
3. Third embodiment (example in which statistics within a frame period are calculated and an input control unit controls an image sensor)
4. Fourth embodiment (an example in which a simple detector and a detailed detector for calculating statistics within a frame period are provided in an SNN circuit)
5. Fifth embodiment (an example in which a simple detector and a detailed detector for calculating statistics within a frame period are provided in an image sensor)
6. Sixth embodiment (example of selecting a rule based on a detection result of a simple detector that calculates statistics within a frame period and a sensor signal)
7. Examples of applications to moving objects

1. First embodiment
[Configuration example of imaging device]
1 is a block diagram showing an example of a configuration of an imaging device 100 according to an embodiment of the present technology. The imaging device 100 is a device that captures image data, and includes an imaging section 110 and an analysis device 130. The imaging section 110 includes an image sensor 120 and a simple detector 200.

The image sensor 120 generates image data by photoelectric conversion. For example, the image sensor 120 determines whether the amount of light has changed for each pixel address, and generates data that arranges the determination results as image data. Such a sensor is called an EVS (Event-based Vision Sensor) or DVS (Dynamic Vision Sensor). By using an EVS, the input to the simple detector 200 becomes sparse, and even if the input undergoes a significant periodic change, it can be further sparsed within the simple detector 200.

The image sensor 120 is not limited to an EVS, and can also generate a grayscale signal for each pixel according to the amount of light and generate image data by arranging these. When generating a grayscale signal, the image sensor 120 generates an analog signal for each pixel and converts it into a digital signal using an AD (Analog to Digital) converter. Alternatively, the image sensor 120 has a SPAD (Single-Photon Avalanche Diode) for each pixel and counts pulse signals from the SPAD to generate a digital signal.

In addition, the image sensor 120 switches the output destination of the image data to either the simple detector 200 or the analysis device 130 based on the detection results of the simple detector 200.

The image sensor 120 can also change the resolution of the image data it generates based on the detection results of the simple detector 200. The resolution is changed by pixel addition, thinning readout, or the like. In this case, for example, the image sensor 120 lowers the resolution of the image data sent to the simple detector 200 and increases the resolution of the image data sent to the analysis device 130.

The simple detector 200 detects a specific detection target from image data through analysis. The function of the simple detector 200 is implemented in hardware, for example, in an SNN circuit 500. The SNN circuit 500 is an example of a neural network circuit as described in the claims.

In the simple detector 200, for example, a person or a specific preparatory movement is set as a detection target. The simple detector 200 detects the presence or absence of a detection target (such as a person) and supplies the detection result to the analysis device 130 and the image sensor 120. When the detection result is a predetermined value P, the image sensor 120 changes the output destination of the image data to the analysis device 130. Here, P is not limited to one, and may be two or more, such as _P0 and _P1 . A detection target is assigned to each of _P0 and _P1 . The detection result being the predetermined value P indicates that a detection target corresponding to that value has been detected.

The analysis device 130 includes an input control unit 131, a rule selection unit 132, a detail detector 133, and an evaluator 134.

The input control unit 131 transmits and receives data between the outside of the analysis device 130 and the rule selection unit 132, the detail detector 133, and the evaluator 134. This input control unit 131 supplies the detection results from the simple detector 200 to the rule selection unit 132. The input control unit 131 also supplies image data from the image sensor 120 to at least one of the detail detector 133 and the evaluator 134. The input control unit 131 also supplies the detection results from the detail detector 133 to the rule selection unit 132, and transmits logs and alerts from the detail detector 133 and the evaluator 134 to an external server, etc.

The rule selection unit 132 selects a rule that controls the operation of the detailed detector 133 and the evaluator 134 based on the detection results of the simple detector 200. For example, various rules are envisioned, such as "activate the evaluator 134 when a specific detection target is detected by the detailed detector 133" or "output an alert to the outside when a specified evaluation value is obtained by the evaluator 134." The rule selection unit 132 controls the operation of the detailed detector 133 and the evaluator 134 according to the rule selected from among them.

For example, in a certain rule, the rule selection unit 132 stops the operation of the detailed detector 133 and the evaluator 134 when the detection result of the simple detector 200 is not the predetermined value P. On the other hand, when the detection result of the simple detector 200 is the predetermined value P, the rule selection unit 132 causes the detailed detector 133 to start analysis while keeping the evaluator 134 stopped. Then, when the detection result of the detailed detector 133 is the predetermined value Q, the rule selection unit 132 causes the evaluator 134 to start analysis. Here, Q is not limited to one, and may be two or more, such as _Q0 or _Q1 . The detection result being the predetermined value Q indicates that a detection target corresponding to that value has been detected.

The detailed detector 133 detects a specific detection target from image data using a more detailed analysis than the simple detector 200. For example, the presence or absence of people, the number of people, the presence or absence of a specific motion, the location of people within the image, etc. are set as detection targets. Furthermore, when a person is detected, the detailed detector 133 can also detect whether or not it is a specific individual. The detailed detector 133 may perform these analyses on a single piece of image data, or on a video that includes multiple pieces of image data captured in succession.

The detail detector 133 outputs the detection result to the input control unit 131. In addition, when the detail detector 133 detects a specific individual, it can generate a log including the shooting time and output the log to the input control unit 131.

The evaluator 134 generates an evaluation value from image data using a different analysis from that of the detail detector 133. For example, the evaluator 134 analyzes what actions one or more people are performing and what state they are in, and generates an evaluation value that indicates the results. The evaluator 134 may perform this analysis on a single piece of image data, or on a video.

For example, in a certain rule, when the evaluation value is a predetermined value R, the evaluator 134 outputs a log including the shooting time and an alert to the input control unit 131. Here, R is not limited to one, and may be two or more, such as _R0 or _R1 . The evaluation value being a predetermined value P indicates that an analysis object corresponding to that value has been detected.

As described above, a system in which the simple detector 200 continuously detects the detection target and operates the detailed detector 133 and the evaluator 134 when the detection target is detected is called an Always-ON system. The Always-ON system can reduce power consumption while maintaining performance, compared to a system in which the detailed detector 133 and the evaluator 134 are always operated.

In the figure, the imaging unit 110 and the analysis device 130 are arranged in one device, but this configuration is not limited to this. They can also be distributed and arranged in multiple devices. Furthermore, a system in which the imaging unit 110 and the analysis device 130 are arranged is an example of an imaging system described in the claims.

In addition, in the figure, one imaging unit 110 is arranged, but this configuration is not limited. As shown in the example of FIG. 2, multiple imaging units such as imaging units 110-1, 110-2, etc. can also be arranged.

Also, as shown in FIG. 3, the simplified detector 200 and the analysis device 130 can be arranged within a single processor 600.

[Example of SNN circuit configuration]
4 is a diagram showing an implementation example of an SNN circuit 500 according to the first embodiment of the present technology. The SNN circuit 500 includes, for example, an input/output interface 560 and a multi-core array 570.

The input/output interface 560 transmits and receives data between the outside and the multi-core array 570. This input/output interface 560 supplies image data from the image sensor 120 to the multi-core array 570, and supplies the detection results from the multi-core array 570 to the analysis device 130.

In the multi-core array 570, multiple cores 590 are arranged in a two-dimensional lattice. A router 580 is placed adjacent to each core 590.

The router 580 controls the data path. This router 580 includes, for example, FIFO (First In, First Out) memories 581 to 585 and an arbiter 586. In the figure, "E" indicates the east direction of the router 580 in question, and "S" indicates the south direction. "W" indicates the west direction, and "N" indicates the north direction. "L" indicates the direction toward the core 590 adjacent to the router 580.

FIFO memory 581 holds data from the east direction in a first-in-first-out manner and outputs a request to arbiter 586. FIFO memory 582 holds data from the south direction in a first-in-first-out manner and outputs a request to arbiter 586. FIFO memory 583 holds data from the west direction in a first-in-first-out manner and outputs a request to arbiter 586. FIFO memory 584 holds data from the north direction in a first-in-first-out manner and outputs a request to arbiter 586. FIFO memory 585 holds data from adjacent core 590 in a first-in-first-out manner and outputs a request to arbiter 586.

The arbiter 586 arbitrates between requests from each of the FIFO memories 581 to 585 and returns a response. When a response is received, the FIFO memory outputs data to either the east, west, north, or south adjacent core 590 via the arbiter 586.

FIG. 5 is a block diagram showing an example of the configuration of a core 590 in the first embodiment of the present technology. The core 590 includes a core router 591, a neuron I/O (Input/Output) 592, a multiply-accumulate unit 593, a work memory 594, a membrane potential memory 595, and an LIF (Leaky Integrate and Fire) unit 596.

The core router 591 supplies data from adjacent routers 580 to the neuron I/O 592 and supplies data from the LIF unit 596 to the adjacent routers 580.

The multiply-and-accumulate unit 593 uses the work memory 594 to integrate the data from the neuron I/O 592. The membrane potential memory 595 holds the membrane potential obtained by the integration. The LIF unit 596 determines whether the membrane potential has exceeded a predetermined threshold and fired (in other words, a spike has occurred), and supplies the result to the core router 591.

FIG. 6 is a diagram showing an example of a neural network structure of the simplified detector 200 in the first embodiment of the present technology. In the figure, a shows an example of a neural network structure, and b shows an example of a two-layer structure of a in the figure. The circle in b in the figure indicates a neuron.

As shown in FIG. 1A, the neural network in the simplified detector 200 is, for example, a CNN (Convolutional Neural Network), which is composed of multiple layers such as convolutional layers 211 and 212 and a pooling layer 213. Each of the neurons in the first convolutional layer 211 receives the pixel signals of one or more pixels in the image data 700. Note that the neural network is not limited to a CNN.

As shown in FIG. 1B, for example, a predetermined number of ΔΣ neurons 300 are arranged in the convolution layer 212. In FIG. 1B, for simplicity, the channel direction is omitted and the diagram is shown in two dimensions, but the same applies to three-dimensional representation.

As shown in FIG. 3B, a neural network has an array of multiple neurons including one or more ΔΣ neurons 300. As these neurons, for example, spiking neurons in an SNN are used. Note that general neurons in an ANN (Artificial Neural Network) can also be used. Spiking neurons and general neurons can also be combined.

[Example of the configuration of a ΔΣ neuron]
7 is a block diagram showing an example of a configuration of a ΔΣ neuron 300 according to the first embodiment of the present technology. The ΔΣ neuron 300 includes a difference calculator 310, an integrator 320, a limiting section 330, a comparator 340, and a feedback section 350.

The difference calculator 310 calculates the difference B between the sum of a predetermined number of input signals and a feedback signal E from the feedback section 350, and supplies the difference B to the integrator 320. For example, the outputs of the previous neuron are _in0 , _in1 , and _in2 , and the weights are _ω0 , _ω1, and _{ω2 , and in0×ω0 , in1} × _ω1 , and _in2 × _ω2 are input as input signals.

The integrator 320 integrates the difference B and supplies the integral value C to the limiting unit 330.

When the integral value C is outside a predetermined limit range, the limiting unit 330 outputs the value as an output signal OUT to the subsequent pooling layer 213 and the comparator 340. For example, the upper limit threshold of the limit range is Th ₀ , and the lower limit threshold is Th _1. If the integral value C is equal to or greater than Th ₀ and equal to or less than Th ₁ , the limiting unit 330 outputs an output signal OUT of "0". The output signal OUT being "0" means that a significant output signal is not output and is limited. On the other hand, when the integral value C is less than Th ₀ or greater than Th ₁ , the limiting unit 330 outputs the integral value C as it is as the output signal OUT.

Comparator 340 compares output signal OUT with a predetermined comparison threshold TH and supplies a comparison result D to feedback section 350. For example, TH is set to "0", and comparator 340 outputs "1" as the comparison result when output signal OUT is greater than TH, and outputs "0" as the comparison result when output signal OUT is equal to or less than TH.

The feedback unit 350 converts the comparison result D into a feedback signal E and feeds it back to the difference calculator 310. For example, when the comparison result D is greater than "0", the feedback unit 350 outputs a feedback signal E of "-1", and when the comparison result D is "0", it outputs a feedback signal E of "1".

The above-mentioned ΔΣ neuron 300 calculates and outputs an output signal OUT at regular time intervals in synchronization with a clock signal of a predetermined processing frequency. In the above-mentioned configuration, the time derivative of the integral value C is expressed by, for example, the following equation.
dC/dt=-(C-a)/T+Σ(ωj×inj)-Δ(оut)
...Equation 1
In the above equation, t indicates time, a indicates a coefficient, T indicates a time constant, and Δ( ) indicates the comparison result of the comparator 340.

The value of the output signal OUT is expressed by the following equation.
OUT=f(C)...Formula 2
In the above equation, f( ) denotes the activation function of the limiting unit 330 .

In addition, both the input signal and the output signal of the ΔΣ neuron 300 are digital signals. Note that the input signal may be an analog signal and the output signal may be a digital signal. In this case, a 1-bit DAC (Digital to Analog Converter) or the like is used as the feedback section 350. Also, both the input signal and the output signal may be analog signals.

FIG. 8 is a diagram showing an example of an input signal and an output signal of the ΔΣ neuron 300 in the first embodiment of the present technology. In the figure, a shows an example of the trajectory of the input signal over time, and b shows an example of the trajectory of the output signal.

As shown in a and b in the figure, an input signal is input for each unit time, and an output signal is calculated. These input and output signals are, for example, spike signals of n bits (n is an integer). Such neurons are called spiking neurons. Note that the ΔΣ neuron 300 is not limited to spiking neurons, and may be a general neuron whose input and output signals are not spike signals.

Here, we consider a typical ΔΣ neuron without a limiting section 330 as a first comparative example.

FIG. 9 is a block diagram showing an example of the configuration of a ΔΣ neuron in the first comparative example. As shown in the figure, in the first comparative example, the limiting unit 330 is not provided, and the integral value is output as is as the output signal OUT and input to the comparator 340. In this first comparative example, the firing rate of the neuron is controlled by the comparison threshold TH of the comparator 340. Increasing the comparison threshold TH reduces the firing rate, and accordingly, the sparsity can be improved.

In the aforementioned Always ON system, by making the data, including the input from the image sensor 120, sparse as much as possible in both space and time, the Always ON system can operate more efficiently.

In addition, while an ANN can only sparsify processing in the time direction by changing the processing frequency, an SNN can sparsify information in both time and space by preventing each neuron from sending unnecessary information. Unlike an ANN, an SNN can sparsify information spatially, so it is possible to achieve the same level of sparsity as a slow ANN while maintaining the processing frequency.

Therefore, by using an SNN as the simple detector 200, it is possible to maintain the processing frequency while improving processing efficiency.

However, there is a trade-off between sparsity and performance. In the first comparative example, increasing the comparison threshold TH for each neuron reduces the firing rate of the neuron and improves sparsity, but the amount of information transmitted becomes insufficient, resulting in increased errors in the analysis results.

In this first comparative example, the only threshold to be controlled is the comparison threshold TH, whereas in the ΔΣ neuron 300 provided with the limiting unit 330, two types of thresholds can be controlled: the comparison threshold TH and the thresholds of the limiting unit 330 (upper and lower thresholds). In this configuration, the sparsity can be improved by widening the limiting range from the lower threshold to the upper threshold. On the other hand, information loss can be minimized by lowering the comparison threshold TH. Therefore, by outputting the integral value C through the limiting unit 330 rather than directly, information loss when sparsity is increased can be minimized. This makes it possible to suppress performance degradation when sparsity is increased.

FIG. 10 is a graph showing an example of the waveforms of the output signal and the comparison result when a sine wave is input in the first comparative example and the first embodiment of the present technology. In the figure, "a" is an example of the waveforms of the input signal and the output signal when a sine wave is input to the neuron of the first comparative example, and "b" is an example of the waveforms of the input signal and the comparison result in that case.

In addition, c in the figure is an example of the waveforms of the input signal and the output signal when a sine wave is input to the ΔΣ neuron 300 with the limiting unit 330, and d in the figure is an example of the waveforms of the input signal and the comparison result in that case.

The dotted lines a to d in the figure indicate the waveform of the input signal. The solid line a in the figure indicates the waveform of the output signal (i.e., the integral value), and the solid line b in the figure indicates the waveform of the comparison result. The solid line c in the figure indicates the waveform of the output signal from the limiting section 330, and the solid line d in the figure indicates the waveform of the comparison result. The same applies to Figures 11 and 12.

As shown in Fig. 10A to 10D, in a configuration with a limiting unit 330, the compression rate is higher for information with periodicity such as sine waves, and information loss is also smaller than in the first comparative example. Note, however, that information loss is greater for inputs that fluctuate in a complex manner.

FIG. 11 is a graph showing an example of the waveforms of the output signal and the comparison result when a sparse signal is input in the first comparative example and the first embodiment of the present technology. In the figure, "a" is an example of the waveforms of the input signal and the output signal when a sparse signal is input to the neuron of the first comparative example, and "b" is an example of the waveforms of the input signal and the comparison result in that case.

In addition, c in the figure is an example of the waveforms of the input signal and the output signal when a sparse signal is input to the ΔΣ neuron 300 with the limiting unit 330, and d in the figure is an example of the waveforms of the input signal and the comparison result in that case.

As shown in the examples a to d in the figure, in a configuration with a limiting unit 330, the compression rate when a sparse signal is input is almost the same as in the first comparative example.

FIG. 12 is a graph showing an example of the waveforms of the output signal and the comparison result when a stationary signal is input in the first comparative example and the first embodiment of the present technology. In the figure, "a" is an example of the waveforms of the input signal and the output signal when a stationary signal is input to the neuron of the first comparative example, and "b" is an example of the waveforms of the input signal and the comparison result in that case.

In addition, c in the figure is an example of the waveforms of the input signal and the output signal when a stationary signal is input to the ΔΣ neuron 300 with the limiting unit 330, and d in the figure is an example of the waveforms of the input signal and the comparison result in that case.

As shown in the examples a to d in the figure, in a configuration with a limiting unit 330, the compression ratio when a stationary signal is input is almost the same as in the first comparative example.

Next, we will explain the output format of the output layer in the neural network of the simple detector 200.

FIG. 13 is a block diagram showing an example of the configuration of the simple detector 200 in the first embodiment of the present technology and the second comparative example. In the figure, a is a block diagram showing an example of the configuration of the simple detector 200 in the first embodiment.

The neural network in the simple detector 200 comprises an input layer 210, an intermediate layer 220, and an output layer 230. The input layer 210 processes image data from the image sensor 120 and supplies an array of spike signals to the intermediate layer 220. The intermediate layer 220 processes the array of spike signals and supplies the array of spike signals to the output layer 230. This output layer 230 comprises a predetermined number of output neurons 231, a statistical calculation unit 232, and a function calculation unit 233.

Each output neuron 231 generates an output signal every time a unit time elapses based on an input signal from the intermediate layer, and supplies the output signal to the statistical calculation unit 232. If the number of output neurons 231 is M (M is an integer), an array of M output signals is output every unit time. The length of the unit time is, for example, 1 millisecond.

The statistical quantity calculation unit 232 calculates the statistical quantities of the output signals generated within a predetermined frame period that is longer than the unit time for each output neuron 231. The maximum value, sum, and average value are calculated as the statistical quantities. The statistical quantity calculation unit 232 supplies the array of M statistical quantities as Logit to the function calculation unit 233.

The function calculation unit 233 inputs Logit into a predetermined function (such as a softmax function) and outputs the solution value (in other words, the calculated value) obtained as the detection result to the analysis device 130.

Here, a configuration in which the statistical quantity calculation unit 232 is not arranged is considered as a second comparative example. In the same figure, b is a block diagram showing an example of a configuration of the output layer 230 in the second comparative example. In the second comparative example, the array of output signals from the output side neurons 231 is input to the function calculation unit 233 as Logit.

In the figure, the function calculation unit 233 outputs the solution as the detection result, but this configuration is not limited. As will be described later, the simple detector 200 can also output an array of statistics (Logit) to the analysis device 130 without calculating the solution.

FIG. 14 is a diagram showing an example of the output value of the output neuron 231 in the first embodiment of the present technology. In the figure, a shows an example of the output value for each output neuron 231 at a certain time t. For example, eight output neurons 231 are arranged, and their respective identification numbers are "0" to "7." The circles in the figure, a, indicate the output neurons 231. The output values of these neurons are, for example, n-bit spike values.

B in the figure shows the output values for each output neuron 231 in chronological order. As shown in b in the figure, each of the eight output neurons 231 outputs an output value (such as a spike value) per unit time. The numbers in parentheses in b in the figure indicate the identification numbers of the output neurons 231.

15 is a diagram showing an example of an output format in the second comparative example. At time _t9 , an array of eight output values (i.e., Logit) is input to the function calculation unit 233. The function calculation unit 233 outputs, for example, the identification number of the output neuron 231 corresponding to the largest output value among the Logits as a solution. After time _t10 , the same process is repeatedly executed for each unit time.

In the second comparative example, if the sparsity of spike signals in an SNN is improved in the time direction, the amount of zero information increases when the function calculation unit 233 calculates a solution, and there is a risk that an appropriate solution cannot be obtained.

For example, if we assume that "7" is the correct answer during the period from time _t9 to _t20 , then at time _t13 , the function calculation unit 233 erroneously outputs "0." At times _t17 , _t18 , and _t20 , the output value of the output neuron 231 with the identification number "7" does not reach its maximum, and an incorrect answer is output.

To obtain an appropriate solution, the output layer 230 needs to accumulate non-zero information for a sufficiently long period of time, but in this case, the output frequency of the simple detector 200 decreases and delays increase. This is also undesirable because the evaluation of the downstream analysis device 130 will be delayed or inaccurate.

FIG. 16 is a diagram showing an example of an output format in the first embodiment of the present technology. As illustrated in the figure, in the first embodiment, a frame period (such as 4×unit time) longer than the unit time is set. The statistics calculation unit 232 calculates the statistics (such as the maximum value) of the output signal generated within the frame period for each output neuron 231. For example, within a frame period including time t ₉ to t ₁₂ , the output neuron 231 with the identification number “7” outputs four output values in sequence. If the maximum value of these is “8”, the statistics calculation unit 232 acquires “8” as the statistics of that neuron. The statistics calculation unit 232 also obtains statistics for other neurons and supplies the array of the eight statistics to the function calculation unit 233 as Logit. The same process is performed in the next frame period including time t ₁₃ to t _16. The function calculation unit 233 outputs the identification number of the output neuron 231 corresponding to the largest output value of the Logit as a solution.

In the second comparative example, an incorrect answer was output at time _t13 , but as illustrated in the figure, the output layer 230 is able to calculate statistics and output the correct answer, "7", in the frame period including _t13 . Similarly, at times _t17 , _t18 , and _t20 , the output layer 230 is able to output the correct answer.

As shown in the figure, by calculating statistics, it is possible to suppress the output of erroneous solutions when the sparsity is improved along the time axis. In addition, since there is no need to accumulate non-zero information for a long period of time, it is possible to suppress a decrease in output frequency and an increase in delay compared to the second comparative example. This makes it possible to suppress a decrease in performance when the sparsity is improved.

[Example of analysis device operation]
17 is a flowchart showing an example of the operation of the analysis device 130 according to the first embodiment of the present technology. This operation is started, for example, when the simple detector 200 outputs a detection result.

The rule selection unit 132 determines whether the detection result of the simple detector 200 is a predetermined value P (in other words, whether a predetermined detection target has been detected) (step S901). If the detection result is not P (step S901: No), the rule selection unit 132 returns to step S901.

On the other hand, if the detection result is P (step S901: Yes), the rule selection unit 132 selects a rule based on the detection result and, for example, starts the detail detector 133 based on that rule (step S902).

The rule selection unit 132 determines whether the detection result of the detailed detector is a predetermined value Q (step S903). If the detection result is not Q (step S903: No), the rule selection unit 132 repeats steps S901 and onward.

On the other hand, if the detection result is Q (step S903: Yes), the rule selection unit 132, for example, starts the evaluator 134 and determines whether the evaluation value is a predetermined value R (step S904). If the evaluation value is not R (step S904: No), the rule selection unit 132 repeats steps S901 and onward.

On the other hand, if the evaluation value is R (step S904: Yes), the evaluator 134 records a log and outputs an alert (step S905), and the analysis device 130 ends the analysis operation.

The rules shown in steps S903 to S905 above are examples of rules selected in step S902. The content of the rules is not limited to the example shown in the figure. For example, the rule selection unit 132 may activate only one of the detail detector 133 and the evaluator 134 when the detection result is P. Alternatively, the evaluator 134 may only output an alert in step S905.

In this way, according to the first embodiment of the present technology, the statistics calculation unit 232 calculates the statistics of the output signal generated within a frame period for each output neuron 231, so that it is possible to suppress a decrease in performance due to an erroneous solution when improving sparsity.

In addition, the ΔΣ neuron 300 includes a limiting unit 330 that outputs the integral value as an output signal when the integral value is outside a specified range, so that information loss when sparsity is improved can be minimized.

[Modification]
In the first embodiment described above, the statistics calculation unit 232 calculates statistics within a frame period, but the frame period can also be made variable. The imaging device 100 of this first embodiment differs from the first embodiment in that the length of the frame period is controlled in accordance with the output of the simple detector 200.

FIG. 18 is a block diagram showing an example configuration of the imaging unit 110 in a first modified example of the first embodiment of the present technology. The imaging unit 110 in the first modified example of the first embodiment differs from the first embodiment in that it further includes a control unit 510. The control unit 510 is provided, for example, in the SNN circuit 500.

The control unit 510 controls the length of the frame period according to the detection result of the simple detector 200 (i.e., the solution of the function calculation unit 233). For example, the control unit 510 shortens the frame period to less than a predetermined value when the solution is smaller than a predetermined set value, and lengthens the frame period to less than the predetermined value when the solution is equal to or greater than the set value.

The simplified detector 200 can also output Logit. In this case, for example, the control unit 510 finds a solution from Logit and controls the frame period according to the solution.

By making the frame period (in other words, the frame rate) variable, the effect of essentially thinning out the output of the simple detector 200 can be achieved during times when the output of the simple detector 200 is sparse.

In addition, if the output frequency of the simple detector 200 is high, the control unit 510 can determine that important information is being received and increase the operating frequency of the analysis device 130, thereby making the entire system more efficient so that it can perform the necessary operations.

The control unit 510 can also control the threshold of the neuron in the simple detector 200 according to the detection result (solution or Logit) of the simple detector 200. For example, the control unit 510 increases the threshold during the time period when the solution is greater than a set value, and decreases the threshold during the time period when the solution is equal to or less than the set value, thereby reducing the output of the simple detector 200. In the case of the aforementioned ΔΣ neuron 300, each of the comparison threshold and the upper and lower thresholds of the limit range is controlled.

Note that while the control of the frame period and the like is performed by the control unit 510 in the SNN circuit 500, this is not limiting. The frame period and the like can also be controlled by an external device outside the SNN circuit 500, for example, the input control unit 131.

In this way, according to the modified example of the first embodiment of the present technology, the control unit 510 controls the length of the frame period according to the detection result of the simple detector 200, so that an appropriate frame rate can be set according to the detection result.

2. Second embodiment
In the first embodiment described above, the simple detector 200 is provided in the SNN circuit 500, but it is also possible to provide a detailed detector 133 and an evaluator 134. The imaging device 100 in this second embodiment differs from the first embodiment in that the simple detector 200, the detailed detector 133, and the evaluator 134 are provided in the SNN circuit 500.

FIG. 19 is a block diagram showing an example configuration of an imaging device 100 in a second embodiment of the present technology. The imaging device 100 in this second embodiment includes an image sensor 120, a control CPU (Central Processing Unit) 140, and an SNN circuit 500.

The multi-core array 570 of the SNN circuit 500 is divided into multiple regions, and the functions of the simple detector 200, the detailed detector 133, and the evaluator 134 are assigned to these regions. Note that the router and input/output interface are omitted in the figure.

The control CPU 140 has the functions of the input control unit 131 and the rule selection unit 132 of the first embodiment. The control CPU 140 can also control the setting values related to the output of the image sensor 120 based on the output state of at least one of the simple detector 200, the detailed detector 133, and the evaluator 134. For example, the setting values related to the range of pixels to be output and characteristics such as pixel gain are controlled.

As shown in the figure, by providing a simple detector 200, a detailed detector 133, and an evaluator 134 within the SNN circuit 500, there is no need to place the detailed detector 133 and the evaluator 134 outside the SNN circuit 500.

In addition, a modified version of the first embodiment can also be applied to the second embodiment.

In this way, according to the second embodiment of the present technology, the simple detector 200, the detailed detector 133, and the evaluator 134 are provided within the SNN circuit 500, so there is no need to place the detailed detector 133 and the evaluator 134 outside the SNN circuit 500.

3. Third embodiment
In the first embodiment described above, the simple detector 200 calculates a solution from Logit and outputs it as a detection result, but it is also possible to output Logit as it is as a detection result without calculating a solution. The imaging device 100 in this third embodiment differs from the first embodiment in that the simple detector 200 outputs Logit.

FIG. 20 is a block diagram showing an example of the configuration of an imaging device 100 in a third embodiment of the present technology. The simplified detector 200 in this third embodiment differs from the first embodiment in that it does not calculate a solution, but outputs Logit to the input control unit 131 as the detection result.

The input control unit 131 also calculates a solution from Logit, controls the image sensor 120 according to the solution, and changes the output destination.

FIG. 21 is a flowchart showing an example of the operation of the analysis device 130 in the third embodiment of the present technology. The operation in this third embodiment differs from the first embodiment in that steps S910, S911, and S912 are executed instead of step S901.

The input control unit 131 calculates a solution from Logit (step S910) and determines whether the solution is a predetermined value P (step S911). If the solution is not P (step S911: No), the input control unit 131 returns to step S910.

On the other hand, if the solution is P (step S911: Yes), the input control unit 131 controls the output destination of the image sensor 120 to output image data to the detailed detector 133 and the evaluator 134 in the analysis device 130 (step S912). After step S912, steps S902 and onwards are executed.

As described above, by the input control unit 131 calculating the solution from Logit, the simple detector 200 does not need to calculate the solution, and the amount of processing by the simple detector 200 can be reduced.

The second embodiment can be applied to the third embodiment.

In this way, according to the third embodiment of the present technology, the input control unit 131 calculates the solution from Logit, eliminating the need for the simple detector 200 to calculate the solution.

4. Fourth embodiment
In the first embodiment described above, the simple detector 200 is provided in the SNN circuit 500, but it is also possible to provide a detailed detector 133. The imaging device 100 in this fourth embodiment differs from the first embodiment in that the simple detector 200 and the detailed detector 133 are provided in the SNN circuit 500.

FIG. 22 is a block diagram showing an example configuration of an imaging device 100 according to a fourth embodiment of the present technology. The imaging device 100 according to the fourth embodiment differs from the first embodiment in that a detail detector 133 is further provided in the SNN circuit 500.

As shown in the figure, by providing a simple detector 200 and a detailed detector 133 within the SNN circuit 500, there is no need to place the detailed detector 133 outside the SNN circuit 500.

In addition, a modified version of the first embodiment or the third embodiment can also be applied to the fourth embodiment.

In this way, according to the fourth embodiment of the present technology, the simple detector 200 and the detailed detector 133 are provided within the SNN circuit 500, so there is no need to place the detailed detector 133 outside the SNN circuit 500.

<5. Fifth embodiment>
In the above-described first embodiment, the simple detector 200 is disposed outside the image sensor 120, but it is also possible to dispose the simple detector 200 inside the image sensor 120. The imaging device 100 in this fifth embodiment differs from the first embodiment in that the simple detector 200 is disposed inside the image sensor 120.

FIG. 23 is a block diagram showing an example of the configuration of an imaging device 100 in a fifth embodiment of the present technology. The imaging device 100 in this fifth embodiment differs from the first embodiment in that a simple detector 200 is disposed within the image sensor 120.

The image sensor 120 includes, for example, a pixel array unit 121, a memory 122, a simple detector 200, and a controller 123. The simple detector 200 is implemented in an SNN circuit 500.

In the pixel array section 121, a number of pixels are arranged, each of which generates a pixel signal. The memory 122 stores various types of data, such as image data. The controller 123 controls the circuits in the image sensor 120.

By providing the simple detector 200 inside the image sensor 120, there is no need to place the simple detector 200 outside the image sensor 120.

FIG. 24 is a diagram showing an example of a stacked structure of an image sensor 120 in a fifth embodiment of the present technology. The image sensor 120 includes a logic chip 126 and a pixel chip 125 stacked on the logic chip 126. These chips are electrically connected by, for example, Cu-Cu bonding. Note that in addition to Cu-Cu bonding, they can also be connected by vias or bumps.

The pixel array section 121 is arranged on the pixel chip 125. The remaining circuits, such as the simple detector 200, are arranged on the logic chip 126.

Note that although the circuits in the image sensor 120 are arranged on the pixel chip 125 and the logic chip 126, these circuits can also be arranged on a single semiconductor chip.

In addition, a modified version of the first embodiment, or each of the second, third, and fourth embodiments can be applied to the fifth embodiment.

In this way, according to the fifth embodiment of the present technology, since the simple detector 200 is provided inside the image sensor 120, there is no need to place the simple detector 200 outside the image sensor 120.

6. Sixth embodiment
In the first embodiment described above, the rule selection unit 132 selects a rule based on the detection result, but in this configuration, there is a risk of selecting an incorrect rule due to erroneous detection by the simple detector 200. The imaging device 100 in the sixth embodiment differs from the first embodiment in that the imaging device 100 selects a rule based on the sensor signal from the environmental sensor and the detection result.

FIG. 25 is a block diagram showing an example of the configuration of an imaging device 100 in a sixth embodiment of the present technology. The imaging device 100 in the sixth embodiment differs from the first embodiment in that it includes one or more environmental sensors 150.

The environmental sensor 150 generates a sensor signal different from the image data. For example, a temperature sensor, a humidity sensor, an illuminance sensor, a pressure sensor, an acoustic sensor, or a vibration sensor can be used as the environmental sensor 150. In addition, a proximity sensor, a contact sensor, or an inertial sensor attached to the measurement object or the surroundings of the measurement object can also be used as the environmental sensor 150. The sensor signal of the environmental sensor 150 is supplied to the rule selection unit 132 via the input control unit 131.

The rule selection unit 132 selects a rule based on the sensor signal and detection result from the environmental sensor 150. For example, even if the simple detector 200 fails to detect a person approaching, the rule selection unit 132 can detect the person from sensor signals from an acoustic sensor, vibration sensor, etc., and select an appropriate rule. This makes it possible to prevent an incorrect rule from being selected when the simple detector 200 makes a false detection or misses a detection, making the system more robust.

Furthermore, a modified version of the first embodiment, or each of the second, third, fourth, and fifth embodiments can be applied to the sixth embodiment.

In this way, according to the sixth embodiment of the present technology, the rule selection unit 132 selects a rule based on the sensor signal from the environmental sensor 150 and the detection result, making the system more robust.

<7. Examples of applications to moving objects>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, or a robot.

FIG. 26 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology disclosed herein can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in FIG. 26, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside vehicle information detection unit 12030, an inside vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio/video output unit 12052, and an in-vehicle network I/F (interface) 12053.

The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 functions as a control device for a drive force generating device for generating the drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force for the vehicle.

The body system control unit 12020 controls the operation of various devices installed in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, tail lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals from various switches transmitted from a portable device that replaces a key can be input to the body system control unit 12020. The body system control unit 12020 accepts the input of these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the image capturing unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the image capturing unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared light.

The in-vehicle information detection unit 12040 detects information inside the vehicle. To the in-vehicle information detection unit 12040, for example, a driver state detection unit 12041 that detects the state of the driver is connected. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the driver's degree of fatigue or concentration based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

The microcomputer 12051 can calculate control target values for the driving force generating device, steering mechanism, or braking device based on information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an Advanced Driver Assistance System (ADAS), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, or vehicle lane departure warning.

The microcomputer 12051 can also perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

The microcomputer 12051 can also output control commands to the body system control unit 12020 based on information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

The audio/image output unit 12052 transmits at least one output signal of audio and image to an output device capable of visually or audibly notifying the occupants of the vehicle or the outside of the vehicle of information. In the example of FIG. 26, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

FIG. 27 shows an example of the installation position of the imaging unit 12031.

In FIG. 27, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at the front nose, side mirrors, rear bumper, back door, and upper part of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin is mainly used to detect leading vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

Note that FIG. 27 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera consisting of multiple imaging elements, or an imaging element having pixels for phase difference detection.

For example, the microcomputer 12051 can obtain the distance to each solid object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and can extract as a preceding vehicle, in particular, the closest solid object on the path of the vehicle 12100 that is traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km/h or faster). Furthermore, the microcomputer 12051 can set the inter-vehicle distance that should be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving, which runs autonomously without relying on the driver's operation.

For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects, such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects, based on the distance information obtained from the imaging units 12101 to 12104, and can use the data to automatically avoid obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines the collision risk, which indicates the risk of collision with each obstacle, and when the collision risk is equal to or exceeds a set value and there is a possibility of a collision, it can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the image captured by the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the image captured by the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not it is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the image captured by the imaging units 12101 to 12104 and recognizes a pedestrian, the audio/image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio/image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

Above, an example of a vehicle control system to which the technology according to the present disclosure can be applied has been described. Of the configurations described above, the technology according to the present disclosure can be applied to the imaging unit 12031. Specifically, the imaging device 100 in FIG. 1 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, it becomes possible to suppress degradation in system performance when sparsity is improved.

Note that the above-described embodiment shows an example for realizing the present technology, and there is a corresponding relationship between the matters in the embodiment and the matters specifying the invention in the claims. Similarly, there is a corresponding relationship between the matters specifying the invention in the claims and the matters in the embodiment of the present technology that have the same name. However, the present technology is not limited to the embodiment, and can be realized by making various modifications to the embodiment without departing from the gist of the technology.

Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

The present technology can also be configured as follows.
(1) a predetermined number of output neurons that generate output signals every time a predetermined unit time elapses based on input signals from a hidden layer;
and a statistics calculation unit that calculates, for each output neuron, statistics of the output signals generated within a predetermined frame period that is longer than the unit time.
(2) The imaging device according to (1), further comprising a function calculation unit that inputs each of the statistical quantities to a predetermined function and outputs a calculated value obtained by the input.
(3) The imaging device according to (3), further comprising a control unit that controls a length of the frame period in accordance with a detection result of the simple detector.
(4) The imaging device according to (3), further comprising a control unit that controls a threshold value of a neuron in the simple detector in accordance with a detection result of the simple detector.
(5) further comprising an image sensor for generating image data;
The imaging device according to any one of (1) to (4), wherein the simplified detector detects a predetermined detection target from the image data by analysis.
(6) a detailed detector for detecting a predetermined detection target from the image data by a more detailed analysis than the simple detector; and
an evaluator that generates an evaluation value from the image data by an analysis different from that of the detail detector;
The imaging device according to (5), further comprising a rule selection unit that selects a rule for controlling operations of the detailed detector and the evaluator based on a detection result of the simple detector.
(7) The imaging device according to (6), wherein the simple detector and the detailed detector are arranged within a neural network circuit.
(8) The imaging device according to (6), wherein the simple detector, the detailed detector and the evaluator are arranged in a neural network circuit.
(9) The imaging device according to any one of (6) to (8), further comprising an input control unit that controls an operation of the image sensor based on a detection result of the simple detector.
(10) Further comprising an environmental sensor that outputs a sensor signal different from the image data;
The imaging device according to any one of (6) to (9), wherein the rule selection unit selects the rule based on the sensor signal and a detection result of the simple detector.
(11) The imaging device according to any one of (1) to (10), wherein the simple detector is disposed within an image sensor that generates image data.
(12) The imaging device according to any one of (1) to (11), wherein the neuron in the simple detector is a spiking neuron.
(13) a difference calculator that calculates a difference between an added value of a predetermined number of input signals and a feedback signal;
an integrator that integrates the difference and outputs an integral value;
a limiting unit that outputs the integral value as an output signal when the integral value is outside a predetermined range;
a comparator for comparing the output signal with a predetermined value and providing a comparison result;
a feedback section that converts the comparison result into the feedback signal and feeds it back, the feedback section being disposed in at least one of a plurality of neurons in the neural network.
(14) A simplified detector including: a predetermined number of output side neurons that generate output signals every time a predetermined unit time elapses based on an input signal from an intermediate layer; and a statistics calculation unit that calculates, for each of the output side neurons, statistics of the output signals generated within a predetermined frame period that is longer than the unit time;
and an analysis device that analyzes an image based on a detection result of the simple detector.
(15) a step in which a predetermined number of output neurons generate output signals every time a unit time elapses based on input signals from the intermediate layer;
and calculating, for each of the output neurons, a statistical amount of the output signals generated within a predetermined frame period longer than the unit time.

REFERENCE SIGNS LIST 100 Imaging device 110, 110-1, 110-2 Imaging section 120 Image sensor 121 Pixel array section 122 Memory 123 Controller 125 Pixel chip 126 Logic chip 130 Analysis device 131 Input control section 132 Rule selection section 133 Detail detector 134 Evaluator 140 Control CPU
150 Environmental sensor 200 Simple detector 210 Input layer 211, 212 Convolution layer 213 Pooling layer 220 Hidden layer 230 Output layer 231 Output side neuron 232 Statistical quantity calculation unit 233 Function calculation unit 300 ΔΣ neuron 310 Difference calculator 320 Integrator 330 Limiting unit 340 Comparator 350 Feedback unit 500 SNN circuit 510 Control unit 560 Input/output interface 570 Multi-core array 580 Router 581 to 585 FIFO memory 586 Arbiter 590 Core 591 Core router 592 Neuron I/O
593 Product-sum unit 594 Work memory 595 Membrane potential memory 596 LIF unit 600 Processor 12031 Imaging unit

Claims (15)

a predetermined number of output side neurons that generate output signals every time a predetermined unit time elapses based on input signals from the intermediate layer;
and a statistics calculation unit that calculates, for each output neuron, statistics of the output signals generated within a predetermined frame period that is longer than the unit time. The imaging device according to claim 1, further comprising a function calculation unit that inputs each of the statistical quantities into a predetermined function and outputs the calculated value. 4. The imaging device according to claim 3, further comprising a control unit that controls the length of the frame period in response to a detection result of the simple detector. 4. The imaging apparatus according to claim 3, further comprising a control unit for controlling a threshold value of a neuron in the simple detector in response to a detection result of the simple detector. further comprising an image sensor for generating image data;
The imaging device according to claim 1 , wherein the simplified detector detects a predetermined detection target from the image data by analysis. a detailed detector for detecting a predetermined detection target from the image data by a more detailed analysis than the simple detector;
an evaluator that generates an evaluation value from the image data by an analysis different from that of the detail detector;
6. The imaging apparatus according to claim 5, further comprising a rule selection unit that selects a rule for controlling operations of the detailed detector and the evaluator based on a detection result of the simple detector. 7. The imaging device of claim 6, wherein the simple detector and the detailed detector are arranged in a neural network circuit. 7. The imaging device according to claim 6, wherein the simple detector, the detailed detector and the evaluator are arranged in a neural network circuit. The imaging device according to claim 6, further comprising an input control unit that controls the operation of the image sensor based on the detection results of the simple detector. An environmental sensor that outputs a sensor signal different from the image data,
The imaging device according to claim 6 , wherein the rule selection section selects the rule based on the sensor signal and a detection result of the simple detector. The imaging device of claim 1 , wherein the simplified detector is disposed within an image sensor that generates image data. 2. The imaging device according to claim 1, wherein the neurons in the simplified detector are spiking neurons. a difference calculator that calculates a difference between an added value of a predetermined number of input signals and a feedback signal;
an integrator that integrates the difference and outputs an integral value;
a limiting unit that outputs the integral value as an output signal when the integral value is outside a predetermined range;
a comparator for comparing the output signal with a predetermined value and providing a comparison result;
a feedback section that converts the comparison result into the feedback signal and feeds it back, the feedback section being disposed in at least one of a plurality of neurons in the neural network. a simplified detector including a predetermined number of output side neurons that generate output signals every time a predetermined unit time elapses based on input signals from a hidden layer, and a statistics calculation unit that calculates statistics of the output signals generated within a predetermined frame period that is longer than the unit time for each of the output side neurons;
and an analysis device that analyzes an image based on a detection result of the simple detector. A step in which a predetermined number of output neurons generate output signals every time a unit time elapses based on input signals from the hidden layer;
and calculating, for each of the output neurons, a statistical amount of the output signals generated within a predetermined frame period longer than the unit time.

Images