[go: up one dir, main page]

WO2019149375A1 - Procédé et système de traitement de données d'entrée et de propagation de variance dans un réseau neuronal - Google Patents

Procédé et système de traitement de données d'entrée et de propagation de variance dans un réseau neuronal Download PDF

Info

Publication number
WO2019149375A1
WO2019149375A1 PCT/EP2018/052721 EP2018052721W WO2019149375A1 WO 2019149375 A1 WO2019149375 A1 WO 2019149375A1 EP 2018052721 W EP2018052721 W EP 2018052721W WO 2019149375 A1 WO2019149375 A1 WO 2019149375A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
input data
variance
variance value
processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2018/052721
Other languages
English (en)
Inventor
Patrick SAUER
Oleksandr SHEKHOVTSOV
Boris Flach
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Toyota Motor Europe NV SA
Czech Technical University In Prague
Original Assignee
Toyota Motor Europe NV SA
Czech Technical University In Prague
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Toyota Motor Europe NV SA, Czech Technical University In Prague filed Critical Toyota Motor Europe NV SA
Priority to PCT/EP2018/052721 priority Critical patent/WO2019149375A1/fr
Publication of WO2019149375A1 publication Critical patent/WO2019149375A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning

Definitions

  • the present disclosure is related to the field of data processing using neural networks, for example image processing using neural networks.
  • neural networks for example convolutional neural network or feed-forward neural networks, to process data.
  • a scoring system is used to adjust the neural network so that it behaves as expected on these known images.
  • the neural networks are then used in a phase called a testing phase on actual images without any knowledge of the expected output.
  • neural network used in the present application can cover a combination of a plurality of known networks.
  • the current neural network models are deterministic systems which maps inputs (for example images) to outputs (for example regressions or class probabilities). Thus, the presence of noise at the input is critical.
  • noises are replaced with an expectation inside each layer of the neural network.
  • the present disclosure overcomes one or more deficiencies of the prior art by proposing a method for processing input data using a neural network (preferably a feed-forward neural network) comprising a plurality of consecutive neural network layers, the method comprising:
  • processing the input data comprising, by each neural network layer of the plurality of consecutive neural network layers, a processing,
  • the method comprises, after each processing by a neural network layer of the plurality of consecutive neural network layers, a determination of at least an intermediate variance value.
  • the invention therefore proposes to propagate variance values through the neural network, by performing a determination of at least an intermediate variance value after each processing by a neural network layer of the plurality of neural network layers.
  • An uncertain input may be caused by sensor noise if a sensor is used to acquire the input data. It may also be an input from a computational sensor (for example a depth map computed inside a structured light sensor), or an input from another neural network.
  • a computational sensor for example a depth map computed inside a structured light sensor
  • the noise at the input can be modelled using the above method because the first variance value is an approximation of this noise.
  • the neural network layers are of the sigmoid type, then using the variance of the input may improve the stability or the robustness of the neural network.
  • the determination of at least an intermediate variance value is performed through analytical calculations.
  • the analytical calculations use at least an intermediate variance value determined after a processing by the preceding neural network layer in the plurality of consecutive neural network layers (i.e. the neural network layer right before the one where the calculation is performed).
  • the variance is calculated using previously obtained variance values so as to propagate through the neural network until reaching the output where the variance is associated with the output.
  • each neural network layer has a type
  • the analytical calculations take the type of the preceding (the one that has just processed data before the calculation is carried out) neural network layer into account.
  • a different formula may be used for each type of neural network layer.
  • each layer is seen as a module associated with a specific calculation depending on the type of layer.
  • various known neural networks such as known convolutional neural networks or known feed-forward neural networks.
  • the input data has a plurality of input components each associated with a first variance value.
  • each neural network layer receives a respective first number of input components (for example from the preceding neural network layer if there is a preceding neural network layer) and a first or intermediate variance value for each input component,
  • each neural network layer delivers a respective second number of output components, and an intermediate variance value for each output component is determined. It should be noted that the number of components at the input may differ from the number of components at the output for a given layer.
  • the method is performed during a testing phase and the first variance value is predetermined or the first variance values are predetermined.
  • This testing phase is defined as the phase during which input data is inputted without a priori knowledge of the expected output of the neural network. It is the phase which succeeds the training phase which is well known to the person skilled in the art.
  • This predetermined first variance value may be obtained during a calibration step.
  • the noise estimate (another words the variance value) could be obtained by varying the exposure and gain settings of the sensor used to collect this image while observing a calibration pattern.
  • the exposure and gain settings have a direct effect on pixel noise and this could be determined from the recorded images so as to obtained this variance value.
  • each input component of the input data is associated with a respective predetermined first variance value.
  • the input has a plurality of components, it is possible to apply a predetermined variance on each component.
  • each pixel value of this image is a component of the input data, and it is possible to apply a different variance value to each pixel according to, for example, the type of sensor used.
  • the method is performed during a training phase of the neural network, the input data belonging to a set of training input data,
  • the first variance value relates to a variance over the set of training input data or the first variance value is predetermined.
  • a set of known training input data is inputted to the neural network so as to observe the output of the neural network and adjust the neural network.
  • the variance for each component may be the variance for all the possible values of this component in the set of training input data.
  • the first variance value is predetermined, for example set at a constant value in a prior step. This value may be chosen to correspond to sensor noise (if a sensor is used to acquire the input data) or the input variance of a system/sensor that provides an estimate of the component of the input data.
  • At least a first mean value associated with the input data is obtained, the first mean value relating to a mean over the set of training input data.
  • the mean and variance over the set of training input data can be inputted to the neural network in order to estimate the statistics of inner neurons over the dataset. Such statistics are not used for recognition or classification of a specific image, but may be needed in a subsequent normalization (to initialize and better condition the network training).
  • the mean for each component may be the mean for all the possible values of this component in the set of training input data.
  • the method may comprise, after each processing by a neural network layer of the plurality of consecutive neural network layers, a determination of at least an intermediate mean value. That is to say, a propagation of the mean value is obtained.
  • the intermediate mean value may be determined through analytical calculations, for example using at least an intermediate mean value determined after a processing by the preceding neural network layer in the plurality of neural network layers. These analytical calculations may also take the type of the preceding neural network layer into account.
  • Each input component may be associated with a first mean value.
  • the same method may apply with a different first mean value which may be the sensed input if a sensor is used.
  • a first variance value with this means implies that the noise is taken into account and this noise is represented by the first variance value.
  • the mean is propagated along with the variance in a similar manner.
  • Each neural network layer may receive a respective first number of input components and a first or intermediate mean value for each input component, and each neural network layer may deliver a respective second number of output components, and an intermediate mean value for each output component is determined.
  • the input data is an image.
  • the image can be a color image such as an RGB (Red- Green-Blue) image known to the skilled person.
  • RGB Red- Green-Blue
  • the invention also relates to a system, for example implemented by one or more computers, comprising a neural network for processing input data, the neural network comprising a plurality of consecutive neural network layers, the system being configured to:
  • This system can be configured to carry out all the embodiments of the method for processing data described above.
  • the steps of the method for processing input data are determined by computer program instructions.
  • the invention is also directed to a computer program for executing the steps of a method as described above when this program is executed by a computer.
  • This program can use any programming language and take the form of source code, object code or a code intermediate between source code and object code, such as a partially compiled form, or any other desirable form.
  • the invention is also directed to a computer-readable information medium containing instructions of a computer program as described above.
  • the information medium can be any entity or device capable of storing the program.
  • the medium can include storage means such as a ROM, for example a CD ROM or a microelectronic circuit ROM, or magnetic storage means, for example a diskette (floppy disk) or a hard disk.
  • the information medium can be an integrated circuit in which the program is incorporated, the circuit being adapted to execute the method in question or to be used in its execution.
  • FIG. 1 is a block diagram of an exemplary method for processing data according to an example
  • FIG. 2 is a block diagram of an exemplary system for processing data according to an example
  • - figure 3 is a representation of a neural network
  • - figure 4 is a representation of a neural network layer
  • - figure 6 is an example of neural network and of its output.
  • the data may be an image or a set of images.
  • the invention is however not limited to processing images and it may also relate to processing sounds or other types of data which can be processed using neural networks.
  • Figure 1 is a block diagram of an exemplary method for processing input data, which may be data acquired by a sensor during a testing phase, or input data from a set of input data used during a training phase.
  • a first step E01 is carried out in which first mean and variance values are obtained.
  • the method of figure 1 is performed during a training phase.
  • the input data may be an image belonging to a set of training images. This image comprises a plurality of pixels each having a pixel value and for each pixel, the mean is calculated for each pixel over the entire set of training images. The variance is also calculated for each pixel on the basis of the value of this pixel in each training image.
  • the first variance values can be set at predetermined values, for example one value per pixel.
  • the input data is inputted to the neural network after step E01.
  • the first mean and variance values may also be inputted to the neural network. Even if the first mean and variance values are not used by the layers of the neural network, they will be used in intermediate calculations carried out between the layers of the neural network.
  • the propagation method gives identical result to the standard propagation of the mean values. If the variance is positive, the propagation approximates the effect of propagating multiple samples through the layer that are distributed normally with the input mean and variance.
  • processing is performed by a first neural network layer of a neural network, for example a feed-forward neural network.
  • the neural network comprises a plurality of consecutive neural network layers.
  • the first layer may be, for example a linear layer, or even a rectifier linear unit. This first layer processes the input data and transmits the processed input data, along with the first mean and variance values to the following neural network layer.
  • step E03 Before performing processing by the following neural network, a calculation of intermediate variance and mean values is carried out in step E03 (during testing, only the intermediate variances may be calculated). This calculation may be an analytical calculation and the formulas to be used may depend on the type of the preceding neural network layer (the first layer in this case). The calculation uses the first mean and variance values.
  • step E04 The final processing by a neural network layer is performed in step E04 and a final calculation of intermediate variances and means is carried out in step E05.
  • This calculation takes into account the type of neural network layer of the final neural network layer and it also uses the variance and mean values, which have been determined in relation to the neural network layer which precedes the final neural network layer.
  • each neural network receives a respective first number of input components and, for each input component, a first or intermediate variance value (only the first neural network layer will receive a first variance value, the following neural network layers will receive intermediate variance values) and a first or intermediate mean value.
  • each neural network layer delivers a respective second number of output components, and in the subsequent calculation step, an intermediate variance value and an intermediate mean value for each output component is determined.
  • the steps of the method described in reference to figure 1 can be determined by computer instructions. These instructions can be executed on one or more computers. In the example of figure 2, a single computer is used to perform the method.
  • This system comprises an image sensor 201 which acquires input data in the form of images when a neural network 203 comprised in the system is to be tested.
  • a module 201' is used to obtain an image from a set of training images.
  • Module 202 is used to obtain one or more variance value and one or more mean values (especially during training).
  • the modules 201, 20 and 202 can be implemented by computer instructions which can be executed by processor 210.
  • the neural network 203 comprises a first neural network layer 204, a module 205 for calculating intermediate variance and mean values associated with this first neural network layer 204.
  • the neural network 203 may comprise a plurality of additional layers and calculation modules which have not been represented for the sake of conciseness.
  • the final neural network layer 206 has been represented with the associated module for calculating intermediate variance and mean values.
  • Figure 3 shows a Bayesian network representation of a neural network wherein the input/outputs of neural network layers are denoted by X k with each component being represented by a circle (with k an index indicating the layer). Each neural network layer is represented by the probability of having a corresponding output based on the input of the layer, namely p(X k+1
  • X k ) may define a deterministic relation, for example if:
  • X k ) may also define a stochastic relation, for example if:
  • Z ⁇ N (0,1) means that Z is a random variable that has a standard normal distribution.
  • the inventors of the present invention have noted that this representation allows expressing the probability p(X k ⁇ x° for a given output component noted with the index i of a given neural network layer through a similar probability of the preceding layer, leading to the recurrent (which may also be called feed-forward) calculation:
  • E is the expectation, and capital X k denotes a random variable, while small x k denotes a specific value taken by X k , identified also with the event that that variable takes this value.
  • Factorized approximations are commonly used in variational inference methods. The approximation is possible, because in many recognition or regression problems, when the network input x° is specified, the corresponding class label or the regression value is unique. When the input x° is specified with some uncertainty, the states of variables in the hidden layers tend to have unimodal distributions concentrated around their mean values. According to the present disclosure, the factorized approximation q can be computed layer by layer by substituting the already approximated quantities in the exact recurrent expression above. The following equation is obtained:
  • Figure 4 is a schematic illustration of a neural network layer. This layer receives four input component noted with the index i and having a mean m, and a variance s £ .
  • the neural network layer delivers four output components each having a mean m'; and a variance s .
  • X is a vector of all the input components
  • Y is a vector of all the output components of the neural network layer.
  • - MC is the result of a Monte-Carlo simulation obtained by inputting several times (for example 10000 times) this image with random noise each time without using the method of the invention. The results are sampled each time and the output is represented.
  • the method of the invention provides distributions (Gaussian using the mean and variances) which correspond to MC.
  • the histogram shows what is the probability to see a particular value of the neuron output in a network with noisy input. It is seen that AP2 provides usable estimates of this probability distribution because it is close to MC.
  • the invention provides a good approximation of the output of a neural network by propagating the mean and the variance, and, during training, the mean.
  • This method may be used to perform normalisation as disclosed in the international patent application filed by the same applicants on the same day as the present application and titled "Method and system for processing input data using a neural network and normalizations", which is incorporated entirely to the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne un système et un procédé de traitement de données d'entrée faisant appel à un réseau neuronal comprenant une pluralité de couches consécutives de réseau neuronal, le procédé comprenant les étapes consistant : - à obtenir (E01) au moins une première valeur de variance associée aux données d'entrée, - à traiter (E02, E04) les données d'entrée et au moins la première valeur de variance au moyen du réseau neuronal, le traitement des données d'entrée comprenant un traitement effectué par chaque couche de réseau neuronal de la pluralité de couches consécutives de réseau neuronal. Le procédé consiste, après chaque traitement effectué par une couche de réseau neuronal de la pluralité de couches consécutives de réseau neuronal, à déterminer (E03, E05) au moins une valeur de variance intermédiaire.
PCT/EP2018/052721 2018-02-02 2018-02-02 Procédé et système de traitement de données d'entrée et de propagation de variance dans un réseau neuronal Ceased WO2019149375A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/EP2018/052721 WO2019149375A1 (fr) 2018-02-02 2018-02-02 Procédé et système de traitement de données d'entrée et de propagation de variance dans un réseau neuronal

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2018/052721 WO2019149375A1 (fr) 2018-02-02 2018-02-02 Procédé et système de traitement de données d'entrée et de propagation de variance dans un réseau neuronal

Publications (1)

Publication Number Publication Date
WO2019149375A1 true WO2019149375A1 (fr) 2019-08-08

Family

ID=61168099

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2018/052721 Ceased WO2019149375A1 (fr) 2018-02-02 2018-02-02 Procédé et système de traitement de données d'entrée et de propagation de variance dans un réseau neuronal

Country Status (1)

Country Link
WO (1) WO2019149375A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112882053A (zh) * 2021-01-21 2021-06-01 清华大学深圳国际研究生院 一种主动标定激光雷达和编码器外参的方法

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160217368A1 (en) * 2015-01-28 2016-07-28 Google Inc. Batch normalization layers
WO2016145516A1 (fr) * 2015-03-13 2016-09-22 Deep Genomics Incorporated Système et procédé d'apprentissage de réseaux neuronaux

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160217368A1 (en) * 2015-01-28 2016-07-28 Google Inc. Batch normalization layers
WO2016145516A1 (fr) * 2015-03-13 2016-09-22 Deep Genomics Incorporated Système et procédé d'apprentissage de réseaux neuronaux

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
"JOURNAL OF MACHINE LEARNING RESEARCH", vol. 28 (2), 17 June 2013, MIT PRESS, CAMBRIDGE, MA, US, ISSN: 1532-4435, article SIDA I WANG ET AL: "Fast dropout training", pages: 118 - 126, XP055515749 *
AHMED H ABDELAZIZ ET AL: "Uncertainty propagation through deep neural networks", INTERSPEECH 2015, 6 September 2015 (2015-09-06), Dresden, Germany, pages 1 - 6, XP055516033 *
DROPOUT, A SIMPLE WAY TO PREVENT NEURAL NETWORKS FROM OVERFITTING
SRIVASTAVA ET AL.: "Dropout: A Simple Way to Prevent Neural Networks from Overfitting", JOURNAL OF MACHINE LEARNING RESEARCH, vol. 15, 2014, pages 1929 - 1958, XP055193568
WANG, S; MANNING, C: "Fast dropout training", PROCEEDINGS OF THE 30TH INTERNATIONAL CONFERENCE ON MACHINE LEARNING, PMLR, vol. 28, no. 2, 2013, pages 118 - 126
YARIN GAL ET AL: "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning", 4 October 2016 (2016-10-04), XP055467519, Retrieved from the Internet <URL:https://arxiv.org/pdf/1506.02142.pdf> [retrieved on 20180416] *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112882053A (zh) * 2021-01-21 2021-06-01 清华大学深圳国际研究生院 一种主动标定激光雷达和编码器外参的方法
CN112882053B (zh) * 2021-01-21 2023-07-18 清华大学深圳国际研究生院 一种主动标定激光雷达和编码器外参的方法

Similar Documents

Publication Publication Date Title
Gawlikowski et al. A survey of uncertainty in deep neural networks
Chen et al. Weak in the NEES?: Auto-tuning Kalman filters with Bayesian optimization
Castellano et al. Variable selection using neural-network models
Dera et al. PremiUm-CNN: Propagating uncertainty towards robust convolutional neural networks
Kwoh et al. Using hidden nodes in Bayesian networks
US10610109B2 (en) Emotion representative image to derive health rating
US20140343903A1 (en) Factorial hidden markov models estimation device, method, and program
Wawrzyniak et al. Data-driven models in machine learning for crime prediction
EP3975071A1 (fr) Identification et quantification de polarisation parasite sur la base de connaissances d&#39;expert
Zeng et al. Gait recognition across different walking speeds via deterministic learning
CN117763307A (zh) 一种基于lstm的异味检测方法、系统、设备及介质
Titensky et al. Uncertainty propagation in deep neural networks using extended kalman filtering
Sathyavathi et al. An intelligent human age prediction from face image framework based on deep learning algorithms
Ibragimovich et al. Effective recognition of pollen grains based on parametric adaptation of the image identification model
CN115954109A (zh) 一种提高疫情防控精度的方法、装置、电子设备及介质
WO2019149375A1 (fr) Procédé et système de traitement de données d&#39;entrée et de propagation de variance dans un réseau neuronal
CN112598020B (zh) 目标识别方法及系统
Movellan A learning theorem for networks at detailed stochastic equilibrium
US20240378866A1 (en) Cell nuclei classification with artifact area avoidance
Cho et al. Understanding token probability encoding in output embeddings
Abbasi et al. Long-term prediction of blood pressure time series using multiple fuzzy functions
Kumar et al. Robust adaptive fuzzy identification of time-varying processes with uncertain data. Handling uncertainties in the physical fitness fuzzy approximation with real world medical data: An application
Sun Revisiting Human Activity Recognition with Explainable AI Approaches
Xi et al. Parameter estimation of RBF-AR model based on the EM-EKF algorithm
Vera et al. Optimizing a dental milling process by means of Soft Computing Techniques

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18703563

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18703563

Country of ref document: EP

Kind code of ref document: A1