[go: up one dir, main page]

WO2019067831A1 - Générateurs multi-objectifs en apprentissage profond - Google Patents

Générateurs multi-objectifs en apprentissage profond Download PDF

Info

Publication number
WO2019067831A1
WO2019067831A1 PCT/US2018/053295 US2018053295W WO2019067831A1 WO 2019067831 A1 WO2019067831 A1 WO 2019067831A1 US 2018053295 W US2018053295 W US 2018053295W WO 2019067831 A1 WO2019067831 A1 WO 2019067831A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
objective
machine
learning
generator
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2018/053295
Other languages
English (en)
Inventor
James K. Baker
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
D5AI LLC
Original Assignee
D5AI LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from PCT/US2018/051069 external-priority patent/WO2019067236A1/fr
Application filed by D5AI LLC filed Critical D5AI LLC
Priority to US16/646,096 priority Critical patent/US11074505B2/en
Priority to CN201880067035.3A priority patent/CN111226236B/zh
Priority to EP18861823.5A priority patent/EP3688677A4/fr
Publication of WO2019067831A1 publication Critical patent/WO2019067831A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions

Definitions

  • Generative modeling is an active and growing subfield within unsupervised and semi-supervised machine learning.
  • the goal of generative models is to represent the latent variables that describe a data distribution that can produce or reproduce data examples in high-dimensional space.
  • Two methods, Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) have emerged as leading techniques for generative modeling within artificial intelligence and deep learning.
  • VAEs Variational Autoencoders
  • GANs Generative Adversarial Networks
  • One typical example is images, where generative models can be used to produce new images that appear realistic and are highly similar to those in a set of training examples, without exactly reproducing the existing images on which the generative model was trained.
  • Trained with stochastic gradient descent in coordination with deep neural networks that serve as universal function approximators, VAEs and GANs can be used to generate many types of realistic data.
  • a key aspect of generative models is that they should not simply reproduce data examples used in training, but should be able to generate novel data that is similar to, but different from, any example in the training set.
  • Both VAEs and GANs are trained by attempting to minimize objective cost functions that achieve a minimum by exactly reproducing training data.
  • GANs frequently suffer from mode collapse, where the GAN generates representatives of only a small proper subset of the modes of a multi-modal distribution, in some cases generating representatives of only a single mode. This reduces diversity among generated data and limits the usefulness of GANs in some applications.
  • the present invention in one general aspect, is designed to overcome these and other limitations in VAEs and GANs by using an additional objective to avoid generating data that is too similar to any previously known data example. This prevents plagiarism or simple copying of existing data examples, enhancing the ability of a generator to usefully generate novel data.
  • a formulation of GANs learning as the mixed strategy minimax solution of a zero-sum game solves the convergence and stability problem of GANs learning. At the same time, the present invention does not suffer from mode collapse, the way GANs can.
  • FIG. 1 illustrates a block diagram of an autoencoder having an additional objective, according to one aspect of this disclosure
  • FIG. 2 illustrates a block diagram of a generator including a real-vs-generated discriminator, according to one aspect of this disclosure
  • FIG. 3 illustrates a block diagram of a generator including a real-vs-generated discriminator, according to one aspect of this disclosure
  • FIG. 3 A illustrates a block diagram of a system with a plurality of generators and a plurality of classifiers according to one aspect of this disclosure
  • FIG. 4 is a diagram of a computer system that may be used to implement various embodiments, according to one aspect of this disclosure.
  • FIG. 5 is a diagram illustrating a deep neural network of the type that might be used in various aspects of various embodiments of the invention.
  • FIGs 1, 2, and 3 are block diagrams of machine-learning systems according to various embodiments of the invention that may be implemented on a computer system, such as the computer system 400 in Figure 4.
  • Many components of the machine-learning systems in Figures 1-3 are neural networks. Before explaining further the components of the current invention, a summary explanation of neural networks is provided.
  • a diagram of an example neural network is shown in Figure 5.
  • a neural network is a collection of nodes and directed arcs.
  • the nodes in a neural network are often organized into layers.
  • the layers may be numbered from bottom to top, when diagramed as in Figure 5.
  • Each directed arc in a layered feed-forward neural network goes from a source node in a lower layer to a destination node in a higher layer.
  • the feed-forward neural network shown in Figure 5 has an input layer, an output layer, and three inner layers (although a deep neural network could be fewer or more inner layers).
  • An inner layer in a neural network is also called a "hidden" layer.
  • Each directed arc is associated with a numerical value called its "weight.”
  • each node other than an input node is associated with a numerical value called its “bias.”
  • the weights and biases of a neural network are called “learned” parameters.
  • the values of the learned parameters are adjusted by a computer system such as the computer system 400 shown in Figure 4.
  • Other parameters that control the training process are called hyperparameters.
  • a feed-forward neural network may be trained by the computer system 400 using an iterative process of stochastic gradient descent with one iterative update of the learned parameters for each minibatch.
  • the full batch of training data is typically arranged into a set of smaller, disjoint sets called minibatches.
  • An epoch comprises the computer system 400 doing a stochastic gradient descent update for each minibatch contained in the full batch of training data.
  • the computer system 400 estimates the gradient of the objective (or cost) function for a training data item by first computing the activation of each node in the network using a feed-forward activation computation.
  • the computer system 400 then estimates the partial derivatives of the objective with respect to the learned parameters using a process called "back-propagation," which computes the partial derivatives based on the chain rule of calculus, proceeding backwards through the layers of the network.
  • back-propagation a process called "back-propagation,” which computes the partial derivatives based on the chain rule of calculus, proceeding backwards through the layers of the network.
  • the processes of stochastic gradient descent, feed-forward computation, and back-propagation are well-known to those skilled in the art of training neural networks.
  • Some of the components of the machine-learning systems of Figures 1 through 3 that can comprise neural networks include the encoders 102, 202, 302; the decoders 104, 204, 304; the real v. generated discriminators 219, 319; the classifier 321; the machine-learning systems 208, 328; and the distortion (or noise) detector 322, for example.
  • FIG 1 shows a block diagram of an autoencoder.
  • An autoencoder is a type of machine learning system that can learn efficient data coding and may be used to generate data examples.
  • Use of an autoencoder as a generator is well-known to those skilled in the art of neural networks.
  • VAE variational autoencoder
  • SCAN stochastic categorical autoencoder network
  • Typical auto-encoders include, as depicted in Figure 1, an encoder 102 that is trained to produce, from training examples 101, a layer 103 that is a bottle-neck layer in the case of a conventional autoencoder, or a stochastic layer in the case of a VAE or SCAN. A decoder 104 is then trained to produce the training examples 101 from the bottle-neck or stochastic layer 103, as the case may be.
  • An autoencoder network is sometimes said to learn by "unsupervised” training, because it does not require labeled training data. More precisely, an autoencoder network learns by "self-supervised” training. The learning of an autoencoder network may use stochastic gradient descent training just as in supervised training. However, the autoencoder provides its own objective target, which is just a copy of the input data 101.
  • the computer system provides the input 101 to the autoencoder system 111.
  • the computer system 400 also supplies a copy of input 101 as a target for the output of decoder 104, which is also the output of autoencoder system 111.
  • the autoencoder system 111 is a conventional deterministic autoencoder
  • the autoencoder system 111 computes its output by feed forward activation of the nodes in the neural networks 102, 103, and 104.
  • the autoencoder system 111 is a stochastic autoencoder, such as a VAE or a SCAN
  • the encoder 102 produces as output the values of parameters that determine a parametric probability distribution.
  • the parameters may specify the means and standard deviations (or other central tendency and dispersion statistics) for a set of Gaussian random variables. If the autoencoder system 111 is a VAE, these parameters are trained to minimize the
  • the computer system 400 generates a set of random variables according to the probability distributions specified by the parameters output by the encoder 102 for the input 101. The values of these random variables are then the input for the decoder 104.
  • the output of encoder 104 which is also the output for the autoencoder system 111, may then be computed by the computer system 400 as a feed-forward activation.
  • a difference between the illustrative embodiment shown in Figure 1 from a regular deterministic or stochastic autoencoder comprises the blocks 105 (nearest neighbor finder), 106 (similarity computation) and 107 (minimum distance constraint). These blocks, which represent processes and/or constraints (as opposed to machine learning systems or networks) provide an additional objective to the autoencoder 111 for training the autoencoder 111 as described below. To generate the additional objective, the computer system 400 first sets a minimum distance criterion in block 107. The minimum distance criterion may be determined by a hyperparameter.
  • the computer system 400 finds the nearest neighbor it can in a set of nearest neighbor candidates 113 to the output pattern generated by the decoder 104 for an input data item 101. Any suitable algorithm can be used to find the nearest neighbor, such as the k-nearest neighbors algorithm.
  • the computer system 400 at the nearest neighbor finder block 105 finds near neighbor candidates by retrieving them as the output from a robust auto-associative memory 112, such as a Hopfield network or a robust auto-associative memory as described in PCT Application No. PCT/US 18/51683, filed September 19, 2018, entitled Robust Auto- Associative Memory with Recurrent Neural Network, which is incorporated herein in its entirety.
  • a robust auto-associative memory 112 such as a Hopfield network or a robust auto-associative memory as described in PCT Application No. PCT/US 18/51683, filed September 19, 2018, entitled Robust Auto- Associative Memory with Recurrent Neural Network, which is incorporated herein in its entirety.
  • the dotted arrow from the decoder 104 to the nearest neighbor finder block 105 indicates that the nearest neighbor finder block 105 receives data from the decoder 104 (e.g., the output of the decoder 104) but does not directly back propagate partial derivatives to the decoder 104. Instead, conditional on detection of a violation of the minimum distance constraint at the minimum distance constraint block 107, partial derivatives of a penalty term may be propagated from the minimum distance constraint block 107 directly to the decoder 104.
  • the computer system 400 computes the distance between (i) the nearest neighbor found at the nearest neighbor finder block 105 and (ii) the output pattern from the decoder 104 based on a distance measure.
  • a distance measure may be used, such as the Euclidean distance measure or some other metric in the data space of the input variables.
  • it may be a distance measure in some encoding, such as a feature vector.
  • a feature vector may be obtained, for example, by an encoder trained as a non-stochastic autoencoder with the feature vector as the bottleneck layer.
  • the feature vector may be obtained as an intermediate stage in a classifier, such as, for example, the vector representation in a Word2Vec system.
  • the computer system 400 compares the distance computed at the similarity computation block 106 to a constraint that sets a minimum allowed value for the distance.
  • the computer system 400 may add an extra, positive penalty term to the cost function for the back-propagation of the generator/autoencoder 111 if the minimum distance constraint is violated (i.e., if the distance is less than the minimum allowed distance). This prevents the generator 111 from simply copying the input and helps the generator 111 learn to generalize from the training data 101.
  • Autoencoder systems including an additional objective function such as the system described in connection with Figure 1, could be utilized, for example, as a system to detect plagiarism by being able to detect patterns that are close, but not exact, matches to the original pattern.
  • the penalty term can be constant for any violation, or it could vary based on the scope of the violation.
  • the penalty term could be smaller when the distance to the nearest neighbor is close to (but still less than) the minimum distance and greater when the distance to the nearest neighbor is close to zero.
  • the variability of the penalty could vary linearly or non-linearly with the difference between the distance to the nearest neighbor and the minimum distance constraint.
  • the penalty term added by the minimum distance constraint block 107 has its maximum when the distance between the distance between the output of decoder 104 and the nearest neighbor, as determined by the nearest neighbor finder block 105, is zero. There is no penalty if the distance is greater than the minimum distance constraint.
  • Figures 2 and 3 are generators that use a real-vs-generated discriminator such as is used in a Generative Adversarial Network (GAN).
  • the generator 211, 311 in Figure 2 and 3 may be a conventional auto-encoder, a VAE, a SCAN or a GAN.
  • the generators 211, 311 each also have other objectives that help prevent mode collapse during their training.
  • Figure 2 is similar to Figure 1, except for the addition of the real-vs-generated discriminator 219.
  • Various embodiments of Figure 2 can represent enhancements to a VAE, SCAN or a GAN.
  • the encoder 202 and original patterns 201 are only used in training the generator 211.
  • a GAN as the generator 211, they are not used at all; a GAN has a stochastic layer 203 as its input to the decoder 204 and does not have an encoder 202.
  • the different types of generators have different stochastic layers 203.
  • a GAN generates a vector of random variables directly without training latent variables as output from the encoder 202.
  • a SCAN and a VAE have different constraints in the training of the latent variables at the stochastic layer 203.
  • the system of Figure 2 uses the steps at blocks 205, 206 and 207, which are similar to blocks 105, 106 and 107 of Figure 1, to allow the sub-system to function as an anti -plagiarism detector.
  • Output from the real-vs-generated discriminator 219 is also back-propagated to the generator 211. That is, the real-vs-generated
  • discriminator 219 can determine (or "discriminate") whether the output of the generator 211 is real or generated. If the output of the generator 211 looks generated to the real-vs- generated discriminator 219, the real-vs-generated discriminator 219 can back propagate an additional penalty term to the generator 211 in the training thereof, thereby effectively rewarding the generator 211 for generating realistic output patterns. Blocks 205-207 also prevent the real-vs-generated discriminator 219 from causing the generator 211 to merely memorize the training data 101.
  • the dotted arrow from the decoder 204 to the nearest neighbor finder block 205 indicates that the nearest neighbor finder block 205 receives data from the decoder 204 but does not directly back propagate partial derivatives to the decoder 204. Instead, conditional on detection of a violation by the minimum distance constraint block 207, partial derivatives of a penalty term may be propagated from the minimum distance constraint block 207 directly to the decoder 204.
  • the real-vs-generated discriminator 219 has been trained on a dataset disjoint from the nearest neighbor set 213.
  • the system of Figure 2 may also comprise an additional machine learning system 208 that supplies an additional objective for the training of the generator 211 (e.g., an objective in additional to input data target and the real-vs-generated discriminator 219).
  • the machine learning system 208 may comprise a classifier that has already been trained, before training the generator, or machine learning system 208 may be trained on the data received from the output of the decoder 204. In either case, the machine learning system 208 back propagates the partial derivatives of its classification objective to the output of the decoder 204. This additional objective helps train the generator 21 1 to learn to generate data that is useful for distinguishing the categories in the classifier 208.
  • the machine learning system 208 may be an ensemble of classifiers or a plurality of classifiers with different objectives or different categories.
  • the additional objectives 321 and 322 shown in Figure 3 may alternately be viewed as possible examples of other objectives from the machine learning system 208 in Figure 2.
  • the extra objectives from the machine learning system 208 help any of the types of generator 211 generalize from the training examples 201 and help prevent the mode collapse that might be caused by the real-vs-generated objective 219.
  • Figure 3 is a block diagram of another system that can represent an enhanced version of any of the generator types.
  • Figure 3 has a real-vs-generated discriminator 319 and components, e.g., blocks 305, 306 and 307, that can be utilized to allow the subsystem to function as an anti -plagiarism detector. These components operate similarly to blocks 105, 106, and 107 of Figure 1 and prevent the real-vs-generated discriminator 319 from causing the generator 311 to merely memorize the training data.
  • the dotted arrow from the decoder 304 to the nearest neighbor finder block 305 indicates that the nearest neighbor finder block 305 receives data from the decoder 304 but does not directly back propagate partial derivatives to the decoder 304. Instead, conditional on detection of a violation of a minimum distance constraint block 307, partial derivatives of a penalty term may be propagated from the minimum distance constraint block 307 directly to the decoder 304.
  • Figure 3 adds two new types of objectives.
  • a machine-learning distortion (or noise) detector 322 may be trained to detect noise or distortion in the output of the decoder 304.
  • the distortion (or noise) detector 322 is trained as a
  • the distortion (or noise) detector 322 may back propagate a penalty term to the generator 311 when the distortion (or noise) detector 322 determines that the output of the decoder 304 has above a threshold level of noise, thereby effectively rewarding the generator 311 for generating patterns without noise.
  • a machine-learning classifier 321 provides a different kind of objective; it may be a classifier that is trained separately from the generator 311, and is not necessarily being trained during the training process for the system illustrated in Figure 3.
  • the objective from which classifier 321 back propagates to the generator 311 is a multi -target objective. That is, the target output for the classifier 321 is not a single category but rather includes target activation levels for multiple categories.
  • the objective may be for category A and category B to both have an activation of 0.5.
  • the classifier 321 imposes an implicit constraint that the generated pattern from the decoder 304 not match the training data for the classifier 321.
  • the training data for the classifier 321 can include the training data for the real-vs-generated discriminator 319.
  • the multi-category target of the classifier 321 then prevents an output pattern from the decoder 304 that matches an example in the training data for the discriminator 319 from being an optimum for the overall objective.
  • the real-vs-generated discriminator 319 objective rewards the generator 311 for producing realistic patterns, including the training data for the discriminator 319
  • the objectives from the minimum distance constraint 307 and from the classifier 321 discourage the generator 311 from exactly matching the training data 301 and encourage the decoder 304 to generalize to new patterns.
  • the system illustrated in Figure 3 may have still additional back- propagation training objectives from a machine-learning system 328.
  • the machine-learning system 328 may comprise a classifier or a plurality of classifiers, as explained in association with block 208 of Figure 2.
  • the additional objectives from the machine-learning system 328 may have a regularizing effect, making the training of the generator 311 less likely to overfit the training data and less likely to have mode collapse.
  • Figures 2 and 3 provide two illustrative examples of ways in which multi- objective enhancements to GANs, SCANs, and VAEs can be trained to generalize from their training data and to avoid mode collapse with a real-vs-generated discriminator as one of their objectives.
  • Having multiple objectives helps to avoid mode collapse even without the anti- plagiarism subsystems (e.g., components 205, 206, 207 in Figure 2 or components 305, 306, 307 in Figure 3).
  • the subsystem comprising components 205, 206, 207 in Figure 2 or components 305, 306, 307 in Figure 3 may be omitted.
  • having the other multiple objectives, even without the minimum distance constraint blocks 207 or 307 will enhance the ability of the system (e.g., the data generators 211, 311) to learn to generalize to new data.
  • the multiple objectives will also provide regularization, reducing the effective number of degrees of freedom and reducing the tendency to overfit the training data.
  • the multiple objectives in addition to the real- vs-generated discriminators 219 or 319 will also reduce the tendency of mode collapse in GAN training.
  • Figure 3 A illustrates another aspect of the invention in which there are multiple generators 331, 332, 333 as well as multiple detectors 341, 342, 343, which may each embody more robust versions of the multiple objectives illustrated in Figures 2 and 3.
  • Figure 3 A is a block diagram of an illustrative embodiment of a generative model of a collection of data as a mixture of clusters, with each cluster represented by a respective generator-detector pair 92A-C.
  • each category or other collection of input data is modeled as a mixture of clusters, and there is a generator- detector pair for each cluster.
  • the system 90 is called a mixture of generators model (MGM).
  • MGM mixture of generators model
  • each detector 341, 342, or 343 is not a simple detector but rather may be a complex multi-objective classifier with each output node of each of the classifiers replaced by a more robust 3 -node output distinguishing three outcomes, detection (D), rejection (R), or neutral (N, no decision).
  • Each detector 341, 342, or 343 may embody one or more of the categorical objectives in Figures 2 and 3, that is, the real-vs-generated discriminator (219 or 319), the classifier with a multi-class target 321, other objectives (228 or 328), or the minimum distance constraint (207 or 307). For simplicity and clarity, only one set of three nodes D, R, and N is shown for each detector.
  • the 3-node output of a detector 341, 342, 343 may be a single node, in which case the detector is essentially equivalent to a subsystem of Figure 3, namely a decoder, such as decoder 304 of Figure 3, and one or more objectives, such as 307, 319, 321, 322, and 328 of Figure 3.
  • a two-node output with a softmax constraint is effectively equivalent to a single output node with a sigmoid activation function.
  • a three-node output with a softmax constraint and a "neutral" alternative is a more robust version of the one- or two-node output, in which some data is not classified as either a detection or a rejection. It is more robust in the sense that the category "neutral" creates a margin or buffer region between the accept vs reject alternatives.
  • There are various embodiments for training three-node detectors One embodiment is simply to label some training data as neutral. Another embodiment is during training to label as neutral any data that is classified as "detected" by a plurality of cluster detectors of different categories or that is not labeled as
  • Yet another embodiment is to dynamically relabel training data as “detected,” “rejected,” or “neutral” based on optimizing performance in testing on development data that is disjoint from the training data.
  • Each generator-detector pair 92A-C may comprise one generator and one detector as shown in Figure 3 A, and there preferably is one generator-detector pair 92A- C for each cluster in the category.
  • generator 1 (block 331) and detector 1 (block341) form the generator-detector pair 92A for cluster 1
  • generator 2 (block 332) and detector 2 (block 342) form the generator-detector pair 92B for cluster 2
  • generator 3 (block 333) and detector 3 (block 343) form the generator-detector pair 92C for cluster 3.
  • the generator-detector pairs are individually trained by the computer system 400 as described in connection with Figures 2 and 3.
  • a transmission switch 340 Under control of the computer system 400, a transmission switch 340
  • each generator (331, 332, or 333) is connected to its corresponding detector (341, 342, or 343).
  • a generator may be cross-connected to a different detector, for testing how well each generalizes to data that was used for training the other.
  • the set of real data training data for each generator- detector pair is disjoint from the data for other pairs.
  • a generator is trained to optimize its objectives, such as producing realistic data to fool the real-vs- generated discriminator 207 or 307 in the case of a GAN or minimizing the KL- divergence of the stochastic layer 203 or 303 in the case of a VAE, but a generator is not explicitly constrained from producing data that might also be produced by another generator.
  • each detector 341, 342, and 343 is a multi- objective detector comprising a real-vs-generated detector and other objectives, so each generator 331, 332, and 333 is a "cluster" GAN, for generating data examples within a unique collection of data (e.g., a data category having multiple clusters), with additional objectives.
  • the cluster GANs 331, 332, 333 may constitute a composite GAN.
  • the computer system 400 selects one of the clusters in the category. Each cluster may be selected based on a probability distribution specified through computer system 400. In an illustrative embodiment of GAN training as a two-person game, there are two probability
  • the collection of generators 331-333 represents one player.
  • the computer system 400 trains the probabilities as learned parameters that specify how often the composite GAN generates data from each cluster, representing a von Neumann mixed strategy for the player.
  • the generator for the selected cluster say generator 2 for cluster 2, etc.
  • the computer system 400 uses the generator for the selected cluster, say generator 2 for cluster 2, etc., the computer system 400 generates a data example for selected cluster (e.g., cluster 2) that is sent to the transmission switch 340.
  • the computer system 400 can receive real or generated data from block 335.
  • the real or generated data 305 can be stored in an on-board and/or off-board of the computer system 400. If the data 305 is generated data, it may be generated by a data generator (not shown).
  • the switch 340 sends the data from block 305 to each of the detectors 111-113, one for each cluster.
  • the computer system 400 uses a similar set of probabilities 350 for selecting each of the training data items of real data, with the probabilities over the real data summing to 1.0, thereby representing a von Neumann mixed strategy for the second player, the real-vs-generated discriminator.
  • the optimum strategy for the player represented by the collection of generators 331-333 is to generate only the training data items and to generate each of the training data items with equal probability.
  • the optimum strategy is to reject any data item that does not match a training data item and to specify the probability for each real data item to be equal to the probability of each of the other real data items.
  • the additional objectives force the generator-detector pairs to generalize from the training data to jointly satisfy the multiple objectives.
  • the computer system 400 preferably feeds the "Detect" activation to two nodes.
  • One destination is "Max Node” 351.
  • the activation of Max Node 351 is the maximum of the activations of the "Detect” outputs of all the clusters in a specific category. For example, as shown in Figure 3 A, the activation value of the max node 351 equals the single greatest activation value for the Detect outputs for Detectors 1 through 3.
  • the activation of node 351 may be used as the score for the category in a category classification task.
  • the node 351 preferably receives back propagation during training for the classification task and in turn the computer system 400 back propagates the error cost function derivatives to the detector 341-343 that has the maximum value.
  • the second destination, in the cluster classification mode, is a dedicated node in the node set 361.
  • the computer system 400 sends the "Detect" activation of each cluster detector 341-343 to its respective, dedicated node in the node set 361.
  • the computer system 400 performs a softmax operation for the node set 361; that is, it normalizes the activations of its nodes to sum to one.
  • the node set 361 is trained by the computer system 400 for cluster classification.
  • the target for the node set 361 is a value of one for the correct cluster and a value of zero for all the other nodes.
  • the computer system 400 back propagates this objective to the cluster detectors 341, 342 and 343, respectively.
  • a plurality of the generators 331, 332, 333 in Figure 3 A are GANs (i.e., "cluster GANs")
  • the introduction of probabilities in the node 330 fundamentally changes the composite GAN learning process.
  • the probabilities in node 330 represent the probabilities of a von Neumann mixed strategy for choosing clusters by the player represented by the cluster GAN generators.
  • the probabilities in node 350 represent the probabilities of a von Neumann mixed strategy for choosing real data 355 by the player represented by the real-vs-generated discriminators in the multi-objective detectors.
  • the other objectives in the multi-objective detectors help improve the ability of the system 90 to learn to generalize but are not controlled by the "players" in the notional two-person game of GAN training.
  • the fundamental theorem of the mathematical theory of games is von Neumann's fixed-point minimax theorem, which guarantees that there is a unique value for any two-person zero-sum game and that each player has a probabilistic mixed strategy that guarantees at least that unique value, no matter what strategy the other player may use.
  • This theorem means that a GAN learning method based on von Neumann mixed strategies, such as illustrated in Figure 3 A, will converge to the von Neumann fixed-point value.
  • a GAN learning method based on an embodiment such as illustrated in Figure 3 A avoids the stability problem of other GAN learning methods. In particular, it avoids the cyclical non- convergence behavior of the original GAN implementation based on the players
  • each detector 341, 342, 343 may simply be a copy of a decoder with multiple objectives, such as decoder 304 with the objectives 307, 319, 321, 322, and 328 shown in Figure 3.
  • the system illustrated in Figure 3 A may avoid mode collapse by having a separate cluster generator 331, 332, 333 for each mode.
  • FIG 4 is a diagram of a computer system computer system 400 that could be used to implement the embodiments described above.
  • the illustrated computer system 400 comprises multiple processor units 402 A-B that each comprises, in the illustrated embodiment, multiple (N) sets of processor cores 404A-N.
  • Each processor unit 402A-B may comprise on-board memory (ROM or RAM) (not shown) and off-board memory 406A-B.
  • the on-board memory may comprise primary, volatile and/or non-volatile storage (e.g., storage directly accessible by the processor cores 404A-N).
  • the off-board memory 406A-B may comprise secondary, non-volatile storage (e.g., storage that is not directly accessible by the processor cores 404A-N), such as ROM, HDDs, SSD, flash, etc.
  • the processor cores 404A-N may be CPU cores, GPU cores and/or AI accelerator cores. GPU cores operate in parallel (e.g., a general-purpose GPU (GPGPU) pipeline) and, hence, can typically process data more efficiently that a collection of CPU cores, but all the cores of a GPU execute the same code at one time.
  • AI accelerators are a class of microprocessor designed to accelerate artificial neural networks. They typically are employed as a co-processor in a device with a host CPU 410 as well. An AI accelerator typically has tens of thousands of matrix multiplier units that operate at lower precision than a CPU core, such as 8-bit precision in an AI accelerator versus 64-bit precision in a CPU core.
  • the different processor cores 404 may train and/or implement different networks or subnetworks or components.
  • the cores of the first processor unit 402 A may implement the encoder 102 in Figure 1 and the second processor unit 402B may implement the decoder 104.
  • different sets of cores in the first processor unit 402A may be responsible for the encoder 102 and the decoder 104.
  • one or more of the processor cores 404 and/or one or more of the processor units could implement other components in the systems herein, such as the near neighbor finder 105, the real-vs-generated discriminator 219, the distortion or noise detector 322, the multi -category classifier 321, etc.
  • One or more host processors 410 may coordinate and control the processor units 402 A-B.
  • the system 400 could be implemented with one processor unit 402.
  • the processor units could be co-located or distributed.
  • the processor units 402 may be interconnected by data networks, such as a LAN, WAN, the Internet, etc., using suitable wired and/or wireless data communication links. Data may be shared between the various processing units 402 using suitable data links, such as data buses (preferably high-speed data buses) or network links (e.g., Ethernet).
  • the software for the various computer system 400s described herein and other computer functions described herein may be implemented in computer software using any suitable computer programming language such as .NET, C, C++, Python, and using conventional, functional, or object-oriented techniques.
  • Programming languages for computer software and other computer-implemented instructions may be translated into machine language by a compiler or an assembler before execution and/or may be translated directly at run time by an interpreter.
  • Examples of assembly languages include ARM, MIPS, and x86; examples of high level languages include Ada, BASIC, C, C++, C#, COBOL, Fortran, Java, Lisp, Pascal, Object Pascal, Haskell, ML; and examples of scripting languages include Bourne script, JavaScript, Python, Ruby, Lua, PHP, and Perl.
  • the present invention is directed to machine- learning computer systems and computer-implemented methods for training a machine- learning data generator.
  • the method comprises training, with a computer system 400 that comprises a set of processor cores 404A-N, a machine-learning data generator 111, 211, 311, where training the data generator comprises training the data generator through machine-learning with multiple objectives, including: a first objective that the data generator generate, as output patterns, data patterns that match data patterns in a target set of data patterns; and a second objective that is different from the first objective.
  • a computer system comprises a set of processor cores 404A-N and computer memory (e.g., on-board or off- board memory 406A-B) that is in communication with the set of processor cores.
  • the computer memory stores software that when executed by the set of processor cores, causes the set of processor cores to train the machine-learning data generator 111, 211, 311 through machine-learning with multiple objectives, including: a first objective that the data generator generate, as output patterns, data patterns that match data patterns in a target set of data patterns; and a second objective that is different from the first objective.
  • the data generator comprises a feed-forward neural network and training the data generator comprises training the data generator using stochastic gradient descent.
  • the first objective of the data generator may be to generate, as the output patterns, input patterns that are input to the data generator.
  • the data generator may comprise a deterministic autoencoder or a stochastic autoencoder, such as a variational autoencoder or a stochastic categorical autoencoder network.
  • the data generator may comprise a generative adversarial network.
  • the second objective may comprise an objective from an anti-plagiarism subsystem (e.g., components 105-107 of Figure 1).
  • the objective from the anti -plagiarism subsystem may back propagate partial derivatives of an anti -plagiarism penalty term through the data generator when an output pattern from the data generator is within a minimum threshold distance of a nearest neighbor pattern that is a nearest neighbor in a set of nearest neighbor candidate patterns to the output pattern.
  • the computer software may store software that when executed by the set of processor cores causes the set of processor cores to generate the set of nearest neighbor candidate patterns with a Hopfield network or with an auto-associative memory.
  • the computer memory may store software that when executed by the set of processor cores causes the set of processor cores to compute a distance between the output pattern and the nearest neighbor candidate pattern.
  • the distance may comprise a Euclidean distance between the output pattern and the nearest neighbor candidate pattern.
  • the anti -plagiarism penalty term preferably is inversely related to the distance between the output pattern and the nearest neighbor candidate pattern.
  • the multiple objectives may comprise a third (or second as the case may be) objective that is an output of a real-vs-generated discriminator, such that a generated-data penalty term is back-propagated to the data generator when an output pattern from the data generator is determined to be generated by the real-vs-generated discriminator.
  • the multiple objectives may comprise an additional objective that is an output of a machine learning sub-system that comprises a classifier, such that partial derivatives of a classification objective of the machine-learning sub-system are back-propagated to the data generator.
  • the machine-learning sub-system may comprise an ensemble of classifiers or a multi -target classification objective.
  • the multiple objectives may comprise an objective that is an output of a machine-learning distortion detector, such that this objective back propagates an excess distortion penalty term to the data generator.
  • the present invention is directed to a computer- implemented method that comprises the step of training, by a computer system 400 that comprises a set of processor cores 402A-N, a composite generative adversarial network (GAN) for producing data examples in a collection of data.
  • the composite GAN comprises N generator-detector pairs 92A-C, where N > 1.
  • Each of the N generator-detector pair comprises: a cluster GAN 331-333 for generating data for one cluster in the collection of data, such that there are N clusters, one for each generator-detector pair; and a corresponding multi -objective classifier 341-343 for classifying data in the cluster of the corresponding cluster GAN of the generator-detector pair.
  • Each of the N multi -objective classifiers 341-343 comprises a real-vs-generated discriminator for discriminating whether input data to the multi-objective classifier is real data or generated data.
  • one of the N clusters is selected based on a first specified probability distribution.
  • a probability that a training example for the multi-objective classifiers is a specific example of real data is set based on a second specified probability distribution.
  • the present invention is directed to a computer system 400 that comprises a set of processor cores 402A-N and computer memory (e.g., off-board memory 406A-B) that stores software that, when executed by the set of processor cores, causes the set of processor cores to train a composite generative adversarial network (GAN) for producing data examples in a collection of data as described above.
  • GAN composite generative adversarial network
  • the first specified probability distribution corresponds to a first von Neumann mixed strategy of a first player in a two-player GAN training game
  • the second specified probability distribution corresponds to a second von Neumann mixed strategy of a second player in the two-player GAN training game.
  • the collection of data may comprise a data category
  • a machine-learning data generator can avoid generating data that is too similar to any previously known data example.
  • Such generated, or “synthetic,” data can be used in a wide variety of applications, such as to train computer-vision systems, image classifiers or other types of classifiers, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

Selon la présente invention, des générateurs de données d'apprentissage automatique utilisent un objectif supplémentaire pour éviter la génération de données qui sont trop similaires à un quelconque exemple de données déjà apprises. Ceci empêche le plagiat ou la simple copie d'exemples de données existantes, ce qui permet d'améliorer la capacité d'un générateur à générer de manière utile de nouvelles données. Une formulation d'apprentissage de réseau contradictoire génératif (GAN) en tant que solution minimax à stratégie mixte d'un jeu à somme nulle résout le problème de convergence et de stabilité de l'apprentissage des GAN, sans subir un effondrement de mode.
PCT/US2018/053295 2017-09-28 2018-09-28 Générateurs multi-objectifs en apprentissage profond Ceased WO2019067831A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US16/646,096 US11074505B2 (en) 2017-09-28 2018-09-28 Multi-objective generators in deep learning
CN201880067035.3A CN111226236B (zh) 2017-09-28 2018-09-28 深度学习中的多目标生成器
EP18861823.5A EP3688677A4 (fr) 2017-09-28 2018-09-28 Générateurs multi-objectifs en apprentissage profond

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201762564754P 2017-09-28 2017-09-28
US62/564,754 2017-09-28
USPCT/US2018/051069 2018-09-14
PCT/US2018/051069 WO2019067236A1 (fr) 2017-09-28 2018-09-14 Mélange de modèles de générateurs

Publications (1)

Publication Number Publication Date
WO2019067831A1 true WO2019067831A1 (fr) 2019-04-04

Family

ID=65902125

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/053295 Ceased WO2019067831A1 (fr) 2017-09-28 2018-09-28 Générateurs multi-objectifs en apprentissage profond

Country Status (1)

Country Link
WO (1) WO2019067831A1 (fr)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110766071A (zh) * 2019-10-21 2020-02-07 北京工业大学 一种基于森林自编码器的脑网络数据增强方法
US10832137B2 (en) 2018-01-30 2020-11-10 D5Ai Llc Merging multiple nodal networks
US10885470B2 (en) 2017-06-26 2021-01-05 D5Ai Llc Selective training for decorrelation of errors
US10922587B2 (en) 2018-07-03 2021-02-16 D5Ai Llc Analyzing and correcting vulnerabilities in neural networks
US10929757B2 (en) 2018-01-30 2021-02-23 D5Ai Llc Creating and training a second nodal network to perform a subtask of a primary nodal network
US10956818B2 (en) 2017-06-08 2021-03-23 D5Ai Llc Data splitting by gradient direction for neural networks
US11003982B2 (en) 2017-06-27 2021-05-11 D5Ai Llc Aligned training of deep networks
US11010670B2 (en) 2018-08-27 2021-05-18 D5Ai Llc Building a deep neural network with diverse strata
US11037059B2 (en) 2018-08-31 2021-06-15 D5Ai Llc Self-supervised back propagation for deep learning
US11074505B2 (en) 2017-09-28 2021-07-27 D5Ai Llc Multi-objective generators in deep learning
US11074502B2 (en) 2018-08-23 2021-07-27 D5Ai Llc Efficiently building deep neural networks
US11195097B2 (en) 2018-07-16 2021-12-07 D5Ai Llc Building ensembles for deep learning by parallel data splitting
US11270188B2 (en) 2017-09-28 2022-03-08 D5Ai Llc Joint optimization of ensembles in deep learning
US11295210B2 (en) 2017-06-05 2022-04-05 D5Ai Llc Asynchronous agents with learning coaches and structurally modifying deep neural networks without performance degradation
US11321612B2 (en) 2018-01-30 2022-05-03 D5Ai Llc Self-organizing partially ordered networks and soft-tying learned parameters, such as connection weights
CN114880924A (zh) * 2022-04-22 2022-08-09 东莞中科云计算研究院 基于深度学习的牙齿修复体自动设计方法及系统
CN114970818A (zh) * 2022-05-25 2022-08-30 重庆大学 一种减少生成对抗网络模式崩溃的方法
US11501164B2 (en) 2018-08-09 2022-11-15 D5Ai Llc Companion analysis network in deep learning
US11676026B2 (en) 2018-06-29 2023-06-13 D5Ai Llc Using back propagation computation as data
US11836600B2 (en) 2020-08-20 2023-12-05 D5Ai Llc Targeted incremental growth with continual learning in deep neural networks

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6128606A (en) * 1997-03-11 2000-10-03 At&T Corporation Module for constructing trainable modular network in which each module inputs and outputs data structured as a graph
US8527276B1 (en) * 2012-10-25 2013-09-03 Google Inc. Speech synthesis using deep neural networks
US20140358828A1 (en) * 2013-05-29 2014-12-04 Purepredictive, Inc. Machine learning generated action plan

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6128606A (en) * 1997-03-11 2000-10-03 At&T Corporation Module for constructing trainable modular network in which each module inputs and outputs data structured as a graph
US8527276B1 (en) * 2012-10-25 2013-09-03 Google Inc. Speech synthesis using deep neural networks
US20140358828A1 (en) * 2013-05-29 2014-12-04 Purepredictive, Inc. Machine learning generated action plan

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
SALUNKHE, S ET AL.: "A Plagiarism Detection Mechanism using Reinforcement Learning", INTERNATIONAL JOURNAL OF ADVANCE RESEARCH IN COMPUTER SCIENCE AND MANAGEMENT STUDIES, vol. 1, no. 6, November 2013 (2013-11-01), pages 125 - 129, XP055585865, Retrieved from the Internet <URL:http://www.ijarcsms.com/docs/paper/volume1/issue6/V1I6-0019.pdf> [retrieved on 20181031] *
See also references of EP3688677A4 *
SPRINGENBERG, J ET AL.: "UNSUPERVISED AND SEMI-SUPERVISED LEARNING WITH CATEGORICAL GENERATIVE ADVERSARIAL NETWORKS", CONFERENCE PAPER AT ICLR 2016, 30 April 2016 (2016-04-30), pages 1 - 20, XP055506693, Retrieved from the Internet <URL:htts://arxiv.or/df/1511.06390.pdf> [retrieved on 20181031] *

Cited By (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12271821B2 (en) 2017-06-05 2025-04-08 D5Ai Llc Training an autoencoder with a classifier
US12061986B2 (en) 2017-06-05 2024-08-13 D5Ai Llc Adding a split detector compound node to a deep neural network
US11790235B2 (en) 2017-06-05 2023-10-17 D5Ai Llc Deep neural network with compound node functioning as a detector and rejecter
US11562246B2 (en) 2017-06-05 2023-01-24 D5Ai Llc Asynchronous agents with learning coaches and structurally modifying deep neural networks without performance degradation
US11392832B2 (en) 2017-06-05 2022-07-19 D5Ai Llc Asynchronous agents with learning coaches and structurally modifying deep neural networks without performance degradation
US11295210B2 (en) 2017-06-05 2022-04-05 D5Ai Llc Asynchronous agents with learning coaches and structurally modifying deep neural networks without performance degradation
US10956818B2 (en) 2017-06-08 2021-03-23 D5Ai Llc Data splitting by gradient direction for neural networks
US10885470B2 (en) 2017-06-26 2021-01-05 D5Ai Llc Selective training for decorrelation of errors
US11003982B2 (en) 2017-06-27 2021-05-11 D5Ai Llc Aligned training of deep networks
US11270188B2 (en) 2017-09-28 2022-03-08 D5Ai Llc Joint optimization of ensembles in deep learning
US11074505B2 (en) 2017-09-28 2021-07-27 D5Ai Llc Multi-objective generators in deep learning
US11748624B2 (en) 2018-01-30 2023-09-05 D5Ai Llc Evaluating the value of connecting a selected pair of unconnected nodes of a nodal network
US11087217B2 (en) 2018-01-30 2021-08-10 D5Ai Llc Directly connecting nodes of different copies on an unrolled recursive neural network
US11093830B2 (en) 2018-01-30 2021-08-17 D5Ai Llc Stacking multiple nodal networks
US11151455B2 (en) 2018-01-30 2021-10-19 D5Ai Llc Counter-tying nodes of a nodal network
US12182712B2 (en) 2018-01-30 2024-12-31 D5Ai Llc Node-splitting for neural networks based on magnitude of norm of vector of partial derivatives
US10929757B2 (en) 2018-01-30 2021-02-23 D5Ai Llc Creating and training a second nodal network to perform a subtask of a primary nodal network
US12430559B2 (en) 2018-01-30 2025-09-30 D5Ai Llc Accelerated training of a neural network via regularization
US11321612B2 (en) 2018-01-30 2022-05-03 D5Ai Llc Self-organizing partially ordered networks and soft-tying learned parameters, such as connection weights
US11010671B2 (en) 2018-01-30 2021-05-18 D5Ai Llc Iterative training of a nodal network with data influence weights
US12288161B2 (en) 2018-01-30 2025-04-29 D5Ai Llc Creating diverse neural networks with node tying
US10832137B2 (en) 2018-01-30 2020-11-10 D5Ai Llc Merging multiple nodal networks
US11461655B2 (en) 2018-01-30 2022-10-04 D5Ai Llc Self-organizing partially ordered networks
US11676026B2 (en) 2018-06-29 2023-06-13 D5Ai Llc Using back propagation computation as data
US10922587B2 (en) 2018-07-03 2021-02-16 D5Ai Llc Analyzing and correcting vulnerabilities in neural networks
US11195097B2 (en) 2018-07-16 2021-12-07 D5Ai Llc Building ensembles for deep learning by parallel data splitting
US11501164B2 (en) 2018-08-09 2022-11-15 D5Ai Llc Companion analysis network in deep learning
US11074502B2 (en) 2018-08-23 2021-07-27 D5Ai Llc Efficiently building deep neural networks
US11010670B2 (en) 2018-08-27 2021-05-18 D5Ai Llc Building a deep neural network with diverse strata
US11037059B2 (en) 2018-08-31 2021-06-15 D5Ai Llc Self-supervised back propagation for deep learning
CN110766071A (zh) * 2019-10-21 2020-02-07 北京工业大学 一种基于森林自编码器的脑网络数据增强方法
CN110766071B (zh) * 2019-10-21 2023-04-28 北京工业大学 一种基于森林自编码器的脑网络数据增强方法
US11836600B2 (en) 2020-08-20 2023-12-05 D5Ai Llc Targeted incremental growth with continual learning in deep neural networks
US11948063B2 (en) 2020-08-20 2024-04-02 D5Ai Llc Improving a deep neural network with node-to-node relationship regularization
US12205010B2 (en) 2020-08-20 2025-01-21 D5Ai Llc Targeted incremental growth with continual learning in deep neural networks
US12346792B2 (en) 2020-08-20 2025-07-01 D5Ai Llc Accelerated training of neural networks with regularization links
CN114880924A (zh) * 2022-04-22 2022-08-09 东莞中科云计算研究院 基于深度学习的牙齿修复体自动设计方法及系统
CN114970818A (zh) * 2022-05-25 2022-08-30 重庆大学 一种减少生成对抗网络模式崩溃的方法

Similar Documents

Publication Publication Date Title
US11074505B2 (en) Multi-objective generators in deep learning
WO2019067831A1 (fr) Générateurs multi-objectifs en apprentissage profond
US20200410090A1 (en) Robust von neumann ensembles for deep learning
US11270188B2 (en) Joint optimization of ensembles in deep learning
Kotsiantis Combining bagging, boosting, rotation forest and random subspace methods
US11250328B2 (en) Cooperative evolution of deep neural network structures
US11087086B2 (en) Named-entity recognition through sequence of classification using a deep learning neural network
Bhowan et al. Evolving diverse ensembles using genetic programming for classification with unbalanced data
Risi et al. How novelty search escapes the deceptive trap of learning to learn
US11195097B2 (en) Building ensembles for deep learning by parallel data splitting
US12450339B2 (en) Diversity for detection and correction of adversarial attacks
Lu et al. Counting crowd by weighing counts: A sequential decision-making perspective
Utkin et al. A deep forest improvement by using weighted schemes
Tovias-Alanis et al. Learning smooth dendrite morphological neurons for pattern classification using linkage trees and evolutionary-based hyperparameter tuning
Rani et al. Searching and evolving test cases using moth flame optimisation for mutation testing
Smart Genetic programming for multiclass object classification
Curry et al. One-class genetic programming
Kim et al. Robust inference of Bayesian networks using speciated evolution and ensemble
Lai et al. Tracking-removed neural network with graph information for classification of incomplete data
ChingHao et al. Fair Multi-exit Framework for Facial Attribute Classification
Trivedi Comparison of Evolutionary Strategies and Reinforcement Learning Algorithms on Custom and Non-Conventional Environment
Soler Estimation of Distribution Algorithms for Generative Adversarial and Convolutional Neural Network Hyper-Parameter Optimization
Dutta et al. Forward-Cooperation-Backward (FCB) learning in a Multi-Encoding Uni-Decoding neural network architecture
Kumar The Multi-Objective Genetic Algorithm Based Techniques for Intrusion Detection
Cordero Towards Robust Design and Training of Deep Neural Networks

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18861823

Country of ref document: EP

Kind code of ref document: A1

DPE1 Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101)
NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2018861823

Country of ref document: EP

Effective date: 20200428