JP7507791B2 - SYSTEM AND METHOD FOR SYNTHESISING APPAREL ENSEMBLES FOR MODELS - Patent application - Google Patents

Description

Continuation and Priority Claims This international patent application claims priority to U.S. Patent Application No. 16/422,278, filed May 24, 2019. The parent application is a continuation-in-part of U.S. Patent Application No. 15/634,171, filed June 27, 2017 (U.S. Patent No. 10,304,227, published May 28, 2019). The prior disclosures are incorporated by reference.

TECHNICAL FIELD The present invention relates to automatic image generation. More particularly, the present invention relates to a method for creating a composite image based on a training set of image pairs. Once trained, an embodiment can accept a new image that is similar to one member of the image pair and create a composite image that represents what the other image in the pair might look like.

Neural networks are computer systems inspired by biological processes. In particular, they comprise multiple modules that are designed to behave similarly to the way biological neurons are thought to function. Neurons are often modeled as multiple-input, single-output thresholding accumulators that "fire" if enough inputs are sufficiently active (the output of one model neuron may be connected to many subsequent neuron model inputs, or even to previous neuron models in a feedback loop). While the functioning of a single neuron model is very simple, it has been observed that a large number of models operating simultaneously, properly constructed or "trained", can produce surprisingly good results for problems that are difficult to address using traditional data processing or programming techniques.

One common neural network task is image recognition. Multiple neurons are arranged in a pyramid-like hierarchy with an array of input neurons for each pixel in the image, followed by one or more layers of neurons of decreasing size, and an output neuron designated to indicate whether the input image has a property of interest. This type of network can be "trained" by exposure to a set of images in which the property of interest is either present or absent. For example, the property might be whether a penguin is present in the image. Once trained, the network can be able to determine with considerable accuracy whether a penguin is depicted in a new image that was not part of the training set.

Neural networks can also be used to generate or synthesize new information based on traditional training and random seeds. For example, biophysicist Dr. Mike Tyka has attempted to train a neural network to generate images that resemble artistic depictions of human faces, although the images do not contain any actual subjects and any resemblance to individuals is purely coincidental.

Although neural networks are not infallible (a recognizer can misidentify a target, or a generative network can construct an output that cannot be used for its intended purpose), they are often practical in applications where deterministic methods are too slow, too complex, or too expensive. Thus, neural networks can bridge the gap between mechanical methods that are easy to implement on a computer, and labor-intensive methods that rely on human judgment to achieve the best results.

The neural network is trained with a set of images, where two or more images in each set show an item of apparel (clothing, shoes, accessories, etc.) and one image shows a model wearing the item of apparel. Once trained, a new set of "apparel" images is presented to the network and a synthetic image that resembles the model wearing the item of apparel is automatically generated. The synthetic image is displayed to the user.

1 is a flow chart outlining a method according to one embodiment of the present invention. 2 is a simplified representation of data flow through a neural network implementing one embodiment of the present invention. 13 shows the range of synthetic images that are produced as the "model pose" image control parameter is varied. 1 is a flow chart outlining a more comprehensive application of one embodiment. 11 is a flow chart outlining another application of an embodiment. 2 is a simplified representation of data flow through a neural network implementing one embodiment of the present invention. 4 is a more detailed representation of data flow through a neural network implementing one embodiment of the present invention. 1 is a flow chart outlining an efficient method for training a neural network implementing an embodiment of the present invention.

Most people wear clothes, and many people select and purchase some of their clothing from retailers that do not (or cannot) offer customers the opportunity to try on the clothing before purchasing. Catalogs and online retailers (among others) often go to great lengths to produce photographs of clothing worn by representative models and presented in typical contexts. These photographs are important for conveying an impression of the fit and appearance of the garment to customers who cannot see it in person.

An embodiment of the invention uses neural networks to synthesize an image of a model wearing a garment from a lower-cost image of the garment alone. Beneficially, the synthesis process exposes controls that can be used to adjust the properties of the synthetic image. For example, the skin tone, body shape, posture, and other properties of the model in the synthetic image may be varied over a useful range. This allows a user (e.g., a customer) to adjust the synthetic "as worn" image to more closely resemble themselves. Thus, in addition to reducing the cost of generating representative photographs, an embodiment allows a user to better understand how a particular garment might look when worn by the user. This improved representation may improve a retailer's chances of a sale (or avoid a buyer returning a garment that they don't like when they try it on).

Embodiments achieve this synthesis by harnessing the power of software systems known as "neural networks." Neural networks are often used to perform automatic recognition of objects and features in images; they can answer questions such as "Does this image show the Eiffel Tower?" or "Does this image contain a person wearing a hat?" Neural networks can also be configured to generate, or synthesize, new images that are similar to other images they were trained on. When these two types of neural networks (recognition and synthesis) are inferred together, they form a generative adversarial network ("GAN"), where the synthesis part optimizes its output to generate images that cause the recognition part to answer "yes," and the recognition part optimizes its analysis to reduce the chance of being "fooled" by the synthetic image. Thus, one possible GAN can be trained to generate better images of people wearing hats (it should be recognized that some synthetic images of "people wearing hats" may not look like people wearing hats and may fool the recognizer, but in general, if the training of the network is performed carefully, many or most synthetic images will be useful in a process that requires images that humans recognize as "photos of people wearing hats"). Figure 1 shows an overview of the core of the operation of one embodiment. The method begins with initializing a neural network (100). The network is trained (110) with pairs of images, one image of each pair showing the garment (e.g., the garment laid out flat on a neutral surface) and the other image of each pair showing a model wearing the garment. Once training is complete, useful parameters are identified (120) from the "Z-vector" (described below).

Now, in a typical use, a clothing image is provided to the trained network (130). The clothing image does not have to be one of the training images (but preferably is not one of the training images), but is instead an image without a corresponding counterpart. The network synthesizes (140) this clothing image onto the model based on the network's training and Z-vector parameters. The synthesized image is displayed (150) to the user. The user may adjust (160) the Z-vector parameters, and the new image is synthesized (140) and displayed (150). The adjustment and resynthesis may be repeated as many times as desired to generate various synthesized images of the model wearing the clothing.

The Z-vector parameters may control characteristics such as the model's skin color, body shape and size, pose or position, and even accessories (e.g., shoe style, handbag, glasses, scarf or jewelry), so that the user may be able to control the compositing process to produce an image that more closely resembles what the garment would look like if worn by the user.

2 shows a conceptual representation of a generative adversarial network, which is one type of neural network that can be used in an embodiment of the present invention. Information can be understood as generally passing from left to right across the drawing. An image of clothing 210 is passed to an input layer 221 of a first neural network 220. The input layer may have approximately as many input elements as there are pixels in the input image (i.e., more elements than are shown in this drawing). Alternatively, the input layer may comprise an element for each color channel of each input pixel (e.g., red, green and blue elements per pixel). The input layer 221 is coupled to a middle layer 223 by a network of variable connection weights 222, which in turn is coupled to an output layer 225 by a similar network 224. Although the connections are shown all-to-all, each connection may be associated with a variable weight, so that some connections may be effectively absent (weight=0). In addition to layer-to-layer connections, an embodiment may use feedback connections 226, which may be made to the immediately preceding layer or to an earlier layer. Each neural network may have more than the three layers shown here; in the preferred embodiment, about seven layers are used.

Note that each layer of neural network 220 (progressing from left to right) is smaller than the previous layer. Thus, the network can be thought of as performing a kind of compression, resulting in Z-vector 250, which in many embodiments is a vector of real numbers comprising the output of the last layer of network 220.

The Z-vector 250 is used as input to a second neural network 230. This network (with layers of model neurons 231, 233 and 235 interconnected by variable weight connections 232, 234) is similar in structure to network 220, but the number of elements in each layer is increased (in the direction of data flow). Like network 220, network 230 may include feedback connections (e.g. 236) that carry data in the "reverse" direction. The output of network 230 is a synthetic image 240 that represents what the garment 210 would look like if worn by the model once the GAN has been trained.

In addition to the various intra-network connections, an embodiment may use inter-network connections 227, 228, or 237. These typically connect layers of similar depth (227, 237), although connections between different depths (228) may be provided (where "depth" means "a level away from the input of the left network 220 or a level away from the output of the right network 230," acknowledging the fact that the networks are to some extent mirror images of each other). These connections may be referred to as "skip connections." Conceptually, skip connections provide a simple and direct way for the network to pass information that does not significantly affect the "compression" and "composition" operations. For example, clothing color is largely orthogonal to the question of how fabrics tend to hang and drape in the figure. A red dress and a blue dress of equivalent construction will look nearly identical on the model, except for the color itself. Thus, rather than relying on the input network to recognize the color and convey it to the generative network through the Z vector, the skip connections can carry the color of the sample image directly up to the output network.

Returning to the idea that the network 220 performs something like compression, we see that some of the Z-vector elements encode properties that the GAN has learned to recognize (and use in image synthesis) after processing the training image pairs. For example, one Z-vector element may control the skin color of the model. Another element may control the model's pose (including whether the model is facing the camera or to one side, or the position of the arms or legs). An element may control the style of shoes depicted in the synthetic image. These may resemble one of the styles learned from the training images, but the shoes in the output image are not simply replicated from one of the legs of the training image model. Instead, the generative network 230 builds an image including the shoes or other accessories, which looks like what might have been seen in the training images if the input image 210 had an image of one of the corresponding model clothing pairs. And crucially, adjusting the Z-vector elements can cause the generative network to create new images with different skin tones, model poses, or shoe styles. The Z-vector elements may also control details, such as the length of the dress, the length of the sleeves, or the style of the collar. That is, the synthesized images may show how the clothing may be altered as well as how the model may be altered. In one exemplary embodiment, even if the input dress image is shown from the front, the Z vector component can control the rotation of the model to generate a range of plausible images of the model wearing the dress with left to right model rotations across that range. An example of this is shown in Figure 3. All 25 images were synthesized from a single image of a black dress by a network trained with photographs of various outfit/model pairs.

Although a generative adversarial network ("GAN") as depicted and described with reference to FIG. 2 is one type of neural network suitable for use in one embodiment, other well-known types can also be used. For example, network configurations known in the art as recurrent neural networks ("RNN"), recurrent interfering machines ("RIM"), and variational autoencoders ("VAE") can all be trained with pairs of images as described above to generate new synthetic images that may be plausible counterparts of new input images, exposing quantitative control parameters that can be varied proportionally to adjust the features or characteristics of the synthetic images. The above-described aspects of neural networks are important for use in embodiments of the present invention, namely:
● It is trainable by image pairs ● It generates a synthetic image from a new input image, where the synthetic image resembles the "pair" image that corresponds to the new input image (i.e., the new input image and the synthetic image could plausibly have been paired in the training set)
• Expose quantitative control parameters that can be manipulated to change the properties of the composite image, including properties such as:
○ Model's skin tone ○ Model's posture ○ Model's weight, body shape ○ Accessories ○ Garment length ○ Garment sleeve style ○ Garment collar style ○ Garment fit

One feature of neural networks that is suitable for use in one embodiment was mentioned briefly above, but is now considered further. In FIG. 2, each node at a particular level was depicted as being connected to all nodes at the next level. In addition, some inter-level connections were shown. However, neural networks may be fully connected. The weighted output of each node may form part of the input signal for all other nodes (even including the node itself). This type of network is very difficult to depict in a two-dimensional diagram, but it is a common topology well known to those skilled in the art. Another alternative is the convolutional neural network. Because this topology is more space-efficient than a fully connected network, it can often operate on larger input images (and generate higher resolution output images). Convolutional networks are also well known to those skilled in the art and can be used effectively by adhering to the principles and methods described herein.

In an exemplary embodiment of the invention using a generative adversarial network, the input image may be approximately 192x256 pixels with three color channels (red, green and blue) for each pixel. The input layer therefore comprises 192x256x3 = 147,456 neuron model elements. The next layer reduces by a factor of two, and the Z vector ends up as 512x4x3 = 6,144 scalars. The generative network may mirror the input network, starting with the Z vector and emitting a 192x256 synthetic color image.

Not all elements of a Z-vector correspond to recognizable features such as "skin color," "dress length," "shoe style," or "model pose." The influence of individual vector elements (and subsets of vectors) may be determined empirically or by providing additional information about training image pairs (e.g., by tagging the images with feature descriptions) and propagating the additional information through the network during training.

One preferred method of determining the influence of the Z vector components is to perform a principal components analysis ("PCA") of the Z vector and identify a smaller vector Z' whose components are primarily linearly independent. The components of Z' may be tested to determine their influence, and those that affect the property of interest may be exposed to the user to control the synthetic image generation.

Figure 4 outlines the methodology used by a complete application built around an image synthesis unit according to one embodiment of the present invention. To begin, the system operator initializes and trains the neural network as described above using a training set of image pairs, where one member of each pair depicts a garment and the other member of each pair depicts a model wearing the garment (410). Next, a database of garment images is populated (420). These images are similar to the first image in the training set and may even include training images. These may be, for example, images of garments sold by the system operator.

When a customer visits the operator's system (e.g., when the customer accesses an e-commerce website), she may search or browse a catalog of clothing (430) using any suitable prior art method. For example, clothing may be categorized and presented by color, style, weight, designer, size, price, or any other desired arrangement. When the user selects a garment (440), the system composites and displays an image of the garment onto the model (450). The user may be provided with control arrays that are coupled to the appropriate elements of the Z vector, and she may adjust their parameters as desired (460). Once the parameters are adjusted, the system composites and displays a new image of the garment onto the model (450).

If the user decides not to purchase the garment (470), for example by clicking a "back" button or returning to the list of search results, she may continue to view other garments for sale (430). If the user decides to purchase the garment (480), information about the selected garment (e.g., SKU) is communicated to the prior art order fulfillment process for further action (490).

An embodiment of the invention may combine clothing selection controls with controls for other aspects of image generation (skin tone, pose, body shape, accessories, etc.). Then, by manipulating individual controls of this plurality of controls, the user can change clothing (leaving only skin tone, pose and accessories) or switch accessories (leaving only skin tone, pose and clothing). This embodiment allows for rapid, spontaneous comparison among complete "outfits" or "looks" possibilities that are provided at great expense by human fashion coordinators and therefore often unavailable to shoppers of ordinary or modest means.

Figure 5 outlines another application of the image synthesis network described above. As described above, the method begins by initializing and training the image synthesis network with a training set of image pairs (500). The system then acquires clothing images (510), synthesizes the clothing images onto the model (520), and stores the synthesized images (530). If there are more clothing images to process (540), these steps are repeated, resulting in a library of synthesized images showing the various garments whose images have been acquired and processed. Image synthesis may use randomly selected Z-vector parameters that result in a variety of different model skin tones, body shapes, poses, and accessories in the synthesized image library.

When there are no garment images to process (550), the composite images from the library may be incorporated into the catalog layout (560), printed (570), or static web pages may be generated (580) with one or more composite images, which may be served to visitors to the website (590). This process may reduce the cost of generating a catalog of product images or a website that displays many garments.

It should be appreciated that the method may be integrated with prior art garment processing sequences, for example a consignment store that receives many outsourced garments from various manufacturers. These garments may be passed through a steam room to eliminate storage or transportation odors and to remove wrinkles. The garments may be placed on a mannequin during this process, and an image of the freshly steamed garment may be automatically captured at the end. This image may be delivered to the process outlined in FIG. 5 as an "acquired image" at 510. Multiple views of the garment may be acquired, allowing the synthesis of more diverse "garment on model" images once the neural network has been trained with correspondingly more diverse image pairs.

The neural network-based image synthesis system described in this application can be extended to perform more complex and useful tasks without substantially departing from the concepts described above. Extension generally involves training a neural network with a set of input images, which are classified into a number of categories. After training, the neural network is used to create new synthetic images from the inputs, preferably including never-before-seen images that fit the training categories. To make this concrete, recall that the system described above is trained with pairs of images, the first of which shows only clothing, and the second of which shows a model wearing the clothing. Once trained, a new image of only the clothing is presented, and the network synthesizes a new image that resembles the hypothetical model wearing the clothing. The system can generate synthetic images of models that include features other than the input clothing, provided that some of the training images showed such elements. For example, as described above, an element of the Z vector may control the style of shoes depicted. However, the shoes, as well as the model itself, were entirely the creation of the image synthesis unit. The input analyzer or compressor encountered training images containing models wearing shoes and "learned" to generate plausible synthetic images that might contain these kinds of shoes.

In the system described below, the training images include a group or set of two or more related "component" images: clothing, shoes, and accessories (e.g., handbags, bracelets, hats, etc.), and a "target" image of a model wearing the clothing, shoes, and accessories. The goal of training is to "teach" the network to recognize features in the target images that indicate the model is wearing the apparel items depicted in the component images. Finally, following the conventions of generative adversarial networks ("GANs"), the image synthesizer network is contrasted with the recognizer so that the synthesizer learns to create new images that the recognizer recognizes as valid or plausible photographs of a model wearing the input clothing and accessories. Once the system is trained, it can generate images that look like a model wearing the clothing, shoes, and accessories, without the need for a real model to wear the items and appear in a photo session. Of course, the synthesis control parameters (i.e., elements of the Z vector) described above can be used to adjust the images synthesized by this more complex system as well.

Figure 6 shows an overview of the system of the present invention with sample input categories on the left, sample output (or training) images on the right, and a neural network 600, represented as an input recognition ("compression") network 620 with a reduced topology generating Z-vectors 650, operated by an output "composition" network 630 with an expanded topology to generate composite images showing plausible models wearing various input items of apparel. One embodiment of the system accepts at least two different types of input apparel data and generates at least one composite image that combines the inputs in a plausible way given the training of the network. Here, "plausible" does not mean "a copy of a previously seen training image with some parts replaced by parts of the input image". Instead, a "plausible" image is one that the recognizer (after training) would assess as being an image showing a model wearing the input clothing and accessories.

The black box simple implementation of the neural network of FIG. 6 may simply synthesize all of the input image data into a single large image, then rely on a training process to teach the network to recognize different parts of the larger image, and then synthesize images of a model wearing all of the input items. However, the more advanced implementation shown in FIG. 7 may deliver the input information to the neural network more efficiently for simpler training. It also exposes more useful control features to the user, allowing the system to be used in an almost "modular" manner.

In a preferred embodiment, two or more input networks 721, 722, ..., 729 receive different categories of inputs. One network 721 may receive images of clothing 711, while the other network 729 may receive images of shoes 719a,b (the inventors note that the performance of the "shoes" network can be improved by providing multiple images of the same shoe, e.g., front and side views). Each input network may be a separate multi-layer neural network with a reduced number of nodes at each level, distilling the information of its respective input into a corresponding Z-vector 751, 752, ..., 759 by a process such as compression as described above.

From these Z-vectors, multiple output networks 731, 732, ..., 739, operating independently according to basic inventive principles, can be trained to generate composite images that include the model and their respective inputs. The "clothing" network can generate images of the model wearing the input clothing, and the "shoes" network can generate images of one (or both) legs of the model wearing the input shoes. However, in this preferred embodiment, the separate Z-vectors 751, 752, ..., 759 are combined (e.g. concatenated) to form a combined Z-vector 760, which is delivered to a multi-element image combiner neural network 770. This output network creates an image of the model wearing the clothing and shoes (plus any other accessories whose images are provided to other input networks and concatenated into the composite Z-vector 760, e.g. pants 712, whose images are provided through the input neural network 722).

When the input networks are separated in this way, they can be used separately and trained/retrained independently. This configuration also simplifies the identification of various Z-vector elements that affect aspects of the output image. These elements may be identified via Principal Component Analysis (PCA) as described above. Elements that control useful properties of the composite image may be described as "outcome-influencing". For example, one Z-vector element (or a set of co-varying elements) may be effective in controlling the skin color of the model. Another outcome-influencing variable may change the pose of the model (e.g., whether it looks left or right, or changes the position of its arms or legs, etc.). Some outcome-influencing variables may control the properties of an item of apparel, such as the softness or stiffness of the pleats of a fabric, the length of a sleeve or collar, or the heel height of a shoe.

In this regard, it should be appreciated that two particularly useful outcome-influencing variables control the (fictional, synthetic) model's body size or proportions and the tightness of the clothing. Together, these variables allow the user to visualize the fit of different clothing sizes.

Note that an input "recognizer" network 729 is proposed that receives multiple images depicting different views of shoes 719a and 719b and generates a Z-vector 759 so that a photo of the shoe can be synthesized onto the model's foot 749. The foot is indicated by three black dots for illustrative purposes. These dots are actually recorded in the image as invisible metadata. They carry additional information about the pose of the synthesized model, so that (for example) the synthesis network can avoid generating synthetic images where the model's body parts are in an impossible configuration (e.g. the left foot points forward but the right foot points backward). In addition to the "pose" information, other information that helps to recognize or synthesize images may also be provided to the neural network to improve its performance.

It should be recognized that adding new categories of inputs increases the complexity of the training problem and increases the number of training images required for the neural network to be able to recognize and synthesize the desired images. For example, for a dress, a pair of shoes, and a handbag, it may be preferable to present a "model" image with only the dress (the model is barefoot and without the bag), the model with the shoes but without the bag, the model with the bag but without the shoes, and the model with the shoes and the bag. If the system is to learn and synthesize multiple model poses, more model images may be needed, with the training image requirements growing exponentially with the number of apparel articles and model body shapes, sizes, and poses that the system is desired to handle.

In the preferred training process outlined in FIG. 8, training images are synthesized from 3D models of various input-category items. Although these 3D models are expensive and time-consuming to create, they can be automatically assembled into different groupings, and photorealistic renderings can be automatically created with parametrically generated model features and poses. Thus, an arbitrarily large number of training images can be created and presented to the neural network. Moreover, these training images can be created without lighting artifacts, minor pose changes, and other defects that are present in actual photographs of live models. This fine-grained control over training images and the ability to automatically generate any number of images allows the neural network to be trained efficiently and reduces the risk of inadvertently training the network to recognize or respond to irrelevant information.

According to this process, a user builds 3D models (800) from several items of apparel. These may be, for example, a shirt, pants, a jacket, a dress, a scarf, a hat, or shoes. The models may contain information that supports rendering a photorealistic image, such as material properties, weave, color, pattern, etc. The models can be automatically manipulated, for example, by simulating their physical characteristics and forces, such as the effects of gravity and inertia on their materials. The size and dimensions of the models can also be automatically manipulated. For example, the hem length of a skirt or the circumference size of a sleeve can be increased or decreased.

Next, images of the model are rendered (810). These images are preferably in an orientation and condition that is inexpensive to mimic when producing photographs of the actual garment. For example, a photograph of a dress laid flat against a background is less expensive to produce than a photograph of the same dress placed on a mannequin. Therefore, for similarly rendered images, it is preferable to show the dress against a flat surface. Even when the three-dimensional model is placed against a flat surface, information about the garment (material, color, etc.) can be used to render realistic layers and wrinkles (i.e., the rendering does not need to be perfectly flat, but rather should be indicative of what the real garment would look like if laid flat on a flat surface, without being pressed or restrained).

Finally, a 3D poseable model of a human figure is created (820) in preparation for the generation of many training images. This figure may include information such as height, weight, lengths and girth of body parts, hair and skin color, etc. (indeed, models of many different human figures may be created). These models are also automatically manipulable, in that their parts can be positioned or posed as they would be for a real person.

The human figure model is then posed (830) and dressed (840) in some or all of the input items of apparel for as many different training images as necessary. By "dressed," we mean combining the human figure model with the apparel model using an interference recognition software process that places the apparel at an appropriate distance from the surface of the human figure model, simulating effects such as gravity so that the rendered image of the clothed model appears realistic.

Here, photorealistic images of the posed and clothed models are created by rendering (850), and the input images from 810 and the posed and clothed "target" images from 850 are used to train (860) the neural network. Additional training images can be created by reposing and reclothing the human figure models and rendering other target images. Beneficially, since the posing can be done automatically and parametrically by the software, target images can be generated that have little or no difference between the intended poses. In other words, for example, in the case of two human figure models of different apparent weights or plumpness, both models can be posed in exactly the same position, so that any differences between the rendered images, such as with respect to the weight of the models, can be learned by the neural network, rather than being an artifact of inadvertent differences between the poses of two real models attempting to assume the same position in a real photo shoot.

Once the neural network is trained (which may require thousands or tens of thousands of automatically generated training images), photographic images of several real items of apparel are obtained (870). As mentioned above, these images should be created in conditions similar to those in which the training apparel items were prepared in 810. These photographic images are provided to the trained neural network, which delivers corresponding synthetic images that appear to show a model wearing the apparel in the photographic images (880). In this synthetic image, some or all of the apparel may be real (these images have been provided to the neural network), but the model is not. The neural network, due to its training, is able to layer clothes appropriately (e.g., by drawing a jacket over a shirt, or shoes over socks). Finally, the synthetic image may be displayed (890).

It should be noted that while training a network on photorealistic renderings of three-dimensional models of clothing, shoes, accessories, etc. is very time-consuming and resource-intensive, the resulting trained network can generate the desired composite images from inputs that are significantly easier and less expensive to obtain. Once trained, the network can operate on essentially "flat" images of clothing (i.e., images of clothing laid out flat against a planar background), front and side images of shoes, and one or a small number of images of each accessory type. The training process can include learning options for "no accessories,""noshoes," so that the compositor can generate those images, as well as images with any desired combination of shoes, bags, jewelry, and other features. A compositing network according to an embodiment of the present invention includes an input sub-network and can receive images and associated information, including:
● Flat garment image ● Mannequin garment image ● Shoe image (preferably 2 or 3 views)
Handbag images (preferably two or three images)
●Hat image ●Necklace image ●Bracelet image ●Ring image ●Scarf image ●Tie image

In addition to training images, an embodiment of the present invention can learn from text and other data sources. This allows the synthesis or image generator network to produce better results when the corresponding text and/or other data is available in later operations. For example, text "tags" associated with images are sometimes available. In a "clothing" application, the tags may describe such things as color, pattern, size, dimensions, collar style, fabric composition or properties, etc. These tags may help the recognizer determine whether the images plausibly depict all of the input items. If the tags describe the clothing color as "blue" or "patterned," an output image showing red or plain clothing is unlikely to be considered plausible. Improved recognition/discrimination performance forces the generator portion of the network to generate better synthetic images and "trick" the recognizer. Since the desired output of the system is a synthetic image that "looks" like a real image, providing the network with additional information (such as text tags) will allow the synthesizer to produce better results.

It should be appreciated that in a multiple-input neural network as described herein, one of the inputs may be an image of a person (nude or wearing a neutral bodysuit) standing in a neutral pose. If this image is provided to the input recognizer and the network is trained to generate a matching output image, the entire system can synthesize an image that appears to be a specific individual wearing a selection of apparel. Thus, instead of just showing a generic model (with controllable height, weight, skin tone, pose, and other characteristics), the synthetic image may show a specific individual who is provided with a photo of a neutral pose and wearing the same apparel. In this embodiment, it is still possible to adjust the pose of the (real) model or her apparent height or weight, using the same Z-vector control that affects any other synthetic image generated by an embodiment. Thus, the output is not simply a morphing or interpolation of clothing that occludes the model image. Instead, all of the input images (including images of real people) are "compressed" down to a Z-vector, and the generative network synthesizes a new image based on the (possibly modified) combined Z-vector. An obvious distinguishing property of this kind of synthetic image is that the model is in a different pose than the input model image. For example, the input model image may be standing upright and facing forward, while the synthetic image may be facing left or right, or have its arms or legs in a different position. However, the Z vector can be adjusted to generate an image showing how a person might look if they gained or lost weight. This use model can be particularly useful.

The inventors have identified one additional type of input data that can be provided to the network during training to improve the quality of the synthetic images. This is "pose" data. When a three-dimensional model of the human figure is used, it is relatively common to specify its pose via reference points for skeletal joints. For example, the positions of the head, shoulders, elbows, wrists, hips, knees and ankles can only vary over a limited range dictated by the nature of the associated joints. By specifying the angles, orientations and joint-to-joint distances, one can effectively define the posture of the person to train. If this information is provided during training along with clothing, tag and other information, the neural network can learn to generate synthetic images that depict the fictional model in various poses. It is even possible to generate a series of successive pose images, which may be displayed in succession to generate an animation of the model showing clothing and accessories, as in a fashion show. Body part girth data may be used to help the neural network learn to depict the model at different weights, as may posture or joint position data.

Some or all of the above-described concepts and procedures can be incorporated in a variety of ways to form a useful system. For example, in one aspect, the system may initialize a multi-layer neural network having a reduced topology input section that generates Z-vectors and an augmented topology output section that generates a composite image based on the Z-vectors, and train the multi-layer neural network with training data sets, each training data set comprising: (i) a first training image showing a first item of apparel; (ii) a second training image showing two views of a second, different item of apparel; and (iii) a training target image showing a model wearing the first item of apparel and the second, different item of apparel, the training step being for generating a trained image synthesis network, and then using the system to generate an instance data set. A set of instance images can be provided to a trained image synthesis network, the instance dataset comprising: (I) a first instance image showing a first instance item of apparel; and (II) a second instance image showing two views of a second, different instance item of apparel. The system can be used to obtain an instance Z vector from the reduced topology input section based on the instance dataset, and to export a composite instance image from the expanded topology output section based on the instance Z vector, the composite instance image appearing to show a model wearing the first instance item of apparel and the second instance item of apparel.

In a system as described above, further refinements may include identifying factors that affect the Z-vector outcome during training and adjusting factors that affect the instance Z-vector outcome to alter characteristics of the synthetic instance image. Other refinements may include using training model images showing a human figure in a neutral pose, the training target images (iii) showing a human figure wearing a first item of apparel and a second, different item of apparel, and the instance dataset further includes (III) a model instance image showing the person in a neutral pose, and the synthetic image appears to show a person wearing the first instance item of apparel and the second instance item of apparel, and the pose of the person in the synthetic image is different from the neutral pose of the model instance image.

Another improvement to the system is to include in the training data at least one non-image data related to the first training image. The non-image data may be text tags, clothing materials, clothing sizes and clothing dimensions, among others. A useful way to apply the system is to use images in which the apparel is flat, but unconstrainedly depicted relative to a flat surface. Items of apparel may include, for example, shirts, blouses, dresses, skirts or pants. Items of apparel that would benefit from images captured from multiple viewpoints may include, for example, shoes, handbags, hats, bracelets and other jewelry.

In another aspect, a system according to an embodiment may create a training image set by constructing an automatically steerable model of a first item of apparel, rendering an image of the first item of apparel to generate a first training image, constructing an automatically steerable model of a second, different item of apparel, rendering two images of the second, different item of apparel to generate a second pair of training images, constructing an automatically steerable model of a human figure, automatically steering the model of the human figure to generate a posed figure model, automatically dressing the posed figure model with the model of the first item of apparel and the model of the second item of apparel to generate a dressed figure model, rendering an image of the dressed figure model to generate a third training image, and applying the first, second, and third training images to a neural network to train the neural network and synthesize an image similar to the third training image from an instance image similar to the pair of the first training image and the second training image.

A neural network trained with the image set generated as described may be used to obtain a first instance image of a first instance item of apparel by photographing the first instance item of apparel, obtain a second instance pair of images of a second instance item of apparel by photographing the second instance item of apparel from two different viewpoints, deliver the first and second instance image pairs to the trained neural network, and receive a target image from the trained neural network, the target image resembling a non-existent human figure wearing the first instance item of apparel and the second instance item of apparel.

The neural network training images may include rendered neutral pose images of a model of a human figure along with a model of a neutral pose, and the trained network may receive an image of a particular person in a neutral pose so that the composite image appears to show that particular person wearing the item of apparel.

The neural network trained as described above may be incorporated into a commercial workflow by acquiring a first instance image of a first instance item of apparel by photographing the first instance item of apparel, acquiring a second instance pair of images of the second instance item of apparel by photographing the second instance item of apparel from two different viewpoints, acquiring a third instance image of a person by photographing the person in a neutral pose, delivering the pair of the first and second instance images and the third instance image to the trained neural network, and receiving a target image from the trained neural network. The target image resembles a person wearing the first instance item of apparel and the second instance item of apparel, and a pose of the person in the target image differs from a neutral pose or an apparent weight of the person in the target image differs from an apparent weight of the person in the third instance image.

Embodiments of the invention may be machine-readable media, including but not limited to non-transitory machine-readable media, for storing data and instructions and for causing a programmable processor to perform the operations as described above. In other embodiments, the operations may be performed by specific hardware components that include hardwired logic. Alternatively, the operations may be performed by any combination of programmed computer components and custom hardware components.

Instructions for a programmable processor may be stored in a form that is directly executable by the processor ("object" or "executable" form), or the instructions may be stored in a human-readable text format called "source code" that can be automatically processed by development tools commonly known as "compilers" to generate executable code. Instructions may also be specified as differences or "deltas" from a given version of the underlying source code. Using deltas (also called "patches"), instructions can be prepared to implement one embodiment of the present invention, starting from a publicly available source code package that does not include one embodiment.

In some embodiments, the instructions for the programmable processor may be treated as data and used to modulate a carrier signal that can then be transmitted to a remote receiver, the signal demodulated to recover the instructions, and the instructions executed at the remote receiver to implement the method of an embodiment. In technical terms, this type of modulation and transmission is known as "providing" the instructions, while the receiving and demodulation is often referred to as "downloading." In other words, an embodiment "provides" (i.e., encodes and transmits) the instructions of an embodiment to a client, often over a distributed data network such as the Internet. Thus, the transmitted instructions may be stored on a hard disk or other data storage device at the receiver, creating other embodiments of the invention and fulfilling the description of a non-transitory machine-readable medium storing data and instructions to perform some of the operations described above. As a result of compiling (if necessary) and executing this type of embodiment at the receiver, the receiver may perform operations according to the third embodiment.

In the previous description, numerous details have been set forth. However, it will be apparent to those skilled in the art that the present invention may be practiced without some of these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.

Some portions of the detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here and generally understood to be a self-consistent sequence of steps leading to a desired result. The steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared and otherwise manipulated. It has proven convenient at times, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

However, it should be borne in mind that all of these similar words and phrases should be associated with the appropriate physical quantities and are primarily convenient labels to apply to these quantities. Unless otherwise specifically stated as is clear from the preceding discussion, throughout the description, discussion utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" will relate to the actions and processes of a computer system or similar electronic computing device, transmission or display apparatus, and that the computer system manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data represented in the same manner as physical quantities in the computer system's memory or registers or other such information storage devices.

The present invention also relates to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer-readable storage medium, including, without limitation, any type of disk, i.e., floppy disk, optical disk, compact disk with read-only memory ("CD-ROM") and magneto-optical disk, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory ("EPROM"), electrically erasable read-only memory ("EEPROM"), magnetic or optical card, or any type of medium suitable for storing computer instructions.

The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform some of the method steps. The required structure for a variety of these systems is set forth in the following claims. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein.

Applications of the present invention have been described primarily by reference to specific examples and in terms of specific allocations of functionality to particular hardware and/or software components. However, those skilled in the art will recognize that configurable composite images of a model wearing clothing based on images of the clothing alone may be generated by software and hardware that distribute the functionality of embodiments of the present invention differently than described herein. Such variations and implementations are to be understood as protected in accordance with the following claims.

Claims (12)

1. A method comprising:
initializing a multi-layer neural network having a reduction topology input section for generating a Z-vector and an augmentation topology output section for generating a composite image based on the Z-vector;
training the multi-layer neural network with training data sets, each training data set comprising:
i) a first training image showing a first item of apparel;
ii) two second training images showing two views of a second different item of apparel; and
iii) training target images showing a model wearing the first item of apparel and the second, different item of apparel;
The training step is for generating a trained image synthesis network;
The method includes providing an instance dataset to the trained image synthesis network, the instance dataset comprising:
I) a first instance image showing a first instance item of apparel;
II) two second instance images showing two views of a second different instance item of apparel;
The method comprises:
obtaining an instance Z vector from the reduced topology input section based on the instance data set;
and exporting a composite instance image from the augmented topology output section based on the instance Z vector.
the composite instance image appears to show a model wearing the first instance item of apparel and the second instance item of apparel.
Method. Identifying factors that influence the outcome of the Z vector during training;
adjusting factors that influence the outcome of the instance Z vector to change characteristics of the composite instance image;
Further comprising:
The method of claim 1. Each training dataset is
iv) further comprising a training model image showing a human figure in a neutral pose;
the training target images (iii) depict the human figure wearing the first item of apparel and the second, different item of apparel;
The instance data set includes:
III) further comprising a model instance image showing a person in a neutral pose;
the synthetic image appears to show the person wearing the first instance item of apparel and the second instance item of apparel, and a pose of the person in the synthetic image differs from the neutral pose of the model instance image.
The method of claim 1. Each training dataset is
v) further comprising at least one non-image data related to the first training image;
The method of claim 1. the at least one non-image data is selected from the group consisting of text tags, clothing materials, clothing sizes, and clothing dimensions;
The method according to claim 4. said first item of apparel being selected from the group consisting of a shirt, a blouse, a dress, a skirt and pants;
The method of claim 1. the first item of apparel is flat but depicted unconstrained relative to a flat surface;
The method according to claim 6. the second distinct item of apparel is selected from the group consisting of shoes, handbags, hats and bracelets;
The method of claim 1. 1. A method for training a neural network to synthesize an image of a model wearing multiple items of apparel, comprising:
building an automatically operable model of a first item of apparel;
rendering an image of the first item of apparel to generate a first training image;
building an automatically operable model of a second, different item of apparel;
rendering two images of the second different item of apparel to generate a second training image pair;
constructing an automatically operable model of the human figure;
automatically manipulating the model of the human figure to generate a posed figure model;
automatically dressing the posed figure model with the model of the first item of apparel and the model of the second item of apparel to generate a dressed figure model;
rendering images of the clothed model to generate third training images;
applying the first, second and third training images to a neural network, training the neural network to synthesize an image similar to the third training image from instance images similar to the pair of the first training image and the second training image;
The method includes: The applying step produces a trained neural network, the method comprising:
acquiring a first instance image of a first instance item of apparel by photographing the first instance item of apparel;
obtaining a second instance pair of images of a second instance item of apparel by photographing the second instance item of apparel from two different viewpoints;
delivering the pair of the first instance image and the second instance image to the trained neural network;
receiving a target image from the trained neural network, the target image resembling a non-existent human figure wearing the first instance item of apparel and the second instance item of apparel.
The method of claim 9. The applying step produces a trained neural network, the method comprising:
rendering a neutral pose image of the model of the human figure with the model in a neutral pose;
applying the neutral pose image along with the first, second and third training images to the neural network.
The method of claim 9. The applying step produces a trained neural network, the method comprising:
acquiring a first instance image of a first instance item of apparel by photographing the first instance item of apparel;
obtaining a second instance pair of images of a second instance item of apparel by photographing the second instance item of apparel from two different viewpoints;
acquiring a third instance image of the person by photographing the person in a neutral pose;
delivering the pair of the first and second instance images and the third instance image to the trained neural network;
receiving a target image from the trained neural network, the target image resembling the person wearing the first instance item of apparel and the second instance item of apparel;
the pose of the person in the target image is different from the neutral pose, or
an apparent weight of the person in the target image is different from an apparent weight of the person in the third instance image;
The method of claim 11.

Images