Generative Adversarial Networks

Generative adversarialnetworks, first described by Ian Goodfellow in 2014 are often considered to be the state-of-the-art models for image generation. They are, however, notoriously difficult to train due to their most unique design feature. Though a GAN is just a simple neural network, training it requires a special auxilliary network, which must be trained alongside with the main network. These are known as the generator and the discriminator, and they play a zero-sum game between themselves to learn how to generate realistic outputs.

The basic principle is quite simple. In an ideal setting, we could train generative networks by having them output images, then having human volunteers evaluate which one is real and which one is fake. In reality, this would be prohibitively expensive, so we train a neural network to distinguish between real and artificial samples, though this presents a serious problem. After every few training iterations, the outputs of the generator become radically different, thus the discriminator has to be retrained to be effective. Furthermore, if the discriminator becomes too good at distinguishing the fake samples from the real ones, the generator cannot learn, since there is no gradient for it to follow - whatever small change it makes, it will never achieve a lower loss.

To solve the above two issues, we train the two networks together, so they can both evolve dynamically and keep up with the other. Still, training GANs is unusually difficult, even when compared to other neural networks, and aside from some best practices, we stil do not have exact formulae for getting good results.

Before moving on to generating images, we will start with a simple 1D
problem. We will assume that there is a data set of real samples that
are drawn from a normal distribution with a particular mean value.
These samples are in the sample domain x.

The generator network does not know anything about the distribution
of the real samples in the sample domain x, but will try to come up
with a transformation that maps random noise from a distribution z
to samples that seem to come from the real sample distribution.

GAN paper: https://arxiv.org/pdf/1406.2661.pdf6.2661.pdf

In the figure above, you can see the process of training a GAN:

 - black dotted line - a sampling of the real data;

 - solid green line - generative distribution (sample generated by the generator);

 - blue dashed line - discriminative distribution (distribution generated by the discriminator: distinguishes between fake and real samples);

Most deep learning tutorials build a discriminative model that is able
to classify an image into categories. In this exercise, we are going to do
the inverse. Given a label and a point in the latent space (which in our case will be
a multi-dimensional uniform distribution), we are going to train the network
to generate a realistic image for this point. We will use the images and labels in the Fashion-MNIST data set to train a conditional GAN. The conditioning will allow us to control what object
we want to synthesize with the generator. To achieve this, we will modify the generator and the discriminator from the previous part to use convolutional layers and accept additional input, which will represent the label of an object.

cGAN paper: https://arxiv.org/pdf/1411.1784.pdf