Configurations

Hypergan model author JSON reference

ConfigurableComponent

Hypergan comes with a simple DSL to design your own network architectures. A ConfigurableComponent can be used to create custom designs when building your models.

ConfigurableDiscriminator

For example, a discriminator component can be defined as:

...
  "discriminator": 
  {
      "class": "class:hypergan.discriminators.configurable_discriminator.ConfigurableDiscriminator",
      "layers":[
        "conv 32",
        "relu",
        "conv 64 ", 
        "relu",
        "conv 128",
        "relu",
        "conv 256",
        "relu",
        "flatten",
        "linear 1"
      ]

  }

This means to create a network composed of 4 convolution layers that decrease along stride and increase filter size, ending in a linear without activation. End the discriminator in a logit(output of conv/linear/etc)

ConfigurableGenerator

  "generator": {
    "class": "class:hypergan.discriminators.configurable_discriminator.ConfigurableDiscriminator",
    "layers": [
      "linear 128 name=w",
      "relu",
      "resize_conv 256",
      "const 1*1*128",
      "adaptive_instance_norm",
      "relu",
      "resize_conv 256",
      "adaptive_instance_norm",
      "relu",
      "resize_conv 256",
      "adaptive_instance_norm",
      "relu",
      "resize_conv 256",
      "adaptive_instance_norm",
      "relu",
      "resize_conv 256",
      "adaptive_instance_norm",
      "relu",
      "resize_conv 128",
      "adaptive_instance_norm",
      "relu",
      "resize_conv 64",
      "adaptive_instance_norm",
      "relu",
      "resize_conv 32",
      "adaptive_instance_norm",
      "relu",
      "resize_conv 3",
      "tanh"
    ]

  }

This is a generator. A generator takes in a latent space and returns a data type that matches the input.

adaptive_instance_norm looks up the style layer (default named 'w') and uses it to perform the adaptive instance norm.

HyperGAN defaults to the image space of (-1, 1), which is the same range as tanh.

Layers

Common layer types:

linear

linear [outputs] (options)

Creates a linear layer.

conv

conv [filters] (options)

A convolution. Stride is applied if it is set. For example: conv [filters] filter=5 stride=1 will run set stride to 1 and with a filter size of 5.

deconv

deconv [filters] (options)

Doubles the width and height of your tensor. Called conv2d_transpose in literature

resize_conv

resize_conv [output filters] (options)

reshape

reshape [size]

size can be:

• -1

• * delimited dimensions.

flatten

Same as reshape -1

const

const [size]

size is * delimited dimensions

Configuration

Configuration in HyperGAN uses JSON files. You can create a new config with the default template by running hypergan new mymodel.

You can see all templates with hypergan new mymodel -l.

Architecture

A hypergan configuration contains all hyperparameters for reproducing the full GAN.

In the original DCGAN you will have one of the following components:

• Distribution(latent space)

• Generator

• Discriminator

• Loss

• Trainer

Other architectures may differ. See the configuration templates.

GANComponent

A base class for each of the component types listed below.

Generator

A generator is responsible for projecting an encoding (sometimes called z space) to an output (normally an image). A single GAN object from HyperGAN has one generator.

Discriminators

A discriminator's main purpose(sometimes called a critic) is to separate out G from X, and to give the Generator a useful error signal to learn from.

DSL

Each component in the GAN can be specified with a flexible DSL inside the JSON file.

Losses

WGAN

Wasserstein Loss is simply:

 d_loss = d_real - d_fake
 g_loss = d_fake

d_loss and g_loss can be reversed as well - just add a '-' sign.

Least-Squares GAN

 d_loss = (d_real-b)**2 - (d_fake-a)**2
 g_loss = (d_fake-c)**2

a, b, and c are all hyperparameters.

Standard GAN and Improved GAN

Includes support for Improved GAN. See hypergan/losses/standard_gan_loss.py for details.

Boundary Equilibrium Loss

Use with the AutoencoderDiscriminator.

See the began configuration template.

Loss configuration

attribute	description	type
batch_norm	batch_norm_1, layer_norm_1, or None	f(batch_size, name)(net):net
create	Called during graph creation	f(config, gan, net):net
discriminator	Set to restrict this loss to a single discriminator(defaults to all)	int >= 0 or None
label_smooth	improved gan - Label smoothing.	float > 0
labels	lsgan - A triplet of values containing (a,b,c) terms.	[a,b,c] floats
reduce	Reduces the output before applying loss	f(net):net
reverse	Reverses the loss terms, if applicable	boolean

Trainers

Determined by the GAN implementation. These variables are the same across all trainers.

Consensus

Consensus trainers trains G and D at the same time. Resize Conv is known to not work with this technique(PR welcome).

Configuration

attribute	description	type
learn_rate	Learning rate for the generator	float >= 0
beta1	(adam)	float >= 0
beta2	(adam)	float >= 0
epsilon	(adam)	float >= 0
decay	(rmsprop)	float >= 0
momentum	(rmsprop)	float >= 0

Alternating

Original GAN training. Locks generator weights while training the discriminator, and vice-versa.

Configuration

attribute	description	type
g_learn_rate	Learning rate for the generator	float >= 0
g_beta1	(adam)	float >= 0
g_beta2	(adam)	float >= 0
g_epsilon	(adam)	float >= 0
g_decay	(rmsprop)	float >= 0
g_momentum	(rmsprop)	float >= 0
d_learn_rate	Learning rate for the discriminator	float >= 0
d_beta1	(adam)	float >= 0
d_beta2	(adam)	float >= 0
d_epsilon	(adam)	float >= 0
d_decay	(rmsprop)	float >= 0
d_momentum	(rmsprop)	float >= 0
clipped_gradients	If set, gradients will be clipped to this value.	float > 0 or None
d_clipped_weights	If set, the discriminator will be clipped by value.	float > 0 or None

Fitness

Only trains on good z candidates.

Curriculum

Train on a schedule.

Gang

An evolution based trainer that plays a subgame between multiple generators/discriminators.