WIKT Challenge

11/2017 Viktor Bachraty

Problem description

• Train a binary classifier to distinguish original MNIST images (human drawn numbers) from artificial (generated) ones.
• What we have:
• Embedding vectors of length 32 for each picture
• Decoder to convert the vectors back into images
• Labelled training a validation sets
• Test set we need to label
• Baseline solutions with given metrics

​

Simple multilayer dense network

• The baseline solutions didn’t include a neural network.
• Datasets are sufficiently small to be trained on CPU. Using sklearn it’s possible to test this model within a few minutes.
• Having done this gives us a much better understanding of the complexity/difficulty of this task.
• F1 score: ~95,3 (+1.4) 😊

​

Simple multilayer dense network

• In order to be able to tweak model with something more sophisticated, we need a framework like TF, Keras, Cognitive toolkit etc..
• These frameworks also allow training on GPU that significantly speeds up the rate we can iterate on models.
• Adding regularization in form of dropout and fine tuning hyperparameters (number of hidden layers, neurons, learning rate) improves performance to ~95.7 (+0.4) 😊
• We would like to have something more sophisticated, what can we come up with?

​

Hypotheses

• MNIST is a well researched dataset. There is list of top ranking models¹ on this dataset.
• The baseline model with 3 convolutional layers yields 99.2% on digit classification.
• Almost state-of-the-art is the new approach from Hinton group - CapsNet with ~99.75%
• We can easily modify them to binary classification instead of their original purpose to classify digits and train them on the image representation instead of the embedding.
• Maybe we can use their inability to classify the digit as a signal that the digit was generated?

​

​

¹ rodrigob.github.io

CapsNet

(Image credit: Sabour et. al)

Capsule

• A set of neurons or a multidimensional neuron - transforms a vector of vectors (matrix) into a vector.
• A layer of capsules transforms a matrix to a matrix.
• Weights of a capsule layer compose a 4D tensor.
• Activation is a vector-valued function applied to all components in the capsule (much like softmax in case of a layer of neurons)

Verification of hypotheses

• Attempts to use the image representation failed:
• Convolutional baseline ~90 (-3.9) ☹
• CapsNet ~91 (-2.9) ☹
• CapsNet can be forced to create a representation similar to the input vector in the digit caps layer (by adding an additional loss). Assuming input vectors are somewhat aligned, it could make sense to train the network to do so for one class and have a perpendicular vector representation in the other. This can be achieved by adding a cosine distance loss.
• Unfortunately, this lead to a minor improvement only.

Image representation

• How does the dataset look like?

Converter tool

• We have many good looking numbers in the generated class, and bad looking ones in the ‘handwritten’ class.
• Why is that? This is the provided decoder:

​

​

​

​

​

​

​

Converter tool

• The images were generated using an autoencoder. The encoding part, that generated the embeddings makes a lossy compression. The decoding part, that generates images, should not throw away information.
• But what happens if you give the decoder something completely different, something it wasn’t trained on? I tries to find one of the learned patterns, thus reducing the differences between inputs from the training set and other, previously unseen inputs.
• Possibly some subsets of the dataset were generated using different models. The provided decoder is optimized only for one of those subsets.

​

​

​

​

​

​

​

Autoencoder

• Since the provided decoder doesn’t provide a representation we can use, can we train the network to make a better one?
• We can can do so, using the following model
• F1score ~96.2 (+0.5) 😊

​

Deconvolutional autoencoder

• What if we create an ‘inverse’ autoencoder - one that has the embeddings on input/output and the internal representation is an image?
• We can use conv2d_transposed to do so.
• The model does not overfit so easily so it can be trained longer yielding better results.
• F1 score ~96.7 (+0.5) 😊

​

Regularization

• A commonly used technique to improve image classification robustness is augmenting the dataset by generating randomly transformed images e.g. applying random shift by a small distance.
• How to find good values to add noise to our embeddings?
• We can find distributions for each component of the vector.
• If the data was created using a variational autoencoder, finding the normal distribution of each latent variable is essential.
• Basically, we are training a denoising autoencoder.

Regularization

• Visualising the data helps - tSNE shows, the data doesn’t cover the space evenly, but is clustered (unfortunately not by class)

• We can estimate the mean and variance of the data by first clustering it using k-means.
• Then we find means/variances.
• The best results we got using 120 clusters.
• F1 score 97.5 (+0.8) 😊

Post-competition improvement

• How can we further improve the model?
• Simple voting ensemble - 3 models can yield f1 score ~97.67

​

• Replacing voting with a gating network to learn the weights for each prediction bumps f1 score over ~97.7 (on validation set, need to verify test set)

Mixture of Experts

• Each tuple (noise gen, dedcoder, classifier, regularizer) is a single expert
• Gating network learns to combine them
• Regularization error is also gated

​

Possible future improvements

​

​

• GMM - we have falsely assumed, that components of the latent vector are independent random variables. A multivariate mixture of gaussians could give us a better estimation of variance within clusters than kmeans. Alternatively, maybe we could force the model to learn optimal noise params.
• Using more regularizers - e.g. batch normalization or an activation regularizer - this could speed up learning and result in more fine grained distributed internal representation.
• Boosting - well known technique, in some ways similar to MoE, but with focus on misclassified input examples.
• Massive parallel hyperparameter search - we could get some gains by using a semi-automated search in the state space of models.

​

Lessons learned

​

​

• There are seemingly more options that don’t work compared to those that work just because you stop searching once you find one that does.
• More data gives better results, even if it is synthetically generated.
• Sequential tweaking and testing is slow and error prone. Automated grid search for hyperparameter tuning would be tremendously useful.
• If your model doesn’t use all available computational resources then probably there is a more accurate model that does.

​

​