wevi: Word Embedding Visual Inspector

FAQ

What is this tool?

What does it do?

How do I get started?

What are some interesting results?

Is Firefox supported?

Can I specify my own training data?

What do the colors mean?

Why PCA? Why not t-SNE, MDS, etc.?

Why do neurons turn black during training?

What neural network package did you use?

Doesn't word2vec take in very different inputs from what is in wevi?

Can I help improve the tool?

Any future development plan?

Any documentation on the code?

How do you implement this?

I found a bug, where do I report?

References?

How to contact you?

Other questions?

FAQ

What is this tool?

This tool visualizes the basic working mechanism of word2vec, a popular word embedding model, originally published by Mikolov et al. (https://code.google.com/p/word2vec/). It is the result of a few days' hack during my preparation for the a2-dlearn (Ann Arbor Deep Learning Meetup) event hosted by Daniel Pressel.

Here are the video and slides of my talk. To read more about this work, please read my paper explaining the parameter learning of word embedding. I am planning to keep adding functionalities to the tool, and it is greatly appreciated if you could help in the development process! The repository is at https://github.com/ronxin/wevi.

What does it do?

It learns word vectors by running the most basic word2vec learning algorithm on a very tiny vocabulary and input sets. The whole point of the tool is for any user who wishes to understand neural word embedding techniques to get an intuitive interpretation of what the model is actually doing. You may specify the inputs (i.e., training data), the number of hidden layers, and the learning rate. You can train the model. It runs word2vec in your browser (using JavaScript, of course). You can then visually inspect the results using either the weight matrices, vector scatter plots, as well as directly checking the neurons (by hovering your mouse over them).

How do I get started?

The model by default has randomly initialized weights, so you will need to train it in order to see some interesting patterns emerge. You can simply click on the  button, and it will train the model using 500 training instances. When it stops, you can hover your mouse over some neuron on the input layer, and that neuron will be set as active, and you can inspect the activation levels of the neurons, like this:

You may also click on the  button, and it will take one training instance at a time, so that you can inspect it at your own pace. Clicking on the  button will make it recalculate PCA given the current positions of the vectors. This will usually give you a better layout.

What are some interesting results?

Load the "King and Queen" preset, train 1000 iterations (click on the  button, wait until it stops, and clicks  again). Then click on the  button. Look into the lower-right scatterplot, check the blue markers (input vectors), and you can see the infamous analogy: "king - queen = man - woman."

In addition to this, each of the presets bears some interestingness in it. You are encouraged to explore on your own!

Is Firefox supported?

Yes. But Chrome is the recommended browser. The font looks weird on Firefox.

Can I specify my own training data?

Yes, you may follow the same format in the preset training data textbox. Note that you may also specify multiple words as context or target words. Just separate the words with ^. Input pairs should be separated by |. Vocabulary will be automatically derived based on your given training data. Below are some examples:

What do the colors mean?

Why PCA? Why not t-SNE, MDS, etc.?

I only spent a few days on this, so I picked a linear transformation that I was most familiar with. These more advanced dimensionality reduction methods are on the TODO list, you are welcome to help me by just sending in pull requests!

What neural network package did you use?

I implemented weight initialization, feedforward and backpropagation by myself, because they are actually very simple to implement. I used an existing package for PCA.

Doesn't word2vec take in very different inputs from what is in wevi?

The inputs and outputs of wevi are identical to the neural network portion of word2vec. Of course, the latter has components that actually process input sentences and generate context/target word pairs. Below is an illustration of the entire word2vec working pipeline, and wevi simulates the part in the red circle.

Can I help improve the tool?

Yes! Please do whatever you want! Pull requests on github are more than welcome! The repository is at https://github.com/ronxin/wevi.

Any future development plan?

Yes. Here are a list of things on my mind. Please let me know if you would like to pick up any of these and just shoot me an email (see "How to contact you").

Any documentation on the code?

Sorry... I did not (and perhaps will not) have much time working on that. But the code should be quite self-explanatory. I tried to leave useful comments here and there.

How do you implement this?

Visualization is completely written using JavaScript D3. Feedforward and backpropagation are implemented in JavaScript as well. For PCA, I used an existing JavaScript implementation found here. Web page layout uses Bootstrap.

I found a bug, where do I report?

Please create an "issue" in https://github.com/ronxin/wevi.

References?

How to contact you?

Other questions?

Just send me an email to xrong14@gmail.com and I will be happy to reply.