TensorFlow - Part 2
It’s finally a plugin
Last time we talked about Plugins
-Blueprint spaghetti and features can be encapsulated and shared
Last time we talked about Plugins
-Blueprint spaghetti and features can be encapsulated and shared
-Can add external libraries and features e.g. novel input devices
Last time we talked about Plugins
-Blueprint spaghetti and features can be encapsulated and shared
-Can add external libraries and features e.g. novel input devices
-Easy to start - use the plugin wizard
Last time we connected TensorFlow to UE4
Why Tensorflow?
By far the biggest library
What can you use it for?
Sound generation, Wavenets - https://deepmind.com/blog/wavenet-generative-model-raw-audio/
Speech recognition - https://github.com/tensorflow/models/tree/master/syntaxnet
What can you use it for?
Sound generation, Wavenets - https://deepmind.com/blog/wavenet-generative-model-raw-audio/
Speech recognition - https://github.com/tensorflow/models/tree/master/syntaxnet
Image detection
What can you use it for?
Sound generation, Wavenets - https://deepmind.com/blog/wavenet-generative-model-raw-audio/
Speech recognition - https://github.com/tensorflow/models/tree/master/syntaxnet
Image detection
Self driving cars
What can you use it for?
Sound generation, Wavenets - https://deepmind.com/blog/wavenet-generative-model-raw-audio/
Speech recognition- https://github.com/tensorflow/models/tree/master/syntaxnet
Image detection
Self driving cars
Cancer detection
Examples of Machine Learning
Sound generation, Wavenets - https://deepmind.com/blog/wavenet-generative-model-raw-audio/
Speech recognition
Image detection
Self driving cars
Cancer detection
Anything with a pattern and a lot of data
Examples of Machine Learning - What about 3d?
Examples of Machine Learning - What about 3d?
Examples of Machine Learning - What about 3d?
Examples of Machine Learning - What about 3d?
So how does it work?
Fork of https://github.com/20tab/UnrealEnginePython for making everything work
And https://github.com/getnamo/socketio-client-ue4 for BP json struct support
Python 3.5
TensorFlow Python
Tensorflow C++ Library
CudaNN
UnrealEngine Python Plugin
SocketIO Client Plugin (JSON)
TensorFlow
UE4 Plugin
User Project
Blueprints
BP <-> Python complex data via JSON
So what has changed?
-Scripts and code fully plugin encapsulated.
-Multi-threading. No longer blocks your game while doing tensorflow things
-Pip support. You can install and uninstall python from the commandline
-Auto import changes. You can rerun your code many times and see small changes you make each time
So what has changed?
-Upyjson dependency list. Plugin can define its dependencies and will be fetched on run
-GPU support.
-Tensorflow 1.3 and Keras
-Support for audio recording -> float array. Now you can input both images and sound
So what has changed?
-API updated to allow for easy multithreaded calls
-API missing? allow custom function calls
-Want async updates? callEvent() which forwards a UE4 event dispatch on your tensorflow component
Ok great, how do I get it?
Ok great, how do I get it?
Ok great, how do I get it?
Easy! Well one gotcha...
If you see this when importing tensorflow
Re-launch your project. You only need to do this once.
How do I use it?
Thanks to the python plugin you can use it directly in the editor by typing in the python console
import tensorflow as tf
Write some operations, then sess.run(myOperation)
How do I use it?
How do I encapsulate my own script in e.g. an actor?
How do I use it?
How do I encapsulate my own script in e.g. an actor?
2. Class defaults -> change your module name to your script e.g. ExampleAPI (leave out .py)
How do I use it? Your own script
Implement TFPluginAPI in your script.
onSetup(self):
Called when you begin play.
How do I use it? Your own script
Implement TFPluginAPI in your script.
onSetup(self):
Called when you begin play.
onBeginTraining(self):
Called when you call start training on your component, or on begin play if set to auto-train.
How do I use it? Your own script
Implement TFPluginAPI in your script.
onSetup(self):
Called when you begin play.
onBeginTraining(self):
Called when you call start training on your component, or on begin play if set to auto-train.
onJsonInput(self, jsonInput):
Called when you pass data in
How do I use it? Your own script
Implement TFPluginAPI in your script.
These two have to match.
It’s normal to leave this unchanged unless you rename your class for other uses.
How do I use it? Your own script
Full TFPluginAPI adding example.
How do I use it? Your own script
Full TFPluginAPI adding example.
We store our tf.Session, a, and b placeholders as well as our operation a + b in c.
How do I use it? Your own script
Full TFPluginAPI adding example.
We store our tf.Session, a, and b placeholders as well as our operation a + b in c.
When we receive an input in json format we make a feed_dict that links the values to the inputs e.g. self.a : jsoninput[‘a’].
Run the session to get the results and pass them back as our answer in json format
How do I use it? Your own script
Full TFPluginAPI adding example.
We store our tf.Session, a, and b placeholders as well as our operation a + b in c.
When we receive an input in json format we make a feed_dict that links the values to the inputs e.g. self.a : jsoninput[‘a’].
Run the session to get the results and pass them back as our answer in json format.
We have a custom function call to change the operation from + to -
How do I use it? Your own script, blueprint side
In blueprint we subscribe to the input result event dispatch.
We decode our json string and fetch the answer, in this instance as a string.
How do I use it? Your own script, blueprint side
We send input via a blueprint defined struct holding a and b.
Which we encode automagically to json
How do I use it? Your own script, blueprint side
Note the struct can hold arrays for a and b and no code change would be necessary!
How do I use it? Your own script, blueprint side
To swap the input method we call our custom function and pass in - or + as string data
That’s it!
Everything else is just making the Tensorflow side more complicated and managing training or stopping.
That’s it!
Everything else is just making the Tensorflow side more complicated and managing training or stopping.
Well you can also send async events with data back to UE4. E.g. previewing training images
That’s it!
Everything else is just making the Tensorflow side more complicated and managing training or stopping.
Well you can also send async events with data back to UE4. E.g. previewing training images
Since Tensorflow 1.1 you can also use Keras high level library which makes things simple and powerful.
Time to show it live?
Thankfully no memes this time.
Time to show it live?
Thankfully no memes this time. I lied.
What’s next?
-Blueprint TF api?
-Optimizations? (buffer*)
-Tensorboard inside UE4?
-Speech recognition?
-Handwriting recognition?
-Self driving cars in UE4?
Production Ready! Ship it! :p
Speech Recognition - Not quite production ready
-State of the art speech recognition takes ~20days to train on a Titan X
-Pretrained examples available have WER of 50%, CER of 20%.
-Waiting on pretrained examples from Mozilla:
Questions?
Which machine learning features would you like to see in UE4?
Full presentation at: https://skillsmatter.com/skillscasts/10929-tensorflow-plugin
Ask belated questions @getnamo twitter, github, unreal forums
Links
Two Minute papers - https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg
Learning AI if you suck at math - https://hackernoon.com/learning-ai-if-you-suck-at-math-8bdfb4b79037
Awesome Tensorflow - https://github.com/jtoy/awesome-tensorflow
Beginner mnist - https://www.tensorflow.org/tutorials/mnist/beginners/
Udacity - https://www.udacity.com/course/deep-learning--ud730
Deep Mind blog - https://deepmind.com/blog/