Your phone is learning!
Introduction to Tensorflow on Android
Bartosz Kraszewski
about.me/bartoszk
About me
What is Machine Learning?
“Machine learning is the science of getting computers to act without being explicitly programmed.” – Stanford
“Machine learning is based on algorithms that can learn from data without relying on rules-based programming.”- McKinsey & Co.
Example
Image classification
(Food Recognition)
It's a cat!
Another example
Face emotion recognition
Something from my phone
Let’s imagine complex system using ML
Collect Data
Label Data
Train Model
Export to cloud
Export to mobile
Google Sketch
Captcha
Use trained model in production
Input test data
Trained Model
Output decision
What’s mobile developer part in this process?
Just give me data
Just use my model
No questions!
Start career in Data Science
Start career in Data Science
We are still part of a process!
Online Courses!
How mobile world meet Machine Learning?
Introduction to Tensorflow
Tensorflow
Google Samples for Tensorflow
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/android
TF Classify
TF Detect
TF Stylize
Want more?
Mnist Data Set
60k training images, 10k test images
Number representation
28x28 2d float array flattened to 784 elements array with grayscale values:
0.0 - white pixel, 1.0- black pixel
also has antialiasing supported
Model Representation
0
1
2
…
…
783
0
1
2
3
4
5
6
7
8
9
Class index, probability (0, 1)
Pixel address, color value (0, 1.0)
Model Representation
0
1
2
…
…
783
0 - 0.0
1 - 0.0
2 - 0.0
3 - 0.0
4 - 0.997
5 - 0.01
6 - 0.001
7 - 0.0
8 - 0.0
9 - 0.0
Class index, probability (0, 1)
Pixel address, color value (0.0, 1.0)
Magic
Dragons
Even more magic
Model source
Step one - Android Studio
implementation 'org.tensorflow:tensorflow-android:+'
Step two - Implement UI
Create custom draw component
public class DrawingView extends View {
private Bitmap bitmap;
private Canvas canvas;
private Path path;
private Paint bitmapPaint;
private Paint paint;
public DrawingView(Context c, AttributeSet set) {
super(c, set);
path = new Path();
bitmapPaint = new Paint(Paint.DITHER_FLAG);
}
https://github.com/bkraszewski/MachineLearningDemo
Step three - load model
executor.execute(() -> {
try {
classifier = Classifier.create(getApplicationContext().getAssets(),
MODEL_FILE,
LABEL_FILE,
INPUT_SIZE,
INPUT_NAME,
OUTPUT_NAME);
} catch (final Exception e) {
throw new RuntimeException("Error initializing TensorFlow!", e);
}
});
Initialization params
INPUT_SIZE = 28;
INPUT_NAME = "input";
OUTPUT_NAME = "output";
MODEL_FILE = "file:///android_asset/expert-graph.pb";
LABEL_FILE = "file:///android_asset/labels.txt";
Step fourth - initialize classifier
Classifier classifier = new Classifier();
classifier.tfHelper = new TensorFlowInferenceInterface(assetManager, modelPath);
Let’s prepare data...
How Android Stores bitmap?
ARGB - 32 bit integer
Alpha: 0 - 255
Red: 0 - 255
Green: 0 - 255
Blue: 0 - 255
In Hex: #FFFF0000
What data classifier requires?
Step fifth - convert bitmap to Tensor
Bitmap original = drawingView.getBitmap();
Bitmap scaled = Bitmap.createScaledBitmap(original, 28, 28, false);
int width = 28;
int[] pixels = new int[width * width];
scaled.getPixels(pixels, 0, width, 0, 0, width, width);
private float[] createInputPixels(int[] pixels) {
float[] normalized = ColorConverter.convertToTfFormat(pixels);
return normalized;
}
Read color values
for (int i = 0; i < argbPixels.length; i++) {
int aargbPixel = argbPixels[i];
int alpha = (aargbPixel >> 24) & 0xff;
int r = (aargbPixel >> 16) & 0xff;
int g = (aargbPixel >> 8) & 0xff;
int b = aargbPixel & 0xff;
}
Ignore transparent pixels
if (alpha == 0) {
ret[i] = 0.0f;
continue;
}
Convert to grayscale
int avg = (r + g + b) / 3;
Convert to float and add alpha
float grayscaled = avg / 255.0f;
withAlpha = grayscaled * (alpha / 255.0f);
Invert value
ret[i] = 1.0f - grayscaled;
Let’s unit test converter!
int[] blackPixels = new int[4];
for (int a = 0; a < blackPixels.length; a++) {
blackPixels[a] = Color.BLACK;
}
float[] tfInput = ColorConverter.convertToTfFormat(blackPixels);
assertEquals(blackPixels.length, tfInput.length);
for (int a = 0; a < tfInput.length; a++) {
assertEquals(1.0f, tfInput[a], ACCEPTED_DELTA);
}
int[] blackPixels = new int[4];
for (int a = 0; a < blackPixels.length; a++) {
blackPixels[a] = Color.BLACK;
}
float[] tfInput = ColorConverter.convertToTfFormat(blackPixels);
assertEquals(blackPixels.length, tfInput.length);
for (int a = 0; a < tfInput.length; a++) {
assertEquals(1.0f, tfInput[a], ACCEPTED_DELTA);
}
Step sixth - communicate with model
final float[] pixels …
tfHelper.feed(inputName, pixels,
inputSize * inputSize);
tfHelper.run(outputNames);
tfHelper.fetch(outputName, output);
Step sixth - communicate with model
final float[] pixels …
tfHelper.feed(inputName, pixels,
inputSize * inputSize);
tfHelper.run(outputNames);
tfHelper.fetch(outputName, output);
Step sixth - communicate with model
final float[] pixels …
tfHelper.feed(inputName, pixels,
inputSize * inputSize);
tfHelper.run(outputNames);
tfHelper.fetch(outputName, output);
Tensorflow data type: Tensor
N dimensional array of data type
Each Tensor has:
Let’s use theory
public Classification recognize(final float[] pixels) {
tfHelper.feed(inputName, pixels, inputSize * inputSize);
Rank 1 Tensor
Shape of tensor is 28 * 28 = 784
Step sixth - display classification results
Classification classification = classifier.recognize(retPixels);
String result = String.format("It’s a %s with conf: %f", classification.getLabel(), classification.getConf());
Toast.makeText(this, result, Toast.LENGTH_SHORT).show();
It's working!
Let me do some Machine Learning...
Install Tensorflow
Hello world
import tensorflow as tf
hello = tf.constant('Hello,TensorFlow!')
sess = tf.Session()
print(sess.run(hello))
Experiment with existing implementations:
Export model to .pb format
Android Demo App:
https://github.com/bkraszewski/MachineLearningDemo
Interesting materials
Explore, experiment, ask why?
Thanks!