Welcome to TensorFlow!
CS 20: TensorFlow for Deep Learning Research
Lecture 1
1/12/2018
1
Thanks Danijar Hafner for the logo!
2
Agenda
Welcome
Overview of TensorFlow
Graphs and Sessions
3
What’s TensorFlow™?
“Open source software library for �numerical computation using data flow graphs”
4
Launched Nov 2015
5
Why TensorFlow?
6
Why TensorFlow?
7
Why TensorFlow?
8
Companies using TensorFlow
9
Demand for tutorials on TensorFlow
10
Some cool projects using TensorFlow
11
Classify skin cancer
12
Dermatologist-level classification of skin cancer with deep neural networks (Esteva et al., Nature 2017)
WaveNet: Text to Speech
13
Wavenet: A generative model for raw audio (Oord et al., 2016)
It takes several hours to synthesize 1 second!
Drawing
14
Draw Together with a Neural Network (Ha et al., 2017)
Neural Style Translation
15
Image Style Transfer Using Convolutional Neural Networks (Gatys et al., 2016)
Tensorflow adaptation by Cameroon Smith (cysmith@github)
I hope that this class will give you the tool to build cool projects like those!
16
Goals
17
CS20
18
Staff
Chip Huyen�huyenn@stanford.edu
Michael Straka�mstraka2@stanford.edu
Pedro Garzon�pgarzon@stanford.edu
19
Logistics
20
Grading
21
Resources
22
Permission Number
23
Many of you are ahead of me in academia so I probably need more of your help than you do mine
24
Getting Started
25
import tensorflow as tf
26
Graphs and Sessions
27
Data Flow Graphs
TensorFlow separates definition of computations from their execution
Graph from TensorFlow for Machine Intelligence
28
Data Flow Graphs
Phase 1: assemble a graph
Phase 2: use a session to execute operations in the graph.
Graph from TensorFlow for Machine Intelligence
29
Data Flow Graphs
Phase 1: assemble a graph
Phase 2: use a session to execute operations in the graph.
Graph from TensorFlow for Machine Intelligence
30
This might change in the future with eager mode!!
What’s a tensor?
31
What’s a tensor?
An n-dimensional array
0-d tensor: scalar (number)
1-d tensor: vector
2-d tensor: matrix
and so on
32
Data Flow Graphs
import tensorflow as tf�a = tf.add(3, 5)
33
Visualized by TensorBoard
Data Flow Graphs
import tensorflow as tf�a = tf.add(3, 5)
34
Why x, y?
TF automatically names the nodes when you don’t explicitly name them.
x = 3
y = 5
Visualized by TensorBoard
Data Flow Graphs
import tensorflow as tf�a = tf.add(3, 5)
35
Interpreted?
5
3
a
Nodes: operators, variables, and constants
Edges: tensors
Data Flow Graphs
import tensorflow as tf�a = tf.add(3, 5)
36
Nodes: operators, variables, and constants
Edges: tensors
Tensors are data.
TensorFlow = tensor + flow = data + flow
(I know, mind=blown)
Interpreted?
5
3
a
Data Flow Graphs
import tensorflow as tf�a = tf.add(3, 5)�print(a)
37
>> Tensor("Add:0", shape=(), dtype=int32)
(Not 8)
5
3
a
How to get the value of a?
Create a session, assign it to variable sess so we can call it later
Within the session, evaluate the graph to fetch the value of a
38
How to get the value of a?
Create a session, assign it to variable sess so we can call it later
Within the session, evaluate the graph to fetch the value of a
import tensorflow as tf�a = tf.add(3, 5)�sess = tf.Session()�print(sess.run(a))�sess.close()
39
The session will look at the graph, trying to think: hmm, how can I get the value of a,
then it computes all the nodes that leads to a.
How to get the value of a?
Create a session, assign it to variable sess so we can call it later
Within the session, evaluate the graph to fetch the value of a
import tensorflow as tf�a = tf.add(3, 5)�sess = tf.Session()�print(sess.run(a))�sess.close()
40
>> 8
8
The session will look at the graph, trying to think: hmm, how can I get the value of a,
then it computes all the nodes that leads to a.
How to get the value of a?
Create a session, assign it to variable sess so we can call it later
Within the session, evaluate the graph to fetch the value of a
import tensorflow as tf�a = tf.add(3, 5)�sess = tf.Session()�with tf.Session() as sess:� print(sess.run(a))�sess.close()
41
8
tf.Session()
A Session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated.
42
tf.Session()
A Session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated.
�Session will also allocate memory to store the current values of variables.
43
More graph
x = 2�y = 3�op1 = tf.add(x, y)�op2 = tf.multiply(x, y)�op3 = tf.pow(op2, op1)�with tf.Session() as sess:� op3 = sess.run(op3)
44
Visualized by TensorBoard
Subgraphs
x = 2�y = 3�add_op = tf.add(x, y)�mul_op = tf.multiply(x, y)�useless = tf.multiply(x, add_op)�pow_op = tf.pow(add_op, mul_op)�with tf.Session() as sess:� z = sess.run(pow_op)
45
Because we only want the value of pow_op and pow_op doesn’t depend on useless, session won’t compute value of useless
→ save computation
useless
add_op
mul_op
pow_op
Subgraphs
x = 2�y = 3�add_op = tf.add(x, y)�mul_op = tf.multiply(x, y)�useless = tf.multiply(x, add_op)�pow_op = tf.pow(add_op, mul_op)�with tf.Session() as sess:� z, not_useless = sess.run([pow_op, useless])
46
tf.Session.run(fetches,
feed_dict=None,
options=None,
run_metadata=None)
fetches is a list of tensors whose values you want
useless
add_op
mul_op
pow_op
Subgraphs
Possible to break graphs into several chunks and run them parallelly across multiple CPUs, GPUs, TPUs, or other devices
Example: AlexNet
Graph from Hands-On Machine Learning with Scikit-Learn and TensorFlow
47
Distributed Computation
To put part of a graph on a specific CPU or GPU:
# Creates a graph.
with tf.device('/gpu:2'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='b')
c = tf.multiply(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print(sess.run(c))
48
What if I want to build more
than one graph?
49
You can
but you don’t need more than one graph
The session runs the default graph
50
But what if I really want to?
51
URGH, NO
52
53
BUG ALERT!
I insist ...
54
tf.Graph()
create a graph:
g = tf.Graph()
55
tf.Graph()
to add operators to a graph, set it as default:
g = tf.Graph()�with g.as_default():� x = tf.add(3, 5)�sess = tf.Session(graph=g)�with tf.Session() as sess:� sess.run(x)
56
tf.Graph()
To handle the default graph:
g = tf.get_default_graph()
57
tf.Graph()
Do not mix default graph and user created graphs
g = tf.Graph()
# add ops to the default graph�a = tf.constant(3)
# add ops to the user created graph�with g.as_default():� b = tf.constant(5)
58
Prone to errors
tf.Graph()
Do not mix default graph and user created graphs
g1 = tf.get_default_graph()�g2 = tf.Graph()
# add ops to the default graph�with g1.as_default():� a = tf.Constant(3)
# add ops to the user created graph�with g2.as_default():� b = tf.Constant(5)
59
Better
But still not good enough because no more than one graph!
60
Why graphs
61
Why graphs
62
Why graphs
63
Why graphs
A neural net graph from Stanford’s CS224N course
64
Next class
Basic operations
Constants and variables
Data pipeline
Fun with TensorBoard
Feedback: huyenn@stanford.edu
Thanks!
65