This file contains figures from the book:
Information Theory
A Tutorial Introduction
by
Dr James V Stone
2015
Sebtel Press.
Copyright JV Stone.
These figures are released for use under the Creative Commons License specified on next slide.
A copy of this file can be obtained from
http://jim-stone.staff.shef.ac.uk/BookInfoTheory2013/InfoTheoryBookFigures.html
Chapter 1
A
B
D
0 1 1 = 3
0 0 0 = 0
0 0 1 = 1
0 1 0 = 2
1 0 0 = 4
1 0 1 = 5
1 1 0 = 6
1 1 1 = 7
C
-----
-----
-----
--
-----
-----
-----
-----
Left
Right
0
1
0
0
1
1
1
1
1
1
0
0
0
0
0 1 1 = 3
0 0 0 = 0
0 0 1 = 1
0 1 0 = 2
1 0 0 = 4
1 0 1 = 5
1 1 0 = 6
1 1 1 = 7
Fish
Bird
Dog
Car
Van
Truck
Bus
No
Yes
Yes
Yes
Q1
Q2
Q3
Cat
Chapter 2
Random variable X
Experiment
Coin flip
Outcome
= head
Mapping
X(head) = 1
Random variable X
Experiment
Coin flip
Outcome
= tail
Mapping
X(tail) = 0
Output y
Noise η
Channel
Encoder
x = g(s)
Data s
Data s
Decoder
Input x
Chapter 3
Noise
Channel
Encoder
Data
Data
Decoder
Input
Output
X
A 0.87
B 0.04
C 0.04
D 0.03
E 0.02
0.08
0.05
0.13
1.00
0
1
1
0
1
1
0
0
from xkcd.com
From xkcd.com
Chapter 4
Output y
Noise η
Channel
Encoder
x = g(s)
Data s
Data s
Decoder
Input x
a
b
c
H(Y)
H(X)
I(X,Y)
H(Y|X)
H(X|Y)
H(X,Y)
H(Y)
H(X)
I(X,Y)
H(Y|X)
H(X|Y)
H(X,Y)
2H(Y|X)
2H(X|Y)
Number of
outputs = 2H(Y)
Number of
inputs = 2H(X)
Number of possible outputs
for one input = 2H(Y|X)
Number of possible inputs
for one output = 2H(X|Y)
Inputs X
Outputs Y
Output
entropy H(Y)
Mutual information I(X,Y)
Output noise
entropy H(Y|X)
Input
entropy H(X)
Mutual information I(X,Y)
Input noise
entropy H(X|Y)
(a) (b)
A
B
C
D
E
F
G
H
I
B
E
H
Chapter 5
Residual uncertainty
Residual
uncertainty
Residual uncertainty
b) After receiving 1 bit, U=50%
a) After receiving 2 bits, U=25%
c) After receiving 1/2 a bit, U=71%
Noise
Channel
Encoder
Data
Data
Decoder
Input
Output
Chapter 6
Output y
Noise η
Channel
Encoder
x = g(s)
Data s
Data s
Decoder
Input x
H(Y)
H(X)
I(X,Y)
H(Y|X)
H(X|Y)
H(X,Y)
Chapter 7
Chapter 8
Warm
Warm
A
B
Cold
Hot
A
B
Chapter 9
a
b
c
d
e
82
83
Blue-Yellow
channel
Luminance
channel
Red-Green
channel
S-cone
M-cone
L-cone
THE END.