Bayesian Decision Theory

Bayesian Decision Theory

• Design classifiers to recommend decisions that minimize some total expected ”risk”.
• The simplest risk is the classification error (i.e., costs are equal).
• Typically, the risk includes the cost associated with different decisions.

​

​

​

Terminology

• State of nature ω (random variable):
• e.g., ω₁ for sea bass, ω₂ for salmon

​

• Probabilities P(ω₁) and P(ω₂) (priors):
• e.g., prior knowledge of how likely is to get a sea bass or a salmon

​

• Probability density function p(x) (evidence):
• e.g., how frequently we will measure a pattern with feature value x (e.g., x corresponds to lightness)

​

Terminology (cont’d)

• Conditional probability density p(x/ω_j) (likelihood) :
• e.g., how frequently we will measure a pattern with feature value x given that the pattern belongs to class ω_j

​

e.g., lightness distributions

between salmon/sea-bass

populations

Terminology (cont’d)

• Conditional probability P(ω_j /x) (posterior) :
• e.g., the probability that the fish belongs to class ω_j given measurement x.

​

​

Decision Rule Using Prior Probabilities

Decide ω₁ if P(ω₁) > P(ω₂); otherwise decide ω₂

or P(error) = min[P(ω₁), P(ω₂)]

​

• Favours the most likely class.
• This rule will be making the same decision all times.
• i.e., optimum if no other information is available

Decision Rule Using Conditional Probabilities

• Using Bayes’ rule, the posterior probability of category ω_j given measurement x is given by:

​

​

where (i.e., scale factor – sum of probs = 1)

​

Decide ω₁ if P(ω₁ /x) > P(ω₂ /x); otherwise decide ω₂ or

Decide ω₁ if p(x/ω₁)P(ω₁)>p(x/ω₂)P(ω₂) otherwise decide ω₂

Decision Rule Using �Conditional pdf (cont’d)

P(ω_j /x)

p(x/ω_j)

Probability of Error

• The probability of error is defined as:

​

​

​

or

​

• What is the average probability error?

​

​

​

• The Bayes rule is optimum, that is, it minimizes the average probability error!

​

P(error/x) = min[P(ω₁/x), P(ω₂/x)]

Where do Probabilities Come From?

• There are two competitive answers to this question:

​

(1) Relative frequency (objective) approach.

• Probabilities can only come from experiments.

​

(2) Bayesian (subjective) approach.

• Probabilities may reflect degree of belief and can be based on opinion.

Prior and Posterior Probabilities

Example (objective approach)

• Classify cars whether they are more or less than $50K:
• Classes: C₁ if price > $50K, C₂ if price <= $50K
• Features: x, the height of a car

​

• Use the Bayes’ rule to compute the posterior probabilities:

​

​

​

• We need to estimate p(x/C₁), p(x/C₂), P(C₁), P(C₂)

Example (cont’d)

• Collect data
• Ask drivers how much their car was and measure height.
• Determine prior probabilities P(C₁), P(C₂)
• e.g., 1209 samples: #C₁=221 #C₂=988

Example (cont’d)

• Determine class conditional probabilities (likelihood)
• Discretize car height into bins and use normalized histogram

Example (cont’d)

• Calculate the posterior probability for each bin:

A More General Theory

• Use more than one features.
• Allow more than two categories.
• Allow actions other than classifying the input to one of the possible categories (e.g., rejection).
• Employ a more general error function (i.e., “risk” function) by associating a “cost” (“loss” function) with each error (i.e., wrong action).

Terminology

Conditional Risk (or Expected Loss)

• Suppose we observe x and take action α_i

​

• Suppose that the cost associated with taking action α_i with ω_j being the correct category is λ(α_i / ω_j)

​

• The conditional risk (or expected loss) with taking action α_i is:

​

​

​

Overall Risk

• Suppose α(x) is a general decision rule that determines which action α_1, α_{2, …,} α_l to take for every x; then the overall risk is defined as:

​

​

• The optimum decision rule is the Bayes rule

​

​

Overall Risk (cont’d)

• The Bayes decision rule minimizes R by:

(i) Computing R(α_i /x) for every α_i given an x

(ii) Choosing the action α_i with the minimum R(α_i /x)

​

• The resulting minimum overall risk is called Bayes risk and is the best (i.e., optimum) performance that can be achieved:

​

Example: Two-category classification

• Define
• α₁: decide ω₁
• α_2: decide ω₂
• λ_ij = λ(α_i /ω_j)

​

• The conditional risks are:

​

(c=2)

Example: Two-category classification (cont’d)

• Minimum risk decision rule:

or (i.e., using likelihood ratio)

or

>

threshold

likelihood ratio

Special Case:�Zero-One Loss Function

• Assign the same loss to all errors:

​

​

​

• The conditional risk corresponding to this loss function:

Special Case:�Zero-One Loss Function (cont’d)

• The decision rule becomes:

​

​

​

​

​

​

• In this case, the overall risk is the average probability error!

or

or

Example

(decision regions)

Decide ω₁ if p(x/ω₁)/p(x/ω₂)>P(ω₂ )/P(ω₁) otherwise decide ω₂

Assuming zero-one loss:

assume:

Assuming general loss: