Probability, Statistics and Errors�in High Energy Physics

Wen-Chen Chang

Institute of Physics, Academia Sinica

章文箴

中央研究院 物理研究所

Outline

• Errors
• Probability distribution: Binomial, Poisson, Gaussian
• Confidence Level
• Monte Carlo Method

Why do we do experiments?

1. Parameter determination: determine the numerical value of some physical quantity.
2. Hypothesis testing: test whether a particular theory is consistent with our data.

Why estimate errors?

• We are concerned not only with the answer but also with its accuracy.
• For example, speed of light 2.998x10⁸ m/sec
• (3.09±0.15) x10⁸:
• (3.09±0.01) x10⁸:
• (3.09±2) x10⁸:

​

Source of Errors

• Random (Statistic) error: the inability of any measuring device to give infinitely accurate answers.
• Systematic error: uncertainty.

Systematic Errors

Systematic effects is a general category which includes effects such as background, scanning efficiency, energy resolution, angle resolution, variation of counter efficiency with beam position and energy, dead time, etc. The uncertainty in the estimation of such as systematic effect is called a systematic error

Orear

Systematic Error: reproducible inaccuracy introduced by faulty equipment, calibration, or technique

Bevington

Error=mistake?

Error=uncertainty?

Experimental Examples

• Energy in a calorimeter E=aD+b

a & b determined by calibration expt

• Branching ratio B=N/(ηN_T)

η found from Monte Carlo studies

• Steel rule calibrated at 15C but used in warm lab

If not spotted, this is a mistake

If temp. measured, not a problem

If temp. not measured guess →uncertainty

​

Repeating measurements doesn’t help

The Binomial

n trials r successes

Individual success probability p

​

​

Variance

V≡σ²=<(r- μ )²>=<r²>-<r>²

=np(1-p)

Mean

μ=<r>=ΣrP( r )

= np

1-p ≡p ≡ q

A random process with exactly two possible outcomes which occur with fixed probabilities.

Binomial Examples

n=10

p=0.2

p=0.5

p=0.8

p=0.1

n=20

n=50

n=5

Poisson

‘Events in a continuum’

The probability of observing r independent events in a time interval t, when the counting rate is μ and the expected number events in the time interval is λ.

Mean

μ=<r>=ΣrP( r )

= λ

Variance

V≡σ²=<(r- μ )²>=<r²>-<r>²

=λ

λ=2.5

More about Poisson

• The approach of the binomial to the Poisson distribution as N increases.
• The mean value of r for a variable with a Poisson distribution is λ and so is the variance. This is the basis of the well known n±√n formula that applies to statistical errors in many situations involving the counting of independent events during a fixed interval.
• As λ→∞, the Poisson distribution tends to a Gaussian one.

Poisson Examples

λ=25

λ=10

λ=5.0

λ=0.5

λ=2.0

λ=1.0

Examples

• The number of particles detected by a counter in a time t, in a situation where the particle flux φ and detector are independent of time, and where counter dead-time τ is such that φ τ <<1.
• The number of interactions produced in a thin target when an intense pulse of N beam particles is incident on it.
• The number of entries in a given bin of a histogram when the data are accumulated over a fixed time interval.

Binomial and Poisson

From an exam paper

A student is standing by the road, hoping to hitch a lift. Cars pass according to a Poisson distribution with a mean frequency of 1 per minute. The probability of an individual car giving a lift is 1%. Calculate the probability that the student is still waiting for a lift

(a) After 60 cars have passed

(b) After 1 hour

1. 0.99⁶⁰=0.5472

b) e^-0.6 * 0.6⁰ /0! =0.5488

Gaussian (Normal)

Probability Density

​

Mean

μ=<x>=∫xP( x ) dx

=μ

Variance

V≡σ²=<(x- μ )²>=<x²>-<x>²

=σ²

Different Gaussians

There’s only one!

Normalisation (if required)

Location change μ

Width scaling factor

Falls to 1/e of peak at x=μ±σ

Probability Contents

68.27% within 1σ

95.45% within 2σ

99.73% within 3σ

​

90% within 1.645 σ

95% within 1.960 σ

99% within 2.576 σ

99.9% within 3.290σ

​

These numbers apply to Gaussians and only Gaussians

Other distributions have equivalent values which you could use of you wanted

Central Limit Theorem

Or: why is the Gaussian Normal?

If a variable x is produced by the convolution of variables x₁,x₂…x_N

I) <x>=μ₁+μ₂+…μ_N

2. V(x)=V₁+V₂+…V_N
3. P(x) becomes Gaussian for large N

Multidimensional Gaussian

Chi squared

Sum of squared discrepancies, scaled by expected error

Integrate all but 1-D of multi-D Gaussian

​

​

About Estimation

Theory

Data

Statistical

​

Inference

Theory

Data

Probability

​

Calculus

Given these distribution parameters, what can we say about the data?

Given this data, what can we say about the properties or parameters or correctness of the distribution functions?

What is an estimator?

An estimator (written with a hat) is a function of the data whose value, the estimate, is intended as a meaningful guess for the value of the parameter . (from PDG)

What is a good estimator?

A perfect estimator is:

• Consistent
• Unbiassed

​

• Efficient

minimum

​

Minimum Variance Bound

​

​

One often has to work with less-than-perfect estimators

The Likelihood Function

Set of data {x₁, x₂, x₃, …x_N}

Each x may be multidimensional – never mind

Probability depends on some parameter a

a may be multidimensional – never mind

Total probability (density)

P(x₁;a) P(x₂;a) P(x₃;a) …P(x_N;a)=L(x₁, x₂, x₃, …x_N ;a)

The Likelihood

​

Maximum Likelihood Estimation

In practice usually maximise ln L as it’s easier to calculate and handle; just add the ln P(x_i)

ML has lots of nice properties

​

Given data {x₁, x₂, x₃, …x_N} estimate a by maximising the likelihood L(x₁, x₂, x₃, …x_N ;a)

​

Properties of ML estimation

• It’s consistent

(no big deal)

• It’s biased for small N

May need to worry

• It is efficient for large N

Saturates the Minimum Variance Bound

• It is invariant

If you switch to using u(a), then û=u(â)

​

u

Ln L

û

More about ML

• It is not ‘right’. Just sensible.
• It does not give the ‘most likely value of a’. It’s the value of a for which this data is most likely.

​

• Numerical Methods are often needed
• Maximisation / Minimisation in >1 variable is not easy
• Use MINUIT but remember the minus sign

ML does not give �goodness-of-fit

• ML will not complain if your assumed P(x;a) is rubbish
• The value of L tells you nothing

Fit P(x)=a₁x+a₀

will give a₁=0; constant P

L= a₀^N

Just like you get from fitting

Least Squares

• Measurements of y at various x with errors σ and prediction f(x;a)
• Probability
• Ln L

​

​

• To maximise ln L, minimise χ²

x

y

So ML ‘proves’ Least Squares. But what ‘proves’ ML? Nothing

Least Squares: �The Really nice thing

• Should get χ²≈1 per data point
• Minimise χ² makes it smaller – effect is 1 unit of χ² for each variable adjusted. (Dimensionality of MultiD Gaussian decreased by 1.)

N_{degrees Of Freedom}=N_{data pts} – N _parameters

• Provides ‘Goodness of agreement’ figure which allows for credibility check

​

Chi Squared Results

Large χ² comes from

1. Bad Measurements
2. Bad Theory
3. Underestimated errors
4. Bad luck

Small χ² comes from

1. Overestimated errors
2. Good luck

Fitting Histograms

Often put {x_i} into bins

Data is then {n_j}

n_j given by Poisson,

mean f(x_j) =P(x_j)Δx

4 Techniques

Full ML

Binned ML

Proper χ²

Simple χ²

​

x

x

What you maximise/minimise

• Full ML

​

• Binned ML

​

​

• Proper χ²

​

​

• Simple χ²

​

Confidence Level:�Meaning of Error Estimates

• How often we expect to include “the true fixed value of our paramter” P₀, within our quoted range, p±δp, for a repeated series of experiments?
• For the actual value P₀, the probability that a measurement will give us an answer in a specific range of p is given by the area under the relevant part of Gaussian curve. A conventional choice of this probability is 68%.

The Straightforward Example

Apples of different weights

Need to describe the distribution

μ = 68g σ = 17 g

50 100

All weights between 24 and 167 g (Tolerance)

90% lie between 50 and 100 g

94% are less than 100 g

96% are more than 50 g

Confidence level statements

Confidence Levels

• Can quote at any level

(68%, 95%, 99%…)

• Upper or lower or two-sided

(x<U x<L L<x<U)

• Two-sided has further choice

(central, shortest…)

​

U

L

U’

Maximum Likelihood and Confidence Levels

ML estimator (large N) has variance given by MVB

​

At peak For large N

Ln L is a parabola (L is a Gaussian)

​

a

Ln L

Falls by ½ at

Falls by 2 at

Read off 68% , 95% confidence regions

Monte Carlo Calculations

• The Monte Carlo approach provides a method of solving probability theory problems in situations where the necessary integrals are too difficult to perform.
• Crucial element: random number generator.

​

​

An Example

References

• Lectures and Notes on Statistics in HEP, http://www.ep.ph.bham.ac.uk//group/locdoc/lectures/stats/index.html

​

• Lecture notes of Prof. Roger Barlow, http://www.hep.man.ac.uk/u/roger/
• Louis Lyons, “Statistics for Nuclear and Particle Physicists”, Cambridge 1986.
• Particle Data Group, http://pdg.lbl.gov/2004/reviews/contents_sports.html#mathtoolsetc