1 of 83

Random Variables

Fall 2025

Instructor:

Ajit Rajwade

2 of 83

Topic Overview

Random variable: definition
Discrete and continuous random variables
Probability density function (pdf) and cumulative distribution function (cdf)
Joint and conditional pdfs
Expectation and its properties
Variance and covariance
Markov’s and Chebyshev’s inequality
Weak law of large numbers
Moment generating functions

3 of 83

Random variable

In many random experiments, we are not always interested in the observed values, but in some numerical quantity determined by the observed values.

Example: we may be interested in the sum of the values of two dice throws, or the number of heads appearing in n consecutive coin tosses.

Any such quantities determined by the results of random experiments are called random variables (they may also be the observations themselves).

4 of 83

Random variable

Value of X (Denoted as x) where X = sum of 2 dice throws	P(X=x)
2	1/36
3	2/36
4	3/36
5	4/36
6	5/36
7	6/36
8	5/36
9	4/36
10	3/36
11	2/36
12	1/36

This is called the probability mass function (pmf) table of the random variable X. If S is the sample space, then

P(S) = P(union of all events of the form X = x) = 1 (verify from table).

5 of 83

Random variable: Notation

A random variable is usually denoted by an upper case alphabet.

Individual values the random variable can acquire are denoted by lower case.

6 of 83

Random variable: discrete

Random variables whose values can be written as a finite or infinite sequence are called discrete random variables.

Example: results of coin toss or random dice experiments

The probability that a random variable X takes on value x, i.e. P(X=x), is called as the probability mass function.

7 of 83

Random variable: continuous

Random variables that can take on values within a continuum are called continuous random variables.

Example: the dimensions (length, height, width, weight) of an object are usually continuous quantities, direction of a vector, amount of water that can be stored in a 4 litre jar is a continuous random variable in the interval [0,4].

8 of 83

Random variable: continuous

For a continuous random variable, the probability that it takes on any particular value within a continuum is zero!

Why? Because there are infinitely many values – say in the interval [0,4] in the example on the previous slide. Each value will be equally likely.

Note: Zero probability in case of continuous random variables does not mean the event will never occur! This differs from the discrete case.

9 of 83

Random variable: continuous

Hence for a continuous random variable X, we consider the cumulative distribution function (cdf) F_X(x) defined as P{X ≤ x}.

The cdf is basically the probability that X takes on a value less than or equal to x.

The cdf can be used to compute cumulative interval measures, that is the probability that X takes on a value greater than a and less than or equal to b, i.e. P(a < X ≤ b) = F_X(b) -F_X(a ).

10 of 83

Random variable: continuous - example

Consider a cdf of the form:

F_X(x) = 0 for x ≤ 0, and

F_X(x) = 1-exp(-x²) otherwise

To find: probability that X exceeds 1

P(X > 1) = 1-P(X ≤1)=1-F_X(1) = e^-1

11 of 83

Probability Density Function (pdf)

The pdf of a random variable X at a value x is the derivative of its cumulative distribution function (cdf) at that value x.

It is a non-negative function f_X(x) such that for any set B of real numbers, we have

Properties:

12 of 83

The area beneath the blue curve in between the lines x = a and x = b is the cumulative interval measure P(a < X ≤ b) = F_X(b) -F_X(a ).

f_X(a)dx = probability that the random variable X takes on values between a and a+dx.

f_X(x)

13 of 83

Probability Density Function

Another way of looking at this concept:

14 of 83

Examples: Popular families of PDFs

Gaussian (normal) pdf:

15 of 83

Examples: Popular families of PDFs

Bounded uniform pdf:

16 of 83

Expected Value (Expectation) of a random variable

It is also called the mean value of the random variable.

For a discrete random variable X, it is defined as:

For a continuous random variable X, it is defined as:

The expected value should not be (mis)interpreted to be the value that X usually takes on – it’s the average value, not the “most frequently occurring value”.

17 of 83

Expected Value (Expectation) of a random variable

For some pdfs, the expected value is not always defined, i.e. the integral below may not have a finite value.

One example is the pdf for the Pareto distribution (under some parameters) given as:

x_m and α are parameters of the pdf for the Pareto distribution. Verify this result for E(X) on your own.

18 of 83

Expected Value (Expectation) of a random variable

Likewise for some discrete random variables which take on infinitely many values, the expected value may not be defined, i.e. we may have

Example:

See here.

19 of 83

Expected Value: examples

The expected value that shows up when you throw a die is 1/6(1+2+3+4+5+6) = 3.5.

The game of roulette consists of a ball and wheel with 38 numbered pockets on its side. The ball rolls and settles on one of the pockets. If the number in the pocket is the same as the one you guessed, you win $35 (probability 1/38), otherwise you lose $1 (probability 37/38). The expected value of the amount you earn after one trial is: (-1)37/38 +(35)1/38 = $-0.0526

20 of 83

A Game of Roulette

https://en.wikipedia.org/wiki/Roulette#/media/File:Roulette_casino.JPG

21 of 83

Expected value of a function of random variable

Consider a function g(X) of a discrete random variable X. The expected value of g(X) is defined as (provided the summation is well-defined):

For a continuous random variable, the expected value of g(X) is defined as (provided the integral is well-defined):

This is called the Law of the Unconscious Statistician (LOTUS). It is something which requires proof, but is stated as if it were obvious.

22 of 83

Proof: Law of the Unconscious Statistician

We are starting with a conventional definition, which actually requires rigorous proof. We will show on the next slide that this is equal to the expected value of Y based on the fundamental definition of expectation.

23 of 83

Proof (continued): Law of the Unconscious Statistician

This justifies the previous formula for E(g(X)). Most textbooks just gloss over the formula, ignoring the fact that it requires proof. We have established that the accepted expression obeys the rigorous definition of expectation.

This step assumes that g is a strictly increasing function. There is a fix to this, if g were strictly decreasing. What is it?

24 of 83

Properties of expected value

This property is called the linearity of the expected value. In general, a function f(x) is said to be linear in x is f(ax+b) = af(x)+f(b) where a and b are constants. In this case, the expected value is not a function but an operator (it takes a function as input). An operator E is said to be linear if

E(af(x) + b) = a E(f(x)) + E(b). This is equal to aE(f(x)) + b for the expectation operator.

25 of 83

Properties of expected value

Consider a set of random variables X₁, X₂,…, X_n; a set of functions g₁, g₂,…, g_n. Then we have:

This also forms a notion of the linearity of the expectation operator.

Note: for a general nonlinear function g, we have:

a_i, b_i are scalars

26 of 83

Properties of expected value

Suppose you want to predict the value of a random variable with a known mean. On an average, what value will yield the least squared error?

The expected value is the value that yields the least mean squared prediction error!

27 of 83

The median

What minimizes the following quantity?

28 of 83

The median

In this derivation, we are assuming that the two definite integrals of q(x) exist! This proof won’t go through otherwise.

29 of 83

The median

This is the median – by definition and it minimizes J(c). We can double check that J’’(c) >= 0. Notice the peculiar definition of the median for the continuous case here! This definition is not conceptually different from the discrete case, though. Also, note that the median will not be unique if F_X is not differentiable at c. This happens when F_X is not strictly increasing in some interval – say K = [c,c+ε] or [c-ε,c]. In such cases, all y ϵ K will qualify as medians and all of them will produce the same value of J(y). This is because f_x(y) = 0 for y ϵ K.

30 of 83

What cost function does a quantile minimize?

We obtained the median by minimizing the following cost function w.r.t. c (go back a few slides)

31 of 83

Variance

The variance of a random variable X tells you how much its values deviate from the mean – on an average.

The definition of variance for a continuous r.v. with mean μ is:

For a discrete r.v., the integration is replaced by a summation:

The positive square-root of the variance is called the standard deviation.

Low-variance probability mass functions or probability densities tend to be concentrated around one point. High variance densities are spread out.

32 of 83

Existence?

For some distributions, the variance (and hence standard deviation) may not be defined, because the integral may not have a finite value.

Example: Pareto distribution (see slides on expectation for definition) for α < 2.

Note in some cases the mean is defined, but the variance is not. In some cases both are undefined. However, if the mean is undefined, then the variance will be undefined too (why?).

33 of 83

Variance: Alternative expression

The definition of variance is:

Alternative expression:

34 of 83

Variance: properties

Property:

35 of 83

Probabilistic inequalities

Sometimes we know the mean or variance of a random variable, and want to guess the probability that the random variable can take on a certain value.

The exact probability can usually not be computed as the information is too less. But we can get upper or lower bounds on this probability which can influence our decision-making processes.

36 of 83

Probabilistic inequalities

Example: Let’s say the average annual salary offered to a CSE Btech-4 student at IITB is $100,000. What’s the probability that you (i.e. a randomly chosen student) will get an offer of $110,000 or more? Additionally, if you were told that the variance of the salary was 50,000, what’s the probability that your package is between $90,000 and $110,000?

37 of 83

Markov’s inequality

Let X be a random variable that takes only non-negative values. For any a > 0, we have

Proof: next slide

38 of 83

Markov’s inequality

Proof:

39 of 83

Chebyshev’s inequality

For a random variable X with mean μ and variance σ², we have for any value k > 0,

Proof: follows from Markov’s inequality

40 of 83

https://en.wikipedia.org/wiki/Andrey_Markov

Andrey Andreyevich Markov^[a] (14 June [O.S. 2 June] 1856 – 20 July 1922) was a Russian mathematician celebrated for his pioneering work in stochastic processes. Among his teachers were Yulian Sokhotski (differential calculus, higher algebra), Konstantin Posse (analytic geometry), Yegor Zolotarev (integral calculus), Pafnuty Chebyshev (number theory and probability theory), Aleksandr Korkin (ordinary and partial differential equations), Mikhail Okatov (mechanism theory), Osip Somov (mechanics), and Nikolai Budajev (descriptive and higher geometry).

41 of 83

Chebyshev’s inequality: another form

For a random variable X with mean μ and variance σ², we have for any value k > 0,

If I replace k by kσ, I get the following:

42 of 83

Back to counting money! ☺

Let X be the random variable indicating the annual salary offered to you when you reach BTech-4 ☺�
Then

43 of 83

Back to the expected value

When I tell you that the expected value of a random die variable is 3.5, what does this mean?

If I throw the die n times, and average the results, I should get a value close to 3.5 provided n is very large (not valid if n is small).

As n increases, the average value should move closer and closer towards 3.5.

That’s our basic intuition!

44 of 83

https://en.wikipedia.org/wiki/Law_of_large_numbers

45 of 83

Back to the expected value: weak law of large numbers

This intuition has a rigorous theoretical justification in a theorem known as the weak law of large numbers.

Let X₁, X₂,…,X_n be a sequence of independent and identically distributed random variables each having mean μ. Then for any ε > 0, we have:

46 of 83

Back to the expected value: weak law of large numbers

Let X₁, X₂,…,X_n be a sequence of independent and identically distributed random variables each having mean μ. Then for any ε > 0, we have:

Proof: follows immediately from Chebyshev’s inequality

Empirical (or sample) mean

47 of 83

Comments on weak law of large numbers

The assumption of “identically distributed” is not strictly necessary (even though it is stated that way in many texts), but the random variables should have the same mean μ.

The previous proof assumes they have the same variance – if not the nσ² term in the RHS numerator would be replaced by n times the average variance of the random variables. The proof as such still goes through.

The independence assumption is not strictly necessary – the random variables just need to be pair-wise uncorrelated (more on this in later slides) – so that the variances can add up as shown in the proof.

The law assumes that the random variables have a well-defined expected value and variance. Otherwise, the law may not hold.

48 of 83

The strong law of large numbers

The strong law of large numbers states the following:

This is stronger than the weak law because this states that the probability of the desired event (that the empirical mean is equal to the actual mean) is equal to 1 given enough samples. The weak laws states that it tends to 1 given enough samples.

The proof of the strong law is formidable and beyond the scope of our course.

49 of 83

(The incorrect) Law of averages

As laymen we tend to believe that if something has been going wrong for quite some time, it will suddenly turn right – using the law of averages.

This supposed law is actually a fallacy – it reflects wishful thinking, and the core mistake is that we mistake the distribution of samples among a small set of outcomes for the distribution of a larger set.

This is also called as Gambler’s fallacy.

50 of 83

(The incorrect) Law of averages

Let’s say a gambler independently tosses an unbiased coin 20 times, and gets a head each time. He now applies the “law of averages” and believes that it is more likely that the next coin toss will yield a tail.

The mistake is as follows: The probability of getting all 21 heads = (1/2)²¹. The probability of getting 20 heads and 1 tail also = (1/2)²¹. �

51 of 83

Joint distributions/pdfs/pmfs

52 of 83

Jointly distributed random variables

Many times in statistics, one needs to model relationships between two or more random variables – for example, your CPI at IITB and the annual salary offered to you during placements!

Another example: average amount of sugar consumed per day and blood sugar level recorded in a blood test.

Another example: literacy level and crime rate.

53 of 83

Joint CDFs

Given continuous random variables X and Y, their joint cumulative distribution function (cdf) is defined as:

The distribution of either random variable (called as marginal cdf) can be obtained from the joint distribution as follows:

These definitions can extended to handle more than two random variables as well.

I’ll explain this a few slides further down

54 of 83

Joint PMFs

Given two discrete random variables X and Y, their joint probability mass function (pmf) is defined as:

The pmf of either random variable (called as marginal pmf) can be obtained from the joint distribution as follows:

Why?

55 of 83

Joint PMFs: Example

Consider that in a city 15% of the families are childless, 20% have only one child, 35% have two children and 30% have three children. Let us suppose that male and female child are equally likely and independent.
What is the probability that a randomly chosen family has no children?
P(B = 0, G = 0) = 0.15 = P(no children)
Has 1 girl child and no boy child?
P(B=0,G=1)=P(1 child) P(G=1|1 child) = 0.2 x 0.5 = 0.1
Has 3 girls?
P(B = 0, G = 3) = P(3 children) P(G=3 | 3 Children) = 0.3 x (0.5)³
Has 2 boys and 1 girl?
P(B = 2, G = 1) = P(3 children) P(B = 2, G = 1| 3 children) = 0.3 x (1/8) x 3 = 0.1125 (all 8 combinations of 3 children are equally likely. Out of these there are 3 of the form 2 boys + 1 girl)

56 of 83

Joint PDFs

For two jointly continuous random variables X and Y, the joint pdf is a non-negative function f_XY(x,y) such that for any set C in the two-dimensional plane, we have:

The joint CDF can be obtained from the joint PDF as follows:

57 of 83

The joint probability that (X,Y) belongs to any arbitrary-shaped region in the XY-plane is obtained by integrating the joint pdf of (X,Y) over that region (eg: region C)

58 of 83

Joint and marginal PDFs

The marginal pdf of a random variable can be obtained by integrating the joint pdf w.r.t. the other random variable(s):

59 of 83

Independent random variables

Two continuous random variables are said to be independent if and only if:

i.e., the joint pdf is equal to the product of the marginal pdfs.

For independent random variables, the joint CDF is also equal to the product of the marginal CDFs:

Try proving this yourself!

60 of 83

Independent random variables

Some n continuous random variables X₁, X₂, …, X_n are said to be mutually independent if and only if for any finite subset of k random variables X_i₁, X_i₂,…, X_ik and finite sequence of number x₁, x₂,…, x_k , the events X_i₁ ≤ x₁, X_i₂ ≤ x₂,…, X_ik ≤ x_k are mutually independent.

As a consequence

i.e., the joint pdf is equal to the product of all n marginal pdfs.

Note that this condition is stronger than pairwise independence!

61 of 83

Independent random variables

Mutual independence between n random variables implies that they are pairwise independent, or in fact, k-wise independent for any k < n.

But pairwise independence does not necessarily imply mutual independence.

Example: Consider a sample space {1,2,3,4} where each singleton element is equally likely to be chosen.

62 of 83

Independent random variables

Consider A = {1,2}, B = {1,3}, C = {1,4}.

Then P(A) = P(B) = P(C) = 1/2. P(ABC) = P({1}) = ¼ ≠ P(A)P(B)P(C) implying that A,B,C are not mutually independent.

But P(AB) = ¼ = P(A)P(B) and likewise for AC, BC.

Let us define the random variables EA, EB, EC which acquire the value 1 if events A,B,C respectively occur, and 0 otherwise.

We can see that EA, EB, EC are pair-wise independent random variables, but not mutually independent.

63 of 83

Concept of covariance

The covariance of two random variables X and Y is defined as follows:

Further expansion:

64 of 83