Probability/Random Variables


DefinitionEdit

Definition. (Random variable) Formally, a random variable on a probability space   is a measurable real function   defined on   (the set of possibleh outcomes)

Remark.

  • The property of measurability means that for each real  , the set
 , i.e. is an event in the probability space.
  • Measurability will not be emphasized in this book.
  • In some definitions, the codomain of the random variable is defined as  , namely the extended real number line.
  • Usually, a capital letter is used to represent a random variable, and the small corresponding letter is used to represent a value taken by the random variable, e.g.   is a value taken by random variable  .

Since random variable maps the outcomes in   to a certain number, it can quantify the outcomes in  , which can be useful. Another function which is related to random variable in some sense is indicator function. It is useful in many situations.

Definition. (Indicator function) For each statement (which is usually an event)  , the indicator function is

 

Example. Let   be the number of heads facing up from tossing an unfair coin one time. Then,   is a random variable, since

 
Also, if we let   be the number of times the coin is tossed, then   is still a random variable, since   is  , and  , even if   only contains one element.

Remark. Actually,  .

Proposition. (Properties of indicator function) For each event  ,

  • (Complementary event)  

For each event  ,

  • (Intersection of events)  
  • (Union of events)   for mutually exclusive events  

Proof. Outline:

Complementary event:   is true   is false, and   is true   is false.

Intersection of events: when one of events   is false,   is false, and the product at right hand side becomes zero as well.

Union of events: since the events are mutually exclusive, at most one of the events is true, so the sum of the right hand side cannot be larger than 1. Also, if one of the events   is true, then the union of events   is also true, and the sum at right hand side becomes one as well.

 

Exercise.

A fair six-faced dice is thrown one time. Define   be the number facing up. Which of the following is (are) true?

 
 
 
 
 
 
 


Cumulative distribution functionEdit

Definition.

 
Three examples of cdf, which are illustrated by the red lines and dots between two blue lines.

(Cumulative distribution function) The cumulative distribution function (cdf) of random variable   is

 
in which  .

Remark.

  • Cdf completely determine the random behaviour of a random variable.

Example. Suppose we toss a coin two times, then the sample space   is   in which   means head and tail come up in first and second toss respectively, other notations are defined similarly. If we define   to be the number of heads, and

 
Show that the cdf of   and   are
 

Proof. For cdf of  , first,  ,   for each   and  .

If  ,   since  .

If  ,   since  .

If  ,   since  .

If  ,   since  .

Similarly, we can get the desired cdf of  , by considering   for   in different ranges.

 

Remark. Graphically, the cdf of   and   is step function.

Exercise.

Suppose cdf of a random variable   is  .

1 Given that  , compute  .

0
0.1
0.9
1

2 In addition to  , it is further given that  . Compute  .

0
0.01
0.1
0.2
0.8

3 Which of the following is (are) possible?

 
 
 
 
 


In the following, we will discuss three defining properties of cdf.

Theorem. (Defining properties of cdf) A function   is the cdf of a random variable   if and only if

(i)   for each real number  .

(ii)   is nondecreasing.

(iii)   is right-continuous.

Proof. Only if part (  is cdf   these three properties):

(i) It follows the axioms of probability since   is defined to be a probability.

(ii)

 

(iii) Fix an arbitrary positive sequence   with  . Define   for each positive number  . It follows that  . Then,

 
It follows that
 
for each   with   as  . That is,
 
which is the definition of right-continuity.

If part is more complicated. The following is optional. Outline:

  1. Draw an arbitrary curve satisfying the three properties.
  2. Throw a fair coin infinitely many times.
  3. Encode each result into a binary number, e.g.  
  4. Transform each binary number to a decimal number, e.g.  . Then, the decimal number is a random variable  .
  5. Use this decimal number as the input of the inverse function of the arbitrarily drawn curve, and we get a value, which is also a random variable, say  .
  6. Then, we obtain a cdf of the random variable    , if we throw a fair coin infinitely many times.

 


Sometimes, we are only interested in the values   such that  , which are more 'important'. Roughly speaking, the values are actually the elements of the support of  , which is defined in the following.

Definition. (Support of random variable) The support of a random variable  ,  , is the smallest closed set   such that  .

Remark.

  • E.g. closed interval is closed set.
  • Closedness will not be emphasized in this book.
  • Practically,   (which is the smallest closed set).
  •   is probability mass function for discrete random variables;
  •   is probability density function for continuous random variables.
  • The terms mentioned above will be defined later.

Example. If

 
then  , since   and this set is the smallest set among all sets satisfying this requirement.

Remark.   etc. also satisfy the requirement, but they are not the smallest set.

Exercise.

Suppose we throw an unfair coin. Define   if head comes up and   otherwise. Let   be the cdf of  .

1 Find  .

 
 
 
It cannot be determined since the probability that head comes up is not given.

2 Suppose  , compute  .

0
0.3
0.5
0.7
1

3 Suppose  . Which of the following is (are) true?

 
 
 
  if the coin is fair instead.
 


Discrete random variablesEdit

Definition. (Discrete random variables) If   is countable (i.e. 'enumerable' or 'listable'), then the random variable   is a discrete random variable.

Example. Let   be the number of successes among   Bernoulli trials. Then,   is a discrete random variable, since   which is countable.

On the other hand, if we let   be the temperature on Celsius scale,   is not discrete, since   which is not countable.

Exercise.

Which of the following is (are) discrete random variable?

Number of heads coming up from tossing a coin three times.
A number lying between 0 and 1 inclusively.
Number of correct option(s) in a multiple choice question in which there are at most three correct options.
Answer to a short question asking for a numeric answer.
Probability for a random variable to be discrete random variable.


Often, for discrete random variable, we are interested in the probability that the random variable takes a specific value. So, we have a function that gives the corresponding probability for each specific value taken, namely probability mass function.

Definition.

 
An example of pmf. This function is called probability mass function, since the value at each point may be interpreted as the mass of the dot located at that point.

(Probability mass function) Let   be a discrete random variable. The probability mass function (pmf) of   is

 

Remark.

  • Alternative names include mass function and probability function.
  • If random variable   is discrete, then   (it is closed).
  • The cdf of random variable   is  . It follows that the sum of the value of pmf at each   inside the support equals one.
  • The cdf of a discrete random variable   is a step function with jumps at the points in  , and the size of each jump defines the pmf of   at the corresponding point in  .

Example. Suppose we throw a fair six-faced dice one time. Let   be the number facing up. Then, pmf of   is

 

Exercise.

1 Which of the following is (are) pmf?

 . It is given that   is countable.
 
 
 
 

2 Compute   such that the function   is a pmf.

 
 
 
 


Continuous random variablesEdit

Suppose   is a discrete random variable. Partitioning   into small disjoint intervals   gives

 
In particular, the probability per unit can be interpreted as the density of the probability of   over the interval. (The higher the density, the more probability is distributed (or allocated) to that interval).

Taking limit,

 
in which, intuitively and non-rigorously,   can be interpreted as the probability over 'infinitesimal' interval  , i.e.  , and   can be interpreted as the density of the probability over the 'infinitesimal' interval, i.e.  .

These motivate us to have the following definition.

Definition. (Continuous random variable) A random variable   is continuous if

 
for each (measurable) set   and for some nonnegative function  .

Remark.

  • The function   is called probability density function (pdf), density function, or probability function (rarely).
  • If   is continuous, then the value of pdf at each single value is zero, i.e.   for each real number  .
  • This can be seen by setting  , then   (dummy variable is changed).
  • By setting  , the cdf  .
  • Measurability will not be emphasized. The sets encountered in this book are all measurable.
  •   is the area of pdf under  , which represents probability (which is obtained by integrating the density function over the set  ).

The name continuous r.v. comes from the result that the cdf of this kind of r.v. is continuous.

Proposition. (Continuity of cdf of continuous random variable) If a random variable   is continuous, its cdf   is also continuous (not just right-continuous).

Proof. Since   (Riemann integral is continuous), the cdf is continuous.

 

Example. (Exponential distribution) The function   is a cdf of a continuous random variable since

  • It is nonnegative.
  •  . So,  .
  • It is nondecreasing.
  • It is right-continuous (and also continuous).

Exercise.

1 Which of the following is (are) pdf?

 
 
 
 
 

2 Compute   such that the function   is a pdf.

 
 
 
 
There does not exist such  .

3 Compute   such that the function   is a cdf.

 
 
 
 
There does not exist such  .

4 Which of the following is (are) true?

If the support of a random variable is countable, then it is discrete.
If the support of a random variable is not countable, then it is continuous.
If the support of a random variable is not countable, then it is not discrete.


Proposition. (Finding pdf using cdf) If cdf   of a continuous random variable is differentiable, then the pdf  .

Proof. This follows from fundamental theorem of calculus:

 

 

Remark. Since   is nondecreasing,  . This shows that   is always nonnegative if   is differentiable. It is a motivation for us to define pdf to be nonnegative.

Without further assumption, pdf is not unique, i.e. a random variable may have multiple pdf's, since, e.g., we may set the value of pdf to be a real number at a single point outside its support (without affecting the probabilities, since the value of pdf at a single point is zero regardless of the value), and this makes another valid pdf for a random variable. To tackle this, we conventionally set   for each   to make the pdf become unique, and make the calculation more convenient.

Example. (Uniform distribution) Given that

 
is a pdf of a continuous random variable  , the probability  

Exercise.

It is given that the function   is a pdf of a continuous random variable  .

1 Compute  .

 
 
 
 
 

2 Compute  .

 
 
 
 
 

3 Compute  .

 
 
 
 
 



Mixed random variablesEdit

You may think that a random variable can either be discrete or continuous after reading the previous two sections. Actually, this is wrong. A random variable can be neither discrete nor continuous. An example of such random variable is mixed random variable, which is discussed in this section.

Theorem. (cdf decomposition) The cdf   of each random variable   can be decomposed as a sum of three components:

 
for some nonnegative constants   such that  , in which   is a real number,   is cdf of discrete, continuous, and singular random variable respectively.

Remark.

  • If   and  , then   is a mixed random variable.
  • We will not discuss singular random variable in this book, since it is quite advanced.
  • One interpretation of this formula is:
     
  • If   is discrete (continuous) random variable, then   ( ).
  • We may also decompose pdf similarly, but we have different ways to find pdf of discrete and continuous random variable from the corresponding cdf.

An example of singular random variable is the Cantor distribution function (sometimes known as Devil's Staircase), which is illustrated by the following graph. The graph pattern keeps repeating when you enlarge the graph.

 
Cantor distribution function

Example. Let  . Let  . Then,   is a cdf of a mixed random variable  , with probability   to be discrete and probability   to be continuous, since it is nonnegative, nondecreasing, right-continuous and  .

Exercise. Consider the function  . It is given that   is a cdf of a random variable  .

(a) Show that  .

(b) Show that the pdf of   is

 

(c) Show that the probability for   to be continuous is  .

(d) Show that   is  .

(e) Show that the events   and   are independent if   .


Proof.

(a) Since   is a cdf, and   when  ,

 

(b) Since   is a mixed random variable, for the discrete random variable part, the pdf is

 
On the other hand, for the continuous random variable part, the pdf is
 
Therefore, the pdf of   is
 

(c) We can see that   can be decomposed as follows:

 
Thus, the probability for   to be continuous is  .

(d)

 

(e) If  ,  . Thus,

 
i.e.   and   are independent.