Probability/Joint Distributions and Independence


MotivationEdit

Suppose we are given a pmf of a discrete random variable   and a pmf of a discrete random variable  . For example,

 
We cannot tell the relationship between   and   with only such information. They may be related or not related.

For example, the random variable   may be defined as   if head comes up and   otherwise from tossing a fair coin, and the random variable   may be defined as   if head comes up and   otherwise from tossing the coin another time. In this case,   and   are unrelated.

Another possibility is that the random variable   is defined as   if head comes up in the first coin tossing, and   otherwise. In this case,   and   are related.

Yet, in the above two examples, the pmf of   and   are exactly the same.

Therefore, to tell the relationship between   and  , we define the joint cumulative distribution function, or joint cdf.

Joint distributionsEdit

Definition. (Joint cumulative distribution function) Let   be random variables defined on a sample space  . The joint cumulative distribution function (cdf) of random variables   is

 

Sometimes, we may want to know the random behaviour in one of the random variables involved in a joint cdf. We can do this by computing the marginal cdf from joint cdf. The definition of marginal cdf is as follows:

Definition. (Marginal cumulative distribution function) The cumulative distribution function   of each random variable   is marginal cumulative distribution function (cdf) of   which is a member in the   random variables  .

Remark. Actually, the marginal cdf of   is simply the cdf of   (which is in one variable). We have already discussed this kind of cdf in previous chapters.

Proposition. (Obtaining marginal cdf from joint cdf) Given a joint cdf  , the marginal cdf of   is

 

Proof. When we set the arguments other than  -th argument to be  , e.g.   , the joint cdf becomes

 
 

Remark. In general, we cannot deduce the joint cdf from a given set of marginal cdf's.

Example. Consider the joint cdf of random variables   and  :

 
The marginal cdf of   is
 

Similar to the one-variable case, we have joint pmf and joint pdf. Also, analogously, we have marginal pmf and marginal pdf.

Definition. (Joint probability mass function) The joint probability mass function (joint pmf) of   is

 

Definition. (Marginal probability mass function) The marginal probability mass function (marginal pmf) of each   which is a member of the   random variables   is

 

Proposition. (Obtaining marginal pmf from joint pmf) For discrete random variables   with joint pmf  , the marginal pmf of   is

 

Proof. Consider the case in which there are only two random variables, say   and  . Then, we have

 
Similarly, in general case, we have
 
Then, we perform similar process on each of the other variables (  left), with one extra summation sign added for each process. Thus, in total we will have   summation sign, and we will finally get the desired result.  

Remark. This process may sometimes be called 'summing over each possible value of other variables'.

Example. Suppose we throw a fair six-faced dice two times. Let   be the number facing up in the first throw, and   be the number facing up in the second throw. Then, the joint pmf of   is

 
in which  , and   otherwise. Also, the marginal pmf of   is
 
in which  , and   otherwise.

By symmetry (replace all   with   and replace all   with  ), the marginal pmf of   is

 
in which  , and   otherwise.

Exercise. Suppose there are two red balls and one blue ball in a box, and we draw two balls one by one from the box with replacement. Let   if the ball from the first draw is red, and   otherwise. Let   if the ball from the second draw is red, and   otherwise.

1 Calculate the marginal pmf of  .

 
 
 
 

2 Calculate the joint pmf of  .

 )
 )
 )


Exercise. Recall the example in the motivation section.

(a) Suppose we toss a fair coin twice. Let   and  . Show that joint pmf of   is

 

(b) Suppose we toss a fair coin once. Let   and  . Show that joint pmf of   is

 

(c) Show that marginal pmf of   and   are

 
in each of the situations in (a) and (b). (Hint: for part (b), we need to put value in the variable in the indicator)


Proof.

(a) Since the support of   is  , the joint pmf of   is

 

(b) Since the support of   is  , the joint pmf of   is

 

(c) Part (a): marginal pmf of   is

 
and marginal pmf of   is
 

Part (b): The marginal pmf of   is

 
Similarly, the marginal pmf of   is
 

For jointly continuous random variables, the definition is generalized version of the one for continuous random variables (univariate case).

Definition. (Jointly continuous random variable) Random variables   are jointly continuous if

 
for some nonnegative function  .

Remark.

  • The function   is the joint probability density function (joint pdf) of  .
  • Similarly,   can be interpreted as the probability over the 'infinitesimal' region  , and   can be interpreted as the density of the probability over that 'infinitesimal' region, i.e.  , intuitively and non-rigourously.
  • By setting  , the cdf

 
which is similar to the univariate case.

Definition. (Marginal probability density function) The pdf   of each   which is a member of the   random variables   is the marginal probability density function (marginal pdf) of  .

Proposition. (Obtaining marginal pdf from joint pdf) For continuous random variables   with joint pdf  , the marginal pdf of   is

 

Proof. Recall the proposition about obtaining marginal cdf from joint cdf. We have

 

 

Proposition. (Obtaining joint pdf from joint cdf) If a joint cdf   of jointly continuous random variables has each partial deriviative at  , then the joint pdf is

 

Proof. It follows from using fundamental theorem of calculus   times.

 

Example. If the joint pdf of jointly continuous random variable   is

 
the marginal pdf of   is
 
Also,
 

Exercise. Let   and   be jointly continuous random variables. Consider the joint cdf of  :

 

1 Calculate the joint pdf of  .

 
 
 
 

2 Calculate the marginal pdf of  .

 
 
 
 



IndependenceEdit

Recall that multiple events are independent if the probability for the intersection of them equals the product of probabilities of each event, by definition. Since   is also an event, we have a natural definition of independence for random variables as follows:

Definition. (Independence of random variables) Random variables   are independent if

 
for each   and for each subset  .

Remark. Under this condition, the events   are independent.

Theorem. (Alternative condition for independence of random variables) Random variables   are independent if and only if the joint cdf of  

 
or the joint pdf or pmf of  
 
for each  .

Proof. Partial:

Only if part: If random variables   are independent,

 
for each   and for each subset  . Setting  , and we have
 
Thus, we obtain the result for the joint cdf part.

For the joint pdf part,

 

 

Remark.

  • That is, if joint cdf (joint pdf (pmf)) can be factorized as the product of marginal cdf's (marginal pdf's (pmf's))
  • Actually, if we can factorize the joint cdf or joint pdf or joint pmf as the product of some functions in each of the variables, then the condition is also satisfied.

Example. The joint pdf of two independent exponential random variables with rate  ,   and   is

 
(Random variables   and   are said to be independent and identically distributed (i.i.d.) in this case)

In general, the joint pdf of   independent exponential random variables with rate  ,   is

 
(Random variables   are also i.i.d. in this case)

On the other hand, if the joint pdf of two random variables   and   are

 
random variables   and   are dependent since the joint pdf cannot be factorized as the product of marginal pdf's.

Exercise. Let   be jointly continuous random variables. Consider a joint pdf of  :

 

1 Calculate  .

1
2
3
4

2 Are   independent?

yes
no


Consider another joint pdf of  :

 

1 Calculate  .

1
2
3
4

2 Are   independent?

yes
no


Consider another joint pdf of  :

 

1 Calculate  .

1
2
3
4

2 Are   independent?

yes
no


Proposition. (Independence of events concerning disjoint sets of independent random variables) Suppose random variables   are independent. Then, for each   and fixed functions  , the random variables

 
are independent.

Example. Suppose   are independent Bernoulli random variables with success probability  . Then,   and   are also independent.

On the other hand,   and   are not independent. A counter-example to the independence is

 
Left hand side equals zero since  , but  .

Right hand side may not equal zero since  , and  . We can see that   may not equal zero.

Exercise.

Let   be i.i.d. random variables, and   also be i.i.d. random variables. Which of the following is (are) true?

  and   are independent.
  and   are independent.
  and   are independent.
  and   are independent if   are independent.


Sum of independent random variables (optional)Edit

In general, we use joint cdf, pdf or pmf to determine the distribution of sum of independent random variables by first principle. In particular, there are some interesting results related to the distribution of sum of independent random variables.

Order statisticsEdit

Definition. (Order statistics) Let   be   i.i.d. r.v.'s (each with cdf  ). Define   be the smallest, second smallest, ..., largest of  . Then, the ordered values   is the order statistics.

Proposition. (Cdf of order statistics) The cdf of   (  is an integer such that  ) is

 

Proof.

  • Consider the event  .
                          Possible positions of x
                      |<--------------------->
    *---*----...------*----*------...--------*
X  (1)  (2)          (k)  (k+1)             (n)
                      |----------------------> when x moves RHS like this, >=k X_i are at the LHS of x
  • We can see from the above figure that  .
  • Let no. of  's that are less than or equal to   be  .
  • Since   (because for each  , we can treat   and   be the two outcomes in a Bernoulli trial),
  • The cdf is

 

 

Example. Let   be i.i.d. r.v.'s following  . Then, the cdf of   is

 

Exercise.

Calculate  .

0.000665
0.000994
0.036296
0.963704
0.999335




Poisson processEdit

Definition.

 
Illustration of Poisson process. Each circle indicates one arrival. The arrivals occur at common rate  , and the successive interarrivial times are independent.

If successive interarrival times of unpredictable events are independent random variables, with each following an exponential distribution with a common rate  , then the process of arrivals is a Poisson process with rate  .

There are several important properties for Poisson process.

Proposition. (Time to  -th event in Poisson process) The time to  -th event in a Poisson process follows the   distribution.

Proof.

  • The time to  -th event is  , with each following  .
  • It suffices to prove that  , and then the desired result follows by induction.
  •  
which is the pdf of  , as desired.

 

Remark. The time to  -th event is also the sum of the   successive interarrival times before the  -th event.

Proposition. (Number of arrivals within a fixed time interval) The number of arrivals within a fixed time interval of length   follows the   distribution.

Proof. For each nonnegative integer  , let   be the interarrival time between the  -th and  -th arrival, and   be the time to  th event, starting from the beginning of the fixed time interval (we can treat the start to be time zero because of the memoryless property). The joint pdf of   is

 
Let   the number of arrivals within the fixed time interval. The pmf of   is
 
which is the pmf of  . The result follows.

 

Proposition. (Time to the first arrival with   independent Poisson processes) Let   be independent random variables with  , in which  . If we define   (which is the time to the first arrival with   independent Poisson processes), then  .

Proof. For each  ,

 

 

Example. Suppose there are two service counters, counter A and B, with independent service times following the exponential distribution with rate  . In the past 10 minutes, John and Peter are being served at counter A and B respectively.

First, the time you need to wait to be served (i.e. the time for one of John and Peter leaves the counter) is the minimum value of the service time for John and Peter counting from now, which are independent and follow the exponential distribution with rate  . Thus, your waiting time follows the exponential distribution with rate  .

Suppose now John leaves the counter A, and you are currently being served at counter A. Then, the probability that you leave the counter first is  , by memoryless property and symmetry (the chances that Peter and you leave the counter first are governed by the same chance mechanism), counterintuitively.

Exercise. Suppose the process of arrivals of car accidents is a Poisson process with unit rate. Let   be the time to the  -th car accidents, and   be the interarrival time between the  -th and  -th accidents.

1 Which of the following is (are) true?

 
 
 
 
 

2 Which of the following is (are) true?

 
 
 
 

3 Which of the following is (are) true?

 
 
 
The pmf of the number of arrivals within a fixed time interval of length   is  .