$\newcommand{\br}{\\}$ $\newcommand{\R}{\mathbb{R}}$ $\newcommand{\Q}{\mathbb{Q}}$ $\newcommand{\Z}{\mathbb{Z}}$ $\newcommand{\N}{\mathbb{N}}$ $\newcommand{\C}{\mathbb{C}}$ $\newcommand{\P}{\mathbb{P}}$ $\newcommand{\F}{\mathbb{F}}$ $\newcommand{\L}{\mathcal{L}}$ $\newcommand{\spa}[1]{\text{span}(#1)}$ $\newcommand{\dist}[1]{\text{dist}(#1)}$ $\newcommand{\max}[1]{\text{max}(#1)}$ $\newcommand{\min}[1]{\text{min}(#1)}$ $\newcommand{\supr}[1]{\text{sup}(#1)}$ $\newcommand{\infi}[1]{\text{inf}(#1)}$ $\newcommand{\ite}[1]{\text{int}(#1)}$ $\newcommand{\ext}[1]{\text{ext}(#1)}$ $\newcommand{\bdry}[1]{\partial #1}$ $\newcommand{\argmax}[1]{\underset{#1}{\text{argmax }}}$ $\newcommand{\argmin}[1]{\underset{#1}{\text{argmin }}}$ $\newcommand{\set}[1]{\left\{#1\right\}}$ $\newcommand{\emptyset}{\varnothing}$ $\newcommand{\tilde}{\text{~}}$ $\newcommand{\otherwise}{\text{ otherwise }}$ $\newcommand{\if}{\text{ if }}$ $\newcommand{\proj}{\text{proj}}$ $\newcommand{\union}{\cup}$ $\newcommand{\intercept}{\cap}$ $\newcommand{\abs}[1]{\left| #1 \right|}$ $\newcommand{\norm}[1]{\left\lVert#1\right\rVert}$ $\newcommand{\pare}[1]{\left(#1\right)}$ $\newcommand{\brac}[1]{\left[#1\right]}$ $\newcommand{\t}[1]{\text{ #1 }}$ $\newcommand{\head}{\text H}$ $\newcommand{\tail}{\text T}$ $\newcommand{\d}{\text d}$ $\newcommand{\limu}[2]{\underset{#1 \to #2}\lim}$ $\newcommand{\der}[2]{\frac{\d #1}{\d #2}}$ $\newcommand{\derw}[2]{\frac{\d #1^2}{\d^2 #2}}$ $\newcommand{\pder}[2]{\frac{\partial #1}{\partial #2}}$ $\newcommand{\pderw}[2]{\frac{\partial^2 #1}{\partial #2^2}}$ $\newcommand{\pderws}[3]{\frac{\partial^2 #1}{\partial #2 \partial #3}}$ $\newcommand{\inv}[1]{{#1}^{-1}}$ $\newcommand{\inner}[2]{\langle #1, #2 \rangle}$ $\newcommand{\nullity}[1]{\text{nullity}(#1)}$ $\newcommand{\rank}[1]{\text{rank }#1}$ $\newcommand{\nullspace}[1]{\mathcal{N}\pare{#1}}$ $\newcommand{\range}[1]{\mathcal{R}\pare{#1}}$ $\newcommand{\var}[1]{\text{var}\pare{#1}}$ $\newcommand{\cov}[2]{\text{cov}(#1, #2)}$ $\newcommand{\tr}[1]{\text{tr}(#1)}$ $\newcommand{\oto}{\text{ one-to-one }}$ $\newcommand{\ot}{\text{ onto }}$ $\newcommand{\ceil}[1]{\lceil#1\rceil}$ $\newcommand{\floor}[1]{\lfloor#1\rfloor}$ $\newcommand{\Re}[1]{\text{Re}(#1)}$ $\newcommand{\Im}[1]{\text{Im}(#1)}$ $\newcommand{\dom}[1]{\text{dom}(#1)}$ $\newcommand{\fnext}[1]{\overset{\sim}{#1}}$ $\newcommand{\transpose}[1]{{#1}^{\text{T}}}$ $\newcommand{\b}[1]{\boldsymbol{#1}}$ $\newcommand{\None}[1]{}$ $\newcommand{\Vcw}[2]{\begin{bmatrix} #1 \br #2 \end{bmatrix}}$ $\newcommand{\Vce}[3]{\begin{bmatrix} #1 \br #2 \br #3 \end{bmatrix}}$ $\newcommand{\Vcr}[4]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \end{bmatrix}}$ $\newcommand{\Vct}[5]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \br #5 \end{bmatrix}}$ $\newcommand{\Vcy}[6]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \end{bmatrix}}$ $\newcommand{\Vcu}[7]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \br #7 \end{bmatrix}}$ $\newcommand{\vcw}[2]{\begin{matrix} #1 \br #2 \end{matrix}}$ $\newcommand{\vce}[3]{\begin{matrix} #1 \br #2 \br #3 \end{matrix}}$ $\newcommand{\vcr}[4]{\begin{matrix} #1 \br #2 \br #3 \br #4 \end{matrix}}$ $\newcommand{\vct}[5]{\begin{matrix} #1 \br #2 \br #3 \br #4 \br #5 \end{matrix}}$ $\newcommand{\vcy}[6]{\begin{matrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \end{matrix}}$ $\newcommand{\vcu}[7]{\begin{matrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \br #7 \end{matrix}}$ $\newcommand{\Mqw}[2]{\begin{bmatrix} #1 & #2 \end{bmatrix}}$ $\newcommand{\Mqe}[3]{\begin{bmatrix} #1 & #2 & #3 \end{bmatrix}}$ $\newcommand{\Mqr}[4]{\begin{bmatrix} #1 & #2 & #3 & #4 \end{bmatrix}}$ $\newcommand{\Mqt}[5]{\begin{bmatrix} #1 & #2 & #3 & #4 & #5 \end{bmatrix}}$ $\newcommand{\Mwq}[2]{\begin{bmatrix} #1 \br #2 \end{bmatrix}}$ $\newcommand{\Meq}[3]{\begin{bmatrix} #1 \br #2 \br #3 \end{bmatrix}}$ $\newcommand{\Mrq}[4]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \end{bmatrix}}$ $\newcommand{\Mtq}[5]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \br #5 \end{bmatrix}}$ $\newcommand{\Mqw}[2]{\begin{bmatrix} #1 & #2 \end{bmatrix}}$ $\newcommand{\Mwq}[2]{\begin{bmatrix} #1 \br #2 \end{bmatrix}}$ $\newcommand{\Mww}[4]{\begin{bmatrix} #1 & #2 \br #3 & #4 \end{bmatrix}}$ $\newcommand{\Mqe}[3]{\begin{bmatrix} #1 & #2 & #3 \end{bmatrix}}$ $\newcommand{\Meq}[3]{\begin{bmatrix} #1 \br #2 \br #3 \end{bmatrix}}$ $\newcommand{\Mwe}[6]{\begin{bmatrix} #1 & #2 & #3\br #4 & #5 & #6 \end{bmatrix}}$ $\newcommand{\Mew}[6]{\begin{bmatrix} #1 & #2 \br #3 & #4 \br #5 & #6 \end{bmatrix}}$ $\newcommand{\Mee}[9]{\begin{bmatrix} #1 & #2 & #3 \br #4 & #5 & #6 \br #7 & #8 & #9 \end{bmatrix}}$
Definition: Cumulative Distribution Function (CDF)

The cumulative distribution function (CDF) of an random variable $X $ is denoted $F_X $ and is defined as:

$$F_X(x)=\b{P}(X \leq X) = \b{P}(\set{ \omega \in \Omega \mid X(\omega) \leq x})$$

Properties

Let $F $ be a cumulative distribution function (CDF), $f $ be the corresponding probability distribution function (PDF).

  1. $F $ is non-decreasing.
  2. $\limu{x}{ -\infty } F(x) = 0$.
  3. $\limu{x}{ +\infty } F(x) =1$.
  4. If $X$ is discrete, then $X$ is piecewise constant and continuous from the left.
  5. If $X$ is continuous and $f_X$ is continuous, then $F$ is differentiable.
  6. If $X$ is continuous, $F'_X(x) = f_X(x)$.
  7. if $X$ is discrete and $X$ is integer-valued, then

$$\begin{align*} \b{P}(X=n) &= \b{P}(X \leq n) - \b{P}(X \leq n-1) \br &= F_X(n) - F_X(n-1) \end{align*}$$

Definition: Uniform Distribution
Definition: Exponential Distribution
Definition: Normal (Gaussian) Distribution

$$f_X(x) = \frac{1}{\sqrt{2\pi}{ \sigma }}e^{-\frac{(x-{ \mu })^2}{2{ \sigma }^2}}$$

Example

For any $B \subseteq R $, 2 e.g. condition on value of $X$.

$ f_{X \mid \set{ X \in E }}(x) = \begin{cases} \frac{ f_X(x)}{ \b{P}(x \in E)}, \if x \in E \br 0, \otherwise \end{cases} $

Example

The time $T$ until a call comes into a call center is modeled by an exponential random variable with parameter $\lambda > 0$, you turn on phone, leave for $t$ units of time and return. The phone didn’t ring that time. Let $X$ be additional time until phone rings. Let’s find CDF of $X$ given $T>t$,

$$f_T(s) = \begin{cases} e^{- \lambda s}, s \geq 0 \br 0 , \otherwise \end{cases} $$

$$f_T(s) = \begin{cases} 1-e^{- \lambda s}, s \geq 0 \br 0 , \otherwise \end{cases} $$

We seek $F_{X| T > t} (x) = \b{P}(X \leq x | T > t)$

Note: $t + X = T = \b{P}(T - t \leq x | T > t) $

$$\begin{align*} F_{X| T > t} (x) &= \b{P}(X \leq x | T > t) \br &= 1- \b{P}(X > x | T > t) \br &= 1 - \b{P}(T > x + t | T > t) \br &= 1 - \b{P}(T > x + t | T > t) \br &= 1 - \frac{ \b{P}(T > x + t \cap T > t)}{ \b{P}(T > t)} \br &= 1 - \frac{ P (T > x + t)}{ \b{P}(T > t)} \br &= 1 - \frac{ e^{- \lambda (x + t)}}{ e^{- \lambda t}} \br &= 1 - e^{- \lambda x } \end{align*}$$

This is the CDF of an exponential with parameter $\lambda$ !

Note: doesn’t depend on $t$. This property is special to exponential.

This property is special to exponential, and is called “memoryless”.

Recall: By definition, $E[X | A] = \int_{ -\infty }^{ +\infty } x f_X|A(x) \d x $

Theorem : Total Probability Theorem

Let $A_1, … , A_n$ be a partition of sample space. Then,

1.$f_X(x) = \sum_{ i=1 }^{ n } f_{X \mid A_i}(x) \b{P}(A_i)$

Example

Your arrival time at train stop is uniformly distributed between 7:10 to 7:30am. Trains arrive at 7:15am and 7:30am. Let $y $ be amount of time you wait for train. Let’s find PDF of $Y $ and $E[Y]$.

Let $A $ be the event that you arrive between 7:10 & 7:15am.

$f_Y(y) = f_{y \mid A}(y)\b{P}(A) + f_{y \mid A^c}(y)\b{P}(A^c)$

$f_\set{Y|A}$ is uniform in $[0, 5]$.

$f_\set{Y|A^c}$ is uniform in $[0, 15]$.

so $f_{y|A}(Y) = \begin{cases} \frac{ 1 }{ 5 }, \if y \in[0, 5] \br 0, \otherwise \end{cases}$

so $f_{y|A^c}(Y) = \begin{cases} \frac{ 1 }{ 15 }, \if y \in[0, 15] \br 0, \otherwise \end{cases}$

$\b{P}(A) = \frac{ 5 }{ 20 } = \frac{ 1 }{ 4 }$

For, $y \in [0, 5]$,

$f_Y(y) = \frac{ 1 }{ 5 } \cdot \frac{ 1 }{ 4 } + \frac{ 1 }{ 15 } \frac{ 3 }{ 4 } = \frac{ 1 }{ 10 }$

For, $y \in (5, 15]$,

$f_Y(y) = 0 + \frac{ 1 }{ 15 } \frac{ 3 }{ 4 } = \frac{ 1 }{ 20 }$

so

$$ f_Y(y) = \begin{cases} \frac{ 1 }{ 10 }, y \in [ 0, 5 ] \br \frac{ 1 }{ 20 }, y \in [5, 15] 0, \otherwise \end{cases} $$

Theorem : Total Expectation Theorem

Let $A_1, …, A_n$ partitions sample space. Then,

$E[X] \sum_{ i=1 }^{ n } E[X|A_i] \b{P}(A)$

Proof
  </span>
</span>
<span class="proof__expand"><a>[expand]</a></span>

Integrate the equality in total expectation theorem.

$\b{P}(X \leq x) = \sum_{ i=1 }^{ n }\b{P}(X \leq x | A_i)\b{P}(A_i)$

$\begin{align*} \int_{ -fin }^{ x } f_X(y) \d y = \sum_{ i=1 }^{ n } \int_{ -\infty }^{ x } f_{X \mid A}(y)\b{P}(A) \d y \end{align*}$

take d/dx to get 1.

Example: continued

$$\begin{align*} F[y] &= E[Y|A]\b{P}(A) + E[Y|A^c]\b{P}(A^c) \br &= \frac{ 5 }{ 2 } \frac{ 1 }{ 4 } + \frac{ 15 }{ 2 } \frac{ 3 }{ 4 } \br &= \frac{ 50 }{ 8 } \br &= \frac{ 25 }{ 4 } \end{align*}$$

Example

Let $X, Y $ be continuous random variables. Want to condition $X $ on $Y = y$(say) that $\b{P}(Y = y) = 0$ for any $y$. How to do this?

Definition

For any $y$ with $f_Y(y) > 0$. The conditional PDF of $X$ given $Y=y$ is

$$f_{X \mid Y}(x \mid y) = \frac{f_{X,Y}(x,y)}{ f_Y(y)}$$

To justify this, can consider for $\delta \approx 0 $

$$\b{P}(x \leq X \leq x + \delta \mid y \leq Y \leq y+ \delta) $$

Apply old definition, and take $\delta \to 0$.

Theorem : Total Probability Theorem

$f_X(x) = \int_{ -\infty }^{ +\infty } f_{X \mid Y}(x|y)f_Y(y) \d y$

Proof
  </span>
</span>
<span class="proof__expand"><a>[expand]</a></span>

$$\begin{align*} f_X(x) &= \int_{ -\infty }^{ +\infty } f_{X, Y}(x, y) \d y \br &= \int_{ -\infty }^{ +\infty } f_{X \mid Y}(x \mid y) f_Y(y) \d y\ (definition) \end{align*}$$

  • In examples or modeling, it can be natural to define one random variable and then describe a second one in terms of the first.

i.e. know $f_X, f_{y|X}$.

Can use this to find $f_{X,Y}, f_y, etc. $

e.g. The speed of a typical vehicle driving put a police officer is modeled by an exponential random variable $X$ with mean $50$ mph.

The radar’s measurement of the speed $y$ has an error that’s normal with mean $0 $ and speed equal to $\frac{ 1 }{ 10 }$ of vehicle’s speed. Find $f_{X, Y}$(From this case find $f_Y$)

We see if speed is $x $, then $y = x + N$

note:

$x+N $ is also normal, with mean $x$ and variance (= speed^2)

$$f_{Y \mid X}(y \mid x) = \frac{ 1 }{ \sqrt{2 \pi} \frac{ x }{ 10 }} e^{ \frac{ -(y-x)^2 }{ x^2 /50 }}$$

Since $X $ exponential means,

$$ f_X(x) = \begin{cases} \lambda e^{- \lambda x}a, \if x \geq o \br 0, \otherwise \end{cases} $$

In our situation, $E[X] = 50$, hence, $\frac{ 1 }{ \lambda } = 50$ and $\lambda = \frac{ 1 }{ 50 }$

so $f_X(x) = \begin{cases} \frac{ 1 }{ 50 }e^{-x /50}, \if x \geq 0 \br 0, \otherwise \end{cases}

f_{X, Y}(x,y) = \frac{ 1 }{ sqrt{2 \pi} \frac{ x }{ 10 }} e^{-50 (y-x)^2 / x^2} \frac{ 1 }{ 50 } $