$\newcommand{\br}{\\}$ $\newcommand{\R}{\mathbb{R}}$ $\newcommand{\Q}{\mathbb{Q}}$ $\newcommand{\Z}{\mathbb{Z}}$ $\newcommand{\N}{\mathbb{N}}$ $\newcommand{\C}{\mathbb{C}}$ $\newcommand{\P}{\mathbb{P}}$ $\newcommand{\F}{\mathbb{F}}$ $\newcommand{\L}{\mathcal{L}}$ $\newcommand{\spa}[1]{\text{span}(#1)}$ $\newcommand{\set}[1]{\{#1\}}$ $\newcommand{\emptyset}{\varnothing}$ $\newcommand{\otherwise}{\text{ otherwise }}$ $\newcommand{\if}{\text{ if }}$ $\newcommand{\proj}{\text{proj}}$ $\newcommand{\union}{\cup}$ $\newcommand{\intercept}{\cap}$ $\newcommand{\abs}[1]{\left | #1 \right |}$ $\newcommand{\norm}[1]{\left\lVert#1\right\rVert}$ $\newcommand{\pare}[1]{\left (#1 \right)}$ $\newcommand{\t}[1]{\text{ #1 }}$ $\newcommand{\head}{\text H}$ $\newcommand{\tail}{\text T}$ $\newcommand{\d}{\text d}$ $\newcommand{\limu}[2]{\underset{#1 \to #2}\lim}$ $\newcommand{\inv}[1]{{#1}^{-1}}$ $\newcommand{\inner}[2]{\langle #1, #2 \rangle}$ $\newcommand{\nullity}[1]{\text{nullity}(#1)}$ $\newcommand{\rank}[1]{\text{rank}(#1)}$ $\newcommand{\var}[1]{\text{var}(#1)}$ $\newcommand{\tr}[1]{\text{tr}(#1)}$ $\newcommand{\oto}{\text{ one-to-one }}$ $\newcommand{\ot}{\text{ onto }}$ $\newcommand{\Re}[1]{\text{Re}(#1)}$ $\newcommand{\Im}[1]{\text{Im}(#1)}$ $\newcommand{\Vcw}[2]{\begin{pmatrix} #1 \br #2 \end{pmatrix}}$ $\newcommand{\Vce}[3]{\begin{pmatrix} #1 \br #2 \br #3 \end{pmatrix}}$ $\newcommand{\Vcr}[4]{\begin{pmatrix} #1 \br #2 \br #3 \br #4 \end{pmatrix}}$ $\newcommand{\Vct}[5]{\begin{pmatrix} #1 \br #2 \br #3 \br #4 \br #5 \end{pmatrix}}$ $\newcommand{\Vcy}[6]{\begin{pmatrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \end{pmatrix}}$ $\newcommand{\Vcu}[7]{\begin{pmatrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \br #7 \end{pmatrix}}$ $\newcommand{\vcw}[2]{\begin{matrix} #1 \br #2 \end{matrix}}$ $\newcommand{\vce}[3]{\begin{matrix} #1 \br #2 \br #3 \end{matrix}}$ $\newcommand{\vcr}[4]{\begin{matrix} #1 \br #2 \br #3 \br #4 \end{matrix}}$ $\newcommand{\vct}[5]{\begin{matrix} #1 \br #2 \br #3 \br #4 \br #5 \end{matrix}}$ $\newcommand{\vcy}[6]{\begin{matrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \end{matrix}}$ $\newcommand{\vcu}[7]{\begin{matrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \br #7 \end{matrix}}$ $\newcommand{\Mqw}[2]{\begin{bmatrix} #1 & #2 \end{bmatrix}}$ $\newcommand{\Mqe}[3]{\begin{bmatrix} #1 & #2 & #3 \end{bmatrix}}$ $\newcommand{\Mqr}[4]{\begin{bmatrix} #1 & #2 & #3 & #4 \end{bmatrix}}$ $\newcommand{\Mqt}[5]{\begin{bmatrix} #1 & #2 & #3 & #4 & #5 \end{bmatrix}}$ $\newcommand{\Mwq}[2]{\begin{bmatrix} #1 \br #2 \end{bmatrix}}$ $\newcommand{\Meq}[3]{\begin{bmatrix} #1 \br #2 \br #3 \end{bmatrix}}$ $\newcommand{\Mrq}[4]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \end{bmatrix}}$ $\newcommand{\Mtq}[5]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \br #5 \end{bmatrix}}$ $\newcommand{\Mqw}[2]{\begin{bmatrix} #1 & #2 \end{bmatrix}}$ $\newcommand{\Mwq}[2]{\begin{bmatrix} #1 \br #2 \end{bmatrix}}$ $\newcommand{\Mww}[4]{\begin{bmatrix} #1 & #2 \br #3 & #4 \end{bmatrix}}$ $\newcommand{\Mqe}[3]{\begin{bmatrix} #1 & #2 & #3 \end{bmatrix}}$ $\newcommand{\Meq}[3]{\begin{bmatrix} #1 \br #2 \br #3 \end{bmatrix}}$ $\newcommand{\Mwe}[6]{\begin{bmatrix} #1 & #2 & #3\br #4 & #5 & #6 \end{bmatrix}}$ $\newcommand{\Mew}[6]{\begin{bmatrix} #1 & #2 \br #3 & #4 \br #5 & #6 \end{bmatrix}}$ $\newcommand{\Mee}[9]{\begin{bmatrix} #1 & #2 & #3 \br #4 & #5 & #6 \br #7 & #8 & #9 \end{bmatrix}}$
Theorem : Cauchy-Schwarz Inequality

$E[XY]^2 \leq E[X^2] E[Y^2]$

Example

$E[(X - \frac{ E[XY] }{ E[Y^2] })^2] \geq 0$

Proof [expand]
Definition: Correlation Coefficient

$\rho(X, Y) = \frac{ cov(X, Y)}{ \sqrt{ var(X)var(Y)}} $

Prop. $-1 \leq \rho(X, Y) \leq 1 \iff \rho(X, Y)^2 \leq 1$

$\rho(X, Y)^2 = \frac{ cov(X,Y)^2 }{ var(X)var(Y)} = \frac{ E[(X - EX)(Y - EY)]^2 }{ E[(X-EX)^2]E[(Y-EY)^2] } \leq 1$ By Cauchy-Schwarz

Recall, for $\overline{ Y } = a \overline{ X } \pare{\text{linear relationship}} \implies \rho(X, Y) = \pm 1 = sgn(a)$

Converse If $\rho(X, Y)^2 = 1 $.

$E[(\overline{ X } - \frac{ E[ \overline{ X } \overline{ Y } ] }{ E[Y^2] } \overline{ Y })^2] = E[ \overline{ X } ^2 ]$

Definition: Conditional Expectation

Discrete $E[X|Y = y] = \sum_{ x }^{ ‘ } x P(X = x | Y = y) = \sum_{ x } x \frac{ P(X = x, Y = y)}{ P(Y=y)}$

continuous $E[X|Y = y] = \int_{ -\infty }^{ +\infty } x f_{X|Y} (x|y) \d x = \int_{ -\infty }^{ +\infty } \frac{ f_{X,Y}(x,y)}{ f_Y(y)} \d x$

Conditional expectation of $X $ given $Y $, $E[X | Y]$ as follows: let $g(y) = E’[X | Y = y] $ function of $y $ then $E[X|Y] = g(Y)$

Example

$Y$-uniform on $[0, 1] $. Take $X $-uniform on $[0, Y] $.

$\implies E[X|Y = y] = \frac{ y }{ 2 } \implies E[X | Y] = \frac{ y }{ 2 }$

$E[X] = E[E[X|Y]] = E[ \frac{ Y }{ 2 } ] = \frac{ 1 }{ 2 } E[Y] = \frac{ 1 }{ 2 } \frac{ 1 }{ 2 } = \frac{ 1 }{ 4 }$

Recall Total Expectation Theorem

$$ E[X] = \sum_{ y } E[X|Y = y]p_X(y) $$ discrete $$ E[X] = \int_{ \R } \d E[X|Y = y]f_X(y) \d y $$ discrete

Theorem : Law of Iterated Expectation

$E[E[X|Y]] = E[X]$

Definition: Conditional Variance

$$ var(X|Y = y) = E[(X - E[X|Y])^2 | Y = y] = g(y) $$

Then $ var(X|Y) = g(Y) $

Theorem : Law of Total Variance

$var(X) = var(X|Y) = \frac{ Y^2 }{ 12 }$, $Y$ uniform $(0,1]$, $X$ uniform $(0, Y]$