Theorem : Cauchy-Schwarz Inequality

$E[XY]^2 \leq E[X^2] E[Y^2]$

Example

$E[(X - \frac{ E[XY] }{ E[Y^2] })^2] \geq 0$

Proof [expand]
Definition: Correlation Coefficient

$\rho(X, Y) = \frac{ cov(X, Y)}{ \sqrt{ var(X)var(Y)}}$

Prop. $-1 \leq \rho(X, Y) \leq 1 \iff \rho(X, Y)^2 \leq 1$

$\rho(X, Y)^2 = \frac{ cov(X,Y)^2 }{ var(X)var(Y)} = \frac{ E[(X - EX)(Y - EY)]^2 }{ E[(X-EX)^2]E[(Y-EY)^2] } \leq 1$ By Cauchy-Schwarz

Recall, for $\overline{ Y } = a \overline{ X } \pare{\text{linear relationship}} \implies \rho(X, Y) = \pm 1 = sgn(a)$

Converse If $\rho(X, Y)^2 = 1$.

$E[(\overline{ X } - \frac{ E[ \overline{ X } \overline{ Y } ] }{ E[Y^2] } \overline{ Y })^2] = E[ \overline{ X } ^2 ]$

Definition: Conditional Expectation

Discrete $E[X|Y = y] = \sum_{ x }^{ ‘ } x P(X = x | Y = y) = \sum_{ x } x \frac{ P(X = x, Y = y)}{ P(Y=y)}$

continuous $E[X|Y = y] = \int_{ -\infty }^{ +\infty } x f_{X|Y} (x|y) \d x = \int_{ -\infty }^{ +\infty } \frac{ f_{X,Y}(x,y)}{ f_Y(y)} \d x$

Conditional expectation of $X$ given $Y$, $E[X | Y]$ as follows: let $g(y) = E’[X | Y = y]$ function of $y$ then $E[X|Y] = g(Y)$

Example

$Y$-uniform on $[0, 1]$. Take $X$-uniform on $[0, Y]$.

$\implies E[X|Y = y] = \frac{ y }{ 2 } \implies E[X | Y] = \frac{ y }{ 2 }$

$E[X] = E[E[X|Y]] = E[ \frac{ Y }{ 2 } ] = \frac{ 1 }{ 2 } E[Y] = \frac{ 1 }{ 2 } \frac{ 1 }{ 2 } = \frac{ 1 }{ 4 }$

Recall Total Expectation Theorem

$$E[X] = \sum_{ y } E[X|Y = y]p_X(y)$$ discrete $$E[X] = \int_{ \R } \d E[X|Y = y]f_X(y) \d y$$ discrete

Theorem : Law of Iterated Expectation

$E[E[X|Y]] = E[X]$

Definition: Conditional Variance

$$var(X|Y = y) = E[(X - E[X|Y])^2 | Y = y] = g(y)$$

Then $var(X|Y) = g(Y)$

Theorem : Law of Total Variance

$var(X) = var(X|Y) = \frac{ Y^2 }{ 12 }$, $Y$ uniform $(0,1]$, $X$ uniform $(0, Y]$