$\newcommand{\br}{\\}$ $\newcommand{\R}{\mathbb{R}}$ $\newcommand{\Q}{\mathbb{Q}}$ $\newcommand{\Z}{\mathbb{Z}}$ $\newcommand{\N}{\mathbb{N}}$ $\newcommand{\C}{\mathbb{C}}$ $\newcommand{\P}{\mathbb{P}}$ $\newcommand{\F}{\mathbb{F}}$ $\newcommand{\L}{\mathcal{L}}$ $\newcommand{\spa}[1]{\text{span}(#1)}$ $\newcommand{\dist}[1]{\text{dist}(#1)}$ $\newcommand{\max}[1]{\text{max}(#1)}$ $\newcommand{\min}[1]{\text{min}(#1)}$ $\newcommand{\supr}[1]{\text{sup}(#1)}$ $\newcommand{\infi}[1]{\text{inf}(#1)}$ $\newcommand{\ite}[1]{\text{int}(#1)}$ $\newcommand{\ext}[1]{\text{ext}(#1)}$ $\newcommand{\bdry}[1]{\partial #1}$ $\newcommand{\argmax}[1]{\underset{#1}{\text{argmax }}}$ $\newcommand{\argmin}[1]{\underset{#1}{\text{argmin }}}$ $\newcommand{\set}[1]{\left\{#1\right\}}$ $\newcommand{\emptyset}{\varnothing}$ $\newcommand{\tilde}{\text{~}}$ $\newcommand{\otherwise}{\text{ otherwise }}$ $\newcommand{\if}{\text{ if }}$ $\newcommand{\proj}{\text{proj}}$ $\newcommand{\union}{\cup}$ $\newcommand{\intercept}{\cap}$ $\newcommand{\abs}[1]{\left| #1 \right|}$ $\newcommand{\norm}[1]{\left\lVert#1\right\rVert}$ $\newcommand{\pare}[1]{\left(#1\right)}$ $\newcommand{\brac}[1]{\left[#1\right]}$ $\newcommand{\t}[1]{\text{ #1 }}$ $\newcommand{\head}{\text H}$ $\newcommand{\tail}{\text T}$ $\newcommand{\d}{\text d}$ $\newcommand{\limu}[2]{\underset{#1 \to #2}\lim}$ $\newcommand{\der}[2]{\frac{\d #1}{\d #2}}$ $\newcommand{\derw}[2]{\frac{\d #1^2}{\d^2 #2}}$ $\newcommand{\pder}[2]{\frac{\partial #1}{\partial #2}}$ $\newcommand{\pderw}[2]{\frac{\partial^2 #1}{\partial #2^2}}$ $\newcommand{\pderws}[3]{\frac{\partial^2 #1}{\partial #2 \partial #3}}$ $\newcommand{\inv}[1]{{#1}^{-1}}$ $\newcommand{\inner}[2]{\langle #1, #2 \rangle}$ $\newcommand{\nullity}[1]{\text{nullity}(#1)}$ $\newcommand{\rank}[1]{\text{rank }#1}$ $\newcommand{\nullspace}[1]{\mathcal{N}\pare{#1}}$ $\newcommand{\range}[1]{\mathcal{R}\pare{#1}}$ $\newcommand{\var}[1]{\text{var}\pare{#1}}$ $\newcommand{\cov}[2]{\text{cov}(#1, #2)}$ $\newcommand{\tr}[1]{\text{tr}(#1)}$ $\newcommand{\oto}{\text{ one-to-one }}$ $\newcommand{\ot}{\text{ onto }}$ $\newcommand{\ceil}[1]{\lceil#1\rceil}$ $\newcommand{\floor}[1]{\lfloor#1\rfloor}$ $\newcommand{\Re}[1]{\text{Re}(#1)}$ $\newcommand{\Im}[1]{\text{Im}(#1)}$ $\newcommand{\dom}[1]{\text{dom}(#1)}$ $\newcommand{\fnext}[1]{\overset{\sim}{#1}}$ $\newcommand{\transpose}[1]{{#1}^{\text{T}}}$ $\newcommand{\b}[1]{\boldsymbol{#1}}$ $\newcommand{\None}[1]{}$ $\newcommand{\Vcw}[2]{\begin{bmatrix} #1 \br #2 \end{bmatrix}}$ $\newcommand{\Vce}[3]{\begin{bmatrix} #1 \br #2 \br #3 \end{bmatrix}}$ $\newcommand{\Vcr}[4]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \end{bmatrix}}$ $\newcommand{\Vct}[5]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \br #5 \end{bmatrix}}$ $\newcommand{\Vcy}[6]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \end{bmatrix}}$ $\newcommand{\Vcu}[7]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \br #7 \end{bmatrix}}$ $\newcommand{\vcw}[2]{\begin{matrix} #1 \br #2 \end{matrix}}$ $\newcommand{\vce}[3]{\begin{matrix} #1 \br #2 \br #3 \end{matrix}}$ $\newcommand{\vcr}[4]{\begin{matrix} #1 \br #2 \br #3 \br #4 \end{matrix}}$ $\newcommand{\vct}[5]{\begin{matrix} #1 \br #2 \br #3 \br #4 \br #5 \end{matrix}}$ $\newcommand{\vcy}[6]{\begin{matrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \end{matrix}}$ $\newcommand{\vcu}[7]{\begin{matrix} #1 \br #2 \br #3 \br #4 \br #5 \br #6 \br #7 \end{matrix}}$ $\newcommand{\Mqw}[2]{\begin{bmatrix} #1 & #2 \end{bmatrix}}$ $\newcommand{\Mqe}[3]{\begin{bmatrix} #1 & #2 & #3 \end{bmatrix}}$ $\newcommand{\Mqr}[4]{\begin{bmatrix} #1 & #2 & #3 & #4 \end{bmatrix}}$ $\newcommand{\Mqt}[5]{\begin{bmatrix} #1 & #2 & #3 & #4 & #5 \end{bmatrix}}$ $\newcommand{\Mwq}[2]{\begin{bmatrix} #1 \br #2 \end{bmatrix}}$ $\newcommand{\Meq}[3]{\begin{bmatrix} #1 \br #2 \br #3 \end{bmatrix}}$ $\newcommand{\Mrq}[4]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \end{bmatrix}}$ $\newcommand{\Mtq}[5]{\begin{bmatrix} #1 \br #2 \br #3 \br #4 \br #5 \end{bmatrix}}$ $\newcommand{\Mqw}[2]{\begin{bmatrix} #1 & #2 \end{bmatrix}}$ $\newcommand{\Mwq}[2]{\begin{bmatrix} #1 \br #2 \end{bmatrix}}$ $\newcommand{\Mww}[4]{\begin{bmatrix} #1 & #2 \br #3 & #4 \end{bmatrix}}$ $\newcommand{\Mqe}[3]{\begin{bmatrix} #1 & #2 & #3 \end{bmatrix}}$ $\newcommand{\Meq}[3]{\begin{bmatrix} #1 \br #2 \br #3 \end{bmatrix}}$ $\newcommand{\Mwe}[6]{\begin{bmatrix} #1 & #2 & #3\br #4 & #5 & #6 \end{bmatrix}}$ $\newcommand{\Mew}[6]{\begin{bmatrix} #1 & #2 \br #3 & #4 \br #5 & #6 \end{bmatrix}}$ $\newcommand{\Mee}[9]{\begin{bmatrix} #1 & #2 & #3 \br #4 & #5 & #6 \br #7 & #8 & #9 \end{bmatrix}}$
Example: Roulette

[1|2|3|4| … |36|0|00]
1, 3, …, 35 are red (odd)
2, 4, …, 36 are black (even)
0, 00 are green.
Payout on red or black is $1:1$ i.e. if you bet $1 and win, you win $1.
Payout on $a \neq (1, …, 36)$ is $1:35$ i.e. if you bet $1 and win you win $35.

Let $X$ be the outcome of a roll.

$X(\set{0})= 37, X(\set{00})=38$

A bet is a function of $X$.

e.g. 1: bet $10 on red.

$G(X) = \begin{cases} 10 , \if x = 1,3,…,35 \br -10 , \otherwise \end{cases}$

2: bet $10 on 2.

$W(X) = \begin{cases} 350, \if x = 2 \br -10, \otherwise \end{cases}$

We assume it’s a fair wheel, i.e.

$\b{P}(X=k) = \begin{cases} \frac{1}{ 38 }, \if 1, 2, …, 38 \br 0, \otherwise \end{cases}$

Let’s compute the expected.

Let’s compute the expected outcome of each of these bets.

  1. let $Y = g(X)$, recall by definition, $E[Z]= \sum_{z}\b{P}(Z=z)$ also recall theorem E[g(X)] = \sum_{x} g(x)\b{P}(X=x) $E[y] = E[g(X)] = \sum_{x} g(x)\b{P}(X=x) = 10 \sum_{ x=1,3,…,35 } \frac{1}{ 38 } + (-10) \sum_{ x = 2,4, …, 36,37,38 } \frac{1}{ 38 }$

$ E[h(X)] = \sum_{x} h(x)\b{P}(X=x) = \sum_{ x=2 } 350 \cdot \frac{1}{ 38 } + \sum_{ x \in \set{ 1, 38 }} (-10) \frac{1}{ 38 }$

try calculating variance and standard deviation.

Properties: Key Properties of $E[X]$
  1. $E[X+Y] = E + E[Y]$
  2. if $a \in R$, then $E[aX] = aE $
  3. if $a \in R$, then $E[a] = a$

I summarize (2) - (3) as $E[aX + c] = aE[X] + c$

for variance, $var(aX + c) = a^2 var(X)$

Definition: Variance

$var(X) = E[(X-E[X])^2]$

Equivalently, $var(X) = E [X^2 -2XE[X] + (E[X])^2]$ $=E[X^2] - 2E[X]E[X] + E[X]^2 = E[X^2] -2E[X]^2$

One experiment, $2$ random variable $s$.

Definition: Joint Probability Mass Function

The joint probability mass function of two discrete random variable $X,Y$ associated to one experiment is $P_{X,Y}(x,y) = \b{P}(X=x, Y=y)$.

Definition: Joint Probability Density Function
Definition: Marginal Distributions (Discrete)

$$P_{X_i}(x) = \sum_{ x_1 \in \R } … \sum_{ x_{i - 1} \in \R } \sum_{ x_{i + 1} \in \R }\sum_{ x_{n} \in \R } P_{ x_1, x_2, …, x_{ n }}(x_1, _2, … x, …, _{ n })$$

Definition: Marginal Distribution (Continuous)
Example

magnetic board, some parts are stronger than others, I throw a metallic object.

I record the probability of objects landing on the each of the squares.

0 1
1 1/2 1/8
0 1/8 1/4

Let X be the horizontal grid value, y = the vertical grid value.

$\b{P}(X=1)= \b{P}(X=1) = \frac18 + \frac14 = \frac38$

How to find $P_X, P_Y$?

In general, say $X, Y$ have joint probability mass function $ P_{X, Y}$ we can find $P_X$, and $P_Y$(called marginals) as follow: $P_X(x) = \b{P}(X=x) = \b{P}(\cup_y (X=x \cap Y = y)) = \sum_{y} p_{X,Y}(x,y)$

Similarly, $P_Y(y) = P_{X,Y}(x,y)$

$P_X(1) = \sum_{y} P_{1,Y} = \b{P}(X=1, Y=0) + \b{P}(X=1, Y=1) = \frac{1}{4} + \frac{1}{8} = \frac{3}{8}$

$P_X(0) = \sum_{y} P_{0,Y} = \b{P}(X=0, Y=0) + \b{P}(X=0, Y=1) = \frac{1}{8} + \frac{1}{2} = \frac{5}{8}$

$P_Y(1) = \sum_{x} P_{X,1} = \b{P}(X=0, Y=1) + \b{P}(X=1, Y=1) = \frac{1}{2} + \frac{1}{8} = \frac{5}{8}$

$P_Y(0) = \sum_{x} P_{X,0} = \b{P}(X=0, Y=0) + \b{P}(X=1, Y=0) = \frac{1}{8} + \frac{1}{4} = \frac{3}{8}$

Example

Another example, with same marginals but different probability for each grid.

0 1
1 25/64 15/64
0 15/64 9/64

$P_X(1) = \sum_{y} P_{1,Y} = \b{P}(X=1, Y=0) + \b{P}(X=1, Y=1) = \frac{3}{8}$

$P_X(0) = \sum_{y} P_{0,Y} = \b{P}(X=0, Y=0) + \b{P}(X=0, Y=1) = \frac{5}{8}$

$P_Y(1) = \sum_{x} P_{X,1} = \b{P}(X=0, Y=1) + \b{P}(X=1, Y=1) = \frac{5}{8}$

$P_Y(0) = \sum_{x} P_{X,0} = \b{P}(X=0, Y=0) + \b{P}(X=1, Y=0) = \frac{3}{8}$

Theorem

Let $g: \R \times \R \to \R$ i.e. g is a real-valued function of 2 variables. Let X, Y have joint probability mass function, $P_{X,Y}$.

$$E[g(X,Y)] = \sum_{ x,y } g(x,y) P_{X,Y}(x, y)$$

Example

Let $X$ be binomial random variable, with parameters $n, p$ i.e. $X$ is number of $H$ in $n$ tosses of a coin.

Let random variable $X_i$ take the value $1$ if the $i$th toss is $\head $, and $0$ if the $i$th toss is $\tail $.

Note: $X = \sum_{ i=1 }^{n} X_i$

So $E[X] = \sum_{ i=1 }^{n} E[X_i]$

$\forall i, E[X_i] = \sum_{x} x p_X(x) = 10 p + 0(1-p) =p $

so $E[X] = np$

summaries:

if $X$ binomial with parameters $n, p$. New $E[X]= np$.

note: if $p= \frac{ 1 }{ 2 }, E[X] = \frac{n}{2}$

Definition: Poisson Random Variables

Let $X$ be number of typos in a book with $n$ letters, where each letter is wrong is $p$. All typos are independent.

$\b{P}(X=k) = \binom{n}{k} p^k (1-p)^{n-k}, k \in \set{ 0, 1, 2, …, {n}}$

Let $\lambda = np$ for $k \in \set{ 0, …, n }$

For large $n, p \in [0, 1]$:

$$e^{-\lambda} \frac{ \lambda^k }{ k! } \approx \binom{n}{k} p^k (1-p)^{n-k}$$

Let $Y$ be random variable with probability mass function:

$$p_Y(k) = e^{-\lambda} \frac{ \lambda^k }{ k! }, k= 0,1,2,…$$

$Y$ is called Poisson with parameter $\lambda > 0$.

check:

$\sum_{ k=0 }^{ \infty } p_y(k) = \sum_{ k=0 }^{ \infty } e^{-\lambda} \frac{ \lambda^k }{ k! } = e^{-\lambda} \sum_{ k=0 }^{ \infty } \frac{ \lambda^k }{ k! } = e^{-\lambda} e^\lambda = 1$