STT 861 Theory of Prob and STT I Lecture Note - 13

2017-11-29

Recap of linear predictor; almost surely convergence, converge in probability, converge in distribution; central limit theorem, theorem of DeMoivre-Laplace.

Portal to all the other notes

Lecture 13 - Nov 29 2017

For Video Recording

Linear Prediction

Recall $X, Y$ let

g (x) = E [Y | X = x]

The r.v. $g (x)$ is the best predictor of $Y$ given $X$ in the least square sense: $g$ minimizes $E (g (X) - Y)^{2})$ = MSE.

But what about making this MSE as small as possible when $g$ is linear? So use notation $h (x) = a + b x$ (instead of $g$ ).

We want to minimize

M S E = E ((Y - (a + b X))^{2})

Find $a$ and $b$ to make this as small as possible. Let

Z_{Y} = \frac{Y - μ_{Y}}{σ_{Y}}

Z_{X} = \frac{X - μ_{X}}{σ_{X}}

We also know,

E (Z_{X} Z_{Y}) = C o r r (X, Y) = ρ

Y - (a + b X) = (Z_{Y} - c Z_{X}) σ_{Y} + (d - a)

where $c = b \frac{σ_{Y}}{σ_{X}}$ , and $d = μ_{Y} - b μ_{X}$ ,

\begin{aligned} M S E & = σ_{Y}^{2} E ((Z_{y} - c Z_{X})^{2}) + (d - a)^{2} \\ = σ_{Y}^{2} (1 - 2 c ρ + c^{2}) + (d - a)^{2} \\ = σ_{Y}^{2} (1 - ρ^{2} + (c - ρ)^{2}) + (d - a)^{2} \end{aligned}

We see immediately that this is minimal for $a = d$ and $c = ρ$ .

Therefore,

b = ρ \frac{σ_{Y}}{σ_{X}}

This answers the question of what the best linear predictor of $Y$ given $X$ is the mean square sense.

We see the smallest MSE is therefore $σ_{Y}^{2} (1 - ρ^{2})$ .

Therefore, we see that the proportion of $Y$ ’s variance which is not explained by $X$ is

\frac{M S E}{σ_{Y}^{2}} = 1 - ρ^{2}

Finally, the proportion of $Y$ ’s variance which is explained by $X$ is $ρ^{2}$ .

Chapter 6 - Convergences

Definition: We say that the sequence of r.v.’s $(X_{n})_{n \in N}$ converges (“a.s.”(almost surely)) to the r.v. $X$ if

lim_{n \to \infty} X_{n} = X

with probability 1. In other word,

P (lim_{n \to \infty} | X_{n} - X |) = 1

Definition: (A weaker notion of convergence) A sequence of r.v.’s $(X_{n})_{n \in N}$ converges in probability to $X$ if

\forall ε > 0 : P (| X - X_{n} | > ε) \to 0

Note: [Convince yourself as an exercise at home] Convergence in probability is (easier to achieve) than converge a.s.

Definition: (even weaker version) Let sequence of r.v.’s $(X_{n})_{n \in N}$ as above but now let $F$ be the CDF of some distribution. We say $X_{n}$ converges in distribution to the law $F$ is

F_{X_{n}} (X) \to F (X)

as $n \to \infty$ for every fixed $x$ where $F (x)$ is continuous.

Note: unlike the previous two notions, here there is no need for a limiting r.v. $X$ and the $X_{n}$ ’s. Do not need to share a probability space with $X$ or anyone else.

Example 1

Let $Y_{i}$ , $i = 1, 2, 3, \dots$ be i.i.d with $μ$ and $σ^{2}$ is finite. We proved that, with

X_{n} = \frac{Y_{1} + Y_{2} + \dots + Y_{n}}{n} = μ

then

P (| X_{n} - μ | > ε) \leq \frac{σ^{2}}{ε^{2} n}

(By Chebyshev’s inequality)

The whole thing goes to 0 as $n \to \infty$ , this proves that $X_{n} \to μ$ in probability.

Note: assuming only $μ$ exists ( $σ^{2}$ could be infinity), conclusion still holds. See W. Feller’s book in 1950.

Let $X_{n} = U n i f o r m {1, 2, \dots, n}$ . This is a stepper function, with $1 / n$ increment each step. Let’s try to find out $F_{X_{n}} (x) = \frac{1}{n} [x]$ for $x \in [0, n]$ (integer function, the integer larger than $x$ ).

For fixed $x \in R^{+}$ , as $n \to \infty$ , $F_{X_{n}} (x) \to 0 (⋆)$ .

Since the function $⋆$ , is not the CDF of any random variable, this proves that $X_{n}$ does not converge in distribution, And therefore, $X_{n}$ cannot converges in any stronger sense (in probability or a.s.).

How about $Y_{n} = U n i f o r m {1 / n, 2 / n, \dots, n / n}$ ?

Since

F_{Y_{n}} (x) \to x

for $x \in [0, 1]$ , this is the CDF of $U n i f o r m (0, 1)$ .

Example 2

Let $U_{n} \sim U n i f (0, 1)$ , i.i.d. Let $M_{n} = max_{i = 1, 2, \dots, n} (U_{i})$ . We can tell that $M_{n} \to 1$ in some sense. Let’s prove it in probability:

Let $ε > 0$ , $\begin{aligned} P (| M_{n} - 1 | > ε) & = P (1 - M_{n} > ε) \\ = P (M_{n} < 1 - ε) \\ = P (max_{i = 1, 2, . . ., n} U_{i} < 1 - ε) \\ = P (\forall i : U_{i} < 1 - ε) \\ = P (\cap_{i = 1}^{n} {U_{i} < 1 - ε}) \\ = \prod_{i = 1}^{n} P (U_{i} < 1 - ε) = (1 - ε)^{n} \\ lim P (| M_{n} - 1 | > ε) & = 0 \end{aligned}$

We proved that $M_{n} \to 1$ in probability.

Now consider $Y_{n} = (1 - M_{n}) n$ . Let’s see about CDF of $Y_{n}$ .

\begin{aligned} 1 - F_{Y_{n}} (y) & = P ((1 - M_{n}) > y) \\ = P ((1 - M_{n}) > \frac{y}{n}) \\ = P (M_{n} < 1 - y / n) \\ = P (\prod {U_{i} < 1 - y / n}) \\ = (1 - y / n)^{n} \\ \to e^{- y} \end{aligned}

This CDF is exponential. Thus $Y_{n} \to E x p (λ = 1)$ in distribution.

Theorem (6.3.6): Let $(X_{n})_{n \in N}$ be a sequence of r.v.’s. If $X_{n}$ has MGF $M_{X_{n}} (t)$ and $M_{X_{n}} (t) \to M_{X} (t)$ for $t$ not 0, then $X_{n} \to X$ in distribution.

Example 3

Let $X_{n}$ be Bin( $n, p = λ / n$ ). We know that the PMF of $X_{n}$ converges to the PMF of Poisson( $λ$ ). Try it again here. We will find

M_{X_{n}} (t) = e^{λ} (e^{t} - 1)

and here we recognize that this is the MGF of Poisson( $λ$ ). By Theorem 6.3.6, $X_{n} \to P o i s s (λ)$ in distribution.

Let $X_{i}, i = 1, 2, \dots, n$ be i.i.d with $E (X_{i}) = μ$ and variance $V a r (X_{i}) = σ^{2}$ .

Let $Z_{i} = \frac{X_{i} - μ}{σ}$ , $S_{n} = \frac{\sum_{i = 1}^{n} Z_{i}}{\sqrt{n}}$ ,

(We divide by $\sqrt{(} n)$ as a standardization and we know $V a r (S_{n}) = 1$ ).

Central Limit Theorem (CLT)

As $n \to \infty$ , then $S_{n} \to N o r m a l (0, 1)$ in distribution.

This means:

lim P (S_{n} \leq x) = F_{N (0, 1)} (x) = \int_{- \infty}^{x} \frac{1}{\sqrt{2 π}} \exp (- z^{2} / 2) d z

The proof just combines the Taylor formula up to the order 2 and Theorem 6.3.6.

Next, consider

Y_{n} = \frac{(\sum_{i = 1}^{n} x_{i}) - n μ}{\sqrt{n}} = σ S_{n}

A corollary of the CLT says this:

Y_{n} \to N (0, σ^{2})

in distribution.

Proof:

\begin{aligned} P (Y_{n} \leq x) & = P (σ S_{n} \leq x) \\ = P (S_{n} \leq \frac{x}{σ}) \\ = \int_{- \infty}^{x / σ} \frac{1}{\sqrt{2 π}} \exp (- z^{2} / 2) d z \end{aligned}

and conclude by changing of variable.

Theorem of DeMoivre-Laplace

Let $X_{i}, i = 1, 2, 3, . ., n$ be i.i.d Bernoulli. Let $W_{n} = X_{1} + X_{2} + \dots + X_{n}$ . We know $W_{n} \sim B i n (n, p)$ .

Let $S_{n} = \frac{W_{n} - n p}{\sqrt{n p (1 - p)}}$ ,

Therefore, by CLT, $S_{n} \to N (0, 1)$ as $n \to \infty$ in distribution.

In other words, to be precise,

lim P (S_{n} \leq x) = \frac{1}{\sqrt{2 π}} \int_{- \infty}^{x} \exp (- z^{2} / 2) d z

P (\frac{W_{n} - n p}{\sqrt{n p (1 - p)}} \leq x) = P (W_{n} \leq x \sqrt{n p (1 - p)} + n p)

The speed of convergence in the CLT is known as a “Berry-Esseen” theorem. But the speed of convergence for Binomial CLT is much faster and rule of thrum is $p \in [1 / 10, 9 / 10]$ and $n \geq 30$ . CLT is good within $1 %$ convergence error.

Example 4

Let

M_{n} = \frac{U_{1} + U_{2} + \dots + U_{n}}{n}

where $U_{i}$ ’s are i.i.d Uniform(0, 1). We know by Weak law of large numbers, that $M_{n} \to \frac{1}{2}$ in probability as $n \to \infty$ . But how spread out is $M_{n}$ around 1/2? For example, can we estimate the chance that $M_{n}$ is more than 0.02 away from its mean value 1/2?

Answer:

\begin{aligned} P (| M_{n} - \frac{1}{2} | > 0.02) & = 1 - P (- 0.02 < M_{n} - \frac{1}{2} < 0.02) \\ = 1 - P (- 0.02 < \frac{\sum (U_{i} - 1 / 2)}{n}) \\ = 1 - P (- 0.02 \sqrt{n} < \frac{\sum (U_{i} - 1 / 2)}{\sqrt{n}} < 0.02 \sqrt{n}) \\ = 1 - P (- 0.02 \sqrt{n} / \sqrt{1 / 12} < \frac{\sum (U_{i} - 1 / 2)}{\sqrt{n}} \sqrt{1 / 12} < 0.02 \sqrt{n} \sqrt{1 / 12}) \\ \approx 1 - P (- \frac{0.02 \sqrt{n}}{\sqrt{1 / 12}} < N (0, 1) < \frac{0.02 \sqrt{n}}{\sqrt{1 / 12}}) \end{aligned}

We will feel comfortable if $n$ is large enough to make this greater than 0.95. What value should $n$ at least be.

In order to get this setup, it is known that the right-hand value should be = 1.96.

This value

0.975 = P (N (0, 1) \leq 1.95)

So 1.96 is therefore know as the 97.5 $^{t h}$ percentile of $N (0, 1)$ . Therefore, we must take

\frac{0.02 \sqrt{n}}{\sqrt{1 / 12}} \geq 1.96 \Rightarrow n > 800.33

← Older-Things to Do After Installing Ubuntu

STT 861 Theory of Prob and STT I Lecture Note - 14-Newer →

Disqus Comment （0）

LanternD's Castle

An electronics enthusiast - survive technically