[1] 3 1 2[1] "integer"SMaC: Statistics, Math, and Computing
APSTA-GE 2006: Applied Statistics for Social Science Research
Introduction to Vectors
- When is dot product zero?
- These vectors are perpendicular (orthogonal)
Introduction to Matrices
- A good way to think about the Matrices is that they act on vectors and transform them in some way (stretch or turn them)
- You can think of this transformation as encoding the eventual locations of the basis vectors
- Standard basis vectors in \(R^2\) (two-dimensional space), are commonly called \(\hat{i}\) with location \((1, 0)\), and \(\hat{j}\) with location \((0, 1)\).
- Notice their dot product is zero: \(\hat{i} \cdot \hat{j} = 0\) and so they are orthogonal
- Let’s look at one simple transformation – rotation by 90 degrees counter-clockwise
- Recall, vectors \(v\) and \(w\):
\[ v = \left[ \begin{matrix}1\\2\end{matrix} \right] w = \left[ \begin{matrix}3\\1\end{matrix} \right] \]
- The following matrix encodes a 90-degree, counterclockwise rotation. Why?
\[ R = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix} \]
- Let’s see how this matrix will act on vectors \(v\) and \(w\). That’s where multiplication comes in.
\[ Rv = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = 1 \begin{bmatrix} 0 \\ 1 \end{bmatrix} + 2 \begin{bmatrix} -1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 1 \end{bmatrix} + \begin{bmatrix} -2 \\ 0 \end{bmatrix} = \begin{bmatrix} -2 \\ 1 \end{bmatrix} \]
- We scale the first vector, then scale the second, and then add.
Matrix Multiplication
- Our matrix encodes a rotation of every single vector in \(R^2\).
- In particular, it rotates every point \((x, y)\) in \(R^2\) by 90 degrees
- We can do this transformation in one go by applying the Rotation matrix to another matrix comprising our vectors \(v\) and \(w\)
- The number of columns in the Rotation matrix has to match the number of rows in the \(K = [v, w]\) matrix
The resulting matrix \(K\), has the correctly rotated \(v\) in the first column and rotated \(w\) in the second column.
Another way to think about this operation is to encode two transformations in \(K'\) — the \(K\) transformation followed by the \(R\) transformation. We can now use the resulting \(K'\) matrix and apply these two transformations in one swoop to any vector in \(R^2\).
Think about why matrix multiplication, in general, does not commute — \(RK \neq KR\)
- Rotating red (\(v = [1 \ 2]\)) and blue (\(w = [3 \ 1]\)) vectors by 90 degrees
Your Turn: Come up with a 90-degree clockwise rotation matrix and show that it sends \((2, 2)\) to \((2, -2)\)
Now pick three vectors in \(R^2\) and rotate all three at the same time
Linear Independence and Determinants
- Think of a determinant as a (signed) area scaling factor from the basis \((\hat{i}, \hat{j})\) to the transformed vectors
- The area represented by two \(R^2\) basis vectors is 1
- What happens if the columns of \(X\) are linear combinations of each other?
- Geometrically, that means that in \(R^2\), they both sit on the same line
Linear Systems and Inverses
- One of the key ideas from Linear Algebra that is relevant to statistics is solving linear systems
- This is where estimating coefficients in \(X\beta = y\) comes from
- If \(X\) is a square matrix, the problem is not statistical but algebraic
- The reason is that if we have the same number of equations as we have unknowns, we can solve the system exactly unless X is not full rank
- For example, try to solve this system:
\[ 2x + y = 1 \\ 4x + 2y = 1 \]
- Question: Geometrically, what does it mean to solve a system of equations?
- You should now have a deeper understanding of why there is no solution
- Before we talk about overdetermined systems (more equations than unknowns) that we see in statistics, let’s see how it works in the square matrix case
- To see what a matrix inverse does, let’s bring back the 90-degree counterclockwise rotation matrix \(A\)
- The idea of an inverse is to reverse the action of this transformation
- In this case, we need a clockwise 90-degree rotation matrix
- We can find that matrix by directly tracing the basis vectors
\[\begin{aligned} A^{-1} = \left[\begin{matrix}0 & 1\\-1 & 0\end{matrix}\right] \end{aligned}\]
- Let’s check our results in R. When you multiply a matrix by its inverse, you get back the identity matrix (which is like when you divide a scalar by its reciprocal, you get 1)
- We are now ready to solve the square linear system \(Ax = b\), in the case when A is full rank
\[ \begin{eqnarray} Ax & = & b \\ A^{-1}Ax & = & A^{-1}b \\ x & = & A^{-1}b \end{eqnarray} \]
- The general solution in two-dimensional case:
- In R,
solve(A)inverts the matrix, andsolve(A, b)solves \(Ax = b\).
\(X\hat{\beta} = y\)
If we can not reach \(y\), we need to find a vector in the column space of \(X\) that is closest (in a certain sense) to \(y\)
The projection of \(y\) onto this plane gives us the answer
Recovering g from the Moon Drop
- Recall the Moon mission from Session 2: fixed marks give known distances \(x_i\), while stopwatch readings \(t_i\) have a constant delay and random noise
gravity-flash-162-01.csvcontains 17 recorded readings from one actual trial- We are now in a position to solve the problem
Recovering g from the Moon Drop
The marks \(x_i\) are fixed; let \(\alpha\) be a constant stopwatch delay
Model the observed time as \[ t_i=\alpha+\sqrt{\frac{2x_i}{g}}+\varepsilon_i, \qquad \E(\varepsilon_i)=0. \]
Set \(z_i=\sqrt{x_i}\) and \(\beta=\sqrt{2/g}\) to obtain \[ t_i=\alpha+\beta z_i+\varepsilon_i. \]
With \(\boldsymbol\theta=(\alpha,\beta)^\mathsf{T}\), the matrix model is \[ \mathbf{t}=X\boldsymbol\theta+\boldsymbol\varepsilon, \qquad X=\begin{pmatrix} 1 & \sqrt{x_1}\\ \vdots & \vdots\\ 1 & \sqrt{x_n} \end{pmatrix}. \]
Recover gravity from the slope: \(\hat g=2/\hat\beta^2\)
[1] 17[1] 2.8435 3.6596 4.6100 5.3433 intercept sqrt_x
[1,] 1 2.236068
[2,] 1 3.162278
[3,] 1 3.872983
[4,] 1 4.472136 [,1]
[1,] 2.8435
[2,] 3.6596
[3,] 4.6100
[4,] 5.3433- The least-squares estimate is \(\hat{\boldsymbol\theta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\mathbf{t}\)
[,1]
intercept 0.3771
sqrt_x 1.0875delay g
0.377 1.691 (Intercept) sqrt(x)
0.3771303 1.0874652 - \(\hat\alpha=0.377\text{ s};\quad \hat g=1.691\text{ m/s}^2\).
fit_data <- transform(
moon_drop,
t_hat = delay_hat + beta_hat * sqrt_x
)
p_linearized <- ggplot(
fit_data,
aes(sqrt_x, observed_time_s)
) +
geom_point(size = 0.7) +
geom_line(
aes(y = t_hat),
color = "red",
linewidth = 0.35
) +
labs(
x = expression(sqrt(x)),
y = "observed time t (s)",
title = "Linearized fit"
)
