[1] 3 1 2[1] "integer"SMaC: Statistics, Math, and Computing
Applied Statistics for Social Science Research
Introduction to Vectors
- When is dot product zero?
- These vectors are perpendicular (orthogonal)
Introduction to Matrices
- A good way to think about the Matrices is that they act on vectors and transform them in some way (stretch or turn them)
- You can think of this transformation as encoding the eventual locations of the basis vectors
- Standard basis vectors in \(R^2\) (two-dimensional space), are commonly called \(\hat{i}\) with location \((1, 0)\), and \(\hat{j}\) with location \((0, 1)\).
- Notice their dot product is zero: \(\hat{i} \cdot \hat{j} = 0\) and so they are orthogonal
- Let’s look at one simple transformation – rotation by 90 degrees counter-clockwise
- Recall, vectors \(v\) and \(w\):
\[ v = \left[ \begin{matrix}1\\2\end{matrix} \right] w = \left[ \begin{matrix}3\\1\end{matrix} \right] \]
- The following matrix encodes a 90-degree, counterclockwise rotation. Why?
\[ R = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix} \]
- Let’s see how this matrix will act on vectors \(v\) and \(w\). That’s where multiplication comes in.
\[ Rv = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = 1 \begin{bmatrix} 0 \\ 1 \end{bmatrix} + 2 \begin{bmatrix} -1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 1 \end{bmatrix} + \begin{bmatrix} -2 \\ 0 \end{bmatrix} = \begin{bmatrix} -2 \\ 1 \end{bmatrix} \]
- We scale the first vector, then scale the second, and then add.
Matrix Multiplication
- Our matrix encodes a rotation of every single vector in \(R^2\).
- In particular, it rotates every point \((x, y)\) in \(R^2\) by 90 degrees
- We can do this transformation in one go by applying the Rotation matrix to another matrix comprising our vectors \(v\) and \(w\)
- The number of columns in the Rotation matrix has to match the number of rows in the \(K = [v, w]\) matrix
The resulting matrix \(K\), has the correctly rotated \(v\) in the first column and rotated \(w\) in the second column.
Another way to think about this operation is to encode two transformations in \(K'\) — the \(K\) transformation followed by the \(R\) transformation. We can now use the resulting \(K'\) matrix and apply these two transformations in one swoop to any vector in \(R^2\).
Think about why matrix multiplication, in general, does not commute — \(RK \neq KR\)
- Rotating red (\(v = [1 \ 2]\)) and blue (\(w = [3 \ 1]\)) vectors by 90 degrees
Your Turn: Come up with a 90-degree clockwise rotation matrix and show that it sends \((2, 2)\) to \((2, -2)\)
Now pick three vectors in \(R^2\) and rotate all three at the same time
Linear Independence and Determinants
- Think of a determinant as a (signed) area scaling factor from the basis \((\hat{i}, \hat{j})\) to the transformed vectors
- The area represented by two \(R^2\) basis vectors is 1
- What happens if the columns of \(X\) are linear combinations of each other?
- Geometrically, that means that in \(R^2\), they both sit on the same line
Approximating a Solution to Overdetermined \(X\hat{\beta} = y\)
If we can not reach \(y\), we need to find a vector in the column space of \(X\) that is closest (in a certain sense) to \(y\)
The projection of \(y\) onto this plane gives us the answer
Motion in the Straight Line
- Recall, from lecture 1, our example of a car moving in a straight line where we have noisy measurements of its position over time
- At the time, we assumed we were able somehow to find the intercept and slope of the equation
- We are now in a position to solve the problem
Motion in the Straight Line
This is a quadratic function. How can we solve it using linear regression (the least squares method)?
We assumed the equation of motion was \(x(t) = a + bt^2\), which we will write as \(y(t) = \beta_0 + \beta_1 t^2\)
The unknowns are \(\beta_0\) and \(\beta_1\)
Let’s express it in the matrix notation
What is our design matrix \(X\)? We have 51 observations, so \(X\) will have two columns and 51 rows: a column of 1s for the intercept and a column of times squared \(t^2\)
Our unknown vector \(\hat{\beta}\) has two elements \((\hat{\beta_0},\, \hat{\beta_1)}\)
\(y\) is our outcome vector of length 51, capturing the car’s position at each time point \(t\)
\(y = X\hat{\beta}\) captures our system in matrix form
[1] 51[1] 0.0 0.1 0.2 0.3 0.4 0.5 [,1] [,2]
[1,] 1 0.00
[2,] 1 0.01
[3,] 1 0.04
[4,] 1 0.09
[5,] 1 0.16
[6,] 1 0.25[1] 51[1] -2.4417028 6.7615084 -0.7502334 -0.4900157 -3.5129263 1.9331119- Recall, that the least squares estimate of \(\beta\) is given by \(\hat{\beta} = (X^TX)^{-1}X^Ty\)
[,1]
[1,] 1.46
[2,] 3.01data <- data.frame(y = y, t_squared = t^2)
# compare to R's lm()
fit <- lm(y ~ t_squared, data = data)
arm::display(fit)lm(formula = y ~ t_squared, data = data)
coef.est coef.se
(Intercept) 1.46 0.54
t_squared 3.01 0.05
---
n = 51, k = 2
residual sd = 2.60, R-Squared = 0.99- Warning: do not use \((X^TX)^{-1}X^Ty\) directly as above; it’s computationally inefficient
