Interactive Linear Algebra

Definition. A vector is an element of a vector space. We begin with vectors in \(\mathbb{R}^n\), written as ordered lists of real numbers: \(v=(v_1,\ldots,v_n)\). In two and three dimensions, a vector can be visualized as an arrow with magnitude and direction.

Geometrically. Think of an arrow drawn from the origin to a point. Its length is the magnitude and the way it points is the direction. For example, \(v=(3,1)\) points three units to the right and one unit upward.

Algebraically. Its Euclidean length, \(\|v\|=\sqrt{v_1^2+\cdots+v_n^2}\), is the Pythagorean theorem extended to \(n\) coordinates. The same definition works in four or four thousand dimensions, even when there is no picture we can draw.

\[\begin{aligned} v &= (v_1, v_2, \ldots, v_n) \in \mathbb{R}^n \\ \|v\| &= \sqrt{v_1^2+v_2^2+\cdots+v_n^2} \end{aligned}\]

Another description: polar coordinates. In two dimensions we can replace \((x,y)\) with a magnitude \(r=\|v\|\) and an angle \(\theta\): \(v=(r\cos\theta,r\sin\theta)\). Cartesian and polar coordinates describe the same arrow. This is our first example of an important theme: the representation may change while the object does not.

Why the trigonometric functions appear. The value \(\cos\theta\) measures alignment through \(\cos\theta=(a\cdot b)/(\|a\|\|b\|)\), while \(\sin\theta\) controls the area spanned by two vectors through \(\|a\times b\|=\|a\|\|b\|\sin\theta\). Finally, \(\tan\theta=y/x\) is the slope of the direction, and \(\operatorname{atan2}(y,x)\) recovers the angle from the coordinates.

\[\begin{aligned} x &= r\cos\theta \\ y &= r\sin\theta \\ r &= \|v\| \\ \theta &= \operatorname{atan2}(y,x) \end{aligned}\]

Drag the vector and watch its Cartesian coordinates, magnitude, and angle change together. The unit-circle panel shows the corresponding sine, cosine, and tangent.

Definition. Vector addition and subtraction operate componentwise. Scalar multiplication multiplies every component by the same number: \(v+w\), \(v-w=v+(-w)\), and \(kv\). The average \((v+w)/2\) is simply a sum scaled by one half.

In the picture. Scaling changes the arrow’s length by a factor of \(|k|\); a negative \(k\) also reverses its direction. To add two vectors, place the tail of the second at the tip of the first. The arrow from the origin to the final tip is \(v+w\), also the diagonal of their parallelogram.

Subtraction has an equally useful interpretation: \(v-w\) points from the tip of \(w\) to the tip of \(v\), so \(\|v-w\|\) measures the distance between the two endpoints. Meanwhile, \((v+w)/2\) lands at their midpoint.

Algebraically. \(v+w=(v_1+w_1,\ldots,v_n+w_n)\), \(v-w=(v_1-w_1,\ldots,v_n-w_n)\), and \(kv=(kv_1,\ldots,kv_n)\). Expressions such as \(c_1v+c_2w\) lead to linear combinations. Averaging many vectors gives their centroid, a simple way to construct a representative embedding.

\[\begin{aligned} v + w &= (v_1+w_1, \ldots, v_n+w_n) \\ v - w &= (v_1-w_1, \ldots, v_n-w_n) \\ kv &= (kv_1, \ldots, kv_n) \\ \text{avg} &= \frac{v+w}{2} \end{aligned}\]

Drag \(v\) and \(w\), then adjust \(k\) to compare the sum, difference, scaled vector, and midpoint.

Definition. A norm \(\|v\|\) measures the size of a vector with a nonnegative scalar. The familiar Euclidean, or L2, norm is only one choice. Common examples are \(\|v\|_1=\sum_i|v_i|\), \(\|v\|_2=\sqrt{\sum_i v_i^2}\), and \(\|v\|_\infty=\max_i|v_i|\).

Geometrically. Compare all points whose norm is one. In two dimensions these unit spheres are a circle for L2, a diamond for L1, and a square for L∞. Their interiors, \(\{v:\|v\|\leq1\}\), are the corresponding unit balls.

What makes a norm a norm? It must satisfy positivity, \(\|v\|\geq0\) with equality only at zero; absolute homogeneity, \(\|kv\|=|k|\|v\|\); and the triangle inequality, \(\|v+w\|\leq\|v\|+\|w\|\). The L2 norm connects directly to the next section because \(\|v\|_2=\sqrt{v\cdot v}\).

Embedding normalization. Dividing a nonzero embedding by its norm gives a unit vector in the same direction: \(\hat v=v/\|v\|\). The inner product of two L2-normalized embeddings is their cosine similarity, \(\hat v\cdot\hat w=(v\cdot w)/(\|v\|\|w\|)=\cos\theta\). Contrastive models such as CLIP and SimCLR use this idea so that direction, rather than arbitrary magnitude, drives the comparison.

What changes in high dimensions? The definitions do not: magnitude is still \(\sqrt{\sum_i v_i^2}\), and the inner product still accumulates how corresponding components align. Features in a neural representation are usually distributed across many coordinates rather than assigned one meaning per dimension. Random vectors from a high-dimensional isotropic distribution have cosine values concentrated near zero, but learned embeddings may be anisotropic, so a cosine score should be interpreted relative to the model and its data.

Why probabilities use L1 normalization. A probability vector has nonnegative entries and must satisfy \(\sum_i p_i=1\). Because its entries are nonnegative, this is exactly \(\|p\|_1=1\). Dividing a positive vector by its L1 norm creates a probability distribution. Softmax follows the same recipe: exponentiate to make every entry positive, then divide by their sum.

\[\begin{aligned} \|v\|_1 &= |v_1|+\cdots+|v_n| \\ \|v\|_2 &= \sqrt{v_1^2+\cdots+v_n^2} = \sqrt{v\cdot v} \\ \|v\|_\infty &= \max(|v_1|,\ldots,|v_n|) \\ \hat{v} &= v/\|v\| \end{aligned}\]

Drag the vector to compare the three norms, or play the L2 embedding and L1 probability normalization animations.

Definition. For real coordinate vectors, \(r\cdot v=\sum_i r_i v_i\).

Geometrically. \(r\cdot v=\|r\|\|v\|\cos\theta\). The value is positive for acute angles, zero for orthogonal vectors, and negative for obtuse angles.

Algebraically. For fixed \(r\), the map \(v\mapsto r\cdot v\) is a linear functional. The signed projection length of \(v\) onto the direction of \(r\) is \((r\cdot v)/\|r\|\).

Why are those the same number? "Multiply the coordinates and add them up" and "\(\|r\|\|v\|\cos\theta\)" look unrelated. The bridge between them is that a coordinate product is a per-direction measurement. Split \(v\) into \(v_1\hat\imath+v_2\hat\jmath\); linearity then gives \(r\cdot v=v_1(r\cdot\hat\imath)+v_2(r\cdot\hat\jmath)=v_1r_1+v_2r_2\), so the coordinate formula is nothing but "measure along each axis separately, then add." Meanwhile \(r\cdot\hat u\) is exactly the length of the shadow \(r\) casts along a unit direction \(\hat u\). The two expressions therefore measure one quantity in two different bases, and the length of a shadow does not care how we rotated the axes.

Intuition: an alignment score. Read the dot product as "how much do these two vectors look the same way," weighted by both lengths, and the signs become obvious: positive when they agree, zero when they are perpendicular, negative when they oppose. The coordinate formula tells the same story — \(r_iv_i\) contributes a positive amount only when the two vectors agree in sign along axis \(i\), so the sum is a vote tallied across the axes over how much they agree. Embedding search scores similarity by running that vote across hundreds of dimensions at once.

\[\begin{aligned} r\cdot v &= r_1v_1+r_2v_2 = \|r\|\|v\|\cos\theta \\ \text{proj length} &= \frac{r\cdot v}{\|r\|} \end{aligned}\]

Drag both vectors and observe the projection and cosine value.

Definition. For \(a,b\in\mathbb{R}^3\), the cross product \(a\times b\) is perpendicular to both vectors.

Geometrically. Its magnitude is \(\|a\times b\|=\|a\|\|b\|\sin\theta\), the area of the parallelogram spanned by \(a\) and \(b\). Reversing the operands reverses the direction.

Algebraically. \(a\times b=(a_2b_3-a_3b_2,\ a_3b_1-a_1b_3,\ a_1b_2-a_2b_1)\). For vectors in the xy-plane, its z-component is the corresponding 2×2 determinant.

\[a\times b = (a_2b_3-a_3b_2, \ a_3b_1-a_1b_3, \ a_1b_2-a_2b_1)\]

Drag the two planar vectors and inspect the perpendicular green vector in the 3D view.

Definition. An expression \(c_1v_1+\cdots+c_kv_k\), with scalar coefficients \(c_i\), is a linear combination.

Geometrically. Each coefficient controls how far and in which orientation to move along its corresponding vector.

Algebraically. Matrix–vector multiplication is a linear combination of the columns of \(A\): \(Ax=x_1a_1+\cdots+x_na_n\). Thus \(Ax=b\) asks whether \(b\) lies in the span of those columns.

\[Ax = x_1a_1 + x_2a_2\]

(\(a_i\) is the \(i\)-th column of \(A\).)

Adjust the coefficients to construct different points from the two columns.

Linear independence. Vectors \(v_1,\ldots,v_n\) are independent if \(c_1v_1+\cdots+c_nv_n=0\) forces every coefficient to be zero. If a nontrivial choice also gives zero, the vectors are dependent: at least one direction was already obtainable from the others.

Span. The span is the set of every linear combination: \(\operatorname{span}\{v_1,\ldots,v_n\}=\{c_1v_1+\cdots+c_nv_n:c_i\in\mathbb R\}\).

Basis and dimension. A basis is both independent and spanning. Every basis of a finite-dimensional space contains the same number of vectors, and that number is the dimension.

In the picture. One nonzero vector spans a line through the origin. Two nonparallel vectors span the whole plane and form a basis of \(\mathbb R^2\). As they become parallel, the span collapses back to a line: the second vector adds no genuinely new direction.

The determinant test. In \(n\) dimensions, place \(n\) vectors in the columns of a square matrix. Then \(\det A\neq0\), linear independence, spanning the whole space, and forming a basis are equivalent statements. If \(\det A=0\), the span has dimension below \(n\).

\[c_1v_1+\cdots+c_nv_n=0 \implies c_1=\cdots=c_n=0\] (definition of linear independence)

Drag the vectors until they become parallel and observe the determinant, span, and rank change together.

Definition. A function \(f\) is linear when \(f(x+y)=f(x)+f(y)\) and \(f(cx)=cf(x)\).

Geometrically. A linear function maps the origin to zero and preserves the linear structure of the domain. For \(f(x)=r\cdot x\), equal-valued inputs form parallel hyperplanes perpendicular to \(r\).

Algebraically. The map \(f(v)=r\cdot v\) is a standard example. Adding a nonzero constant, as in \(g(x)=r\cdot x+b\), produces an affine rather than a linear function.

\[f(x+y) = f(x)+f(y), \qquad f(cx) = c f(x)\]

Drag \(A\) and \(B\) to verify \(f(A+B)=f(A)+f(B)\).

Definition. A map between vector spaces is linear if it preserves addition and scalar multiplication. After bases are chosen, every finite-dimensional linear map is represented by a matrix \(A\).

Geometrically. The images of the standard basis vectors are the columns of \(A\). Once those images are known, linearity determines the image of every vector.

Algebraically. \(A(x,y)^\top=xA\hat\imath+yA\hat\jmath\), a linear combination of the transformed basis vectors.

\[T(x,y) = A[x,y]^\top = x\cdot A_{:,1} + y\cdot A_{:,2}\]

Modify the four matrix entries or select a preset to transform the grid.

First, what is a matrix? An \(m\times n\) matrix is a rectangular table with \(m\) rows and \(n\) columns; \(A_{ij}\) denotes the entry in row \(i\), column \(j\). A column vector is simply an \(n\times1\) matrix.

Definition. If \(A\in\mathbb{R}^{m\times n}\) and \(B\in\mathbb{R}^{n\times p}\), then \(AB\) is the \(m\times p\) matrix with \((AB)_{ik}=\sum_j A_{ij}B_{jk}\). The inner dimensions must match.

As composition. The order in \((AB)x=A(Bx)\) runs from right to left: apply \(B\) first, then \(A\). Deforming a grid in those two stages gives exactly the same result as applying \(AB\) once. Since changing the order usually changes the transformation, \(AB\neq BA\) in general.

As row–column products. Entry \((i,k)\) is the dot product of row \(i\) of \(A\) and column \(k\) of \(B\).

As combinations of columns. Column \(k\) of the product is \((AB)_{:,k}=AB_{:,k}\): a linear combination of the columns of \(A\), with the entries of \(B_{:,k}\) as coefficients. These are two views of the same operation, not two different algorithms.

\[\begin{aligned} (AB)_{ik} &= \sum_j A_{ij} B_{jk} = \text{row}_i(A)\cdot\text{col}_k(B) \\ (AB)_{:,k} &= A \cdot B_{:,k} \end{aligned}\]

Play the composition and switch between \(AB\) and \(BA\) to see why order matters.

Definition. If \(AB=BA=I\), then \(B=A^{-1}\).

Geometrically. An inverse restores the original grid. If \(\det(A)=0\), the transformation collapses the space into a lower dimension and cannot be reversed.

For a 2×2 matrix. If \(A=\begin{bmatrix}a&b\\c&d\end{bmatrix}\), then \(A^{-1}=\frac{1}{\det A}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}\), provided \(\det A\neq0\).

\[\begin{aligned} A^{-1} &= \frac{1}{\det(A)}\begin{bmatrix}d&-b\\-c&a\end{bmatrix} \\ A\cdot A^{-1} &= I \end{aligned}\]

Apply \(A\) followed by \(A^{-1}\), then move the determinant toward zero.

Definition. A nonzero vector \(x\) is an eigenvector of \(A\) if \(Ax=\lambda x\). The scalar \(\lambda\) is the corresponding eigenvalue.

Geometrically. The vector remains on the same line. A positive eigenvalue preserves its orientation, a negative value reverses it, and zero collapses it to the origin.

Algebraically. Nonzero solutions of \((A-\lambda I)x=0\) exist when \(\det(A-\lambda I)=0\), the characteristic equation.

Why a vanishing determinant? That condition is not a new rule; it reuses the story from the inverse section. In \((A-\lambda I)x=0\), the vector \(x=0\) always works — what we want to know is whether some nonzero \(x\) works too. But if a matrix \(M\) sends two different inputs (\(0\) and \(x\)) to the same output \(0\), it has crushed the space and lost information, so it cannot have an inverse — and having no inverse is precisely \(\det M=0\). In short, an eigenvalue is a scalar you can subtract from \(A\) to make it degenerate, and \(\det(A-\lambda I)=0\) is simply "when does it degenerate?" written as an equation in \(\lambda\).

Why this matters. Along an eigenvector, matrix multiplication collapses into ordinary multiplication by a number, which makes applying \(A\) repeatedly easy: such a component grows as \(A^kx=\lambda^kx\). Decompose any vector into eigenvector components, and after many applications the direction with the largest \(|\lambda|\) dominates everything else. That is why the long-run behaviour of a repeated linear update — whether it converges or blows up, and which direction it settles into — can be read off the eigenvalues alone, and why PCA picks the leading eigenvector of the covariance matrix as the direction of greatest spread.

\[Ax = \lambda x \iff \det(A-\lambda I) = 0\]

Move the vector onto an eigenvector direction and apply the transformation.

Definition. For \(A\in\mathbb{R}^{m\times n}\), the row space and null space lie in \(\mathbb{R}^n\), while the column space and left null space lie in \(\mathbb{R}^m\). The first two are generated by the rows and columns; the others are \(\{x:Ax=0\}\) and \(\{y:A^\top y=0\}\).

Orthogonal structure. The input space splits orthogonally into the row space—the directions whose information survives—and the null space—the directions sent to zero. The output space splits into the reachable column space and its perpendicular complement, the left null space.

A rank-one example. For \(A=\begin{bmatrix}1&2\\3&6\end{bmatrix}\), the second row is three times the first, so \(\operatorname{rank}(A)=1\). The row space follows \((1,2)\), its null space follows \((2,-1)\), the column space follows \((1,3)\), and the left null space follows \((3,-1)\).

Intuition: surviving dimensions plus destroyed dimensions equals the input dimension. Every direction in the input space does one of two things — it leaves a trace after the map (row space) or it is crushed to zero (null space). There is no middle case and no overlap, so the two dimensions must add up to the input dimension: \(\operatorname{rank}(A)+\dim(\text{null})=n\). This is the rank–nullity theorem, and it says that "how much information is preserved" and "how much is discarded" are each other's remainder. In the example above the input is two-dimensional with rank one, so the null space has no choice but to be one-dimensional — flattening a plane onto a line sacrifices exactly one direction.

Why the split is orthogonal. Saying \(Ax=0\) is saying that every row of \(A\) has zero dot product with \(x\). So a null-space vector is by definition perpendicular to every row vector, and therefore to the entire space the rows generate. Orthogonality is not a decoration added afterwards; it follows immediately from the definition of the null space. Apply the same argument to \(A^\top\) and you get column space ⊥ left null space — the same sentence read once more, transposed.

Under the map. Every input decomposes into row-space and null-space components. The null component disappears, while the surviving component produces an output in the column space.

\[\begin{aligned} \text{row space} &\perp \text{null space} & &(\text{input}, \mathbb{R}^n)\\ \text{column space} &\perp \text{left null space} & &(\text{output}, \mathbb{R}^m) \end{aligned}\]

Drag the input vector to inspect its decomposition and resulting output.

Orthogonality. Vectors \(a\) and \(b\) are orthogonal when \(a\cdot b=0\), written \(a\perp b\). A collection is orthogonal when every pair is orthogonal, and orthonormal when those vectors also have unit length.

Projection. The projection of \(b\) onto \(\operatorname{span}\{a\}\) is the point \(p=\lambda^*a\) on that line closest to \(b\). In other words, choose \(\lambda\) to minimize \(\|b-\lambda a\|\).

Orthogonal complement. For a subspace \(W\), the set \(W^\perp=\{v:v\perp w\text{ for every }w\in W\}\) contains all directions perpendicular to it. The row/null and column/left-null pairs in the previous section are exactly such complements.

Geometrically. Drop a perpendicular from \(b\) to the line spanned by \(a\). The landing point is \(p\), and \(b=p+e\) with residual \(e=b-p\) perpendicular to \(a\). If it were not perpendicular, we could move along the line to find a closer point.

Algebraically. Minimizing \(\|b-\lambda a\|^2\) gives \(\lambda^*=(a\cdot b)/(a\cdot a)\). Thus \(p=Pb\), where \(P=aa^\top/(a^\top a)\), and a projection matrix satisfies \(P^2=P=P^\top\).

Least squares. For a subspace spanned by the columns of \(A\), the projection becomes \(P=A(A^\top A)^{-1}A^\top\) when the inverse exists. If \(Ax=b\) has no exact solution, the closest attainable point is found from the normal equations \(A^\top A\hat x=A^\top b\).

\[\begin{aligned} \lambda^* &= \frac{a\cdot b}{a\cdot a} \\ p &= \lambda^* a = Pb \\ P &= \frac{aa^\top}{a^\top a} \\ P^2 &= P = P^\top \end{aligned}\]

Drag \(a\) and \(b\) and verify that the residual remains perpendicular to \(a\).

Definition. A property is invariant under an operation if it remains unchanged; it is variant if its representation changes.

Transformation invariance. An eigenvector direction is invariant under \(A\) because \(Ax=\lambda x\).

Basis dependence. The coordinate lists \([v]_E\) and \([v]_B\) differ, but they describe the same vector. Distinguishing an object from its coordinates is central to linear algebra.

\[\begin{aligned} \text{(a)} \quad &Ax=\lambda x \\ \text{(b)} \quad &[v]_E \neq [v]_B \end{aligned}\]

(a) the direction is invariant; (b) there is one vector \(v\) but two coordinate lists.

Switch between the transformation and basis views.

Definition. Let \(B=\{b_1,b_2\}\) be a basis and \(P=[b_1\ b_2]\). The coordinate vector in basis \(B\) is \([v]_B=P^{-1}v\), while \(v=P[v]_B\).

Geometrically. The same arrow is measured against a different grid. Its coordinate values change because the unit directions have changed.

Algebraically. Solving \(P[v]_B=v\) determines the coefficients of \(v\) as a linear combination of the new basis vectors.

\[\begin{aligned} P &= [b_1\ b_2] \\ [v]_E &= P[v]_B \\ [v]_B &= P^{-1}[v]_E \end{aligned}\]

Drag the vector or either basis vector and observe the coordinates in the new basis.

einsum

Definition. Einstein summation describes tensor contractions by naming axes with indices. Indices repeated across inputs are multiplied; indices omitted from the output are then summed away.

An intuitive reading. Think of a tensor as a table with several axes. An einsum expression says which axes survive in the result and which ones are contracted. Dot products, outer products, matrix multiplication, and transposition are all special cases of this one notation.

For example, torch.einsum('i,i->', a, b) computes a dot product, torch.einsum('i,j->ij', a, b) an outer product, and torch.einsum('ij,jk->ik', A, B) matrix multiplication.

Select an equation to inspect its indices and output.

permute, view, and reshape

permute. Reorders tensor dimensions by changing shape and strides without copying the underlying storage. For a matrix, permute(1, 0) is a transpose.

view. Reinterprets the same storage with a different shape. The number of elements must remain fixed, and the requested shape must be compatible with the existing strides. A contiguous tensor usually permits the expected views, although non-contiguous does not automatically mean that every view must fail.

reshape. Returns a view when possible and otherwise copies the data. Code should not assume that its result always shares storage with the input.

A useful mental model. Picture storage as one row of boxes. permute changes how many boxes each logical index step jumps over; view regroups those same boxes when the existing step pattern allows it. reshape may first copy them into a compatible order.

Memory layout. A contiguous tensor of shape (3, 2) has stride (2, 1). The transposed example has stride (1, 3), so its logical traversal order differs from its storage order even though the buffer itself is unchanged.

Back to the mathematics. For a matrix, permute(1, 0) produces \(A^\top\). In a real inner-product space this transpose is the adjoint, satisfying \(\langle Ax,y\rangle=\langle x,A^\top y\rangle\). Higher-dimensional permutations follow the same index logic; whether PyTorch must copy depends on the concrete shape and strides.

Compare view(3, 2) with permute(1, 0). Their shapes agree, but their values occupy different logical positions.