best approximation in inner product spaces
The study of best approximations in inner product spaces^{} has a very elegant treatment with profound consequences. Most of the theory of Hilbert spaces^{} depends on this study and several approximation problems are better understood using this techniques and results.
For example: least square fitting, linear regression, approximation of functions by polynomials, among many other problems, can be seen as particular cases of the general study of best approximation in inner product spaces.
Some of the above problems are going to be discussed later in this entry.
1 Existence and Uniqueness
Our fundamental result on the existence and uniqueness of best approximations is the following (we postpone its proof to this attached entry (http://planetmath.org/ProofOfExistenceAndUniquenessOfBestApproximations)):
Theorem  Let $X$ be an inner product space and $A\subseteq X$ a complete^{} (http://planetmath.org/Complete), convex and nonempty subset. Then for every $x\in X$ there exists a unique best approximation (http://planetmath.org/BestApproximation) of $x$ in $A$, i.e. there exists a unique element ${a}_{0}\in A$ such that
$$\parallel x{a}_{0}\parallel =d(x,A)=\underset{a\in A}{inf}\parallel xa\parallel .$$ 
2 Geometric Interpretation
The following result gives a very geometric interpretation^{} of the best approximation when $A$ is a subspace^{} of $X$. We also postpone its proof to an attached entry.
Theorem  Let $X$ be an inner product space, $A\subseteq X$ a subspace and $x\in X$. The following statements are equivalent^{}:

•
${a}_{0}\in A$ is the best approximation of $x$ in $A$.

•
${a}_{0}\in A$ and $x{a}_{0}\u27c2A$.
Thus, the best approximation of $x$ in a subspace $A$ is just the orthogonal projection of $x$ in $A$.
3 Calculation of Best Approximations
When the $A$ is a complete subspace of $X$, the best approximation can be ”calculated” explicitly. Recall that, in this case, $A$ becomes an Hilbert space (since it is complete) and therefore it has an orthonormal basis^{}.
Again, we postpone the proof of the next result to an attached entry.
Theorem  Let $X$ be an inner product space and $A\subseteq X$ a complete subspace. Let ${({e}_{i})}_{i\in J}$ be an orthonormal basis for $A$. Then for every $x\in X$ the best approximation ${a}_{0}\in A$ of $x$ in $A$ is given by
$${a}_{0}=\sum _{i\in J}\u27e8x,{e}_{i}\u27e9{e}_{i}.$$ 
One can also write the best approximation in of any other basis (not necessarily an orthonormal one). For simplicity we present here how that can be done when $A$ is a finite dimensional subspace of $X$.
Theorem  Let $X$ be an inner product space and $A\subseteq X$ a finite dimensional subspace. Let ${v}_{1},\mathrm{\dots},{v}_{n}$ be a basis for $A$. Then for every $x\in X$ the best approximation ${a}_{0}\in A$ of $x$ in $A$ is given by
$${a}_{0}=\sum _{i=1}^{n}{a}_{0}^{i}{v}_{i}$$ 
where the coefficients ${a}_{0}^{i}$ are the solutions of the system of equations
$$\left(\begin{array}{ccc}\hfill \u27e8{v}_{1},{v}_{1}\u27e9\hfill & \hfill \mathrm{\cdots}\hfill & \hfill \u27e8{v}_{1},{v}_{n}\u27e9\hfill \\ \hfill \mathrm{\vdots}\hfill & \hfill \mathrm{\ddots}\hfill & \hfill \mathrm{\vdots}\hfill \\ \hfill \u27e8{v}_{n},{v}_{1}\u27e9\hfill & \hfill \mathrm{\cdots}\hfill & \hfill \u27e8{v}_{n},{v}_{n}\u27e9\hfill \end{array}\right)\left(\begin{array}{c}\hfill {a}_{0}^{1}\hfill \\ \hfill \mathrm{\vdots}\hfill \\ \hfill {a}_{0}^{n}\hfill \end{array}\right)=\left(\begin{array}{c}\hfill \u27e8x,{v}_{1}\u27e9\hfill \\ \hfill \mathrm{\vdots}\hfill \\ \hfill \u27e8x,{v}_{n}\u27e9\hfill \end{array}\right).$$ 
$Remark$ The above matrix is a symmetric^{} positive definite^{} (http://planetmath.org/PositiveDefinite) matrix, which implies that the system has a unique solution as expected.
4 Applications
There are several applications of the above results. We explore two of them in the following.
4.0.1  Approximation of functions by polynomials :
Suppose we want to find a polynomial of degree $\le n$ that approximates in the best possible way a given function $f$. We are in fact trying to find a point in the subspace of polynomials of degree $\le n$ that is closest to $f$, i.e. we are trying to find the best approximation of $f$ in that subspace.
For example, let $f\in {L}^{2}([0,1])$. Consider the basis ${v}_{k}(t)={t}^{k},0\le k\le n,$ of the subspace of polynomials of degree $\le n$.
The best approximation of $f$ by these polynomials is the function ${a}_{0}(t)={a}_{0}^{1}+{a}_{0}^{1}t+\mathrm{\dots}+{a}_{0}^{n}{t}^{n}$, where the coefficients ${a}_{0}^{1},\mathrm{\dots},{a}_{0}^{n}$ are the solutions of the system
$$\left(\begin{array}{ccc}\hfill 1\hfill & \hfill \mathrm{\cdots}\hfill & \hfill \frac{1}{n+1}\hfill \\ \hfill \mathrm{\vdots}\hfill & \hfill \mathrm{\ddots}\hfill & \hfill \mathrm{\vdots}\hfill \\ \hfill \frac{1}{n+1}\hfill & \hfill \mathrm{\cdots}\hfill & \hfill \frac{1}{2n+1}\hfill \end{array}\right)\left(\begin{array}{c}\hfill {a}_{0}^{1}\hfill \\ \hfill \mathrm{\vdots}\hfill \\ \hfill {a}_{0}^{n}\hfill \end{array}\right)=\left(\begin{array}{c}\hfill {\int}_{0}^{1}f(t)\mathit{d}t\hfill \\ \hfill \mathrm{\vdots}\hfill \\ \hfill {\int}_{0}^{1}{t}^{n}f(t)\mathit{d}t\hfill \end{array}\right).$$ 
$Remark$ Instead of polynomials we could approximate $f$ by any other of functions using the same procedure.
4.0.2  Best Fitting Lines :
Suppose we want to find the line that best fits some given points $({t}_{1},{y}_{1}),\mathrm{\dots},({t}_{n},{y}_{n})$, i.e. the affine function ${a}_{0}(t)=\alpha t+\beta $ that minimizes $\sum _{k=1}^{n}}{{a}_{0}({t}_{k}){y}_{k}}^{2$.
We are then led to consider the inner product^{}
$$\u27e8f,g\u27e9=\sum _{k=1}^{n}f({t}_{k})g({t}_{k})$$ 
in the space of functions $h:\{{t}_{1},\mathrm{\dots},{t}_{k}\}\u27f6\mathbb{R}$.
With this setting we are then looking for the best approximation of the function $f({t}_{k})={y}_{k}$ on the subspace of affine functions.
A base for the subspace of affine functions is given by the functions ${v}_{1}(t)=1$ and ${v}_{2}(t)=t$.
The best approximation of $f$ on this space is the function ${a}_{0}(t)=\beta +\alpha t$, where the coefficients $\beta ,\alpha $ are the solutions of the system
$$\left(\begin{array}{cc}\hfill n\hfill & \hfill {\sum}_{k=1}^{n}{t}_{k}\hfill \\ \hfill {\sum}_{k=1}^{n}{t}_{k}\hfill & \hfill {\sum}_{k=1}^{n}{t}_{k}^{2}\hfill \end{array}\right)\left(\begin{array}{c}\hfill \beta \hfill \\ \hfill \alpha \hfill \end{array}\right)=\left(\begin{array}{c}\hfill {\sum}_{k=1}^{n}{y}_{k}\hfill \\ \hfill {\sum}_{k=1}^{n}{y}_{k}{t}_{k}\hfill \end{array}\right).$$ 
Thus, the function ${a}_{0}(t)=\beta +\alpha t$ obtained by the above procedure provides the line that best fits the data $({t}_{1},{y}_{1}),\mathrm{\dots},({t}_{n},{y}_{n})$.
Title  best approximation in inner product spaces 
Canonical name  BestApproximationInInnerProductSpaces 
Date of creation  20130322 17:32:16 
Last modified on  20130322 17:32:16 
Owner  asteroid (17536) 
Last modified by  asteroid (17536) 
Numerical id  12 
Author  asteroid (17536) 
Entry type  Feature 
Classification  msc 41A65 
Classification  msc 46C05 
Classification  msc 46N10 
Classification  msc 49J27 
Classification  msc 41A52 
Classification  msc 41A50 
Defines  approximation by polynomials 
Defines  best fitting lines 