Tikhonov regularization

Tikhonov regularization, named for Andrey Tikhonov, is the most commonly used method of regularization of ill-posed problems. In statistics, the method is known as ridge regression, in machine learning it is known as weight decay, and with multiple independent discoveries, it is also variously known as the Tikhonov–Miller method, the Phillips–Twomey method, the constrained linear inversion method, and the method of linear regularization. It is related to the Levenberg–Marquardt algorithm for non-linear least-squares problems. Tikhonov regularization, named for Andrey Tikhonov, is the most commonly used method of regularization of ill-posed problems. In statistics, the method is known as ridge regression, in machine learning it is known as weight decay, and with multiple independent discoveries, it is also variously known as the Tikhonov–Miller method, the Phillips–Twomey method, the constrained linear inversion method, and the method of linear regularization. It is related to the Levenberg–Marquardt algorithm for non-linear least-squares problems. Suppose that for a known matrix A {displaystyle A} and vector b {displaystyle mathbf {b} } , we wish to find a vector x {displaystyle mathbf {x} } such that The standard approach is ordinary least squares linear regression. However, if no x {displaystyle mathbf {x} } satisfies the equation or more than one x {displaystyle mathbf {x} } does—that is, the solution is not unique—the problem is said to be ill posed. In such cases, ordinary least squares estimation leads to an overdetermined, or more often an underdetermined system of equations. Most real-world phenomena have the effect of low-pass filters in the forward direction where A {displaystyle A} maps x {displaystyle mathbf {x} } to b {displaystyle mathbf {b} } . Therefore, in solving the inverse-problem, the inverse mapping operates as a high-pass filter that has the undesirable tendency of amplifying noise (eigenvalues / singular values are largest in the reverse mapping where they were smallest in the forward mapping). In addition, ordinary least squares implicitly nullifies every element of the reconstructed version of x {displaystyle mathbf {x} } that is in the null-space of A {displaystyle A} , rather than allowing for a model to be used as a prior for x {displaystyle mathbf {x} } .Ordinary least squares seeks to minimize the sum of squared residuals, which can be compactly written as where ‖ ⋅ ‖ 2 {displaystyle |cdot |_{2}} is the Euclidean norm.