Download Linear Systems. LU.

Tutorial 4. Least Squares.
1. Reminder: Least squares solution
A x  b  min A x  b 2  minA x  bT A x  b
2
x

min x T A T A x  x T A T b  b T A x  b T b
x
Taking the derivative we obtain:
The normal equation:
T

2A T A x  2A T b  0
A T Ax  A T b
 
T
If A A  0 then there is a unique global solution: x*  A A
1
ATb
If A is square and invertible, then the unique solution is x*  A 1 b
2
General Quadratic Equation (K positive semi-definite):

min x T K x  2x T f  c
x
The derivative this time is given by:

2K x  2f  0
If K  0 then there is a unique global solution: x*  K 1 f
If K is singular, any solution satisfying K x*  f leads to the
minimal value, and there are infinitely many of those.
3
2. Derivatives of Matrix-Vector Expressions
Given the function:

T

T
T
T

f ( x)  x A x  2b x  c x Dx  2e x  f

 

α( x )
With symmetric matrices:
β( x )
T
A  A;
T
D D
f ( x )  2A x  b β( x )  2Dx  e α( x )
The minimizer of f(x) should satisfy the cubic equation:
β(x)A  α(x)Dx  β(x)b  α(x)e 
4
(continue)
For the special case:

T
T
f ( x)  x A x  2b x  c
We obtain the requirement:

2
αx A x  b   0
Thus, the possible solutions (zeroing the
gradient of the function) are:
1. αx   0
2. A x  b  0
If x αx   0, the first solution is
meaningless, and second solution is the
correct one.
If x αx   0 , the first solution is the
correct one, and the second represents a
local maximum!!!!!!
5
Example:

T
T
f ( x)  x A x  2b x  c
A=[4 1; 1 2];
b=[2;1];

2
c=-50;
for ind1=1:100
x1=ind1/10-5;
for ind2=1:100
x2=ind2/10-5;
h(ind1,ind2)=[x1,x2]*A*[x1;x2]-b'*[x1;x2]+c;
end;
end;
figure(1); meshc(h);
figure(2); meshc(h.*h);
axis([0 100 0 100 0 3000]); caxis([0 3000]);
Note: In this case, the value of (x) can get
below zero, as the next figures show
6
7
3. Perform: Polynomial fitting
50
40
x = (-3:.06:3)'; PointNum=size(x,1);
C=[-3 1 2 -1];
y0=C(1)+C(2)*x+C(3)*x.^2+C(4)*x.^3;
y=y0+10*randn(PointNum,1);
figure(1); clf; plot(x,y0,'r');
hold on; plot(x,y,'.b');
30
20
10
0
-10
-20
Lets search by curve fitting the best
Polynomial of orders 2-10, and see how they
perform (visually and by evaluating the error):
-30
-3
-2
-1
0
1
2
3
A=[ones(PointNum,1),x,x.^2,x.^3,x.^4,x.^5,x.^6,x.^7,x.^
8,x.^9,x.^10];
c10=inv(A'*A)*A'*y;
c8=inv(A(:,1:8)'*A(:,1:8))*A(:,1:8)'*y;
c6=inv(A(:,1:6)'*A(:,1:6))*A(:,1:6)'*y;
c4=inv(A(:,1:4)'*A(:,1:4))*A(:,1:4)'*y;
c2=inv(A(:,1:2)'*A(:,1:2))*A(:,1:2)'*y;
8
(continue)
disp(sqrt(mean((y0-A*c10).^2)));
figure(2); clf;
plot(x,y0,'r'); hold on;
plot(x,y,'.b');
50
plot(x,A*c10,'g');
40
2.2604
30
20
10
0
-10
-20
-30
-3
-2
-1
0
1
2
3
9
(continue)
disp(sqrt(mean((y0-A*c8).^2)));
figure(3); clf;
plot(x,y0,'r'); hold on;
plot(x,y,'.b');
50
plot(x,A(:,1:8)*c8,'g');
40
1.9877
30
20
10
0
-10
-20
-30
-3
-2
-1
0
1
2
3
10
(continue)
disp(sqrt(mean((y0-A*c6).^2)));
figure(4); clf;
plot(x,y0,'r'); hold on;
plot(x,y,'.b');
50
plot(x,A(:,1:6)*c6,'g');
40
1.8918
30
20
10
0
-10
-20
-30
-3
-2
-1
0
1
2
3
11
(continue)
disp(sqrt(mean((y0-A*c4).^2)));
figure(5); clf;
plot(x,y0,'r'); hold on;
plot(x,y,'.b');
50
plot(x,A(:,1:4)*c4,'g');
40
0.8114
30
20
10
0
-10
-20
-30
-3
-2
-1
0
1
2
3
12
(continue)
disp(sqrt(mean((y0-A*c2).^2)));
figure(6); clf;
plot(x,y0,'r'); hold on;
plot(x,y,'.b');
50
plot(x,A(:,1:2)*c2,'g');
40
6.9345
30
20
10
0
-10
-20
-30
-3
-2
-1
0
1
2
3
13