Transcript
one and okay. Here is lecture 16. And if you remember, I ended up the last lecture with this formula for what I called a projection matrix. And maybe I could just recap for a minute. What is that What is that magic formula doing? For example, it's supposed to be it's supposed to produce a projection. If I multiply by a B, so I take P times B, I'm supposed to project that vector B to the nearest point in the column space. Okay. Can I just One way to recap is to take the two uh extreme cases. Suppose the vector B is in the column space. Then what do I get when I apply the projection P? So I'm projecting into the column space, but I'm starting with a vector in this case that's already in the column space.
So of course, when I project it, I get B again, right. And we want to I want to show you how that comes out of this formula. Let me do the other the other extreme. Suppose that vector is perpendicular to the column space. So imagine this column space as a plane, and imagine B is sticking straight up perpendicular to it. What's the nearest point in the column space to B in that case? So what's the projection onto the plane, the nearest point in the plane, if the vector B that I'm looking at is got no component in the column space. It's sticking completely 90° with it. Then PB should be zero, right.
So those are the two extreme cases. The average vector has a component P in the column space and a component perpendicular to it. And what the projection does is it kills this part and it preserves this part. Okay. Can we just see why that's true? Just just that formula uh ought to work. Let me start with this one. What what vectors are in the are are perpendicular to the column space? Uh how do I see that I really get zero? I have to think, what does it mean for a vector B to be perpendicular to the column space? So if it's perpendicular to all the columns, then it's in some other space. We've got our four spaces.
So the reason I do this is it's perfectly using what we know about our four spaces. What's What vectors are perpendicular to the column space? Those are the guys in the null space of A transpose, right? That's the first section of this chapter. That's the key geometry of these spaces. If I'm perpendicular to the column space, I'm in the null space of A transpose. Okay. So if I'm in the null space of A transpose, and I multiply this big formula times B, so now I'm getting PB. This is now the projection PB. Do you see that I get zero? Of course I get zero. Right at the end there, A transpose B will give me zero right away.
So that's why that zero is here. Cuz if I'm perpendicular to the column space, then I'm in the null space of A transpose, and A transpose B is zilch. Okay. What about the other possibility? What What How do I see that this formula gives me the right answer if B is in the column space? So what's a typical vector in the column space? It's a combination of the columns. How do I write a combination of the columns? So So tell me, how would I write you know, your everyday vector that's in the column space? It would have the form A times some X, right? That's what's in the column space, A times something.
That makes it a combination of the columns. So these B's were in the in the null space of A transpose, these guys in the column space, those B's are AX's. Right? If B is in the column space, then it has the form AX. I'm going to stick that on the quiz or the final for sure. That you have to real- realize, cuz we've said it like a thousand times, that the things in the column space are vectors A times X. Okay. And do you see what happens now if we use our formula? There's an A transpose A gets canceled by its inverse. We're left with an A times X. So the result was AX, which was B. Do you see that it works?
This is that whole business. Cancel, cancel, leaving AX. And AX was B. So that turned out to be B in this in this case. Okay. So geometrically, what we're seeing is we're taking a vector we we've got the we've got the column space. And perpendicular to that is the null space of A transpose. And our typical vector B is out here. There's zero, so there's our typical vector B. And what we're doing is we're projecting it to P. And the And of course, at the same time, we're finding the other part of it, which is E. So the diff- the the two pieces, the projection piece and the error piece, add up to the original B.
Okay. That's That's like what our matrix does. So this is P P is This P is AB is Sorry, is PB. It's the projection applied to B. And this one is Okay, that's a projection, too. That's a projection down onto that space. What's a good formula for it? Suppose I ask you for the projection of the of the projection matrix onto the this space, this perpendicular space. So if this projection was P, what's the projection that gives me E? It's uh what I want is to get the rest of the vector, so it'll be just I minus P. times B. That's a projection, too. That's the projection onto the perpendicular space.
Okay. So if P is a projection, I minus P is a projection. If P is symmetric, I minus P is symmetric. If P squared equals P, then I minus P squared equals I minus P. It's just the algebra is is only doing what your your uh uh picture is is completely telling you. It's But the algebra leads to this uh expression. That expression for P given given uh a basis for the subspace. Given the matrix A whose columns are a basis for our column space. Okay. That's recap, cuz you you need to see that formula more than once. And now can I pick up on using it? So now and I and the It's It's like uh Let me do that again.
I'll do go go right through a problem that I started at the end, which is find a best straight line. You remember that problem? I I picked a particular uh set of points. They weren't specially brilliant. T equal 1 2 3. The heights were 1 2 and then 2 again. So they were heights were that point that point, which makes it look like I've got a nice 45° line, but then the third point didn't lie on the line. And I wanted to find the best straight line. So I'm looking for the the this line Y equals C plus DT. And it's not going to go through all three points, cuz no line goes through all three points.
So I'm going to pick the best line. The the best being the one that makes the the overall error as small as I can make it. Now I have to tell you, what is that overall error? And uh uh Cuz that that determines what's the winning line. If we don't know I mean we have to decide what we mean by the error. What and then we minimize and we find the right the best C and D. Okay. So, if again if if I went through this if I went through that point, I would solve the equation C plus D equals one. Because at T equal to one, I'd have C plus D and it would come out right. If if if it went through this point, I'd have C plus 2D equal to two.
Because at T equal to two, I would like to get the answer two. At the third point, I have C plus 3D because T is three, but the the answer I'm shooting for is two again. So, those are my three equations. And they don't have a solution. But they've got a best solution. What do I mean by best solution? So, let me take time out to remember what I'm talking about for best solution. So, this is my equation B. A is this matrix 1 1 1 1 2 3. X is my only have two unknowns C and D. And B is my right hand side 1 2 3. Okay. No solution. Three I have a 3 by 2 matrix. I do have two independent columns. So, I do have a basis for the column space.
Those two columns are independent. They're a basis for the column space, but the column space doesn't include that vector. So, best possible in this What would best possible mean? The the way that comes out to linear equations is I I want to minimize the sum of these. I'm going to make an error here. I'm going to make an error here. I'm going to make an error there. And I'm going to sum and square and add up those errors. I So, it's a sum of squares. It's a least squares solution I'm looking for. So, if I those errors are the difference between AX and B. That's what I want to make small. And the way I'm measuring this, this is a vector, right?
This is E1 E2 E3. AX minus B, this is the this is the E the error vector. And small means its length. The length of that vector. That's what I'm going to try to minimize. And it's convenient to square. If I make something small, I make This is a This is a non This is a never negative quantity, right? The length of that vector. The length will be zero exactly when the when I have the zero vector here. That's exactly the case when I can solve exactly. B is in the column space. All great. But I'm not in that case now. I'm going to have an error vector E. What's this error vector in my picture? I guess what I'm trying to say is there's there's two pictures of what's going on.
There's two pictures of what's going on. One picture is in this in this is the three points and the line. And in that picture, what are the three errors? The three errors are what I missed by in this equation. So, it's this this little bit here. That vertical distance up to the line. There's one Sorry, there's one and there's C plus D and it's that difference. Here's two and here's C plus 2D. So, vertically, it's that distance. That's that's That that little error there is E1. This little error here is E2. This little error coming up is E3. E3. And what's my to- overall error is E1 squared plus E2 squared plus plus E3 squared.
That's what I'm trying to make small. I Some statisticians This is a big part of statistics. Fitting straight lines is a big part of science and specifically statistics where the right word to use would be regression. I'm doing regression here. Linear regression. And I'm and I'm using this sum of squares as the measure of error. Again, some statisticians would be They would say, "Okay, I'll solve that problem because it's the clean problem. It leads to a beautiful linear system." But they would be a little careful about least squares for in this case. If one of these points was way off. Suppose I had a measurement at T equal zero that was way off.
Well, would the straight line with the best line be the same if I had this fourth point? Suppose I have this fourth data point. No, certainly the line would it wouldn't be that That wouldn't be the best line. Because that line would have a giant error and when I squared it, it would be like way out of sight compared to the others. So, the this This would be called by statisticians an outlier. And they would not be happy to see the whole problem turned topsy-turvy by this one outlier, which could be a mistake after all. So, they wouldn't So, they wouldn't like maybe squaring. If there were outliers, they would want to identify them.
Okay. I'm not going to I'm I'm taking the the but the but I I don't want to suggest that least squares isn't used. It's the most used, but it's not exclusively used because it's a little overcompensates for outliers. But because of that squaring. Okay. So, suppose we don't have this guy. We just have these three equations. And I want to make minimize this error. Okay. Now, what I said is there's two pictures to look at. One picture is this is this one. The three points, the best line. And the errors. Now, on this picture, what are these points on the line? The points that are really on the line.
So, they're they're points. Let me call them P1, P2, and P3. Those are three numbers. So, this this height is P1. This height is P2. This height is P3. And what are those guys? Suppose those were the three values instead of There's B1. Everybody's seeing all these Sorry, the my art is as usual not the greatest, but there's the given B1, the given B2, and the given B3. I promise not to put a single letter more on that picture. Okay. There's There's B1. P1 is the one on the line and E1 is the distance between. And same at points two and same at points three. Okay. So, what's up? What's up with those P's?
P1, P2, P3, what are they? They're the components. They lie on the line, right? They're the They're the points which if I if if instead of 1 2 2, which were the B's. Suppose I put P1, P2, P3 in here. I'll figure out in a minute what those numbers are. But I just want to get the picture of what I'm doing. If I put P1, P2, P3 in those three equations, what would be good about the the three equations? I could solve them. A A line goes through the P's. So, the P1, P2, P3 vector, that's in the column space. That is a combination of these columns. It's the closest combination. It's this picture. See, I've got the two pictures.
Like here's the picture uh that shows the points. This is a a picture in a in a blackboard plane. Here's a picture that's showing the vectors. The vector B, which is in this case in this example is the vector 1 2 2. The column space is in this case spanned by the Well, you see A there. The column space of the matrix 1 1 1 1 2 3. And this picture shows the nearest point. There's the that point P1 P2 P3, which I'm going to compute before the end of this hour, is the closest point in the column space. Okay. Let me Let me not I don't dare leave it any longer. Like can I just compute it now? So, I want to compute find P.
All right, find P. Find X, which is CD. Find P and P. Okay, and I really should put these little hats on to to remind myself that they're the estimated. The best The best line, not the perfect line. Okay. Okay, how do I proceed? Let's just run through the mechanics. What's the equation for X? The equa- X hat. The equation for that is A transpose A X hat equals A transpose X. A A transpose B. The most I I I will venture to call that the most important equation in statistics. And in estimation. And whatever you're Wherever you've got error and noise, this is the estimate that you use first. Okay.
Whenever you're fitting things by a few parameters, that's the equation to use. Okay, let's solve it. Uh what is A transpose A? So, I have to figure out what these matrices are. 111123 and 111123. That gives me some matrix. That gives me a matrix. What do I get out of that? Three uh six six and four and nine 14. Okay. And what do I expect to see in that matrix that I do see it just before I keep going with the calculation? I expect that matrix to be symmetric. I expect it to be invertible. And near the end of the course, I'm going to say I expect it to be positive definite, but that's uh I that's a future fact about this crucial matrix A transpose A.
Okay, and now let me figure A transpose B. So, let me Can I tack on B as an extra column here? 122 and tack on the extra A transpose B is looks like five and one and four and six 11. So, I think my equations are 3C + 6D = 5 and 6D plus four uh 6C plus 14D is 11. Can I just for safety see if I did that right? 111 * 122 is five. 123, that's one four and six 11. Looks good. These are my equations. That's my my They're called the normal equations. I'll just write that word down because it So, I solve them. I solve that for C and D. I I would like to Before I solve them, could I do one thing that's on the that's just about here?
I would like to I'd like to find these equations from calculus. I'd like to find them from this minimizing thing. So, what's the first error? The first error is what I missed by in the first equation. C plus D minus one squared. And the second error is what I missed in the second equation. C plus 2D minus two squared. And the third error squared is C plus 3D minus two squared. That's my overall squared error that I'm trying to minimize. Okay. So, How do How would you minimize that? Okay, linear algebra has has given us the equations for the minimum. But we could use calculus, too. That's That's a function of two variables, C and D.
And we're looking for the minimum. So, how do we find it? Directly from calculus, we take partial derivatives, right? We've got two variables, C and D. So, take the partial derivative with respect to C and set it to zero, and you'll get that equation. Take the partial derivative with respect I'm I'm not going to write it all out, just you will. The partial derivative with respect to D, it You know, it's going to be linear. That's the beauty of least squares, is that if if I have the square of something and I take its derivative, I get something linear. And this is what I get. So, this is the derivative of the error with respect to C being zero, and this is the derivative of the error with respect to D being zero.
Wherever you look, these equations keep coming. So, now I guess I'm going to solve it. What will I do? I'll subtract I'll do elimination, of course, cuz that's only thing I know how to do. Two of these away from this would give me Let's see, six. So, would that be two Ds equals one? Huh. So, I wasn't I I was afraid these numbers were going to come out awful. But if I take two equa- two of those away from that, the equation I get left is 2D equals one. So, I think D is a half. And C is whatever back substitution gives. 6D is three. So, 3C plus three is five. I'm doing back substitution now, right?
Three Can I do it in light letters? 3C plus that 6D is three equals five. So, 3C is two. So, I think C is 2/3. 1/2 and 2/3. So, the best line the best line is the best line is the constant 2/3 plus 1/2 T. And am I Is my picture more or less right? Let me write Let me copy that best line down again. 2/3 and 1/2. Let me I'll put in the 2/3 and the half. Okay. So, what's this P1? That's the value at T equal to one. At T equal to one, I have 2/3 plus 1/2, which is What's that? 4/6 and 3/6. So, P1 Oh, I promised not to write another thing on this. I'll Look, I'll I I'll erase P1 and I'll put 7/6. Okay.
And Yeah, it's above one. And E1 is 1/6. Right. You see it all, right? What's P2? Okay, at the point T equal to two, where's my line here? At T equal to two, it's 2/3 plus one, right? That's 5/3. 2/3 and T is two, so that's 2/3 and one make 5/3. And that's sure enough, that's smaller than the exact two. And then the final P3, when T is three Oh, what's 2/3 plus 3/2? It's the same as 3/2 plus 2/3. It's um So, maybe 4/6 and 9/6, maybe 13/6. Okay. And again, look. Oh, look at this. Okay. You have to admire the beauty of this answer. Uh this what's this first error? So, here are the errors, E1, E2, and E3.
Okay, what was that first error, E1? Well, if if we decide the errors counting up, then it's 1/6. And the last error, 13/6 minus the correct two is 1/6 again. And what's this error in the middle? Let's see, The correct answer was two two. And we got 5/3 and it's the other direction minus 1/3 minus 2/6. That's our error vector. In our picture, in our other picture, here it is. We just found P and E. E is this vector 1/6 minus 2/6 1/6. And P is this guy. Well, maybe I have the signs of E wrong. I think I have. Let me Let me fix them. Because I would like this 1/6 I would like this plus the P to give the original B.
I want P plus E to match B. So I want minus a sixth plus 7/6 to give the correct B equal 1. Okay. Now Okay, my my I'm going to take a deep breath here and ask what do we know about this error vector E? You have seen now this whole problem worked completely through and I even think the numbers are right. So this P So let me I'll write I'll write out what if I can put it down here. B is P plus E. B I believe was 1 2 2. The nearest point had 7/6 Um what were the others? 5/3 and 13/6. And the E vector was minus a sixth 2/6 1/3 in other words and minus a sixth. Okay. Tell just like tell me some stuff about these two vectors.
Tell me something about those two vectors. Well, they had to be right. Great. Okay. What else? What else about those two vectors? The P the the projection vector P and the error vector E. What else do you know about them? They're perpendicular. Right. Do we dare verify that? Can we take the dot product of those vectors? I'm like getting like minus 7 over 36. Can I change that to 10/6? Oh God, come out right here. Uh minus 7 over 36 plus 20 over 36 minus 13 over 36. Thank you God. Okay. And what else should we know about that vector? Actually, we know I I I got to say we know even a little more.
This vector E is perpendicular to P. But it's perpendicular to other stuff, too. It's perpendicular not just to this guy in the column space. This is in the column space for sure. This is perpendicular to the column space. So like give me another vector it's perpendicular to. Another cuz it's perpendicular to the whole column space, not just to this this particular projection that that is in the column space. But it's it's perpendicular to other stuff. Whatever is in the column space. So tell me another vector in the col- Oh well, I've written down the matrix. So tell me another vector in the column space.
Pick a nice one. 1 1 1. It's what everybody's thinking. Okay. 1 1 1 is in the column space and this guy is supposed to be perpendicular to 1 1 1. Is it? Sure. If I take the dot product with 1 1 1, I get minus a sixth plus 2/6 minus a sixth. Zero. And it's perpendicular to 1 2 3. Cuz if I take the dot product with 1 2 3, I get minus 1 plus 4 minus 3 zero again. Okay. Do you see the I hope you see the two uh pictures. The picture here for vectors and the picture here for the best line and it's the same picture. Just like it's in This one's in the plane and it's showing the line. This one never did show the line.
This In this picture, C and D never showed up. In this picture, C and D were you know, they determined that line. But the two are exactly the same. C and D is the combination of the two columns that gives P. Okay. So that's uh least squares and the special but most important example of fitting by straight line. So the homework that's coming then Wednesday ask you to fit by straight lines. So you're just going to end up solving the key equation. You're going to end up solving that key equation and then P will be A X hat. That's it. Okay. Now can I put in a little piece of linear algebra that I mentioned earlier, mentioned again, but I never did write.
And I've I should do it right. It's the It's about this matrix A transpose A. There. I was sure that that matrix would be invertible. And of course, I wanted to be sure it was invertible because I planned to solve the system with that with that matrix. So and I announced like before the as the chapter was just starting I announced that it would be invertible. But now I Can I come back to that? Okay. So So what I said was that if A has independent columns then A transpose A is invertible. And I would like to first repeat that important fact that that's the requirement that makes everything go here.
It's the It's this independent columns of A that that guarantees everything goes through. And think why. Why does Why does this matrix A transpose A Why is it invertible if the columns of A are independent? Okay. There's I guess I'm I'm I'm wondering So So if it wasn't invertible So I want to prove that. If If it isn't invertible, then what? I want to reach uh I want to follow that follow that line of of thinking and see what I come to. Suppose So proof. Suppose A transpose A X is zero. I'm trying to prove this. This is now to prove. I I don't like hammer away at too many proofs in this course.
But this is like the central fact and it brings in all the stuff we know. Okay. So I'll start the proof. Suppose A transpose A X is zero. What what and I and I'm aiming to prove A transpose A is invertible. So what do I want to prove now? So I'm I'm aiming to prove this fact. I'll use this and I'm aiming to prove that this matrix is invertible. Okay. So if I suppose A transpose A X is zero, then what conclusion do I want to reach? I'd I'd like to know that X must be zero. I want to show X must be zero. To show now, to prove X must be the zero vector. Is Is that right? That's what we worked in the previous chapter to understand.
That a matrix was invertible when its null space is only the zero vector. So, that's what I want to show. How come if A transpose AX is zero, how come X must be zero? What's going to be the reasoning? Actually, I have two ways to do it. Let me show you one way. This is So, here trick. Take the dot product of both sides with X. So, I'll multiply both sides by X transpose. X transpose A transpose AX equals zero. I shouldn't have written trick. That makes it sound like just a dumb idea. Brilliant idea, I should have put. Okay. I'll just put idea. Okay. Now, I got to that equation X transpose A transpose AX equals zero and I'm hoping you can see a a the right way to to look at that equation.
It What can I What can I conclude from that equation? That if I have X transpose A Well, what is X transpose A transpose AX? Does that Do you see the Do you see what it what it what it's giving you? I'm It's again going to be putting in parentheses. I'm looking at AX, and what am I seeing here? It's transpose. So, I'm seeing here this is AX transpose AX equaling zero. Now, if AX transpose AX, so like let's call it Y or something. If Y transpose Y is zero, what does that tell me? That the vector has to be zero, right? And this is the length squared. That's the length of the vector AX squared. That's AX times AX.
So, I conclude that AX has to be zero. Well, I'm getting somewhere. Uh now that I know AX is zero, now I'm going to use my little hypothesis. Somewhere every mathematician has to use the hypothesis. Right? Now, if A has independent columns and we've we're at the point where AX is zero, what does that tell us? I could I mean, that could be like a a fill-in question on the final exam. If A has independent columns and if AX equals zero, then what? Please say it. X is zero. Right. Which was just what we wanted to prove. That Do you you see why that is? If AX equals zero, now we're using Here we used This was the square of something.
So, I I'll put in little parentheses the the the observation we made. That was a square, which is zero, so the thing has to be zero. Now we're using the hypothesis of independent columns that A has independent columns. If A has independent columns, this is telling me X is in its null space, and the only thing in the null space of such a matrix is the zero vector. Okay. So, that's the the argument, and you see how it really used our understanding of the of the uh sub of the null space. Okay. That's that's great. All right. So, where are we then? That that board is like the backup theory that tells me that this matrix had to be invertible because these columns were independent.
Okay. Uh there's one case of independent There's one case where the geometry gets even better. When the When I There's one case when columns are sure to be independent. And let me put that Let me write that down, and that'll be the subject for next time. Columns are sure are certainly independent, definitely independent. If if they're perpendicular. Oh, I've got to rule out the zero column. Let me give them all length one, so so they can't be zero. If they are if they are perpendicular unit vectors. Like the vectors 1 0 0 0 1 0 and 0 0 1. Those vectors are unit vectors. They're perpendicular, and they certainly are independent.
And what's more, suppose they're Oh, it's so nice. I mean, what is A transpose A for that matrix? For the matrix with these three columns, it's the identity. So, here's the key to the to the lecture that's coming. If we're dealing with perpendicular unit vectors, and the word for that will be ortho See, I could have said orthogonal, but I said perpendicular. Ortho, and this unit vectors gets put in as the word normal. Orthonormal vectors. Those are the best columns you could ask for. Matrices with whose columns are orthonormal. They're perpendicular to each other, and they're unit vectors. Well, they don't have to be those three.
Let me do a final example over here. Uh How about one at an angle like that, and one at 90 degrees? That vector would be cos theta sin theta. A unit vector. And this vector would be minus sin theta cos theta. That is our absolute favorite pair of orthonormal vectors. They're both unit vectors, and they're perpendicular. That angle is 90 degrees. So, like our job next time is first to see why orthonormal vectors are great, and then to make vectors orthonormal by picking the right basis. Okay. See you. Thanks.