Transcript
Okay. Uh Cameras are rolling. This is lecture 14. Starting a new chapter. Chapter about orthogonality. What it means for vectors to be orthogonal. What it means for subspaces to be orthogonal. What it means for bases to be orthogonal. So, a 90° This is This is a 90° chapter. So, what does it mean? Uh let me let me jump to subspaces because I've drawn here the big picture. This is the This is the 1806 picture here. And uh Hold it down, guys. Uh so, this is the picture and we know a lot about that picture already. We know the dimension of every subspace. We know that these dimensions are r and n minus r.
We know that these dimensions are r and m minus r. What we What I want to show now is what this figure is saying that the angle betw- wait, the figure is like just my attempt to draw what I'm now going to say that the angle between these subspaces is 90°. And the angle between these subspaces is 90°. Now, I have to say, what does that mean? What does it mean for a subspaces to be orthogonal? But I hope you appreciate the beauty of this picture. That that those subspaces are going to come out to be orthogonal. Those two and also those two. So that's like one point one important point to step forward in understanding those subspaces.
We knew what each subspace was like. We could compute bases for them. Now we know more. Or we will in a few minutes. Okay, I have to say first of all, what does it mean for two vectors to be orthogonal? So let me start with that. Orthogonal vectors. The word orthogonal is is just another word for perpendicular. It means that in n-dimensional space the angle between those vectors is 90°. It means that they form a right triangle. It It even means that the going way back to the Greeks that this uh angle that this triangle of vector x of vector y and the vector x + y of course that'll be the hypotenuse.
So what was it the Greeks figured out and it's neat. It's the fact that the So these are orthogonal. This is a right angle. If So So let me put the great name down, Pythagoras. I'm I'm looking for What am I looking for? I'm looking for the condition if you give me two vectors, how do I know if they're orthogonal? How can I tell two perpendicular vectors? And actually, you probably know the answer. Let me write the answer down. Orthogonal vectors, what's the test for orthogonality? I take the dot product, which I tend to write as X transpose Y, cuz that's a row times a column, and that matrix multiplication just gives me the right thing, that X1Y1 + X2Y2 and so on.
So, these vectors are orthogonal if this result X transpose Y is zero. That's the test. Okay. Can I connect that to other things? I mean, it's like it's amazing. It's It's just beautiful that here we have and we're in N dimensions, we've got a couple of vectors, we want to know the angle between them, and the right thing to look at is the simplest thing that you could imagine, the dot product. Okay. Now, why? So, I'm answering the question now, why let's just some Let's add some uh justification to this fact that that's the test. Okay. So, Pythagoras would say we've got a right triangle if that length squared plus that length squared equals that length squared.
Okay. Can I write it as X squared plus y squared equals x plus y squared. Now, don't please don't think that this is always true. This is only going to be true in this It's going to be equivalent to orthogonality. For other triangles, of course, it's not true. For other triangles, it's not. But for a right triangle, somehow that fact should connect to that fact. Can we just make that connection? What's the connection between this test for orthogonality and this statement of orthogonality? Well, I guess I have to say, what is the length squared? So, let's continue on the board underneath with that equation.
Give me another way to express the length squared of a vector. And let me just give you a vector. The vector 1 2 3. That's in three dimensions. What is the length squared of the vector x equal 1 2 3? So, how do you find the length squared? Well, really you're just you want the length of that vector that goes along one, up two, and out three. And And we'll come come back to that right triangle stuff. The length squared is This is exactly x transpose x. Whenever I see x transpose x, I know I've got a number that's positive. It's a length squared unless it unless x happens to be the zero vector. That's the one case where the length is zero.
So, right this is just X1 squared plus X2 squared plus so on plus Xn squared. So, one in the example I gave you, what was the length squared of that vector 1 2 3? So, you square those, you get 1 4 and 9, you add, you get 14. So, the vector 1 2 3 has length 14. So, let me just let me just put down a vector here. Let X be the vector 1 2 3. Let me cook up a vector that's orthogonal to it. So, what's the vector that's orthogonal So, let write down here X squared is 1 plus 4 plus 9 14. Let me cook up a vector that's orthogonal to it. We'll get Right, that that those two vectors are orthogonal. The length of Y squared is five.
And X plus Y is 1 and 2 making 3, 2 and minus 1 making 1, 3 and 0 making 3, and the length of this squared is 9 plus 1 plus 9 19. And sure enough I haven't proved anything. I've just like checked to see that my my my test X transpose Y equals 0, which is true, right? Everybody sees that X transpose Y is 0 here. That that's maybe the main point that you should get really quick at doing X transpose Y. So, it's just this plus this plus this, and that's zero. And sure enough, that clicks with uh 14 + 5 agreeing with 19. Now, let me just do that in letters. So, that's Y transpose Y. And this is X + Y transpose X + Y.
Okay. So, I'm looking again, this isn't always true. I repeat. This is going to be true when we have a right angle. And let's just Well, of course, I'm just going to simplify this stuff here. There's an X transpose X there, and there's a Y transpose Y there, and there's an X transpose Y, and there's a Y transpose X. I knew I could do that simplification because I'm just doing matrix multiplication, and I've just followed the rules. Okay. So, X transpose X is cancelled, Y transpose Y is cancelled, and what about these guys? What can What can you tell me about the inner product of X with Y and the inner product of Y with X?
Is there a difference? I think if we if while we're doing real vectors, which is all we're doing now, there isn't a difference. There's no difference. If I take X transpose Y, that'll give me zero. If I took Y transpose X, I would have the same X1Y1 and X2Y2 and X3Y3. It would be the same. So, this is this is this is the same as that. This is really all I'll knock that guy out and say two of these. So, actually that's the this this equation boiled down to this thing being zero. Right? Everything else canceled and this equation boiled down to that one. So, that's really all I wanted. I just wanted to check that Pythagoras for a right triangle led me to this Now, of course, I canceled the two now, no problem.
2x transpose y = 0 as the test. Fair enough. Okay. You knew it was coming. The dot product of orthogonal vectors is zero. It's just I just want to say that's really neat. It comes that it comes out so well. All right. Now, what about So, now I know if two vec what it means when two vectors are orthogonal. By the way, what about if one of these guys is the zero vector? Suppose x is the zero vector and y is whatever. Are they orthogonal? Sure. In math, the one thing about math is you're supposed to follow the rules. So, you're supposed to If If x is the zero vector, you're supposed to take the zero vector dotted with y and of course, you get always get zero.
So, just so we're all sure, the zero vector is orthogonal to everybody. But what I want to what I I want to uh speak about is subspaces. What does it mean for me to say that some subspace is orthogonal to some other subspace? So, okay. Now, I've got to write this down. So, cuz we're defining definition of subspace S is orthogonal, so to subspace let's say T. So, I've got a couple of subspaces. And what should it mean for those guys to be orthogonal? It's just a na- it's sort of what's the natural extension from orthogonal vectors to orthogonal subspaces? Well, and in particular let's think of some orthogonal subspaces like uh uh this wall.
I I let's say in three dimensions. So, the blackboard extended to infinity, right? Is a is a subspace, a plane, a two-dimensional subspace. It's a little bumpy, but anyway, it's a think of it as a subspace. Um let me take the floor as another subspace. Again, it's not a great subspace. MIT only built it like so-so, but uh and I'll put the origin right here. So, the origin of the world is right there. Okay. Thereby giving linear algebra its proper importance in in this. Okay. So, there's one subspace, there's another one, the floor. And are they orthogonal? What does it mean for two subspaces to be orthogonal in a and in that special case, are they orthogonal?
All right. Let let's finish this sentence. What does it mean? Means We have to We have to know what we're talking about here. So, what would be a reasonable idea of orthogonal? Well, let me put the right thing up. It means that every vector in S every vector in S is orthogonal to to What am I going to say? Every vector in T. That That's a reasonable and it's a good and it's the right definition for two subspaces to be orthogonal. But I just want you to see, hey, what does that mean? So, answer the question about the the blackboard and the floor. Are those two subspaces They're They're two-dimensional, right?
And we're in R3. It's like the XZ plane or something and the XY plane. Are they orthogonal? Is every vector in the blackboard orthogonal to vector in the floor? Starting from the origin right there. Yes or no? I could take a vote. I'll be getting some yeses and some nos. No is the answer. They're not. You can tell me a vector in the blackboard and a vector in the floor that are not orthogonal. Uh What's What's Well, you can tell me quite a few, I guess. Maybe like I could take some 45° guy in the blackboard and some thing in the floor that aren't at 90°, right? In fact, even more, you could tell me a vector that's in both the blackboard plane and the floor plane, so it's certainly not orthogonal to itself.
So, for sure, those two planes aren't orthogonal. What would that be? So, what's a vector that's in the black- in in both of those planes? It's this guy running along the crack there, where the intersection the intersection. A vector Look, if two subspaces meet some vector, well, then for sure they're not orthogonal because that vector is in one and it's in the other and it's not orthogonal to itself unless it's zero. So, the only I mean, so orthogonal If for me to say these two subspaces are orthogonal, first of all, I'm certainly saying that they don't intersect in any uh nonzero vector. But also, I mean more than that just not intersecting isn't good enough.
So, give me an example Oh, let's say in the plane. Well, well. Well, when do we have orthogonal subspaces in the plane? Yeah, tell me in the plane. So, we don't there aren't that many different subspaces in the plane. What What have we got in the plane that's as possible subspaces? The zero vector, real small, uh a line through the origin, or the whole plane. Okay. Now, so when is a line through the origin orthogonal to the whole plane? Never, right? Never. When is a line through the origin orthogonal to the zero subspace? Always. Right. When is a line through the origin orthogonal to a different line through the origin?
Well, that's the case that we all have a clear picture of. They the two lines have to meet at 90°. They have only the So, that So, that's like this simple case I'm talking about. There's one subspace, there's the other subspace, they only meet at zero, and they're orthogonal. Okay. Now, so we now know what it means for two subspaces to be orthogonal. And now I want to say that this is true for the row space and the null space. Okay. So, that's the that's the neat fact. So, row space is orthogonal to the null space. Now, how did I come up with that? But, you see that like it's right. That means that these that these subspaces are just the right things.
They're just cutting the cutting the whole space up into two perpendicular subspaces. Okay, so what? Well, what have I got to work with? All I know is the null space. The null space has vectors that solve Ax = 0. So, this is a this is a guy x. x is in the null space. Then, Ax is 0. So, why is it orthogonal to the rows of A? If I write down Ax = 0, which is all I know about the null space, then I guess I want you to see that that's telling me just that equation right there is telling me that the rows of A, let me write it out. There's row one of A. Row two. Row m. of A. That that's A. And it's multiplying x and it's producing zero.
Okay. Written out that way, you'll see it. So, I'm saying that a vector in the row space is perpendicular to this guy x in the null space. And you see why? Cuz this equation is telling you that the row one of A multiplying, that's a that's a dot product, right? Row row one of A dot product with this X is producing this zero. So, X is orthogonal to the first row. And to the second row. Row two of A X is giving that zero. Row M of A times X is giving that zero. So, X is The equation is telling me that X is orthogonal to all the rows. Right? It's just sitting there. That's all we we it had to be sitting there cuz we didn't know anything more about the null space than this.
And now I guess to be totally complete here, I've now checked that X is orthogonal to each separate row. But what else strictly speaking do I have to do? The the what to show that those subspaces are orthogonal, I have to take this X in the null space and show that it's orthogonal to every vector in the row space. Every vector in the row space. So, what what else is in the row space? Th- this row is in the row space, that row is in the row space. They They're all there. But that's not the whole row space, right? We just have to like remember, what does it mean What does that word space telling us?
And what else is in the row space? Besides the rows? All there combination. So, I really have to check that sure enough, if X is perpendicular row one, row two, all the all the different separate rows, then also X is perpendicular to what combination of the rows. And that's just matrix multiplication again. You know, I have row one transpose X is zero. So on, row two transpose X is zero. So, I'm entitled to multiply that by some C1, this by some C2, I still have zeros. I'm entitled to add. So, I have C1 row one. So So, all this when I put that together, that's big parentheses C1 row one plus C2 row two and so on transpose X is zero.
Right? I just added the zeros and got zero, and I just added these following the rules. No No big deal. The whole point was right sitting in that. Okay. So if I come back to this figure now I'm like a happier person. Cuz I have this uh the I now see how those subspaces are oriented. And these subspaces are also oriented. Well, actually why why is that orthogonality? Well, if the same statement for A transpose as that one was for A. So, I won't take time to prove it again because it's cuz we checked it for every matrix, and A transpose is just as good a matrix as A, so we're orthogonal over there.
So, we really have carved up this these this This was like carving up M dimensional space into two subspaces. And this one was carving up n-dimensional space into two subspaces. And Well, one more thing here. One more important thing. Let Let me move into into into three dimensions. Tell me a couple of orthogonal subspaces in three dimensions that somehow don't carve up the whole space. There's stuff left there. I'm thinking of a couple of orthogonal lines. If I Suppose I'm in three dimensions, R3. And I have one line, one one-dimensional subspace, and a perpendicular one. Could those be the row space and the null space?
Could those be the row space and the null space? Could I Could I be in three dimensions and have a row space that's a line and a null space that's a line? No. Why not? Because the dimensions aren't right, right? The dimensions are no good. The The dimensions here are r and n minus r. They add up to three. They add up to n. If I take uh Let's just if I just follow that example. If If If If the row space is one-dimensional, suppose A is What's a good in I'm in R3. I want a one-dimensional row space. Let me take 1 2 5 2 4 10. What's the dimension of that row space? One. What's the dimension of the null space?
What What tell What's the null space look like in that case? The row space is a line, right? One-dimensional. It's just a line through 1 2 5. Geometrically, what's the row space look like? It's a What's its dimension? So So here are Here n is 3. The rank is 1. So the dimension of the null space, so I'm looking at this x x1 x2 x3 to get 0. So the dimension of the null space is we all know is two. Right. It's a plane. And now, actually, we know we see better what plane is it. What plane is it? It's the plane that's perpendicular to 1 2 5. Right? We now see In fact, the 2 4 10 didn't actually have any effect at all.
I could have just ignored that. That didn't change the row space or the null space. I'll just make that one equation. Yeah. Okay. Sure. That's That's the easiest to deal with. One equation, three unknowns, and um I want to ask uh What's What does the equation give me? It gives me the null space. And you would have said back in September, you would have said it gives you a plane. And we're we're completely right. And the plane it gives you, the normal vector, you remember in calculus there was this dumb normal vector called N. Well, there it is, 1 2 5. Okay. So, uh What what is the What's the point I want to make here?
I want to make I want to emphasize that not only are the Let me write it in words. So, I want to write the null space and the row space are orthogonal. That's That's this neat fact which we we just checked from AX = 0. But now I want to say I want to say more because there's a little more that's true. Their dimensions add to the whole space. So, that's like a little extra information. That It's not like I could I couldn't have a line and a line in three dimensions. Those Those don't add up. One and one don't add to three. So, I use the word orthogonal complement. in Rn. And the idea of this word complement is that the orthogonal complement of a row space contains not just some vectors that are orthogonal to it, but all.
So, so what does that mean? That means that the null space contains all not just some, but all vectors that are perpendicular to the row space. Okay. Really, what I've done in this half of the lecture is just notice some of the nice geometry that that we didn't pick up before because we didn't discuss perpendicular vectors before. But, it was all sitting there, and now we've picked it up. That these vectors are orthogonal complements. And I guess I even call this part one of the fundamental theorem of linear algebra. The fundamental theorem of linear algebra is about these four subspaces. So, part one is about their dimension.
Maybe I should call it part two now. Their dimensions we got. Now we're getting their orthogonality. That's part two. And part three will be about bases for this. Orthogonal bases. So, that's uh coming up. Okay. So, I'm I'm uh I'm happy with that geometry right now. Okay. Okay. Now, what's my next goal in this chapter? Here's the It's the main problem of the chapter. The main problem of the chapter is So, this is coming. Coming attraction. This is this is the very last chapter that's about AX = B. I would like to solve that system of equations when there is no solution. You may say what a ridiculous thing to do, but I have to say I it's done all the time.
In fact, it has to be done. You get So so the problem is solve as the best possible solve I'll put quotes AX = B when there is no solution. And of course, what does that mean? B isn't in the in the column space. And it's quite typical if this matrix A is rectangular, if I maybe I have M equations and that's bigger than the number of unknowns then for sure the rank is not M. The rank couldn't be M now. So there will be a lot of right hand sides with no solution, but Here's an example. Some satellite is buzzing along. You measure its position. You make a thousand measurements. So that gives you a thousand equations for the for the parameters that that that give the position.
But there aren't a thousand parameters, there's just maybe six or so. Or you're measuring the you're doing questionnaires. You're you're measuring resistances. You're taking pulses. You're measuring somebody's pulse rate. Okay, there's just one unknown, the pulse rate. So, you measure it once, okay, fine. But, if you really want to know it, you measure it multiple times. But then, the measurements have no reason. So, there's The problem is that in many, many problems, we got too many equations, and they've got noise in the right-hand side. So, Ax = b, I can't expect to solve it exactly, right?
Cuz I don't even know what the there's error there's there's measurement mistake in b, but there's information, too. There's a lot of information about x in there. And what I want to do is like separate the noise, the junk, from the information. And so, this is a straightforward linear algebra problem. How do I solve What's the best solution? Okay. Now, let me um I want I I want to say, so that's like describes the problem in a algebraic way. I've got some equations. I'm looking for the best solution. Well, one way to find it is one one way to start one way to find a solution is throw away equations until you've got a nice square invertible system and solve that.
That's not satisfactory. There's no reason in these measurements to say these measurements are perfect and these measurements are useless. We want to use all the measurements to get the best information, to get the maximum information. But how? Okay. Let me anticipate a matrix that's going to show up. This A is typically a rectangle. But a matrix that shows up whenever you have and we've chapter three was all about rectangular matrices. And we know when this is solvable, you could do elimination on it, right? But I'm thinking, hey, you do elimination and you get equation zero equal other non-zeros.
I'm thinking we really elimination is going to fail. So, that's our question. Elimination will get us down to to will tell us if there is a solution or not, but I'm now thinking not. Okay, so what are we going to do? All right. I'm going to I'm going to tell you to jump ahead to the matrix that will play a key role. So, this is the matrix that you want to understand for this chapter four. And it's the matrix A transpose A. What's Tell me some things about that matrix. So, A is this m by n matrix, rectangular. But now I'm saying that the good matrix that shows up in the end is A transpose A. So, tell me something about that.
Is it Yeah, tell me what's the first thing you know about A transpose A? It's square, right? Square because this is m by n and this is n by m. So, this is the result is n by n. Good. Square. What else? It's symmetric. Good. It's symmetric. Cuz you remember how to do that. If we transpose that matrix, let's transpose it. A transpose A, if I transpose it, then that comes first, transpose, this comes second, transpose, and then transposing twice is leaves it brings it back to the same, so it's symmetric. Good. Now, we now know how to ask more about a matrix. I'm interested in is it invertible? If not, what's its null space?
So, I want to know about cuz cuz you're going to see Well, let me let me even Well, I shouldn't do this, but I will. Let me tell you what equation to solve when you can't solve that one. The good equation comes from multiplying both sides by A transpose. So, the the the good equation that you get to is this one. A transpose A x = A transpose B. That will be the central equation in the chapter. So, I think why not tell it to Why not admit it right away? Okay. I have to I should really give X uh I I want to sort of indicate that this X isn't I mean this X was the solution to that equation if it existed, but probably didn't.
Now, let me give this a different name, X hat. Because I'm hoping this one will have a solution. And I'm saying that it's my best solution. I'll have to say what does best mean. But that's going to be my my my plan. I'm going to say that the best solution solves this equation. So, you see right away why I'm so interested in this matrix A transpose A and in its invertibility. Okay. Now, when is it invertible? Okay. Let me take a case Let Let me just do one example and then uh uh just I I'll just pick a matrix here. Just so we see what A transpose A looks like. So, let me take a matrix A 1 1 1 1 2 5 just to invent a matrix.
So, there's a matrix A. Notice that it has m = 3 rows and n = 2 columns. Its rank is The rank of that matrix is two, right? Yeah, the columns are independent. Does AX = B If I look at AX = B, so X is just X1 X2 and B is B1, B2, B3. Do I expect to solve AX = B? What what's No way, right? I mean linear algebra is great, but solving three equations with only two unknowns usually we can't do it. We can only solve it if this vector B is what? I can solve that equation if that vector B1, B2, B3 is in the column space. If it's a combination of those columns, then fine. But usually it won't be. So the combinations just fill up a plane and most vectors aren't on that plane.
So what I'm saying is that I'm going to work with the matrix A transpose A and I just want to figure out in this example what A transpose A is. So it's 2 by 2. The first entry is a three. The next entry is an eight. This entry is a What's that entry? Eight. For sure. We knew it had to be and this entry is one. Now getting out my trusty calculator. 30. Is that right? 30. And is that matrix invertible? There's an A transpose A. And it is invertible, right? 38 is not a multiple of 830 and and it's invertible. And that's the normal That's what I expect. So this is what I what I want to I want to show.
It So here's the final Here's the key point. The null space of A transpose A It It It's not going to be always invertible. Tell me a matrix Yeah. Now, I I have to I have to say that I can't say A transpose A is always invertible. Cuz that's asking too much. I mean, what could the matrix A be, for example, so that A transpose A was not invertible? Well, even could be the zero matrix. I mean, that's like extreme case. Suppose I make this rank Suppose I change to that A. Now, I figure out A transpose A again, and I get What do I get? I get nine. I get nine, of course, and here I get What's that entry?
27. And is that matrix invertible? No. And why do I I I I knew it wouldn't be invertible anyway. Cuz this matrix only has rank one. And if I have a product of matrices of rank one, the product is not going to have a rank bigger than one. So, I'm not surprised that the answer only has rank one, and that's what I'm al- always happens. That the rank of A transpose A comes out to equal the rank of A. So, uh yeah. So, the null space of A transpose A equals the null space of A. The rank of A transpose A equals the rank of A. So, let's I'll I'll tell you if in a as soon as I can why that's true, but let's draw from that what the fact that I want.
This tells me that this square symmetric matrix is invertible if So, here's my conclusion. A transpose A is invertible if exactly when exactly if this null space is only got the zero vector. Which means the columns of A are independent. So, A transpose A is invertible exactly if A has independent columns. That's the That's the That's the the the fact that I need about A transpose A, and then you'll see next time how A transpose A enters everything. Next Next lecture is actually a crucial one. Here I'm preparing for it by getting us thinking about A transpose A. And its rank is the same as the rank of A, and we can decide when it's invertible.
Okay, so see you Friday. Thank you.