# 33. Left and Right Inverses; Pseudoinverse

**Channel:** MIT OpenCourseWare
**Source:** https://www.youtube.com/watch?v=Go2aLo7ZOlU
**Transcript page:** https://www.withtranscript.ai/video/Go2aLo7ZOlU

## Chapters

- 0:03 — Chapter 1 (0:03)
- 5:58 — Chapter 2 (5:58)
- 13:07 — Chapter 3 (13:07)
- 17:37 — Chapter 4 (17:37)
- 18:50 — Chapter 5 (18:50)
- 25:35 — Chapter 6 (25:35)
- 28:53 — Chapter 7 (28:53)
- 32:39 — Chapter 8 (32:39)
- 36:03 — Chapter 9 (36:03)

## Transcript

**[0:03]:** yeah okay 4 3 2 1 okay I see you guys are in a happy mood I don't know if that means 1806 is ending or uh the quiz was good uh my birthday conference was going on at the time of the quiz so in the conference of course everybody had to say nice things uh but I was wondering what would my 180 sixth class be saying because it was at the exactly the same time um but uh what I know from the grades so far they're basically close to and maybe slightly above the grades that you got on the on quiz two so uh very satisfactory um and then we have a final exam coming up uh and today's lecture as I told you by email will be a first step in the review and then on Wednesday I'll do all I can in reviewing the whole course so but the best so my topic

**[1:14]:** today is actually this is a lecture I have I've never given before in this in this um uh kind of in this way uh and it will well four subspaces that's certainly fundament metal and you you you know that so I want to speak about left inverses and right inverses and then something called pseudo inverses and pseudo inverses let let me say right away that comes in near the end of chapter 7 and that would not be expected on the final uh but you'll see that what I'm talking about is really the basic stuff that that uh for an m byn Matrix of rank R we're going back to the most fundamental picture in the in in linear algebra nobody could forget that picture right when you're when you're my age even you'll remember the row space and the null space orthogonal complements over there the column space and the null space of a transpose

**[2:22]:** column orthogonal complements over here and I want to speak about inverses okay and I want to identify the different possibilities so first of all when does a matrix have a just a perfect inverse two-sided you know so the the the the two-sided inverse is the it's what we just call inverse right and uh so that means that that there's a matrix that produces the identity whether whether we write it on the left or on the right and just tell me how is how are the numbers R the rank n the number of columns M the number of rows how are those numbers related when we have an invertible Matrix so this is the Matrix which was chapter 2 was all about matrices like this the the the beginning of the course what was the relation of the of r m and n for for the nice case they're all the same all equal so this

**[3:39]:** is the case when Ral m equal n Square Matrix full rank period just so I'll I'll use the words full rank okay good everybody knows that okay then chapter three got us we began to deal with matrices that were not of full Rank and we we they could have any Rank and we knew learned what the rank was and then we focused if you remember on some cases like full column rank can now can you remember what was the deal with full column rank so now this is going to be the case I think this is the case in which we have a left inverse and I'll try to find it

**[4:32]:** so we have a and and but what was the situation there so that it's the case of full column Rank and that means what does that mean about R it equals the what's the deal with r now if if the we have full column rank I mean the columns are independent but maybe not the rows so what is r equal to in this casee n thanks N R equals n the N columns are independent but probably we have more rows what's the picture and then what's the null space for this so the N columns are independent what's the null space in this case um what so of course you know what I'm asking you're saying why is this guy asking something I know that you know I I think about in my sleep right so the null space of this Matrix if the rank is n the null space

**[5:42]:** is what vectors are in the null space just the zero Vector right the columns are independent independent

**[5:58]:** columns no no combination of the columns gives zero except that one and what's my picture over my my let me redraw my picture my picture is I have the the row space is everything the row space is all of no is that right the row space Oh let's see I often often get these turned around right so what's the deal the rank the The Columns are independent right so the rank should be the full number of columns so what does that tell us there's no null space right okay the row space is the whole thing yeah I won't even draw the picture so and what was the deal with and these were very important in Le squares problems because so what what was the what what more uh is true here if we have full column rank the null space is zero we have independent columns uh the unique so we have zero or one

**[7:20]:** solutions to ax equal B there may not be any solutions but if there's a solution there's only one solution solution because other Solutions are found by adding on stuff from the null space and there's nobody there to add on so the the the particular solution is the solution if there is a particular solution of course the rows might not be in probably not independent and therefore some right hand sides won't end up with a 0 equals 0 after elimination so some sometimes we have may have no solution or one solution okay and what I want to say is that for this Matrix a oh oh yeah tell me something about a transpose a in this case so this this whole part of the board now is devoted to this this case what's the deal with a transpose a I've emphasized over and over how important that combination is for a rectangular Matrix a transpose a

**[8:32]:** is is the good thing to look at and if the rank is n if the null space has only zero in it then the same is true of a transpose a that's the beautiful fact that if the rank of a is n well we know this will be an N byn symmetric Matrix and it will be full rank so this is invertible this is this Matrix is invertible that Matrix is in invertible and now I want to show you that a itself has a onesided inverse here it is the inverse of that which exists times a transpose there is a onesided there is a should I call it a inverse left of the Matrix a why do I say that because if I multiply a if I multiply this guy by a what do I get what does that multiplication give of course you know it instantly because I just put the parentheses there I

**[9:44]:** have a transpose a inverse time a transpose a so of course it's the identity so it's a left inverse and so we can we have and this was the totally crucial case for leas squares CU you remember that lease squares the whole the central equation of lease squares had this Matrix a transpose a as its coefficient Matrix and in the case of full column rank that Matrix is invertible and we're go so that's the case where there is a left inverse so we a does whatever it does we can find a matrix that that comes that that brings it back to uh to the identity now is it true that in the other order so a inverse left time a is the

**[10:40]:** identity th right this Matrix is M by n this Matrix is n by m the identity Matrix is n byn all good all good if you're n but if you try to put that Matrix on the other side

**[11:02]:** it would fail if if the r if the full column rank if this is smaller than M the case where they're equal is the beautiful case where that's all set now we're looking at the case where the columns are independent but the rows are not so this may so this is invertible but what Matrix is not invertible a a transpose is bad for this case a transpose a is good so we can multiply on the left everything good we get the left inverse but it would not be a a two-sided inverse a rectangular Matrix can't have a two-sided inverse because there's got to be some null space right if if I have a matrix that's rectangular then either that Matrix or it's or its transposed has some null space because if n and M are different then there's going to be some free variables around and we'll have some null space in that direction okay

**[12:10]:** tell me the corresponding picture for the opposite case so now I'm going to ask you about right inverses a right inverse and you can do you can fill this all out this is going going to be the case of full row Rank and then the r is equal to M now the M rows are independent but the thing but the columns are not so what's the deal on that well just exactly the flip of this one the null space of a transpose contains only zero because there are no combination of the rows that give the zero row we have independent

**[13:07]:** rows and in a minute I'll let's give an example of all these how many so how many solutions to ax equal B in this

**[13:22]:** case it the rows are independent so we can always solve ax equal B we never we never elimination never produces a zero row so we never get into into that zero equal one problem so ax equal B always has a solution but too many so it's probably it will there'll be some null space the null space of a what what will be the dimension of the null of of A's n space how many free variables have we got how many special Solutions in that null space have we got so how how many free variables in this set up we got n columns so n variables and this tells us how many are pivot variables that tells us how many pivots there are so there are n minus M free variables so they're infinitely many solutions to axal B we have n minus M free variables in this case okay now I wanted to ask about

**[14:42]:** this this idea of a right inverse okay so I'm going to have a a matrix a my Matrix a and now there's going to be some inverse on the right that will give the identity Matrix so it'll be a time a inverse on the right will be I and can you tell me what just by by uh comparing with what we had up there what will be the the right inverse we even have a formula for there will be other actually there are other left inverses that's our favorite there will be other right inverses but tell me our favorite here what's the what's the nice right inverse the nice right inverse will be well there we had a transpose a was good now it'll be a a transpose that's good so the the good Matrix the good right The Good the thing we can invert is a a transpose so now if I just do

**[15:55]:** it that way there sits the right inverse you see how completely parallel it is to the one

**[16:07]:** above right so that's the that's the right inverse so that's the that's the case when uh there when there's in terms of this picture tell me what the null spaces are like so far for these three cases for what about case one where we had a two-sided inverse full rank everything everything great the null spaces were like gone right the null spaces were just the zero vectors then I took case two this null space was gone case three this null space was gone and then case four is like the most General case when when this picture is all there when the when when all the N when the null spaces this has Dimension R of course this has Dimension n minus r this has Dimension R this has Dimension M minus r and the final case will be when this uh um when when when R is smaller than m and n but let

**[17:30]:** can I just before I leave

**[17:37]:** here look a little little more moment at this at this one at that at this case of full column rank so a inverse on the left this this it has this left inverse to give the identity I said if we multiply them in the other order we wouldn't get the identity but then I just realized that I should ask you what do we get so if I put them in the other order so if I continue this down below but I write a time a inverse

**[18:16]:** left so there's a times the left times the left inverse but it's not on the left anymore so it's not going to come out perfectly but everybody in this room Auto

**[18:34]:** recognize that Matrix right that let's see is that the guy we know am I am I okay

**[18:50]:** here what what is that Matrix P thanks P that Matrix it's a projection it's the projection onto the column space it's trying to be the identity Matrix right it that's a projection Matrix is as like as it's it tries to be the identity Matrix but you've given it uh an impossible job so it's the it's the identity Matrix where it can be and elsewhere it's the zero Matrix so this is p right the projection onto the column space okay and if I asked you this one and put these in the in the opposite order so this came from up here and similarly if I if I tried to put the right inverse on the left so can I so that like came from above this coming from this side what happens if I try to put the right inverse on the left then I would have a transpose a a transpose inverse a if if

**[20:05]:** this Matrix is now on the left what do you figure that Matrix

**[20:13]:** is it's going to be a projection too right it looks very much like this guy except the only difference is a and a transpose have been reversed so this is a projection this is another projection onto to the row space it's it's again it's trying to be the identity but there's only so much the Matrix can do and this is the projection onto the column space so let me now go back to the main picture and tell you about this the general case the pseudo inverse what's what so this these are cases we know so this was important review you got to know the these business about these ranks and the free variables really this is linear algebra coming together and you know it's one nice thing about teaching 1806 you you it's not trivial but it's I don't know somehow it's nice when it comes out right I mean well I shouldn't say anything

**[21:30]:** bad about calculus but I will uh I mean like you know you have formulas for surface area and and other awful things and and you know they do their best in calculus but uh it it's not uh elegant and um linear algebra just is well you know linear algebra is about the nice part of calculus where everything's like flat and uh uh the formulas come out right and you can go into high Dimensions where in calculus you really you know you trying to visualize these things well two or three dimensions is kind of the limit but here we don't you know i' I've stopped doing two by tws I'm just talking about the general case okay now I really will speak about the general case here what what could be the inverse what's a kind of reasonable inverse for a matrix for the the completely General Matrix where there's a rank R but it's smaller

**[22:37]:** than n so there's some null space left and it's smaller than M so a transpose has some null space and it's those null spaces that are screwing up inverses right because if a matrix takes a vector to zero well there's no way an inverse can like bring it back to life

**[23:00]:** so now I'm my my topic is now the pseudo inverse and and let's just by a picture see what what's the best inverse we could have so here's a vector X in the row space I multiply by a now the one thing everybody knows is you take a vector you multiply by a and you get an output and where is that output where is ax always in the column space right ax is a combination of the columns so ax is somewhere here so I could take all the vectors in the row space I could multiply them all by a I would get a bunch of vectors in the column space and what I think is I'd get all the vectors in the the column space just right I think that this connection between an X in the row space and an ax in the colum space this is one: one there's exact I mean well look we got a chance

**[24:13]:** because they have the same dimmension that's an R dimensional space and that's an R dimensional space and somehow The Matrix a it's got these null spaces hanging around where it's knocking vectors to zero and then it's got all the vectors in between which is almost all vectors almost all vectors have a row space component and a null space component and it's killing the null space component but if I look at the vectors that are in the row space with no null space component just in the row space then they all go into the column space so if I put another Vector let's say y in the row space I am positive that wherever a Y is it won't hit ax you see what I'm saying let let's let's see why all right so if so here's what I said If X and Y are in the row

**[25:18]:** space then ax is not the same as a y they're both in the column space of course but they're

**[25:35]:** different that would be a perfect question on a final exam because that's that's what I'm teaching you in that material of chapter 3 and chapter 4 especially chapter 3 If X and Y are in the row space then ax is different from a y and what so so what this means is that and and we'll see why what this means is that in words from the row space to the column space a is perfect it's an invertible Matrix if we if we like limited it to those spaces and then its inverse will be what I'll call the pseudo inverse so that's what the pseudo inverse is it's the inverse so a goes this way from X to y sorry X to Ax from y to a y that's that's a going that way then in the other direction anything in the column space comes from somebody in the row space and the reverse there is

**[26:43]:** what I'll call the pseudo inverse and the the simple notation has accepted notation is A+ so so y will be a + x sorry no y will be A+ times whatever it started with a Y do you see what you see my picture there same same of course for x and ax this way a does it the other way is the pseudo inverse and the pseudo inverse just kills this stuff and the Matrix just kills this stuff so everything that really serious here is going on in the row space and the column space and now tell me so this is the fundamental fact that between those two R dimensional spaces our Matrix is perfect why suppose suppose they weren't why do I get into trouble suppose so proof I haven't written down proof very much but I'm going to use that word once suppose they were the same

**[28:02]:** suppose so this is these are supposed to be two different vectors maybe I better make the statement correctly If X and Y are different vectors in the row space maybe I'll better put if x is different from y both both in the row space so I'm starting with two different vectors in the row space I'm multiplying by a so these guys are in the column space everybody knows that and the point is they're different over there so suppose they weren't suppose ax equals a

**[28:43]:** y suppose well that's the same as saying a x - Y is

**[28:53]:** z so what so what do I I know now about xus Y what do I know about this Vector it's well I can see right away where what space is it in it's sitting in the null space right so it's in the null space but what else do I know about it here here was X in the row space y in the row space what about xus y it's also in the row space SP right heck that thing is a is a vector space and if if a vector space is anything at all if x is in the row space and Y is in the row space then the difference is ALS so it's also in the row

**[29:46]:** space so what now I've got a vector xus y that's in the null space and that's also in the row space so what Vector is it it's the zero vector so I would conclude from that that xus y had to be the zero Vector X was y so so in other words if I start from two different vectors I get two different vectors if these vectors are the same then those vectors had to be the same that's that's like the algebra proof which we understand completely because we really understand these subspaces of what I said in words that a matrix a is really a nice invertible mapping from row space to column space if if the null spaces keep out of the way then we have an inverse and that inverse is called the pseudo inverse and it's very very useful in application statisticians discovered oh boy this is the thing that we needed all our lives and here it finally showed up the pseudo inverse is is the right

**[31:00]:** thing statisticians why do statisticians need it and and uh because statisticians are like lease squares happy I mean they're they're always doing lease squares and so this is their Central linear regression statisticians who may watch this on video please forgive that uh description of your interest one of your interests is linear regression and this problem but if this problem is only okay provided we have full column Rank and statisticians have to worry that all the time about oh God maybe we just repeated an experiment you know you you you're taking all these measurements maybe you just repeat them a few times maybe you know maybe they're not independent well in that case that that a transpose a matrix that they depend on becomes singular so then that's when they needed the pseudo inverse it just arrived at the right moment and it's it's the the right quantity okay so I'll I'll now that you know

**[32:17]:** what the pseudo inverse should do let me see what it is can we find it so this is my my the to complete the lecture is how do I find this pseudo

**[32:39]:** inverse A+ okay okay well here's one way and I'm always everything I do today is to try to review stuff one way would be to start from the SVD the singular value decomposition and you remember that that factored a into an orthogonal Matrix times this diagonal matrix times this orthogonal Matrix but what did this diagonal guy look like this diagonal guy Sigma has some nonzeros and they you remember they came from a transpose a and a a transpose these are the good guys and then some more zeros and all zeros there and all zeros there so you can guess what the pseudo inverse is I just invert stuff that's nice to invert well what's the pseudo inverse of this that's that's what the problem comes down to what's the pseudo inverse of this beautiful diagonal matrix but it's got a no space right what's the rank of this Matrix what's the rank of this of this uh diagonal matrix R of course it's got R nonzeros and then it's otherwise zip so it's got

**[34:12]:** It's got n columns it's got M rows and it's got rank R it's the best example the simplest example we could ever have of our of our uh General setup okay so what what's the what what what's the pseudo inverse what's the Matrix so I'll erase our columns because right below it I want to write the pseudo inverse okay we you can make a pretty darn good guess if it was a proper diagonal matrix invertible if there weren't any zeros down here if if it was Sigma 1 to Sigma n then everybody knows what the inverse would be the inverse would be say one over Sigma 1 down to one over but I'll of course I'll have to stop at Sigma R and it'll be the rest zeros again of course and now this one was m byn and this one is meant to have a slightly different you know transposed shape n by n they both have that rank

**[35:35]:** R what is so if so my idea is that the pseudo inverse is the best is the closest I can come to an inverse so what is Sigma times its pseudo inverse can you multiply Sigma by its pseudo inverse multiply that by that what Matrix do you

**[36:03]:** get they're diagonal rectangular of course but of course we're going to get ones R ones and all the rest zeros and the shape of that this was M this was M by this this whole Matrix will be M by m and suppose I did it in the other order suppose I did Sigma plus Sigma why don't I do it right underneath what would what in the O in the opposite order see this Matrix hasn't got a left inverse it hasn't got a right inverse but every Matrix has got a pseudo inverse if I do it in the order of Sigma plus Sigma what do I get square Matrix this is M byn this is n by m my result is going to be M by m is going to be n byn and what is it those are diagonal matrix matrices it's going to be ones and then

**[37:13]:** zeros it's not the same as that it's a different size it's the projection it's a projection Matrix onto one is the projection Matrix onto the column space and this one is the projection Matrix onto the row space that's the best the pseudo inverse can do so the what the pseudo inverse does is if you multiply on the left you don't get the identity if you multiply on the right you don't get the identity what you get is the projection it it it brings you into the two good spaces the row space and column space and it just wipes out the null space so that's what the pseudo inverse of this diag one is and then the pseudo inverse of a itself this is perfectly invertible what's the inverse of V transpose just another tiny bit of review that's an orthogonal Matrix and its inverse is V good this guy is has got all the trouble

**[38:20]:** in it all the null spaces is is is this is responsible for so we can't it doesn't have a true inverse it has a pseudo inverse and then the inverse of U is U transpose thanks or of course I could write U invert so that's the question of how do you find the pseudo inverse what so what statisticians do when they're in this so this is like the case of where least squares breaks down because the rank is you don't have full Rank and but the the beauty of the singular value decomposition is it puts all the problems into this diagonal matrix where where it's clear what to do it's the best inverse you could think of is clear uh you see there there could be other I mean we could put some stuff down here it would multiply these zeros and wouldn't wouldn't have any effect but then but the right the the good

**[39:26]:** pseudo inverse is the one is the one with no extra stuff it's sort of like as small as possible it it has to have those to produce the ones if it had other stuff it would just be like it would be a larger Matrix so this this pseudo inverse is kind of the minimal Matrix that gives uh the best results Sigma Sigma plus being r1s okay uh so I I guess I'm hoping pseudo inverse again let me repeat what I said at the very beginning the this pseudo inverse which appears at the end which is in section

**[40:16]:** 7.4 and probably I did more with it here than I did in in that in the book I'm not the the word Pudo inverse will not appear on on an exam in this course but I think if you see these this all will appear because this is all what the course was about chapters 1 2 3 4 but uh so if you but if you see all that then you probably see well okay the general case had both null spaces around and this is the natural thing to do yeah so uh that's this is one way to find the pseudo inverse there I could I could the the point of a pseudo inverse of of computing a pseudo inverse is to get some factors where you can find the pseudo inverse quickly and this is like the champion because this is this is one where we can invert those and those two easily just by

**[41:21]:** transposing and we know what to do with the diagonal okay that that's like uh as much review maybe let's let's have a five minute holiday in 1806 and uh I'll see you Wednesday then for the rest of this course thanks
