# 9. Constraints: Visual Object Recognition

**Channel:** MIT OpenCourseWare
**Source:** https://www.youtube.com/watch?v=gvmfbePC2pc
**Transcript page:** https://www.withtranscript.ai/video/gvmfbePC2pc

## Chapters

- 0:09 — Chapter 1 (0:09)
- 0:53 — Chapter 2 (0:53)
- 3:57 — Chapter 3 (3:57)
- 5:35 — Chapter 4 (5:35)
- 8:48 — Chapter 5 (8:48)
- 11:01 — Chapter 6 (11:01)
- 12:37 — Chapter 7 (12:37)
- 16:51 — Chapter 8 (16:51)
- 19:34 — Chapter 9 (19:34)
- 22:31 — Chapter 10 (22:31)
- 26:10 — Chapter 11 (26:10)
- 28:46 — Chapter 12 (28:46)
- 34:29 — Chapter 13 (34:29)
- 35:13 — Chapter 14 (35:13)
- 38:31 — Chapter 15 (38:31)
- 41:27 — Chapter 16 (41:27)
- 42:28 — Chapter 17 (42:28)
- 44:55 — Chapter 18 (44:55)
- 46:08 — Chapter 19 (46:08)
- 47:34 — Chapter 20 (47:34)
- 48:14 — Chapter 21 (48:14)
- 50:32 — Chapter 22 (50:32)

## Transcript

**[0:09]:** You know, it's unfortunate that politics has become so serious. Back when you were little, it was a lot more fun. You could make fun of politicians. Here's a politician some of you may recognize.

**[0:27]:** But uh it's convenient to be able to vary uh what this particular politician looks like. For example, we can uh go from cookie baker to radical.

**[0:43]:** We can go from uh superwoman uh to bimbo.

**[0:53]:** Uh socialite. I like the socialite end of this. There she is. Or we can move the slider over the other way. Get the bag lady. Uh alert, asleep. Sad, happy. How do How does How does that work? I don't know, but I bet by the end of this hour you'll know how that works. And not only that, you'll understand something about uh what it takes to recognize faces. It turns out that some theories of face recognition are based on the same principles that um this um that this program is based on. But you can kind of guess what's happening here. There are several many stored images and when I move those sliders, it's interpolating amongst them. So that's how that works. But the main subject of today um is the the uh matter of uh recognizing objects. Faces could be the objects, but they don't have to be.

**[2:04]:** This could be an object that you might want to recognize. And I want to talk to you a little bit about the history of uh this problem and where it stands today. Still not solved. Uh but it's an interesting exercise to see how the attempts at solution have evolved slowly uh over the past 30 years. So slowly in fact that I think if someone told me how where how long it would take to get to where we are, I uh 30 years ago, I think I would have hung myself. But um things do move slowly uh and it's important to see how slowly they move because they will continue to move slowly in the future and you have to understand that that's the way things work sometimes. So uh to start this all off, we have to go back to the um ideas of the legendary David Marr who dropped dead from leukemia in about 1980.

**[2:56]:** Uh I say the gospel according to Marr because he was such a um a powerful and central figure that uh almost anything he said was believed by a a large um collection of devotees. But Marr um articulated a a set of ideas about how computer vision would work that started off by suggesting that uh with the input from the camera you form a kind of you look for edges and you find edge fragments and normally they wouldn't be even as even as um well drawn as uh I've done them now or as badly drawn as I've done them now. But uh the first step then in visual recognition would be to form this edge-based description of what's out there in the world. And Marr uh called that the primal sketch.

**[3:57]:** And from the pri- primal sketch, the next step was to decorate the primal sketch with some vectors, some surface normals showing where the faces on the object were oriented. He called that the two and a half D sketch.

**[4:21]:** Uh why is it two and a half D? Well, it's it's sort of two D in the in the in in the sense that it's still camera-centric in its its way of presenting information, but at the same time it attempts to say something about the three-dimensional or or arrangement of the faces. So the speculation was that you couldn't get to where you wanted to go in one step. So you needed several steps to get from the image to something that you could recognize. And the third step was to convert the primal the two and a half D sketch into generalized cylinders.

**[5:03]:** And the idea is this, if you have a regular cylinder you can think of it as a as a circular area moving along an axis like so. So that's a description of a cylinder. A circular area moving along an axis. You can get a different kind of cylinder if you go along the same axis but you allow the size of the circle to change as you go. So for example, if you were to describe a wine bottle

**[5:35]:** there would be a function of distance along the axis that would shrink the uh the uh the circle appropriately to match the uh dimensions of a wine bottle. A fine burgundy, I perceive. In any case, this one once converted into a generalized cylinder when matched against a library of such descriptions results in recognition.

**[6:04]:** Great theory based on the idea that you start off by looking at edges and you end up in several steps of transformation producing something that you can look you can look up in a library of descriptions. Great idea. Trouble is no one could make it work. It was too hard to do this. It was too hard to do that. And the generalized cylinders produced, if any were too coarse. You couldn't tell the difference between a Ford and a Chevrolet or between a Volkswagen and a Cadillac because they're just too coarse. So although it was a great idea based on the idea that you have to do recognition in several transformations of representational of rep- of representational apparatus, just didn't work. So much later, maybe 15 years later or so uh we get to the next um the next part of our story which is the alignment theory uh alignment theories, uh most notably the one produced by Shimon Ullman, one

**[7:09]:** of Marr's students. So the um alignment theory of recognition is based on a very strange and exotic idea. It doesn't seem strange and exotic to mechanical engineers for a while because they're used to mechanical drawings. But here's the strange and miraculous idea. Imagine this object. You take three pictures of it. You can reconstruct any view of that object. Now I have to be a little bit careful about how I say that. First of all, some of the vertexes are not visible in the views that you have. So of course you can't do anything with those. So let's say that we have a transparent object where you can see all the vertexes. If you have three pictures of that you can reconstruct any view of that object. Now I have to be a little careful about how I say that because it's not true. What's true is you can produce any view of that in orthographic projection.

**[8:11]:** So if you're close enough to the object that you get perspective, it doesn't work. But for the most part you can neglect perspective after you get about two and a half times as far away as the object is big and you can presume that you got orthographic projection. So that's a strange and exotic idea. So how But how can you make a recognition theory out of that? So let me show you. Uh here's one drawing of the object. I need two more. Oh, let's see. Let's have this one.

**[8:48]:** And maybe one that's tilted up a little bit.

**[8:58]:** It's important that these It's important that these pictures not be not be just rotations in one axis because they wouldn't form what you might think of as a kind of basis set. So there are three pictures. We'll call them A, B, and C.

**[9:15]:** And then we want a a fourth picture which will look like this. Doesn't have to be too precise. And we'll call that the unknown. And what we really want to know is if the unknown is the same object that these three pictures were made from.

**[9:41]:** So let me uh begin with an assertion. I'll need four colors of chalk to make this assertion. What I want to do is I want to I want to pick a particular place on the object like this one.

**[9:55]:** And maybe the same place on this over here. Those are corresponding places, right? So, I can now write an equation that the X coordinate of that unknown object is equal to oh, I don't know, alpha X of A plus alpha plus beta, I mean, X of B plus gamma X sub C plus some constant tau. Well, of course that's obviously true because I'm I'm letting you take those alpha, beta, gamma, and tau and make them anything you want. So, although that's conspicuously obviously true, it's not interesting. So, let me take another point. And of course I can write the same equation down for this for this purple point.

**[11:01]:** And now that I'm on a roll and having a great deal of fun with this I can take this point

**[11:12]:** and make a blue equation.

**[11:26]:** And you know I'm I'm destined to do it. So, I've got one more color. I might as well use it. Let's just make sure I've got something that works here. That's this one. That's this one. I hope I've got these correspondences right. Have I got one off? Which what which color? Okay, so this one goes with this one goes with this one. No, is that one wrong? Oh, oh, oh, of course it's wrong. Excuse me. Goes down here. Right? And then this one is off as well. I wouldn't get a very good recognition scheme if I can't get those correspondences right. Which is um one of the lessons of today. Okay, now I've got them right and now that equation is correct. I think I've got this one right already. So, now I can just write that down. I'm on a roll. I can just I'm just copying this.

**[12:37]:** So, those are a bunch of equations. And now the astonishing part is that I can choose alpha beta gamma and tau to be all the same.

**[12:59]:** That is there's one set of alpha, beta, gamma, and tau that works for every for all four points. So, you you look at that puzzled and that's okay to be puzzled because I certainly haven't proved it. I'm asserting it. But right away there's something interesting about this and that is that the relationship between the points on the unknown object and the points in the stored library of of of images are related linearly. That's true because it's orthographic projection. Linearly related. So, I can generate the points in the in in in some fourth object from the points in three sample objects with a linear operations. Christopher? Is that the X coordinate of It's the X It's the X coordinate. Each of those X coord Christopher asked about the X coordinates. Each of these X coordinates are meant to be color coded. It gets a little complicated with the notation and stuff.

**[13:59]:** Uh so, that's the reason I I'm color kind of coloring coding the coordinates so that the so the orange X sub U is the X coordinate of that particular point. In three No, in not in 3D space, in the image. So, it's a 3D projection of it. It's a 2D projection of it, an orthographic projection. Okay? So, what we're looking at drawings and those coordinates over there are the two-dimensional coordinates in the drawing. Just as if it were on your retina. All right.

**[14:36]:** projection or can curved surfaces also So, he asked about curved surfaces and the answer is that you have to find corresponding points on the object. So, if you had a curved If if you have a totally curved surface and you can't identify any corresponding points, you lose. But if you consider our faces, there are some obvious points even though our faces are not by any means flat like these objects. We have the tip of our nose in the center of our eyeballs and so on. So, uh if that's true uh what does that mean about recovering alpha, beta, gamma, and tau? Can we Can we find them? Now, what do you think? How would we go about finding them? You're nodding your head in the right direction.

**[15:26]:** Splendid. It's four equations and four unknowns. Four linear equations and four unknowns. So, obviously you can solve for alpha, beta, gamma, and and and tau if you know that these equations are correct. So, how does that help us with recognition? It helps us with recognition because we can take another point.

**[15:47]:** Let me say this square point here and this corresponding square point here and this corresponding square point here. And what can we do with those three points now? We've got alpha, beta, gamma, and tau. So, we can predict where it's going to be in the fourth image. So, we can predict that that square point is going to be right there. And if it isn't we are highly suspicious about whether this object is the kind of object we think it is. So, you look at me in disbelief. You'd like me to demonstrate this, I imagine. Okay. Let me see if I can demonstrate this.

**[16:51]:** So, I'm going to do this um in a slightly simplified uh in a slightly simplified version. I'm only going to allow rotation around the um around the vertical axis. And just so you know I'm not cheating, there's a little slider here that rotates that that third object. And let's see, why are there just two two known objects and one unknown? Well, that's because I've restricted the motion to rotation around the vertical axis and some translation. So, now that I've spun it around a little bit let me pick some corresponding points. Oops. What's happened?

**[17:41]:** Well, let me run that by you again.

**[18:04]:** Okay, so there's one point I've selected from the model objects and the corresponding point over here on the unknown is right there. I'm going to be a little off, but that's okay. So, let me just pick that one. And then this that corresponds to this one. Krishna, would you like me to specify a point so people know know I'm just cheat No, I'm not not cheating? Pick a point. Pick a point, Krishna. Oh, the right the The right? This one? Oops, I got to pick it out of the models first. Now, I'll pick it over here. Boom. So, all the points are where they're supposed to be. Not cool? Well, let's suppose that the unknown is something else. This is a carefully selected object because the points are all the correct positions vertically um but they're not necessarily the correct positions in the other two dimensions.

**[19:00]:** So, let me pick a this point and this point and this point and this point and Krishna had me pick this point. So, we pick this point. So, if it thinks that the unknown is one of these obelisk objects then we would expect to see all of the corresponding points correctly identified. But boom. All the points are off.

**[19:34]:** So, it seems to work in this particular example. I find the alpha and beta using two images and I predict the locations of the other points and I determine whether those positions are correct and if they are correct, then I have a pretty good idea that I have fact identified the object on the right as either an obelisk or an organ depending on which of the model choices and the um unknown choices I've selected. So uh all that I have to do left to do is to demonstrate that what I said about this is true. So I'm going to actually demonstrate that what I said about this is true uh using the configuration in this demonstration because it's much too hard for me to remember uh matrix transformations for generalized rotation around three in three dimensions. So here's how it's going to work.

**[20:33]:** The Z axis is going up that way. Or it's going to be pointing toward you. And what I'm going to do is I'm going to rotate around this axis. And what I'm going to do what I want to do is I want to find out how the X coordinate in the image of the points move as I do that rotation. So here's the X axis. This is the coordinate that you can see. Here is the Y axis. That's in depth, so you can't tell how far away it is. And the Z axis X Y Z axis must be pointing out that way toward you. So now I'm going to consider just a single point on the object and see what happens to it. So uh I'm going to s- I'm going to say to myself, let's put it in some kind of standard Let's put the object in some kind of standard position.

**[21:32]:** I don't care what it is, but it could be just random. Just spin it around. Some position. We'll call that that the uh standard position S and that means that the X coordinate of the standard position is X sub S and the Y coordinate of the standard position is uh Y sub S. And now I'm going to rotate the object uh three times. Once to form the A picture, once to form the B picture, and once to form the C picture. And you can make those choices. I don't have You know, I don't have to depend on Those can be anything, right? So let's say that the um A picture is out here. So that's X That's That's the A picture. The B picture is out here. And the unknown is up that way.

**[22:31]:** And so what I want to know depends on these vectors. We'll call that theta sub A. And this is theta sub B. Oops. Yeah, that's theta sub B. And this one is theta sub U. So I would like to know how uh X sub A depends on X sub S and Y sub S. And I can never remember how to do that cuz I can never remember the transformation equations for rotation. So I have to figure it out every time. And this is no exception. So what I'm going to say is that this vector that goes out to S consists of two pieces. There's the X part and the Y part. And I know that I can rotate this vector by alpha sub A by rotating this vector and rotating that vector and adding up the results, right? So if I rotate this vector by alpha sub A then the contribution of that to the X coordinate uh of A is going to be given by the cosine of theta sub A multiplied by X sub S.

**[23:56]:** Right? So you can just exaggerate that mo- motion say, well, if I pitch it up that way, then the projection down on the on the X axis is going to be this length of the vector times the cosine of the angle. Right? Now there's also going to be a dependence on Y sub S. Let's figure out what that's going to be. I've got this vector here and I'm going to rotate it by theta sub A as well. If I rotate that by theta sub A and see what the projection is on the X axis that's going to be given by the of the angle, right? But it's actually it's going the wrong way, so I have to subtract it off. So that's how I don't have to remember what the signs are on these equations.

**[24:44]:** Well, that was good. And now I can Now I'm Now that I'm off and running, I can do what I did before. Makes it easy to do give the lecture cuz this is going to be X sub B is equal to X sub S times the cosine of theta sub B minus Y sub S times the cosine of theta Oh, you're letting me make mistakes. Shame. This is I can generally tell by all the troubled looks, but there should be some shouting as well. It's the sign and that's the sign. And one more time X sub U is equal to X sub S times the cosine of theta sub U minus Y sub S times the sine of theta sub U and I forgot the B up there. So there's some equations and we don't know what we're doing. We're just going to stare around them a while and see if they sing us a song.

**[25:45]:** So let's see if they sing us a song. Uh what about X sub A and X sub B? These are things that we see in the image. These are things that we can measure.

**[26:10]:** What about all those cosines and sines and theta A's and theta B's? Well, we have no idea what they are, but one thing is clear. They're true for all of the points on the object. Because when we rotate the object around by angle theta we're rotating all of the points the same but through the same angle, right? So with respect to any particular view of the object here we are in the standard position. Here we are in position A. The vectors to all of the points on the object are rotated by the same angle when we go from the standard position to the A position. So that means that for all of the

**[26:59]:** all of the images in this particular rendering with a particular rotation by theta A, theta B, and theta U those are constants.

**[27:15]:** Now remember this is for a particular theta A, a particular theta B, and a particular theta U. As long as the we're talking about all of the points for each of those rotations those angles and cosines are going to be the same for all possible for all possible points on the object. Okay, so now we go back to our high school algebra experts and we say, "Look at these first two equations."

**[27:46]:** We can get rid of We can We We can express We got two equations in what we can now conceive screw to be two unknowns. What are the unknowns that are left? We can measure A and B. Whatever the cosines are, they're the same for all the pictures. So if we treat those as constants, then we can solve for X sub S and Y sub S. Right? We can solve for X sub S and Y sub S in terms of X sub A and Y and X sub B and and and a whole bunch of constants. But I don't know a whole bunch of constants. Let's see. We can gather up all of those cosines and ratios of sines and cosines and all that stuff. Uh and they're all Put them all together because they're all constants. And then we can do this. We can say X sub U

**[28:46]:** is equal to Well, it's going to depend on X sub A and X sub B. And by the time we wash uh manipulate or screw around with all of those cosines uh we can say that the multiplier for X sub A is some constant alpha and the multiplier for X sub B is some constant beta. So that's not a sleight of hand. That's just linear manipulation of those equations. And that's what we wanted to show. That for orthographic projection, which is the which this is There's no perspective involved here. We're just taking the projection along the X axis. For that kind of projection, then we can demonstrate for this simplified situation that that equation must hold. Now let me I want to give you a few puzzles cuz this stuff is so simple. Suppose I allow translation as well as rotation. What's going to happen?

**[29:42]:** You just get the tau. Basically, you get a Yeah, you add a You add a constant tau, but what do we need to do in order to solve it? Subtraction. out. I mean, you subtract two equations and Let's see. Now I've got three unknowns, right? I don't know tau, I don't know X of S, I don't know Y of S. So, I need another equation. Where do I get the other equation? From another picture.

**[30:09]:** That's why up up there, I needed four points. That covers the situation where I've got three degrees of rotation and translation. Here I got by with just two pictures uh in this illustration. That one involved a tau, a translational element, so I needed three pictures, and this one's got full rotation, so I needed four. So, um great idea. Uh works fine. Uh trouble is it doesn't work so fine on natural objects. It works fine on things that are manufactured because they all have identical dimensions. So, if I made a million of these in a factory I'd have no trouble recognizing them because all I'd have to do is take three pictures record coordinates of some of the points and I'll be done. Trouble is, the natural world isn't like this. And you aren't like this either. I don't know if you know, if I'm trying to recognize faces it's a it's not that easy to to do all this.

**[31:23]:** First of all, it's it's a little difficult to find the exact point. The exactly corresponding points. I made a I made a mistake in doing it myself, and if the computer made a mistake, it would be certainly make an error because it would be using non-corresponding points to make predictions, it would be way off. But this is still in the tradition of working from the from from local features in the objects toward recognition. So having having looked at that theory, um we also find it a little wanting. Works great in some circumstances. Doesn't seem to solve the whole recognition problem.

**[32:07]:** Years pass. Shimon Ullman comes up with another theory that's not so much based on edge fragments or the location of particular features but rather on correlation. Taking a picture of say Krishna's face taking a picture of the whole class and then using that as a kind of correlation mask running it all over the the picture of the class, seeing where it maximizes out. Now, that's vague. I'll explain what I'm talking about with respect to correlation in a minute. But it's basically saying do I have if if I have a picture of Krishna where do I find him? I'll find him in one place. But you know what? Krishna doesn't look like anybody else. So, I might not find any other faces. And if my objective is to find all the faces then maybe that idea won't work either. Or to take another example here's a dollar bill.

**[33:13]:** We haven't had raises in quite a while, so this is my last one. Uh it's got a picture of George Washington on it. And I can look over all over the class and and if I use this as a face detector I'd be sorely disappointed because I wouldn't find any faces. Because thank god, nobody looks exactly like George Washington, so the correlation wouldn't wouldn't work very well. So, that idea's a loser.

**[33:41]:** But wait a minute. I don't have to look for the whole face. I could I could just look for uh eyeball for eyes and then I could look for noses and maybe mouths, and maybe I have a library of 10 different eyes and 10 different noses and 10 different mouths. Uh would that idea work?

**[34:06]:** Probably not so well. Uh trouble with that one is I'd find eyeballs in every doorknob. There's just not enough stuff there to uh to uh give me a a reliable correlation. So, let's uh make this a little more concrete by drawing some pictures.

**[34:29]:** Halloween is approaching. So here's a face.

**[34:42]:** All right? Here's another face.

**[34:55]:** So, those are might be faces in my pre-recorded library of pumpkin faces. Now, along comes this face.

**[35:13]:** What's going to happen? Well I don't know. Let's try to get another face.

**[35:31]:** I don't know, that could be pretty weird pumpkin face, I suppose, but I'm I mean it to be something that doesn't look very much like a face. So, if I'm doing a complete correlation with either of these faces in my library, either one will match this one very well. Right? If I'm looking for fine features like eyes then I've got these eyes everywhere. So, it doesn't help very much. So, you can see where I'm going, and you can reinvent Ullman's great idea. What is it? We don't look for big features like whole faces. We don't look for small features like individual eyes. We look for intermediate features. Like two eyes and a nose or a mouth and a nose. So, when we do that then we can say

**[36:31]:** well here are two eyes and a nose. Ah, that's found in this one.

**[36:42]:** And what about the combination of that nose and that mouth? Oh, that's over here. And neither of those features can be found in the third image in the in the fourth image. So, that's the Goldilocks principle. When you're doing this sort of thing, you want things that are not too small, not too big. I've got the the Rumpelstiltskin pic uh the Rumpelstiltskin principle up there, too, by the way, because I I meant to mention that Marr was a genius at naming things. And even though um many of his theories uh have faded he's still known for these names like primal sketch and two and a half-D sketch because he was such an artist at naming the concepts. He even got credit for a lot of stuff that he didn't do. Uh not because he was deliberately trying to to uh to get it inappropriately, but just because he was so good at naming stuff.

**[37:36]:** So, we had the Rumpelstiltskin principle back then, and now we have the Goldilocks principle. Not too big not too small. But that leaves us with the uh final question, which is okay, so if what we want to do is look for intermediate size features uh how do we actually find them in a sea of a sea of faces out there? See, I might have a library I might take 10 of you and record your eyes take another 10, record your mouths and they may be put together in a in a in a unique way for each of you out there, but it's it's it's likely that I'll find Lana's eyes somewhere else in the in the crowd. And uh Nicholas' mouth somewhere else in the crowd. So, how how do we in fact go about finding them? And I mentioned the term correlation a couple of times now.

**[38:24]:** Let me let me make that concrete.

**[38:31]:** So, let's consider a um a one-dimensional face that looks like this.

**[38:47]:** It's just a signal. And I'm going to consider a one-dimensional image. And in that one-dimensional image

**[39:03]:** I've got a facsimile of the face. And the question is, what kind of algorithm can I use to determine the offset in the image where the face occurs. And so, you can you can see that one possibility is you just do an integral of the signal in the face and the signal out here over the extent of the face and and see how it multiplies out. Or to make it less Lorelei and more MIT-ish, let's say that what we're going to do is we're going to maximize

**[39:40]:** over some parameter X the interval over X of some face which is a function of X and the image G which is a function of X minus that offset. So, when T when the offset T is equal to this offset then we're essentially multiplying the thing by itself and integrating over the extent of the face. And that gives you a very big number if they're lined up and a very small number if they're not. And it's even true if we add a whole lot of noise to the images. But these are images they're not one-dimensional. That's okay. It's easy enough to make a modification here. I'm going to maximize over translation parameters X and Y and these are no longer functions of just X. They're also functions of Y.

**[40:56]:** Like so. All right. So, that's that's basically how it works. So, we won't go into details about normalization and all that sort of thing because that's the stuff of which other courses uh remain the custodians. So, would you like to see a demonstration? Okay.

**[41:27]:** Let's see where we are here.

**[41:36]:** All right. So, without realizing it Nicola and Erica have loaned us their pictures and they are embedded in that big field of noise. And it's pretty easy to pick out Erica and Nicola, right? Because we are actually pretty good at at at at picking faces out of these out of these images. So, um let's add some noise.

**[42:05]:** It's a little harder now. What I'm going to do is I'm going to run this correlation program over this whole image using Nicola's face as a mask and seeing where the correlation pit peaks up in spite of all the noise that's in there.

**[42:28]:** Boom, there he is. Okay? I don't know. Maybe we can find Erica, too. I forgot where she was. Uh can't find her.

**[42:44]:** There she is. So, unfortunately the parameters aren't very good here. Do you see that Do you see that? Let me get a another version of this. Just do some Should we do some real-time programming?

**[43:08]:** I've been trying to reset the parameters so that they come out so so that the images the demonstration comes out clearly up there. See if this works a little better. Okay, so let's add some noise. And let's find Erica.

**[43:28]:** There she is. Okay? There's some other things that look a little bit like Erica, but nothing looks quite exactly like Erica. So, let's try uh Nicola's eyes.

**[43:45]:** So, they stand out pretty clearly against the background. Let's see if we can find Erica's eyes.

**[43:54]:** So, they stand out pretty clearly against the background. Notice that it also gets Nicola's eyes. So, two eyes is an intermediate size constraint. It's loose enough that it will match more than one person but it's not so loose that it's as bad as looking for one eye. See, they're all over the place. So, two eyes and a nose a mouth and a nose that would be even better as an intermediate feature. But it doesn't matter what the best ones are because you can work that out experimentally. So, that's how correlation works and it's just amazing how much noise you can add and they'll still pick out the right the right stuff. There's Nicola. Boom. Very clear. Want to add some more noise? I don't know. I I can see it, but that's because I'm a pretty good correlator, too.

**[44:55]:** Boom. I don't know. Let's add some more noise.

**[45:04]:** It's just hell hard to get rid of it. I mean, it's just amazing how well it picks it up. That's That's good. That's cool. Now, how could you uh but this the reason that this is 30 years and we're still not done is there's still some questions. This is recognizing stuff straight on. How is it I can recognize you in the hall from the side? Nobody knows. One possibility is that you have an ability to make those transformations. If so, then that alignment theory has a role to play. Another theory is that well, after I've seen you once I can watch you turn your head and keep recording what you look like at all possible angles. That would work. Trouble is is there enough stuff in there? Maybe. We don't know. Uh what would it take to break this mechanism? Well, I don't know. Let's see if we can break the mechanism.

**[46:03]:** Um

**[46:08]:** Let's see if you can recognize some well-known faces.

**[46:15]:** Who's that? Quick. Oh, that's too easy. Hm. Well, we'll see if we can make some harder ones. Yeah, that's Obama. Who's this? Oh boy, you're really good at this. That's Bush. How about this guy?

**[46:38]:** Okay, now I've got it. Some people are starting to turn their heads and that's not fair. That's not fair. Cuz you see what's happened is that if this if this kind of pumpkin theory is correct then when you turn the face upside down you lose the correlation of those features that have vertical vertical components. So, if you have two eyes and a nose it won't match two eyes and a nose when they're turned upside down. Well, let's see. Try some more. Who's that? Gorbachev. Who said that? Leonid, where are you? It's Gorbachev, right? You can recognize him because of that little birthmark on his top of his head. One more. Who's Oh, that's that's easy, right? Who is it? That's that's Clinton. How about this one?

**[47:34]:** You see how insulting it is to be at MIT? That's me.

**[47:43]:** And you didn't even know. Oh god.

**[47:52]:** So, this suggests that this it might be evidence for the correlation theory, but of course turning the face upside down would make it very difficult to do alignment, too. So, it would break alignment theory as well. Let me get that after class. Was there a mistake or

**[48:14]:** It would would break the stretch? What? Stretching? Uh the Elliot asked if stretching would break the correlation and and the answer is I think stretching in the vertical dimension is worse than stretching in the horizontal dimension. Cuz you get a certain amount of stretching in the horizontal dimension when you just turn your head. By the way, since our faces are basically mounted on a cylinder uh this kind of transformation uh might actually work. That's a sidebar to the answer to your question, Elliot. But now you say well, okay. So, this is not completely solved. You can you can work this out. Uh but if you really want to work something out let me tell you what the current what the current questions are in computer vision. People have worked for an awful long time in this recognition stuff.

**[49:13]:** And to my mind I've neglected the more serious questions. And the more serious questions are how do you visually determine what's happening? If you could write a program that would reliably determine when these verbs are happening in your field of view, I will I will sign your PhD thesis tomorrow. There are 48 of them there. And that is today's challenge. But since we're short on time, I want to skip over that and perform an experiment on you. I want you to tell me what I'm doing. So, the best single word answer is M- Manisha? Okay, this is not a trick question. Okay, the best single word answer. Christopher, what do you think? Toast. Christopher. Well, you. It You. Toasting. What? Is it going to fall? Toasting. Okay, not a trick question. What's happening here?

**[50:18]:** Best single word answer? Drinking. It's drinking. Which pair look more alike?

**[50:32]:** So, that cat is drinking and nobody has any trouble recognizing that. And I believe it's because I believe it's because you're telling a story. So, our power of storytelling even reaches down into our visual apparatus. So, the story here is that some animal has evidently had an urge to find something to drink and water is passing through that animal's mouth. That's the drinking story. So, even though they look enormously different visually, the stuff at the bottom of our vision system provides enough evidence for our story apparatus so that we can give the left one and the right one different labels and recognize the cat is drinking.

**[51:16]:** And that's the end of the story.
