Transcript
The following content is provided under a Creative Common License. Your support will help MIT Open Courseware continue to offer highquality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT Open Courseware at ocw.mmit.edu. It was um in 2010. Yes, that's right. It was in 2010. We were having our annual discussion about what we would dump from 6034 in order to make room for some other stuff. And we almost killed off neural nets. Now that might seem strange because um you know our heads are stuffed with neurons. If you open up your skull and pluck them all out, you don't think anymore.
So it would seem that um neural nets uh would be a fundamental and unassalable topic. But many of us felt that the uh neural models of the day weren't much um in the way of um faithful models of what actually goes in inside our heads. And besides that, nobody had ever made a neural net that was worth darn for doing anything. So, um we almost killed it off. But then we said, well, the everybody will feel cheated if they take a course in artificial intelligence, don't learn anything about neural nets, and then they'll go off and invent them themselves and waste all sorts of time. So, we kept the subject in.
Uh, then two years later, Jeff Hinton from the University of Toronto stunned the world uh with some real net work he had done on recognizing and classifying pictures. and he published a paper from which I am now going to show you a couple of examples. Uh Jeff's neural net by the way had 60 million parameters in it and it was its purpose was to determine which of a thousand categories best characterized a picture. So there it is. There's a sample of things that um the Toronto neural net uh was able to uh recognize or make mistakes on. I'm going to blow that up a little bit. I think I'm going to look particularly at the uh example there labeled containership.
So what you see here is that the uh program returned its best estimate of what it was ranked first five according to the likelihood of probability or the certainty that it felt that a particular class was characteristic of the picture. And so you can see this one is extremely confident that it's a container ship. It also was uh fairly u moved by the idea that might be a lifeboat. Now, I'm not sure about you, but I don't think this looks much like a lifeboat, but it does look like a container ship. So, if I look at only the best choice, it looks pretty good. Here are the other things that it did pretty well on.
Got the right answer as the first choice is this first choice. So, over on the left, you see that it's decided that the picture is a picture of a might. Uh the might is not anywhere near the center of the picture, but somehow it managed to find it. The container ship. Again, there's a motor scooter, couple people sitting on it, but it correctly characterized the picture as a motor scooter. And then on the right, a leopard, and everything else is a cat of some sort. So, it seems to be doing pretty well. In fact, it does do pretty well. But anyone who uh does this kind of work has an obligation to show you some of the stuff it doesn't work so well on or doesn't get quite right.
And so, these pictures also occurred in Hinton's paper. So the first one is characterized as a grill but the right answer was supposed to be convertible. Oh no. Yes. Yeah. Right answer was convertible. In the second case um the characterization is of a mushroom and the alleged right answer is agaric. Is that pronounced right? It turns out that's a kind of mushroom. So no problem there. In the next case it said it was a cherry but it was supposed to be a dalmatian. Now, I think a dalmatation is a perfectly legitimate answer for that particular picture. So, hard to fault it for that. And in the last case, it didn't, you know, the the correct answer was not in any of the top five.
I I'm not sure if you've ever seen a Madagascar cat, but that's a picture of one. And it's interesting to compare that with the uh first choice of the program, squirrel monkey. This is the two side by side. So, in a way, it's not surprising that it thought that the Madagascar cat was a picture of a squirrel monkey. So, pretty impressive. It blew away the competition. It did so much better that second place wasn't even close. And for the first time, demonstrated that a neural net could actually do something. And since that time, in the three years since that time, there's been an enormous amount of effort put into neural net technology, which some say is the answer.
So what we're going to do today and uh tomorrow uh is have a look at this stuff and uh ask ourselves why it works when it might not work what needs to be done what has been done and all those kinds of questions will emerge. So I guess the first thing to do is think about what it is that we are being inspired by. uh we're being inspired by those things that are growing that are inside our head all 10 to the 11th of them. And so if we take one of those 10 to the 11th and look at it uh you know from uh 70 something or other approximately what a neuron looks like. And by the way I'm going to teach you uh in this lecture how to answer questions about neurobiology with an 80% probability that you will give the same answer as a neurobiologist.
Okay. So let's go. So, here's a a neuron. It's got a cell body. In there is a nucleus. And then out here is a long thing which divides maybe a little bit but not much. And we call that the axon. So then over here we got this uh much more branching type of structure that looks maybe a little bit like so. I don't know, maybe like that. And this stuff branches a whole lot. And that part is called the dendritic tree. Now, uh there are a couple of things we can uh note about this is that these guys are connected axon to dendrite. So over here there'll be a a so-called presaptic thickening and over here will be some other neurons dendrite and likewise over here some other neurons axon is coming in here and hitting the dendrite of our of of the one that occupies most of our So if there's enough uh stimulation from this side in the axonal tree or in the dendritic tree then a spike will go down that axon.
It acts like a transmission line and then after uh that happens uh the act the neuron will go quiet for a while as it's kind of recovering it strength. is called the refractory period. Now, if we look at um that connection in a little more detail, this little piece right here, it sort of looks like this. Here's the axon coming in. It's got a whole bunch of little vesicles in it. And then there's a dendrite over here. And when a and when the axon is stimulated, it dumps all these vesicles into this inner synaptic space. For a long time, it wasn't known whether those things were actually separated.
I think it was Ramoni Kahal who demonstrated that the that one neuron is actually not part of the next one. They're actually separated by these synaptic uh by these synaptic gaps. So, oh, there it is. How can we model that sort of thing? Well, here's what's usually done. Here's what is done in the neural net literature. First of all, uh we've got some kind of binary input because these things either fire or they don't fire. So it's an all ornone kind of situation. So over here we have some kind of input value. We'll call it x1 and it's either a zero or a one. So it comes in here and then it gets multiplied times some kind of weight.
We'll call it W one. So this this part here is sort of modeling this this synaptic connection. It may be more or less strong and if it's more strong this weight goes up and if it's less strong this weight goes down. So that's the that's the that's the influence that that reflects the influence of the synapse on on on whether or not the whole axon decides it's stimulated. And we got other inputs down here. X subn also 01. It's also multiplied by a weight. We'll call that W subn. And now we have to somehow uh represent the way in which these uh these inputs uh are collected together. How how they have collective force and we're going to model that very very simply just by saying okay we're running through a summer like so.
But then we have to decide if the collective influence of all those uh inputs is sufficient to make the neuron fire. So we're going to do that by running this guy through a threshold box like so. Here is what the box looks like in terms of the relationship between the input and the output. And what you can see here is that nothing happens until the input exceeds some threshold t. If that happens then the output Z is a one otherwise it's a zero. So binary in binary out. We model the synaptic weights by these multipliers. We model the accumulating accumulative effect of all that input to the neuron by a summer.
We decide if it's going to be the an all or none one by running it through this threshold box and seeing if the sum products add up to more than the threshold. If so, we get a one. So what in the end are we in fact modeling? Well, with this model we have number one all or none. Number two cumulative influence. Number three. Oh, I suppose synaptic weight. But that's not all that there might be to model in a real neuron. We might want to deal with the refractory period. Not noted in these biological models that we build neural nets out of. We might want to model axonal bifurcation. We do get some division in the axon of the neuron.
And it turns out that that pulse will either go down one branch or the other. And which branch it goes down depends on electrical activity in the vicinity of the division. So these things might actually be fantastic coincidence detectors, but we're not modeling them. We don't know how it works. So axonal bifurcation might be modeled. We might also have a look at time patterns. See what we don't know is we don't know if the timing of the arrival of these pulses in the dendritic tree has anything to do with what that neuron is going to recognize. Right? So there a lot of unknowns here. And now I'm going to show you how to answer a question about neurobiology with 80% probability you'll get it right.
Just say we don't know. And that'll be with 80% probability what the neurobiologist would say. So this is a model inspired by what goes on in our heads. But it's far from a it's far from clear if what we're modeling is the essence of why those guys make possible what we can do. Nevertheless, that's where we're going to start. That's where we're going to go. So, we've got this model of what it's of what a neuron does. So, what about what does a collection of these neurons do? Well, we can think of your skull as a big box full of neurons. Uh maybe a better way to think of this is that your head is full of neurons and they in turn are full of weights and thresholds like so.
So into this box come a variety of inputs X1 through X M. And these find their way to the inside of this gaggle of neurons. And out here come a bunch of outputs Z1 through Z N. And there a whole bunch of these maybe like so. And there are a lot of inputs like so. And somehow uh these inputs um through the influence of the weights and the thresholds come out as a set of outputs. So we can we can write that down a little fancier by just saying that Z is a vector which is a function of um well certainly the input vector but also the weight vector and the threshold vector. So that's all a neural net is.
And when we train a neural net, all we're going to be able to do is adjust those weights and thresholds so that what we get out is what we want. So a neural net is a function approximator. It's good to think about that. It's a function approximator. So maybe we've got uh some sample data uh that gives us an output vector that's desired as a function as another function of the input forgetting about what the weights and the thresholds are. That's what we want to get out. And so how well we're doing can be figured out by comparing uh the desired value with the actual value. So we we might uh think then that we can get a handle on how well we're doing by constructing some performance function which is determined by the desired vector uh and uh the input vector sorry the the desired vector and the actual output vector for some particular input or for some set of inputs.
And the question is what what should that function be? How should we measure performance given that we have what we want out of here and what we actually got out of here? Well, one simple thing to do is just to measure the magnitude of the difference. That makes sense. But of course uh that would give us a uh performance function that is a function of the distance between those vectors would look like this. But this turns out to be um mathematically inconvenient in the end. So how do you think we're going to tart it up a little bit? What's that? Well, I don't know. How about just we square it that way?
We're we're we're going to go from this little sharp point down there to something looks more like that. So, it's best when uh the difference is zero, of course, and it gets worse as you move away from zero. Um but what we're trying to do here is we're trying to get to a minimum value. And and and I I hope you'll forgive me. I I just don't like that the direction we're going here because I like to think in terms of improvement as going uphill instead of downhill. So I'm going to dress this up just one more step. Put a minus sign out there. And then our performance function looks like this. It's always negative and it gets and and the best value it can possibly be is zero.
So that's what we're going to use just because I am who I am. It doesn't matter, right? still you're you're trying to either minimize or maximize some performance function. Okay, so what are we going to do? I guess what we could do is we could treat this thing. Well, we already know what to do. I'm not even sure why we're devoting a lecture to this because it's clear that what we're trying to do is we're trying to take our weights and our thresholds and adjust them so as to maximize performance. So we can make a little contour map here with a simple neural net with just two weights in it. And maybe it looks like this contour map.
And at any given time we've got a particular W1 and particular two W2 and we're trying to find a better W1 and W2. So here we are right now and there's the contour map and this is 6034. So what do we do? a simple simple matter of hill climbing, right? So, we'll take a step in every direction. If we take a step in that direction, not so hot. That actually goes pretty bad. These two are really ugly. Ah, but that one that one takes us up a hill a little bit. So, we're done. Except that I just mentioned it. Hinton's neural net has 60 million parameters in it. So we're not going to hill climb with 60 million parameters because that you know it explodes exponentially in the number of weights you've got to deal with the number of steps you can take.
So that so this approach is computationally intractable. Fortunately you've all taken 801 1801 or the equivalent thereof. So you have a better idea. Instead of just taking a step in every direction, what we're going to do is we're going to take some partial derivatives and we're going to see what the they suggest to us in terms of how we're going to get get around in this space. So we might have the partial of that performance function up there with respect to W1 and we might also take the partial derivative of that guy with respect to W2. And these will tell us how much improvement we're getting by making a little movement in those directions, right?
How much how much of changes given that we're just going right along the axis. So maybe what we ought to do is if this guy is much bigger than this guy, it would suggest that we mostly want to move in this direction. Or to put it in 1801 terms, what we're going to do is we're going to follow the gradient. And so the change in the W vector is going to equal to this partial derivative time I plus this partial derivative time J. So what we're going to end up doing in this particular case by following that formula is moving off in that direction right up right up the steepest part of the hill. And you know how much we move is a is a question.
So, let's just have a rate constant R that decides how big our step is going to be. And now you think we were done. Well, too bad for our side. We're not done. Uh there's a reason why we can't use gradient ascent or in the case that I've drawn or gradient descent. If we take the performance function the other way, why can't we use it? Uh the remark is local maxima and that is certainly true but it's not our first obstacle when does gradient descent work ah there's something wrong with our function that's right it's nonlinear or it's rather it's discontinuous so gradient descent requires a continuous space continuous surface So too bad for our side.
It isn't. So what to do? Well, nobody knew what to do for 25 years. People were screwing around with training neural nets for 25 years before Paul Werbos sadly at Harvard in 1974 gave us the answer. Now I want to tell you what the answer is. First part of the answer is those thresholds are annoying. They're just they're just extra baggage to deal with. What we really like instead of Z being a function of X, W, and T was like C prime to be a function FP prime of X and the weights. But we got to account for the threshold somehow. So here's how you do that. What you do is you say, let us add another input to this neuron and it's going to have a weight w.
All right? And it's going to be connected to an input that's always minus one. You with me so far? We're just going to add another. Now what we're going to do is we're going to say let w 0 equal t. What's that do to to movement of the threshold? What it does is it takes that threshold and moves it back to zero. So this little trick here takes this pink threshold and re redo it so that the that the new threshold box looks like this. Right? Think about it. If if this is t and this is minus one, then this is minus t. And so this thing ought to fire if everything's over if the sum is over zero. So it makes sense.
It gets rid of the it gets rid of the threshold thing for us. So now we can just think about weights. But still we've got that we've got that uh we've got that step function there. And uh that's not good. So what we're going to do is we're going to smooth that guy out. So this is trick number two. Instead of a step function, we're going to have this thing we uh lovingly call a sigmoid function because it's kind of got an S-type shape. And the function we're going to use is this one. One well better make it a little bit different. 1 over 1 + e to the minus whatever the input is. Let's call the input alpha.
Does that make sense? Let's see. If alpha is zero, then it's 1 over 1 plus 1. So that's 1/2. If alpha is extremely big, then e to the minus alpha is extremely small and it becomes one. Goes up to an asmtoic value one here. On the other hand, if alpha is extremely negative, then e to the minus alpha is extremely positive and it goes to zero asmtoically. So we got the right look to that function. It's a very convenient function. Did God say that neurons ought to be has that threshold ought to work like that? No. God didn't say so. Who said so? The math says so. it has the right shape and look and the math and it turns out to have the right math.
We'll as we'll see in a moment. Okay, so let's see where are we? We decided that what we'd like to do is take these partial derivatives. We noted that it was awkward to have those thresholds, so we got rid of them. And we noted it was impossible to have the step function, so we got rid of it. Now we're in a situation where we can actually take those partial derivatives and see if it gives us a way of training the neural net. so as to bring the actual output into alignment with what we desire. All right? So to deal with that, we're going to have to work with the world's simplest neural net. Now, if we've got one neuron, it's not a net.
But if we've got two word neurons, we've got a net. Turns out that's the world's simplest neuron. So we're going to look at it. Not 60 million parameters, but just a few. Actually, just two parameters. So let's draw it out. We've got an input X that goes into a multiplier and it gets multiplied times W1. Then that goes into a sigmoid box like so. We'll call this P1, by the way, product number one. Out here comes Y. Y gets multiplied times another weight. We'll call that W2. The neck produces another product which we'll call P2 and that goes into a sigmoid box. And then that comes out as Z. And Z is the number that we use to determine how well we're doing.
And our performance is actually going to be our performance function P is going to be 12 minus 12 because I like things are going in that direction times the difference between the desired output and the actual output squared. Okay. So now let's decide what those partial derivatives are going to be. Oh, let me do it over here. So, what are we trying to compute? Partial of the performance function P with respect to W2. Okay. Well, let's see. you know, partial. We're trying to figure out how much this wiggles when we wiggle that, right? But, you know, it goes through this variable P2. And so, maybe what we can do is figure out how much this wiggles when we wiggle, how much Z wiggles when we wiggle P2, and then how much P2 wiggles when we wiggle W2.
And just multiply those together. I forget what's that called. in 180 something or other the chain rule. So what we're going to do is we're going to rewrite that der that partial derivative using chain rule. And all it's doing is saying that there's an intermediate variable and we can compute how much that end wiggles with respect to how much that end wiggles by multiplying the how much of the other guys wiggle. Let me write it down. It makes more sense in mathematics. So that's going to be equal to the partial of P with respect to Z times the partial of P. Oh, sorry. Partial of Z with respect to P2.
Ah, keep me on track here. Partial of Z with respect to W2. Now, I'm going to do something for which I will hate myself. I'm going to erase something on the board. I I don't like to do that. But but you you you know what I'm going to do, don't you? I'm going to say this this this is true by a chain rule. But look, I can take this guy here and screw around with it with a chain rule, too. And in fact, what what what I'm going to do is I'm going to replace that with partial of Z with respect to P2 and partial of P2 with respect to W2. Right? So, I didn't erase it after all, but you can see what I'm going to do next.
I'm going to do the same thing with the other partial derivative, but this time instead of writing down and writing over, I'm just going to expand it all out in one go. I think so. Partial of P with respect to W1 is equal to the partial of P with respect to Z. The partial of Z with respect to P2. The partial of P2 with respect to what? Y. partial of y with respect to p1 partial of p1 with respect to w1. So that's just kind of going like a zipper down that that string of variables expanding each by using a chain rule until we got to the end. So there are some expressions that provide those partial derivatives.
But now, if you don't if if you'll forgive me, it was convenient to write them out that way because that that matched the intuition in my head, but I'm just going to turn them around. I'm not just a it's just a product. I'm just going to turn them around. So partial P2 partial W2 times partial of Z partial P2 times the partial of P with respect to Z. Same thing. And now this one keep me on track because if there's a mutation here it will be fatal. partial of P1 partial W1 partial of Y partial P1 partial P2 partial of Y partial of Z times a partial of P2 partial of performance function with respect to Z.
Okay. And now all we have to do is figure it out. Figure out what those partials are. And we uh have solved this little simple neural net neural net. So it's going to be easy. Oh, where where is my board space? Let's see. Partial of P2 with respected to um what partial that's that's the product. Partial of Z partial of the performance function with respect to Z. Oh, now I can see why I wrote it down this way. Let's see. is going to just be d minus e. We can do that one in our head. What about the partial of p2 with respect to w2? Well, p2 is equal to y * w2. So that's easy. That's just y. And now all we have to do is figure out the partial of Z with respect to P2.
Oh crap. All it's just it's going through this this threshold box. So I don't know exactly what that partial derivative is. So we'll have to figure that out, right? Because uh the function relating them is this is this guy here. And so we have to figure out the partial of that with respect to alpha. All right. So we got to do it. There's no qu there's no way around it. So, uh, we have to destroy something. Okay, we're going to destroy our neuron. So the function we're dealing with is we'll call it beta is equal to 1 / 1 + e to the minus alpha. And we want what we want is the derivative with respect to alpha of beta and that's equal to d by d alpha of you know I I can never remember those quotient formulas.
Uh so I'm going to rewrite it a little different way. I'm going to write it as 1 minus e to the minus alpha to the minus one. So I because I can't remember the formula for differentiating quotient. Okay. So let's differentiate it. So uh that's equal to 1 - alpha 1 - e to the minus alpha to the minus 2 and we got that minus comes out out of that part of it. Then we got to differentiate the it's got to differentiate the inside. of that expression and when we differentiate the inside of that expression we get e to the minus alpha. Yeah. Oh yeah. Sorry. Thank you. That was that was one of those fatal mistakes you just prevented.
So that's one plus. That's one plus here too. Okay. So so we we've differentiated that. We've turned that into a minus2. We brought the minus sign outside. Then we're differentiating the inside. The derivative of an exponential is an exponential. Then we got to differentiate that guy and that just helps us get rid of the minus sign we introduced. So that's the derivative. I'm not sure how much that helps except that I'm going to perform a polar trick here and rewrite that expression thusly. I'm going to say that's going to be um eus alpha over 1 + e to the minus alpha times 1 / 1 + eus alpha that okay I've got a lot of nodding heads here so I think I'm safe on safe ground but now I'm going to perform another polar trick I'm going to add one, which means I also have to subtract one.
All right, that's legitimate, isn't it? So now I can rewrite this as 1 + eus alpha over 1 + eus alpha minus 1 / 1 + eus alpha times 1 over 1 + e to the minus alpha. Any high school kid could do that. I think I'm on safe ground. Oh, wait. This is beta. This is beta. Oh, sorry. Wrong wrong wrong side. Better make this beta. And this one. Any high school kid could do it. Okay. So, what we've got then is that this is equal to 1 minus beta time beta. That's the derivative. And that's weird because the the derivative of the output with respect to the input is given exclusively in terms of the output.
Strange doesn't really matter, but it's a curiosity. And what we get out of this is that that partial derivative there that's equal to well the output is P2. No, the output is Z. So it's Z * 1 - Z. So whenever we see the output derivative of one of these sigmoids with respect to its input, we can just write the output times one minus the output. We've got it. So that's why it's mathematically convenient. It's mathematically convenient because when we do this differentiation, we get a very simple expression in terms of the output. We get a very simple expression. That's all we really need. So would you like to see a demonstration?
This is a demonstration of world's smallest neural net in action. Where is neural net? There we go. So there's our neural net. And what we're going to do is we're going to train it to do absolutely nothing. What we're going to do is train it to have make the output the same as the input. Not what I call a fantastic leap of intelligence, but let's see what happens. Wow. Nothing's happening. Well, it finally got to the point where the maximum error, not the performance, but the maximum error went below a threshold that I had previously determined. So if you look at the input here and compare that with the desired output on the far right, you see it produces an output which compared with the desired output is pretty close.
So we can test uh the other way like so. And we can see that the desired output is pretty close to the desire actual output in that case too. And it took 694 iterations to get that done. Let's try it again. Oh, to 823. Of course, this is all a consequence of just starting off with random weights. By the way, if you started off with the weights, all the weights being the same, what would happen? Nothing, because it'd always stay the same. So, you got to put some randomization into the beginning. So that took a long time. Maybe the problem is that our rate constant is too small. So let's try let's crank up the rate constant a little more a little bit and see what happens.
Ah that was pretty fast. Let's see if it was a consequence of uh random chance uh run. No, it's pretty fast there. 57 times 57 iterations. Third try 67. So it looks like my initial rate constant was too small. So if uh 0.5 was not as good as 5.0, how why don't we crank it up to 50 and see what happens? Oh, in this case 124. Let's try it again. H in this case 117. So it's actually gotten worse. And not only has it gotten worse, you'll see that there's a little bit of uh a little bit of instability showing up as it courses along its way toward a solution. So what it looks like is that if you've got a rate rate constant too small, it takes forever.
You got a rate constant that's too big, it can sort of jump too far. As in my diagram, which is somewhere underneath a board, you can you can go all the way across the hill and get to the other side. So you have to be careful about the rate constant. So what you really want to do is you want your rate constant to vary with what's happening in the in as you progress toward uh an optimal part an optimal performance. So if you if your if your performance is going down when you make the jump, you know you've got a rate constant is too big. If your performance is going up when you make a jump, maybe you want to increase bump it up a little bit until it until it until till it doesn't look so good.
Okay. So uh is that all there is to it? Well, not quite because uh this is the world's simplest neural net. And maybe we ought to look at the world's second simplest neural net. Let's call this uh well, let's call this X. What we're going to do is we're going to have a second input. And I'm just I don't know, maybe this is screwy. I'm just going to use color coding here to differentiate between the two inputs and and the and the stuff they go through. Maybe I'll call this Z2 and this Z1 and this X1 and X2. Now, if I do that, if I've got two inputs and two outputs, then then uh my performance function is going to have two numbers in it, the two desired values and the two actual values.
And I'm going to have two inputs. But, you know, it's the same stuff. I just repeat the I just repeat what I did in white only I make it orange. Oh, but what happens if um what happens if I I I do this? So I put a little cross connections in there. So these these these two streams are going to interact. And then there might be some you know this this Y can go into another multiplier here and go into a sum here. And likewise this Y can go up here and into a multiplier like so. And there are weights all over the place like so. This guy goes up into into here. And now what happens now? We've got a disaster on our hands because there are all kinds of paths through this network.
And you can imagine that if this was not just two neurons deep, but three neurons deep, what I would find is expressions that look like that. But, you know, you could go this way and then down through here and out here. Or you could go this way and then back up through here. So, it looks like there is an exponentially growing number of paths through that network. And so we're back to an exponential blow up and it won't work. Uh yeah, it won't work. Except that we need to let the math sink to us a little bit and we need to look at the picture. And the reason I turned this guy around was actually because from the point of view of letting the math sink to us, this piece here is the same as this piece here.
So part of what we needed to do to calculate the partial derivative with respect to W1 has already been done when we calculated the partial derivative with respect to W2. And not only that, if we calculated the partial with respect to these green W's at both levels, what we would discover is that that sort of repetition occurs over and over again. And now I'm I'm going to try to give you an intuitional idea of what's going on here rather than just write down the math and salute it. And and here's here's here's a way to think about it from an intuitional point of view. Whatever happens to this performance function that's back of this row back back of these P's here the stuff over there can influence P only only by going through can influence performance only going through this column of PS and there's a fixed number of those it depends on the width not the depth of the network So the influence of that stuff back there on P is going to end up going through these guys and it's going to end up being so that we're going to discover that a lot of what we need to compute in one column has already been computed in the column on the right.
So it isn't going to explode exponentially because the influence let me say it one more time. The influence of changes the influences of changes in P on the performance is all we care about when we come back to this part of the network because this stuff cannot influence the performance except by going through this column of peace. So it's not going to blow up exponentially. we were going to be able to reuse a lot of the computation. So, it's the reuse principle. Have we ever seen the reuse principle at work before? Not exactly. But you remember that little business about the extended list? We we we we know that we've seen how we we know we've seen something before.
So, we have we can stop computing. It's like that. We're going to be able to reuse the computation. So we've already done so as to prevent an exponential blow up. By the way, for those of you who know about fast for transform, same kind of idea, reuse of partial results. So in the end, what can we say about this stuff? In the end, what we can say is that it's linear in depth. That is to say, if we increase the number of of layers to so-called depth, then we're going to increase the amount of computation necessary in a linear way because the computation we need in any column is going to be fixed.
What about how it goes with respect to with respect to the width? Well, with respect to the width, any neuron here can be connected to any neuron in the next row. So, the amount of work we're going to have to do will be proportional to the number of connections. So, with respect to width, it's going to be W ^2. But the fact is that in the end this stuff is readily computed and this phenomenally enough was overlooked for 25 years. So, what did it take? What what is it in the end? In the end, it's extremely simple idea. All great ideas are simple. How come there aren't more of them? Well, because frequently that simplicity involves finding a couple tricks and making a couple of observations.
So, usually we we we humans hardly ever go beyond one trick or one observation. But if you cascade a few together, sometimes something miraculous falls out that looks in retrospect extremely s simple. So, that's why we got the reuse principle at work. Run a reuse computation. In this case, the miracle was a consequence of two tricks plus an observation. And the overall idea is all great ideas are simple and easy to overlook for a quarter century.