Transcript
The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseware continue to offer high-quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseware at ocw.mit.edu. Good morning, everyone. Today, we're going to talk about boosting. Boosting is pretty awesome, and it's not as easy as it it's not as hard as it might seem. It's actually pretty easy. As long as you do it right. So, let's take a look at this boosting problem. There's really mainly, just like with ID trees, you may have noticed there's like the ID tree problem where there's a graph and all of the tests are X and Y axis tests on that graph.
And then there's the ID tree problem where there are a lot of crazy different classifiers about sort of characteristics that are discrete, and then all of the sort of ID tree stumps, if you will, are built out of those discrete qualities. This is an example of a boosting problem of the of the second type with a bunch of discrete qualities like evil, emo, transforms, sparkly. And it has a numerical quality, number of romantic interests. So, it's one of basically the two kinds of boosting problems you might see. Another is a is the um two axis Cartesian problem. A good example of that is the hours of sleep versus coffee um problem, which I for one am planning on doing a tutorial on Monday, so that you guys sort of get a sense of both types of problem.
This one looks big. I think it looks I mean, it has 10 different vampires or non-vampires to classify. It has a whole bunch of possible classifiers, but if you do it right, you can do it fast. So, here's the prompt for the problem. Let's see. After graduating MIT, you get a job working for Van Helsing and Summers, a famous vampire hunting consulting agency. Gabriel Van Helsing, one of the two founders, once attended several 6034 lectures as a guest, and he remembers Professor Winston's vampire identification tree lecture. He assigns you the task of creating a superior classifier for vampires by using boosting on the following data.
So, we've got the ID number, which we can use to just write things out in shorthand. We've got the name of several vampires and non-vampires. Then, you see whether they're a vampire or not. That's whether their classification is like a plus or a minus for for the quality vampire. After that, there's uh a bunch of possible ways to classify whether they're a vampire or not. There's whether or not they're evil, whether or not they're emo, whether or not they transform, and whether or not they're sparkly, as well as the number of romantic interests that they have. So, for instance, on the one hand, you have Dracula, who's evil, but he's not emo.
He can transform into a bat or a a cloud of mist. He He does not sparkle, and he has five romantic interests, those three vampire chicks at the beginning, Wilhelmina, Murray, and Lucy Westenra. So, um on the other hand, you have like Squall Leonhart, who's the protagonist of Final Fantasy VIII, is extremely emo and doesn't have any of the other characteristics. And he's not a vampire. However, he's a nice counterexample for a possible rule that all emo people are vampires, because he's very very emo and he's not a vampire. So, how will we go about tackling this problem with boosting? Well, there's a whole bunch of different classifiers, and if you think this is all of them, like evil, emo, transforms, sparkly, romantic interests, and true, it's actually only half of them.
The other half are the opposite versions. But we'll ignore them for now. So, if you look at these, you can probably figure out what they mean. Evil equals yes means vampire is what we're saying here, except maybe true. You might be saying, "Why is there one that just says true?" The one that just says true says that everybody's a vampire. You might think, "Oh, that sucks." But it's not that bad since seven of the 10 samples are vampires. The key crucial thing about boosting is that for any possible classifier, like classifying on the evil dimension, which actually sounds like some kind of weird place that you'd go in a comic book, but classifying on the emo dimension or whatever, as long as it's not a 50/50 split of the data, you're guaranteed to get to be able to use it in some way for boosting.
If it is a 50/50 split, it's like flipping a coin. So, it's useless, because if you say, "Okay, if you had some other thing like gender equals male or female." And let's say that was 50/50. It's not, but let's say it was 50/50 between vampire and non-vampire. It's a useless classifier, because it would be just the same as flipping a coin. You get no information. Now, you might say, "Wait a minute. What about classifiers that get worse than 50/50?" What about them? Might not they be even worse than a 50/50 classifier? I claim a classifier that gets less than 50/50 is still better than a classifier that gets exactly a 50/50 split.
Uh is there a question? Uh So, the question is, in the ID tree example, you might use 50/50 classifiers in later rounds. If, for instance, um there's a 50/50 classifier except for that most of the things off of that side have been already removed. Let's say there's 20 data points, and there's a classifier that splits it 10 and 10. Um it splits it 10 and 10, and it it gets half plus half minus on both sides. Um but all of the pluses from the right side have been removed by some other classifier, you might use it. That's true, but in boosting, you'll never use something that's a 50/50 classifier.
So, you never use something that has um exactly a 50/50 chance of being correct. Because if it has a 50/50 chance of being correct, it's useless. And if it has a 50/50 If it has a 50/50 chance uh No, sorry. Let me Let me specify again. You'll never use something that has a 50/50 that has a 50/50 chance of giving you the right answer given the weights. That's very, very important, and I that may be what your what your question was getting at. As I'm about to show you, and as uh Patrick told you in the lecture, in later rounds of boosting, you change the weights of each of the 10 data points. At first, you start with all the weights being 1/10.
The weights have to add up to one. In this case, you never ever choose a classifier that gets five of them right and five of them wrong. In later rounds, you'll never ever choose a classifier that gets half of the weight wrong. Exactly half of the weight wrong. But half of the weight may not be half of the data points. So, it's possible to choose a classifier that that gets half of the data points wrong if it doesn't get half of the weight wrong. And that's similar to an ID trees when um you've already gotten things right before, because you'll see that the weight is going to go to the ones you got wrong.
So, I'm not saying that you should throw out right away anything that gets five of the points wrong. Hell, you shouldn't even throw out right away something that gets seven of the points wrong. It's possible possible that you can get seven of the points wrong while getting less than half of the weight wrong. If those other three points are really, really annoying to get right. And we'll see that later on. But for insight, at every step along the way, we're willing to choose We're willing to choose any classifier that doesn't get 50/50. However, um we want to choose the classifier that gets the most of the weight right.
By most of the weight, at first, we mean most of the points right. Later, we will mean but exactly what I said, most of the weight. And if you don't understand that, it's sometimes um hard to get it right away um when Patrick just lectures through and introduces the new concept. If you don't understand that, you'll see what I mean when we go through. All right? So, my my um point I was making before is what about things that get less than half of the weight right? Well, those are always okay, because you can just flip them around, use their inverse, and that gets more than half of the weight right.
It's sort of like um Yeah, it's sort of like um my girlfriend always tells me that she is more than 50% likely to choose the wrong direction when you're trying to go between two places, which I'm kind of skeptical of. But I said, "If that's really true, then we can just go wherever you didn't say to go, and we'll be more likely to go the right way. So, you're actually really good at finding the place that we want to go." And then she's like, "No, that won't work, because then I'll I'll know that you're going to do that, and I'll say the I'll double say the wrong way, and then you'll go the wrong way again." But that notwithstanding, you can you can see you can apply the same concept to boosting, and that's why underneath of this, I have all the opposite versions of all of these tests.
So, what should we be doing to um to solve this problem more quickly? First, let's figure out which data points are misclassified by each of these classifiers. In other words, if we say all the evil things are vampires and all the non-evil things are not vampires, what do we get wrong? And we're If we do that for every classifier, then that'll make it faster later on because later on we're going to go through classifiers and we're going to have to add up the ones they got wrong. So, this chart over here is going to be extremely useful. And on the test that this appeared in, they even made you fill it to help yourself out.
So, let's see. If we said that all the evil equals yes are vampires and all the evil equals no are not vampires, then I'll do the first one for you. So, we get all of the non-vampires correct because they are all evil equals no. But unfortunately, we get uh we get Angel, Edward Cullen, Saya Otonashi, and Lestat de Lioncourt wrong because they are vampires. And their evil equals no. Apparently, Lestat is iffy, but I never I never read those books and the Wikipedia article said that in the end he wasn't that evil. So, there we go. Evil equals yes misclassifies 2 3 4 and 5. All right. So, let's try emo equals yes.
I'll have someone else do it. So, uh see if you guys got it. So, if we say that all the emo people are vampires and all the non-emo people are not vampires, what do we get wrong? 1 6 7 9. That's exactly right and fast. Good. We get 1 6 7 and 9 wrong. 1 6 and 7 are wrong cuz they are not emo, but they're vampires. 9 is wrong because Squall is emo and he's not a vampire. Good. Okay. What if we said that exactly the transforming characters are vampires and the ones that do not transform are not vampires? Which ones would get wrong? Transform is just the next one. All of the ones So, which ones would we get wrong if we said that the transforms yes are vampires and transforms no are not vampires?
Brown. We definitely get A wrong because she's not a vampire. Mhm. Hold on, but it's it's on the It's actually on there. And three and four are also Yeah. Three and four are also going to get Yep. Yes, exactly. Oh, yeah. Oh, man, you you didn't see the chart. You were just like, "Mhm." You You saw on the left you were just like, "Mhm, which of these are the transforming characters?" Okay, that's pretty hard to work. But yeah, 3 4 5 and 8. No, no, it's definitely given to you. That would be like the worst test ever for for international students. Uh if you don't know these 10 characters that Mark found as vampires, you lose.
All right. So, what about that sparkly equals yes is a vampire and if it's not sparkly, it's not a vampire. This is not This is guaranteed not to go well. What do you think it's going to get wrong? The sparkly Yeah, sparkly equals yes are the only vampires. They're going to get it wrong. And they're going to get it wrong. Yep. Uh-huh. Saya Mhm. So, 1 3 4 5 Mhm. 6 Yep. 7 Uh yeah, okay. And 8. Yes, that's right. It gets 1 2 4 5 6 7 and 8 wrong. That's pretty awful. But damn it, it gets Edward Cullen right and he's hard to get correct. Due to the fact that he's not really much like a vampire. He's more of a superhero who says he's a vampire.
Okay. So, next, number of romantic interests greater than two. So, if they have more than two romantic interests, they're a vampire and otherwise they're not a vampire. So, which ones would those get wrong? Would that get wrong? Three and 10. Just 3 and 10. That's right. Because Circe had Odysseus, she had Telemachus actually, she had that guy she turned into a woodpecker, she had that other guy who was a sea god who caused her to turn Scylla into the nine-headed thing, and probably at least one other person. So, Circe gets wrong and it also gets Edward Cullen wrong because he only has one. So, 3 and 10.
You can tell I thought about this problem when I was writing it up. I wrote this one. All right. Number of romantic interests greater than four. So, it's a little bit different this time. Now you have to have at least four romantic interests or actually greater than four, but there are none that are exactly four to be classified as a vampire. Which ones do you think it's going to get wrong? Three, four, 10. Yep. It is going to get 3 4 and 10 wrong. Cuz now you run into the fact that Saya has that blond chevalier guy, and Kai. So, the last of the positive ones because I I claim I can derive all the negative ones if you guys give me the positive ones.
Last of the positive ones is everybody's a vampire. Who does that get wrong? Um 8 9 Yes. Okay. Now, I can derive all the negative ones from this without a sweat. Evil equals no, well, it's 1 6 7 8 9 10. Without looking at the chart. Raise your hand if you don't see why Raise your hand if you see why it's 1 6 7 8 9 10 without looking at the chart. Raise your hand if you don't. Nobody. Okay. Wait, one hand. Okay, I saw another one back there too later. They were just more tentative. Okay, it's the complement of A. Because A is evil equals yes is a vampire. It gets 2 3 4 and 5 wrong. So, therefore evil equals no is a vampire is going to guarantee to get all the opposite ones.
Oh, like we could have looked at that. Yeah, we're looking here, but we're not looking at the big chart there. Oh, yeah, if you can't look at anything, then you're screwed. Unless you've also not only are you as hardcore as this guy, but you've also memorized the numbers. Uh all right. So, emo equals no is going to be 2 3 4 5 8 10 Transforms equals no is 1 2 6 7 9 10 Sparkly equals no is 3 9 10 Romantic interests less than two is everything except 3 and 10. 1 2 4 5 6 7 8 9 1 2 5 6 7 8 9 and then finally, everything but 8 9 and 10. So, 1 2 3 4 5 6 7. All right. So, we've started off. We know what everything gets wrong.
I then make a bold claim because there is there N of these, which is 14. I make the claim that there only six that in your wildest dreams you would ever possibly even consider using ever. And the rest you would never ever use. Uh question? Yeah, I just have a question about the number of romantic interests. Yes. You negated it without an equal sign on either side. That's true. Uh that works only if there are none that are two or four. But it should have been negated with the less than or equal. I'm copying off of the quiz. But yes. I noticed that this morning when I was go putting myself through my paces.
I'm like, "There's not a less than or equal to. Wait a minute." And then, "Oh, wait. It doesn't have any twos or fours." Actually, I don't remember writing them all as fives and threes. It's possible that somebody else in the post-editing process changed them all to be about the same number and then changed the less than or equal twos to be less confusing. It's possible I had Circe at four and there was an equal to somewhere and they were like, "Forget it. Cuz I can't think of the fifth romantic interest for her." So, yes. Normally, you would have to negate it with the equal to sign, but there happen to not be any things that are equal to four or two here.
So, they get away with it this time. But good practice is good practice. So, I'm claiming that in our wildest dreams we'd only ever want to use six of these ever. And the other eight, forget it. So, let's see. I will call on people at random to the first people obviously getting it really easy to tell me which of these you think that you might ever want to use. Give me one you might ever want to use. Why E? Of course, the best one. Yes, that's one that you might ever want to use. I'll circle the ones that you might ever want to use. E. It only gets 3 and 10 wrong. That's amazing. It's like the best classifier of all.
Okay. So, give me another one that you might ever want to use. F. Let's see. F. F is great. It only gets three wrong. Do people agree that you would ever want to use F? Everyone's saying no. Why not? It's like E except worse. It's guaranteed at every step, no matter what the weights are, to have a worse accuracy than E. It is definitely good. If E wasn't around, it would be one of our best classifiers of all. But, actually F is not one of the six. This is why I wrote had them write on the test that there are six cuz people might not have found all six. Because people who did figure out not to include F might not have figured out to include some of the ones you want to include.
So, I don't understand why you can't use Why you can't use F? Okay. So, we start off with 1/10 weight for all of our data points. But, let's say during our time of boosting that all 10 of the data points have now different weights. So, we'll call whatever the weight of three is, which you're going to get wrong. You want to minimize the error, right? So, the weight of three, which goes into the error of E, is X. The weight of 10 can be Y. So, if you get If you're thinking of choosing three, you know your Oh, sorry. If you're thinking of choosing three, your error is X + Y. If you're thinking of choosing F, your error is X + Y + Z, where Z is the error of four.
And since you're never going to have a negative weight, X + Y + Z is always greater than X + Y. That's a Yes, you would You would probably choose something without the three and the 10 that didn't get the three and the 10 wrong. But, you would certainly never choose F, ever. Because it's always worse than E. In fact, this is exactly the process that it will allow you to find the correct six. And by will, I mean can. And by can, I mean let's see if you guys get it. Give me another one of the six that you might keep. K. K is the claim. Sparkly. Okay, K I'm going to say will lose for the same reason as F.
It's three, nine, and 10. Um it's essentially similar to three, four, and 10. So, oh by the way, we should not be only going for the ones with the fewest wrong with the fewest incorrect. Where you need to be going for ones that do not have something that is strictly better. In this case, three and 10 wrong is strictly better than getting three, nine, and 10 wrong. Question. You were going to say transform. You were going to be correct. Transform is one of the ones we need, C. Three, four, five, and eight, there's nothing down here that gets fewer than those wrong. Like there's nothing that gets us three, four, five wrong, for instance.
There's no Yeah, there's no way to get three, four wrong without without getting either 10 wrong or five and eight wrong. What? Why not G? Why not? Why not G? Let's include G, too. We need six. No, I just said give me any, and someone gave me the easiest one, E. Question. Why not B? B looks great. I love B. Let's include B. Does someone else want to give another one that they want to include? A. Why not A? Sure. I mean, it's hard to see down here cuz there might be something better on the bottom, but yeah, there's not. So, let's include A. Why not A? I love A. A is great. Okay. So, that is now That is now five.
There's one more that we need. Is by far the hardest one to find. Find me one more that there's nothing better than it. There's nothing that has a strict subset of the same ones wrong. What? Okay, why would you choose A if you've already chosen C? Let's say eight was a real problem for you. And you were just getting Let's say that three, four, and five, they weren't that bad. They weren't that bad. They weren't that bad. Okay, you got them wrong here with transform, you chose C. But, sometime later, eight was just by far your issue. All right. Three, four, and five, and eight Three, four, and five are much smaller weights than eight.
And then after you got them You got three, four, five, and eight wrong. Three, four, and five were still not that bad, and eight you still was a high number. And then sometime later down the line, you're looking at things, and you're saying, "You know what? I really don't want to get eight wrong again, but I I don't mind if I get three, four, and five wrong. Maybe I'll get it wrong with two, which I've never gotten wrong yet." Actually, none of the ones we've circled here get two wrong. So, it's probably not that bad to get two wrong. So, that's why. Because it also It doesn't get eight wrong.
If A was two, three, four, five, eight, you would never take it. You see what I mean? Oh, did someone raise their hand? Did someone find it? But, it's strictly worse. But, you should always choose E because three, 10 is Getting only three, 10 wrong is better than getting three, nine, and 10 wrong any universe you pick. You see what I mean? It might not be that much worse. It might be only a little bit worse to choose K, but it's always worse. So, question? Okay. Okay, that's a pretty good insight. What are you thinking about? You're trying to justify D. D is huge. It gets more than half of them wrong.
But, you know what? It gets three right. You know what? It gets 10 right. And unlike our other things that get three and 10 right, which is which is B, also gets nine right. D is the last classifier. You got it. It is hard to choose one that has seven of them wrong. But, D is the last one you might pick. There's It turns out there's nothing better than this for classifying correctly those annoying data points of Edward Cullen and Squall. And also C C, who's not that annoying, but she tends to be a problem when romance is concerned. So, um these are the six that we might use. We can now ignore the rest of them forever.
Or at least until someone reads this problem or something like that. But, we can ignore everything except A, B, C, D, E, G. In fact, why did I even bring that up? All the ones we want are on the front. I'll bring it back down. Then I'll cross this off with reckless abandon. That even broke off a piece of my chalk. Now, these are the ones we're actually thinking about using. There is a chart over here already prepared to do some boosting with these six classifiers. All right. So, let's give it a try. Now, remember, in boosting, we always try to choose whichever classifier has the least error. Is there a question?
Yes. Okay. You want to take every classifier You You generally want to take classifiers. So, I'll tell you what lets you cross off a classifier. That may be a better way to do it. You can cross off a classifier as useless if um if And by the way, this is only useful if you can do it faster than just wasting your time looking at all of them. Because if you can't cross cross off some of them as useless, usually on the test they won't make you. You can just waste your time and have 14 instead of six possibilities every step of the boosting. But, take a look at this one, one, two, three, four, five, six, seven.
Then see, do you have anything here that has a strict subset of these right wrong? Oh, look. Two, three, four, five is a strict subset. This can be crossed off. One, two, five, six, seven, eight, nine. Anything that's a strict subset? Yes, one, six, seven, nine. So, it can be crossed off. One, two, four, five, six, seven, eight, nine. Um let's see. One, six, seven, nine is a strict subset. Three, nine, 10. Three, 10 is a strict subset. One, six, seven, nine strict subset. Three uh four, five, eight is a strict subset as is two, three, four, five. One, six, seven, nine is a strict subset. And up here, three, four, 10.
Three, 10 is a strict subset, but the others don't have one. Even one, two, four, five, six, seven, eight. So, you look You In general, you want to keep them. You want to keep every classifier you might use. The only ones you'll never use are ones that there's something else that's just better always by having a strict subset of them wrong. Hopefully, that was more clear. It's a It's tricky. Some I'll Very few people realized were brave enough to take sparkly even when it got seven things wrong. So, let's start out some boosting. This wasn't boosting. This was setting yourself up. That It was setting yourself up with the knowledge of how boosting works.
Less knowledge, less search. Now, we only have to search six things. I I mean, more knowledge, less search, not less knowledge, less search. So, we start off with all weights being equal. And since there's 10 data points, all 10 of the data points are weighted 1/10. Okay. We're now weighting all of them equally. Since we're weighting all of them equally, when we want to find the classifier that gets the least error, we just want to find the one that gets the fewest points wrong. Which one is that? That's our friend E, the first one that people realized it a good one. So, we're going to choose classifier E.
What's our error? It's just the sum of the ones we get wrong. So, what's our error this time? It's 1/5. We got points 3 and 10 wrong. They both have a weight of 1/10. 1/10 + 1/10 is 1/5. So, I'll put 1/5 and alpha. Alpha is sort of a vote that will be used at the very end to sort of aggregate our classifier. Alpha is 1/2 log of 1 minus the error over the error. However, I have a little trick for you. It's not that impressive of a trick, but it's a little fun. So, since error is 1/2 Oh, sorry, not error, alpha. Alpha is 1/2 natural log of 1 minus error over error. If error is in the form if error is 1 over X, then alpha is 1/2 natural log of X minus 1.
That just follows from the math. It's a little shortcut. If error is in the form of 1 over X, then it's just 1/2 natural log of X minus 1, which means since this is in the form of 1/5, our alpha is 1/2 ln4. Okay, 1/2 ln4. So, now we come to the part in boosting that many people consider to be the hardest part, and I'm going to show you how to do it more easily. This is that part where we try to make all the ones we got right, we change their weights to be 1/2, and all the ones we got wrong, we change their weights to be 1 and 1/2. Here is my automated process. It's called the numerator stays the same method.
Here's how it works. Here's our 10 data points. Their current weight is 1/10. All of them. We're about to re-weight them for the next step. So, you agree they're all 1/10. They're equal to start off. So, step one, erase the denominators. Screw fractions. I don't like them. There's division, multiplication. It's a pain. I just want to add whole numbers. That's what we're going to do. So, which ones did we get wrong? 3 and 10. Circle those. All right. Add the numbers in the circles and multiply by two. What does that give you? Four. That's the new denominator. You always multiply by two. Add the numbers not in the circles.
Multiply by two. What does that give you? 16. That's the new denominator. The final crucial step, so that we can do this again next round, is by far the most mathematically complicated thing here because we have to actually do something with fractions, but it's not too bad, is we then change everything to be with the same denominator. So, the 1/4 become 4/16. All right. I can also uncircle these for next. Ah, I hit the the that button. All right. All right. New weights. 1/16, 1/16, 4/16, 1/16, 1/16, 1/16, 1/16, 1/16, 4/16. Note, the weights add up to one. The ones you got wrong add up to 1/2. The ones you got right add up to 1/2.
You're happy. So, now that you get 4/16 of an error for getting three wrong, 4/16 of an error for getting 10 wrong, take a look at these six. I'm not going to call on someone, just whoever's good at not whoever's good at math and can add these up more quickly, just 3 and 10 count as four. All the others count as one. Add them up. Tell me which one's the lowest. What did you say? You go with B. It doesn't get three wrong. That sounds pretty good to me. Does everyone else like B as well? I like it. I mean, of our ones that don't get three wrong or 10 wrong, we're only looking at B and D, and D just has D has seven.
B has four. So, B is the best. B gets 4/16 wrong. Does everyone see that? Cuz even getting one of three or 10 wrong is as bad as all the ones that B gets wrong cuz of the new weights. So, cool. Let's choose B. That's right. And I sort of gave it away. What's the error that B has? It has four of the four wrong, each of which are worth 1/16. The error is What? 4/16 or 1/4. Whichever's your favorite. Which means that the alpha is 1/2 ln3. Bingo. Final round. Okay, we What did we get wrong? We got 1 6 7 and 9 wrong. Oh, yeah, we can erase the denominators. All right. What are the numbers in the circles summed up?
Multiplied by two? That's eight. 1/8. And what about the numbers not in the circle summed up? Multiplied by two? That's right, 24. Which means I'm going to have to change all the numbers in the circle to 3/24, except I guess I don't because this is the last round. But if I was going to do another round, let's prepare in case we were. Change all of these to 3/24. Besides, it makes it easier to calculate which one is the best classifier cuz you can just use the numerators and add and sort of add them up. So, while I'm writing that up, you guys figure out which one you like for classifier and call it out to me when I'm done.
3/24, 1/24, 4/24, 1/24, 1/24, 3/24, 3/24, 1/24, 3/24. Wait, I'm off by one here. 3 1 4. Aha! You're right. W1 is not assigned to any to anything, so W W2 is really W1. Um yeah. Yes, that's true. Okay, well, you get it. H. So, what is the best H? You got it cuz it's right here. See? The process is so foolproof, even a fool like me can get it right while they have the chart wrong. All right. So, what's the best classifier? You say C? I say that seems pretty reasonable. Only gets three, four, five, and eight wrong. Does anyone else get a different answer? Someone else gets A. I like A. Who said A?
A lot of people said A. Well, let's figure it out. So, A A gets one five, six, seven. C gets four, five, six, seven. They're in fact equal. Tie break goes to the lower letter because that's what we said. In fact, I didn't tell you, but that's what we said. Question? So, when we're deciding which classifier to use, Mhm. do we only look at the weights, or do we also have to look at the ones that we screwed up in previous rounds? Ignore all the previous rounds. The question is, do you only look at the current weights when determining classifier, or do you look at the previous rounds as well? You got to ignore the previous rounds.
Trust me, they will will be used later in the vote, but it's sort of like tainting the jury a little bit to use the previous rounds um when you're doing the current round because you want to start fresh with these new weights, get a new classifier, and then later everyone will get to make their vote. So, you only do it based on the current weights. You don't take into consideration the last round. Six was wrong. Nope. Although, the weights are going to increase when it's wrong. It's going to increase. Question? Could you Could you theoretically reuse a classifier? Question is, could you theoretically reuse a classifier?
Answer, you absolutely can. When that happens, it essentially gets extra weight cuz you used it again. But, you can never ever use it twice in a row. Here's why. Let's say that we wanted to use um which was the one we used last over there, B. Let's say we want to use B again. What does it give us? 50/50. If we want to use B and then B, 3, 6, 9, 12 wrong. Always guaranteed to give you 50/50 which is the only is the only way that you can be sure you'll never use it. In fact, that's by design. You could reuse it, but never not twice in a row. It could be used later on down the stream and it will be used because if you do seven rounds, one of them has to be reused.
It just gives more weight to whichever one is reused. But yes, A wins against um C. C was a perfectly good answer as well. Question. Wait, if you reuse one of these three What? It's either A, R, or C. Okay. If you could reuse, why didn't you pick E? E gets eight out of 24 wrong. It's one worse than A and C. That's the only reason. Next step we'll probably use A. Uh sorry, the next step we'll probably use E. Frankly. Although maybe not because we picked uh we still we got three wrong on A. But but pretty soon we would use E again because E is pretty awesome. Anyway, here we use A. And we said we got 7/24's wrong.
Oh man, we can't use my little shortcut. So the answer it has to be 17/7's or 1/2 natural log of 17/7's. Okay. So there we go. Now we have to ask what is the final classifier of that we created from all these things? All we do is we sum up all the classifiers we chose and we multiply them times their weight alpha. So 1/2 ln4 times E whether or not E returns true. Right? Plus 1/2 ln3 times B plus 1/2 ln 17/7's times A. Is our final classifier. Where E returns a plus one if it's a vampire if if E thinks it's a vampire and a minus one if E thinks it's not. Same for B and A. All right? And then we take the sign of this and I don't mean sine and cosine.
I mean just is it positive or negative? Okay? So the question is now on the uh on the exam is how many of the 10 data points we get right if we use this? Let's give it a look-see. E is So we have romantic interest greater than two. We have emo, yes, and we have evil, yes. So oh my gosh. Logarithms, they're sometimes annoying. Do we have to actually add them up? I claim we don't. Here's a nice special case of having three logarithms on the board. One of two things is true. Either one of those three logarithms is so large that it's bigger than the other two combined. In which case if that one returns a positive or negative it's just positive or negative because that one's so big.
Or it's not one is not that large and in which case any two can dominate the other one and so it just is equivalent to a majority vote. So I I can we never have to add them when there's only three. You guys see what I mean? Like let's say one of them was 1/2 log of a billion and the others were 1/2 log of three and 1/2 log of four. Obviously whatever the 1/2 log of a billion says, which is multiplied by 1/2 log of a billion, is it's just going to be that and the others will be ignored. However, if it's not the case that one of them is larger than the other two combined, then it's a simple vote between the three because any two can outvote the other one if they work together.
And in this case let's see. 17/7's is not quite three. However, log of four is certainly not better than log of three plus log of 17/7's. It's not even log of four is equal to log of two plus log of two and these are both bigger than log of two. Um that's rules of logs. Log of four equals log of two two squared and you can take the two out. So these are not big enough that one of them is bigger than the other two combined. So it's just going to be a simple vote. So let's go through. Dracula. Okay. He's got tons of his little vampirettes. He's not emo, so that gets it E gets it right. He's not emo, so that gets it wrong, but he is evil.
That gets it right. Two out of three vote that he's a vampire. Correct. Next. Angel. Okay. Well, he's in a long-running series. He's got plenty of romantic interests. So that gets it right. Um he is certainly emo. That gets it right. And even though he's not evil, two out of three says he's a vampire. So correct. Next. Edward Cullen. Well Twilight, here we come. Uh let's see. He only has one romantic interest, so that gets it wrong. Okay. He's emo, so that gets it right. But he's not evil, so two wrong. So Edward's not a vampire according to our final classifier, but he is. So we got one of the data points wrong.
You guys see that? Because two out of three of our classifiers here said that he was not a vampire. All right, let's see. Saya. Well, she has more than two romantic interests and she's emo. So even though she's not evil, we got it right. Okay. Let's see. Lestat. He is also has many love interests, is emo, and is not evil. So you got it right. Okay. Bianca is evil with many love interests. Uh even though she's not emo, two out of three you got it right. All right. Um Carmilla Mercalla Karnstein is basically exactly the same as Bianca with the number of romantic interests fixed the way it is. So she will always do the same thing that Bianca does.
It's why six and seven always travel together. So we get it right. Sailor Moon is supposed to be not a vampire. So her number of love interests say that she's not a vampire because she only has one. The fact that she's not evil and not emo says that actually she's perfectly not a vampire. They all agree and that's correct. Squall has only one love interest, Rinoa, and he is not evil. Both of which say he's not a vampire, but he is emo, but two out of three say he's not a vampire. We get it correct. And C C, despite her many romantic interests which says she's might be a vampire, is neither evil nor emo and is not a vampire.
So we got everything right except Edward Cullen, which perhaps says more about Stephanie Meyer's writing than about our boosting. All right, final question. Wesley Windham Price, a fellow consultant, has noticed a few correlations between some of the classifiers you used. He suggests using a new set of weak classifiers consisting of a pair of your original classifiers such as the um um or a pair of your classifiers that are logically ended and or together. For instance, two of the new classifiers would be emo equals yes or evil equals yes. Or sparkly equals no and transforms equals yes. So you know, that would cut out Sailor Moon from the transforms cloud.
He believes that you'll be able to classify large vampire data sets, larger than this one anyway, more quickly with fewer rounds of boosting using his system. Do you agree or disagree with Wesley? Explain your arguments. So this was you know, the tough concept question. Does anyone have have a just an instinctual thing other than like oh man, it's Wesley, he must be wrong. You'll you'll probably use fewer rounds of boosting because you have more classifiers, but you'll have to search through more classifiers. Aha! That is the rare full point answer. Very few people um realized that Wesley was partially right.
They either said something about him being completely wrong, which was wrong, or said he was completely right. Yes, it will use fewer rounds of boosting because of the fact that you can um essentially one of the things boosting already does is sort of gets things to vote together in an and-so-like fashion. So it may it'll use approximately half the number of rounds of boosting by being able to combine into two, but there's a lot of ands and ors. There's in fact N choose two where N is the number of vampires. And since using half the number of rounds, but taking N choose two time for each round, is not less time.
So that's exactly correct. Not that many people got full credit on on that one because sometimes they were seduced by Wesley's idea or they just feel like it's Wesley, he's wrong or some or just some other funny answer. Um Any questions about boosting? Question. How do you know how many rounds of boosting to do? How do you know The question is how do you know how many rounds of boosting to do? The answer is so on the quiz it tells you that you have three. In real life you kind of you might want to just kind of keep it running until it converges. That's one possibility. Keep it running until it converges to an answer and it doesn't do anything anymore.
Patrick has a little widget on the 6034 website I think that lets you plunk down some data points and run boosting on them. And you can see that eventually it converges um the boosting converges to an answer and it doesn't change. The um basically the class not the class files you picked of course or the weights. But what converges is which ones of the of your data set you get correct. Because he does his in in two-dimensional space rather than like this and he shows you the lines the boosting is drawing between classification and colors things in green and red or something like that. And eventually it converges where the lines are and which ones it's getting right.
It generally converges to getting everything correct. Um and when that happens, then you can stop. But that's a good question, and it's not always that easy in the real world. You have to sometimes just say, "This is enough for me. I've given it a number of rounds, and that's much more than the number of classifiers, so maybe it won't get anything better."