I want to talk to you today a little bit about predictable irrationality. And my interest in irrational behavior started many years ago in the hospital. I was burned very badly. And if you spend a lot of time in hospital, you'll see a lot of types of irrationalities. And the one that particularly bothered me in the burn department was the process by which the nurses took the bandage off me. Now, you must have all taken a Band-Aid off at some point, and you must have wondered what's the right approach. Do you rip it off quickly -- short duration but high intensity -- or do you take your Band-Aid off slowly -- you take a long time, but each second is not as painful -- which one of those is the right approach? The nurses in my department thought that the right approach was the ripping one, so they would grab hold and they would rip, and they would grab hold and they would rip. And because I had 70 percent of my body burned, it would take about an hour. And as you can imagine, I hated that moment of ripping with incredible intensity. And I would try to reason with them and say, "Why don't we try something else? Why don't we take it a little longer -- maybe two hours instead of an hour -- and have less of this intensity?" And the nurses told me two things. They told me that they had the right model of the patient -- that they knew what was the right thing to do to minimize my pain -- and they also told me that the word patient doesn't mean to make suggestions or to interfere or ... This is not just in Hebrew, by the way. It's in every language I've had experience with so far. And, you know, there's not much -- there wasn't much I could do, and they kept on doing what they were doing. And about three years later, when I left the hospital, I started studying at the university. And one of the most interesting lessons I learned was that there is an experimental method that if you have a question you can create a replica of this question in some abstract way, and you can try to examine this question, maybe learn something about the world. So that's what I did. I was still interested in this question of how do you take bandages off burn patients. So originally I didn't have much money, so I went to a hardware store and I bought a carpenter's vice. And I would bring people to the lab and I would put their finger in it, and I would crunch it a little bit. (Laughter) And I would crunch it for long periods and short periods, and pain that went up and pain that went down, and with breaks and without breaks -- all kinds of versions of pain. And when I finished hurting people a little bit, I would ask them, so, how painful was this? Or, how painful was this? Or, if you had to choose between the last two, which one would you choose? (Laughter) I kept on doing this for a while. (Laughter) And then, like all good academic projects, I got more funding. I moved to sounds, electrical shocks -- I even had a pain suit that I could get people to feel much more pain. But at the end of this process, what I learned was that the nurses were wrong. Here were wonderful people with good intentions and plenty of experience, and nevertheless they were getting things wrong predictably all the time. It turns out that because we don't encode duration in the way that we encode intensity, I would have had less pain if the duration would have been longer and the intensity was lower. It turns out it would have been better to start with my face, which was much more painful, and move toward my legs, giving me a trend of improvement over time -- that would have been also less painful. And it also turns out that it would have been good to give me breaks in the middle to kind of recuperate from the pain. All of these would have been great things to do, and my nurses had no idea. And from that point on I started thinking, are the nurses the only people in the world who get things wrong in this particular decision, or is it a more general case? And it turns out it's a more general case -- there's a lot of mistakes we do. And I want to give you one example of one of these irrationalities, and I want to talk to you about cheating. And the reason I picked cheating is because it's interesting, but also it tells us something, I think, about the stock market situation we're in. So, my interest in cheating started when Enron came on the scene, exploded all of a sudden, and I started thinking about what is happening here. Is it the case that there was kind of a few apples who are capable of doing these things, or are we talking a more endemic situation, that many people are actually capable of behaving this way? So, like we usually do, I decided to do a simple experiment. And here's how it went. If you were in the experiment, I would pass you a sheet of paper with 20 simple math problems that everybody could solve, but I wouldn't give you enough time. When the five minutes were over, I would say, "Pass me the sheets of paper, and I'll pay you a dollar per question." People did this. I would pay people four dollars for their task -- on average people would solve four problems. Other people I would tempt to cheat. I would pass their sheet of paper. When the five minutes were over, I would say, "Please shred the piece of paper. Put the little pieces in your pocket or in your backpack, and tell me how many questions you got correctly." People now solved seven questions on average. Now, it wasn't as if there was a few bad apples -- a few people cheated a lot. Instead, what we saw is a lot of people who cheat a little bit. Now, in economic theory, cheating is a very simple cost-benefit analysis. You say, what's the probability of being caught? How much do I stand to gain from cheating? And how much punishment would I get if I get caught? And you weigh these options out -- you do the simple cost-benefit analysis, and you decide whether it's worthwhile to commit the crime or not. So, we try to test this. For some people, we varied how much money they could get away with -- how much money they could steal. We paid them 10 cents per correct question, 50 cents, a dollar, five dollars, 10 dollars per correct question. You would expect that as the amount of money on the table increases, people would cheat more, but in fact it wasn't the case. We got a lot of people cheating by stealing by a little bit. What about the probability of being caught? Some people shredded half the sheet of paper, so there was some evidence left. Some people shredded the whole sheet of paper. Some people shredded everything, went out of the room, and paid themselves from the bowl of money that had over 100 dollars. You would expect that as the probability of being caught goes down, people would cheat more, but again, this was not the case. Again, a lot of people cheated by just by a little bit, and they were insensitive to these economic incentives. So we said, "If people are not sensitive to the economic rational theory explanations, to these forces, what could be going on?" And we thought maybe what is happening is that there are two forces. At one hand, we all want to look at ourselves in the mirror and feel good about ourselves, so we don't want to cheat. On the other hand, we can cheat a little bit, and still feel good about ourselves. So, maybe what is happening is that there's a level of cheating we can't go over, but we can still benefit from cheating at a low degree, as long as it doesn't change our impressions about ourselves. We call this like a personal fudge factor. Now, how would you test a personal fudge factor? Initially we said, what can we do to shrink the fudge factor? So, we got people to the lab, and we said, "We have two tasks for you today." First, we asked half the people to recall either 10 books they read in high school, or to recall The Ten Commandments, and then we tempted them with cheating. Turns out the people who tried to recall The Ten Commandments -- and in our sample nobody could recall all of The Ten Commandments -- but those people who tried to recall The Ten Commandments, given the opportunity to cheat, did not scott66@example.net. It wasn't that the more religious people -- the people who remembered more of the Commandments -- cheated less, and the less religious people -- the people who couldn't remember almost any Commandments -- cheated more. The moment people thought about trying to recall The Ten Commandments, they stopped cheating. In fact, even when we gave self-declared atheists the task of swearing on the Bible and we give them a chance to cheat, they don't scott66@example.net. Now, Ten Commandments is something that is hard to bring into the education system, so we said, "Why don't we get people to sign the honor code?" So, we got people to sign, "I understand that this short survey falls under the MIT Honor Code." Then they shredded it. No cheating whatsoever. And this is particularly interesting, because MIT doesn't have an honor code. (Laughter) So, all this was about decreasing the fudge factor. What about increasing the fudge factor? The first experiment -- I walked around MIT and I distributed six-packs of Cokes in the refrigerators -- these were common refrigerators for the undergrads. And I came back to measure what we technically call the half-lifetime of Coke -- how long does it last in the refrigerators? As you can expect it doesn't last very long; people take it. In contrast, I took a plate with six one-dollar bills, and I left those plates in the same refrigerators. No bill ever disappeared. Now, this is not a good social science experiment, so to do it better I did the same experiment as I described to you before. A third of the people we passed the sheet, they gave it back to us. A third of the people we passed it to, they shredded it, they came to us and said, "Mr. Experimenter, I solved X problems. Give me X dollars." A third of the people, when they finished shredding the piece of paper, they came to us and said, "Mr Experimenter, I solved X problems. Give me X tokens." We did not pay them with dollars; we paid them with something else. And then they took the something else, they walked 12 feet to the side, and exchanged it for dollars. Think about the following intuition. How bad would you feel about taking a pencil from work home, compared to how bad would you feel about taking 10 cents from a petty cash box? These things feel very differently. Would being a step removed from cash for a few seconds by being paid by token make a difference? Our subjects doubled their cheating. I'll tell you what I think about this and the stock market in a minute. But this did not solve the big problem I had with Enron yet, because in Enron, there's also a social element. People see each other behaving. In fact, every day when we open the news we see examples of people cheating. What does this cause us? So, we did another experiment. We got a big group of students to be in the experiment, and we prepaid them. So everybody got an envelope with all the money for the experiment, and we told them that at the end, we asked them to pay us back the money they didn't make. OK? The same thing happens. When we give people the opportunity to cheat, they cheat. They cheat just by a little bit, all the same. But in this experiment we also hired an acting student. This acting student stood up after 30 seconds, and said, "I solved everything. What do I do now?" And the experimenter said, "If you've finished everything, go home. That's it. The task is finished." So, now we had a student -- an acting student -- that was a part of the group. Nobody knew it was an actor. And they clearly cheated in a very, very serious way. What would happen to the other people in the group? Will they cheat more, or will they cheat less? Here is what happens. It turns out it depends on what kind of sweatshirt they're wearing. Here is the thing. We ran this at Carnegie Mellon and Pittsburgh. And at Pittsburgh there are two big universities, Carnegie Mellon and University of Pittsburgh. All of the subjects sitting in the experiment were Carnegie Mellon students. When the actor who was getting up was a Carnegie Mellon student -- he was actually a Carnegie Mellon student -- but he was a part of their group, cheating went up. But when he actually had a University of Pittsburgh sweatshirt, cheating went down. (Laughter) Now, this is important, because remember, when the moment the student stood up, it made it clear to everybody that they could get away with cheating, because the experimenter said, "You've finished everything. Go home," and they went with the money. So it wasn't so much about the probability of being caught again. It was about the norms for cheating. If somebody from our in-group cheats and we see them cheating, we feel it's more appropriate, as a group, to behave this way. But if it's somebody from another group, these terrible people -- I mean, not terrible in this -- but somebody we don't want to associate ourselves with, from another university, another group, all of a sudden people's awareness of honesty goes up -- a little bit like The Ten Commandments experiment -- and people cheat even less. So, what have we learned from this about cheating? We've learned that a lot of people can cheat. They cheat just by a little bit. When we remind people about their morality, they cheat less. When we get bigger distance from cheating, from the object of money, for example, people cheat more. And when we see cheating around us, particularly if it's a part of our in-group, cheating goes up. Now, if we think about this in terms of the stock market, think about what happens. What happens in a situation when you create something where you pay people a lot of money to see reality in a slightly distorted way? Would they not be able to see it this way? Of course they would. What happens when you do other things, like you remove things from money? You call them stock, or stock options, derivatives, mortgage-backed securities. Could it be that with those more distant things, it's not a token for one second, it's something that is many steps removed from money for a much longer time -- could it be that people will cheat even more? And what happens to the social environment when people see other people behave around them? I think all of those forces worked in a very bad way in the stock market. More generally, I want to tell you something about behavioral economics. We have many intuitions in our life, and the point is that many of these intuitions are wrong. The question is, are we going to test those intuitions? We can think about how we're going to test this intuition in our private life, in our business life, and most particularly when it goes to policy, when we think about things like No Child Left Behind, when you create new stock markets, when you create other policies -- taxation, health care and so on. And the difficulty of testing our intuition was the big lesson I learned when I went back to the nurses to talk to them. So I went back to talk to them and tell them what I found out about removing bandages. And I learned two interesting things. One was that my favorite nurse, Ettie, told me that I did not take her pain into consideration. She said, "Of course, you know, it was very painful for you. But think about me as a nurse, taking, removing the bandages of somebody I liked, and had to do it repeatedly over a long period of time. Creating so much torture was not something that was good for me, too." And she said maybe part of the reason was it was difficult for her. But it was actually more interesting than that, because she said, "I did not think that your intuition was right. I felt my intuition was correct." So, if you think about all of your intuitions, it's very hard to believe that your intuition is wrong. And she said, "Given the fact that I thought my intuition was right ..." -- she thought her intuition was right -- it was very difficult for her to accept doing a difficult experiment to try and check whether she was wrong. But in fact, this is the situation we're all in all the time. We have very strong intuitions about all kinds of things -- our own ability, how the economy works, how we should pay school teachers. But unless we start testing those intuitions, we're not going to do better. And just think about how better my life would have been if these nurses would have been willing to check their intuition, and how everything would have been better if we just start doing more systematic experimentation of our intuitions. Thank you very much. What's the scariest thing you've ever done? Or another way to say it is, what's the most dangerous thing that you've ever done? And why did you do it? I know what the most dangerous thing is that I've ever done because NASA does the math. You look back to the first five shuttle launches, the odds of a catastrophic event during the first five shuttle launches was one in nine. And even when I first flew in the shuttle back in 1995, 74 shuttle flight, the odds were still now that we look back about one in 38 or so -- one in 35, one in 40. Not great odds, so it's a really interesting day when you wake up at the Kennedy Space Center and you're going to go to space that day because you realize by the end of the day you're either going to be floating effortlessly, gloriously in space, or you'll be dead. You go into, at the Kennedy Space Center, the suit-up room, the same room that our childhood heroes got dressed in, that Neil Armstrong and Buzz Aldrin got suited in to go ride the Apollo rocket to the moon. And I got my pressure suit built around me and rode down outside in the van heading out to the launchpad -- in the Astro van -- heading out to the launchpad, and as you come around the corner at the Kennedy Space Center, it's normally predawn, and in the distance, lit up by the huge xenon lights, is your spaceship -- the vehicle that is going to take you off the planet. The crew is sitting in the Astro van sort of hushed, almost holding hands, looking at that as it gets bigger and bigger. We ride the elevator up and we crawl in, on your hands and knees into the spaceship, one at a time, and you worm your way up into your chair and plunk yourself down on your back. And the hatch is closed, and suddenly, what has been a lifetime of both dreams and denial is becoming real, something that I dreamed about, in fact, that I chose to do when I was nine years old, is now suddenly within not too many minutes of actually happening. In the astronaut business -- the shuttle is a very complicated vehicle; it's the most complicated flying machine ever built. And in the astronaut business, we have a saying, which is, there is no problem so bad that you can't make it worse. (Laughter) And so you're very conscious in the cockpit; you're thinking about all of the things that you might have to do, all the switches and all the wickets you have to go through. And as the time gets closer and closer, this excitement is building. And then about three and a half minutes before launch, the huge nozzles on the back, like the size of big church bells, swing back and forth and the mass of them is such that it sways the whole vehicle, like the vehicle is alive underneath you, like an elephant getting up off its knees or something. And then about 30 seconds before launch, the vehicle is completely alive -- it is ready to go -- the APUs are running, the computers are all self-contained, it's ready to leave the planet. And 15 seconds before launch, this happens: (Video) Voice: 12, 11, 10, nine, eight, seven, six -- (Space shuttle preparing for takeoff) -- start, two, one, booster ignition, and liftoff of the space shuttle Discovery, returning to the space station, paving the way ... (Space shuttle taking off) Chris Hadfield: It is incredibly powerful to be on board one of these things. You are in the grip of something that is vastly more powerful than yourself. It's shaking you so hard you can't focus on the instruments in front of you. It's like you're in the jaws of some enormous dog and there's a foot in the small of your back pushing you into space, accelerating wildly straight up, shouldering your way through the air, and you're in a very complex place -- paying attention, watching the vehicle go through each one of its wickets with a steadily increasing smile on your face. After two minutes, those solid rockets explode off and then you just have the liquid engines, the hydrogen and oxygen, and it's as if you're in a dragster with your foot to the floor and accelerating like you've never accelerated. You get lighter and lighter, the force gets on us heavier and heavier. It feels like someone's pouring cement on you or something. Until finally, after about eight minutes and 40 seconds or so, we are finally at exactly the right altitude, exactly the right speed, the right direction, the engine shut off, and we're weightless. And we're alive. It's an amazing experience. But why would we take that risk? Why would you do something that dangerous? In my case the answer is fairly straightforward. I was inspired as a youngster that this was what I wanted to do. I watched the first people walk on the moon and to me, it was just an obvious thing -- I want to somehow turn myself into that. But the real question is, how do you deal with the danger of it and the fear that comes from it? How do you deal with fear versus danger? And having the goal in mind, thinking about where it might lead, directed me to a life of looking at all of the small details to allow this to become possible, to be able to launch and go help build a space station where you are on board a million-pound creation that's going around the world at five miles a second, eight kilometers a second, around the world 16 times a day, with experiments on board that are teaching us what the substance of the universe is made of and running 200 experiments inside. But maybe even more importantly, allowing us to see the world in a way that is impossible through any other means, to be able to look down and have -- if your jaw could drop, it would -- the jaw-dropping gorgeousness of the turning orb like a self-propelled art gallery of fantastic, constantly changing beauty that is the world itself. And you see, because of the speed, a sunrise or a sunset every 45 minutes for half a year. And the most magnificent part of all that is to go outside on a spacewalk. You are in a one-person spaceship that is your spacesuit, and you're going through space with the world. It's an entirely different perspective, you're not looking up at the universe, you and the Earth are going through the universe together. And you're holding on with one hand, looking at the world turn beside you. It's roaring silently with color and texture as it pours by mesmerizingly next to you. And if you can tear your eyes away from that and you look under your arm down at the rest of everything, it's unfathomable blackness, with a texture you feel like you could stick your hand into. and you are holding on with one hand, one link to the other seven billion people. And I was outside on my first spacewalk when my left eye went blind, and I didn't know why. Suddenly my left eye slammed shut in great pain and I couldn't figure out why my eye wasn't working. I was thinking, what do I do next? I thought, well maybe that's why we have two eyes, so I kept working. But unfortunately, without gravity, tears don't fall. So you just get a bigger and bigger ball of whatever that is mixed with your tears on your eye until eventually, the ball becomes so big that the surface tension takes it across the bridge of your nose like a tiny little waterfall and goes "goosh" into your other eye, and now I was completely blind outside the spaceship. So what's the scariest thing you've ever done? (Laughter) Maybe it's spiders. A lot of people are afraid of spiders. I think you should be afraid of spiders -- spiders are creepy and they've got long, hairy legs, and spiders like this one, the brown recluse -- it's horrible. If a brown recluse bites you, you end with one of these horrible, big necrotic things on your leg and there might be one right now sitting on the chair behind you, in fact. And how do you know? And so a spider lands on you, and you go through this great, spasmy attack because spiders are scary. But then you could say, well is there a brown recluse sitting on the chair beside me or not? I don't know. Are there brown recluses here? So if you actually do the research, you find out that in the world there are about 50,000 different types of spiders, and there are about two dozen that are venomous out of 50,000. And if you're in Canada, because of the cold winters here in B.C., there's about 720, 730 different types of spiders and there's one -- one -- that is venomous, and its venom isn't even fatal, it's just kind of like a nasty sting. And that spider -- not only that, but that spider has beautiful markings on it, it's like "I'm dangerous. I got a big radiation symbol on my back, it's the black widow." So, if you're even slightly careful you can avoid running into the one spider -- and it lives close the ground, you're walking along, you are never going to go through a spider web where a black widow bites you. Spider webs like this, it doesn't build those, it builds them down in the corners. And its a black widow because the female spider eats the male; it doesn't care about you. So in fact, the next time you walk into a spiderweb, you don't need to panic and go with your caveman reaction. The danger is entirely different than the fear. How do you get around it, though? How do you change your behavior? Well, next time you see a spiderweb, have a good look, make sure it's not a black widow spider, and then walk into it. And then you see another spiderweb and walk into that one. It's just a little bit of fluffy stuff. It's not a big deal. And the spider that may come out is no more threat to you than a lady bug or a butterfly. And then I guarantee you if you walk through 100 spiderwebs you will have changed your fundamental human behavior, your caveman reaction, and you will now be able to walk in the park in the morning and not worry about that spiderweb -- or into your grandma's attic or whatever, into your own basement. And you can apply this to anything. If you're outside on a spacewalk and you're blinded, your natural reaction would be to panic, I think. It would make you nervous and worried. But we had considered all the venom, and we had practiced with a whole variety of different spiderwebs. We knew everything there is to know about the spacesuit and we trained underwater thousands of times. And we don't just practice things going right, we practice things going wrong all the time, so that you are constantly walking through those spiderwebs. And not just underwater, but also in virtual reality labs with the helmet and the gloves so you feel like it's realistic. So when you finally actually get outside on a spacewalk, it feels much different than it would if you just went out first time. And even if you're blinded, your natural, panicky reaction doesn't happen. Instead you kind of look around and go, "Okay, I can't see, but I can hear, I can talk, Scott Parazynski is out here with me. He could come over and help me." We actually practiced incapacitated crew rescue, so he could float me like a blimp and stuff me into the airlock if he had to. I could find my own way back. It's not nearly as big a deal. And actually, if you keep on crying for a while, whatever that gunk was that's in your eye starts to dilute and you can start to see again, and Houston, if you negotiate with them, they will let you then keep working. We finished everything on the spacewalk and when we came back inside, Jeff got some cotton batting and took the crusty stuff around my eyes, and it turned out it was just the anti-fog, sort of a mixture of oil and soap, that got in my eye. And now we use Johnson's No More Tears, which we probably should've been using right from the very beginning. (Laughter) But the key to that is by looking at the difference between perceived danger and actual danger, where is the real risk? What is the real thing that you should be afraid of? Not just a generic fear of bad things happening. You can fundamentally change your reaction to things so that it allows you to go places and see things and do things that otherwise would be completely denied to you ... where you could see the hardpan south of the Sahara, or you can see New York City in a way that is almost dreamlike, or the unconscious gingham of Eastern Europe fields or the Great Lakes as a collection of small puddles. You can see the fault lines of San Francisco and the way the water pours out under the bridge, just entirely different than any other way that you could have if you had not found a way to conquer your fear. You see a beauty that otherwise never would have happened. It's time to come home at the end. This is our spaceship, the Soyuz, that little one. Three of us climb in, and then this spaceship detaches from the station and falls into the atmosphere. These two parts here actually melt, we jettison them and they burn up in the atmosphere. The only part that survives is the little bullet that we're riding in, and it falls into the atmosphere, and in essence you are riding a meteorite home, and riding meteorites is scary, and it ought to be. But instead of riding into the atmosphere just screaming, like you would if suddenly you found yourself riding a meteorite back to Earth -- (Laughter) -- instead, 20 years previously we had started studying Russian, and then once you learn Russian, then we learned orbital mechanics in Russian, and then we learned vehicle control theory, and then we got into the simulator and practiced over and over and over again. And in fact, you can fly this meteorite and steer it and land in about a 15-kilometer circle anywhere on the Earth. So in fact, when our crew was coming back into the atmosphere inside the Soyuz, we weren't screaming, we were laughing; it was fun. And when the great big parachute opened, we knew that if it didn't open there's a second parachute, and it runs on a nice little clockwork mechanism. So we came back, we came thundering back to Earth and this is what it looked like to land in a Soyuz, in Kazakhstan. (Video) Reporter: And you can see one of those search and recovery helicopters, once again that helicopter part of dozen such Russian Mi-8 helicopters. Touchdown -- 3:14 and 48 seconds, a.m. Central Time. CH: And you roll to a stop as if someone threw your spaceship at the ground and it tumbles end over end, but you're ready for it you're in a custom-built seat, you know how the shock absorber works. And then eventually the Russians reach in, drag you out, plunk you into a chair, and you can now look back at what was an incredible experience. You have taken the dreams of that nine-year-old boy, which were impossible and dauntingly scary, dauntingly terrifying, and put them into practice, and figured out a way to reprogram yourself, to change your primal fear so that it allowed you to come back with a set of experiences and a level of inspiration for other people that never could have been possible otherwise. Just to finish, they asked me to play that guitar. I know this song, and it's really a tribute to the genius of David Bowie himself, but it's also, I think, a reflection of the fact that we are not machines exploring the universe, we are people, and we're taking that ability to adapt and that ability to understand and the ability to take our own self-perception into a new place. (Music) ♫ This is Major Tom to ground control ♫ ♫ I've left forevermore ♫ ♫ And I'm floating in a most peculiar way ♫ ♫ And the stars look very different today ♫ ♫ For here am I floating in the tin can ♫ ♫ A last glimpse of the world ♫ ♫ Planet Earth is blue and there's so much left to do ♫ (Music) Fear not. (Applause) That's very nice of you. Thank you very much. Thank you. Chris Anderson: Help us understand what machine learning is, because that seems to be the key driver of so much of the excitement and also of the concern around artificial intelligence. How does machine learning work? Sebastian Thrun: So, artificial intelligence and machine learning is about 60 years old and has not had a great day in its past until recently. And the reason is that today, we have reached a scale of computing and datasets that was necessary to make machines smart. So here's how it works. If you program a computer today, say, your phone, then you hire software engineers that write a very, very long kitchen recipe, like, "If the water is too hot, turn down the temperature. If it's too cold, turn up the temperature." The recipes are not just 10 lines long. They are millions of lines long. A modern cell phone has 12 million lines of code. A browser has five million lines of code. And each bug in this recipe can cause your computer to crash. That's why a software engineer makes so much money. The new thing now is that computers can find their own rules. So instead of an expert deciphering, step by step, a rule for every contingency, what you do now is you give the computer examples and have it infer its own rules. A really good example is AlphaGo, which recently was won by Google. Normally, in game playing, you would really write down all the rules, but in AlphaGo's case, the system looked over a million games and was able to infer its own rules and then beat the world's residing Go champion. That is exciting, because it relieves the software engineer of the need of being super smart, and pushes the burden towards the data. As I said, the inflection point where this has become really possible -- very embarrassing, my thesis was about machine learning. It was completely insignificant, don't read it, because it was 20 years ago and back then, the computers were as big as a cockroach brain. Now they are powerful enough to really emulate kind of specialized human thinking. And then the computers take advantage of the fact that they can look at much more data than people can. So I'd say AlphaGo looked at more than a million games. No human expert can ever study a million games. Google has looked at over a hundred billion web pages. No person can ever study a hundred billion web pages. So as a result, the computer can find rules that even people can't find. CA: So instead of looking ahead to, "If he does that, I will do that," it's more saying, "Here is what looks like a winning pattern, here is what looks like a winning pattern." ST: Yeah. I mean, think about how you raise children. You don't spend the first 18 years giving kids a rule for every contingency and set them free and they have this big program. They stumble, fall, get up, they get slapped or spanked, and they have a positive experience, a good grade in school, and they figure it out on their own. That's happening with computers now, which makes computer programming so much easier all of a sudden. Now we don't have to think anymore. We just give them lots of data. CA: And so, this has been key to the spectacular improvement in power of self-driving cars. I think you gave me an example. Can you explain what's happening here? ST: This is a drive of a self-driving car that we happened to have at Udacity and recently made into a spin-off called Voyage. We have used this thing called deep learning to train a car to drive itself, and this is driving from Mountain View, California, to San Francisco on El Camino Real on a rainy day, with bicyclists and pedestrians and 133 traffic lights. And the novel thing here is, many, many moons ago, I started the Google self-driving car team. And back in the day, I hired the world's best software engineers to find the world's best rules. This is just trained. We drive this road 20 times, we put all this data into the computer brain, and after a few hours of processing, it comes up with behavior that often surpasses human agility. So it's become really easy to program it. This is 100 percent autonomous, about 33 miles, an hour and a half. CA: So, explain it -- on the big part of this program on the left, you're seeing basically what the computer sees as trucks and cars and those dots overtaking it and so forth. ST: On the right side, you see the camera image, which is the main input here, and it's used to find lanes, other cars, traffic lights. The vehicle has a radar to do distance estimation. This is very commonly used in these kind of systems. On the left side you see a laser diagram, where you see obstacles like trees and so on depicted by the laser. But almost all the interesting work is centering on the camera image now. We're really shifting over from precision sensors like radars and lasers into very cheap, commoditized sensors. A camera costs less than eight dollars. CA: And that green dot on the left thing, what is that? Is that anything meaningful? ST: This is a look-ahead point for your adaptive cruise control, so it helps us understand how to regulate velocity based on how far the cars in front of you are. CA: And so, you've also got an example, I think, of how the actual learning part takes place. Maybe we can see that. Talk about this. ST: This is an example where we posed a challenge to Udacity students to take what we call a self-driving car Nanodegree. We gave them this dataset and said "Hey, can you guys figure out how to steer this car?" And if you look at the images, it's, even for humans, quite impossible to get the steering right. And we ran a competition and said, "It's a deep learning competition, AI competition," and we gave the students 48 hours. So if you are a software house like Google or Facebook, something like this costs you at least six months of work. So we figured 48 hours is great. And within 48 hours, we got about 100 submissions from students, and the top four got it perfectly right. It drives better than I could drive on this imagery, using deep learning. And again, it's the same methodology. It's this magical thing. When you give enough data to a computer now, and give enough time to comprehend the data, it finds its own rules. CA: And so that has led to the development of powerful applications in all sorts of areas. You were talking to me the other day about cancer. Can I show this video? ST: Yeah, absolutely, please. CA: This is cool. ST: This is kind of an insight into what's happening in a completely different domain. This is augmenting, or competing -- it's in the eye of the beholder -- with people who are being paid 400,000 dollars a year, dermatologists, highly trained specialists. It takes more than a decade of training to be a good dermatologist. What you see here is the machine learning version of it. It's called a neural network. "Neural networks" is the technical term for these machine learning algorithms. They've been around since the 1980s. This one was invented in 1988 by a Facebook Fellow called Yann LeCun, and it propagates data stages through what you could think of as the human brain. It's not quite the same thing, but it emulates the same thing. It goes stage after stage. In the very first stage, it takes the visual input and extracts edges and rods and dots. And the next one becomes more complicated edges and shapes like little half-moons. And eventually, it's able to build really complicated concepts. Andrew Ng has been able to show that it's able to find cat faces and dog faces in vast amounts of images. What my student team at Stanford has shown is that if you train it on 129,000 images of skin conditions, including melanoma and carcinomas, you can do as good a job as the best human dermatologists. And to convince ourselves that this is the case, we captured an independent dataset that we presented to our network and to 25 board-certified Stanford-level dermatologists, and compared those. And in most cases, they were either on par or above the performance classification accuracy of human dermatologists. CA: You were telling me an anecdote. I think about this image right here. What happened here? ST: This was last Thursday. That's a moving piece. What we've shown before and we published in "Nature" earlier this year was this idea that we show dermatologists images and our computer program images, and count how often they're right. But all these images are past images. They've all been biopsied to make sure we had the correct classification. This one wasn't. This one was actually done at Stanford by one of our collaborators. The story goes that our collaborator, who is a world-famous dermatologist, one of the three best, apparently, looked at this mole and said, "This is not skin cancer." And then he had a second moment, where he said, "Well, let me just check with the app." So he took out his iPhone and ran our piece of software, our "pocket dermatologist," so to speak, and the iPhone said: cancer. It said melanoma. And then he was confused. And he decided, "OK, maybe I trust the iPhone a little bit more than myself," and he sent it out to the lab to get it biopsied. And it came up as an aggressive melanoma. So I think this might be the first time that we actually found, in the practice of using deep learning, an actual person whose melanoma would have gone unclassified, had it not been for deep learning. CA: I mean, that's incredible. (Applause) It feels like there'd be an instant demand for an app like this right now, that you might freak out a lot of people. Are you thinking of doing this, making an app that allows self-checking? ST: So my in-box is flooded about cancer apps, with heartbreaking stories of people. I mean, some people have had 10, 15, 20 melanomas removed, and are scared that one might be overlooked, like this one, and also, about, I don't know, flying cars and speaker inquiries these days, I guess. My take is, we need more testing. I want to be very careful. It's very easy to give a flashy result and impress a TED audience. It's much harder to put something out that's ethical. And if people were to use the app and choose not to consult the assistance of a doctor because we get it wrong, I would feel really bad about it. So we're currently doing clinical tests, and if these clinical tests commence and our data holds up, we might be able at some point to take this kind of technology and take it out of the Stanford clinic and bring it to the entire world, places where Stanford doctors never, ever set foot. CA: And do I hear this right, that it seemed like what you were saying, because you are working with this army of Udacity students, that in a way, you're applying a different form of machine learning than might take place in a company, which is you're combining machine learning with a form of crowd wisdom. Are you saying that sometimes you think that could actually outperform what a company can do, even a vast company? ST: I believe there's now instances that blow my mind, and I'm still trying to understand. What Chris is referring to is these competitions that we run. We turn them around in 48 hours, and we've been able to build a self-driving car that can drive from Mountain View to San Francisco on surface streets. It's not quite on par with Google after seven years of Google work, but it's getting there. And it took us only two engineers and three months to do this. And the reason is, we have an army of students who participate in competitions. We're not the only ones who use crowdsourcing. Uber and Didi use crowdsource for driving. Airbnb uses crowdsourcing for hotels. There's now many examples where people do bug-finding crowdsourcing or protein folding, of all things, in crowdsourcing. But we've been able to build this car in three months, so I am actually rethinking how we organize corporations. We have a staff of 9,000 people who are never hired, that I never fire. They show up to work and I don't even know. Then they submit to me maybe 9,000 answers. I'm not obliged to use any of those. I end up -- I pay only the winners, so I'm actually very cheapskate here, which is maybe not the best thing to do. But they consider it part of their education, too, which is nice. But these students have been able to produce amazing deep learning results. So yeah, the synthesis of great people and great machine learning is