# Media

url: https://www.youtube.com/watch?v=1JT_lTfK69Q
title: DEF CON 31 - Infinite Money Glitch - Hacking Transit Cards - Bertocchi, Campbell, Gibson, Harris
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=1JT_lTfK69Q",
    "title": "DEF CON 31 - Infinite Money Glitch - Hacking Transit Cards -  Bertocchi, Campbell, Gibson, Harris",
    "description": "Who likes paying to ride the subway? Sure, you could hop the fare gates, but that can be athletically challenging and simply isn\u2019t cool enough for our tastes. What\u2019s a mischievous and miserly rider to do, then? Hack the fare system of course!\n\nIn this talk we'll walk you through how we, four high school students and cybersecurity noobs became the first to fully reverse engineer Boston\u2019s CharlieCard fare system and earn ourselves free rides for life\u2026 or at least until the system gets fixed, whichever comes first.\n\nWe\u2019ll start by exploring the trials and tribulations of exploring the hardware behind the CharlieCards. Next, we\u2019ll dive into the emotional rollercoaster of reverse engineering the black box that is a transit card system older than us. We\u2019ll then explain the process of disclosing our findings to a government agency without having to hire a legal team. Finally, we\u2019ll show you a demo of some of the tools we made, including our own portable fare machine! By the end of our talk, regardless of whether you\u2019re an avid RFID hackerman, or a complete noob, we\u2019ll leave you with useful reverse engineering strategies, tips for working with a government agency, and if nothing else, a fun story.",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2704,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230823"
  }

  This text was transcribed using whisper model: large-v2

   This is like an incredible honor to be here. I had no idea there would be this many people.  This is amazing. We're incredibly honored, excited, and absolutely terrified to be here  right now. So, as I think he alluded to earlier, I'm going to start off with a quick question.  How many of you guys like taking public transportation? Okay, all right, all right. Now, how many  of those same people enjoy paying for public transportation? I see a couple hands. Okay.  Well, for those of you with your hands up, it's a good opportunity to avert your eyes if you want.  But for everybody else, I actually have a little bit of bad news to start off with, because  as our local transit agency's lawyers have instructed us to say and put on our title slide,  the MBTA does not endorse or encourage this type of conduct. We're debating fares and  legal, and the MBTA takes these matters seriously. Anyone caught engaging in this type of behavior  will be referred to the appropriate law enforcement agency. MBTA has implemented  mitigation measures to address some of these concerns. Okay. So, now we can begin.  So, are any of these, no, these aren't even up here. Let's see. The slide should be appearing,  right? I think we're having a bit of trouble. Okay. We're on it. Good. Yeah. So, I'll just  explain who we are. So, we don't actually have any real, like, qualifications or anything. We  don't work at any fancy cybersecurity companies, not yet anyway. We're vocational high school  students from Medford, Massachusetts. We call ourselves amateur miscreants. We really like to  make stuff, but we also like to break stuff, and we don't really like being told what to do,  and we're also part of an underwater robotics team at our school called Sunk Robotics. We  participate in a competition called the Mate ROV competition. A lot of fun, and it's brought the  four of us a lot closer together. So, let's see. It's helpful. Yeah. Okay. So, I have pictures  up here of us, but you'll just have to take a look at us here. So, it's me, Zach, Scott,  and Noah. So, you might be wondering what you're actually going to get out of this talk. Well,  if you live in Boston and you're paying attention, you'll get free rides on the MBTA.  You also, if you don't, you can get lessons on how to reverse engineer a transit card,  and more generally, some lessons on reverse engineering. You'll also get a lesson on how  to talk to a government agency and how to not end up in handcuffs when you do so,  and if nothing else, you'll hopefully at least get a fun story.  So, and I should mention, as I just did, that this is illegal. Don't do it, or if you do,  don't tell us about it. Okay? So, let's see. We'll figure it out. Okay. Brief introduction.  So,  one second. Let's just see.  Okay.  Okay.  This wouldn't be DEF CON without computer glitches.  So,  okay.  Is it seeing the second display?  Started detecting it. Couldn't find another display.  Okay. Okay. Give me one second. Okay.  So,  ah, okay. Well, can you guys wait? Go for it. All right.  Okay. Do your best. We'll just get you back.  Hi. Sorry about the whole thing here. Falling from the last bit. Meet the team. The  Massachusetts Bay Transportation Authority is the public transit agency in Boston.  They manage all the buses and the subways. It's also the oldest transit system in the country.  We have a picture of the TA's logo and the subway map here, but these are the slides.  Sorry. So, sounds great. How can you hop on? Option one, you have traditional fare evasion,  which is like hopping the gates or forcing them open, or even just waving your hand over the  sensor to make them open. Paying also works. Subway fare is $2.40, and bus fare is $1.70.  But there's some problems with these strategies. Traditional fare evasion, it's too easy. It's  really boring, and you also might get caught. Paying is way too legal. It's also way too boring,  and importantly, it costs money. So, what do you do? You hack the fare system. Duh.  So, how many of you enjoy reading random Wikipedia articles in your spare time?  Alright. Okay. Very good. I certainly do as well. And this whole project started by me actually  reading a Wikipedia article, a trolley card Wikipedia article, and I went down to the  criticism section, the best part of any article, and I saw a nice security concerns, and a couple  things stuck out to me. One, DEF CON, MIT students, and a federal lawsuit. Let's see if anyone here  remembers that. And Zach can tell you a little bit more about some of these security concerns.  So, just to get a feel for things, does anybody remember that? He does. Okay. So, around 15 years  ago, back when dinosaurs roamed the earth and we were two years old, the MIT students figured out  how to hack the trolley cards, or more specifically the trolley tickets, the mag stripe tickets,  and add values up to $600 on them. So, they had the same ambition as us and wanted to come here  to present. The MBTA did not like that. So, they put out a restraining order against these kids  from MIT so they couldn't come here and present. So, if the slides were working, we'd have our  comically timed gavel and a picture of their restraining order, but you know, they got sued.  So, we actually were inspired by them a little bit, and when they got sued, their slides, all  their research was just put straight into public domain, which allowed people like us to get our  grimy hands on them. So, let's try to be like them, maybe minus the lawsuit part. So, step one,  go for the low-hanging fruit. Trolley tickets, they have awful security, and the MIT students,  they found out how to clone and reverse engineer them. So, seems easy enough, free rides for life.  So, step two, choose your weapon. If the slides were working, you'd be seeing a picture of a mag  stripe reader on eBay. It was $88, a little pricey, but it's an investment. Riding the subway is also  pricey and an investment. So, we're thinking, do we buy a mag stripe reader? The answer is no.  So, around the time when we started this project, about two years ago, the MBTA actually phased out  mag stripe tickets. So, any disposable paper tickets were out and actually tappable, which  renders them useless to us. So, it's time for a new plan. Yeah, so now that trolley tickets were  out of the question, and the new tappable tickets that replaced them seemed like they actually  might have had some decent security, we decided to move on to the trolley cards. So, if you're not  familiar, there would be a picture up here, but the trolley card is a contactless smart card. It was  introduced in 2004, before any of us were born. You just add money at vending machines and you tap  cards at fare gates. There you go. But how does that actually work? Well, when we first started  doing our research about two years ago, we weren't 100% sure. There wasn't very much published  research about it. We kind of assumed that it would use NFC or RFID. How else would you be able to  tap it? But we didn't know much more than that until we took a look at the MIT students and their  slides. And they described that it used MyFair Classic. And so, what is MyFair? So I had to  read another Wikipedia article, and I can tell you that it's a standard for data storage and  communication. It's made by a company called NXP. There's a lot of different flavors of it, MyFair  Plus, Ultralight, Desfire, that all serve different purposes. But the most famous, or rather infamous  flavor is MyFair Classic. And that's the one they use in the Charlie cards. It's the oldest flavor,  and it has NXP's homegrown encryption algorithm called Crypto One. And fortunately for us, it's  proprietary, 48-bit, and relies on security through obscurity. Now, if you don't actually know  anything about crypto, which I'm sure most of you do, that might sound like a good thing, because  proprietary formulas are touted as, like, good or something. 48's a big number. But any  encryption algorithm worth its salt is going to have millions of eyes looking for ways to  improve it. And 48-bit encryption can be cracked in, like, an hour. So unsurprisingly, some  researchers found some holes in it, and they tried to publish it. And they got met with a whole  mess of lawsuits around it. So now that we know what kind of technology we're dealing with, how  do we actually talk to these cards? Well, the first step, and I would have a picture here, was  grabbing an NFC reader. I found one on Amazon for, like, 40 bucks. It had a convenient USB  interface, and there was software out there for it. I then set out trying to figure out how to use  that software. It's called LibNFC. But I then sent it right back, because I couldn't figure out  how to use it. It was too complicated for me. So I had to buy another one. This one's called the  Proxmark 3. I don't know how many of you are familiar with it, but it's kind of, like, before the  Flipper Zero came around, like, Swiss Army Knife. It was 40, 80 bucks, although some of the  really expensive ones can be, like, $300 or $400. It has some really nice software that's really  easy to use. The only problem I had is that it didn't work. It came to my doorstep, and I could,  like, I opened it up, I could connect to it, but I couldn't really do that whole, you know, reading  and writing cards thing that I was supposed to do. So I had to buy another one, about 40 bucks,  this time from AliExpress. I'm sure that's not going to be an issue at all. And it wasn't. It  worked just fine. I could read and write to cards, and I had a blast with it, until, and I also  have a picture for this, but the USB, the micro USB header came clean off, and I tried soldering  it back onto the board, and I couldn't do that. So guess what? I bought another reader. This one's  PN532. It's a nice, small, red thing. It's about 9 bucks, and which might be concerning, given  that we got burned by the last two 40-dollar readers, but if you are familiar with it, you'll  know why it's 9 bucks. There's no USB. It doesn't even come pre-soldered, and it also only works  with the Raspberry Pi, but it does, like, actually work this time, like, not just break when you  turn your back on it, and what I thought would have been as simple as just buying one NFC reader  from Amazon and calling it a day turned into, like, a two-to-three-month ordeal, where I can  say that I did at least learn a lot about NFC and NFC readers, maybe a little more than I would  have liked to. Now that we have an NFC reader, we can actually start trying to talk to cards,  but before we do that, we need to get the encryption keys. Unlike regular encryption  that just has, like, one key or two keys, My Fair Classic has 32 keys in total, so you actually  need to go out and find them, and while we've established that the encryption they use is  pretty terrible, it's still, like, encryption, so you actually need to get the keys if you want  to have any fun. So how do we do that? Well, I found a slide show from Black Hat called  Hacking My Fair Classic Cards. I thought it would be kind of promising, and I took a look,  and it described two different attacks, nested and dark side attack, and it described some  software that implemented that. So I figured, all right, let's give them a go. So the first tool I  tried was called MFOC, implements that nested attack, and it does some fancy magic that I don't  really understand to get a card's keys, and I gave it a go, and it didn't work because you need an  initial key. So how do we get that? Well, we can try implementing the dark side attack.  Fortunately, there's some software called MFCuck that gets the keys kind of, like, out of thin  air. It's kind of cool. I don't know how it works. And I gave it a try, and it didn't work. I left  it running for, like, 24 hours, and I got nothing out of it. And by this point, I was thinking, am I  really going to have to buy my third Proxmark and try an attack where I put it in between the card  and the turnstile and try to sniff the keys? But then I thought, you know, it would be kind of  nice if the T just kind of left the back door open and used, like, default keys. So I made a  quick Google search or DuckDuckGo search for my classic default keys, and I found a pretty  awesome repository of, like, a thousand different keys in it. And I fed them into MFCuck, which  can take a key list as an input, and there they were. Right there. All 16 of them, A and B,  right there in that list. And I was like, holy shit, this is awesome. So now that we have the  keys, we can actually start reading and writing the cards. So we can grab a binary dump. Here I  would have a picture of a hex editor where you can see the binary data. But obviously you can't  see that. So now that we have a binary dump, we can actually attempt to implement a similar  attack that the MIT students did with their cloning. And before you do that, you need  yet more special equipment. You can't use any old regular MyFairCard. You have to use these  special Chinese back door magic cards that let you edit the UID for copying. And I had  fortunately accumulated a bunch of them with all the, like, bajillion different NFC readers I had  bought. So what I did was I took a dump from a Charlie card, took that hex dump, put it on a  magic card. We're going to get this fixed real quick. All right. We're going to use some  hand waving magic to make the slides change. So, yeah. You know, classic DEF CON fashion.  Okay. So what I did was I took a dump from a Charlie card, put that on a magic card, and  I tapped it on a reader. And sure enough, it actually worked. I would have a picture of it.  And we might in a second, but we'll probably be on that. But I have a nice white card tapped  on a reader. It's very obviously not a Charlie card. But crucially, the reader seemed to  believe it was. And I noticed that when I went to tap that clone card on a reader, it worked.  It deducted money. But the card that I copied it from still had its initial value. So if  you're kind of paying attention here, you'll notice that means that the source of truth isn't  in some database somewhere. It's actually right on the card. And the T is trusting people to  just not change it. But you can't trust us. So we figured, okay, that means we can actually  change it. So let's see. I'm trying to remember. I know I might have the slides up in front  of me. But, yeah, that means we can actually attempt to change that. But before I did any  of that, I wrote a little program that we call Charlie Clone written in C. I spent a long  time making a little progress bar for it. But it just copies cards pretty efficiently. And we  don't use that for any future cloning endeavors. And one of the issues with cloning cards,  well, it feels kind of like we found the infinite money glitch because you can just copy  cards. There are a couple of issues. Namely that if you want a $50 card, there we have it.  Yay! All right. This is great. This is going to be a lot easier now. So we need to go on.  I need to change the next slide. All right. So let's see. There are a couple of issues. Mainly  that you need an up front investment. If you want a $50 card, you're going to need to spend $50  to do so. And, you know, it might set off some fraud alarms here and there. Because they  can detect clone cards. We've actually gotten a couple disabled. But let's see. Okay. Yeah.  So there's a couple issues. Yeah. They can detect it. And when the cards do get disabled, you  have to spend yet another 50 bucks. Okay. Nice. Okay. Here we go. This is what I just  described. All right. Now we're in business. This is great. Okay. So as you can tell, there's a  couple issues with cloning cards. Right? So what's better than that? Forging cards.  Actually trying to put our own data on there and have some fun. So if we want to forge cards,  that means that we actually have to reverse engineer the cards and figure out how they're  storing data and figure out what the hell they're doing. So where do we start for that? Well,  it makes sense to look at the hex dump. After all, the money is stored on the card. So it must  be in here somewhere. I looked around for a couple things that stood out. I noticed a serial  number. When you convert the first four bytes, that's a UID, into decimal, they match the  number printed on the card. I then tried to change some data. Just, you know, I put in some  data and just tried to see what happened. And the reader seemed to disagree a little  bit. As you can see, it says, cannot read your card or ticket. And then I remembered something.  That the Charlie tickets, the old ones, had a checksum in them. It was pretty terrible, but  there was still some attempt to avoid people just tampering with it or accidental tampering.  So I quickly realized that the data, that the last two bytes that I've highlighted up here in  yellow, they might be a checksum. Because if you take a look at the red lines that are just  zeroes, they still have some of those two weird bytes at the end. And I thought to myself,  okay, well, checksum, that's not too bad, right? I mean, maybe they're just using some standard  algorithm and we can just plop the data into the checksum algorithm and have a blast. So I  tried some common algorithms. And you can see a couple pictures of them up here. I tried  like 20 or 30 of them. And sure enough, none of them worked. So at this point, I didn't  really know much about checksum algorithms or much of anything. And I thought, okay, this  checksum's a little bit evil. I don't know what they're doing. They're probably having their  own algorithm. And it's kind of above my pay grade of like zero dollars to try to figure that  out. So I figured, okay, cloning cards, there's some issues, but it's just all right. I'll take  them and just put this whole project to rest until I met Scott.  So I had received a student card in Boston area. If you live far enough away from your school,  they'll give you a Charlie card that gives you free transit within a given area, which happens  to be where I live. And I wanted to clone these cards and maybe share them with my friends. We  all get free rides, yay. And in passing, I mentioned that idea to Maddy at Robotics Club. We  didn't really know each other that well at the time. But that started a lifelong friendship,  didn't it, Maddy? I've got to really press it. All right. So I'm going to show you a  demo. He'd already figured out how to clone the cards. So we wanted to maybe take the project a  little bit further. So a reasonable step seemed to be let's find where the money is, kind of get  our bearings with the data. So we looked at some hex data together. For many hours and  hours, what we were trying to do is figure out where the money is. We would take a dump of one  card, save it, add a bit of money, take another dump, and then compare them in a program  called vbindeth and look at the differences. And then, like, we're trying to figure out which one of  these bytes is the money. There's another accurate 3D rendering. And about 2,000 years later,  after much trial and tribulation, we found nothing. So here's a sample of the  project. So here's a screenshot from that MIT DEF CON slide presentation we were talking  about earlier. And how do you reverse engineer things again? Oh, wait. You've got to change it.  So we had to figure out how to do that. But that pesky checksum is in the way. So let's try to  crack the checksum. So if you look at these two screenshots of binary dumps, there's, as you  see in the yellow, that's where the checksum is. And then there's all those zeros leading up to  those two checksums. Those are on two different cards. But even though all the data is the same,  the checksum is different. So that's something that's kind of weird. And it probably means  they're adding cryptographic salt. And for those of you who don't know, cryptographic salt is a  little bit of extra data that you mix in just to make it a little bit harder to go backwards.  You do it when you're storing passwords. And most of you are probably familiar with that. So we  tried those off-the-shelf checksum calculators again, but with a bit of salt. And none of them  worked. I spent a little bit of time in my English class doing this. And by a little bit of  time, I mean three straight weeks of all day. But I still managed to pass. So that brings us to  our first breakthrough. Up until this point, we'd been replicating earlier research. But now  we want to try to do something new. So if you take those two checksums at the end of the line  with all those zeros, we thought maybe there's a mathematical relationship between the  checksums between the two cards. So we tried a bunch of operations on them. Addition,  subtraction, multiplication, division. And eventually we landed on the bitwise operation XOR.  For those of you who don't know what XOR is, it's just like addition or subtraction but for  binary. So if you look at these two hex dump screenshots, you take the checksums. If you  XOR those checksums together, you get a value. That's not surprising. But if you take two  different lines that have identical data from two different cards and you XOR those  together, you get the same value. So that's kind of interesting. Maybe it's something we can  work with. So how do we use it? The first step to our process is you find two identical lines.  So go back to all those zeros. And then you find a checksum modifier. That's what we call the  value you get when you XOR the two checksums together. And we can use it to transfer lines from  one card to another card. So you copy any line from those two cards. You copy the whole line  over. Checksum and all. Then you XOR the checksum that you copied over with the checksum  modifier and you get a new checksum. And you write that into the checksum slot. And boom, the  line's copied. So just to recap, here are your instructions if you want to replicate this,  which don't. It's bad. But you start with two cards. You find two identical lines. XOR the  two different checksums to get a checksum modifier. Find a line that you want to move. Copy  that line onto the other card. XOR the old checksum by the checksum modifier. And profit. But  don't profit. So what can we do with this? You might be thinking, how is this any better than  XOR? But remember how those cards get disabled sometimes when you clone them? Now if your card  gets disabled, say you put $50 on the card you were cloning. It gets disabled, those $50 are  gone. But we can copy that money line from the old card to the new card. And we can also edit  data so it's easier to move from here in reverse engineering. So that moves us on to the next  step of trial and error. So up to this point, we had been just copying data from card to card.  But we still haven't been able to actually fine tune and edit values. So if you want a $50  card, you still have to spend that initial $50. So we had also been mostly just sitting at  home staring at hex data. And eventually if you want to get things done, you got to go to  some readers and tap them. And where do you do that? At a subway station. So if you want to  have your own subway station adventure, the first thing you need is an NFC reader and a  portable one. We didn't have any portable ones at this point. All of them had died on us. So  we had to make our own. We took the PN532 which works with the Raspberry Pi. Here's a picture  if you hadn't been able to see that. Hook it up to a battery bank and toss that whole  contraption into your backpack and refrain from boarding any airplanes. Next step is to spend  hours or better yet days just sitting at a train station trying to just kind of reverse engineer  stuff. More about that in the next slide. You can see some of the benches we sat at at those  stations. And we got quite familiar with the smell of subway stations. And while we were  sitting there, we tried to make a lot of random or better yet educated guess based changes to  try to figure out mainly how the checksum was working. How it was translating data into  those two bytes at the end. We would mostly try changing some data, seeing what happens or  just try changing the checksum directly. And what we do is we take all of the changes, flash  them on to some magic cards, run up to the reader, tap them, they'd make a loud and then we'd  run back to our seats, whip out our laptops and do that like 20 times. All the while we tried  to hide the cards in our hands to make it not obvious even though we're still running up to  the reader, tapping and running back and whipping out our laptops. But we never got stopped. No  one so much as even asked us what we were doing. And so eventually after doing this for long  enough, we made a bit of a breakthrough in terms of figuring out where the money is. So we  knew that the money is stored on the card. And so that means if you take the same card with  different amounts of money, we know that the money must have changed. So we can take a look at  some of the binary differences. This program right here highlights the data in red. That's  different. And I stumbled upon the bytes that I've highlighted in yellow. And I noticed  something about them when you convert them from hex into decimal that you get these numbers  here. And they don't quite look like anything. But if you divide by two, now they start to  look a little bit familiar. And upon trying this with other cards, we quickly realized that  this is where, those yellow bytes are where the money is stored. And it's also like in half  pennies for some reason. We also found that there's two transaction registers, one of the  current and last values and that Charlie cards are just kind of weird. So now that we know  where the money is, why don't we try to change it? So I'm going to go a little bit into the  weeds here. So either get your weed whacker ready or just tell the person next to you, wake  me up when I'm done. So the first step when you're trying to figure something out like this  is to isolate variables. As you can see here, the only data that's different between these two  lines is the stuff highlighted in red. That's the money on the left and the check sum on the  right. And we tried XORing the two values together between the different versions. And we  chose XOR because that seemed to be the key between copying data between cards. So we  figured it might be the key for copying data within a card. Or editing data within a card.  And we XORed the two money values together and we got a value highlighted in green that we  called the data modifier. We then XORed the two check sums together and got a value  highlighted in red that we called the check sum modifier. We call them modifiers because if  you kind of shift your perspective in terms of how we change the top line to get to the bottom  line, you XOR them by the modifiers. And that's a valid way to change data within this  line. Now, of course, there's a lot of valid ways to change data between any two lines. But  what we wanted to see is if this would work with other data. So we took a card with  nothing on it, zero dollars, zero cents, and we XORed the data by the data modifier and the  check sum by the check sum modifier and we got this new line here. And we weren't sure if it  worked. But when we went to tap it, it sure did. And so while this isn't anything crazy, it's  not even enough to ride the subway. It's still our own homegrown value right here. The next  step is to add a quarter. It's a little bit confusing. So if you don't fully understand it,  just trust us. But you need to add a quarter because this method relies on XORing things  together. And with XOR, you can only, like, if you get saturated with one, as you can see  with the example with 15, there's no way you can keep XORing it to get higher and higher. So  you have to add an external value to get another bit and then you can keep increasing that. So  what we did was we added a quarter and we got a card with $2.60 and then we did the same  thing by XORing the values to get a data modifier, XORing the check sums to get a check sum  modifier and then we XORed, we applied the strategy we did earlier by taking a card with zero,  XORing the money by the data modifier and the check sum by the check sum modifier and we got a  new line with $4.95 on it. And we went to tap that. That worked as well. So it means that we  have a repeatable strategy that we can just rinse and repeat and try as many times to see how  high we can get. And we did just that. And we got up to $163.84. And as you can see, the  buttons are grayed out because past $100, you can't even add any more money. I noted by  writing down the data and seeing how they grow, I saw a pattern that they were doubling and  I tried that again one more time and I got up to $327.67, which if you're good with your  powers of two, you know that's a signed 16-bit integer limit. And no, we tried. You can't have a  negative Charlie card. So we eventually refined this process so we don't actually have to add  quarters every time by writing down all the modifiers and building a big table of them. But  we got it down to a process like this. You buy a card for a quarter. You just casually set  the value to $327. And then you profit. We chose to open all the fare gates because we're so  nice. But if you were a little less scrupulous than us, you could use them for your own  personal use or you could sell them and make the T very mad at you. So how does this  actually work? So pardon me if I go a little bit fast. We only have 10 minutes left. So we  wanted to, that's how we did it, but we wanted to have a more refined version to use in  writing programs so we can do this more automatically. So here's the vocabulary you need to  know. The existing data is what's there. Target data is what you want to be. The data  modifier is what you get when you XOR the target data and existing data. The existing  checksum is what's there. Like the checksum is already on the card. The target checksum is  what we need to make the target data valid. The checksum modifier is what ‑‑ I'm just  going to skip part of this down to this slide. So this is our process. You XOR the existing  data with the target data to get data modifier. Plug the data modifier and the column the data  is in into our look up table. Our look up table is a super secret black box. META  specifically requested that we did not share this. But what that really means is come  after maybe 10 bucks and we'll talk. And that spits out a checksum modifier. Finally you  XOR the existing checksum with the checksum modifier to get a target checksum. So  there's an example of steps. Read really fast. And a few more station adventures later. We  were able to use this to reverse engineer a lot quicker. Because we can just edit data and  see what it does. Here's the charlie card. The coolest parts in my opinion are the money and  the card type. The money allows us to make a $300 charlie card. That's cool obviously. And  the card type allows to make employee cards which are our favorite. Blind cards, senior,  student cards, whatever we want. We can make it now pretty much. So. Okay we're on a time  crunch here. So what button is it? Okay. Fun gadgets. We did all this stuff but we need to prove  that we can do it. That's me. I thought about it. Built a machine. Version 1, bad. Version 2,  bad. Version 3, horrible. 4, little better, getting there. Version 5, the first one, the first  version that we showed to the MBTA. The screen's upside down and I broke it. Version 6, I  broke it on the way here. But it worked. It did everything. Changed the data such as the  expiration date, money, and it looks cool. So ingredients, just read blood, sweat, tears, don't  do it. So we're going to go pretty fast here. We chose tarry. It lets you do background  computation and front end computation separates them. We had a program that reads NFC and  MyFairCards on the back end and sends it to basically a web design front end. We also have an  Android app. It was slapped together in like two nights and super ugly. It's slow too but it  does the job. Part 5. Cool. We're here today. We didn't end up in handcuffs. So do we go to  jail or do we not go to jail? We could have published this, given it to you guys, come to  Boston, you know, no more MBTA, no more paying for it at least. But we decided to go to them.  Here's an article about Bobby Roush. He actually found out how to clone the cards, went to the  MBTA, wrote this nice article about him. So we say, hey, Bobby, talk to us, man. How do we do  this? How do we get in touch? So this is a complaint form for the MBTA. They don't really have a  we hacked your transit card form. So we filled it out under question, topic, others said, hey,  we did something. So we thought they were going to react like this, shoo us away. But they  responded. Not quite like this, though. That's an email inviting us to their headquarters. So  there you are. We went to the meeting. They brought in a bunch of execs. We talked to them  about what we'd been working on, how we did it, and how they could prevent us from doing it  next time. So that's a real picture of us. Just kidding. They didn't allow us to take pictures  in there. So post-meeting fun. Quickly, picture, picture. The cops got called on us.  Whole story. Ask us afterwards. Come on. Demo. What's our time? Do we have time to play it?  Four minutes. Do we do it? Okay. Can they do it over there? Can they play the video for us?  It's on the laptop.  Is there a laptop? Check files, maybe? Not quite. You see it? Okay. Maybe that other tab of  files. There's two. There you go. Yeah. All right. Wrong one. Wrong one. No, the other video.  Yeah, there I am. Okay. I don't even know if you're going to be able to hear it. We're just going  to roll. This is me buying a Charlie card. I bought it. Blah, blah, blah. There's, you know,  audio. 25 cents for cheapskates.  So right now there's 25 cents on it. It's verified. Okay. So you'll see the card type is adult.  Has one use. The money on the card is 25 cents. Expiration date January 14th, 2033. And it has  no passes on it. So let's say we want to edit the money value. Just click the money values button.  Okay. And we'll go, I don't know, just click some numbers. $251.53. So let's press edit. You'll  tap it on the reader. Then press your little edit button here. Give it a second. Tap it again.  And give it another second. And another. There we go. There we go. Okay. So let's check if that  edited anything. Okay. So as you'll see, the money value has changed to $251.53. So how about  the card type? Card type, another big button. Let's change it to an employee because, you know,  why not? So we click employee. Click edit. Put it on the reader. Then press our little button. Tap  again. Okay. So you'll see the card type, employee with passback. Again, number of uses, one.  Money on card, $251.53. So one more thing. Can we cut the video? So that's our time. Thank you  guys for being here. Then stand up. Give it up for this guy, these guys. Thank you guys so much.  Thank you guys.


url: https://www.youtube.com/watch?v=rimtaSgGz_4
title: DEF CON 31 - An Audacious Plan to Halt the Internet's Ensh*ttification - Cory Doctorow
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=rimtaSgGz_4",
    "title": "DEF CON 31 - An Audacious Plan to Halt the Internet's Ensh*ttification  - Cory Doctorow",
    "description": "The enshittification of the internet follows a predictable trajectory: first, platforms are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die. It doesn't have to be this way. Enshittification occurs when companies gobble each other up in an orgy of mergers and acquisitions, reducing the internet to \"five giant websites filled with screenshots of text from the other four\" (credit to Tom Eastman!), which lets them endlessly tweak their back-ends to continue to shift value from users and business-customers to themselves. The government gets in on the act by banning tweaking by users - reverse-engineering, scraping, bots and other user-side self-help measures - leaving users helpless before the march of enshittification. We don't have to accept this! Disenshittifying the internet will require antitrust, limits on corporate tweaking - through privacy laws and other protections - and aggressive self-help measures from alternative app stores to ad blockers and beyond!",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2742,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230915"
  }

  This text was transcribed using whisper model: large-v2

   Let's give Mr. Cory Doctorow a giant round of applause.  Thank you very much, hello, hi, thank you, hi, hi, hi, I have a confession to make, I  am now old.  I turned 52 last month, I am now the full deck of cards, I have two artificial hips,  I have cataracts in both my eyes, I am old as dirt.  And you may know that the AARP has these junk mail like doxing ninjas who track you down  on your 50th birthday to try and sell you a membership.  What's less well known is that if you buy the membership on that day, you get issued  a card that lets you complain that things used to be better when you were younger.  I got that card, and I know that the complaint is trite to say that things used to be better  in the old days, but I really think it's true when it comes to the internet.  I really do think the internet used to be better back before it turned out into what  the Kiwi hacker Tom Eastman calls five giant websites each filled with screenshots of text  from the other four.  And I miss the good old internet.  But this isn't a talk about bringing the old good internet back, it's a talk about what  a new good internet could be, and why we don't have it yet, and how we can get it.  So this is a talk about inshittification.  That's a term I coined to describe how platforms die.  And platforms, they're the endemic form of the internet.  A platform is a firm that mediates between end users and business customers.  Uber's got drivers and riders, Amazon and eBay have sellers and buyers, Google and Facebook  have publishers and advertisers and users on the other side.  The platform sits between those two different groups and mediates between them.  Remember when we said that the internet was going to disintermediate everything?  It did disintermediate everything and then it promptly re-intermediated everything.  Platforms are just intermediaries.  Now here's how a platform dies.  First it's good to its users.  Then it abuses those users to make things better for business customers.  It abuses the business customers to claw back all the value that had once been allocated  to those end users and then to those business customers, allocates it to themselves, and  then there's no value left, it turns into a pile of shit, and it dies.  And we are living through mass end-stage platform decay.  We are in the great inshittening.  And I'm going to explain how those three stages of the process of platform inshittification,  and then I'm going to explain the policy choices that let them get away with it, and  then I'm going to tell you what policy changes would let us seize the means of computation  and build a new good internet that is the worthy successor to the old good internet.  It's a talk about how we make the inshitternet that we have now an intermediate stage between  that old good internet and a new good internet.  So I'm going to start with a case study, Facebook.  Now Facebook is a company that was founded to non-consensually rank the fuckability of  Harvard undergraduates, and it only got worse after that.  Now when Facebook started, it was only open to people who had .edu addresses and K-12  U.S. addresses, American students.  But in 2006, Facebook opened to the general public.  It told users, yeah, I know you all use MySpace, but have you ever thought about how MySpace  is owned by an evil, senescent, crapulent Australian billionaire who spies on you with  every hour that God sends?  Come use Facebook.  We are the privacy-forward alternative to MySpace, and we will never spy on you.  Just come and tell us who in this world matters to you, put their names into Facebook, and  we will compose a personal feed for you consisting solely of what those people publish for consumption  by the people who follow them.  Now that was stage one.  Facebook had this surplus.  It had investor cash from people like Peter Thiel, and it allocated that surplus to us,  the end users.  And then we, the end users, we locked ourselves into Facebook.  Now Facebook, like most tech businesses, has network effects on its side.  A product or a service has network effects if it gets better when more people use it.  So you joined Facebook because there was someone there you wanted to talk to, and then once  you were there, you were a reason for someone else to join to come and talk to you.  But Facebook doesn't just have high network effects.  It also enjoyed high switching costs.  Switching costs, they're everything that you have to give up when you leave a product or  a service.  Now in Facebook's case, switching away from Facebook, the cost was access to all the people  that you hung out with on Facebook, the people who followed you and the people that you followed.  Now in theory, you could have all gotten together and agreed on where to go next and left the  platform together, but in practice, you were hamstrung by an insurmountable collective  action problem.  It's hard to get a lot of people to do the same thing at the same time.  You and your six hacker buddies are going to struggle to figure out where you're going  to go to dinner tonight.  How the fuck are you and your 200 Facebook friends going to agree on when it's time to  leave Facebook and where you should go next?  So Facebook's end users engaged in mutual hostage taking and kept themselves glued to  the platform.  And Facebook, it saw the hostage situation and exploited it, withdrawing the surplus  from those end users and allocating it to two important groups of business customers,  publishers and advertisers.  To the advertisers, Facebook said, hey, remember when we told those rubes that we weren't going  to spy on them?  We lied.  We will spy on them from asshole to appetite.  We will sell you access to that surveillance data in the form of fine-grained ad targeting,  and we will devote substantial engineering resources to combating ad fraud.  The ads will be dirt cheap to serve, and we will spare no expense to make sure that when  you buy an ad, a person sees it.  Now to the publishers, Facebook said, hey, remember when we told those rubes we were  going to only show them the things that they asked us to show them?  We lied.  Upload short excerpts from your website, put a link at the bottom, and we'll non-consensually  cram it into the eyeballs of people who never asked to see it.  We are offering you a free traffic funnel that will drive millions of users to your  website that you can monetize as you please.  And if those users subscribe to your feed, they'll be stuck to you forever.  And so advertisers and publishers, they got stuck to the platform too, dependent on those  users.  So the users were holding each other hostage, and the hostages took the publishers and the  advertisers hostage too, and everyone was locked in, which meant it was time for the  third phase of inshittification, withdrawing surplus from everyone and handing it to Facebook  shareholders.  For the users, that meant that the feed that you got, the quantum of stuff that you would  ask to see, was dialed down to a homeopathic dose, so that there could be a resulting void  that could be filled with ads and pay to boost content from publishers.  For advertisers, that meant the prices went up and the ad fraud policing went down.  So advertisers paid much more for ads, and many or most of those ads never got shown  to a human being.  For publishers, Facebook reached in and algorithmically suppressed the reach of content, unless larger  and larger excerpts were put in the posts that were posted to Facebook, until anything  less than a full text feed would likely not reach even your own subscribers, let alone  be boosted to other people through algorithmic recommendation, and then Facebook put the  knife in.  They started punishing publishers who put links to their own content, to their own website,  in those posts.  So they were corralled into posting full text feeds with no links back to their website,  becoming commodity suppliers to Facebook, entirely dependent on Facebook, both for reach  and for monetization, which was only available through that crooked advertising system.  When any of those groups squawked, Facebook just repeated that lesson that they'd learned  on day one of the Darth Vader MBA, I have altered the deal, pray I don't alter it any  further.  Now Facebook has entered the final and most dangerous phase of inshittification.  It wants to withdraw all available surplus and leave just enough residual value in the  service to keep end users stuck to each other and business customers stuck to those end  users, without leaving anything extra on the table, so that every extractable penny is  drawn out and returned to its shareholders.  But that is a very brittle equilibrium, because the difference between, I hate this service,  but I can't bring myself to quit it, and Jesus Christ, why did I wait so long to quit  this shithole, get me the fuck out of here, is razor thin.  All it takes is one Cambridge Analytica scandal, one whistleblower, one live-streamed mass  shooting and users bolt for the exits.  And Facebook discovers that network effects are a double-edged sword.  If users can't leave because everyone is staying, then once everyone leaves, there's no reason  to stay.  That is the terminal phase of inshittification, the phase when the platform becomes a pile  of shit.  And that phase is usually accomplished by panic, which in tech circles we call pivoting.  Which is how...  Thank you.  All right.  Which is how we get pivots like, in the future, all internet users will voluntarily transform  themselves into legless, sexless, low-polygon, heavily surveilled cartoon characters in a  virtual world we ripped off from a 25-year-old satirical science fiction novel.  Now, inshittification is not inevitable.  Plenty of tech platforms in history died without inshittifying.  But when three criteria are satisfying, inshittification always ensues.  The first is a general lack of competition.  Forty years ago, the Carter administration started pulling Jenga blocks out of American  antitrust enforcement.  Then Reagan started pulling them out by the fistful.  And every administration since, but not including the Biden administration, has continued to  nerf antitrust, making it progressively weaker, until every industry, not just tech, is dominated  by a tiny handful of companies.  Whether we're talking about pharma, health insurance, appliances, athletic shoes, books,  shoes, drugstores, office supplies, eyeglasses, vitamin C, bottle caps, airlines, railroads,  rental cars, mattresses, champagne, candy, and professional wrestling.  These companies grew by doing things that were illegal until the 1980s, which is not  coincidentally the time that we got the first PCs.  PCs, the internet, and antitrust drawdown, all those phenomena occurred at the same time.  Tech is the first industry in a century to be born even as antitrust was getting weaker.  Now, what laws were those companies allowed to violate?  Well, first they were able to sell goods below cost.  And that meant that if you had deep pockets, you could bankrupt your competitors and prevent  new companies from entering the market.  So some of you may remember when Amazon tried to buy a company called diapers.com.  Diapers.com said, we don't want to sell.  We like being our own business.  So Amazon lit $100 million on fire in the next few months, selling diapers below cost  until diapers.com went bust.  So this is not about the best company succeeding.  It's the company whose shareholders are willing to lose the most money that gets to survive.  But more than anything, these companies were able to merge with their major competitors  and buy out small ones.  So think about Google.  Google made one good product 25 years ago, a really amazingly great search engine.  And that amazingly great search engine opened a conduit to the capital markets.  And that gave Google an effectively blank check to buy competitors.  So it didn't matter that virtually everything Google developed in-house was a failure.  Videos, social media, Wi-Fi balloons, smart cities, they couldn't even keep an RSS reader  running.  It didn't matter because they could buy other people's companies.  And that's how they got a mobile operating system, an ad tech stack, video, maps, documents,  satellites, server management.  Google is not Willy Wonka's idea factory.  They're just rich uncle penny bags spending other people's money to buy the products that  they themselves are too ossified and lumbering to create.  And that's not just Google.  Apple buys 90 companies per year.  Tim Cook brings home a new company more often than you bring home a bag of groceries.  Eliminating competition is the first step in inshittification.  Because it's a lot easier to treat your customers and suppliers like shit when you're the only  game in town.  But all industries have consolidated, and not just tech.  And shitification is what happens when you stir two more tech-centric factors into the  mix.  So I want to take a quick interlude from economics here and do some hacker shit.  Remember when I told you about how network effects drive explosive growth?  Well, tech does have amazing network effects, but it has another property, an irreducible  feature that operates as the anti-network effect, which is low switching costs, which  are driven by universality.  Because we only know how to make one computer, right?  The Turing-complete universal binomial machine that can run all the software we know how  to write.  We don't know how to make a computer that runs all the programs except for the one that  makes your shareholders sad.  We just know how to make that one.  If that wasn't the case, we wouldn't see people showing up every year at cons like this going  like, hey, guess what?  I figured out how to write malware in PostScript, right?  So that means that every software-driven product or service is liable to adversarial interoperability.  That's when a hacker uses reverse engineering, scraping, and bots to plug something in that  the OEM doesn't want plugged in.  Remember that Facebook case study?  When Facebook was telling MySpace users that they needed to escape Rupert Murdoch's evil  crapulent Australian social media panopticon, it didn't just say to those MySpacers, hey,  fuck your friends.  Come to Facebook and just hang out looking at our cool new UI until they show up.  Instead, it gave them a bot.  You fed the bot your login credentials, and it would go to MySpace and pretend to be you,  and it would scrape the things waiting in your inbox, and it would copy them into your  Facebook inbox, and then you could reply to it and paste it back into your MySpace outbox.  The explosive growth that platforms get from network effects draws competitors, and those  competitors hack interoperability layers into their products that attack those incumbents  on their highest margin offerings, draining out users, draining off revenues, so that  every company starts with explosive network effects growth, and then it ends with implosive,  low-switching cost-driven contraction.  Every successful tech company started with adversarial interop.  Google presented itself to the world's web servers and said, hey, I'm a user, just give  me your web pages, please.  Apple reversed Microsoft Office and made iWork suite, pages, numbers, keynote, that could  perfectly read and write the files from Word, Excel, and PowerPoint.  Look hard at any tech company, and you'll find them ripping, mixing, and burning the  biggest products and services of the day.  Why do tech companies go after the biggest products and services?  Same reason Willie Sutton robbed banks.  That's where the users are.  But when these companies exploited the dying of our antitrust rules to grow to unimaginable  scale, they joined forces and declared the end of history.  Adversarial interoperability was great when they did it.  It was necessary to humanity's progress.  But anyone tries to do that to them, well, that's a crime, because, you know, every pirate  wants to be an admiral.  An industry with 1,000 small and medium-sized companies has the same collective action problem  you and your friends will have when you're looking for a place to have dinner tonight.  They won't be able to agree on anything.  Not only can they not agree on what the law should be for their industry, they can't even  agree on how to cater the annual meeting where they would discuss the question.  Do you remember the Napster Wars?  Tech was much bigger than the entertainment sector back during the Napster Wars, but big  content kicked tech's ass, because there were five labels and seven studios, which meant  that they could easily agree on a single unified response to P2P.  And today, there's three labels and four studios.  Those companies are so incestuous, they've got the corporate equivalent of a Habsburg  jaw, and they've decided that they're going to replace every creative worker with a chat  bot, which is why the actors and writers in my hometown of Burbank have spent the summer  roasting on treeless sidewalks in front of the studios.  Because when a sector has five companies or four or three or two, it can agree on one  policy direction, and it can screw its customers and suppliers so hard that it amasses a fortune  that lets it get that policy.  Today's tech is even more concentrated than the entertainment sector was back during the  Napster Wars, and they've agreed that history has come to an end.  When Apple reversed Office and built iWork, Microsoft had to just suck it up.  But in the ensuing decades, Apple and Microsoft and Google and Facebook and the other tech  giants have secured changes to law, regulation, and policy that make it illegal to do unto  them as they did unto others.  If you were to reverse the file formats used by iOS and make a runtime for its apps and  a player for the DRM-restricted media that you get through it, Apple would reduce you  to rubble.  They'd come after you with Section 1201 of the Digital Millennium Copyright Act, the  Computer Fraud and Abuse Act, tortious interference with contract, copyright, patent, trademark,  and trade secrecy, the stuff that we call IP, in other words.  Now I know free software hackers hate it when you use the word IP.  They say it doesn't mean anything, but no, IP means something very specific when we're  talking about a business context.  IP is any rule or law that lets me reach outside of the four corners of my business  and exert control over the conduct of my competitors, my customers, and my critics.  Or as Jay Freeman from the Cydia Project puts it, this is felony contempt of business model.  And here's what that looks like.  Today, one in four web users has installed an ad blocker.  Doc Searles calls it the largest consumer boycott in history, and he's not wrong.  Ad blockers are only possible because browsers are open platforms.  You don't have to bypass any IP to mod a browser and change the way the page renders  in the user's browser.  But if you want to add ad blocking to apps, you would commit half a dozen federal crimes.  Bypass the DRM?  Well, that's a DMCA 1201 violation.  It's punishable by a five-year prison sentence and a $500,000 fine for a first offense.  An app is just a name for a web page that's been skinned with enough IP that it will allow  Apple or Google to send you to prison for felony contempt of business model if you mod  it.  So let's get back to inshittification, hacker interlude over.  Inshittification is what happens when companies do not face competition and when they're able  to use that incredible flexibility of our universal computers to twiddle the knobs on  the back end to do Darth Vader shit, altering the deal further, unconstrained by privacy  law, labor law, fair trading law, turning every platform into a rigged Skinner box casino  where the payout schedule is altered from moment to moment, making it impossible for  end users or business customers to figure out whether they're getting a fair deal.  Tech companies can twiddle the knobs whenever they want without explanation, without transparency,  and we can't get a law passed to make them stop compulsively touching their knobs.  Because in a world of five giant websites, each filled with screenshots of text from  the other four, they can easily agree that those rules will be bad and they can mobilize  the monopoly winnings from that rigged casino they've built to make sure that those laws  never pass.  So let's take stock.  Step one, consolidated industries eliminate competition through predatory pricing and  acquisition.  Step two, tech companies play this high-speed shell game on the back end and use their consolidation  to Bigfoot any attempt to constrain their twiddling, like labor, privacy, or fair trading  laws.  Now we come to step three, where tech companies embrace tech laws, but just the laws that  make it illegal for us to twiddle back at them, the IP laws that make felony contempt  of business model a reality, criminalizing adversarial interoperability, this process  that once acted as garbage collection for the inshittified, bloated, top-heavy companies,  letting nimble, innovative players drain off their users, eat their lunch, and dance on  their graves.  Put these three factors together, consolidation, unrestricted twiddling for them, and a total  ban on twiddling for us, and inshittification does become inevitable.  That's how inshittification works.  Now let's talk about how we're going to end it, throw it into reverse, and build a new  good internet that's a worthy successor to that old good internet.  Step one, we've got to halt consolidation and break up big tech companies.  This one is actually going great.  After 40 years, we have the first US administration in two generations to take this seriously,  joined by colleagues who are doing really muscular shit in the US and the EU, and even  in China.  They are blocking mergers, demanding breakups, and they're fighting alongside lawmakers who  are joining hands across the political spectrum.  There's a bill in Congress right now called the America Act that'll break up Facebook  and Google, and its two lead co-sponsors are Ted Cruz and Elizabeth Warren.  The Federal Trade Commission and the DOJ have published new merger guidelines which ban  anti-competitive mergers that have been the norm for 40 years.  Now if you're only cursorily paying attention to this, you might have gotten the impression  that the amazing chair of the FTC, Lena Kahn, is thrashing indiscriminately and losing big  tech mergers, like the Activision-Microsoft merger that she tried to block.  But the reality is, Kahn is trying to make new law.  After four decades of complacency and a bias in favor of monopolies, she is taking swings  no one has taken since the Carter administration.  She is a goddamn American hero, and her colleagues like Rohit Chopra at the CPFB and Jonathan  Cantor at the DOJ are doing the Lord's work.  Thank you.  But it takes a long-ass time to do these breakups.  It took 69 years to break up AT&T.  I don't want to wait that long for a new good internet.  And we don't have to, because tech is different.  It's universal, it's interoperable, and that means we have options that we've never had  before when we were fighting rail barons and oil barons and the Whiskey Trust.  Those options are interoperability-driven, and they will devolve control over technology  from giant companies to small companies or co-ops or non-profits or communities of users  themselves.  Interop is how we seize the means of computation.  So how do we do that?  Well, first things first, we've got to limit twiddling.  We've got to pass comprehensive federal privacy laws with a private right of action, meaning  that you can sue if your privacy is violated, even if your local public prosecutor doesn't  think you deserve it.  We need to end worker misclassification through the so-called gig economy, meaning that every  worker will be entitled to minimum wages, a safe workplace, fair scheduling, and apply  normal consumer protection standards to e-commerce platforms and search engines that ban deceptive  advertising, fake reviews, and misleading search results that put fake businesses and  products ahead of the best matches.  Thank you.  Then we need to open up those walled gardens.  There's laws like the Digital Markets Act and the European Union that are going to force  tech platforms to stand up APIs that allow new platforms to connect to them.  This interop will make switching costs low again, so you can leave Facebook or Twitter  and go to Mastodon Diaspora or Blue Sky or some other new platform and still exchange  messages with the people you left behind and participate in the communities that matter  to you and connect to the customers that you rely on.  These new platforms, they need to be fiddle-constrained the way the big ones should be, subject to  the same privacy, fair trading, and labor rules.  But mandatory APIs, there's a fatal flaw in administering them because they are so easy  to cheat on.  Because if we order Facebook to open an API to allow interoperators to siphon off their  users, it doesn't mean that we don't want them to pull the emergency brake if they think  someone's exploiting it to steal millions or billions of users' data.  And that means that Facebook can cheat because they can claim that they pulled the plug because  they thought there was a breach when really they just wanted to destabilize those new  platforms, teach their founders, their users, their investors, you don't bet against Facebook  if you want to win.  And even if you drag Facebook in front of a regulator to get them punished for this,  it's going to take years to get justice.  Because to a first approximation, everyone who understands Facebook's infra well enough  to determine whether the shutdown was pretextual is a Facebook employee.  So we're going to have disputed shutdowns that turn into years-long fact-intensive inquests  that will make nobody happy.  To make mandatory APIs work, we need to make robust interoperability preferable to that  behind-the-scenes fuckery.  We need to align tech giants' incentives so that they encourage competition rather than  sabotaging it.  And this is where you all come in.  This is the part that we need hackers for.  Because in addition to mandatory interop that's already coming down the pike, we need to restore  the right to mod, tinker, reverse, and hack these services.  I'm going to tell you why and how, and how we're going to make it safe for users.  First, why do we need to do this?  Well, companies do hate competition, but the one thing they hate more than competition  is surprises.  If we have the right to mod existing services to restore busted API functionality, then  any company that is tempted to nerf its APIs has to consider the possibility that you're  going to come along and scrape its site or reverse its app and make that API work again.  And that means that the choice for tech giants isn't keep the API and lose my discontented  users, or nerf the API and screw my competitors.  It becomes this.  Keep the API and lose my discontented users, or nerf the API and get embroiled in an unquantifiable  guerrilla warfare against engineers who have the attacker's advantage, meaning I have to  be perfect and they only need to find one mistake I've made and exploit it.  Tech giants hate surprises because investors hate surprises.  When you get a quarterly earnings call and announce worse news than predicted, your company's  share price tanks.  Remember the start of 2022 when Facebook told its investors that it attracted slightly fewer  American users than it thought it would in the previous quarter?  And they engaged in a mass sell-off that lost a quarter of a trillion dollars off the company's  share price in one day, the single largest tanking of any corporate valuation in the  history of the human race?  Now, the people who make the call to break an API, they're executives at those companies.  Those executives, more than anyone else in the world, have portfolios that are top-heavy  with shares in the companies they work for, meaning that if they do something that creates  a surprise, that creates a mass sell-off that tanks the company's share price, they themselves  are the ones who are going to get hurt.  So you put the decision-makers on the front lines of their own bad decisions, and you  know what?  No one ever lost money betting on the hubris of tech leaders, so maybe they'll go ahead  and do it anyway, and if they do, we'll have adversarial interoperability.  We can hack, scrape, and reverse our way back to the API that they've shut down.  So how do we get adversarial interoperability?  Well, we should roll back every law or regulation that constitutes felony contempt of business  model.  Anti-circumvention, criminalizing terms of service violations, overbroad patents and  copyrights, all of it.  But that's a project of years, and we need adversarial interop now, and here's how we  can do that.  First, we can just wait for these companies to cheat, because they're going to cheat,  right?  When we pass the Digital Markets Act or any other law that constrains their twiddling,  they're going to cheat because they're incapable of not cheating.  They've proven that over and over again, and when they cheat, we'll penalize them.  We can stick them with a special master, this is a kind of court-appointed adult supervisor  who has to approve their legal threats against interoperators and verify that those threats  are about protecting the company's users and not its shareholders.  Now, while we're waiting for them to cheat, we can put the government to work for us,  specifically government procurement.  Governments should require that every tech company that sells them a product or a service  has to promise not to interfere with interoperability.  I mean, that's just good prudent administration.  The Lincoln administration only bought rifles from companies that agreed on standard tooling.  I mean, of course they did.  Wars cancelled, boys.  The bullet factory shut down this week, right?  That has been the bedrock of good public procurement for centuries.  We just forgot it.  Every digital system procured by every level of government should come with a binding covenant  not to impede interoperability, from the cars in your government motor pool to the Google  classroom in our public schools to iPhones in our public agencies.  Now, those companies, they're going to squawk, but no one forces a tech giant to sell to  the American government.  If you're too emotionally fragile to sell to the American public on fair terms, go find  another line of work better suited to your delicate sensibilities.  Your shareholders' priorities are your problem.  Public agencies are charged with doing the people's business.  Thank you.  Okay, so we're going to use adversarial interrupt to keep big companies from sabotaging mandatory  interrupt.  We're going to use procurements, conduct remedies, and new law to get adversarial interrupt.  But how do we keep the interoperators honest?  After all, if you squint just right, Cambridge Analytica is just an interoperator.  Now, remember, I talked about putting limits on twiddling, privacy law, labor law, fair  trading laws.  That's how we do it.  I mean, frankly, it's surreal that the primary mechanism by which we keep Facebook's users  from abusing Facebook's, or Facebook's partners from abusing Facebook's users is by asking  Facebook to decide what is and isn't good for its users.  Remember, Cambridge Analytica was a Facebook partner.  So whether you're using an API or whether you're fielding an interoperable app that  relies on scraping and reversing, we want you bound by those same laws.  But these should be laws that are passed by democratically accountable lawmakers and public  proceedings, not by shareholder accountable executives and closed door boardrooms.  And shitification didn't happen because today's companies are run by evil geniuses.  They are no more wicked than the everyday mediocrities who founded DEC and Sun and AOL.  All of those companies would have happily abolished their competitors, captured their  regulators and abused their users and business customers if they could have gotten away with  it.  We didn't let them get away with it.  But we let the current crop get away with murder.  They're just able to buy their way to dominance, merge with their competitors until they have  the money and unity of purpose to capture our laws, to give them the freedom to abuse  us without limit and criminalize anything we do in our own self-defense.  To stop them, we need to block new mergers and unwind existing ones.  We need to limit their ability to twiddle the back end, keep their users and business  customers in a constant state of confusion, and we need to restore our ability to twiddle  back to give ourselves an internet operated by and for the people who use it.  That new good internet would be a worthy successor to our old good internet.  Now for millennia, the indigenous people of California used controlled burns to wipe  out old and sick trees, opening the canopy for new growth.  But when the settlers banned good fire, California started to accumulate fire debt.  And that means that every year, California burns.  Because the alternative to good fire isn't no fire, it's wildfire.  When tech companies had to contend with the implosive contraction of low switching costs,  they were dynamic, springing up and disappearing all the time.  When we stopped enforcing antitrust law, we ended good fire, and we got wildfire.  Our tech companies have terminal gigantism, and they're on fire all the time.  It's time to stop making the tech giants better, and time to start evacuating them and letting  them burn.  In your heart, you know we could have a better internet than this one, and a better tech  sector.  Do you remember when tech workers dreamed of working for a big company for a few years  before striking out on their own to start their own business that put that big company  out of business?  Then that dream shrank to working for a tech giant for a few years, quitting and doing  a fake startup to get acquihired back by your own boss in the world's most inefficient way  to get a raise.  And then it shrank even further to working for a tech giant for your whole life, but  there'd be free kombucha in the cafeteria and you'd get massages on Wednesdays.  Now that dream is over, and all that's left is, work for a tech giant until they fire  your ass, like those 12,000 Googlers who got fired six months after a stock buyback that  would have paid their salaries for the next 27 years.  We deserve better than this, and we can get it.  I want to take a lesson here from my arch nemesis, a guy called Milton Friedman, the  court sorcerer of Ronald Reagan, architect of the neoliberal revolution, author of Our  Misery.  And he was a monster, but he knew a thing or two.  And when people would ask him, Milton, how will you ever put your kooky, fringed ideas  into operation?  He would say, someday there will come a crisis.  And when crisis comes, ideas that are lying around can move from the fringe to the center  in an instant.  I love quoting Friedman.  I imagine that when he hears his words in my mouth, he looks up from that spit he's  roasting on and gargles a curse at me around that red-hot iron bar sticking out of his  jaws while the demons around him laugh and laugh.  And we are lurching from crisis to crisis, and thus far, we do the same thing with every  crisis.  We do the same thing we did last time, but harder, and hope for a different outcome.  We need to start spreading good ideas lying around so that that next crisis doesn't go  to waste.  Now, I'm not a hacker.  I haven't written commercial software since the last century.  These days, I run my mouth, I write books.  My next book is called The Internet Con, How to Seize the Means of Computation, and it  comes out in three weeks, and it explains this stuff in detail.  But it's not my only book, because I write when I'm anxious, so I came out of lockdown  with eight books.  There's two of them in the dealer's room at the No Starch Press booth.  The first one's a hacker novel about the old good internet and the people who built it  called Red Team Blues that's dedicated to Dan Kaminsky.  I finished it the week he died.  And then there's a book called...  Thank you.  Yeah.  One for Dan.  And then there's a book called Chokepoint Capitalism I wrote with my friend Rebecca  Giblin about creative labor markets and how they were captured by big tech and big content,  how to win them back.  I'm gonna be at the No Starch booth from 2.30 to 3.30 if you wanna come by, and I can make  your books non-returnable with a Sharpie, and you can ask me questions.  But you don't have to buy a book or line up for that.  You can email me.  I'm cory at eff.org.  I've been with EFF since 2002.  It's 21 years now.  And so I remember that old good internet, and with my colleagues there, we're fighting  for a new good internet.  And so the last thing I'll say to you, I don't know if we've got time for questions, but  the last thing I'll say to you is I hope that those of you who do support EFF will continue  to do so, and I thank you for it.  And I hope that if you haven't looked too hard at EFF yet, that you figure out why so  many people are wearing our schwag at this con and find a little more and see if you  can find it in your heart to support us too.  So that's what I had to say to you today.  That's the plan.  Thank you very much for coming and listening.  Thank you.  Ten minutes?  So we got ten minutes, a long rambling statement followed by, what do you think of that as  technically a question, but it's not a good one?  If anyone has questions, I'm happy to take them.  Sir?  Shout and I'll repeat it back.  So the question is, if you're bootstrapping a company and you don't want to get sucked  up by big companies, you're avoiding VCs, what can you do?  Well, there's a whole class of stuff that economists call Ulysses Pact.  So Ulysses, you may know the story, right?  Ulysses was a hacker.  He knew that if you sailed through the sea of the sirens and you heard their song, you  would jump into the sea and that the sailors had a standard protocol for not being lured  to their death.  They'd fill their ears with wax.  But Ulysses, being a hacker, he wanted to hear the song.  So he improvised a solution.  He said, tie me to the mast.  And whatever I do, don't untie me.  I want to hear the siren's song.  And he did.  Ulysses Pact is when you anticipate a time in the future when you can reasonably assume  that you will be weak and you take a step now to prevent yourself, now while you're  strong, from giving into that weakness.  So you ever go on a diet and throw away all the Oreos?  That's a Ulysses Pact.  And we have Ulysses Pacts in tech, right?  The reason GPL and Creative Commons licenses and other free and open licenses are irrevocable  is so when the day comes that your VC says, all right, hippie, we went along with this  bullshit open source stuff at the start, but now you've got to make a choice, right?  We can pull the plug on you, and you can put those 150 people who quit their jobs to  follow you into this startup out on the breadline and cost them their kid's college fund.  Or you can put this source back in the proprietary box.  You can say, you know, I would do it if I could, but I can't.  I tied myself to the mast, and no one's going to untie me.  I have an irrevocable license on my code.  So the answer to this is, like, not being strong, right?  The answer to this is anticipating your weakness and taking steps now while you are strong,  while you are not stressed, while you're not in those circumstances, to ensure that you  won't be tempted, right?  The lesson of inshittification is not that we have uniquely evil tech leaders.  The lesson of inshittification is that we have uniquely weak constraints on ordinary people  that leads them into this absolutely wicked course of action.  Other questions?  I think we've got time for one more.  Yes, ma'am.  Okay.  Right.  So the question is, like, the kind of fediversity,  validity kind of world of cool, open, federatable, hard-to-capture services,  will they be able to find enduring success?  And I would say that I'm way more interested in their success than their enduring success,  because the whole point about federated open services is that they fail really gracefully,  right?  I remember trying to get X to work on my Linux box.  The fact that it doesn't exist anymore is fine, right?  Because we weren't locked in.  If you have an old toilet seat iBook that you put X on,  it's not like you can't put Wayland on now, right?  You can migrate, right?  So I actually think I am way more interested in those services attaining however brief a success,  because they have a nice grease skid into the next open platform and the next and the next.  This often happens that we forget a lesson, right?  We put our rat poison for years.  We don't have any rats.  We take the rat poison away because who needs it, right?  We haven't had rats for years.  And then there's rats everywhere.  We're like, where did these fucking rats come from, right?  We used to know that we needed open platforms in the early days of the web.  We all remembered the hard lessons of CompuServe and Genie and AOL.  And then we forgot, right?  And now we're learning those lessons again.  And so we need to keep those lessons open.  The question of whether they can attain liftoff, you know, it's this hard equilibrium where  if the existing services are so terrible, that people are willing to endure the high  switching cost of leaving their friends and everything that matters to them.  I mean, yeah, those services are going to get a lot of users.  But it will be because we just let this like point of no return arrive,  which is actually not great.  Like, I would much rather make it easy to evacuate those services, you know,  by getting the European Union to mandate interop for them through the Digital Markets Act,  than like wait for them to become so unbearable, that even people for whom they are very important  can no longer stay and end up in the Fediverse like that.  I'm a committed Fediverse user.  But like, that's not how I want people on the platform.  I want to minimize the amount of harm that people have to go through.  On the other hand, you know, let's not let the crisis go to waste.  If it does get that bad, let's make sure we welcome them.  Yeah, last question.  Yeah.  Intransigent.  Yeah, so the question is, the Biden administration is the first administration in two generations to  take antitrust seriously.  How do we hurt the politicians who are standing in their way?  You know, there's an old joke from Ireland, not an Irish joke, a joke from Ireland,  whose punchline is, if you wanted to get there, I wouldn't start from here.  And, you know, it's corollary is like the best time to have done something about corporate  corruption in our politics was 40 years ago.  And the second best time is now.  It's going to be a slow process, right?  Like Chuck Schumer blocked the floor vote for the Access Act last quarter and, you know,  or last session.  It's going to be really hard to say, well, you know, everybody should vote for Mitch  McConnell to show Chuck Schumer, you know, what he's what he deserves.  I mean, not least because it's not a plausible path, right?  I mean, even if you are not partisan in any way, the number of Chuck Schumer voters who  are going to become Mitch McConnell voters off the back of this is going to be very small.  And so, you know, I think a lot of this happens at the primary level.  It happens at the state level.  It happens, you know, Larry Lessig says that we have four forces, norms, laws, code, and  markets.  It happens at the normative level.  Like, I think all of us are the people that are the tech support army for all the normies  in our lives.  And, like, we've had this attitude that, well, it's more important that it be easy so that  you can use it than it be right.  And we all get that that's not a completely indefensible position, but that it also has  limits, right?  Like, your password just can't be password 123 on all your services, no matter how hard  you find it to use a password manager.  And maybe we need to move this question about big tech independence, interop, and so on  out of the realm of things that are so much of a pain in the ass that you shouldn't bother.  This is just for, like, us ninjas and not for you normies.  And maybe we need to do what we did with passwords and start talking about how we make it for  everybody.  They deserve it.  All right.  So, 2.30, I'm going to be down in the dealer's room at NoStarch.  Come by.  Thank you all for coming.


url: https://www.youtube.com/watch?v=GOoqEVhvNw8
title: DEF CON 31 - Demystifying (& Bypassing) macOS's Background Task Management - Patrick Wardle
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=GOoqEVhvNw8",
    "title": "DEF CON 31 - Demystifying (& Bypassing) macOS's Background Task Management - Patrick Wardle",
    "description": "To retain a foothold on an infected system, most Mac malware will persist; installing itself in a manner that ensures it will be automatically (re)launched each time the infected system is rebooted. \n\nIn macOS Ventura, Apple's rearchitected core persistence mechanisms and added a new security mechanism that alerts the user any time an item is persisted. As the former is both undocumented and implemented in a proprietary manner this poses a problem for existing security and forensics tools (that aim to heuristically detect malware via unauthorized persistence events). On the other hand, the latter is problematic to malware authors, who obviously want their malicious creations to persist without an alert being shown to the user. \n\nIn this talk, we'll indiscriminately provide solutions for all! First, we'll dive into the internals of macOS's Background Task Management (BTM) which, as we'll see, contains a central (albeit proprietary) repository of persistent items. Armed with this information, we'll release open-source code capable of programmatically enumerating all persistent items from BTM, ensuring security and forensics tools regain compatibility. We'll also highlight design weaknesses that malicious code could trivially employ to sidestep the new security features of BTM, such that persistence may still be silently achieved.",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2830,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230828"
  }

  This text was transcribed using whisper model: large-v2

   Good morning. Aloha and welcome to my talk on Apple's background task management. My name is  Patrick Wardle. I am the founder of the Objective C Foundation. There I work on free  open source Mac security tools. I write the Art of Mac Malware book series and also  co-found and organized the Objective by the C Mac security conference. So basically all  things Apple, all things Mac. So today what will you learn? What will we dive into? What  am I going to be talking about? Well the cool thing is there's kind of many facets to this  talk. I mean really the goal is to focus and understand on Apple's Mac OS's background  task management, BTM, which of course we'll explain exactly what that is. But we're also  going to cover topics of reversing malware detection, Mac OS internals and more. So  hopefully there's something in this talk for everyone. We're going to start by diving into  the internals of Apple's new background task management. I will caveat this part. We're  going to get into the weeds pretty deep. Some reverse engineering, disassembly. But this is  really important for the remainder of the talk. And with an understanding then of the internals  of this background task management, we'll then show how we can build some neat tools to  leverage this. Then because, you know, DEF CON is a hacker conference, we'll talk about of  course how to fully bypass this new system. And then we'll end by talking about how to  detect our zero day bypasses. So first let's start with an overview. Starting with  defining and talking about what is background task management. So this is something that was  largely introduced in Mac OS 13. So the most recent version of Mac OS. And what it is, it's a  system designed by Apple on Mac OS to govern and consolidate persistent items or in Apple's  parlance background tasks. So what are persistent items? Persistent items in the context of  this talk are simply processes that have installed themselves in a manner that every time the  system is rebooted or the user relogs in are automatically re-executed by the operating  system. So things like an auto updater that needs to run every day is going to persistently  install itself. Of course malware is going to do the same thing because if malware does not  persist and you reboot your system, you've just disinfected yourself. Now what happens when  an item is installed that persistently installs itself is three things. Three visual things  related to BTM. First, and again this happens on the most recent version of Mac OS, there's  going to be a system alert informing the user that whatever they just installed is  persisting. So an alert to the user saying hey, something just added itself to the  operating system is going to be automatically restarted. Second, there is a security message  that is broadcast that security tools can register for which is a notification, a  programmatic notification saying hey, something just persisted. You can see from a security  point of view, from a security tool, this is something that's very useful. And then finally  that item is going to be registered in a database, the BTM database. So why do we even care  about this? Who cares about this? Well, we do. First and foremost as defenders, we now have  a nicely organized subsystem that manages most persistent items. If we can understand and  tap into that, we can build, and I'll show you exactly how, very powerful security tools  that can now allow us to generically detect persistent malware. This is awesome. As hackers  though, BTM definitely poses some problems. Two reasons. There's going to be an alert  telling the user hey, something just persisted. So if you get on a system and try to persist  your back door, the user will now be alerted. And also as I just mentioned, there's a  security message that's broadcasted so any security tools that installed on the system will  get a notification. These are both brand new events on the latest version of Mac OS. So you  can see as hackers, this is a problem for us. So we need to figure out exactly how to get  around this. Now, BTM is largely undocumented. Apple really doesn't describe a lot. So the  first part of this talk is really going to be diving into the internals of this subsystem. Now,  when I want to understand something on Mac OS, I always turn to the logs. And this is  because a lot of components in the operating system, even undocumented ones, give us a lot of  very informative log messages. Unfortunately on Mac OS, a lot of log messages are  redacted. So what we first have to do is install a logging profile which basically tells  the operating system, hey, please unredact and log everything. And there's a link on the  slide how to do that. It's pretty straightforward but just know if you're going to be  splunking around in the logs, make sure you've kind of turned on this full logging  capability. So now that we have logging off and running, we just install a launch  statement, which is a persistent component. And then look in the logs and see what is  shown. So we can see on the slide there's a few very interesting log messages that are  directly related to the BTM subsystem. And what we're going to do is we're going to take  these log messages and find the binaries that are responsible for generating them and look in  their disassembly at the functions that are broadcasting or logging these messages to  figure out exactly what's going on. So they're great sign posts. So we can see two things  are happening. First, there are two binaries, two processes that appear to be  responsible for BTM. One is called SMD and the other one is called background task  management D. We'll dive into both of these. And based on the method names, we first see  one that's called launch D will scan your path. And then the second one registered  launch item. So it appears that the operating system has noticed that a new item was  persisted. It's going to scan that directory and then register that item. This makes sense.  So let's first talk about SMD. This is the system management demon. And the first thing it  does is we see that it generates a log message saying got a FS event. FS stands for file  system. And if we debug this, we can see that when this log message is called, if we  poke around and look at some of the registers, we can see that one of the registers contains  an XPC dictionary with an event that is related to watch paths. Now if we look at SMD's  property list, which contains information about how it should be launched and some other  activities, we can see it contains key value pairs that tell the operating system, please  invoke me any time something is written, for example, to the launch demon directory. If  you're not familiar with Mac OS, writing something to the launch demon directory is how you  persist an item as a demon. So what this is telling the operating system is, hey, I am  interested any time something is added to this directory. If we dig in a little more, looking  at some of the method names that we saw in the log messages, we see that SMD will generate  a XPC connection to a service called com.apple.background task management. Now XPC is a  Mac specific IPC, inter process communication mechanism that allows one process to talk to  another. To both send messages but also to invoke methods and code in the remote process.  And we'll see BTM utilizes this heavily. Again, if we poke around in the debugger, we can  see that the scan path that SMD is looking at and is going to talk to the other component is  the launch demon directory, which is where our persistent item was just placed. Let's look a  little bit more at these XPC messages to confirm who SMD is sending them to. We can use  Mac OS's LSMP command. This is LS list MP mock port. This will show you who is sending  mock messages and to whom. So we get the PID of background task management, run LSMP and  we can see, yes, there's a bunch of mock messages, XPC messages coming from SMD. And if  we set a break point on the BTM daemon, we can see that the way its methods are invoked are  via this XPC message. So really SMD is just talking to the BTM daemon. So now let's look at  this BTM daemon. This is like the second piece of the puzzle, kind of the core component of  the background task management subsystem. If we look at its property list, we can see it has  various mock interfaces. These are the XPC interfaces that processes such as SMD can  connect and talk to. And then also the path to its binary. Let's take a closer look at its  binary. We can see first that it's going to be running with root privileges. This makes  sense because it needs to monitor system activity, manage these persistent items. If we look  at its entitlements which are capabilities, additional features that the operating system  will give it, we can see it has some related to managing BTM. Also there's an interesting  one and we'll dive into this more showing it has an entitlement allowing it to submit  end point security events. We'll see this is how that security message is generated. If we  look at its launch item registration function, we can see that what it's going to do is once  it gets a message from SMD it's going to look at for example the launch daemon directory and  find the new item that was added. So for example when I install Zoom on my system, Zoom  legitimately installs a launch daemon, a persistent component, we can see in a debugger that  the BTM daemon has now the property list of that new item that was registered. We now have  another player that enters the scene and this is the BTM agent. So if we go back to the  logs we can see that there's a log message from the BTM daemon that says should post  advisory and in this case it is set equal to true. Next we see a log message from a new  process called the background task management agent and we can see a log message saying  posting new advisory notification. So at the bottom of the slide we have kind of now a full  diagrammatic overview where SMD is going to get the initial notification, send an XPC message  to the BTM daemon. The BTM daemon is then going to talk to this new process, the BTM agent,  which is then going to show the alert to the user. So let's briefly look at this agent. It's  the last piece of the BTM puzzle. If we look at its property list we can see it has some mock  and XPC interfaces. Again these are exposed so that processes such as the BTM daemon can  send a message to it. And then also the path to its binary. If we look at its binary we  notice it's running as not root but as the user. This makes sense because the agent is  responsible for displaying the notification on the user's desktop which you have to be  running in the user session in order to do that. In terms of entitlement we can see that it  has various BTM specific entitlements and also then a private Apple entitlement that  allows it to generate system notifications. On Mac OS you need these entitlements for  example to show a system prompt. Okay we're kind of wrapping up the internals of this. Kind  of one of the last pieces. We need to look a little closer at how this message is actually  displayed. So we know that the BTM daemon is going to send a message to the BTM agent and if  we look at the agent's disassembly we can see it has a method aptly named post advisory for  content. And in the bottom I've debugged this and set a break point on this method and we can  see that the argument that it's passed is the method is sorry the message that should be  displayed in the notification. So this is just the internals of how the agent then will  display the message to the user alerting them as to the fact that something was just  persistently installed. Once that has been displayed we see one other log message which  shows that the background task management daemon, it's kind of the core piece, is actually  updating some undocumented proprietary database with what appears to be the item that was  registered. And we'll dig into this more when we dive into writing a tool. But basically what  the BTM daemon is doing is now storing information about the item that was just persisted  into a new database. So that was kind of an overview. We kind of got into the weeds but it's  important again to understand these different components because as we now talk about  writing tools to interact with them we have to understand those and also when we want to  bypass these we're going to have to have this understanding as well. So that's out of the  way. Let's kind of pull it back a little bit and talk first about how we can write some neat  tools to leverage BTM to detect persistent malware. Now why this is such a great idea in my  opinion is because the vast majority of malware is going to persist. Again persistence is a  way to ensure that you are automatically re-executed by the operating system each time the  system is rebooted. So this is something that the vast, vast majority of malware strives to  achieve. I study the Numack malware of each year. It's kind of one of my end of the year  highlights sadly. And I've noticed some interesting trends and this is not surprising to me  but again this just reiterates that most malware is going to persist. So in 2015 which is now  almost ten years ago I was like hey since all Mac malware is persisting I should write a  tool that monitors for these persistent events and then when malware persists regardless of  how it got onto your system via zero day, a trojanized installer, a supply chain attack, like I  might not be able to detect all that but as soon as it persists I can alert the user. So I  released this free open source tool called BlockBlock and BlockBlock does exactly kind of as  the name implies. It will monitor for persistent activity and then alert the user. So this is  what Apple is doing now. It took them ten years or so to realize this is a good idea. I like to  think they were inspired by my tool. But from a security point of view this is great. Now  this is built into the operating system. And to me again this just iterates the fact that Apple  now has also realized that because most Mac malware is going to persist this is a great way to  now alert the user that perhaps something has slipped into their system. Let's look at a  brief example. DazzleSpy, one of my great examples for this because it was deployed via zero  day Safari bugs. So a very powerful deployment mechanism, right? Users in this case were  almost SOL if they inadvertently browsed to the site, clicked on a link, the zero days would  infect their operating systems, their Mac OS systems even if they were fully patched. If we  look at the embedded strings for DazzleSpy we can see something related to a launch agent  which is another way you can persist. And if we run a file monitor and execute the malware,  for example in a virtual machine, we can passively observe it persisting itself as a  launch agent. So again, even if we can't detect the zero day exploitation or prevent that, as  soon as the malware goes to persist, if we are able to detect that, we can then alert the  user that something is amiss and maybe stop the infection before it's gone too far. So the  first idea I had to leverage this new BTM is I want to dump the database because the BTM  database has a nice comprehensive list of all the persistent items. So the idea is let's  write a tool that you can run on a Mac system that says show me everything that's persisted.  Now this will include legitimate software, for example Zoom we saw, but if there is any  persistent malware on your system it will likely also be in this database. The problem is  this database is proprietary and undocumented. Luckily though, Apple ships a tool called  the SFL tool that allows you to interact with this database. And one of the commands it has  is dump BTM and if you execute this, it will nicely dump the database. So here for example  we can see the information about Zoom's persistent component in the BTM database. Now it's  great that there's a command line tool but I want to be able to programmatically do this so I  can add this to my own security tools. So I said okay, well SFL tool obviously knows how to  grab this database, parse it. Let me reverse engineer and figure out how it does this. So if  we reverse engineer it, we can see in a nutshell that it sends an XPC message to the background  task management daemon and the daemon will then send the database, its contents back to the  SFL tool. We can see that also in the log messages with the BTM daemon saying yes I received  an XPC message to get the contents of this database. So I said awesome, let me implement  this same code. Let me talk to the BTM daemon, kindly ask and say hey I also want the  database. So I mimicked the code of SFL tool, compiled and executed this and it completely  failed. Basically we can see a log message saying Patrick you don't have Apple's private  entitlement. Rude, you can't talk to me. Please go away. And we can confirm this if we  reverse engineer the background task management daemon, we can see it has code to check  who is talking to it. And it will only talk to processes that are privately entitled.  Meaning things like SFL tool. I can't get that private Apple entitlement. I can't spoof it. So  BTM daemon is basically going to tell me to take a hike whereas FSL tool is going to be  allowed to make that request and get the contents of the database. So fail. But can we  just access the database directly? Yeah, as long as we have full disk access we can. So the  first thing I said, let me dig into this database to figure out what exactly it is. I ran  the file tool on it. It told me it was an Apple binary property list. We can use the PLUT  command to convert it to a normal property list. But unfortunately that doesn't really help  because it shows us it's just a NS key archived which is a object that has a bunch of other  serialized objects. So really the database has a bunch of undocumented serialized objects. So  not the best format. But SFL tool can somehow deserialize those objects and display them to  the user. So if we reverse engineer SFL tool some more and figure out how those objects are  deserialized, in theory we should be able to write our own deserializer and then read the  database directly ourselves and display this information to the user. So if we look at SFL's  tool we find it's deserialization code which describes the objects that are in this  database. So what we can do is we can essentially reimplement that deserialization  ourselves. I'm not going to go through all of this. But we can see there's things in these  serialized objects such as an executable path, bundle identifier, et cetera, et cetera. We  can then with this deserialization logic complete write some code that opens the database,  loads it into our own keyed archiver and then invoke the D object, decode object of class  method which will invoke our custom deserializer and deserialize all the objects for us. So I  push this logic into an open source library called dump BTM and with this library now you  can write a 5 or 10 line program that simply can interact with the database. The library  exposes two main methods. The first one is dump BTM. You can either specify the path or look  it up for you. This will print it out to standard out or it will load the contents of this  database into a dictionary that then you can programmatically dig into more. So here's an  example, the output of our dump BTM tool. This should look exactly like SFL tool which was  legal. Another cool thing about it is this doesn't need root access to run where Apple's SFL  tool does. So if we execute this we can see we're opening the background task management  database which contains all the persistent items on the system that we're interested in and  starts printing them out. Again we can see for example the zoom daemon that has installed  itself. So what I then did was I took this library and I implemented it into one of my well  known security tools called knock knock. Knock knock's goal is to simply tell you who's there.  Pretty much saying these are all the persistent items and with the addition of this new  library code it can now also query the undocumented BTM database, get all the persistent  items from that and display that. So on a system where Dazzle Spy has infected via the  Safari zero days we can now enumerate its persistence and alert the user that their system is  infected. Now there's one kind of downside to that approach and that is it's somewhat  reactive, right? You're going to get on a system if you're doing some forensic analysis or  seeing if it's infected and you're going to now query and yes you can get a list of  everything that's persisted but it would be nice to get a notification in real time when  something is persistent. That's a little more proactive. So let's talk about how to do that  now. Now you might remember that I mentioned that the background task management daemon has  an entitlement related to end point security. What is end point security? It's another  subsystem in Mac OS that's designed to be, designed to interact with third party security  tools. The idea is the end point security system and we'll dive into this more, basically  monitors all sorts of activity on the system and then third party security clients can tell  it what they are interested in and when that event occurs the end point security subsystem  will programmatically deliver it to them. So for example we will see, we can tell the end  point security subsystem, hey we are interested whenever something persists and then it will  programmatically call us. So this is really great for a security tool to monitor for  example whenever a persistent item is installed. Whether that is zoom or dazzle spy via  zero days. So what we can do is we can look at the header files for the end point security  system and we note that in Mac OS 13, the most recent version when BTM was introduced, there  are some new end point security events that we can now listen for. And the one we're going to  focus on is the BTM launch item ad. As the name implies, this is a message that will be  delivered anytime a new item is persisted. So the idea is an installer or a piece of  malware comes along and persists itself. If our ES client has registered for these  notifications, Apple will deliver that to us. So you can see if you're writing a security  tool, this is great. You can then perhaps examine that item, alert the user, generate a log  message, et cetera, et cetera. So let's look how to do this programmatically. This is some  standard end point security code. This is how you write a client. Basically three steps. You  tell it what events are of interest to you. Here we want to detect when a new persistent item  is added so we tell it we're interested in the BTM launch item ad event. We create a call  back that will be invoked when this event is delivered to us and then we subscribe which  kicks off basically our listener. Now whenever something is persisted, the operating  system will call us because we said hey, please let us know. And here are some additional  information about the contents of that message. So we can see things like the item, the  executable, the path to its property list. Again, these are the components of the message  that are going to be delivered to us. So now if we add this code and compile and execute, we  are now listening essentially for persistence events. If Dazzle Spy comes along and infects  this system, we can now see that we get a notification from the operating system, from the  end point security subsystem telling us hey, just to let you know, a new persistent item was  added. Here is the path to its executable and also the path to its property list. This is  great because if we look at that, first we can see it's an executable software updater  that's running from a hidden directory. Huge red flag. Its property list is also called  com.apple.software but if we look at the binary software update, it's not signed by  Apple. So another huge red flag. So this is an example of now how we can generically detect  even brand new malware that's being deployed via zero days simply by monitoring for  persistence activities. Now one problematic issue, the other end point security event  related to persistence events, the remove events is broken. Like you can register for it  but when a persistent item is removed, it's just never delivered. So Apple, please fix. And  speaking of broken, the BTM launch ad event was actually horribly broken to begin with in  two ways. First, whenever an item was persisted, the operating system would just call you  with all the items that were persisted which is totally unhelpful. I'm only interested in the  new items, not the 25 other items that are already on the system. And then also if you  persisted as a login item which is another persistence mechanism, you wouldn't get a  notification at all. So I sent a bug report to Apple and I said hey, yo, this is like all  broken. They took their happy time but eventually in Mac OS 13.3 they fixed this. Okay,  so now we've talked about the internals of the BTM subsystem. We've showed how we can write  some pretty powerful detection tools that allow us to detect malware when it persists. So  this is great for defenders. But as I mentioned, DEF CON like it's a hacker conference so we  need to like put on our black hats and talk about now how to break this subsystem. And the  reason we want to break this subsystem as hackers are twofold. First, let's put ourselves in  the shoes of a hacker. We come along, we infect a Mac system with our zero days, all well and  good. We go to persist. Two things now happen because of BTM. First, a big system alert is  generated and displayed to the user saying hey, something just persisted. And like if you're  browsing Safari and all of a sudden this pops up, like yeah, you're probably owned. And then  also as we showed, this will generate a end point security message which means that if any  security system is on your Mac, any security tool, it will pick up this event and also be  like yo, strange, you've essentially been hacked. So as hackers we want to bypass both the  security alert and the end point security event. Two things to point out. These are both  separate so we have to find attacks and bypasses for both of them. But the good news is both  of them are independent on the functionality of the persistent item. Meaning regardless of  what happens to the alert and the security event, they're both notifications. So either way  the item will be successfully persisted. It's not like the operating system is waiting for the  user to click on the event and say allow. So this is good. This means if we can suppress these  messages or somehow prevent them from being delivered, we're great. Our malware will be able  to persist invisibly and still function. So one thing you can do, and this is not really a  bypass per se, it's more just a limitation of BTM, is simply persist in a way that is outside  launch daemons, launch agents or login items which are the three persistent mechanism that  BTM currently monitors. These are the most popular ones but there are a myriad of other ways.  For example, you can persist as a cron job and currently BTM does not monitor cron jobs.  There's also a great blog post by Chaba who basically shows that there's 20 or so more  esoteric and stealthy ways to persist. So if you persist your malware in any of those ways,  again as long as you're not a launch daemon, a launch agent or a login item, BTM doesn't even  come into play. But I wanted to directly bypass that system alert. So method one is  hilariously simple. If we go back to that SFL tool and look at what it supports, one of its  commands is the reset BTM database. And the idea is if your database gets into a funky  state, you can reset it. The operating system will then just rescan all the launch daemon  agent and login items directory and rebuild the database. So that's great. The problem is  when you execute that, it puts the background task management daemon in like a funky  confused state. So if we look at the log messages, what we can first do is reset the  database and then persist an item. And if you look at the log messages, you can see that  should post advisory is now set to false. I don't know exactly why, but I know that if you  reset the database, this will always be the case. So what a hacker can now do, granted with  root privileges, is just reset the database, persist their item, their item will be  successfully persisted and restart every time the system is rebooted, but the alert will  never be delivered. And again, they're simply using Apple's own tool. So this isn't some  crazy like heap overflow. You're basically just saying, hey, please reset the database. And  the system's like, cool. And then you can persist silently. So that's awesome. But the problem  with that is you need root access. So I said, okay, let's do one better. Remember the  BTM daemon will send an XPC notification to the BTM agent and say, please display a  notification to the user. What privilege level is the agent running at? User. And this is  because it needs to display a message on the user's desktop session. So I said, can we just  send the BTM agent a stop signal? Like, yo, go to sleep for a minute. What happens? Well, it  turns out you can do this. The BTM daemon will still deliver the message to the BTM agent and  believe that was successfully delivered. But since the agent is sleeping, it's not really  going to get that message and so it will never be displayed. So what a hacker can do is get  the process ID of the BTM agent, which again is running as user, so lowest privilege level,  send it a stop signal to put it to sleep. Go to persist. BTM daemon will try to deliver the  message and think it succeeded, but the agent is sleeping. And then once they have persisted,  kill the agent, which again you can do because it's running with your same privilege level. You  need to kill the agent because otherwise when you wake it up, the alert will have been  queued and will be displayed. But if you do this, you will now be able to persist and the alert  to the user will never be displayed. Now this is well and good. I'm like, cool, like these  techniques work on the latest version of Mac OS, so hackers take note. I guess Apple take  note too. But I'm like, cool, users might ignore that alert anyways. What I'm really more  interested in is preventing the end point security message because security tools,  including my own, listen for this method. This is how Apple says register for persistence  notification. And you can imagine security tools are not going to necessarily ignore this. A  user might ignore an alert. A security tool, if it gets a persistence event, is going to  examine that very closely. So again as adversaries, hackers, we want to figure out how to  prevent that message from being delivered. And it's a little harder because this is all being  delivered in the root session. So I went back to the BTM daemon and started looking around at  how it submits events to the end point security system when it detects something has  persisted. And unfortunately if we follow the call chain, we can see it calls into the  kernel, which means we're going to have to pop into kernel. However, right below that kernel  call, we can see a log message that says failed to submit event. So I'm like, okay, well, it  looks like there's a failure scenario. I have no idea what that is. But if we can figure out  what that failure scenario is and trigger that, we might be able to prevent the end point  security message from being delivered to any security tool on the system. Which is the  goal. So we pop into the kernel and there's kind of a lot going on in the slide, kind of  follow the black arrows. We can basically see that in the end point security kernel  extension. So this is running in kernel mode. We find a bunch of methods. One is called PDB  read audit token. This reads a process ID from a user mode buffer and then calls another  method create from PID version. And this invokes the proc find API. Given a PID. This is  going to be important. Then it calls another method to get the executable V node. This is  kind of like the file system V node of the process it's just looked up. So basically it's  trying to get a path to a process. What process? Well, if we set a break point in kernel  mode and install something, for example, oversight which is one of my utilities that  installs a launch, a login item, we can see that the process that it's looking up is the  responsible process that has persisted the item. And we can confirm that if we look at the  end point security message in user mode. We can see that the PID and the process match. So  again, what the end point security kernel extension is trying to do is look up the  process that is responsible for generating the persistent event. The installer, the malware,  et cetera, et cetera. This makes sense because they want to deliver that to the security  tool in user mode. So I looked at that a little more and I thought, okay, what happens if the  responsible process is like gone? What if it executes really quickly, right? This is kind  of happening separately. So maybe it will get confused. And so what happens, we can see in  this case, they call proc find with null. They don't have a PID because the process that  persisted, at least I thought, was gone. Now, if you call proc find with null, which is also  zero, you get back a process object but it's the one belonging to the kernel task because the  kernel task is assigned PID zero. It's not really a process per se but this is just how it  works on Mac OS. And that's okay. I mean, they kind of have the wrong process at this point.  But it's problematic then when they call proc get executable V node because the kernel  doesn't have an executable V node. It's not a running process backed by a binary per se. So  that is going to return null which causes them to return error number 3. And if we go back  to the log messages in user mode, we can see a log message that says fail to submit ES  sys call. Error 3 no such process. So I was like, okay, I think I can trigger this but how?  Like how do I develop a proof of concept? And I thought about it and I said, oh, wait a  minute. My own tool block block uses the CP command to install a property list. It copies  that. And that's a very quick command that exits. So I was like, is this going to trigger  this kernel bug and bypass that? So let's do a demo. I haven't prayed to the demo god so  fingers crossed this all worked. But go big or go home. The first thing we're going to do,  I'm going to make this a little bigger, is we're going to execute Apple's ES logger command.  And this is going to, this is a utility, an end point security client that allows you to  listen for end point security events. And we're saying please listen for the BTM launch add  item. So in theory, if an item is persisted, we should get notification. We're then going to  have some log messages running. This is just going to show us the error message. And then  we're going to install block block. Fingers crossed. Put in my password. This is definitely  going to work. Now we do get an alert. We'll address that in a second. These alerts are  again separate from the end point security subsystem. If we go over here though, we see  there's no message. There's supposed to be a message. Why isn't there a message? Well if we  go back to here and look for lib end point, we can see that we have successfully triggered  this kernel bug. Failed to submit event. Syscall failed. Error no such process. And this is  because block block again is a script that executes shell commands which start and stop very  quickly. So the kernel doesn't have time to look that up or isn't tracking those separately.  When it goes to look that up, that process is already gone. And so it doesn't have a PID and so  it errors out. So basically all you have to do to bypass the security events is to do that.  Okay. Let's, I guess we have some time. Let's show how we can block these messages as well.  Yeah. Awesome. Thank you. So I'm going to uninstall block block. Click through this. Just to  kind of reset everything. Uninstall. And we're going to run it again and we're going to this  time try to hide those alerts. So I remember I told you the launch agent, the background, the  BTM agent is responsible for displaying those alerts. And so what we can do is just tell it to  go to sleep. So let's do that exactly. So what we're going to do first over here, I'll make  this a little bigger, is we're just going to get the process ID of the background task  management agent. We can see it has a PID of 86276. It's also running as me as the normal  user. So what we can do is we can do this programmatically or we can just do it via the  kill. We're going to send it the kill stop command which is going to say, hey, go to sleep.  And we can confirm that if we do the process listing again, it now has a T here. T means that  item is now suspended. If we, and I'm going to reset this because the uninstaller for block  block installs something persistently but does it programmatically so this generates an  event. This just proves that normally we should get an event. So I'm going to reset this to  show that the installer doesn't. We're now going to run block block again and the hope is  we A, don't get a security event and B, no, no alert is shown. So fingers crossed. Live  demo number two. Block block is going to install. Hooray. Happy. We're going to get an  alert from block block saying I'm up and running but no alerts from the end point security  system and no alerts from the subsystem. So now what we would do is we would kill the agent to  make sure that those messages are not delivered. What I'm going to do instead, resume it to  show you that if we didn't do those, those would actually be there. So again, normally a  hacker wouldn't like continue the agent. They would kill it at this point which is again  because it's running with the same privileges. But if we continue it, we will see those are  the messages. So they were delivered but because the agent was sleeping, they weren't  actually shown and if we kill it, they will never be displayed. So what we have done is fully  bypassed background task management which is I think kind of problematic. And why is this  problematic? Let me just clear these annoying alerts. It's really problematic because this  is a security mechanism that Apple has basically told us will protect our Macs but in  reality it's implemented incredibly poorly and is trivial to bypass. And this is  problematic because users might now be over confident in the security of their system  because Apple has promised them this security feature. And security tool vendors might also  be leveraging this to detect persistence believing it works when in reality it's broken  and trivial to bypass. So that in my opinion is problematic. So let's end kind of on a more  positive note and show how we can detect these bypasses until Apple gets their act  together and actually starts shipping well secure implementations. So the first thing we  want to do is how do we detect if someone resets the SFL or the BTM database. This is not  something that should ever normally happen, right? It's kind of an undocumented database. No  one really knows about the SFL tool. So if you see someone resetting the database, it's  like probably a hacker trying to leverage one of these bypasses. So what we can do is we can  write a process monitor that can detect when this process is spawned and also look at the  arguments. We can write a process monitor that's either passive that can say hey this  happened or using other end point security events you can actually receive a notification  when a process is about to be spawned and if it's the SFL tool with the dump BTM command you  can actually prevent, sorry with the reset database command you can actually block that  altogether. So this is a way to essentially prevent that command from being run so that  database cannot be reset. What about telling the BTM agent to go to sleep? Well it turns out  we can also use the end point security system to register for signals. Specifically we can  use the event type off signal which will be delivered to us any time a process is trying to  send a signal to another process. What we can then do is we can examine both the signal and  the source and destination process and if the destination process is the background task  management agent and the signal that is about to be sent to it is a stop event to put it to  sleep we can tell the operating system do not allow that to happen. So if we implement this  code and are running this in our security tool you can now see that when the attacker tries  to send that sig stop message to tell the BTM agent to go to sleep our security tool will deny  that and that will actually fail. So now we've prevented the agent from going to sleep.  Again these are things like Apple should be doing. It's like lame that we have to do this but  it is what it is, right? In terms of getting the end point security notification for  persistence this one's a little tougher because it's a kernel flaw. So kind of hard for us to  detect. Well good news is there are other end point security alerts or messages that will also  be delivered that are not susceptible to this bug. For example when a piece of malware  writes a property list to the launch daemon directory to persist, yes in theory this is  supposed to generate a BTM launch item add and will in some cases but we showed how to  trivially bypass this. This will always also generate file notifications. For example  notify create, notify write that will be delivered to us and are not susceptible to the  kernel flaw which allows us then to say hey something was written to those directories. So  what we can do and this is how block block detects persistence is it doesn't use the BTM  launch item add because it's broken currently. I wish it worked. It would be really nice.  But instead it uses the file events to monitor the locations where items can persist. As I  mentioned those events will always be delivered accurately and correctly. So it listens for  those events. If it turns out it's in one of those directories it then alerts the user. The  other thing is unlike Apple's approach it has the ability to block that so you can  actually take some action against that versus just clicking on an alert that's like yeah  cool. Okay. So let's wrap this all up with some takeaways. So we talked about  understanding macOS's BTM subsystem. I think it was important to understand that because  we want to write tools and also understand limitations. So once we had an understanding of  the internals of the background task management we showed how to write some very powerful  malware detection tools that are built on the fact that Mac malware persists and BTM now  manages and governs persistence. Unfortunately we showed that it's trivial to bypass the  BTM subsystem entirely. Again all these approaches worked on all the latest versions of  macOS so go out and cause chaos I guess. No don't. But you know again one of the other  takeaways is like if we're not doing this research who is? Like what is Apple doing? Like  counting their trillions in Cupertino? I don't know but it's really problematic when they  say hey there's this great new security system that's built into macOS. You now are  protected against persistent malware. Well in reality like that's really not true. And again  as I mentioned that's problematic because users are going to have a false sense of security  which is bad. Makes them easier to victimize. And then also security tools might be thinking  they're doing their jobs when in reality they're not at no fault of their own. So again  maybe anything Apple says take with a huge grain of salt and continue to keep poking on  macOS because even new code that Apple releases is well I would say subpar. Now if you're  interested in learning more about Mac malware, reverse engineering, writing security  tools, I've written some free books on the topic. Volume 1 is out. Go to TAOMM.org 100% free  online. I'm also going to be doing a book signing today right after this talk at the No  Starts Press so if you want to pick up Volume 1, get my signature, check it out. I'm also  writing Volume 2 which covers basically the topics we talked about today which are the  programmatic approaches to detecting malware. I'll end by just mentioning briefly again the  non-profit I'm part of because I think we're doing a lot of really cool things that I think  many of you will be interested in. We organized the OBTS security conference. I see a lot of  friends here from that conference so thank you all. We also raised money for college  scholarships for students who are going into technology fields and we have various  diversity programs as well. Our current effort though is raising money for the wild  fowls in Maui. Many of you know I live there. A lot of my friends had everything lost so we  are working to raise money for that. So something very personal. If you feel like you want  to support that, it's awesome or even just share this. It's something that would be very  meaningful to those who lost their houses and their loved ones. So speaking of thank you,  I also want to briefly just end by thanking the companies who support the non-profit. We're  able to organize this great conference. I'm able to release these tools, these free books,  give these talks thanks to them. Really couldn't do that without them so I briefly want to  thank them all and also thank you all for spending this rather early Saturday morning with  me talking nerdy about Mac OS stuff. So thank you again to you all.  Applause.


url: https://www.youtube.com/watch?v=Kb1lKscAMDQ
title: DEF CON 31 - The Internals of Veilid, a New Decentralized Application Framework - DilDog, Medus4
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=Kb1lKscAMDQ",
    "title": "DEF CON 31 - The Internals of Veilid, a New Decentralized Application Framework - DilDog, Medus4",
    "description": "Veilid is an open-source, peer-to-peer, mobile-first networked application framework, with a flagship secure messaging application named VeilidChat. Veilid is conceptually similar to IPFS + Tor, but faster and designed from the ground-up to provide all services over a privately routed network. The network also enables development of distributed applications without a 'blockchain' or a 'transactional layer' at their base. Veilid can be included as part of user-facing applications or run as a standalone server for power users who wish to help build the network.\n\nArchitecturally, it is written in Rust, uses strong encryption, and nodes can run on Linux, Mac, Windows, Android, iOS, and in-browser WASM. Low-level protocols over UDP, raw TCP, Websockets and Secure Websockets. Nodes are optimized for low latency, high node churn, and are particularly capable of dealing with low level network changes, such as switching from cellular to wifi networks mid-communication.\n\nThis talk will focus on the internals of Veilid:\n* How it works as a protocol\n* How it leverages strong cryptography to provide private communications\n* How it provides decentralized storage and cryptographically sound data structures\n* How applications are written to leverage the Veilid Network\n\nWe will demonstrate Veilid Server, and VeilidChat, the application.",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2476,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230901"
  }

  This text was transcribed using whisper model: large-v2

   Hi, my name is Christian Dildog-Ryu and this is Caitlin Medusfor Bowden.  We are members of the Cult of the Dead Cow and here to present to you an unveiling of Veiled.  We've been pumping this thing up for six months online.  A lot of interesting stories about what it could and couldn't be.  We're just ready to pull back the curtain a bit and talk about what is Veiled.  So today we are about to drop some fantastic code.  We are about to drop some really cool apps and things and we are about to make some cool announcements.  But I think above all, I really hope that everybody understands that the biggest thing we're doing here is presenting an idea.  And that idea is that some things are just worth doing.  Even if it's not monetary.  I think we all found ourselves in hacking for one reason or another.  And that was to make a difference.  To do something that's worthwhile.  If you wanted to go make a bunch of money, you'd be over at Black Hat right now.  But you're here.  Okay?  So we want to inspire all of you to hack the planet with us today.  Awesome. Thank you.  So in short, the Veiled mission, we exist to develop, distribute and maintain a privacy-focused communication platform and protocol for the purposes of defending human and civil rights.  There's a big mission there.  We spent a lot of time thinking about this thing before we started writing code on it.  We want to affect a particular outcome on the world.  We're not interested in just writing cool code for the sake of it.  We want to rethink the way applications are built so that the things people take for granted about application development change.  You don't have to monetize everything.  You don't have to sell users out just to make your cloud bill get paid.  That kind of future is the future we should have had.  So let's get into a little bit.  Others have come before us.  We had some inspiration from other tools.  You know, we looked at the world.  We said, you know, we want to build things that are better.  What tools exist already today?  And Tor was there, front and center.  I remember hanging out with Roger and Nick early in their development days and seeing how they had a similar mission to make things private that just weren't back in the day.  I looked at IPFS, which is pretty novel in its use of distributed hash tables and all these crazy technologies to make interplanetary file systems.  I mean, that just sounds huge, doesn't it?  And there's a lot of other efforts in this space.  Peer-to-peer has been bounced around for 20 years since Napster and BitTorrent and everything else.  We are at a time now where everyone has a supercomputer in their pocket.  You've got a terabyte of storage.  And it begs the question, what if the cloud was everyone's computer, not just Jeff Bezos's?  That's right.  Valid is an open-source, peer-to-peer, mobile-first, networked application framework.  What this means is that it's like Tor and IPFS had sex and produced this thing.  Imagine a world where your apps can talk directly to each other, bypassing all the middlemen.  Not taking storage in the cloud or requiring some central authority for things like identity and storage or whatever.  I looked at the Web 3.0 people and the things that are coming down the pipe.  There's been a lot of talk about decentralization lately.  You've seen Mastodon and Blue Sky and these things.  We're moving from centralized networks down to more feudal control.  This is sort of the next logical step toward individual nodes all being peers in the same level.  No hierarchy.  How do you affect that? How do you build that? How do you make it work?  Here's what we're going to do.  We put some design goals together for this project.  Obviously, security was first.  If you're going to be asking people to do something new with their applications or mobile devices,  you have to be thinking like an attacker.  How would you tear this apart?  Security first. We wrote it in Rust.  It is now over 100,000 lines of Rust.  Lovingly hand-coded.  Built for memory and type safety.  It runs everywhere.  Rust is extremely portable.  We have a core that runs on Linux, Mac, Windows, Android, iOS and in browser WASM today that works.  We use standard internet protocols.  You can think of this as an overlay network.  We're not going all the way down and writing a kernel.  We're also not building on top of things like blockchains and transactional protocols and all this.  We use UDP, TCP and WebSockets because these are the tools that are rock solid and available on all these platforms.  It's all in network.  We don't have external services.  We don't rely on Google's stun servers to do hole punching.  We avoid DNS because that's a giant information leak.  There's a tiny little bit of DNS but we'll explain why that doesn't matter.  We have all of the services that make Valid possible inside the Valid network itself.  It's self-contained.  You just link it in. Everything is working.  All nodes are equal.  Privacy focus was really important.  When I looked at IPFS, it was not designed with privacy in mind.  Tor obviously was designed with privacy in mind but it wasn't designed with performance in mind.  It was more about proxying and it really enabled websites to be reached anonymously.  That's not a bidirectional privacy when the NSA runs 100 exit nodes.  What if there were no exit nodes and in fact there were tens of millions of nodes?  It would be really hard to monitor tens of millions of nodes instead of a couple of exit nodes.  Privacy focus from the beginning and keeping nodes and identities distinct.  I'm going to get to why that's important and how we do it.  Resiliency matters.  Mobile networks, if you're going to be mobile first, you have to be able to handle networks that come and go.  Driving through the tunnel, losing your cell connection, coming onto a WiFi, all of that.  It's got to be able to context switch real fast.  We thought about all these things early on and we had to write code to support it.  Feel free to cover this.  When we built this, when Dill first came to me with this idea, he had the tech in his head.  But what it really needed, and I feel that this is really important and something that is often overlooked,  is the human side of things.  What is the point of building these amazing technological fantastic privacy things like Tor  if no one except for the people in this room understand how to use it?  How does that benefit our parents?  How does that benefit the people that might not be as comfortable with the internet?  What we needed to do was really focus on what the user needs.  How to make this approachable.  Because frankly, everybody deserves privacy.  Everybody.  Okay?  When it came...  Not everything does need to be centralized.  We've seen the problems with that.  Human beings are flawed and they are going to put some of themselves into anything that they have control over.  We've seen that with Mark Zuckerberg.  We've seen that with Elon Musk.  To make sure that this is something that no human imperfections can destroy.  There's not going to be an algorithm.  There's not going to be somebody pushing their ideals on you.  This entire network is going to be what we all need it to be.  Alright.  So let's start getting into the nitty gritty of how this works.  All Valid Nodes are applications running the Valid Core.  This means that no nodes are special.  Okay?  There are no special nodes.  You don't have to have an admin running servers or any kind of special...  Everyone's running exactly the same thing.  In fact, every application that is a Valid application looks the same to each other.  So the network gains strength from a single popular app.  Even for the less popular apps, the water level is raised by that one app that's very popular.  So you don't have to build a really popular app and build your own Valid network.  You can if you want.  But the big Valid network is supported by the entire ecosystem.  Not just your app sailing away with a sea of eight users.  So there's basically two kinds of uses today for the Valid Core.  Applications that are intended for end users.  And headless nodes that could be used as a way to contribute to the network.  If any of you guys that are Linux nerds like I am want to take the...  What we're calling Valid Server.  It's not aptly named because it's not really a server per se.  It's just Valid without any app.  It's just running Valid.  Runs the RPC mechanism.  Passes data around.  If you want to help the network out, the way you do it is by running that.  And just let it sit there.  You don't have to do anything.  You're passing encrypted traffic around and storing encrypted things to disk.  And that's it.  Today we have bindings for different languages.  We're going to get to that.  Again, the protocols are all low level.  So UDP and TCP.  Programmers that are writing apps can choose whether they want to...  Talk over a more reliable sequenced protocol.  Or if they would rather just have something be fast and unreliable.  So...  Alright, let's kind of go through.  I'm going to spend a little time on this slide.  But let's go through what happens when you turn a node on.  There is a bootstrap mechanism.  So this is the only time that we're going to use DNS.  And that is when you turn your node on.  It will make a few text message requests to bootstrap.valid.com.  This pulls down some public keys.  And some dial information.  I'll get into what a dial info is exactly.  But you can think of it as an IP address.  With a little more augmentation.  This gets you access to a few other valid nodes.  It just says, oh by the way, the network is over here.  And here is a public key that you can use to make sure that you're talking to the right nodes.  You contact those nodes.  And download a chunk of routing table.  And the valid server stores those routes.  And can then proceed to query all the other nodes in the routing table.  To find more routes.  And it builds this routing table up.  And applies a distance metric.  Which is an XOR of a public key.  That is assigned to each node.  The node generates the public key, private key pair.  Nodes are identified by a public key.  And the distance in the graph of nodes is measured by the XOR.  It's sort of like if you were to subtract the nodes.  Addresses.  But XOR has that property where it doesn't matter.  It's commutative.  So we use that for a nodal distance.  And that's important later on.  All of the nodes do the same thing.  So desktop apps reach out to Bootstrap.  They download the routing table.  The next time you launch it, it doesn't try to Bootstrap again.  It tries to reach out to the previous nodes that it has talked to before.  So Bootstrap operation is very lightweight.  And assuming that you've talked to some nodes that are going to be long lived.  It probably doesn't need to happen with every single time you switch into a Valid app.  Again, there's mobile apps and progressive web apps.  And they all have slightly different network characteristics.  But the first things that are going to happen is we're going to ask those Bootstraps to find themselves.  And when I mean that, you're going to be finding nodes that are close to the Bootstraps.  And then close to yourself.  We're going to be doing a NAT detection operation.  Which is somewhat like STUN.  Have you ever heard of STUN?  It's a detection of your public IP address.  But also kind of tells you how you can be reached.  It's a way of calculating what kind of NAT you're on.  So your network address translation is like you're either fully firewalled.  Or you have a full cone NAT.  Or Porter Address Restricted NAT.  Or Symmetric NAT.  Or you're on one of those weird carrier grade phone networks.  Where every place that you go gives you a different IP address.  Makes it a little more complicated.  But Valid nodes need to understand where they stand in the world.  They need to understand how they could be reached from the outside.  And if they need any help.  Achieving connectivity.  So that operation happens very quickly at the beginning.  About five seconds after a node starts up.  It's figured out it's public IP address.  From other nodes that it knows about.  And calculates it's dial info.  The dial info says.  If I publish this to you.  You'll be able to use this information to be able to reach me.  Somehow.  So basically we sign this information.  With that same public key for the node.  That way when you pass it around.  People can't forge your dial info.  And redirect your traffic.  By claiming that your dial info goes to somewhere else.  No other overlay networks that I know of.  Actually use signing on the routing table itself.  They pass around a lot of information.  But nodes are just trusted to not modify it in transit.  That doesn't really make sense to me.  So all of the node information is signed in Valid.  So yes.  Network class detection.  Determines if you're inbound capable or not.  You know the symmetric NAT.  You just can't be inbound capable without help.  Relays are used in that case.  Every node in Valid is expected to relay encrypted traffic.  To the best of its ability.  To help out other nodes.  So if a node is a cell phone.  On a cell network.  Going through a tunnel.  And it just can't receive inbound traffic.  It's going to use a relay from the network.  To help get inbound traffic.  It will make an outbound connection.  And that relay's IP address.  Is going to be part of its dial info.  As signed and published.  So that way.  You don't have to relay forever.  Relays can start rejecting stuff.  And then the nodes will automatically pick a new one.  So.  After that.  You just have to know whether or not nodes are alive.  Whether or not they've continued to.  Pass packets.  If they're still there.  And we have an exponential back off mechanism.  Of pinging those nodes over the same Valid protocol.  To determine if nodes are still alive.  In the routing table.  This turns out to be a statistical effort.  Almost entirely.  You're really trying to understand.  The performance.  Characteristics of other nodes.  And how.  Alive they are.  Should I talk to them?  Can I trust them to keep being there?  Turns out that like.  If a node's been around for five minutes.  Chances are it's going to be around for another five.  If a node's been around for half an hour.  It's going to be around for another half an hour.  It's just the math behind it sort of.  If a node's only been around for a couple seconds.  You probably shouldn't be relying on it.  For say your private routing.  So we keep this math.  And we do it on a per node basis.  So.  All right.  So what we're here to do with y'all.  Is to help you.  You know inspire you to build stuff on this.  I'm sure you can imagine.  After hearing all of that.  The many many ways that this.  You know can be used and built upon.  And I'm sure everyone is sitting there noodling.  Over what projects could be built with this.  You know whether it's a peer-to-peer messenger.  Whether it's social media.  The possibilities here are endless.  And we really want to give you the tools to do that.  All devices are going to be welcome.  And treated fairly.  It's all equal.  And then.  We're only as strong.  As the weakest node.  Like every node is equal.  And I think that's pretty cool.  And yeah.  You're welcome to use the.  Balint network on its own.  Or you can just build on it.  And we really hope that everybody takes the opportunity.  Okay.  So let's get into some deeper stuff.  What is this cryptography.  That enables what we're doing here.  Devils in the details.  And I know I've already had a ton of lovely.  Contrarian conversations with all of you.  At some point.  How does this actually function.  Okay.  Well we have a cryptography suite.  That we have blessed for this project.  And we're calling it VLD zero.  I'll give you a hint.  If we have to change it.  It would be VLD one.  We have a set of tools.  That we've vetted.  We did not enroll our own crypto.  We know better than that.  But we're hoping to put things together.  That are known good.  Authentication is done.  With ed25519.  On curve 25519.  So we have a very fast.  Public private key authentication.  That we can do.  So we have a very fast.  Public private key authentication.  Signing system.  All node IDs are effectively.  Ed25519 nodes.  Key exchange.  Is done with the DH.  That is.  Came on curve 25519.  Shortly after ed25519.  Was made.  And.  Yeah.  So the same nodes.  Can also be used to generate.  A symmetric key between nodes.  So everything that.  If you have somebody's public key.  And you have your own secret key.  You can DH those to produce the.  A shared secret.  Once you have that.  You have to have a stream cipher.  Or something to.  Encrypt with that in a.  In an appropriate way.  That is.  Invalid.  We're using.  The extended nonce version of.  It is an associated.  Encryption.  Associated data encryption system.  So you can do things like.  Partially encrypt a packet.  Include a header.  In the signing part.  And have the the signing identity.  Is part of the encryption.  All in one go.  You don't have to worry about.  Do I sign first or do I hash Mac.  And all this other stuff.  No it's all just built in.  You don't have to worry about it.  Transfer of information.  You can just use.  That that mechanism.  And then.  Yeah.  So message digested.  Being done with the Blake three.  Which again is a fast.  Fast hash is extremely.  Extremely fast.  Everything in this key space.  By the way is 256 bit.  So all of these algorithms here.  Are tuned for a 256 bit key space.  And key derivation is done with.  The 2015 password.  Hashing competition.  And that is a very slow hash.  Intended to slow down and stymie.  GPU based attacks on passwords.  So all of that is in the VLD zero suite.  For people to use.  As part of our framework.  If we have to change things though.  Valid has been built with.  An upgradable crypto system from the beginning.  We recognized.  That cryptography is going to fail us.  Through the advancement of technology.  CPU speeds.  GPU attacks whatever it is.  Differential this that the other things.  Stuff falls apart.  We know that we'll have to change.  Some of this down the line.  So we built in the ability to upgrade.  The cryptography from day one.  You don't have to understand necessarily.  Every cryptographic decision that's being made.  Every new version of our crypto system.  Is supported alongside the old ones.  So we can.  Add a new crypto system.  Wait for it to become popular.  And then deprecate the old one.  So you know which crypto system was used.  With the key.  We have migration support built in.  So if data was persisted.  With an old crypto system.  It can be automatically re-encrypted.  With the new crypto system.  And we can use again.  Simultaneous crypto systems.  To do transitional.  Have both of them alive at the same time.  This enables a bunch of different.  Kinds of storage.  Valid has secure storage.  We have.  Completely cross platform support.  For device level secrets.  So we're using iOS and macOS key chains.  We have the Android key store.  We have Windows protected store.  Linux secret service.  We have a table store.  Which is an encrypted key value database.  So nothing needs to be stored.  At all on the file system.  Without being encrypted.  Keep all of those private keys.  In the protected store.  Encrypt everything to the table store.  And then that enables.  The actual record store.  And block store.  Which are distributed hash table.  And the content addressable storage.  So on the wire.  Everything.  All these protocols use the same encryption.  Everything is encrypted and signed.  Everything is time stamped.  And all of the node information is signed.  I mean it's very little of what we do.  Has an opportunity to be spoofed.  And it really wouldn't matter if you did.  So.  The reason that I am up here.  Is because at one point.  Somebody got into my ex's phone.  Took some pictures out of it.  That I didn't want to be shared.  And they shared it online.  Because my ex didn't understand.  The importance of his own privacy.  He didn't understand how important it was.  To keep his phone locked.  That is one reason that we really wanted to push.  That all of our storage is encrypted at rest.  It's going to always be safe.  No matter what.  Unless they actually know.  How to get into everything.  It doesn't matter.  It's going to be safe.  And that is the point.  We want to really address the fact.  That everybody.  We want everyone to understand.  The value of their privacy.  That's part of this.  Is to understand what you are giving away.  And what you are trading.  In order to use a lot of these websites.  Your data is protected.  Even if you lose your device.  It's going to be safe.  And things like what happened to me.  Don't happen to other people.  Applause.  And I know I said.  I wasn't going to drop any zero days or anything.  But you really should do an iPhone backup.  Of a lot of your popular encrypted messenger apps.  And take a look at what stuff is actually encrypted on disk.  Because you might be surprised.  About that.  Okay.  RPC protocols.  Okay.  So between nodes.  How are they talking to each other?  Time check.  I'm doing all right.  So how are they talking to each other?  What is on the wire exactly?  Well first off everything is encrypted.  So you run a sniffer.  You'll see chatter.  Encrypted small chunks of stuff.  Going over a bunch of protocols.  I picked port 5150 as the default.  If you kind of want to smell around for it.  But other ports get used if necessary.  You can just use it on the airplane.  Without paying for that.  Anyway.  Applause.  So the schema language.  That we're using is a in schema.  Upgradable.  Documented.  RPC called Captain Proto.  Captain Proto is written by.  The same guy who started the.  Proto buff.  Project at Google.  But it was meant to fix a lot of the.  Schema upgrade problems.  And produce a.  Zero.  Zero copy deserialization system.  So we're talking about something.  That uses the same on.  You know decrypted on the line.  Message format.  As the same as the in memory format.  That way there's no marshaling.  And like moving shit around.  To slow things down.  The RPC itself.  Fully supports private routing.  And we're going to get to that.  And it has schema evolution built in.  So.  I'm going to skip over a couple slides.  Because we have about 15 minutes left.  But effectively.  The RPC operations are a nested.  Set of our.  Captain Proto definitions.  That define a series of operations.  Like pinging and status.  Finding nodes.  And then doing things like.  Distributed hash table.  Value gets and value sets.  And again some RPCs are unidirectional.  Some are bidirectional.  Some are distributed hash tables.  Distributed hash tables.  Get a lot of press.  Because they were like the backbone.  Of a lot of the file sharing tools.  They're used in things like.  Bit torrent.  And old school Nutella.  If you wanted to download executable files.  That said they were MP3s.  DHT is basically just search though.  And the valid DHT.  Is a better DHT.  Because it enforces.  Data locality.  Related data.  Effectively all the DHTs.  That you're going to see out there.  For like IPFS.  Store little tiny bits of stuff.  Very distributed across the.  The hash space.  And what ours does.  Is it actually lets you store.  A bigger chunk of stuff.  We're talking like a megabyte per key.  Across an array of sub keys.  That can be allocated.  On a per writer basis.  And what does that mean?  It's where the public key.  It represents its location in the hash.  And the private key.  Lets you assign the data that's pushed.  To the keys value.  No other DHTs right now.  Let there be multiple writers.  To sub chunks of those keys.  You can build some pretty powerful.  Distributed synchronization.  And data structural paradigms there.  If you can allow people.  To commingle data.  On the same hash key.  That means that you know.  You have a conversation.  You can put it in line in one place.  And have you know.  Sure private routed.  But you can have the data.  Located in the same place.  That allows for that sort of.  Caching to happen.  And for related data.  To be all together in one place.  It means you don't have to search.  The whole network every time.  You need to find a new thing.  This makes our DHT faster.  It's not just about.  Using you know blockchain or a coin.  To me using a coin.  Is an accessibility issue.  It's making sure that people.  That aren't as comfortable with the internet.  Are not going to be using it.  It also is putting a price.  On something that I feel is priceless.  You shouldn't have to pay for privacy.  It should not be behind a paywall.  Privacy should be free accessible.  And available to everybody from the start.  And that is it.  One of the things that we.  Really built in with Valid.  Was to make sure that we are building it.  For the people in this room.  We are building it for the people.  Outside of this room.  We are building it for the people.  That are still on Facebook.  Because that is the only you know app.  That has made itself accessible to them.  We want to build this for the people.  That you know.  Don't want to see the privacy.  Like what goes on behind the curtains.  You click and things are going great.  And that is what they wanted.  They don't want to wait.  And the one thing that I really love.  About what we are doing here.  Is that we are making sure it is seamless.  Privacy is there.  But it is just the baseline.  And that is how it should be.  Applause.  And we can get into.  Why this is not a free lunch.  For infinite storage.  There is some tricks to it.  But generally speaking.  It is free to store a whole lot of stuff.  And the network will support you.  So talk about private routing.  How is this sort of like Tor.  How is this protecting people's IP addresses.  If I launch my valid powered messaging application.  How is it that it maintains its speed.  And keeps people's IP addresses private.  Okay.  I am going to define some terms.  There is a safety route.  A private route.  And a compiled route.  This is a little different than Tor.  Tor has things like guard nodes.  And entry and exit nodes.  If you are familiar with those.  Safety route is sort of like a guard node.  If you are thinking in terms of Tor terminology.  So the idea is that.  If you have two people talking.  If only one end of the conversation.  Can control the entire route.  Then you have lost.  They can pick nodes that are adversarial.  Or phoning home with everything.  And that's no good.  Right?  If you want to truly be able to.  You know.  Keep yourself private.  Neither node needs to.  You should be able to have to trust.  The construction of the route.  So what we do is we build our routes.  In two chunks.  The sender allocates a safety route.  For sender privacy.  And the receiver publishes a private route.  For receiver privacy.  Those are compiled together.  To build the circuit.  That the communication happens over.  And the nodes in the middle.  Don't understand necessarily where.  The end of the route is.  So you have to assume that.  If you look at A, N8 and N7.  A is choosing those nodes.  And N6, N5 and B.  B is choosing those nodes.  But when they snap together.  The nodes don't know any different.  And you know.  You have a circuit.  That is connected.  And.  We actually sort of snap them together.  In this onion style.  Encrypted payload.  Where the addresses.  And public keys of the next hop.  Are embedded.  In a onion style.  Peel off a layer as you go.  You know system.  So this blob of stuff.  Has a private key.  Associated with the route itself.  Not with the nodes.  So everything gets signed.  But.  It's associated with the private route.  Or the safety route.  Not with the node ID.  Or an identity.  And that's allocated on a per route basis.  And these things are coming and going all the time.  This is one of the reasons.  Why we don't necessarily require.  Security.  You know we don't need to worry about.  Long.  Encrypted.  People monitoring.  A single node for a long time.  And then going back in time.  And cracking that conversation.  Because it's moving around all the time.  These routes are coming and going.  You'd have to monitor the entire Valid Network.  To be able to generate enough information.  To be able to utilize some of those.  Differential crypto attacks.  So that envelope that you build.  And an encrypted chunk of stuff.  That is guaranteed.  To be DH'd across just the private route.  And the safety route.  And.  As it goes from node to node.  We're shrinking those payloads.  And eventually it drops off.  Where the end can.  Receive the encrypted blob.  All of these slides are going to be available online.  If you want to tear apart the exact.  Cryptographic implementation.  And of course the code is open too.  If you want to read it.  Toward the future.  There are ways to improve private routing.  We want to do per hop payload keying as well.  We want to minimize the number of things.  That you can glean about traffic.  From sniffing the wire.  Directly.  So reducing the number of things that are common.  Between packets.  Is a good goal over time.  Right now we also do some internal hop counting.  That was for debug purposes.  We can just drop that.  We have bidirectional routes today.  Allocations are done directionally.  We can drop that.  And just go with bidirectional routes to begin with.  And we can do things like hop caching.  And increasing our hop counts.  If we wanted to.  Right now our hop count is effectively equal.  To Tor's hop count.  And they did a whole lot of math.  On why a single guard node.  And a single exit node.  Would be sufficient.  There are two extra nodes in there.  But the reasons why.  Is to make sure that no node can look to the left.  Or to the right.  And know anything about who they're talking to.  As we said in the beginning.  We really wanted to make sure that this is private.  Make sure that nobody can see your location.  And can't get your IP address.  You have to invite somebody.  To be a part of your network.  You have to.  Nothing is going to be forced upon you.  Or put in an algorithm.  So that means that you have a lot more say.  As a user.  In how this is being used.  The no IP address means no tracking.  Collection or correlation.  That's right.  No tracking.  As of right now.  That is the biggest way.  Let's give it up for no tracking.  How nice is that?  A lot of people are monetizing our internet use.  When I was a kid.  I was told about the internet.  Way back in the day.  And I was learning about it.  And I was told it was going to be this open free place.  Where people can learn.  And they can connect to each other.  And instead we have had billionaires.  That have decided to monetize those connections.  And seep every little bit.  Of information and data.  Out of each connection.  And use that they can.  And a lot of people are falling for that.  We have made.  Like I said.  Privacy has to be accessible for everybody.  We have to make sure people understand what that means.  And we have to make things.  That are going to entice them.  That are going to be inviting.  To everybody.  If we do things like.  Build social media networks.  If we get our parents off Facebook.  We're all free.  Okay.  So let's build that.  Let's do this.  Let's do a thing.  Okay.  So how do you do this with us?  Valid loves Rust.  We are crabs.  We have crabs.  We want more crabs.  Valid is written in Rust.  The crates are going to be published.  I wrote this before.  I said the crates are going to be published.  The crates are on the GitLab.  And we'll explain that a little bit more.  We haven't published everything.  We're making sure the documentation is right.  We're getting there.  The crates will be on crates.io in about a month.  As we make sure the docs look as spiffy as they deserve to be.  There is a power user quick start.  When you are ready to run your own Valid server.  You'll be able to do that by either cloning the code.  And running it.  Or you can get Deb, APT or RPM packages.  From our package servers today.  Just download it.  Turn on automatic or unattended upgrades.  Because we're going to keep changing this thing.  And keeping everyone up to date will help the network out.  There is a really cool vapor wave looking CLI.  That you can use to talk to your node.  And send little weird messages and shit.  Knock yourself out with that.  We also love Flutter.  Flutter is a cross platform mobile first application development framework.  Powers all kinds of cool apps like your Instacart.  And DoorDash and all that kind of stuff.  It is Dart internally.  Which is a JavaScript overlay.  That does a really good job of providing asynchronous programming.  To environments that deserve it.  Like the web.  So we have a first class FFI and JavaScript WASM plugin.  For Dart today.  And we have actually written an app using it.  That we will talk about later.  Maybe at the party tonight.  You want to come to the party?  I hope so.  And we love Python too.  If you want to learn how to use Valid.  And just hack on it.  And mess with it.  We support Python today through a JSON API.  That's a no compile way of talking to a Valid server.  So apt install Valid server.  Pull down the Valid bindings for Python.  And you can start writing little apps and scripts and things.  That talk over Valid today.  Our demo lab tomorrow at 10.  Is going to talk and demo the Python side of things.  And show you how you can make your own little chat app.  That is completely private.  Distributed and encrypted.  Both at rest and on the wire.  And so how can you help?  We need coders and hackers.  But more so we need regular people.  Everybody.  Okay we need everybody.  It takes a team.  It takes an entire village to build something.  That is approachable and usable for everybody.  I mean I love all of you coders.  I love all of you hackers.  But we need people.  We need human beings.  Regular people.  We need to speak to human beings.  Everybody here has a voice.  And it all matters.  So you can find us online.  Valid.com is up.  We have a Twitter.  You have probably seen it ad nauseum for the last six months.  But it is there.  We are also on Macedon.  We love Federation.  We want to do more with that.  We have a Discord coming up.  And we have a GitLab that is going to be open now.  So feel free to go poke at it.  And see what we have built.  So if you are here looking for a flag for a certain CTF.  It is not here.  It is in that code.  You might have to read the code.  It is in there somewhere.  So you can see it live tonight.  We have a release party at 8pm.  We are rocking the house.  We got all kinds of cool bands and shit.  It is going to be a wild time.  CDC hijinks.  You name it.  Have you ever heard of a little thing called BO2K?  Let's bring that back.  The world up.  We got the cult of the dead cow.  In force.  They are here.  Come on.  Let's all do a hacktivism together.  Awesome.  See you guys.  If you have any questions.  You can bug us out in the smoking area outside.  In about ten minutes.  Talk to you soon.  Thanks everyone.


url: https://www.youtube.com/watch?v=98hacQ1zpgQ
title: DEF CON 31 - Clocks Have Ears - Timing/ Browser Based Local Network Port Scanner - Dongsung Kim
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=98hacQ1zpgQ",
    "title": "DEF CON 31 - Clocks Have Ears - Timing/ Browser Based Local Network Port Scanner - Dongsung Kim",
    "description": "",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 1090,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230915"
  }

  This text was transcribed using whisper model: large-v2

   Thank you. Well, this is more people than I expected actually. Uh, thank you for coming to my  talk. It's all, it's very great to see you all. I'm very excited to present my research. Your  clocks have ears. Timing based, browser based, local network port scanner. Before we begin,  let me introduce myself. My name is Donnie Kim. Uh, I'm a security, IT security expert at  security office. We are based in Sweden and South Korea and we are part of the Tracer group. If  you have any questions or comments, feel free to reach out and feel free to leave a comment  on over Twitter and or X or whatever. It's going to be called in the next few hours. In May  2020, it was revealed that eBay and other prominent companies were port scanning their visitors'  computers via their websites. The culprit behind the whole saga was called Threatmetrics.  A firm which pride, prides itself on tracking 1.4 billion users online and whose tool suite  included port scanning. The implications of browser based port scanning is twofold. One is  privacy. It can be used to fingerprint visitors, tracking them with characteristics unique to  each user, browser and hardware setup. The other is security. It can be used to uncover  internet services which are a great foothold for performing subsequent attacks like XSS or CSRF.  Various techniques have been circulating for some time. The earliest research I found goes  back to 2006 which utilized the image and iframe tags. Later research added the script tag and  newer exploits employed WebRTC and WebSocket to achieve comparable results. Browser vendors  have implemented mitigations making the mentioned techniques more or less ineffective.  In this presentation, I'll put my focus on timing based techniques for browser based port  scanning. Before we get into the specifics, let's take a look at how the Fetch API is employed  in measuring the timing of a network request. You can see from the function at line 3, we set  up a timer. At line 5, we measure the time. At line 7, we start the fetch. At line 3, we  measure the time again. Finally, the elapsed time in microseconds is returned at line 17.  What we can observe using this function is only the elapsed time of a fetch request due to the  restrictions imposed by browsers and web specifications. We cannot observe any other  metadata such as any of the response data including TCP, TLS, HTTP information, any of  the request data except for the scheme, IP, port, plus any of the error data except for the  generic type error, fail to fetch error string. We can also observe that the request data  is not a generic type error. Now, we can imagine at least 4 components that are each  responsible for a part of a round trip time of a network request. Browser, firewall, TCP, and  TLS plus HTTP. Then, we can think of 3 possible states of a port and hypothesize which  components would take up the observed round trip time. Firstly, if the port is closed and  the firewall drops packets or the host is unreachable, the observed time would be  theoretically infinite but typically an arbitrary timeout would be enforced. Secondly, if  the port is closed and the firewall rejects packets, the browser and the firewall would  take up the observed time. Finally, if the port is open and the process is listening, the  browser, the firewall, and the TCP and the TLS plus HTTP would take up the observed time.  Looking at the hypothesis, we can expect distinct timing differences depending on the  state of a port. Equipped with this knowledge, let's move on to the actual timing based  techniques. We'll try finding open ports on the local host, then we'll try finding the  router, and finally, we'll try finding open ports on all of the LAN clients. First, finding  open ports on the local host. Now, there are 2 things to note about local host. One, the  firewall always rejects packets when the port is closed, and two, the host is always  reachable. Keeping them in mind, we can rule out the first state since it's simply not  possible. For the remaining possible states, if the observed time is fast enough, the port  is closed and the firewall rejects packets. And the observed time is slow enough, the port  is open and the process is listening. So, the algorithm looks like this. Firstly, make  multiple fetches to the port 2, a port that is generally closed, and calculate the average  timing as a control. When a thousand measurements are plotted to a graph, it looks like  this. Secondly, if a fetch takes a similar time as the control, the port can be considered  closed. We do the same measurement to the local host port 65535. You can see from the graph,  the 2 lines are very similar. Finally, if a fetch takes a significantly more time than the  control, the port can be considered open. We do the same measurement to the local host  port 8000, where an HTTP server is running. As you can see from the graph, there is a clear  difference. Repeat the algorithm against various ports of interest, then we get a list of  open ports. Next, finding the router on the LAN. In this case, we only care if the host is  responsive or not, regardless of the port state. If the host is responsive, we found a live  host. So, if an arbitrary timeout is enforced, we can simply say the host is unresponsive.  If the observed time is fast enough so that the port is closed, the port, the firewall  rejects packets, or slow enough so that the port is open and the process is listening, we can  conclude that the host is responsive. So, the algorithm goes like this. Firstly, if a  fetch times out, the host can be considered unresponsive. When we do the measurements  against a potential but non-existent router IP, 10.0.0.1, as you can see from the graph,  they are all above the timeout, which is 50 milliseconds. Secondly, if a fetch ends before  the timeout, the host can be considered responsive. When we do the measurements against the  live router IP, 192.168.0.1, as the graph shows, they are mostly below the timeout. So,  repeat the algorithm against a list of potential router IPs and ports, then we have a list  of the live routers. Next, finding open ports on the LAN clients. Here, we will make an  observation on one more thing. There is this new web standard called private network  access. On Chrome, by default, requests to private network resources are allowed only  from HTTPS web pages, meaning for HTTP requests to any client on the LAN to any local IP from  an HTTP web page would return a CORS error. Below is what it looks like when the error is  returned. Access to fetch at HTTP, this address, from origin HTTP, this address, has been  blocked by CORS policy. The error string itself is still the generic type error, failed to  fetch. Let's look at how the CORS error looks like. First, let's look at the error string  gets returned at the specification level. 3.1.1, secure context restriction. 5, the  HTTP network fetch algorithm is amended to add 3 new steps at 5.2. There is a private  network access check, then 5.3 returns it if the result is a network error. Next, we  look at the error string. But there is something strange. If you read .5 very carefully, the  HTTP network fetch algorithm is amended to add 3 new steps right after checking that the  newly obtained connection is not failure. This means the CORS check happens after the  connection is established and the CORS error happens only when the TCP connection is  successful, when the port is open. With this knowledge, we can think of new states and  hypotheses. When fetching HTTP local IP from an HTTP web page, if the port is open and  the process is listening, the browser and the firewall and the TCP would take up the observed  time. A CORS error is returned, but we can't observe that as we discussed previously. On the  other hand, when fetching HTTPS local IP from an HTTP web page on the same port, the browser,  the firewall, the TCP and the TLS plus HTTP would take up the observed time.  So, this is how we find open ports on the LAN clients. From an HTTP web page, we take a  measurement using HTTP. Then we take another measurement using HTTPS. If there is a  difference between the two observed times, we can say the port is open and the process is  listening. The algorithm is able to detect the error and the process is listening.  The algorithm goes like this. If an HTTPS fetch takes a similar time as an HTTP fetch, the  port can be considered closed and the firewall rejects packets. One line is measurements  against HTTP 10.0.10.7 port 80 and the other line is against the same port but with HTTPS. As  you can see, the two lines are indistinguishable. However, if an HTTPS fetch is  significantly slower than an HTTP fetch, the port can be considered open and the process is  listening. The line above is measurements against HTTPS 10.0.10.12 port 445 and the one  below, almost zero, is against the same port but with HTTP. The graph shows there is about  20 milliseconds of difference. I've created a proof of concept called your LAN.js.  Which utilizes all the three techniques. Let me show you a demo. The video has been sped up  but you can see the actual time it took on the screen. First we try finding open ports on  the local host. It takes about two and a half minutes and it's found Airplay, Discord, an  HTTP server, LINE, Steam and Logi Options Plus. Next we try finding the router. It took  about 10 seconds and the first choice of the possible IPs was the one. Next we try finding  open ports on all the LAN clients. We search the whole subnet against ports of interest. It  takes about 40 minutes and you can see there are many IoT devices on the LAN. Airplay, Google  Cast, LG Smart Device, Dyson, SMB, Homebridge, et cetera. Finally let's explore some of the  found ports. The first one is a router interface. The next one returns HTTP 470 for some  reason. Next one is 404. Next one says nothing to see here. The next one is an actual  homebridge. And the last one is a Philips Hue light bulb. Okay. Let's see if we can find  a port. In conclusion, a website can scan your local network, your local host, your router  and your clients on the LAN. There are some limitations on this research due to the nature of a  timing based attack. There can be false positives and false negatives. While we have  simplified the states into three or four, in actuality there exist many other complicated  scenarios. Also the proof of concept needs further optimization to speed up the scanning  process. Lastly, I'd like to end the talk with takeaways. It's always best to use Tor Browser for  privacy and security. Related to this research, Tor Browser simply declines any request to  local host and local IPs. Also use ad blockers to avoid malicious tracking scripts. Finally,  VPNs can be a mitigation as long as they allow access to the user's local network. And  that's it. I'd like to thank Oscar, Hansol, Luis, Sun and the ACC group for helping out and  Betty for editing. Thank you very much for listening.


url: https://www.youtube.com/watch?v=3T2Al3jdY38
title: DEF CON 31 - Weaponizing Plain Text ANSI Escape Sequences as a Forensic Nightmare - STÖK
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=3T2Al3jdY38",
    "title": "DEF CON 31 - Weaponizing Plain Text  ANSI Escape Sequences as a Forensic Nightmare - STO\u0308K",
    "description": "Logs are a vital component for maintaining application reliability, performance, and security. They serve as a source of information for developers, security teams, and other stakeholders to understand what has happened or gone wrong within an application. However, logs can also be used to compromise the security of an application by injecting malicious content.\n\nIn this presentation, we will explore how ANSI escape sequences can be used to inject, vandalize, and even weaponize log files of modern applications. We will revisit old terminal injection research and log tampering techniques from the 80-90s. Combine them with new features, to create chaos and mischief in the modern cloud cli\u2019s, mobile, and feature-rich DevOps terminal emulators of today.\n\nWe will then provide solutions on how to avoid passing on malicious escape sequences into our log files. By doing so, we can ensure that we can trust the data inside our logs, making it safe for operators to use shells to audit files. Enabling responders to quickly and accurately investigate incidents without wasting time cleaning, or having to gather additional data, while reconstructing events.\n\nWelcome to this \"not so black and white,\" but rather quite colorful ANSI adventure, and learn how to cause, or prevent a forensic nightmare.",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2431,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230915"
  }

  This text was transcribed using whisper model: large-v2

   Am I doing my thing now? Yes? Wow. It's an honor being here. Hi, my name is Stöck. Oh, no. Hi, my name is Fredrik  Alexander. Most people actually know me more by my hackle hand as Stöck. And I'm gonna welcome all  you to my talk, Weaponizing Plain Text, Unseen Scape Sequences as a Forensic Nightmare. And  leading up to this talk, I was kind of told that you need to have this table of content that  explains things that are gonna happen because otherwise people get bored and really sad.  But I figured that we got 281 slides to go through. So screw that other thing and just  head straight into it. Thank you. So I spent a good time of digging into this vulnerability  class CB150 that's been dormant for the last two decades or so. And that was real fun. And  that's what this talk is all about. But I used to travel the world to all these live hacking events  and create educational videos on YouTube. But nowadays I fight cybercrime over at TrueSec.  And I also do shitloads of incident response. So I realized that I'm really passionate about  getting log files because that's kind of the whole part of creating a good timeline.  So I figured that maybe I'm the only one, maybe some other people like log files too. So I went  on the application formerly known as Twitter and asked, are log files important? And I got a bunch  of really good answers coming back. And the short answer is kind of like, yes, log files are  important because logs are a vital component for maintaining application reliability, performance  and security. But it also seems like the common idea is that logs don't lie, people do. And  even though your log files might just not lie to you, do you still trust them? And what happens  if you don't? Because imagine this for a second. You are now the incident response lead of a  global logistics corporation. It's 4 o'clock in the morning and you're sound asleep in your bed,  dreaming about boats or maybe containers in the dockers and stuff. When you get this rude  awakening call from your manager that's pissed because you got breached. And since you are the  incident response lead, it's your responsibility to call up all those tired people that work in  the incident response team and get those starting with collecting all the logs that you need  to be able to create the timeline. You're a manager, so you don't do grunt work, so you log  into your big data app for a thousand. And since you're a professional, you log ship stuff already,  so you export the data, save it into a local file called everything.log. You're happy. But just to  verify that you got any data in there, you decided that maybe I should just cat the file and see  what's in there. In the beginning, everything looks just great. It contains the things that  you wanted, but then suddenly out of nowhere, boom, you're getting an ad for a restoring service  and they're nice enough to provide you with a calculator so you can do the math for how much  it's going to cost you. And that's kind of the situation where you're starting to realize that  shit. And the reason for all this is that it has to do with ANSI escape sequences. And what got  me all into this is this CVE here. And I was just going to read this straight out from my monitor  because I didn't remember it all. Effective version of this package are vulnerable to arbitrary  code injection. There we are interested, right? This is a possible shell escape sequence injection  when built in Racks, Lint, and Common Logger components. Carefully crafted requests can cause  shell escape sequences to be written to the terminal via Racks, Lint, Middleware, and  Common Logger. Middleware. Okay? These escape sequences can be leveraged to possibly execute  commands in the victim's terminal. Audio guy, are you with me? Excuse me, Tiny Sturck. What do you  mean possibly? Either you can or you cannot run arbitrary commands, right? Yeah. Thank you for  that, Giant Sturck, for pitching in and helping out here in the presentation. Well, you're actually  right. It depends. But it apparently depends on the terminal that you're using. And there's a  shitload of different terminals out there today. And some of them come built into the software  that you're using, like VS Code and Docker. Then you have third-party stuff like iTermKitty and  other ones and stuff that people built on their own. Then you have the ones that get shipped in  your operational system, like Windows Terminal, Terminal.app, GNOME, VTE, and other stuff. And  then you've got old stuff like Xterm. So we also got this when we were in the cloud. Maybe you  don't know this, but if you're going to try to use a digital ocean to connect to your droplet or  you're going to do some recovery console stuff, that is actually like an Xterm thingy. Connecting  to that over TTY, you're able to interact with it. It's a terminal. And I wasn't really interested  in the possibly part. I wanted to execute commands in the victim terminals. But I had no idea where  to begin. It sounded cool. So I did everything that every self-respecting researcher do. I  Googled what ANSI escape codes were. I had no idea. And I quickly realized that people actually  like to customize their terminals in the weirdest kind of ways, especially if you're using ARC. And  this is the thing that got me interested. Like, oh, cool, you can change colors and stuff. But I  also realized that in 1978, the Jillstrand boys dropped their Banger album, We're Going to Have  Some Fun Tonight. But even though it would release the same year, the VT100 terminal, and  connecting that using serial cables to a big VAX system is probably not what those guys thought  would be fun to do on a Saturday night, even though I personally would think that was kind of cool.  But I also realized that ANSI escape sequences are the shit that puts stuff on your terminal that  you look at. And it can do stuff like you can move the cursor around. You can remap the keys  using these. And you can actually print stuff. Bing. Interesting. But you can also build whole UIs.  All this UI here is built in ANSI escape codes, positioning of stuff, colors and output and all  that, refreshing, clearing screens. All that is part of something called the control sequences  standard. This is a standard for Xterm, and it's vast and it's super hard to understand. I know  you're all super smart. I wasn't. So I'm going to go through the basics with you. And the first  I started looking through the documentation, it said CSIPM character attribute something,  no idea what that meant, but it's this. So all it actually is is printing out something that  turns into green. Cool. And the reason why this works is due to this funny thing here called the  escape character. That's important to remember. And it's also a non-visible escape character.  And it comes in many shapes and forms, because the reason why they didn't print out exactly the  command is because it depends if you're using octal, hex, unicode, decimal or ASCII. All of  these are the same sequences, different kind of ways to portrait it. And I also realized that  Bash likes octal, Python likes hex, Java and JavaScript likes unicode and PowerShell likes  decimal for some reason. And this escape character here is the one that starts the whole sequence.  It needs to start with that. Then we have this control sequence introducer, which tells the  terminal to do shit. Next one is the parameter, the number. So we'll go with 32, which is green  here. M is to change whatever continues output turning that into green, which is the string we  have here. And then we're just going to close it off with another escape character, another control  sequence introducer. And this is the important part. Here we do zero M, and that means we're  resetting the terminal back. Because if you don't do that, apparently this is going to happen.  Everything, the rest of your shit is going to be all green. Hulk smash. But we need to terminate  this using a ST or string terminator or bell character. And if you're like me and you accidentally  gripped something, and then your terminal starts like ding, ding, ding, that is being interpreted  as the bell character. So we got escape sequences being shoved in there. So I thought, okay, cool,  that's fun. I can put colors on my screen. Fancy. What happens if I put that in something called a  bad log and see what happens and then interact with it? Well, if I'm using Vim, it's going to  show exactly what the escape character is. It's this one here. And that's cool, because it's not  rendering. It just shows it. Same for Nano. If you're using that, you're cool. And if you're  using a browser, instead it's going to show it as Unicode. So that's also interesting to know.  It's getting interpreted differently depending on the system you're using.  But if you cut it, colors are going to get displayed. Grip, tail, the same thing. And even  if you curl it, boom, fun stuff is going to happen. Because all these systems interpret  the sequence and renders it. And at this stage, when I ran into curl, I'm like, shit, what just  happened? What just happened? And I started thinking, where can I stuff this? Where that  would interact with other people's stuff? So I figured, why not do it in a text record? Because  DNS is always the source for all evil. So I put it in there. And I ran NSLookup on OSX. Nothing  happened. You can see here, the extra slash in front of it means that it's escaped. It's  beautifully escaped. This is a real good thing. If you're doing that on Windows, though, it's a  different kind of story. So it depends on what kind of OS you're using. And big ups to David for  showing that. That was real cool. So I was starting to ask myself, okay, cool, that's fun.  But is this even a security issue? Seriously? So this kind of feels like this self-access thing  where you're like, okay, I can put colors on my terminal fun times. I need to figure out,  who would this be an issue for? Where would this be an issue? How would this be an issue? And  what would be the consequences of this? Well, the weird one wasn't that hard to figure out.  Apparently, since we had the common logger thing, log injection would be the way to go. And  who would it affect? I reckon DevOps, sysadmin, IR forensic people, or anyone that interacts  with a log file using a terminal. And we can take a quick break here. How many of you have ever  catted, gripped, or JQ'd a file that you parsed from some other software? I love you.  So to do this, as the scientists we are, researchers, we're professionals. Logging  injection vulnerabilities occur according to OS when data enters an application from an  untrusted source. Yes. The data is written to an application or a system log file. Cool.  I needed to adapt that. This is kind of written to do XSS stuff. So I figured that, okay, to  successfully have this as a measurement, I needed to be able to inject new log files or events by  forging a log. I needed to be able to inject an escape sequence. And I needed to in some way or  form make that run in someone's terminal emulator. And I also needed that injection to execute some  kind of stuff. And I figured, which would be the best test bench for this? And Docker, obviously.  Because it loves to send whatever logs to standard out. It has this real live log that's running  through. They're actually terminal rendering stuff. And it comes with a really nifty getting  started kind of app. Good stuff. I'll go for that. And also, a way for you, if you want to attach  or work with Docker, you would use Docker attach inside a terminal, connect to it, or Docker logs  to do like a tail and see whatever happens inside your container. I'm like, seems tasty enough for  me. But I also realized that I couldn't just take this print F thing and stuff it in a browser  because, hey, shit, you can't do that. No, no, no. So I needed a URL encoded. And for good measure,  we added a new line in there because everyone knows that if you're triaging some kind of bugs,  new lines are very important to show that it's actually able to inject some kind of false log  entries. So I figured, why not see? Let's see what happens if I curl this and you send that straight  in. And I was really excited when I realized, oh, nothing came back. Then I started looking at the  log in over here. It's like, oh, nothing comes in return in the response. But new line has been  entered. The color has been rendered inside the log that's being displayed in real time, which  means on the first try, more or less, we checked all the boxes to prove a successful log injection  because we're injecting new lines. We are enabled to run escape sequences that will be viewed in a  terminal emulator, in this case Docker. And I'm able to inject commands that the terminal emulator  could execute. In this case, we're changing the color. Because remember, even though that was the  only line that I put in, we can create this. Because it's all about the limitations and creativity  of whoever decides to design this. And even though I'm like, fuck yeah, confirmed, the Yieldstrand  boys weren't that impressed. They wanted me to provide more impact. So I'm like, okay, I can do  that. I started once again Googling stuff. And then I stumbled upon this great research from 2003  by H.D. Moore. And he found a really, really interesting, cool way to just misuse the terminal  and use it against them in a way. So what happens here is that they were able to use something  called OSC 2 sequence, where you will actually put this on a command line, and that would change  the title of the terminal you're in. And if you're currently using a terminal and you're changing  directories and stuff, and it used updates there inside the title, I've seen that, or you're  connecting to a shell and changing it, that is actually the sequence doing that. But that wasn't  enough. They digged into the deeper parts of the whole documentation and realized that in this CSI  code here, 21T, you were actually able to report back whatever was put inside the title and push  that to the command line. That could only mean trouble. Because even if this stuff would be put  on the command line, and in this case, you needed to have a social interaction with the user. The  user needs to be fooled into pressing enter. It's still a cool way to just push something in  the title, read it back, and then boom, shit happens. But that's been fixed for ages. 2003, if you  want to feel old, is like two decades ago. So I started searching some more, and I found this  really nice write-up from Evil Alive Jake Linasky that did some cool research on all these different  web servers back in 2010, where they're using the same kind of sequence here. But we know that's  been fixed, so can't do much with that. But then in 2022, Evitar Gursi released some, sorry for  crashing the name, don't know how to pronounce it, a cool way to create a DOS sequence when they  were able to read a bunch of stuff into the title and just creating crashes. And I'm just tasting  buffer overflows here for the future to come. But that's been fixed too. So here I am, feeling like  an epic fail. Can't do the arbitrary command injection. That seems so cool that the person  talked about in the CBE. So I did the thing. I think the end is the only thing you do. You reach out to  the original poster and ask them, can I ask the RC park? And the answer back, I didn't write any, I  used this paper as a reference, and that sucks because we know that's fixed. So here I am, super  excited, but all I got, old stuff. And speaking about old stuff, this is what I looked like in the  80s. No, I'm kidding. I'm probably more like this badass here. And if you look at the angle of the  arm here, that's very important because it's exactly the same angle as this gentleman here in  this band poster. And that's pretty fucking darn cool because he was one of the first people to  sell Diab and Unix computers in Sweden. And not only that, he's my dad. So I'm very grateful to  him to put stuff and give me access to computers at an early age. Thank you.  But I'm starting to look at this picture, I'm starting to realize that I'm actually,  maybe I'm looking more like Lasse that was on the same tour as my dad when he met my mom.  Anyway, the standard is vast and it's deep, and it takes a lot of things to go through, but that  wasn't enough for the fine developers and users of terminals of today. No, they wanted more.  Because all these different terminals have different kind of functionality, and you know,  with more features, we need more stuff. So they invited something called proprietary escape codes,  which is a local edition of the standard. And it works in certain terminals, not all of them.  Some of them are being shared internally and some work over the whole bar. And what that does is  gives you the opportunity to make a fun twirly thing maybe in your title, get notifications and  all this other cool stuff that's in there. Maybe even change your background. Fancy stuff.  But some thought that it was a great idea to always run some processes with arguments.  Why not? Using escape sequences. I mean, that's a really bad idea.  But luckily enough, it's rare to find stuff that runs a really bad idea too  nowadays because it's old and it's been fixed. So I needed to find something that was adapted  by everyone, everybody loved it and trusted it, and they thought it was nice.  That we could weaponize. Why not? Because we are Trixie Hobbit sissies.  And I found OSC 8. I'm like, cool. If you ever run PowerShell and you mess that up,  you'll get this prompt coming back that says, you suck. Click this link and somebody's going  to tell you how to do it. I'm like, cool. That seems doable. And the reason why that works is  because you have this ANSI escape sequence that gives you the HTTP thing and then it has the  title or whatever you want to print out and you close it off and it turns into this link  inside your terminal. And that's great. Initially, that was only allowed to use URI like HTTP and  HTTPS. But recently, the fine people over at Windows Terminal decided that we should allow  file 2. That's even a great idea. Because what could even happen if you typed in CMD there and  clicked it? I wonder. Fun things popping inside user space. And even though this is kind of  filtered in a way where you can start CMD but you can't use any parameters, I'm willing to guess  that some smart person in here is going to figure out how to bypass that, put a link in and pop  shells. It's definitely doable. And the fun thing with OC8 is if you don't terminate it correctly,  you remember what we did with the green when it all went hulk everywhere? If we don't terminate  it correctly, the rest of the output starting from that injection is going to be one big link.  Everything is going to be a link. And if you click that, some terminals will be very nice for  you and just block that out with a warning. And I figure that, okay, that's cool. I wonder if I  can inject this somewhere. Because obviously me putting links on my own shit would be boring. So  what happens if I put that inside Docker? Well, everything is going to turn into one big link  there, too. And that one has a protection in. But it's not everywhere this turns into a protection  because injections can be put in different places and then suddenly you can click links and you can  imagine how fun this could be. But that wasn't enough for people. Because, you know, text is  cool, but inline image support, that's definitely something we need in text space. So the Windows  Terminal people, they decided to adapt the inline support for it in the recent build where you're  able to have six cell and inline support inside the terminal. And that's great. I see no problem  with that. I think that's great. I just wonder what's going to be inside this log file.  So can I get a raise of hands of how many of you wants to see what's in that log file?  I love you. And it's not what you thought it would be. Shame on you!  Then we have OC52, which is great. If you ever used Tmarks or anything and back in the days  you weren't able to copy paste between shells, sucked. Now you can. The reason is OC52. And what  it does is that it takes this sequence, if you have something base 64 coded, it's going to put  that inside user space clipboard. Super sweet. Great. I wonder what happens if we do open a  something calculator app and a new line on the end because we need that new line. If you put that  into CSH and paste it, this is what's going to happen. It requires the users to do something.  And you press that, boom, okay, it pops calc. But if you do the same thing in Bash, Bash honors the  new line. So if you just paste it straight in there, it fires it directly. There are room for  improvement here, though. So I figured I wonder how can I test this in different spaces. I did  this kind of base 64 string with these commands in it and said I'm curious what's going to happen.  And I started to paste that in different places. Docker, that fired directly. That was nice. VS  code, it did it differently depending on what kind of shell you're using. And even in some cases  gave you this warning, which is super good. Of course with your own people. Terminal.app honors  it, runs it directly. And even, you know, inside Windows Terminal you will get even this preview  of showing things that are bad. And the reason for that is that if you wanted to run this command  10 lines and do whom I, Windows Terminal is going to be nice of you to show the first lines and tell  you, yeah, you got your shit going on here. So why not just add seven lines of nothing and see  what happens? Well, it's nothing in there. And everybody knows that nobody is going to scroll  down. And what happens if you add 400 lines in there? Nobody ever going to scroll down. Or you  can even in this add some kind of those ULA things. Then people just press okay.  So with that in there, it's going to be okay just running it. I think that's pretty cool. It's a  decent thing. So I think, okay, so I have this sort of thing for Windows Terminal. I've got one  for Bash. Let's create a polyglot and see what happens. So I figured that why not have a little  bit of an SVG in there. You stop that one off with a good old hash mark at the end. Do a curl  command. Obviously, we want this to run locally and not fuck around on the interwebs. And then  we do a PowerShell kind of mix where we're using CMD that calls PowerShell and stuff because we  want it to work in whatever kind of environment you end up in. And we want to exfiltrate stuff  to know what's happening. So here's the hash that's important. And then eventually this last  one here. Encode that into one big basic support thing and just send that to bad log, cat bad log,  and then start pasting it in places. Well, it obviously worked there. This removed all the  info in the beginning. It's going to pop here too. And if you go to the browser and you paste  that in, you have some JavaScript running, which is great. Then you can just refresh it here and  you'll see that, okay, it's running who am I and different other kind of stuff in there that's  just exfiltrating stuff for you inside the request. And this is just being dormant in some poor  bastard's clipboard by just interacting with the file. Because you can sit there and work, you  catch something, and you decide to paste and boom, it's all you. It's not limited there.  Obviously you can exfiltrate stuff like sending things out and do reverse shells and all that. I  didn't go through that in this demo. You are smart people. You know what to do. But if we're  doing this, we might want to hide our tracks or just mess things up because we can. And there's  different ways to do this. You can use everything from just moving the cursor to a certain  position. You can clear the screen. You can erase stuff or clear the scroll buffer. And to  showcase this, I'm just putting like 15 lines here on the screen for you. And then we're just  going to run the command and it's empty. And that's perfect to have a proof of concept myself.  But what happens if we run that inside, let's say, this Docker instance and we send in the  reset screen and you just go back? Well, this is what's going to happen. It empties the log  file for you. And every time you load it, all the way up to that's getting rendered, it's going  to clean it anyway for you. So you need to sanitize and clean that out. Fun. And that is  somewhat annoying. For anyone that's using that, that's really annoying. But it's not annoying  enough. We need to DOS and brick stuff to really be annoying. And why not use the standard? We  have this print screen functionality that gives us the opportunity, if we're typing that one in,  it will ping, sending the printer prompt for you. Imagine getting that every minute or so. It can  be annoying. But you can also do this cool thing. Instead of doing the 0i, you can do 5i. Because  back in the day, we used to have matrix printers next to our computers that were like, eh, eh, eh,  having a paper copy of said log. We don't have those anymore. So we can't send anything out to  this buffer that doesn't exist, which means, as you can see here, if you send it in, all the  commands are going to send it toopian and it's still going to render, but you can't see shit. So  that's really annoying. And the only way to get rid of that is to close the browser. But it's not  annoying enough. No, we need to be really annoying. And one way to do that is to use the  mouse tracking functionality, which was great. If you ever installed some kind of, I don't know,  Slackware or something or whatever, Debian, and you're in that CLI mode and you suddenly realize  my mouse works and I can click on stuff, that's cool. That is the reason because somebody's  tracking where your mouse are in that state. And if you run that on the terminal and move your  mouse around, it's going to report all that stuff all the time. And if you click, it's going to  execute whatever kind of happened there. And what's really interesting when doing this is that  imagine you're sneaking some other shit in there and somebody clicks stuff and then,  boop, it's going to pop it. Fun. And if you combine those, you see where I'm going, a bunch  of chains here. You can just combine those things together and just create a lot of interesting  stuff. But then I stumbled upon this one here. Looks so sweet. And it's perfect. Repeat the  preceding graphic character X amount of times. And that is perfect. You want to measure how  big the terminal screen are. And so in this case, let's see that we wanted to repeat this 10 times.  It's going to do it 10 times for us in the output of the terminal. But one simply wonders.  A billion times is a lot of fucking piece. And since this is not a single character,  this is Unicode, you get five characters for the price of one. And my recommendation to you is do  not run this in prod. Things will break. Like, I can't open this Docker container again without  it crashing. And when it crashes, it kills Docker desktops. I need to start it all over again.  It can be somewhat annoying. Thank you. So it breaks most terminal. And some terminals,  even like the one I have on my phone here, it crashes that one. So once again, I'm starting  to think, okay, that's cool. I wonder what happens. What can I do to weaponize this?  Maybe if I put it in my service signature on a host and then request a let's encrypt certificate.  I wonder if the bots will come running. I'm betting you there's a bunch of fucking  automation scripts that's going to scrape and look for kind of version I'm running  that's going to get dosed out whenever somebody decides to look at that entry.  Real fun. Okay. You got a question, scary looking kid? No RCE? No. I know. I wasn't able to do any  kind of arbitrary command injection. Sucks. I got malicious stuff but not really bad stuff.  So these were dark times in the middle of my research. The light came in the shape of David  Leadbeater. And he's an amazing fucking dog, sorry, part of my French, at Blue Hat 23 called  Houdini in the Terminal where he doesn't provide just one. No, he brings on six remote code  executions in terminals that are live today. I'm like, mm, tasty. David knows stuff. And he's  doing that, but I'll use kind of the same thing that I did more back in the days with this  CRLF injection. But he's using something called device control request string. I'm not going to  try to pronounce that. If you want to check out his Git repo, it's really, really good. He's got a  lot of cool stuff there. And he's speaking tomorrow here. So please go visit that talk. I'm  sure I'm going to be there. And the yield sign voice is definitely going to be there because he's  going to release some banger stuff, and you don't want to miss that. So this is all David's  work, but I'm going to show it now because, hey, why not? So I figured that he had this thing  provided. I needed to put that in a way that I could infiltrate others. So basics for the string  here, more or less this is kind of what it does. It sends in a bunch of control sequences and  stuff. I'm not going to bore you with that, but more or less what it does, it asks you what kind  of color you have on the screen, but we're feeding that with other stuff instead and then adding  new lines and voila, fun stuff. So if you're using version 3.17 of iTerm, which is not the newest  one, it's been patched now, and you're interacting with the logs using this docker, what's going to  happen if I paste that inside your terminal and you're running that live tail? I'm going to pop  calc for you. Why not? Super simple way to just make sure that things break. So update your iTerms,  people, if you're playing around with that. And the reason why this is working, and I love this,  is simply because what David figured out is that he could sneak in x03 here, which is control C  kind of thing, or control X. So you're escaping out of whatever thing that's running, and imagine  what happens if you're trying to escape out of a docker attach sequence. It's going to shut down  the docker. And after that, we're adding whatever command that we want to run, and then simply we're  going back again, and it tells the system, could you please report back for us whatever kind of  color that we're running? But we already tainted that with other stuff, so it's just going to boom,  which is great. And for me, David was this kind of Mike Metzger moment for me when he did that  stuff, because what Mike did for the motocross circuit was that just before he did the first  backflip in competition, it was like rumors and shit, and nobody thought it would even be remotely  possible. But as soon as he did that, today if you're not doing like a double flip, you suck.  So that was kind of the opener. So I started to look at this, OSC 51.13, which I thought was like,  Kitty has this proprietary escape code. I wonder what that does. And what I figured out real fast  is that test here reflects back in the flow. So I'm starting to think, what happens if we add a new  line and open a bit of calculator there and send that away? Well, let's see what happens. What's  going to happen first is it's going to give you a request. Are you really sure you want to do this?  You said no, but you got a calculator. Because he's adding new lines and he's honoring them.  Time is running out. Thanks, babe. The other one here, and I know you said, yeah, but you  pressed no. What happens if you press yes? Well, I can figure out it's the same. No escaping here.  Yield time, boys. Fuck yeah. I thought that was sweet. So now I'm getting hyped and I'm all happy  about it. And this shows the whole proof of concept. Is it actually possible to execute  commands in the victim terminal? And the short answer to this is yes. But whose fault is it?  This conversation has been going on for like the last 10, 20 years. Because every 10 years or so,  somebody's going to poke around at this stuff. And the terminal developers are going to say,  it is the developer's fault. They need to do their shit. The app people say, no, it's the terminal  people. They need to stop using escape sequences at all. It's a terrible idea. But we all use them  all the time. And then there's the third part. It's these super angry old developers that are  sitting in the back ICU that thinks that everyone with a brain knows that you can't just cut the  file or click on an email link. But that's simply not true. Because we are cybersecurity  professional, some more malicious ones, and we need to protect our users, especially our shell  users, because they also are privileged users using these systems on a daily basis. And they  should be able to grab cat, ork, jq, whatever shit that's running inside the terminal without  any issues, especially if they're downloading files from a third party that took care of them.  And the only way to do that is that we need to sanitize our input and escape our output. And if  we don't do that, it's going to be consequences. And I figured it would be kind of cool, what  happens if I took this web page and stuck a bunch of different kind of escape sequences in there  and see what happens? So I created this evil.terminalinjection.com site. So if you curl  that inside your terminal, any of them, you can see what happens. It's not going to do anything  bad stuff for you. It's going to pop a calc if you're vulnerable. But I'm not exfiltrating your  data. You can just right-click the source and see what's in there. It's cool. And if you run  that on some systems, it's going to be all right. On other systems, when you do it, it's not going  to be as such all right. So it's kind of trusted things, but they will break stuff. So if we're  doing black box testing on this, the whole game, if you're hunting for bugs, is going to 400,  500 errors, because 200 errors usually ends up in access logs, and those are well sanitized and  nice and all. So the easiest way to do this, to break stuff, is to take whatever kind of thing  you have here. This is an API JSON request. You have a key and some value stuff, and you just  append shit on the end and see what happens. Sometimes it will break, and you will get some  information coming back. And if it does contain something like unescaped stuff, because this one  here is a big chance that it actually ended up in the error log, this one here, though, is  properly escaped. So it's a good job on the devs. Fuck yeah. But sometimes, you know, you put that  stuff in, and it's going to sanitize with a backslash. So why not put some invisible characters  and hex in there? The result is going to be really interesting, because then it might just render  anyway, because you're bypassing the whole backslash scenario. Okay. This is very prone to  false positives. So sometimes the developers are really good at just filtrating what kind of output  comes out, stripping the error message, and they're even not displaying any data. We saw that in  the beginning. It was an error, but nothing happened, but it smashed the log. Boom. So even though,  you know, you need to take that into consideration. And even though if you see like 400 errors,  they're rarely logged, so 500 errors is what you're going to be looking for. But this seems to  be everywhere. Everywhere I looked, it just pops and scrambles. And before you create this nuclei  template and go all brr all over the internet, hoping that it's going to end up in somebody's  log somewhere, to verify this, you will need to have access to the access logs and just dig into  that, because otherwise it would be a false positive, because if I wrote the logging system  and I didn't log any output, you can say that it renders, but it's not. So time's running out.  Let's recap real fast on this. People forget. The old is the new new. There's way more things  that's going to come in this area. I hope you are intrigued now to start poking in this, and  let's create a community around this and see whatever fun stuff we can do. And it's like  everywhere. Like I told you, I created this nuclei template, and when I ran it, it rendered in there.  That can be problematic. So if something's like meh today, it can easily turn into oh crap  by tomorrow if somebody decides to put a bunch of creativity in it.  That means your log files, do you still trust them?  I'm stuck with all the social media. Thank you for attending my talk.


url: https://www.youtube.com/watch?v=iprBprAFXCk
title: DEF CON 31 - Warshopping- Phreaking Smart Shopping Cart Wheels Through RF Sniffing - Joseph Gabay
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=iprBprAFXCk",
    "title": "DEF CON 31 - Warshopping- Phreaking Smart Shopping Cart Wheels Through RF Sniffing - Joseph Gabay",
    "description": "Smart shopping cart wheels are electronic wheels with a mechanical braking mechanism meant to prevent cart removal or shoplifting, as well as electronics to provide other tracking functions. In a past talk, I\u2019ve discussed the ultra-low-frequency communication these systems use and how to sniff and replay them (and even use your phone\u2019s speaker to \u201cphreak\u201d your shopping cart!\n\nThis talk explores a new type of smart wheel (the Rocateq system), and focuses on a deeper exploration of the hardware and firmware. On top of capturing new sets of ultra-low-frequency control signals, we\u2019ll look at the 2.4 GHz \u201ccheckout\u201d signal that it receives from the register and reverse engineer the PCB - soldering on \u201cfly-wires\u201d to look at the chip-to-chip communication with a logic analyzer. We\u2019ll also use a PICKIT programmer to dump the firmware from the main microcontroller for basic analysis using Ghidra.\n\nIn addition to the talk, the website where you can play the control signals as audio files on your phone will be updated to include the control codes for the Rocateq brand wheels.",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2304,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230915"
  }

  This text was transcribed using whisper model: large-v2

   So everyone, give a round of applause for Joe.  Alright, we're in. Well uh, everyone thanks for coming. Uh it's great to see you all here. Uh my  name's Joseph Gabay and today I'm gonna be talking to you about smart shopping cart  wheels. Uh if this seems familiar, uh I, 2 years ago, uh my first time at DefCon actually, I  gave a presentation about when I first started my work on these. I did a little bit of early  exploration and uh uh found some cool things you could do with them that I'll be talking  about now. But uh this talk is a continuation of that. Uh I found a completely new brand of  wheel. Uh and did a good amount of reverse engineering and I'd like to take you all on that  journey with me. So before I get any further, I wanna give a quick disclaimer, especially  cause a couple of my coworkers are out in the audience here today. Uh this is a personal  project. This has all been done on my own personal time, with my own personal equipment, uh  with my own personal curiosity. Uh none of this reflects my employer or anything I do  professionally and uh yeah this is just me doing my thing, trying to have some fun and learn  some stuff. So what this talk is, uh I'm gonna be taking you on the journey that I went on  to reverse engineer this wheel. There's a lot of troubleshooting, there's a couple dead  ends, I'm going to be pulling at some stuff that's oh that's neat but it doesn't go anywhere  else and that's the nature of reverse engineering. My ultimate goal with this project was  to try and defeat and bypass the anti-shoplifting features that Rokitech introduces and I  think I got to that point and you know my previous talk was kind of more focused on just  that. This one's going to be bringing you along on how I did it. We're going to be looking at  devices for hardware, firmware and security vulnerabilities. We're gonna be playing  around with debuggers and programmers like PitKits, logic analyzers like Salaze and cool  software stacks like Ghidra and Audacity do some fun stuff. We're going to instrument a  shopping cart with a bunch of SDRs and go war shopping. We're gonna do some old school audio  freaking. What this talk isn't, this isn't leak hacking, I'm not dropping some hot new zero  days. Uh this isn't an endorsement or instructions on how to shoplift anymore than the  lockpicking village is instructions endorsement on burglary. Uh you know we're hackers, we  talk a lot about security systems and how to defeat and bypass them and a lot of times  that's a purely academic and intellectual exercise and this is what this was for me and uh  yeah. At the end of this I'm gonna show you how to defeat the security system, you know use  use that power responsibly. And also this isn't going to be a 100 percent definitive tear  down of how this system works. This is reverse engineering, you know it was limited in  scope. I probably got some things wrong and if there's any Rokitech engineers out there  listening to this you'll be like oh no we didn't do that at all, you got that wrong and yep  that's the nature of the beast. But at the end of the day I I figured I had to walk out of a  shopping center with a out the cart locking up and I consider that a victory. So who am I?  Uh I'm a hacker, I'm a robotics engineer, I don't know why that's cut off for some reason but  uh in my previous life I was a robotics engineer uh uh I worked in a lot of startups uh with  embedded systems and electronics. Uh I jump out of airplanes for fun, I freak shopping  carts and uh I really question has anyone ever seen the other side of the moon? And who I'm  not. I'm not an expert at some of these things. I'm not a binary reverse engineering expert.  You know I do pull some code off of uh the microcontroller and I do open up Engadier and look at it.  It's pretty basic so there isn't a whole lot to do there. You know maybe someone with a sharper  eye than me could see some stuff that I missed but that wasn't instrumental to getting to the  end result but I still think that's an important part of the journey to look at. I'm also not a  radio engineer or an electrical engineer and there's equally some stuff I missed there.  Again, I'm also probably not 100 percent correct here so if you see anything that I missed or I  said anything that's completely wrong, you know I've got my email address at the end of this,  shoot me an email, I'd I'd love to learn stuff. So I talked about how 2 years ago I I talked  about these wheels and I want to give a super quick recap. So for those of you who aren't  familiar with security wheels on shopping carts, they're basically a wheel, they're usually on  either 1 or 2 of the wheels on a shopping cart and they have some mechanism that can lock and  unlock the wheel to immobilize the cart. Uh usually these are used to prevent you know  shopping carts from walking off in urban areas where they're connected directly to sidewalks  but uh some companies advertise that you can have anti-shoplifting features, you know make  sure somebody went through the checkout lane before exiting the store otherwise it'll lock.  And that's one of the big things Rokitech advertises and that we'll be looking at. Uh there's  2 main players in this system, uh gatekeeper systems and I've seen them all across the  country. Uh and then Rokitech which is a smaller player that I've only seen in 1 place, we'll  talk about that in a second. But they have some major commonalities. The big one is that  they have a buried perimeter wire that's broadcasting a radio signal at a very low  frequency. Uh usually around 10 kilohertz and that's a big pain to transmit at, we'll get  into that later but uh it's very similar to those invisible fence systems you'll see for like  dogs. Uh they can receive lock and unlock commands at these ultra low frequency bands as well  as some of them have capabilities of receiving some commands at 2.4 gigahertz which is a  much friendlier band to broadcast at. So they have some locking mechanisms uh these are  both 2 gatekeeper systems wheels, 2 different styles, 1 of them has a internal wheel that  will expand and lock the wheel like a clutch and the other one has a plastic piece that's  spring loaded and will come down and cover the wheel and prevent it from rolling forward. And  their Rokitech has a slightly different one, we'll get into that during the mechanical  teardown. But uh this is a picture of the buried wire system that you'll actually see in the  parking lots. Uh a store nearby me was having their parking lot and sidewalk redone and uh  this wire was nice and exposed for me to uh look at and play with so that was uh pretty  fortuitous but this is what it actually looks like you know underneath that seam you'll see  in the parking lot. Or inside supermarkets you'll sometimes see them integrated directly  underneath like the tile uh you know for like the per check system. And last this is this is  the real fun part of the last talk is the methods I used to capture these signals. So these  signals are in the tens of kilohertz range. That's ultra low frequency you know anything  that we normally deal with is almost always gonna be over at least a megahertz so that's a  couple orders of magnitude lower than we're used to dealing with. And it gets weird dealing  with that. But one of the uh interesting things we'll notice is uh 10 kilohertz is in the  audio range if that were sound waves we'd be able to hear it. So we need some sort of  electronics, we need some device to convert the radio waves into electronic signals and then  something that can deal with 10 kilohertz electronic signals. I don't know like an audio  amplifier. So uh to do this I made a loop stick antenna which is a pretty basic kind of  antenna. They're not very good at transmitting but they're okay at receiving. And uh it's a  ferrite core and some enamel wire both you can grab them off Amazon. Put into a cordless  drill and uh put on some Netflix and just wind the antenna. You know the more loops you  have the more pickup you're gonna get. Uh you have to do a little bit of balancing for the  uh impedance matching and what not. But you can find a ham radio handbook if you really  wanna deep uh dive deep into that math. And uh you take this and you solder it to a  headphone jack. Uh you'll see there's a little resistor that's hooked up there and what  that does is it actually convinces your phone or audio device that instead of a headphone  jack it's a microphone jack. So you can use this to record. So I had to dig through my old  electronics bin to find a phone that actually had a headphone jack. Anyone remember those? Uh  but eventually I found one and uh this is the method that I was using to capture the ultra low  frequency signals. And we'll talk about how to uh broadcast those later. So I mentioned  there's a new kind of wheel and that's Rokotec. Uh how I came across this involves a funny  story with my dad. Uh so my dad's a weird guy as you can imagine if you know me. Uh and he  loves collecting curios. He he recently retired so he's had a lot of time to scour eBay and  local garage sales and uh this is this is his workshop and uh you can see all the fun stuff  he has. And yes that is a functional phone booth in the back. He's got 3 of them. Uh and he  no he doesn't realize how cool that would make him here. He this is completely unrelated to  hacking. He just thinks they're cool and I love him for that. Uh but anyways so one day I  get a call from my dad. Hey Joe uh I got one of those shopping carts that has the smart  wheel but it's locked. Any chance you could help me unlock it? And I'm like yeah sure this is  my thing I guess. Uh so you know I I head up to my parents house and his garage and I he he  not only has the wheel he has the whole cart attached to it. Uh which I I don't ask where he  gets these things and I I think it's better that way. But uh I take a look at the wheel and  lo and behold it's one I've never seen before. Uh and no wonder you know he's trying to use  the website that has the little buttons to unlock it and it wasn't working and he was  wondering why. I said well of course cause I haven't seen this before. So I uh go and look  it up and uh that's how I discovered Rokitech. So this is what the wheel looks like and you  can see uh some of their marketing material. This was actually I took this from a store, a  grocery store near my house. Uh you know Rokitech focuses a lot more heavily on the anti  shoplifting angle in their technology than gatekeeper systems which is more about  retaining your carts on property. Uh so just slight different uh uh approaches but  gatekeeper systems does also advertise that they have a similar system to what we're gonna  talk about but I haven't ever seen it used. So now my curiosity was piqued and also I had  to unlock this cart for my dad. Uh so I need to do some research. There's a couple great  wells of information I go to when I'm trying to do OSINT and uh so publicly inf- publicly  available information uh FCC filings always a huge goldmine. If you have anything that's  gonna be broadcasting, receiving, anything with radio stuff, you have to disclose a huge  amount of information to the FCC about the frequencies, the bands and usually it's easier  just to include like the whole instruction manual cause it's gonna have all the info they  want. You know if you care about security you can you know carefully redact it and only  have what you need but in this case Rokitech had quite a bit of information. Uh what you  can also look at is patent information. You know a lot of times there's very technical  descriptions of how systems work because that's what a patent is. Uh and in return for that  prote- disclosure you get protection but disclosure is disclosure and that can tell us a lot.  And lastly is marketing information. Companies tell us a lot about how things work when  they're trying to get us to buy things. And uh we'll talk about what we found there in the  next couple slides but uh then afterwards for further research you have to do some field  research and get your hands dirty. So that's physical disassembly, taking RF captures and  some reverse engineering. So in terms of marketing material uh uh this is from one of their  brochures and this looks pretty standard and doesn't seem like it has a lot but it has a  couple things that stand out. One uh it claims that it has an RF receiver that can up and  download data. So that immediately people are like alright what data are they uploading? What  data are they you know recording? Are they cause you could be tracking alright where is this  cart going in the store? You could be tracking shopping habits etc. Uh so that piqued my uh  interest and in terms of thinking about some of the design assumptions, it's also battery  powered and it lasts very long. Uh that seems a little silly but what that tells me is it's a  sealed battery system, it's not meant to be serviced or easily replaced very often. So it's  probably going to be designed and optimized around saving power rather than say security  because you know if you're doing encryption every time you send a signal, that's a lot of  CPU time and if you only have that little unrechargeable battery, that's life. So you see a  lot of design decisions and a lot of security decisions get made and eschewed for other  technical design reasons and that's a big thing that you can look at in reverse engineering.  So moving on to the FCC filings uh so the wheel itself had a FCC ID number on it and the first  3 or 5 letters of that are actually common to the company that makes it. So you can punch in  that first 5 and get a list of all of the FCC filings that Rokitech has ever made. So from  this you can pivot to seeing stuff like this little box here which is what they attach to the  checkout counter and when you the payment goes through it sends the signal out and that's  what tells the cart you're allowed to leave. So this is the FCC filing for it and I've taken  out a couple snippets of it. Uh the big part it has uh a couple information up above there it  talks about you know the 2.4 gigahertz of signal, the B signal it emits and it also talks a  little bit about the test procedure. Now this is useful for the 2.4 gigahertz but I also  found the one for the box that actually connects to the buried loop wires and sends out all  of those signals and uh it tells us right there the exact frequency that it's using, 8.13  kilohertz and it also tells us that there's 3 different 8.13 kilohertz signals, the A  signal, the C signal and the H signal. It also tells us a little bit if the relevant shopping  cart casters receive 2 A signals and lock. So alright if it gets 2 of those signals it'll  lock, that's probably the one at the entrance of it and we can start figuring out how this  works and later on we'll turn this into a flow chart and start looking at how we can bypass  it. So I've talked about loop stick antennas, I've talked about the frequency, how are we going  to actually capture and process these? So we can do this in the audio frequency, we can use a  loop stick antenna to capture it and we can actually because it just comes out as an MP3  file and those radio waves get encoded as sound, we can open up an Audacity like and just  treat it as a regular sound file and then we can play it back as a radio file. So we have all  the parts we need, it's time to go war carting. So this is my little setup, I've got my laptop  here, I've got a SDR to capture some of the signals and I've got a loop stick antenna connected  to my phone, a little dab of hot glue so nothing comes out cause uh 3.5 millimeter headphone  jacks did have some flaws. Uh can't do this with Bluetooth though sadly. But anyway so I had  this all running on a audio recorder app and I had a GQRX recording from the SDR for the  uh megahertz and gigahertz signals and uh I put this all into my war cart and you can see the uh  the smart wheel in the corner there and uh just put my backpack down in there, you know  completely innocent, it's where I keep my reusable shopping bags and my reusable shopping SDRs  and all the important stuff there. Uh so what my plan was, my plan was to go into this  grocery store, engineer a situation in which my cart would lock, you know completely  innocently, feign ignorance and wait for the kind clerk to come up and with a remote unlock  the cart for me and at which point I would capture a copy of that unlock signal and I'd have  what I need. Uh with the gatekeeper system thankfully uh uh I was able to buy a couple of the  remotes the stores have uh on eBay. With Rokitech being a bit more obscure I wasn't able  to so I had to go and find these signals a different way. So that was the method and uh it  worked. So I went in uh used the Coinstar machine for a bit, oh no I should probably get a  bigger cart before I go shopping and went back to like the little cart corral area and the  cart locked and the alarms went off and uh the store clerk came rushing out, super  apologetic and I was like no it's okay I promise. Uh but anyways I got it. So this is, you  can see there's a couple different bursts but towards the right there that's uh what the  unlock signal was and we're gonna clean that up in a little bit and look at that a bit  closer but uh as a result of this little war shopping experience I got a good amount of  signals. So I got a copy of, there's a perimeter lock signal around the parking lot itself  so if you try to take the cart out of there the cart will just lock automatically. There's  the arming lock signal which is at the entrance of the shopping area so you take it, bring  it into the market, it gets armed and then if you try to bring it out again it'll lock.  There's obviously the unlock signal which can unlock a locked cart. There's a megahertz  level alarm mute that can mute the alarm uh uh once it gets set to that Intellibox we just  saw. And then there's the per check signal and that per check signal is what the uh uh  checkout counter is sending to the wheel to say hey it's been paid for if you once you get  the uh exit signal you know don't lock. Now you can see here in GQRX I did capture a copy  of the 2.4 gigahertz per check signal. Uh I'm not gonna go too deep into that for reasons  I'll talk about in a second. Uh there actually wasn't a whole lot happening at 2.4 gigahertz  but this signal is also repeated at the kilohertz frequency and that's a lot more useful  for what we're going to do. Uh both because it's paradoxically easier to transmit at  kilohertz for what we're gonna do and because a gigahertz capable SDR is slightly  annoyingly expensive. You know you have to get a hack RF and you know it's you you can't get  that high with the uh real cheap SDRs. So this is what the unlock signal looks like. It's a  8.15 kilohertz tone. Uh it looks like it's kinda broken up a little bit but uh in Audacity I  recreated it just from a pure tone and it was just a solid noise and that consistently  unlocks the wheel so that's uh that appears to be what it's listening for. The arming signal  is uh again at 8.15 kilohertz but it's 50 millisecond length pulses at a 33% duty cycle. So  it's just kind of pulsing that. And you're gonna notice uh all of these signals are pretty  basic. You know they're either solid tones, pulses or very basic shifts. And I'm gonna talk  about this at the very end when I speculate to some of their design decisions but I I think  this is a very intentional design because it's easy to do with low processing power. The  cost is security and that's gonna let us do some fun things later but uh this is the lock  signal. This is a 7 kilohertz pulse, 30 milliseconds with a 50% duty cycle. And then uh this is  the one that I really wanted to get, the per check signal. So you can see there's a couple  things I actually uh uh checked out, bought a pack of gum to do it again so I could see if  there was any difference you know between subsequent ones and the answer is no. And if we uh  adjust the contrast a bit and zoom in we can see what that looks like. Uh it might be a little  hard if you're not used to reading spectrograms but uh what you can see is there's a couple  lines that are parallel and partway through they dip down a little bit and then go back up.  And that's kind of repeated like a ladder across a bunch of different frequencies. So what  this is it's a tone and I'm not sure what frequency it was originally intended to be at.  Things can get a little weird with harmonics but it's a frequency. It drops down by 250  hertz, goes back up. And uh that is the signal that it's allowed to leave. And now if I can I  actually have what that sounds like uh if you you know this was a recreation of it so I  cleaned it up a lot. It was very very faint in the original audio but if you take that  pattern and you uh retune it it sounds like this.  And that's all uh it takes. Okay. That's enough. Uh and that's all it takes to unlock the  shopping cart. So let's get a little freaky. And for those of you uh unfamiliar with what  freaking is it refers back to uh a famous hacker John Draper who figured out that you could  use the toy plastic whistle in a Captain Crunch cereal box. If you played it into an old  school pay phone it was the right frequency as some of the control codes and it would let  you basically access admin mode. And so freaking is kind of one of those fun little you know  using unconventional signal methods to get control that you shouldn't have. And I I think  this is adjacent enough to it where uh uh we can call it that. So a couple things let us do  this really easily. These signals lack any sort of authentication or security. It's the same  signal each time. There's no rolling code. Nothing gets incremented. It's the same each day.  There's no time stamping. Uh and again this was probably done because all of those things  are computationally expensive and every processor cycle takes a couple joules and you have a  limited amount of those in your lifetime and you know if you can make the difference  between a year uh a wheel that lasts 2 years in the field versus one that lasts 5 years  that's huge from a marketing perspective and you know unfortunately a fact that's not uh at  heart to people who aren't in this room is security isn't number one. You know there's so  many other business factors that go into these design decisions and uh a lot of times  security gets put in the wayside. You know I don't think their threat model here was super  sophisticated. I don't think they expected some random nerd to get real curious and reverse  engineer it so I think it could be forgiven but you know you can see the impact of other  factors in design decisions into the security of a system and you know in my day job when I  do assessments of commercial devices you see this a lot more and it's a lot more  pronounced. You know you can start to see corporate politics in in security decisions. So  when we're talking about freaking these wheels we need to talk briefly about what parasitic  EMF is. So whenever you're putting current through a wire you're creating a magnetic and a  radio field. They're kind of the same thing it's a whole whole physics thing but basically you  put current through a wire you're creating radio waves. Uh and normally that's a bad thing.  You know you're putting energy into the air that's not going into the thing you want it to and  uh in the case of a speaker coil you know that's energy that's radio waves just a tiny amount  but that's not going to you know moving your membrane and creating sound. But in our case  since we're trying to transmit a signal at the audio range when we play an audio file the  parasitic EMF is also in the kilohertz range. So if we play the right audio file through our  speaker at the at a loud enough volume and hold it next to one of these wheels we can control  the wheels and send the lock and unlock signals and that's what we're going to be doing. So I  had this successfully working with the gatekeeper systems wheels and uh uh the website I'm  going to link to at the end uh has buttons you can just push play for lock and unlock and it  plays the little audio and you hold it up to the wheel and uh I'll have a demo in the physical  security village if uh people want to try it themselves. But uh I was having inconsistent  results with these Rokitech wheels and I couldn't figure out why. So I needed to go deeper  and uh the only way to do it was physical disassembly. And I have no idea why these things  were on eBay but I'm not complaining. Uh so I got a couple samples that I could uh uh tear  down uh destructively if I needed to and really dive into it. So now it's time to get into  reverse engineering. And uh this is where uh uh the the concept of scope comes in. You know  this is a hobby project. Whenever you're reverse engineering something there's near infinite  scope available. You can go as deep as you want and deep as you have time for. Uh and when  you're reverse engineering you need to carefully select that scope. You know there were  some things that I found was a dead end and I I you know left them as it was. But if I wanted  to I could have gone a lot deeper. You know there's a couple mystery chips that I was like  alright that's what this probably is. And I could have fully traced out the PCB and made more  educated guesses but I didn't because I didn't want to and that's okay. That wasn't in the  scope that I set for this project. Uh you know reverse engineering can be sometimes like  looking at some some footprints and trying to figure out what the person had for lunch. You  know you're looking at some weird things and trying to intuit something completely  orthogonal and if you're good sometimes you can. And uh it wouldn't be a presentation that I  give if I didn't sneak in a Terry Pratchett quote. Uh and I I think this really speaks to the  heart of reverse engineering is what happens to people who ask too many questions? They  need answers and it serves them right. And uh we got some answers. So I got this wheel and I  started tearing it down and mechanically it it's pretty similar to other wheels. You know it  has a battery compartment, a sealed PCB enclosure, uh a sub mechanism to lock it. You know  you have a little locking latch that's controlled by a pawl and that pawl rotates, releases  that latch and then it interferes with that uh internal uh wheel and prevents the wheel from  spinning. Not nothing too crazy. If we look into that sealed PCB area we get to the circuit  board, the real area of interest. So we've got that inductor uh you know that's gonna be the  kilohertz antenna. It's basically the loop stick antenna that I made except it has much  better amplifier equipment designed by a real engineer. Uh we see that underneath it there is a  missing chip footprint. That's gonna be a lot of fun in a couple seconds. Uh we see the main  microcontroller, a PIC a PIC18F23K20. Uh of note is this chip doesn't have 2.4 gigahertz  capability. Uh so the 2.4 gigahertz capability they're advertising is kinda curious since  they need some sort of transceiver unless they're doing something really really basic and  weird with raw signal. But we see there's a header that will later turn out to be a  programming header and we have some wires that connect to the battery connector and the  motor connector. Looking at the back we have a mystery chip. That's gonna be fun. Uh it's  just labeled N100B and uh has DE4 printed on it, nothing else. Uh and we have just a little  uh tactile sensor that's used as a limit switch so it can tell you know when it's in the  locked or unlocked state. So now we get to use a technique that I really like using uh for  tracing out PCBs which is overlaying images. So we have the front and the back of this board  and we can see that the traces jump from the front to the back and it's a pain to trace out  with a multimeter and try to keep track of it. So for smaller boards like this uh uh you can  sometimes do it but even on small boards where you don't have to I find it's really useful if  you take good uh straight on photos of the front and back. You load it up in GIMP or  Photoshop and uh mess with the transparencies and you can get a really cool overlay. And so  you can kind of see where holes, where traces, where vias and everything align and what  components they talk to. And from here you can pull up the data sheet for components that  you can identify and you can start tracing out all of those traces and having that  transparent image and being able to flip back and forth is super useful. So in this case I  traced out these um uh uh pins and I found that that header had a VPP, VDD, PGD and all of  those lovely lovely things that people who work with microcontrollers will recognize as a  debug header. So the next step was to grab a debugger and see what I could find. Uh so this  is a PIC kit. Uh it is the standard debugger for PIC microcontrollers. Uh they're super cheap  on Amazon, like 20 bucks. I got one and hooked it up. Uh word to the wise, hook up all the  wires. I spent about 2 hours wondering why it wasn't working and I didn't have um uh VPP  hooked up properly so that was entirely on me. But uh you hook it up and uh you have to  hook up little tiny uh wires to everything. And then you get it hooked up, all of them hooked  up to the PIC uh PIC kit and you load up PIC kit 2 minus. Uh PIC kit 2 plus is the premium  software that has all the fancy features but PIC kit 2 minus is the free one that doesn't  but it has enough for me. Uh you connect it via USB, take care of the boot loader stuff for  your PIC kit, select your device, go through the rigmarole and if you're lucky, we're in. So  we get a couple cool things. We can see that we've got program memory, we've got some  configuration bits and we've got EEPROM data. Now a quick word about the configuration  bits cause this is also something I see a lot in the commercial world. Uh set your security  bits. You know microcontrollers have code protection. You can code protect it so people  can't pull off the code. You can write protect it so people can't overwrite, overwrite the  code you put on. You can even do external block table reads which I have no idea what they  are. Uh and you can see what they do in the data sheet if you look up that table and I'll  save you the trouble, none of them are set on this. None of the security bits are set. So  that means we can pull off the uh uh code from the microcontroller and open up in a reverse  engineering program like Ghidra. And interestingly there's only about 2 kilobytes of actual  code uh that I managed to pull off. Everything was set to uh OXFF. Uh it looks like it was  mostly the resets and the interrupts that were set and a couple other bits of code. So I'm  not an RE expert. I could be missing things. I had a couple friends who are look at it and  they also agreed that there was not a lot. So based on the fact that this is a really basic  microcontroller, I'm guessing the fact that it's really just handling the you know the limit  switch for the motor, powering the motor and then maybe doing some super basic signal  processing and then firing an interrupt based on that. And you know maybe a couple bits for a  state machine and tracking that. But it really seems like it's that basic and it's not doing  uh uh any encryption, any data uploading or downloading or anything like that. Uh and then  we've got this mystery chip on the back. And we've also got that mystery footprint of an  unpopulated chip. And I'll talk more about this in the speculation section but that's  probably where they'd put the 2.4 gigahertz stuff if they wanted to implement it later. But  uh from tracing it out early, I noticed that all of the pins on that mystery chip get broken  out to both the main microcontroller and also that unpopulated header. And that unpopulated  header makes it real easy to well easy to connect fly wires. Uh this is miserable work. Uh uh  this is my first attempt at it. I ended up having to completely redo it. But uh you take a  little 30 gauge uh uh wire or some uh uh enamel wire if you have it. You wrap it around and  you solder it real tiny. Use lots of flux and a small soldering iron tip. From there you can  hook it up to a Sele logic analyzer. A logic analyzer is basically um uh a whole bunch of  it's like an 8 channel oscilloscope. You know it's not as high resolution but it can do some  really cool stuff. And uh run it through the boot and see what we get. And we get some some  interesting stuff. This is in digital lonely mode. Uh and one thing I noticed is everything  was dead until I touched the wheel. Uh when I moved it, those lines you see the yellow, green  and purple ones lit up. Uh so they were sending something. And I was a little confused. Uh so  it looks like there's some sort of accelerometer and it's only listening for signals when it's  in motion. Again probably a battery saving feature but that would explain why I was getting  inconsistent results with my replay attacks. Because I had it on the benchtop with the  speaker and it wasn't moving so it was just ignoring any signals. But looking at some of  these signals, they're id- nearly identical. Uh uh slightly different but why would they  replicate them? So I switched it to analog mode and we see that they are very similar  waveforms but it's the antenna. I played uh uh a couple of the lock signals and we see  those waveforms and we see on the upper traces similar modified waveforms. My theory is  that they're doing, they're filtering in hardware. So they have passive components and  they're doing their radio signal processing and a couple of their uh uh mathematical  operations at the hardware level so they don't have to do it in the processor. You know if  they need to find the absolute value or the magnitude of it, you can do that with passives  and save yourself compute time. So we we kinda set the Sele to a mixed digital and um uh  analog mode and got a good capture of this is the whole boot sequence, you arm it, you hit put  at the arm thing and you can see it lock and it goes to this whole thing. Uh if you're  wondering if there's any serial data there, no there wasn't any serial on any of those lines.  Uh I think there is chip to chip communication lines there but because that chip is  unpopulated there's just no communication happening. Uh so with this we can make a flow  chart and that's gonna help us bypass this. So we have an idea of you know the basic  states it can operate in, we have an idea of the signals it's listening for and can listen  for and uh we can start thinking of bypasses. So I just put together a quick little flow  chart of how it works. Uh uh one big caveat is uh if it gets the arm signal within 30  seconds of the first one, it assumes that you know you brought it through and then brought  it back quickly and it doesn't lock and it goes back to idle. And that's gonna be  important. So looking at this flow chart we can see 3 possible paths to bypass this. One is is  the very obvious one where we just replay the per check signal to it and say hey yeah you're  okay to leave and you leave and I've tested this in the store you know with an empty cart and  uh successfully been able to get the cart back out without it triggering the alarm. Uh  there's other ways too. Uh the 30 second disarm can be exploited so you bring it out and  then you bring it up to the buried line again and then back it out and it's entered the idle  state again thinking it was back into the little cart corral area but it's actually in the  store and it's not gonna arm till you take it back out. Uh and then the last one and this is  probably the most reliable since the other 2 give you no feedback is to bring it through a  lock cycle. So you have a lock signal and an unlock signal. You bring it into the store, you  play the lock signal to force it to lock, then you play the unlock signal to force it to  unlock and then it's back in its idle state disarmed because it thinks it's already been  through it. And with that you have positive feedback knowing that you know it's working and  you know you're not just playing a speaker sound at it and hoping it picked it up and it's  not gonna alarm. So those those are the 3 main methods for bypassing this that I found. And  now as I promised it's time for a quick little bit of speculation kind of this is me just  shooting off the cuff, stuff I think they're doing or is going on uh uh that I'm not 100  percent sure on. So I mentioned 2.4 gigahertz and how they have it. Uh their FCC documents and  their marketing materials say they do but their PCB says they doesn't. Uh you'll notice  they have uh there's a little trace going around the entire perimeter for that and those are  usually used for gigahertz antennas. Uh but you'll also notice if we zoom in uh where that  connects to the board all of the passive components are depopulated. Uh that combined with  depopulated you know what looks like it could be a microcontroller. I'm going to guess that  they designed this to be able to at some point do 2.4 gigahertz but either for design  reasons, economics or battery saving reasons they said hey we can make this only work off  you know the 8.1 kilohertz range and we don't need to do this radio stuff. And maybe if a  customer did wanna do the advanced stuff and uh be able to upload and download data you  know they could have a revision of this that does have that built in but I haven't been able  to find this and I I've gotten a handful of these wheels off eBay from all different parts of  the country and none of them that I've opened up have had uh those chips populated or any  indicator that it's doing more advanced communications. So that mystery chip on the back?  That could be a a crappy little system on chip that's handling the 2.4 gigahertz. I imagine if  it were it would have been closer to that and those antenna traces. Uh my honest guess is  that's an accelerometer and that's how it's actually sensing when it's in motion. Uh a lot of  accelerometers have ultra low power modes that can send an interrupt signal when it's in  motion so you're not actually doing any communication or like reading and data processing  you're just hey was this in motion within the past couple seconds? And that's consistent with  what I'm seeing on the logic analyzer. Uh and the 2.4 gigahertz could probably be added  later but probably won't because that's gonna use a lot of power that you don't have. Uh I  wanna give a couple thanks and shout outs uh first of all to my dad for like bringing these  wheels to my attention to uh all the Defcon staff and goons for making this happen and for  somehow letting me on stage again to talk about shopping cart like 2 years ago you gave me a  stage and I talked about shopping carts for 20 minutes and like this year you guys have no  excuse. I told you what I was gonna do, you knew what I was gonna do but y'all still showed  up here and uh I'm real thankful for all of you. I'd also like to thank all of my friends and  loved ones some of whom are here uh some of whom are back home uh and we'll see this in  video. Hi mom, hi dad. Uh and then just everyone who's made all of the software stack that I  use this. The people who made Audacity, URH, Ghidra, thanks NSA uh and all the other tools as  well as just the random people on eBay who have these things for sale. I don't know how  people end up with these but that's how it goes. Uh this is just a quick references and uh  that's it. I'm going to have a demo in the physical security village. I'm gonna go set it up  right after this uh where we have I modified some wheels to be transparent so you can see  what's going on inside of them. Uh we have a website bgaydocrime.com slash carts. It has  little play buttons where you can uh play these audio files and uh if you want check us out in  the flamingo in the physical security village. We've got uh I've got 4 of these wheels, some  of the gatekeeper systems, 2 of the Rokitex and uh you can actually try this yourself with  your phone. Uh if anyone has any questions uh uh happy to answer them. I'll also be around  the physical security village if we run out of time. Thank you. Thank you. Thank thank you all  for coming. This was a lot of fun to work on. I had a lot of fun and I'm I'm just real thankful  to be here and that you all let me talk about shopping carts to you for so long. Alright any  questions? Bueller? Bueller? Alright well thank you folks. Have a good one. Uh if you have  any questions feel free to grab me after the talk. I've got pretty distinctive hair. Alright  have a good one folks.


url: https://www.youtube.com/watch?v=sdiHfVhPso4
title: DEF CON 31 - Visual Studio Code is Why I Have Workspace Trust Issues - Chauchefoin, Gerste
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=sdiHfVhPso4",
    "title": "DEF CON 31 - Visual Studio Code is Why I Have Workspace Trust Issues - Chauchefoin, Gerste",
    "description": "Developers are threat actors' targets of choice because of their access to business-critical services. After compromising a single developer, they could push code changes or obtain sensitive information. For instance, a recent campaign attributed to North Korea set up social network profiles to social engineer and infect prominent figures of the developer community with malicious Visual Studio projects and browser exploits. At the same time, modern development tools offer increasingly advanced features and deep integration with ecosystems, sometimes at the cost of basic security measures. Code editors tried to counterbalance it by introducing new lines of defense (e.g., \"Workspace Trust\"), leading to a cat-and-mouse game to restrict access while keeping most features available by default. In this talk, we present the state of the art of Visual Studio Code's security. We go in-depth into its attack surface, how its extensions work, and the technical details of two vulnerabilities we found in Visual Studio Code. These findings, CVE-2021-43891 and CVE-2022-30129, led to a $30.000 bounty with an unexpected twist. We also present 1-days discovered by other researchers to develop the audience's intuition. These concepts apply to most IDEs of the market so everybody will now think twice before opening third-party code!",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2305,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230915"
  }

  This text was transcribed using whisper model: large-v2

   Hello, thank you for attending.  Yeah, first DEF CON, everything is impressive.  The size of the room is also impressive.  So we have Paul and Thomas, who are two VMT researchers  from the SONAR R&D team.  Paul, to you.  Hi, I'm Paul.  And yeah, I come from the CTF scene.  And in general, I like to break JavaScript things.  So VS Code is a very good place for that.  And on my side, I'm more like a PHP guy.  It's been giving me a job in the offensive security  side for like six years.  I always keep on giving for the next decade.  And I also like to do some exploration  in the memory and safety world from time to time.  And our company, SONAR, writes static analytics tools  to help developers write clean code.  And it's like a spell checker, but for code in general.  And we use zero days that we find ourselves  to fuel product innovation.  We found more than 150 zero days over the last two years.  And we were recognized by a nomination for the Bernie  Potts Figure Top 10 Webpacking Techniques.  And we also play point one from time to time.  And because the journey of most developers  starts in the code editor, we're wondering  about the security of these code editors.  And everybody has usually strong opinions  on their favorite code editor.  Like Paul likes Nano, which is weird.  Don't tell me.  But it's like 8% of all developers  pretend to use Nano for some reasons.  But sometimes, you need to get the job done.  And you start using bigger IDs like Visual Studio,  the N2J Suite, or VS Code, which is the most popular one  with like 75% of the market share.  So let's start to do something a bit interactive with VS Code.  You've been cloning some Git repo online,  or somebody just sent you some source code.  And you want to open it in VS Code,  because it's your job as a developer  to read and edit code.  So please, raise your hand if you  think that nothing wrong is going to happen.  So next slide, the conclusion and talk is over.  Nobody, right?  And raise your hand if you think something bad  may happen in this context.  OK, so some people are not sure.  So I'm going to show you a quick demo  to show you that it's real.  So we just received something.  Oh, sorry.  Maybe I can zoom in a bit.  Yeah, OK.  So you just received something, or you just  Git cloned something on your desktop.  And it's just an open me, please.  Just like some HTML file with a Git repo, nothing sensitive.  So I go on VS Code.  I open my folder.  I get a security prompt popping.  But it's calc before this prompt.  I didn't do anything.  So something's fishy.  And we're going to get back on it later.  But it's definitely unsafe to do some things with VS Code.  And this is what we want to show you today.  What's wrong?  There is no standard threat model for developer tools.  We don't know what's unsafe.  And before doing this research, when we found this thing,  we were kind of surprised.  Is it a normal thing to do?  Is it supposed to be unsafe to read somebody else's code?  Because I mean, it's my job.  And every day, I just open random people's PHP code  that I find online.  So if I do it, does it weaken the security of my system?  Can I open somebody else's code without any risk?  Does my IDE run my code?  It just happened with GDB decompiler.  It's used by malware analysts.  But in practice, like in the background,  it would run the code you would decompile,  which is not something nobody wants, I guess.  And also, when you have fancy features like VS Code's  remote development, do you know if it's safe?  Microsoft doesn't say anything on this record.  And also, I think some background  is that we have more and more threat actors' campaigns  against developers.  I would be a threat actor.  I would try to target developers.  It would be an easy target.  They do plenty of things all day with everybody else's code,  with dependencies and stuff.  And recently, we've been seeing threat actors  using Plex one day in the LastPass campaign.  So they compromised DevOps through their Plex media server  instance.  And then I think they used some key logger  to get secrets to access production.  And they compromised everybody's password vault that way.  There was also a Google tag.  So the threat analyst group, they  reported a North Korean campaign.  They say it's North Korea.  And somebody would reach out to some prominent infosec figures  on Twitter and say, oh, yeah, I have this exploit.  I can build it.  Can you help me to do it?  And when you're a nice internet citizen,  you're like, yeah, I'm going to help you.  And if you build a project, it would compromise you  and run some weird PowerShell things in the background.  So it's definitely a thing.  And with this presentation, we want  to show you around the different data  interfaces of these IDs, including VS Code.  It's going to be about VS Code anyway.  Paul will start by showing you how the software is  architectured.  You will see it's a huge beast.  So it needs some background understanding on how it works  and how it's developed and designed.  And then we can start by looking at the most common sources  of risk that we've seen in VS Code  in the core and popular plugins.  And it's based on our own research.  So we will tell you when it's all on bugs.  And also some research done by external researchers.  And we'll credit them every time accordingly.  And then we will look at the reporting process  from Microsoft.  We have a fun anecdote to share.  And most of the bug you will see requires  some degree of interaction.  Like you've seen, I've just opened a folder in my desktop.  And I think it's expected from VS Code.  So we still call everything RCE, because the attacker is remote.  But it would be more like something  like arbitrary code execution that's  done by somebody who's remote, but that's  being carried on locally.  So we call it RCE, ACE.  It's roughly the same thing in this context.  So Paul?  Thank you.  So let's look at how VS Code is built.  First of all, it's based on Electron,  which is basically Node.js and Chromium meshed together.  So it's written in TypeScript and with a little bit  of HTML, CSS, and all the other web technologies  sprinkled on top, which means it can also easily work  in the browser.  So there are two web-based versions of VS Code,  probably even more, on github.dev and vscode.dev.  And they are basically the same experience.  Some things are different, but that's the gist of it.  VS Code is highly extensible.  Everybody can build extensions for it.  And you can also publish them to the marketplace,  where everybody can install them from and search through them.  And one cool thing that came up during the development  of Visual Studio Code was the language server protocol.  It's basically a way how you can write support for language  once, and then all editors that support this protocol  can reuse it instead of having to code it  for every editor again.  VS Code is mostly open source.  It has about 800,000 lines of code in its GitHub repo.  And when you download the version from Microsoft  directly, it has some small proprietary parts.  But you can also download the full OSS version if you want.  Let's look at the components that VS Code is made of.  So first of all, it isolates and splits different things  into different components.  And it uses processes for that.  And there are some privileged processes.  You can see them in red.  There's, first of all, the main process.  This is the one that starts up first  and orchestrates everything else.  Then there's the shared process.  This one hosts the PTYs, the terminals  that you can open in VS Code, for example.  It hosts the FileWatcher, Daemon, and other utilities.  And then there's the extension host process.  And this one, as the name says, hosts  whatever extensions want to run in the background, which  is not UI.  And then there are less privileged parts,  which are the renderer process.  So this is the Chromium part.  And it's basically just like your normal website.  It cannot really access directly the file system,  execute commands, or whatever.  It's just rendering the UI.  And this is how it looks.  I'm sure many of you have seen it.  And the big outer red part is called the Workbench.  This is basically the core UI of VS Code itself.  It's served from the VS Code file protocol, which  is a special protocol.  We will see that later.  And then, for example, an extension here,  the Markdown Preview extension, can also show some UI.  And this will be in a web view, which  is similar to an iframe.  So it's kind of isolated.  And it also comes from the VS Code web view protocol.  So already there, you see that it's  coming from different things.  There are some isolation things going on.  But yeah, if one of the parts, for example, the extension,  wants to directly access the Workbench,  then this will not be allowed by the same origin policy,  because they come from different origins.  And yeah, this is something that comes from the web  and also applies here.  But of course, sometimes the different parts  have to communicate.  And for that, there are different IPC interfaces.  So between origins in the renderers,  there's window.postMessage, which is a regular web message  passing interface.  Then if a UI part, a renderer process,  wants to talk to some of the utility processes,  they can use message ports to talk directly there.  Or if they want to talk to the main process,  then there's the good old electron way  of using preload scripts and the context bridge.  So of course, to hunt bugs and to find them,  you probably want to debug VS Code,  because it's just easier to step through instead  of having to put printf everywhere and recompile.  So for the UI part, you can just use the Chromium developer  tools like you know them from Chrome or any other Chromium  based browsers.  And with that, you can debug extension UI and also  the Workbench UI.  If you want to debug the back end part of extensions,  you have to start VS Code with the dash dash inspect  extensions flag.  And then you can just attach a Chrome DevTools instance  and debug it normally.  Or the easiest way is just you use VS Code to debug VS Code.  It's easy, because they have pre-made launch configurations.  So you just clone the repo, open it in VS Code,  and then you will see in the Launch Conflicts tab  that you can start different processes or everything  together under debugging.  All right, now that we roughly know  how VS Code looks in the inside, Tomas  will start with the first attack surface, which  is exposed network services.  TOMAS LIEBERMANN-WILKIEWICZ So I think it's a bit generic.  It's something you would find in any desktop application.  But in the case of VS Code, it happens a lot.  You may have services that start listening on the network.  You may start wanting to expose a development  server to locals.  You may want to expose the server to everybody.  You may want to expose a debugger  or just to communicate with other components on the system.  And while you would, in theory, to be safe,  you would rather use some OSI PC mechanisms,  it's still JavaScript code, TypeScript code.  And I think people developing VS Code extensions  are maybe web developers or Node.js developers,  but they are not likely to be system-level developers.  So they never really use OSI PC mechanisms.  And in practice, even if you want  to expose a debugger or a local server,  even a Unix socket could be enough.  And it would be safer to not expose anything.  Because if you start exposing pods, even a local host,  there may be a website when you browse.  They may start doing requests to these pods.  And even if it's listening only on local hosts,  you could do C soft attacks.  You could use web sockets to connect to a service.  So listening only on local hosts,  it's not a good solution either.  If you want some examples of CVs that were in plug-ins,  there was one called Rainbow Fart, which  is supposed to play sound when you type keywords  on VS Code, which sounds silly.  But still, they got 128,000 installs.  So people are using this module.  And Kirill Efimov from Snyk found  that it would expose an HTTP port on port 1777.  And if you go on this page, you would  be able to upload a zip file to change to sounds.  It's playing when you type on VS Code.  And there would be a zip-based pass traversal.  So basically, it would browse on some random website online.  And it would do a C soft attack to force  you to send a zip file exploiting  this vulnerability on this page.  And then you would be able to override any file  on the local file system.  So you would override the developer's .bashrc file.  And you would be able to execute arbitrary command from here.  There were also more serious examples in the core,  directly on VS Code.  If you remember, Paul told you you can use the dash-dash  inspect option to expose a Node.js debugger to the outside  to debug VS Code.  And it was enabled by default on VS Code at that point in 2019.  And it was reported in one key by Tavizo and PHRAAAA,  kind of like at the same week or same month.  And in this case, this debugger was  started on localhost in a random port  every time you would start VS Code.  But it's a debugger.  It's meant to chain variables.  It's meant to run code on your behalf.  So if you'd be able to reach out to this port,  you would kind of gain RC easily.  In practice, it's not so easy.  Because the way the Node.js debugger works,  you first have to do an HTTP request  to find out kind of a magic value with a WebSocket URL.  And then you use WebSocket to connect to this destination.  So you cannot just make a request to a local port,  because it's not the same origin.  So you would first use some DNS rebinding to access the service,  get the WebSocket URL.  And then since WebSockets are not  subject to the same origin policy,  you would be able to connect to this thing.  I don't remember seeing any complete exploit for it.  But I think it could have been exploited in that case.  And now we can start to spice things up and dig  into things that are more specific to VS Code  in this case.  Another very interesting feature of Electron  is that it provides a native layer  to work with the underlying operating system  for a better integration.  So you also need it, because you need file access, network  access.  But you can also expose things to the outside.  It's a pretty nice form of IPC that's  not relying on the network.  So VS Code registers is VS Code colon slash slash protocol  handler.  And if you have a VS Code insider,  which is an IP version of VS Code,  it's VS Code dash insiders colon slash slash.  And when you click on one of these links,  the OS gets the event and says, OK,  do I have anybody who registered for this one?  In the case of VS Code, it's going to be VS Code.  And it wakes up VS Code and say, oh, there's something for you.  And so this is one of the bugs that we found  in the Git's built-in extension.  So it's called an extension, but it's built-in,  enabled by default, and you cannot remove it.  It's part of VS Code, but it's still called an extension.  And it exposes a feature that way  to let you clone repositories when you click on a link.  So when you use GitLab, for instance,  when you want to clone something,  it gives you the command line with git clone.  And you can also have this Open in your ID button  that will automatically bring VS Code in front  and clone the repo for you.  So in the way it works, the Git extension  implements a class called UI handler.  And it handles the request that comes  from the operating system.  If it finishes with slash clone, it  calls a method called clone with the full address  that you clicked.  And then it passes URL and calls ID command  called git clone, which is something you can also  invoke yourself when you press Command-Shift-NP.  You can also type git clone, and it's  going to run the same thing.  And then internally, git clone calls something  called clone repository, which is only  a wrapper around a git command.  There is no reason to implement git in TypeScript  just for this feature.  So they simply call git clone for you.  And as you see in the web square, if there is no space,  it puts the URL as is right here.  So there is something interesting here,  because we kind of control the way an external command is  being called.  And there is no risk of command injection here  because it's not being executed in the shell.  It's directly calling git.  But you can still add arbitrary arguments to git.  If you put git dash dash help, it's going to display the help.  In this case, it's a git clone.  So the green part is what's fixed,  what's constant because of VS Code.  And in red is what we control.  And we found out that if you put the option dash u for upload  back, it's going to tell git to, next time you  have to clone something, use this command instead of trying  to use your own internal commands.  So basically, it tricks git into running a command of your choice  instead.  There are some restrictions for exploitation,  which are not that interesting.  It's kind of like CTFE.  You need the colon to trigger this command,  because otherwise, git thinks it's a local repo.  It's not a remote one.  And it's not calling the upload back command.  And you need to avoid spaces to not be well-encoded.  So this is what the final payload looks like.  We have this VS Code colon slash slash thing, VS Code git,  clone, we pass the URL, dash u, and then we  do some tricks to not have spaces.  And the resulting invocation is this git clone dash u thing.  So I'm going to show you what it looks like in practice.  And let's say you go online on some GitLab self-assisted  instance.  You want to use this feature.  So you're going to clone.  You click, I want to open it in my VS Code.  It prompts you, do you want to clone?  There is no way for me to check what's in there.  So I have to trust it.  So I'm going to say yes.  I put it on my desktop.  All right.  And once again, we have this calc that just popped up.  So yeah, kind of unsafe in the end.  And they were a very similar finding, I think,  a year before by Smurry from Shilder.  Also an argument injection when you  would click on links for the remote development extension.  I am not showing code because it's closed source  and we didn't spend time to dig into it and make it readable.  But it's basically the same thing.  The host part is directly being put in the command line,  and you can inject arbitrary arguments.  And you would exploit it with dash o proxy command.  And it's quick at break.  I love argument injection bugs.  I think, Paul, you're getting used to it.  And we found many, many argument injection bugs over the time.  So we created a page that lists all the small dash u, dash o  proxy command, all these argument injection vectors  that's going to help you to execute arbitrary command when  you find argument injection bugs.  So it's free, it's open source, and it welcomes contribution.  So if you go on this page and you find out  that we missed good tricks, feel free to add yours.  And now we can also dig into something  that's even more specific to VS Code,  and that's Workspace Settings.  There is a support for Per Workspace Settings  that comes along your source code repo.  So if you want to share settings with other developers,  like linter configurations, like things to do  and you want to share with everybody else,  you can create a file named .vscode.settings.json.  And it's a way to share the settings,  and it's automatically loaded when you open the folder.  So the question is, if you put your security research hat on,  maybe can I override sensitive settings?  Like, are there maybe sensitive settings?  And in 2017, so a long time ago, Justin Steven  found out that there would be an option called  git.pass, also from the Git's built-in extension,  that you could change to say, oh, no, Git,  it's not in my slash user slash bin slash Git.  It's somewhere else.  So if you ship a folder, a project, and in settings  you override this setting, the git.pass thing,  you could force it to invoke other commands instead.  And as soon as somebody would open a folder,  it would be loaded, and Git would run,  and it would call this other command instead.  There are also some interesting tricks for the exploitation,  because it would have to survive a first command, dash,  dash, version, location, from the vscode install folder,  and then it would be called from the project folder.  So what it did, it was to set git.pass to bash.  So the first code works, and for the other one,  it would plant a file called revpass in this folder,  and it would be called instead, and it would run its payload.  And there were also another example  of a remedy that would use local data directly  found by Didi-Walker in 2020.  When you do Node.js development, you  would use npm to store all your dependencies.  And to this end, you write everything  in a file named package.json.  And if you kind of hover a dependency on this file,  the npm plugin kicks in and tries to fetch information  and to show you information, like the description  of the package, how many installs, everything.  And it would call this command and safely concatenate  the name of a dependency into command,  and there would be a command injection in there.  It was later bypassed by Justin Steven,  again, because they tried to fix it.  They started validating the name of the dependencies,  and it's a fail-open thing.  So basically, if you don't match what  they expect you to be, something unsafe, it would say,  yeah, it's safe.  It's like it's a block list approach instead  of an error list approach.  And it was once again proven to be not the best way  to fix the vulnerabilities.  And now we can get to Workspace Trust, which was also  part of the name of this talk, so I have to explain it to you.  It was a feature introduced in 2021 in VS Code.  Its goal is to reduce impact of malicious folders.  So everything you've seen before,  where we would run arbitrary commands  and where we would use local data,  could be kind of blocked by this new trust-based system.  And you would have a new security assumption.  You would say, all untrusted folders  are safe to open in restricted mode.  But as soon as you start trusting something,  it's not a security bug anymore if somebody executes commands,  in this case.  So trusting a folder is always unsafe,  and untrusted folders are supposed to be safe.  And this is what it looks like.  So this is a prompt that says, do you  trust the author of this file in this folder?  And it pops every time you open a folder.  So you may be influenced and say, oh, I  want to click Yes every single time,  but it's maybe not the best idea.  There are some documentation that say,  this is what you risk if you press Yes,  but you would not expect the No to be unsafe.  And the way it works on the Zoot,  all extensions, including built-in ones,  declare their own capabilities in their own package.json file.  And the default value is false.  That means this extension won't run in untrusted workspaces.  So you miss features from VS Code  just because you clicked No, I don't trust.  So you're not living the life with VS Code  with all the fancy extensions.  And you have a very limited IDE, which is maybe not  what you were looking for in the first place.  So they also introduced something called Limited  that lets extension say, OK, in this case, if it's untrusted,  I'm not going to run this command.  I'm going to remove a few of my features that could be unsafe,  but everything else will run.  And once again, the Gilt built-in extension,  I think it's really a gift for us.  It runs in untrusted workspaces.  I'd say it's supported, so that means  it's going to run in all untrusted workspaces  and untrusted workspaces also.  And when we were trying to see if there were security  issues with this extension, we found out  that Git, so the command Git, supports  several levels of configuration.  There is a system-wide configuration in etc.git.config.  There is a global one in your own folder in .git.config.  And there is a local one in .git.config  of the current folder.  Indeed, but cloning does not retrieve  the local configuration, nor the other kind of configuration.  They have to be files you created yourself on your system.  But you can put Git repos in archives.  So it would come with its own local configuration  that may be unsafe.  And Justin Steven and us, we collided on some research  where we tried to see if we would  be able to hide Git repositories in subfolders of Git  repositories.  So we would be able to clone these files when  you clone the main repository.  And in a Git config, you can have interesting directives.  So like the one we have here, which  is called FS monitor, which is a way to tell Git,  please run this command to see if there were some changes  since the last time you run this command.  The default one is really good.  But if you have a huge monorepo, if you have 1,000 files,  it's going to take some time to find out the differences.  So sometimes you may want something  more specific for better performance,  and not like in the IDE or when you get in your developer  shell.  So it's not specific to VS Code.  And we found out that basically planting  malicious configuration in Git folders,  it's pretty easy to trick people into opening these folders  and to force them to execute arbitrary commands.  In fact, the first demonstration we've shown in the intro  was based on a malicious Git folder  that could have been just a zip file or a sub-repo.  So this was a command, which was a calc.  And we have the Workspace Trust thing.  Because the Git extension runs in trusted workspaces,  it's run before the prompt for display to the user.  So we have no way to prevent everything.  You cannot just review every single Git repo  that you're cloning or that you're  downloading from the internet.  And to show you some side thing that we found,  so we told you it worked in VS Code.  But it also works in developer shells.  So in this case, I have ZSH with all my ZSH team  that includes a Git extension to show in my shell  on which branch I'm running and give me  some contextualized information on my Git repo.  So it runs every time I CD in a new directory.  So if I go on the desktop, and then I  go on my malicious Git repo, it's going to pop again.  So even CDing into something that runs Git in a directory  and having something that runs Git on my behalf, it's unsafe.  So in the end, all these developer tools,  they don't really acknowledge the security risks.  And they say, oh, yeah, it's up to you to not work  with malicious Git repos.  But you can't just review everything by yourself  every single time.  So there are a lot of things out there  that are still affected by this thing,  like see all my ZSH prompt.  And we've also reported it to many IDEs, many shell  integrations.  Some of them tried to fix it.  But in the end, it should be on Git to fix it.  And they said, no, it's an accepted risk.  You should not work with untrusted Git repos.  So Nopal's going to show you a few things  about the excesses in VS Code.  NOPAL MEHTADIAN-SHIRAZIRI Yes, so as I told you earlier,  VS Code has a built-in browser.  Part of it is Chromium.  So of course, there can be cross-site scripting.  And in an IDE, like VS Code, XSS bugs  can be very nasty, because some parts of the UI  always have to be kind of privileged.  I showed you the red part, the workbench.  You will have buttons like Save File or Run My Build Command.  And if you press that button in the UI, that should happen.  So the UI has to have a way to make this happen.  And the other thing is that extensions  can have their own UI part in WebViews,  as I showed you for the Markdown Preview extension.  And even if we would say, OK, all the code of VS Code  itself is clean, no issues, no security vulnerabilities,  all the extensions that you install from the marketplace  could still have something.  And that's then the first foot in the door  to start your XSS attack.  So let's look at the first example.  This is a cool one.  It was found by the Grand Pew and Sirius of ElectroVault.  They presented it here at DevCon last year.  So we will go quickly over it.  Basically, they found a way to control  the content of an extension's WebView, in this case,  the Markdown Preview extension.  And then they used a meta HTML tag to redirect away.  So basically, now their attacker page  is running in this WebView, in this frame.  And from there, they can start their attack.  So they already took control over the frame,  over the WebView.  And then they used the postMessage message passing  interface to talk to the outer part, to the workbench.  And there was an unsafe handler that  allowed them to load any file in that privileged frame.  But there was a clue.  They can't just load their own file from the attacker server  and run any code on the system.  They had to find a way to actually load  the file via the correct origin, via the correct protocol.  And they found a bug in VS Code itself.  So they could do that.  And after that, they could use the node integration, which  is an electron setting, basically,  that says, OK, this origin can just use the Node.js APIs  to run commands and so on.  Then there was a pretty similar finding,  this time by Justin Steven.  He found that in the Jupyter Notebook extension,  you can also render certain things as Markdown.  And then it's the same thing over again.  You can put some HTML.  And first of all, he only found how to leak arbitrary files,  because there's this fun CDN origin that's a special one.  And yes, the plus in there, in the domain name,  is not a mistake.  And you can basically put an absolute path  and read any file by just loading it.  And then Luca Crettoni by Doyensec  used it to improve it to an RCE, basically  using the same last stage as the other guys  to compromise the privileged origin.  And then there was a pretty different finding.  This was by Zemmes of Google.  And they also used the Jupyter Python Notebook Markdown trick  to render something from there as Markdown.  But they didn't have to go up this chain  to first compromise the privileged origin  and then use the node integration.  They found a way to go around that,  which is they just created a link tag with a command link  from the command protocol.  And this is a special one in VS Code.  So this allows you to execute internal VS Code commands.  So not an OS command, but basically something  like, OK, focus the taskbar or open the terminal.  And in this case, they auto-clicked this link  with JavaScript.  So the command would be executed.  And they used the command that opens a new terminal,  because with that, you can specify  which shell executable and which arguments to execute.  And that gives you direct, arbitrary code execution.  And from this, we noticed a lot of the UI of VS Code  is actually Markdown.  The tooltips, the status bar at the bottom, the menus,  a lot of things are just Markdown strings being rendered,  because it's easier than writing HTML, I guess.  And if there's user or attacker-controlled data being  interpolated into this Markdown, then  Markdown injection can happen.  And if that happens, and in the place of the code,  the Markdown string is set to being trusted,  then you can use these command links in Markdown as well.  So if you have Markdown injection,  you can just put in a link to, again,  use this command to open a terminal with your own command,  basically.  But it always requires user interaction,  because the user still has to click on the link.  It can't be auto-clicked unless you have JavaScript execution  in the first place.  So with that in mind, we looked at some third-party extensions.  For example, GitLens.  This is an extension with 25 million installs.  It's pretty common and pretty popular.  And there, we found that while they rendered some hover  tooltips for some commits, we could inject Markdown.  And in this case, the data came from a commit message,  and it was then unsafely interpolated  into a Markdown string, where we could then add a link.  And in this case, we didn't use the other one,  because we also wanted it to work in untrusted workspace  to, again, bypass this workspace trust prompt.  So we used the install extension command.  And because the marketplace is open,  I can just publish any extension there.  You could publish a malicious extension  and then use this command to install it  at the victim's VS Code instance.  And this was fixed pretty quickly in version 14.  CV is still pending, but it's a first good example.  And this is basically how it looks like.  So you open some repo, and you can see the blue bar on top.  So it's still in the untrusted mode.  But if you hover a line and then click  on this very innocent-looking message,  then a calc pops, showing that we could  have executed any command.  And we found a similar thing in the GitHub pull requests  and issues extension, which has 30 million installs, so also  pretty popular.  It's by GitHub themselves.  And again, in a hover tooltip, we found a Markdown injection.  And in this case, it didn't come from the local repository  or the local workspace folder, but it  can be exploded remotely, which means,  let's say a maintainer is working on their project.  They use this extension to look at issues and PRs.  And then I, as an attacker, create a malicious PR,  which has the exploit payload in the body.  And if they then use the extension to look at it,  again, we have the Markdown injection.  And if they click the link, it's GG.  So this one was also fixed pretty soon.  CV is also pending.  And this is what it looks like.  So first, we create this, again, very important-looking issue  on GitHub on some maintainer's project.  Then if they use the extension to look at this,  again, there's a very innocent link.  And if they click on it, a calculator opens.  So yeah, this was basically the last of our attack surfaces.  As we've seen, there's a lot of different things  that can go wrong.  So let's see how things went when  we were reporting the bugs that we found to Microsoft.  So we used the official MSRC platform for that.  It's pretty cool compared to bug bounty platforms,  because it allows you to speak about the bugs  after they were fixed, which we like to do.  For some people, it has a little bit better reputation  because of delays.  But for us personally, we had a good experience, not many  delays.  And it's a nice thing, because everything is centralized  there, even the attribution.  And you can always see, oh, it's in development.  It's being released soon.  So you don't have to wait without a response  from the vendor for months.  So let's look at the bounties we got.  So for the first one, which was the Git local level  configuration bug, we got a 30k bounty, which was pretty cool.  We donated it to charity.  And we even wondered a little bit why it was so much,  because we didn't know.  And it fell into the Azure bug bounty.  And we even asked them why it's so much.  Do they have a hosted VS Code version somewhere?  But we never got a response.  So the year after, we found something again,  which was this time the protocol handler argument injection.  And we thought, OK, easy.  We get another 30k, which we can give to charity.  Pretty cool.  But the response was, oh, wait a minute.  Last time we awarded you the bug bounty in error.  And actually, all extensions, including the built-in ones  that are shipped with VS Code, out of scope.  Sorry, this time you're not getting any bounty.  And yeah, we kind of disagree.  We think extensions that are built in and shipped with it  should be in scope for bug bounty.  But at least they still fix it, right?  And this even brought us onto the MSRC leaderboard in Q2  last year.  And the things that I showed in GitLens and the GitHub  extension brought us on the leaderboard  this year, which was pretty cool,  because we could go to the Microsoft Researcher invite  only party yesterday.  All right, so now let's wrap things up.  We've seen a lot.  So the first things that we learned  is that CVEs hint at buggy components.  A lot of the fixed vulnerabilities  were bypassed by Justin Steven, because he just  looked at the patch and saw that it's not complete.  And that will probably be the same case in the future.  So if you want to find some easy bugs,  just look at what they fix and see if the patch is complete.  The next thing we want you to remember  is if you click Trust, you lose.  For example, the Git local configuration exploit  still works if you trust the workspace,  because the way Git works, and it's basically not really  possible for VS Code to fix it.  And they also say, if you trust, it's your problem now.  We want you.  So please don't click on Trust just  to get this dialogue that negs you away.  Think about it.  And if you don't trust it, press I don't trust it.  And then we found that attacking desktop applications is fun.  And it's a lot different compared  to attacking server applications that you  can reach over the internet, because it  has a whole new set of assumptions and attack  surfaces.  And we've also seen that bug bounty scope doesn't always  reflect the real threat model.  Again, we think that built-in modules should not  be excluded from the bug bounty scope.  We can't change that.  And finally, it's still unclear for many developer tools  what's the security threat model.  They're not built with security in mind.  It's not super clear what's the responsibility of the user  and what's the responsibility of the tool  to make sure everything is safe, no random commands are  executed.  And if you don't know that it's your job as a user,  then you can't really protect yourself.  We personally like the principle of least surprise.  So we think the tool should not do some magic under the hood  that you don't know will happen.  So yeah, it should warn you.  And that's why we think the Workspace Trust that was added  is a very welcome addition and a good start.  And we hope other developer tools will include it as well.  Thank you.


url: https://www.youtube.com/watch?v=2a_vVt96JhE
title: DEF CON 31 - Video Based Cryptanalysis Extracting Keys from Power LEDs - Ben Nassi, Ofek Vayner
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=2a_vVt96JhE",
    "title": "DEF CON 31 - Video Based Cryptanalysis  Extracting Keys from Power LEDs - Ben Nassi, Ofek Vayner",
    "description": "In this talk, we present video-based cryptanalysis, a new method to recover secret keys from a non-compromised device by analyzing video footage obtained from a device\u2019s power LED. We show that cryptographic computations performed by the device\u2019s CPU change the power consumption of the device which affects the brightness/color of the device\u2019s power LED. The changes in the brightness can be detected at a sufficient sampling rate for cryptanalysis by obtaining video footage from a device\u2019s power LED (by filling the frame with the LED) and exploiting the video camera\u2019s rolling shutter, to increase the sampling rate by three orders of magnitude. The frames of the video footage are analyzed in the RGB space, and the RGB values are used to recover the secret key. We demonstrate the recovery of: (1) a 256- bit ECDSA key from a smartcard using video footage obtained from the power LED of the smartcard reader via a hijacked Internet-connected security camera located 16 meters away from the smartcard reader, and (2) a 378-bit SIKE key from a Samsung Galaxy S8 using video footage obtained from the power LED of Logitech Z120 USB speakers (that were connected to the same USB Hub of the Galaxy S8) via iPhone 12. We discuss countermeasures, limitations, and the future of video-based cryptanalysis.",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2624,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230915"
  }

  This text was transcribed using whisper model: large-v2

   Okay. Uh so uh first of all thank you very much for attending this talk and welcome to uh  video based cryptanalysis. Extracting cryptographic keys from uh video footage of a device  that's power LED. I will start by uh introducing ourself. Uh my name is Ben. I'm a black and  bold member. I also do some freelancing and a postdoc at Cornell Tech. I have a PhD in  security and privacy and together with me is uh Ofek which is a master of science student at  uh the Ben Gurion University of the Negev. This work relies on uh two papers that we have  recently published which are video based cryptanalysis and optical cryptanalysis. You can  find them online if you just look for them. And moreover no prior knowledge of cryptography  is required to understand this talk. We try to keep it as simple as possible so the entire  audience will be able to enjoy it. Some of the implement some of the specifics of the  implementations uh are not uh I will not discuss them in the talk. You will be able to find  them in the papers. Now with that in mind let's have some fun. So here is a question for the  audience. Uh what do you associate with the term cryptanalysis? Now some of you probably  associate um servers and quantum computers and data centers and in general you probably tend  to associate high level of computing capabilities with the term cryptanalysis. Others  might associate specialized hardware for example oscilloscopes as the hardware that is  needed to conduct cryptanalysis. Probably supply chain attacks uh is also a term which is  associated with the term cryptanalysis. And also the name of some special agencies um might  appear as well. Um however I tend to believe that the vast majority of you do not associate  this iPhone with uh the term cryptanalysis because smartphones in general have weak  computing capabilities. They are very popular devices. They cannot be used to apply any  complex attack and the vast majority of you do not consider yourself as an entity who owns  a smartphone as uh as uh somebody will uh is interested in uh recovering cryptographic  keys. And here is our message for today. Okay? Think again. By the end of this talk you  will understand that power LEDs pose a great risk to information confidentiality and video  cameras whether they are security video cameras or video cameras of a smartphone provide  the needed infrastructure to exploit this risk. Now by the end of this talk we will discuss  on how to recover cryptographic keys uh from a device by using or by obtaining a video  footage of its power uh LED. But first let's try to answer where did the idea of using a  video camera to recover secret keys from a device's power LED come from? Now this is  actually takes us to 2014 where the visual microphone was published by a group of  researchers uh from MIT. These guys were able to demonstrate a speech recovery technique uh  using a video camera uh where they analyzed the movement of an object. It was demonstrated  on various objects but the most iconic demonstration was the one with the bag of chips uh and  they were able to analyze the movement of a bag of chips from the video footage and recover the  speech played nearby the bag of chips. Okay? And they demonstrated it with the use of two  types of uh video cameras. The first one was a professional video camera. A high uh a high  frequency video camera that ables to provide 20,000 frames per second. And the second one  was a regular video camera and they exploited the rolling shutter in order to do so. And we'll  discuss the rolling shutter later on in this talk. In 2016 I started my uh PhD. And a few years  later uh we um published uh the LEM phone attack. The LEM phone attack is a method to recover  speech by using a photodiode. Now a photodiode is the sensor that you can see on the right  side. Okay it's an optical sensor which converts light into electricity. Okay? Not for  pictures but for electricity. And then it need to be digitized with an A to D. Um we show  that attackers can analyze the movement of a desktop light bulb such as the one that you can  see in the picture. By obtaining optical measurements using the photodiode and then to  recover the speech played by the speakers that you can see uh that are placed nearby the uh  desktop. Okay and we demonstrated speech recovery from various distances up to  35 meters away. You can see the experimental setup at the bottom. The photodiode was  mounted into a telescope. The telescope was directed to the light bulb that was placed on  top of the table on top of the desktop. And the speakers were used in order to play the  speech. Now I want to play to you the speech recoveries uh in order to convince you that you  are able to uh uh recover speech at very high quality. Uh the guys from the audio can I play  it? Okay let's try. We will make America great again. This is the original. This is from 15  meters. 25 meters away. This is the original. We will make America great again. This is the  original. Now the last one was was recovered 35 meters away from the light bulb. And bear in  mind this was recovered from light measurements obtained via the photodiode that was  directed to the rotating or to the vibrating uh light bulb. Now interestingly in one of the  experiment that we conducted we found that not only we can recover speech from by  obtaining optical measurements from the light bulb, we can also recover speech from the  power LED that we used from the power LED of the speakers that we used to project the sound  uh nearby the uh light bulb. Okay? And this is actually led us to write or to publish an  additional research that we named the Glow-O-Matac. Now the Glow-O-Matac is the exact same  technique. Okay we used we recovered speech using optical measurements obtained by an  optical sensor of the photodiode. Um but in this specific case we analyzed the speaker's  power LED intensity. Okay? When speech is played by the speakers and recovers the speech out  of it. And again you can see the exact same threat model. The photodiode is mounted to a  telescope. The telescope is directed towards the power LED. You can see it in here. It's  the uh green LED of the speakers. And we recovered again uh speech at very high quality. I  will play to you the speech recoveries again. Uh that this time we recovered from 5 meters  up to 35 meters away. We will not allow the Glow-O-Matac. We will not allow the Glow-O-Matac. We  will not allow the Glow-O-Matac. We will not allow the Glow-O-Matac. Okay. So we have a  Glow-O-Matac. As you can see the speech or the quality of the speech deteriorates with  distance but then again it was recovered uh by optical measurements obtained by the  photodiode and optical measurements are the uh optical meas- taken from the power LED of the  speakers. Now this is very surprising in a way because probably some of you wonder why can the  intensity of a speaker's power LED can even be used to recover speech? And in order to  answer it we actually conducted an experiment where we took the USB speakers and played  the frequency scan from the USB speakers between 0 to 4 kilohertz. Now we obtained 2 traces.  The first trace was a power trace that we obtained uh via uh an oscilloscope that was  connected to the USB connector of the speakers. And the second trace that we obtained was an  optical trace that was obtained by directing the photodiode towards the LED of the USB  speakers. Okay. You can see them uh at the bottom. Now we made very interesting  observations but I think that the most interesting observation among them was the fact that  you can see that the power consumption correlates with the intensity of the power LED of the  speakers. Okay. Now this is very nice and very interesting but is this correlation between the  intensity of the power LED and the power consumption can also be seen in additional  speakers or is it is it just limited to this specific model that we've used? So then again we  conducted um a few additional experiments. We took various we actually took a few  experiments. We actually repeated the exact same experiment but with different uh models of  speakers. Among them there are Sony uh speakers um Google's assistant. There is also JBL  and Creative. You can see that we took the the best one among them. Okay. You can see that the  correlation appears in each and every one of them. This is by the way spectrograms that were  obtained from the optical measurements. Um and as you can see the intensity of the power  LED of various speakers correlates with the power uh consumption in the range of zero to  four kilohertz. This is what we consider the universal phenomenon. Now some of you still  probably wonder why do we see this correlation? So the fact is that in various electrical  circuits the integrated power LED is connected directly to the power line. Okay. Moreover  dedicated means intended to decouple the correlation between the power consumption and the uh  and the optical and the power correlation are either not integrated to the circuit at all or in  some cases they are integrated but they are ineffective. Okay and as a result the power  consumption of the device which is essentially the power supply to the integrated power LED  affects the intensity of the LED. And due to this fact the optical measurements reflect the  power consumption of the device. Okay. Now at this specific point in time I decided to  leverage these findings to conduct cryptanalysis and uh the idea was to apply the previously  um suggested cryptanalytic attacks that first relied on uh power traces in order to conduct  them. This time we wanted to apply them using optical measurements obtained from the power  LED. And this actually uh led us to optical cryptanalysis. It is a paper again it is about to  be presented at CCS but you can find it online. Optical cryptanalysis is a key recovery  technique to recover the key from a device using optical measurements obtained from the  power LED of the device using a photodiode. Okay. As you can see we directed the uh photodiode  towards the LED of the of uh Raspberry Pi and one of the first observations that we made is  that the intensity of the power LED of various devices it's not only uh Raspberry Pi there are  additional devices as well. Um so the intensity of the power LED correlates with the power  consumption of the device in much wider spectrum that we initially thought. We analyzed  zero to four kilohertz you can see that in this specific case it reaches up to 500  kilohertz which was the upper limit of our equipment. The potential is actually much greater  than 500 kilohertz. Interestingly we conducted an additional experiment where we took um  GNU PG run it on the Raspberry Pi and extracted and obtained optical measurements from the  uh power LED using the photodiode. As you can see in the spectrogram uh that uh appears in  the right side we can distinguish we we can distinguish between the decrypt operations and  the slip uh uh operations of the device which we consider as an idle and moreover as you can  see we can detect the beginning and the end of the cryptographic operations performed by  the CPU. Okay? This is very bad for information confidentiality. This is very bad because um it  now opens a door to conduct the previous timing cryptanalytic attacks that were demonstrated  in the past but in this specific case they can now be demonstrated using uh optical  measurements obtained from the power LED of the uh obtained by the photodiode that was  directed to the power LED of the device. And we use this understanding in order to uh  recover RSA, ECDSA and side keys again you can find this information in the paper. Now this is  very interesting okay? Uh having the ability to recover cryptographic keys using optical  measurements instead of power traces is something which is very interesting. However the  primary disadvantage of uh this entire uh method is the fact that photodiodes aren't  commonly used sensors okay? The vast majority of you do not own a photodiode or at least a  photodiode that can be used to recover this cryptographic keys. Moreover in order to um  obtain optical measurements from a photodiode, attackers must connect them to a dedicated  A to D in order to sample the electricity, the output of the photodiode. Okay? And the idea  we had back then was to go from a photodiode to a video camera. Okay? Instead of using a  photodiode we will use um a video camera to obtain the optical uh to obtain the video  footage from the power LED of the device. Now with that in mind I will let Ofek to discuss uh  the threat models and uh the rolling shutters. Okay so let's discuss the threat models. Our  objective is to perform cryptanalysis and recover secret keys from a device's power LED  using video cameras instead of photodiodes. Now we have two different threat models. The  first one is closed video acquisition. Closed video acquisition uses a smartphone's video  camera. It targets any type of power LED and requires physical access to the device. The  second threat model is over the internet video acquisition. It uses an internet connected  video camera. It targets only type 2 power LEDs which turn on when an operation is executed  and turn off when the operation is stopped being executed and are commonly used in smart  code readers and it can be applied remotely over the internet using a hijacked video  camera. So far we have used photodiodes which are analog sensors that can be sampled at few  gigahertz. Video cameras can record 60 or 120 frames per second. A sampling rate of 60 frames  per second is not sufficient to perform cryptanalysis. Or is it? The third threat model is  rolling shutter. We used to think that a picture is the result of one atomic snapshot taken by  the video camera. However 99 percent of the video cameras use a rolling shutter to record the  video. A single picture is the result of multiple snapshots taken at different times by the  video camera. A video camera uses a rolling shutter to scan an object vertically or  horizontally and combines the scanned pieces into a single frame. Only a few rows are  captured atomically each time and the entire picture consists of multiple snapshots of an  object. Here we can see an example of a picture captured using a rolling shutter. On the left  there is a moving object and on the right there is a captured image. As you can see because  each a couple of rows are captured uh at different times, distortions appears in pictures of  fast moving objects. Such distortions should not appear if the picture was taken using one  atomic snapshot of the object. Now let's describe an experiment we conducted to visualize the  effect of the rolling shutter. First we programmed an Arduino UNO to turn its power lead on  and off every 250 microseconds. And here's what we can see. Now we cannot see the LED turns its  power on and off at 4 kilohertz because the sampling rate is only 60 frames per second. Second  an extra lens was used to direct the camera of a Samsung Galaxy S22 Ultra to the Arduino's  power LED so that the video of the LED fills the entire frame. And here's what you can see.  Now because the LED turns its power on and off a few times in each frame, there are red rows  that were captured when the LED was turned on and black rows that were captured when the LED  was turned off. And this is why we can see those black and red stripes across each frame.  So by filling the frame with the video of the LED we exploit the video camera's rolling shutter to  increase the number of measurements obtained at different times. This allows us to detect the 4  kilohertz flicker. Theoretically by exploiting a video camera's rolling shutter we can increase  the sampling rate by 3 orders of magnitude. From the FPS rate 60 measurements per second to the  rolling shutter rate 60K. In reality there is a delay, a transition time, between  consecutive frames. The transition time between frames is a period of time during which no  object is captured by the video camera. You can see it here in the figure that between two  consecutive frames there is a transition time. This should be taken account of by the  video footage of the power LED. Now I will let Ben to discuss the ACDSA key recovery.  Okay so um before we'll discuss the ACDSA key recovery in order to discuss the ACDSA key  recovery we need to briefly uh discuss the Minerva attack. The Minerva attack introduced or  published 3 years ago and one of the most uh interesting revelations that uh uh researchers  uh did was the fact that they found that various cryptographic libraries uh uh apply some  runtime optimizations in the in the code. Okay? Um and uh I I refer to the code of uh the  elliptic curve 256 R1, the implementation of the ACDSA. And they found that the number of  iterations of the main loop is actually determined by the number of leading zeros in the  nonce. Okay? And as a result it created a time dependency between the number of leading  zeros in the nonce and the execution time of the main loop and essentially the ACDSA  signing time. And the Minerva attack introduced a technique to recover the ACDSA uh um key  by analyzing uh the uh a set of uh by analyzing the um uh ACDSA signing uh measurements  of uh a set of uh ACDSA signatures. You need to have about 4,000 signatures in order to  recover and their associated uh ACDSA signing time in order to recover the complete key.  Okay? Now the original Minerva attack required attackers to obtain uh measurements from the  CPU. The timing measurements obtained by querying the CPU. And essentially it required the  attacker, the attackers to uh compromise the host machine. The authors were unable to  demonstrate the attack remotely over the internet due to the noise added by the network  latency. Okay? The Minerva attack is very sensitive to errors in the timing measurements.  You are about to see it now uh demonstrated over the internet. Um without getting into too  much specifics and details regarding the Minerva attack, you can think about it as a black  box. Okay? The black box receives a set of signatures and their associated uh signing time  and returns the ACDSA key out of them. So essentially what we need to answer is how can the  ACDSA signing time of a signature be estimated from video footage. If we will be able to  answer it, we'll be able to recover the complete ACDSA key using video footage. Now this  was the experimental setup. Uh in our uh in our uh in our um uh experiment. We used it, we  placed an internet connected video camera and directed it to uh the uh the power led of the  smart card reader. Inside the smart card reader there was uh a smart card with an ACDSA  key. And we wanted to recover the ACDSA the ACDSA key from the smart card using video  footage of the power LED of the smart card reader by using this internet connected video  camera that was placed 16 meters away from the uh from the uh from this desktop. Uh  okay and this is the experimental setup. I want to show to you the video. Um you can see on  the left side this was the the card reader that we uh the smart card reader that we used. We  actually focused on the right uh power LED of it of uh it. And let me show it to you now. Now  you can see that we zoom into the uh power LED of the um smart card. And interestingly the  smart card actually provides you with indication regarding whether the the smart card reader  actually provides you with indication regarding the smart card. Whether an operation  actually uh taking place or not. And this is done by uh by changing the color of the LED  between blue or basically turn it off. Okay anytime it's blue the uh smart card is in  idle. Anytime it's off it the smart card is being used to sign. So um as you can see from the  video on the right side there are differences between black and blue colors. Okay which  indicate whether the smart card is currently being used to sign on on stuff or not. So as you  can see this actually uh the the picture on the left side is actually the result of  averaging each row in a frame to a single value and arrange them in a time series. Okay and  we did it for the red green and blue uh channels. You can see that we can distinguish between  the sign operations where the power LED is off and the idle operations where the color of the  power LED is blue based on the blue channel. Now now we need to answer how how can we  calculate the signing time out of it. Now in order to calculate the ACDSA time of a  signature, we extract the series frames associated with each signature from the video. Okay you  can see them you can see uh examples of them uh on the bottom of the slide. Now we kept  series that contain the indication of the switch between the beginning and the end of the  signing. Meaning that we kept series that the beginning and the end of the uh of the  signing were kept in the frames. Where you can see uh in this specific case, this is the  series that you can see on top. Okay? Each series that missed or lacks one of these  indications regarding the switch between the beginning of the signature and the end of the  signature, we filtered it off. Okay? You can see that uh how the in uh the frames uh that  appear in the first series, the color is changed inside the frame between blue and black and  vice versa. Okay this is what we considered as the indication that the operation started and  ended. Okay and these are only the type of series that we kept. Now um first of all we  calculated T1 which T1 in our case is the execution time of all of the uh the full black  frames that you can see in in here. This is actually done just by counting the number of uh  full uh black frames and multiplying them by this constant. This gives you T1. T2 is the  execution time of the first frame. Okay? In which the color changed between as you can  see in here between blue to uh black. So uh we calculated it as follows. We counted the  number of uh rows that are associated with the sign which are you can see it in here. They  are in black. And divided them by the total number of rows in the frame. Okay this is  actually gives us the relative time in the frame. We multiply it by the scanning time and  added the transition time. Now so far Ofek discussed about the scanning time and the  transition time but he did not mention how you can determine the scanning time and the  transition time. So let's discuss how we can find S which stands for the scanning time and T  which stands for the transition time. So I remind you the uh experiment that we did at the  beginning with uh flickering LED of the Arduino. You can just take one of the you can repeat  the exact same experiment with the flicker. Take only one of the um frames from the video.  Count the number of changes between black and red that you have in the frame and multiply it  by the time that the flicker is on and off. In this specific case the on and off uh periods  were equal. Okay? And by doing so this will gives you the scanning time of the video uh the  video camera. And the beauty of it is then when you get the scanning time out of this  experiment you get the transition time for free by deducting the scanning time out of this  constant uh out of this constant. Now with that in mind and returning to T2 you now have S  and T and everything needed to calculate T2. Now T3 is actually calculated at the exact same  way. Okay? Um but in this specific uh case we do not um add the transition time because  the the um ECDSA sign stopped or ended in the middle of the frame so the transition time  happened after the frame so there is no need to add it to this specific uh to T3. And the  sign time is actually the sum of T1, T2 and T3. Now the beauty of this is that you can  do it to enough signatures for about 4,000 signatures. You can see that this is the result  that we got. We were able to recover the ECDSA key out of it. Um using the Minerva uh  script that uh that the others were published. Again using an internet connected video  camera that was directed to a power lead of a smart card reader. Thank you.  Okay. Now moreover there are many additional smart card readers available to purchase on  Amazon that are vulnerable to this attack. And the distance between them and the video  camera might vary based on the intensity of their own power LED but we were able to  recover the same ECDSA key from the smart card inserting it to 5 additional smart card  readers. Again all of them are available online. Okay now let's discuss um recovering a  side key from uh from a device. And again recovering a side key from a device or in order to  understand how to recover a side key from a device we need to briefly discuss the  Herzblied attack. The Herzblied attack introduced uh a year ago. They also presented their  talk at uh Black Hat. And they were able to uh suggest or introduce a new timing attack to  recover the complete side key from a server. The Herzblied attack unlike the Minerva attack  that I showed you before is not the result of the implementation of the code. Okay. It's  the result of the execution of the device. The code should be resilient to timing attack  but the execution actually reveals something in terms of the time that it was uh uh  executed regarding the data being executed by the device. And this is happening due to the  dynamic voltage frequency stabilizers. Maybe you are more familiar with the DVFS. Okay.  Which yields different execution times based on the data that was recovered that uh is  being processed by uh by the device. And uh researchers were able to show that for each  index of the bit they can craft a dedicate a dedicated cryptogram that relies on the um  indexes that they were already recovered. Okay it's an adaptive attack for those of you  familiar with cryptography. And determine whether the value of the bit under attack uh is  similar to the um to the to the value that was recovered in the previous index or changed.  Okay and this is all done based on the timing threshold. Now the original attack was  implemented using timing measurements obtained by querying the API of a server over the  internet. Uh they deducted the request time from the response time. And again due to the  added uh noise that was added by the uh network latency it took a few days to recover the  keys. Now this was our attack. Okay. And this is the experimental setup of our attack. We  tried to recover the secret key, the side key from the Samsung Galaxy S8. The Samsung Galaxy  S8 is the smartphone that you can see on the right side. It's not the smartphone that you can  see on top. The Samsung Galaxy S8 was the device that holded the side key. Okay. Now um  interestingly in this specific case we tried to recover the secret key from a device by  obtaining video footage not from the Samsung Galaxy but from the speakers. Okay the power  lid of the speakers that were connected to the same USB hub that was used to charge the  Samsung Galaxy. And in order in order to make it completely outrageous we did it with an  iPhone. Tried to recover the Samsung the key from the Samsung Galaxy S8. On the right side  you can see the video footage of uh the power lid of the speakers. Okay you can see that this  uh green. This is actually being taken from the speakers as you can see uh on the left side.  Between the speakers and the smartphone you can see a lens which was used to fill the entire  uh frame with the view of the LED. I want to show to you the video. You cannot see anything  right? Seems like completely innocent video with green color. Moreover I will argue that you  cannot see even the difference between the idle and the cycle operations being performed by  the Samsung Galaxy S8. Okay. They cannot be detected by the naked eye. But there is  something very interesting. Okay. We executed 8 consecutive iterations of cycle operations  where each iteration consisted of 100 uh and uh 1100 of cycle operations on the Samsung  Galaxy S8 while obtaining video footage from the power LED of the connected speakers. You  can see that by averaging each frame into a single value and arranging them in a time series  and then again zooming into the green channel you can see that we can detect the beginning  and the end of the iterate of the 8 iterations by analyzing the green channel and again using  this uh by arranging them by arranging the values in a time series. Now this is actually  gives you uh the ability to calculate the iteration time based on the number of values between  each 2 peaks that you can see in here. Okay? Um but then again another question that we need  to answer is how accurate is this calculation? So on the left side you can see the distribution  of the execution time based only on the first iteration. Okay? We apply the attack on by using  only the first iteration and as you can see it's very noisy. The 2 possibilities of the the 2  cases that uh that whether the bit was uh have a similar value or was it changed in comparison to  the previous uh index, you cannot set a threshold that will give you an accurate uh you will be  able to use it in order to accurately distinguish between the 2 specific cases. However when  you use um instead the distribution of the execution time based on the minimum execution  time among these 8 iterations, what you're seeing in the middle, you can see that you can  actually set a threshold in the middle that will uh that will allow you to distinguish between  the 2 possible uh bits of the key uh index with very high accuracy. Okay? 99% accuracy.  Now 99% accuracy is good. It's not excellent. Okay? It's not error prone. And as a result, we  will we will have to uh to use an error detection and correction mechanism throughout the key  reco- throughout the uh key recovery process in order to uh recover the complete key. And as  you can see this is actually what we did. We recovered the complete key, the complete uh uh  psyche from the device using a video footage obtained from the connected speakers. Um then  again we also integrated an error detecting and correcting mechanism that was suggested by  the guys that presented the Hertzbleed attack. Okay so let's briefly discuss the limitations  of uh video based script analysis. So first of all the first limitation is the fact that the  attack mostly targets weak IOT devices. I would say up to a level of a smartphone. I would  be very surprised if somebody will take the exact same video cameras that we used and will  show me that he was able to recover secret keys from uh power led of servers or power led of  laptops. Now moreover, we discussed about it uh earlier, due to the fact that there is a  transition time where uh between each two frames where the object or the power led isn't being  captured in any frame, the sampling uh the uh the distribution of the sampling is semi  uniform. It's not exactly uniform which is a exact which is another issue that we need to  resolve when recovering the secret keys. Now let's discuss the takeaways. I think that now  you are convinced that power leds are much more informative than you initially uh  imagined. Um in some cases attackers can use the information modulated over the power led to  recover speech. In other cases it can be used to recover cryptographic keys as we showed in  this uh in this talk. Now moreover, the potential of the attack is actually much greater  than uh than uh we showed you in this uh work. Okay? We focused on what we consider as  ubiquitous cameras. Okay? Not professional cameras. We use security cameras and  smartphones, video cameras in order to recover the secret keys. Attackers in reality can use  professional video cameras that will provide a high rolling shutter rate, a higher beat  depth and enhance zoom capabilities and by doing so they probably will be able to recover uh  secret keys from a wider range of devices from probably even um a greater distance. And this  is I think the most interesting uh insight or takeaway from this talk. We expect that more  and more devices will be exposed to video based cryptanalysis every year. Okay? This is  actually the result of two interesting facts. First of all uh video cameras specifications  are continuously improving each and every year uh following Moore's law. You can see how  the sensitivity of this picture has improved significantly throughout the years. Moreover,  the number of functional IOT devices with weak CPU power and I'm referring to sensors and  uh smart cards and robotic vacuum cleaners and smartphones and even video uh streamers,  the number of such devices are continuously uh increased each and every day. Their  deployment is expected to grow throughout the next year as well. And as a result,  we will have much improved uh video cameras throughout the years that co-located with a  greater number of weak IOT devices. Okay? And due to this fact, we believe that uh more and  more devices will be exposed to video based cryptanalysis each and every year. This is the  final takeaway. It's mostly uh meant for perspective. It might be easier for this audience  to compromise the target device with the malware and exfiltrate the key over the internet.  Okay? Than applying the attack that I've just, the attacks that I've just mentioned. However,  bear in mind that video based cryptanalysis is intended to extract keys from non-compromised  devices. Okay? And the beauty of it is that it uses popular equipment, not something that we  consider specialized equipment as the equipment that I've discussed at the beginning. Now a  few other uh takeaway, a few uh other things that I would like to mention. Uh this research  was recently awarded with a Pony Award for the best cryptographic attack of 2023. Um you  can find the additional details if you will look for video based cryptanalysis online. And  with that in mind, thank you very much for attending this talk. I will be happy to take any  questions from the audience. Thank you.  >>[applause and cheers from audience and audience members


url: https://www.youtube.com/watch?v=AfMfYOUYZvc
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
title:
  DEF CON 31 - Vacuum Robot Security & Privacy Prevent yr Robot from Sucking Your Data -  Dennis Giese

content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=AfMfYOUYZvc",
    "title": "DEF CON 31 - Vacuum Robot Security & Privacy Prevent yr Robot from Sucking Your Data -  Dennis Giese",
    "description": "Exactly 5 years ago we were presenting ways to hack and root vacuum robots. Since then, many things have changed. Back then we were looking into ways to use the robots' \"dumb\" sensors to spy on the user (e.g. by using the ultrasonic sensor). But all our predictions were exceeded by the reality: today's robots bring multiple cameras and microphones with them. AI is used to detect objects and rooms. But can it be trusted? Where will pictures of your cat end up?\n\nIn this talk we will look at the security and privacy of current devices. We will show that their flaws pose a huge privacy risk and that certification of devices cannot be trusted. Not to worry, though - we will also show you how to protect yourself (and your data) from your robot friends.\n\nYou will learn on how you can get root access to current flagship models of 4 different vendors. Come with us on a journey of having fun hacking interesting devices while preventing them from breaching your privacy. We will also discuss the risks of used devices, for both old and new users.\n\nFinally, we will talk about the challenges of documenting vacuum robots and developing custom software for them. While our Primary goal is to disconnect the robots from the cloud, it is also for users to repair their devices - pwning to own in a wholesome way.",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2665,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230915"
  }

  This text was transcribed using whisper model: large-v2

   All right, thank you very much for being here at 10 a.m.  in the morning on a Sunday.  Everyone who had talked to me over the last past days,  I always say like, hey, don't go to the talks,  just watch a live recording,  but unfortunately I need to be here.  But great that you're here, so thank you very much.  Yeah, so today I will do another vacuum robot talk,  and this time we take a look at robots which have cameras.  And I know this is like every two years  it's more or less sounds like the same talk,  but sadly there's always a lot of development  and the vendors are not the smartest ones always.  So see it as an update.  All right, so some information about me  so that you have some idea what the heck I'm doing.  I'm a PhD student in Boston at Northeastern University,  and I primarily do research in wireless and embedded devices.  So I'm interested in security and privacy.  I think I call myself now also a vacuum robot collector  because I have many, many, many of them.  So some people collect stamps, I collect vacuum robots.  And I'm interested in reverse engineering  interesting devices, robots obviously,  but right now I'm also looking at smart speakers,  flash memory, everything which is kind of interesting,  has internet and can do interesting things.  Some of my past projects to give you some idea  why I'm doing things, how I do them,  I had a paper some time ago about Amazon Echo devices  where it did forensics about 87 or 86 used device  from eBay and was doing forensics on them.  Let's say the outcome was not very good for Amazon.  Some other things which I do right now  is flash forensics, so I look at embedded devices  and see what kind of information I can extract from them.  But I'm looking also at flash firmware itself.  So flash memory has sometimes firmware onto it  and I'm looking if I can use it for something interesting.  Another project which I run primarily is robotinfo.dev.  This is like a website where you have a systematic list  of robots which I have, sorted by operation system,  sensors, vulnerabilities.  My primary focus here is security and privacy.  And one of the things which I also do with that  is also tracking of firmware changes.  We will see later that tracking of firmware changes  can be very, very useful and can save you a lot of time.  Yeah.  One of the stories how I get my information is,  I mean, I have the devices myself  and then as soon as it is connected from the cloud,  I basically emulate them in the cloud.  So the vendor thinks that they're still online  but I just use it to scrape the data.  And this is also like a base for further research.  As you can see in the picture, by the way,  this is like one of my two or three shelves which I have.  So I have from each of the robots which I root,  I have like a reference model.  If I need to test something, just go to my shelf,  just grab one, test whatever I want,  and just put it back on the shelf.  Which means I have no idea which robot cleans the best  because I root them but I don't actually use them  for cleaning.  So I get this question quite often.  I have no idea.  I have one which cleans, it's fine.  So no idea about that.  All right, so what's the goal of this talk?  I want to give you some overview of the development  of vacuum robot hacking over the last five years.  I want to give you also some insight  of vulnerabilities and backdoors.  And I want to give you an idea about the current  routing methods for current robots.  The ultimate goal which we have typically  is that we have root access at some point  on a robot without disassembling it  because disassembly is always warranty seals and stuff.  One quick notice, we don't necessarily hate  the robot companies.  I mean, it's like more or less like a competition.  Sometimes I visited them and had a nice chat  and next week I rooted their devices.  So it's kind of nice.  So it's a nice kind of thing.  So what kind of devices will I cover today in the talk?  And there's a kind of long list.  Technically there's even a longer list,  but primarily we have the companies Roborock,  we have Dreamy.  A lot of things which I do today apply also  to many, many, many other devices.  Not only vacuum robots, but lawnmowers,  smart speakers, and other things.  All the devices which are underlined  are ones which have cameras.  And the ones which are indirectly is like,  technically I have it, but it doesn't make sense  for me to actually root it and develop tooling.  So if you want to port it, you can do it,  but yeah, just as information.  So let's talk about this talk.  So this talk is more or less the result  of like 15 months of research and experiments.  So I do that for a very long time.  And it's a little bit annoying to sit on things  for a very long time.  So if you know, like, okay, I can get access  on this device, but I can't tell anyone how.  But just very annoying.  And also this talk is a collaborative effort  between me and Zoran Bayer.  And Zoran is one of the, or is the developer  of Valetudo, which is the cloud replacement for robots.  So he plays a very, very important role in this endeavor.  The other thing is, all this work wouldn't be possible  if you wouldn't have the community.  So we have thousands of people who are making robots  and who are tinkering around with them.  I cannot have every device so far,  and thankfully people are testing things,  and this is kind of great.  As a general information, in this talk,  you learn about vulnerabilities the same time  as the vendors do.  So right now I know that some companies in China  are watching this talk live,  and I will get some emails afterwards.  But I expect some patches coming in the next days,  so they don't know what we do.  So just as information.  All right, so what's the motivation for this talk?  Well, why do we root devices in the first place?  And a lot of people think like, okay,  you can just use it, why the heck do I need to root it?  Well, this device is a very, very cool hardware.  Think of a Raspberry Pi on tires,  which has tires like cool sensors  and can drive around to your apartment.  It's a very compact, nice unit,  which you can do a lot of experiments with.  The other thing is, we obviously want to stop devices  from constantly phoning home,  and you will be surprised how much we phone home.  So it's like, one device can produce  hundreds of megabytes of log files, maps, whatever,  every month, which is getting uploaded.  So if you have some internet tariff  which has limitations of volume,  you might think about that.  The other thing is, a lot of people like to have  some custom smart home solutions.  So Home Assistant is one of the examples.  It makes it way easier to obviously connect the device  if you have a rooted device.  A thing which becomes more and more important,  also with the right of repair,  by rooting devices or having the tools  at least to be able to do so,  we can diagnose broken devices.  Especially in US, there's a problem  that you have a warranty of one year,  and these robots tend to break after one year.  So it's very useful that we can figure out,  okay, what's going on with devices.  And we want to verify our privacy claims.  Companies can say a lot of things if the day is long,  and marketing people can do that too,  but we want to actually know if it's true or not.  So, why don't we trust the nice companies  and why don't we trust IoT?  Well, imagine this device is connected to your home network  and has access to everything which is in your home  if it's in the same network.  And it has its own internet connection,  basically, back to the cloud.  Which means, most of the time,  this communication is encrypted,  and you have no idea what it's sending back.  It has cameras, it might have microphones,  it has a map of your home,  it can drive around on its own,  and you have no idea what's running on it.  Also, even if the vendor is not malicious,  developing secure hardware and software is really hard.  And if you remember the Mirai botnet,  where basically a lot of IP cameras got hacked  and millions of, you know, basically botnets were created,  this is kind of like a thing  where we want to just make sure that, you know,  there's nothing, you know,  which the vendor accidentally left in there  and forgot about.  And one of the things is also that  a lot of vendor claims contradict each other,  and I have one perfect example for that.  Which I always use, so it's like,  if you were here two years ago,  you probably know it, just, you know, for the newcomers.  Roborock, I mean, I shame a little bit to Roborock,  but all the companies do that, by the way.  They claim for their robots with cameras,  nothing is ever sent to the cloud.  We will never duplicate data,  the data is not stored on the device,  and we don't send camera pictures to the cloud.  So you're safe, the camera is just used  to kind of detect objects.  But if you scroll down on a page back then,  you find, like, oh, by the way,  you can also access the camera remotely to watch your pets.  So, okay, so you're not sending it to the cloud,  but I can still access it remotely, so how does it work?  Another thing which came out a couple months ago,  and I was also a little bit involved in that,  iRobot got caught that they basically upload  pictures of cameras of robots to the cloud,  and then gig workers in Venezuela kind of labeling them.  So there's an interesting article,  I linked it also in the foot line,  and so what happened is basically,  these were development devices,  so people got three devices,  and had to sign some terms of service  where we kind of agreed to that,  but no one actually knew that they agreed to that,  and so people started to kind of,  so these devices started to upload pictures,  and for whatever reason, iRobot was very interested  that you had a fan, and lights,  and whatever, in your apartment.  For whatever reason, they wanted to have it labeled.  A fun fact of that is, as soon as the article came out,  or shortly before that,  a lot of vendors started to panic,  and I saw a lot of firmware changes,  and I saw a lot of changes in privacy policies  for all the companies,  so they got really, really spooked by that.  Not that it actually changes a lot,  but some of the firm has actually started doing pictures  because we got scared they get also caught, like iRobot.  The other interesting thing which we see nowadays  is devices get more and more sensors,  and I have an example here from EchoVox,  which we don't talk about this talk today,  but just as an example.  They have cameras,  but they started also to introduce microphones.  We made, like five years ago, with Dania Wigimer,  who I was starting this research with,  a joke like, oh, we can maybe try to use  the ultrasonic sensor as a microphone  so we can spy on people,  or we can use some other sensors  to get some audio information or visual information.  Nowadays, we are here basically like,  oh, we have cameras and microphones already on board  so we don't need to think of sketchy ways  to get to the audio stream.  Right.  Another thing which we need to talk about  is what are the risks of devices with cameras.  A lot of devices might store pictures indefinitely,  and basically both in the cloud and locally,  and from what I can tell so far is many do so.  So basically, I have a research project running  where we look at what kind of data is stored where,  and a lot of pictures are staying basically forever.  The other thing which might be problematic  is if you are a shopper at Amazon Marketplace  and you get used devices,  I'm not sure if it's a good idea,  partially because whoever had the device before  could have installed a rootkit,  and you as a new owner, you have no idea  what's going on there.  So even if you do a factory reset,  the malicious firmware might still be on.  With the result that you have a malicious device  in your network which has all the access.  By the way, if you learn today how to root devices,  don't buy Amazon devices.  Root them and send them back.  That's evil, that's bad, that's hopefully,  I think that's illegal.  So don't do that, very bad.  All right, so which means rooting is more or less  the only way how you can verify that the device is clean.  So there's literally no other way to kind of know that.  By the way, the vendors are kind of aware  that people and cybersecurity people  are kind of scared of cameras.  So they try to avoid the word camera.  So instead, you know, they say like,  this device doesn't have a camera.  We have optical sensor.  And I have here an article from a German page,  but there's also English-speaking pages  which have marketing material from Roborock.  And it kind of said like, well, we don't have a camera  in the actual sense.  We have an optical sensor which detects a couple things.  Well, on the right you see an output  of this optical sensor.  This looks kind of to me like a picture,  but I mean, it must be an optical sensor.  So yeah, this is from a Roborock S8 Pro Ultra  from the optical sensor.  So yeah, don't believe marketing material.  The other thing is everyone loves certifications.  So all the devices we're talking about today  have some sort of certification.  By my favorite company in Germany, TÜV,  which is certifying a lot of things of the days long.  And all of the devices are certified  by the European cybersecurity standards for IoT products.  So I think because we're certified,  we can stop the talk here because we're secure, I assume.  Yeah, so the takeaway lesson at some point  just to kind of give you already like a spoiler  is don't trust certifications.  They don't necessarily mean anything.  All right, so what happened so far  in the past over the last five years?  Well, one of the general observations which I did  was every time a routing method is presented or released,  the vendors react in one way or the other.  Sometimes we even break things in the process.  So we saw in the past like a vendor  which kind of panicked a little bit  when we released the routing method.  And they created a firmware update  which started to break hundreds of vacant robots,  which we are not recoverable anymore, so yeah.  The best case for us if they fix things  is that the routing method just fails  but the device says like, hey, it's invalid, I don't boot.  The worst case which is, and some vendors do that actually,  the routing succeeds but the device breaks randomly.  So you let it run, it will stop randomly  after five minutes or 10 minutes or whatever.  We will show an example in the next slides  where we saw that.  Or the device might also destroy themselves.  So we saw in one case where basically  if the device detects that it has been routed,  it will just destroy itself, which is bad.  All right, let's look in the past.  So the first work of the vacant cleaning  which I was doing was in 2017.  And this was together with Danny Wigeman,  who has, by the way, a talk at 11 a.m.  So if you're interested in,  I think he does chipsets, like wireless chipsets,  then he has a talk in track three.  And the robot which we were back then analyzing  was the first generation of Xiaomi robots  and the first generation of Roborock robots.  Some of the things which we figured out  is that the firmware updates are unsigned  and encrypted with a very weak key,  which was, I think, the company name, Roborock.  Which wouldn't itself be an issue on its own.  I mean, that's kind of like, not great,  but I mean, the bigger problem was  that you could push a custom firmware  from the local network onto the device.  And because the thing was kind of clear,  so the password was kind of known,  people could do that maliciously.  You cannot only do that from the local network,  but you can do it also for unprovisioned devices.  And I was telling the story like two days ago  in my demo lab where I was in Germany  and I saw one of the neighbors had a Roborock robot  which was unprovisioned,  so it had an open Wi-Fi access point,  and I was like, I didn't do it.  I just want to say I didn't do it,  but you can do that.  So make sure that your device is provisioned,  because otherwise someone else can provision it for you.  Just a side note.  Right, so what was the result?  We could easily root devices  because we could just push firmware updates.  And it was great for the development  of custom software and voice packages.  Right.  And we published that at the CCC Congress in 2017,  and I was giving a talk exactly five years ago  on the day here at DEF CON.  So it's a very long run of research.  So after that, Roborock was, let's say,  not directly happy.  We started to do a couple things.  For example, we blocked the local updates,  which was not necessarily bad.  I mean, I think from a security perspective this was good.  Then they started to introduce with newer models  firmware signatures and signed also voice packages.  And they used different keys for different devices,  so we couldn't decrypt devices,  firmwares for different devices anymore with one key.  One thing which we didn't like,  they started to sign files for region locks.  So if you buy a device from China,  you couldn't run it in US.  They cracked down on that.  On the other side, all the hardware remained the same.  So if you buy a device from five years ago  and you buy a device from two years ago,  they have more or less the same board, the same hardware.  That's kind of a business model  generating revenue.  The bad news after they fixed that  was that we had to basically disassemble the devices,  and this is a little bit of a problem.  So we developed a new routing method,  which worked for Robux S7, S5 Max, S7 and others.  And the initial idea was, okay,  we can just connect over UART  and can do the U-boot shelving,  which you can do for many, many devices to get root access.  However, we blocked that at some point,  and instead we developed a method  where you can basically bring the device  in bootloader mode,  and just start to patch the system from there.  The disadvantage here was, obviously,  we need to disassemble it,  potentially solder things, so it's kind of a problem.  One of the interesting facts is this method  still works today for many, many devices  which are based on the Allwinner R16 chipset.  So even if you buy today a newer Roborock,  like the Q7 or something,  it still works for whatever reason.  So that's a good thing.  After that, they started to react, obviously.  One of the things is we removed the U-boot shell,  so we couldn't do that.  And for their flagship models,  we started to introduce a lot of mean things.  CQ-boot, SA-Linux, DM-Verity,  they put that in the firmware.  And even newer models started to use encrypted file systems  where the user data application was encrypted,  so you couldn't access machine learning models,  you couldn't access anything which is on the device.  And they started to put these keys into a trust zone.  So you need to boot in CQ-boot mode  so that you can access the keys.  So they put a lot of effort into securing that.  One thing which was very mean  is they started to put custom stuff in the kernel.  For example, they added an ELF binary signature check  in the kernel, so you couldn't run any unsigned binaries,  which is a little bit annoying if you got some way in  and tried to run something you have,  and it just says, no, it's not signed, so screw you.  All right, so in 2021, again, here at DEF CON,  I figured out a way how to bypass all this stuff.  The initial idea was basically,  I need to disassemble the device  and modify the configuration partition,  which requires either EMMC desoldering or ISP access,  so which is very, very complicated.  So it's a thing which a very small percentage  of people can do, but it's very tricky.  Also back in 2021, because we got a little bit tired  of Roborock and their counter-measurements,  we figured out that there's a new vendor, which is Dreame,  and their devices were kind of powerful  because they had cameras and other things.  They were easy to root just by a connector, by USB,  but the problem was a little bit  that the devices got soft break from time to time,  but we kind of fixed that.  So what was the Roborock's reaction?  Exactly one day after the DEF CON talk,  I got a nice email from the Roborock CEO.  Thanks for the talk.  Our engineers were watching the talk live,  so I assume they're watching it right now,  and they're fixing right now the vulnerabilities  and pushing out the updates.  So now what we do is all the partitions are encrypted,  except for the system partition,  which we can't do for particular reasons.  We started to obfuscate the ELF binary check verification,  so it cost us more time to figure out what we do,  and we added custom code to detect that we root devices  and to bypass our traffic.  So typically what we do is when we disconnect a device  from the cloud, we reroute the traffic  so that it ends up in our custom software.  They figured out a way to detect that  and basically still send out the traffic  to the real Roborock servers, which was very mean.  I mean, we figured it out, but it was kind of weird.  Wait, we rooted this device,  why is it still sending stuff back home?  So we basically put hooks, obfuscated hooks  in the libcurl library,  so every time HTTP requests happening,  they match for, is it the Roborock URL?  If it is, then do a sign-in DNS check themselves  and kind of send it silently.  And we are big fans of XOR,  so they XORed everything out of all the strings  from all the binaries,  so they really want to make it hard for us,  but we don't use strings on the thing.  As an example for their creativity,  so like I mentioned before,  they added ELF signature checks in the kernel,  and what they do is typically they map the doMMap function,  which creates, I think, the memory area  for a binary if you execute it.  So if you execute an unsigned binary,  you get just a segfault if the signature's invalid.  And they got very, very creative with names,  so I have an example here,  so this security function can be called  like clock set rate DSP,  clock get rate DSP,  and all the other names, right?  So it's kind of like, if you look at that,  you think like, oh yeah, that needs to be there,  but in reality, they try just to kind of be sneaky  and obfuscate things.  They get really, really creative with the names.  But not creative enough.  If you Google for these functions,  you don't find anything except for Rorock,  so, well, maybe you should be a little bit more creative.  Dreamy panicked also at this time.  They were kind of new.  They did a lot of changes to the firmware,  so they pushed like one firmware update after another.  They removed our UART lock-in shell,  and they patched the U-boot shell.  And some of the changes apparently  did break a lot of robots.  There was like some kind of thing  which we didn't test properly.  They pushed a firmware update,  and a lot of robots were kind of very sad afterwards.  A fun fact is, their patches revealed  a way easier meta-to-router devices,  which we actually knew.  So one thing which we do is typically  we compare all the firmware updates  and see like, okay, what did they change?  This happens fully automatic.  And one thing which we noticed like,  oh, they changed this thing.  So typically we would kind of do like a sketchy way  to like of flashing the robot by USB,  and if USB cable's kind of janky,  it might break the robot, so it's very, very dangerous.  Then we saw that they patched a function out,  which basically if you press the reset button  for like one second, it will pop your shell.  And we saw that, and we were like,  great, this is way safer, because UART,  I mean, you cannot break things that easily.  So thanks, Dreamy, for telling us about this thing.  So we weren't even aware of it.  I mean, I didn't have any idea that this exists in there.  In 2022, when we started to kind of more focus  on Dreamy robots and rootless devices,  they introduced more and more countermeasurements.  One of them was, but obviously they enabled secure boot,  and so the eFuses, which was kind of to be expected,  they started to sign the system partition in a weird way  and verified it in a bootloader.  And they started to pair the kernel  with a particular version of file system,  which on itself is not a problem,  but the next step is kind of the annoying part.  They introduced Judge, we call it like that,  countermeasure into the software.  What is this Judge doing?  Well, they introduced it in all the firmwares in 2022,  and it will randomly crash the robot whilst cleaning  if it detects a manipulation of the firmware.  So what it does is it has like an expected char of 256 value  and it baked into the kernel,  and the process, as soon as you start to run it,  it will basically compute the hash of the system partition  and compare them.  If it doesn't match, it will basically create a thread,  which will allocate lots and lots of memory.  So after a random time, it's not even so,  but it's like three minutes, it can be 15 minutes,  it can be anything, the robot will just stop  and crash and reboot.  And imagine how difficult it is to figure that out.  It took us like weeks to figure out like,  wait, is this like a thing which is,  we messed up something?  Is it, what is it?  Because there's no log files, it's not reproducible  like in a very efficient manner,  until on some point we figured out like,  oh, wait, this thing is happening,  and so we patch it out nowadays for like firmwares  which we kind of release.  But this is extremely mean,  this is like one of the cases where I'm saying like,  okay, just tell us that we messed up, it's fine.  Don't be sneaky and you know, kind of annoy us with that.  Okay, so what is the current state of robot security today?  And I have an example here,  which is probably one of the most,  or best protected robots so far in the market,  which is the Roborock S8 Pro Ultra,  which is I think the top on the stack.  It's the current flagship model,  it came out a couple months ago.  It has a quad-core all-winner SOC and multiple MCUs.  They got a little bit cheap,  so the initial models came out with one gigabyte of RAM,  but at some point we figured out like,  oh, hey, we can get away with 512,  so newer releases have like 512 megabyte of RAM.  They have four gigabyte of flash,  they have two cameras, a LiDAR sensor, and two lasers.  Security-wise, we kind of checked everything in the book,  basically secure boot, the MBRT, protected root FS,  look-centric partitions, SA-Linux of signatures,  so the standard stuff, what they have.  Let me give you a quick intro  how the booting process looks like,  and this is a very simplified version.  I know at the end of the day,  it will look very, very complicated,  but just for you to understand,  okay, what are we doing, what is our leverage?  If the device boots up, it boots from a boot ROM,  which is baked into the SOC.  It will then verify the first stage boot loader,  which initializes also RAM,  and the first stage boot loader will then check  the signature of the thing which is called TOC1.  This is just a technical term, ignore that for now.  The TOC1, what it will do is it will start  the trust zone components, Opti in this particular case,  which will then verify U-boot  and launch the actual U-boot boot loader.  The U-boot boot loader gets its configuration,  and basically, for example, which partition to boot,  and will launch and verify the Linux kernel.  The Linux kernel itself will then verify  the root file system and check if it's correctly signed,  and will start all the software,  which it will also check if it's correctly signed.  So the software itself, the elf binaries are signed.  Then we use Opti again to retrieve the keys  for the system partitions and for the data partition,  and then we'll decrypt the software partition  and user data partition.  So there's a lot of stuff in there,  and it's like, all right, everything checks everything,  so what the heck are we doing here, right?  Well, does it, actually.  There's one thing which is maybe not checked  in the way it should be checked.  Let's talk about the boot loader.  So the U-boot boot loader is more or less  the de facto standard thing for embedded devices.  It's a very powerful software.  It sets up some hardware at the boot-up process.  It sends the command line arguments for the kernel.  It verifies the kernel in this particular case,  and boots it, and it uses environment partition  or configuration to configure itself.  For example, it's used for if you want to update  like a partition, you have typically two copies  of a partition.  If you want to update copy B,  you can control it over U-boot.  On the other side, U-boot has a very powerful command set.  It can do a lot of things.  For example, allowing to load partitions,  access memory, changing memory,  because sometimes you need that to set up hardware.  The thing which I was wondering about was like,  huh, okay, it can access and change memory.  I wonder if we can use that for something.  The idea was, can we ask U-boot nicely to modify itself?  The theory is if you have memory reads and writes,  we might be able to overwrite instructions  in the U-boot itself, which for example,  checks, verifies the kernel signature.  We had to do a little bit of math,  so we have to figure out basically  where is the actual signature function,  which was a little bit tricky to figure out  if you don't have the source code.  Then we have to figure out the U-boot location offset,  because there's some technical reason behind that.  We need to figure out where is this function in the memory.  In the end of the day, we figured out where it is,  and then we patched, basically, we set two bytes  in one particular memory location,  and the same was all the verifications forward  after U-boot.  The interesting thing is the device  is still in secure boot mode.  Nothing notices that something is going wrong.  Everything is fine.  What's the consequence of that?  If we go back into our original example,  if we have a malicious U-boot configuration,  U-boot patches itself, so U-boot is compromised.  Because we have U-boot compromised,  we compromised the verification check.  Then we can compromise the kernel,  because nothing is checking it anymore.  If you compromise the kernel,  then the kernel doesn't check anymore  the root file system or the software,  and because Opti never learns that anything bad happened,  it will happily give us the keys,  which we need to do partitions, and everything works.  So this system is more or less screwed.  So what did we achieve with this small one-line thing?  We bypassed the secure boot process.  We modified the kernel,  which means we removed the DM-verity checks,  we removed the ELF binary checks,  and we disabled SE-Linux.  And this allows us to modify the root file system,  so we're back in the game, basically.  And with that, we can also install custom software  and get SSH access.  The interesting thing is when we were playing around  with this model, with this approach,  I mean, it's kind of known that the environment partitions  is kind of like a problem,  but most of the time people use it  to kind of boot their own shell scripts.  But in this particular case,  we can just patch the U-boot persistently,  and not only on raking robots,  but we tested this also for smart speakers,  media devices, and many, many other devices.  So if you happen to have a device which uses U-boot,  and which is kind of locked, you might want to try that.  It's a little bit of effort in getting it to run initially,  but it's definitely worth it.  So we still have a problem.  Well, how do we modify the environment?  Because, I mean, it's like on a flash,  so we don't have access.  Well, without root access, we can't modify it directly.  We cannot also modify the root file system,  so we cannot put our custom software onto it.  And we don't want to disorder the flash.  So the good news is we found for the typical models,  different methods.  For the older Dreamy vacuum robots,  we found to use vStick method,  which, again, they left us, thankfully.  For newer models, which came out very, very recently,  we have like an FAL root, like it's bootloader root.  And for the Roborock models, we have a scary FAL root.  I will explain why we call it scary.  So as a quick recap, if you have been here two years ago,  Dreamy, thankfully, has the back pins,  which are reachable from outside,  so you can just remove like a cover.  It sounds a little bit scary if you kind of pull on it  the first time, but it's very easily removable.  It gives you a UART, it gives you like USB,  it gives you everything.  The same is the case, I mean, we never changed it,  so even for the new models, for the R models,  which are the ones which came out after 2022,  they still use the same thing.  So, again, a thing which we kind of figured out  by looking at firmware updates,  Dreamy left us a nice backdoor.  Basically, if you connect a USB stick,  an empty USB stick, it doesn't matter what USB stick it is,  to the USB port, it will pop a shell.  And at some point, I think they plan  to have some actual verification there,  but as you see, if the verification fails,  it still gets you a shell.  So I think this was a work in progress,  and someone forgot about that.  So that way, as you see, it doesn't matter  if it's doing anything or not, you will always get a shell.  So after you get access, basically, to the shell,  you can just patch environment, and you're good to go.  The sad news is, for whatever reason,  we have two branches, and for the newer R models in 2022,  they have a better check, so it doesn't work there.  Instead, we developed this method,  where we basically bring the device  into a particular bootloader mode,  and then we have a weird, sketchy firmware package  which we installed with a very sketchy software.  And it doesn't actually install any firmware,  but it gives you an interface to the flash  and to some other system functions,  which, surprisingly, if you connect over USB  in the bootloader mode,  you don't need to have signed firmware.  So you can execute unsigned code over USB.  You don't have access to TrustZone,  but that's fine for us.  We just wanna have access to the flash,  and from there, we can just modify the flash.  So I have a how-to here.  The technical process is a little bit complicated,  but we did lots of pictures, and so don't worry.  I mean, many people did that already and tested it,  thankfully, and didn't break their devices.  Well, one did, but we fixed it, so.  It should be safe.  At this time, it should be safe.  I kinda promise.  If you wanna get an adapter, I have some adapters with me.  You can just grab one.  Sadly, I got rid of 100 adapters  two days ago in my demo session,  but I still have some left.  Otherwise, you find the Gerber files on this website.  Let's talk about this scary FAL route.  The problem with Roborock is  they didn't give us any exposed debug pins,  so we have only USB available from outside,  which we can't really use.  So the first approach, which I typically do,  is I buy the device.  I spend $1,000 and disassemble it and analyze it.  I remove the SOC, which is hundreds of pins,  and try to track down which pin is.  I remove also, obviously, the EMMC  and read the EMMC out to kinda figure out what's going on,  and then I start to map all the pins visually,  so I use my favorite hacking software, MSPaint,  and kinda overlay the pictures,  and rotate one so that I can basically trace down,  like, okay, this is the data line from the EMMC.  It goes in this via, and it comes out on the other side.  As you can see, ah, the picture's a little bit weird.  It comes out from the other side,  and so you can basically connect these pictures.  One thing which we figured out,  and Roborock was very, very great with layouting it,  you can access all the EMMC data pins  from the button, which is, you just remove the cover.  There's just some rubber thing.  You don't need to disassemble the robot.  No warranty seals are broken.  You can access the data pins from outside,  which is great for us.  So, as you see, we have a data zero, data one,  and all the other pins, the clock.  We have everything, which is very nice.  So, now we come to the scary part.  We can use a needle to basically  ground the data zero pin to ground,  and that will bring the device into the bootloader mode,  and then we can just flash it from USB.  So, without disassembling,  without touching any single screw,  you can just scratch off a little bit of soda mask.  You connect a wire to it.  Just hold it and power it on,  and then it should be in the bootloader mode.  So, it's more or less the same approach for the Dreamy,  so it's just, in this particular case,  you need to be a little bit more scary.  Why is it scary?  Well, it requests removal of soda mask,  so you need to scratch a little bit.  If you're comfortable with that,  you still need to tear it down,  but if you're unsure, ask other users.  There's a lot of people who do that.  If you have a different robot from Roborock,  like a G10S, which is very often used in Russia right now,  I heard, it's more or less the same process  as for the other robots,  but you need to disassemble it  because we don't have any access from outside.  I checked all the possibilities  of burning holes into the board,  but sadly, that doesn't work.  So, check also my website, robotinfo, for the pinouts,  and ask for help in the community.  Right, what do we do with root access?  So, we have root access now, in some way,  so what do we do now?  We defeated Secure Boot, we can run custom software,  but what kind of software can we run,  and can we build our own software  which kind of does everything?  Well, we are a little bit lazy,  so we actually, our main idea is,  well, we just want to disconnect the cloud,  but keep all the vendor cleaning logic in place,  because, I mean, we are not software engineers,  so we're just hackers,  so we, you know, if the world works,  then, you know, it's still fine.  So, if you get rid of the cloud,  the robot should still work,  but the problem is, like,  well, we don't have live maps,  we don't have advanced features like map editing,  so, you know, why did we even root it  if we just, you know, cut the internet?  So, and what happens if we need to redirect traffic?  So, the problem is, like, we use, obviously, secrets,  or keys, and so we need to extract them from the robot,  and point it, like, to, like, a cloud emulation,  so it's a little bit tricky, right?  But we can do that.  So, the initial approach which we had was quite naive,  so we just redirect DNS to our own software,  and, you know, we just use IP tables,  but Xiaomi kind of figured out that people do that,  so they kind of introduce, like,  hard-coded IP addresses in their binaries.  Our solution was basically to just, you know,  replace their hard-coded IP addresses  with our hard-coded IP addresses,  so it's just an initial step,  so, as you see, we're kind of aware of what we do,  and then we do the same things, basically, in reverse,  which means we can run our own software,  and we can redirect the traffic,  and there's a great software, which is Walletudo,  which is developed by Zoom.  It's a complete replacement of the cloud applications  and the API and the app itself.  So, you can run your vacuum robot completely offline  with all the functionalities which the cloud  and the API, so the smartphone app offers you.  You can see live maps, you can edit the maps,  you can change the robot configuration.  It's a very responsive web interface.  It gives you the API for REST,  and it gives you MQTT functionality,  for example, for Home Assistant.  The problem is, I mean, that's not a problem.  My personal thing is,  Xeron loves to write it in JavaScript,  and so Node.js.  I'm not sure if it's the most performant thing,  but it works, so it's fine with me.  Some screenshots, just to give you information  on how it looks like, so you get, actually, the maps.  You can do a lot of interesting things with that.  The question is now, how do you get the custom firmware?  I made something very, very easy,  which is called DustBuilder.  It's basically a website where you can  let the website build the custom firmware for you.  You just upload the SSH key and do some other stuff,  and it will give you the custom firmware,  which you can use then to push on your robot.  The scripts are technically public on the GitHub,  so you can just, if you want to reproduce that,  you can do that.  So, last couple minutes, that's the reason  why I'm getting a little bit faster.  What else did we find on the robots  while we were doing this research?  Well, all the cameras are, I mean,  it's a Linux operation system on all the robots,  so all the devices use the video for Linux subsystem,  and you can access the cameras  through their device file, basically.  And some of the vendors were even nice enough  to let some debugging tools to debug the camera  on the device.  And I just want to give you some examples here.  This is a G10S from Roborock.  I made, so it's self-aware.  It makes a selfie of itself,  and you see the quality, again, from the optical sensors.  For the L10SU Ultra, they gave it a RGB camera,  which is also nice, so you see, again,  a selfie of a robot with a DEF CON flag.  And on the right, you see what it typically sees  with the camera, if it kind of starts to detect AI,  with the AI objects in your house.  And this is the same example for the Roborock S8.  It has an infrared camera,  so the sad thing is on the right,  as you see there, this looks a little bit bland.  This is the blue elephant,  so you cannot see the blue in the picture,  which is kind of sad, but it's okay.  I mean, you still see that the optical sensor  sees something, right?  So, something about the Dreamy.  The good news is this year,  we didn't find any SSH credentials  to the backend in the firmware, so that's good.  They made a lot of improvements of the software,  but the bad news is they started to introduce  a lot of telemetry in calling home functions,  which is a little bit sad.  And they started to enforce geo-blocking by IP address.  So if you buy a device from China again,  you cannot use it anywhere outside of China.  You need to kind of do some hacking, I guess.  There were a couple problems.  So all the vacuum robots, what they do for legal reasons  is if you access the camera remotely,  the robot will say like, oh, live view is in effect,  so be careful, or whatever,  which is kind of like a legal requirement.  The problem is these audio files  are kind of part of audio packages,  and the audio packages are not signed or encrypted.  So one thing you can do technically,  and don't do that because it's probably illegal  in your jurisdiction,  you can just overwrite this prompt, this warning prompt,  which the robot will kind of blare out every three minutes  by an empty file, and then you can access the camera  without anyone noticing.  But it's super illegal, don't do it.  The thing is, you can do it also for non-routed devices,  which might be a problem.  One interesting thing is Dreamy kind of messed up  by the way the firmware signatures.  So we didn't really use that,  but I mean, it's kind of interesting.  So we use encryption and signing,  and the way how we do that is very interesting.  The AutoZip archive is encrypted with a static password.  Then they created a random file,  which was signed with a private key,  and then they used the random file  as a password for another zip archive  inside of the whole thing,  which is basically the firmware.  The problem is the actual firmware is not signed.  The password is signed, but not the firmware.  So the thing is, if you want to create  your own custom firmware updates,  you just reuse the same password  because it's already signed.  I think someone had a good, I don't know.  I mean, we were not sure if it was a trick  or if they kind of messed up.  It was kind of a little bit weird.  Anyway, so last two slides.  As a summary, we have routing methods  for the most of the current released Dreamy  and Roborock robots, and technically some others too.  We can bypass secure boot.  We can bypass any other security mechanisms.  We even routed devices which came out two weeks ago.  So basically we are at the cutting edge here.  You get persistence, and we can run our custom firmware.  Now we can validate and verify the vendor's claims,  and the bad news is some of them are tricky.  But the interesting thing here is the takeaway message.  Even if you're not doing any robot hacking,  the bootloader attack is technically applicable  for many, many different other devices.  We saw it in, like I said, smart speakers and other things.  This routing allows us basically to do further research  into IoT and AI, because now we have access  to the AI machine learning models,  and we can basically take a look at and compare them  like more in a more qualitative way.  So, the final notes.  Do not use your knowledge for bad things.  I mean, I know people like to have the word burned down,  but please don't do that.  Help us to help others.  So if you have a routing PCB,  share it with other people as soon as you're done.  We will publish the tools in the next couple years,  some of, next couple years, next couple days.  So some of them are already public,  some of them will be enabled in the next couple hours  when I'm kind of getting more cooled down.  So please be patient if some things are not public yet.  If you happen to be a hardware I.O. in the Netherlands  in October or November, in the beginning of November,  feel free to join me.  I might be there with some new things  about regarding robots, which are not these robots,  but some completely new company.  All right, final thing.  I would like to thank Danny Biggema,  with whom I started to research.  I would also like to thank Gevaar Nebier,  Zion Bayer, who is obviously the developer,  and also Mikael Kalkin,  and all the testers in the community.  There were a lot of people who were sacrificed,  or nearly sacrificed their devices  to test our routing methods.  Thankfully, we have no brick devices so far,  I mean, it could have happened,  and this would be kind of sad.  I got an email from one of the people,  like, oh, my wife would kill me if I destroy this device,  so let's hope that it doesn't break.  All right, and that's the end of my presentation.  Thank you very much for being here again  at 10 a.m. in the morning.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.


url: https://www.youtube.com/watch?v=QMO2vMBcx7Y
title: DEF CON 31 - Unlocking Hidden Powers in Xtensa based Qualcomm Wifi Chips - Daniel Wegemer
type: video
author: DEFCONConference
ingestion_date: 2024-08-03
transcription_model: large-v2
content:
  {
    "webpage_url": "https://www.youtube.com/watch?v=QMO2vMBcx7Y",
    "title": "DEF CON 31 - Unlocking Hidden Powers in Xtensa based Qualcomm Wifi Chips - Daniel Wegemer",
    "description": "Wifi chips contain general purpose processors. Even though these are powerful processors, their firmware is closed source and does not allow modifications. This talk explores how the firmware of modern Xtensa based Qualcomm Wifi chips can be modified to allow extending its indented functionality. Such modifications can even be for example leveraged by security researchers to find vulnerabilities in an otherwise closed source Wifi code. During the talk we will also dive into the architecture of Qualcomms Wifi chips as well as the structure of the firmware used withing these chips. We will release a modified version of the Nexmon framework to enable patching of Xtensa based firmware and show all the steps involved to create such patches.",
    "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
    "duration": 2211,
    "channel": "DEFCONConference",
    "uploader": "DEFCONConference",
    "upload_date": "20230915"
  }

  This text was transcribed using whisper model: large-v2

   Hey, thanks everybody for showing up so early.  So yeah, I talk about extensor-based Qualcomm Wi-Fi chips, and so why even looking at Wi-Fi  chips?  So in my opinion, these Wi-Fi chips are pretty powerful chips, so even though they are just  intended to do your Wi-Fi stuff, they are still general purpose CPUs, so you in theory  can run any software on it, but the problem is the proprietary binaries which come with  those chips, they make it difficult to run your own code.  And why would we even want to run our own code?  So what we could do is enable additional functionality, I have a few examples in a few seconds.  And we could also make security research easier, so in case we manage to enable dynamic  analysis, security researchers have it easier to identify bugs and report them to the vendors.  So some examples for additional features which we want to enable is for example, we can enable  monitor mode on chips which is not available by default.  For example, this is monitor mode on a Qualcomm-based chip running on a Nexus 5, which is not available  by default, and then you can use your standard Wi-Fi tools like AeroDump and so on.  We could also build a complete state machine for Berkeley packet filtering inside the firmware,  and then compile our rules for packet filtering on the user space, upload them via the kernel  to the firmware, and run them directly in the firmware.  So that way we could save some power when we do monitor mode basically.  And if we have access to even a deeper layer of firmware code, we can also implement stuff  which is closer to the physical layer of Wi-Fi.  So here's an example of a pilot tone jammer, which only jams the pilot tone of OFDM to  be very power-efficient.  So now that we have all these great goals in mind, let's take a few steps back and look  at the bigger picture.  So in general there are two kinds of Wi-Fi chips, so full-MAC and soft-MAC chips.  We will look at full-MAC chips in this talk.  The biggest difference is that full-MAC chips run the MAC layer inside the firmware.  So this also means if you want to change something in the firmware, in the MAC layer  you have to change the firmware directly.  So it's not possible to do these changes on the driver itself.  So the examples I have shown in the previous slides of stuff which we could want to enable,  this was based on the Nexmon framework.  So I developed this together with a colleague of mine back in my time at university.  So this framework will be used to modify Broadcom chips and we even managed to get deep modifications  working as this jammer I showed earlier.  And you can get it at nexmon.org.  And apart from Broadcom there was also work done on Blackcat 2020 about Intel-based chips  and there was also a talk about Hexagon-based Qualcomm chips on Defcon 27 and Blackcat 2019.  But this talk is the first talk to explain extensor-based Qualcomm Wi-Fi firmware.  So some background, how are Wi-Fi SOCs structured?  So normally you have an application core and then separately you have your Wi-Fi core.  And optionally in some cases you also have a real-time core which handles time-critical  stuff like the distributed coordination function in Wi-Fi.  And now for the chip which we will look at, there are several drivers available.  So there's ADH-10K by Qualcomm directly and also a corresponding firmware with that.  And then there's ADH-10K-CT by Candela Tech.  So this is interesting.  So Candela Tech bought the rights from Qualcomm some time ago and they have their own driver  and their own firmware.  So they got the source code somehow from Qualcomm and added additional features.  And so basically you have a complete separate kind of driver and firmware in addition to  the original Qualcomm driver and firmware.  And lastly there's also the QCAC-LD driver by Qualcomm.  This is used for factory processes.  And also what you can do is you can use the Candela Tech driver and run the Qualcomm  firmware.  This is also possible.  So this is the chip and the board I have looked at.  It's the IPQ4019.  It's based on a development board by 8 devices.  This board is called Habanero.  And in the middle of it you can see the chip we are interested in.  And this is basically a development board for Wi-Fi enabled home routers.  So what does this chip look like from the inside?  So as I said, this is used for Wi-Fi enabled home routers.  And one of the big vendors in Germany is also using it.  So this is AVM, the Fritzbox.  And apart from this, the chip consists of, as I explained before, also of an application  core and multiple Wi-Fi cores.  And the application core runs on OpenWRT, in this case, with a pretty old kernel.  And for Wi-Fi they have one core for 2.4 gigahertz and one for 5 gigahertz.  And they use PCIe to communicate with each other.  So now to the firmware itself.  As I said, it's Extensa-based firmware.  Extensa was initially developed by Tensilica, but now it was bought by Cadence.  And it's a little-endian firmware.  It consists of a ROM part and a RAM part.  The RAM part of this firmware is stored in the file system of the OpenWRT.  And it contains multiple segments, more on that later.  And it's LC77 compressed.  The ROM can be patched.  And there's also a code swap mechanism, which allows the Wi-Fi core to evict code from its  own memory space to the host memory space.  And of course, there's no security enabled by default.  So no secure boot, no stack can erase, no address randomization.  What's also nice is that there is already, by default, there comes this debugfs, which  you can compile into your driver.  And this allows you to get direct memory access via this memvalue file.  So you can directly read and write memory directly into the chip.  There's also a debugmask, which is very useful, which you can set, for example, during initialization  of the kernel module to increase the verbosity of the driver itself.  Now, so the interfaces which are used between basically the application core and the Wi-Fi  core.  So there are two interfaces, the PMI interface, the bootloader management and messaging interface  is used during bootup.  And it's implemented in ROM.  So basically during bootup, this is used to start up the chip.  And then after bootup, the WMI is used to communicate with the chip.  And this is mostly done to send commands like, please now start Wi-Fi scanning, do channel  configuration, stuff like that.  So this is how the loading looks like.  So the driver offers two methods for loading firmware, either via the BMI method or via  a copy engine.  And because in our case we have a compressed firmware, we need to use the BMI method.  And what's basically happening is that the firmware from the file system is passed through  the driver via BMI to the Wi-Fi core and is then uncompressed on the Wi-Fi core itself.  So this is the file structure of the firmware file.  So basically there are two big parts, like this upper one and this lower one.  They are identified with an IE header.  And then after this, there is the segment header.  And this is identified with these metric values here.  It will tell you if it's compressed or not, and on which address it should be loaded into  the chip.  And it will also tell you what's the decompressed size of this segment.  And if you notice already, in this case, the first part and the second part have the  same address, so this means that they get overwritten.  And then, apart from the segment header, there's also some metadata, as I just explained.  And after this, the real data starts, and then it's compressed.  And after this, the next metadata starts in the next part, which is then also of the firmware  data, which is also compressed.  So basically, why these two separate parts is, my assumption is that the first part is  just needed for boot up, and you can also see it's much smaller in size than the second  part.  And after this, we can basically override it with the real firmware code.  And if you look at the driver logs, you will see this whole process being done twice.  And I assume this is done to load up the first Wi-Fi core and the second Wi-Fi core.  If you look at the memory layout, we can see that it's repeating.  So if you just do a long enough read of your address space, you will see these patterns.  And my assumption is that this is used to realize different memory access rights.  So you can have the same memory on different offsets, and depending on offset, you can  realize different access rights.  Then to the extensor architecture.  So there are two major things I want to point out about this architecture, which were new  to me and which also came with some problems.  So the first one is the use of literal pools.  So each load instruction, so this is just a load instruction, instead of being PC dependent,  it's basically independent of that, and you just calculate an offset to a fixed literal  base to get an immediate value.  In addition to that, there is also windowed registers, and so this means that that way  I explained it in a second, but what it's used for is basically to not needing to store  and to save and restore your registers when you call a function.  So here in this example, it's using call 8, so there can be different values here,  or 4 or 12 or whatever, but my firmware used call 8, and if you load a value in A10 and  you call a function, it will be available in A2 of the callee.  So this is how these window registers look like.  So basically you have these overlapping sections in your memory space, and you have way more  registers than you need for one function, and that way they can just shift the window  for each function call, if you have nested functions, and they can basically just rename  it and make it accessible to the next function.  This is how literal pools work, so now the memory space is horizontal, so assume we have  a function myPatch which wants to call a function wlanMain, so the assembler for this will look  like there's first a load instruction which loads intermediate value into a register,  and then the call is just using this register to call the function, and if we want to get  this immediate value, we need to know the literal base, and then we can look up the  offset for this function we want to call, wlanMain in this case, and then we get the  actual address of the function, so for example here.  And this is done for each load instruction.  So what's also the case is that this literal base needs to be set up somewhere, and this  is usually done at the very beginning of the firmware, and this is a fixed value, so for  me it was 0x48001, and this is the code, how it looks like, so you basically have the start  of this literal pool and its size, and then you store it into A2 and write this to the  special register base, and so actually the offsets are negative, this is why you add  the whole size to it and then a load instruction later on will have a negative offset.  So basically the existing firmware does expect that this literal pool is used, so if we want,  and this has some, this comes with some problems if you want to patch existing code.  Another problem was that this lit base is not supported by disassemblers, so for example  IDA77 added support for Extensa but has ignored the lit base call, for JITRA there is an Extensa  plugin but it didn't support lit base either, RADAR2 ignored lit base, has support for Extensa,  and there's also a Binary Ninja plugin but it ignored lit base as well.  So what I need to do is create some patches for Binary Ninja in this case, so in this  patch I just ignore the dependency of the program counter and directly use this fixed  lit base.  And I have a similar, so this is the patch for Binary Ninja and there's also, this is  the patch for JITRA which works the same way, and I have a link to a Git repository  later on where you can find these patches.  So now if you actually want to change some code in the firmware I decided to use the  Nexmon framework I have shown earlier and I wanted to basically modify it to support  Qualcomm based WiFi chips.  So this is how the framework looks like, it's pretty complicated but we only need a subset  of this to make it work.  So what we could do with the original Nexmon was we could extract the ROM, flash patches,  the RAM, the U-code for Qualcomm based WiFi chips and we could use C to write our patches  and of course we could call existing firmware functions and then in the end with this whole  framework we were able to create a firmware file which we could then run on Qualcomm based  WiFi chips.  So how does this Nexmon framework work on a high level?  So we have our own patches which are in patch.c, we have wrapper.c which is used to have some  stubs for the functions which are already existing in the firmware and we use GCC and  the GCC plugin to compile the .o files and also the plugin will help to create this Nexmon.pre  file and this Nexmon.pre file basically contains some metadata which we use later on and for  example we can use this file with an org script to create some linker scripts and then we  can use the linker scripts and the .o file to create an overall patch file.  So this patch file basically includes everything which we need.  And now after this we can use another make file and the elf file to create basically  a blueprint to copy out the relevant section of this elf file.  So we might not need everything, just the patch we wanted to introduce and some stuff  we might want to override in the original function and the original firmware.  So now we have this elf file and this make file which is basically the blueprint for  object copy.  We can copy out the relevant sections and then we can DD them into the firmware binary.  So this is how the original Nexmon worked.  So I needed to do some changes to make it work for Qualcomm.  So first of all there was no support for decompression, the Qualcomm based firmware had no compression  it was just the whole memory, the whole RAM of the chip basically uncompressed.  It also needed to add support for multiple binaries.  So as we have seen earlier the firmware consists of multiple segments and I needed to add support  for that and we also needed support for the LIT base I mentioned earlier.  And then if we make this happen what we could do is compile and link our own patches, we  could patch the second segment I showed earlier and then we can compress it back into the  second segment, add some padding bytes and then write the firmware file basically.  So this is the plan.  So I will not explain the decompression, this is pretty straightforward to implement.  I will start with the support for multiple binaries.  So I needed to extend the GCC plugin which creates this metadata file, the Nexmon.pre  file.  So basically what this GCC plugin does, it looks for this attribute in the source code  and this attribute tells basically where the code after this goes into the firmware in  which chip this is used and in which firmware version this is used.  And I extended this basically to also include the target firmware file.  So after compiling where should this go to and which segment of the firmware file should  this go into.  So basically then after we compile everything we have this new Nexmon.pre file and it basically  contains the address which is here where our code should go, it has basically the type,  so here it's a patch, then if it's a dummy it's basically an existing function in the  firmware, the next column is the .o file that this will be compiled into, this is the name  of the text section in the .l file and now we also added the name of this bin file where  the compiled code should later on be copied into the RTD.  So with this I already wanted to compile some patches, so my goal was to use this pretty  simple patch, so just jump into some code, into this function and then write 1, 2, 3,  4 basically into this memory location and then jump to the original code which is WLAN  main in this case.  I could use the ESP32 GCC compiler to compile my stuff and load it into the chip and I used  the debugfs which I have shown earlier to check basically after the chip was up if the  memory address has changed accordingly.  But this didn't work and the reason is this libbase I explained earlier, so basically  all the load instructions expect this libbase to be set at the very beginning of the firmware  code and so this means like the chip which runs the existing firmware code expects a  load to honour this libbase and I need basically now some way to tell my code either like there  is this existing libbase, please use this and it's like already this filled up.  But unfortunately there is like no such parameter existing in GCC so we need to do something  else and I just highlighted here all the load instructions which are problematic,  as I said the offsets are not right, they need to use this libbase.  So what we could do as an alternative is just avoid load instructions and handcraft our  assembly ourselves, so we just avoid all the references and have basically no immediate  value so this is obviously not something we want to do in the long run.  So I came up with two possible solutions, either as I said earlier we can tell the linker  somehow where the existing libbase is and how full it is or we can use our own libbase  value so I decided to go with the second option, I felt like this is more flexible in the long  run and what this implies is basically that I set the libbase to zero at the entry of  every function, this can be done using the GCC plugin and at the function exit I need  to reset it to its original value.  And I needed to do this into the assembler itself because there is this target dependent  feature which is called relaxation, so if you have a call to a function this assembler  will basically relax this to a load and a call and then we have again a call which is  problematic.  So how does this look like in the assembly, so as I said this relaxation is problematic  and after relaxation we basically have a call extended to two instructions, this load and  this call, and the thing is that in between those two basically we need to reset the libbase  to its original value so that then if I call the existing function that the remaining code  can run as intended.  And it took me a while to find out that I cannot manipulate this behaviour in GCC, it  needs to be patched in the assembler directly because it's target dependent.  So how does this relaxation look like in the assembler, so basically you have this sort  of lookup table, on the left hand side it's looking for a pattern like this, so here it's  looking for a call 8 and if it finds this kind of pattern it will be relaxed to a load  and a call x8.  And what I basically need to do is find a place in the assembler where these built instructions,  so basically the right hand side here, these are called built instructions, where those  are iterated through and then they're applied and pushed on a stack.  And all I did is basically to look like is the current opcode a call x and was the previous  opcode a load instruction and if this is the case then I can basically add an additional  instruction which will reset the lid base with a WSR instruction.  So now that we know all of this and we have patched our assembler we can go back to our  original patching code and we can use this GCC plugin and the patched assembler to compile  this much easier to look at code, which is pretty nice.  So this is how the assembly looks like basically.  After we have saved our parameters we will save the current lid base into A15, we will  write 0 into A14 and write A14 into the lid base value and then we can do all of our load  instructions and at the end we reset or we copy back the value which we have stored  in A15 into the lid base value before we do the final call out.  So there are still some open problems with this, so this implementation is not very good  obviously.  I take away two registers for the assembler, it would be better to have this based on a  stack based implementation.  Also there is this support of disassemblers which I mentioned earlier, so I have these  patches but still, for example you cannot use IDA if you want to and also there is no  support in this GCC I'm using for naked functions, so this is a problem.  So what I do is, you need some way to overwrite an existing call to even get to your patch  and either you are lucky and they are using the literal pool, then you can just overwrite  the value in the literal pool, but if not, if it's a direct call, then you need some  way to compile a call to your own code and you could easily do this with naked functions  but this GCC version, or GCC for Xtensa at least, does not have this option for naked  functions.  And also, what's also missing right now is that there is no text console, so you cannot  just do a printf and somehow dump this in your host system to do some basic debugging,  but we could implement this ourselves now.  So basically, I have this very creative name now for my framework which is called QCAmon  instead of Nexmon, so this is how the folder structure looks like, at the very top you  have some build tools, then the disassembler patches which I mentioned earlier for Jitra  and Banner Ninja and then there is a folder for the original firmwares and a folder for  our patches which we want to apply.  The build tools contain the patched assembler and GCC and also the GCC plugin.  The firmware folder basically contains the original firmware and it already comes with  a structure which would allow different chips and different firmware versions and there  is also a makefile which already extracts the relevant parts and decompresses them.  And then at the end we have our patch folder which we can use to store our source code  which we want to introduce into the firmware.  So now to a brief demo.  So this will basically show the code I had in my slides earlier.  So we just want to set 1, 2, 3, 4 or store 1, 2, 3, 4 into this memory address and then  jump to the original code and in addition to what I've shown earlier we also need these  two lines to basically have an entry point into our firmware code.  So all I'm doing here is creating basically overwriting stuff in the literal pool to point  to my code.  So everything is prepared in the repository.  You just need to run make and this will already compile the C files and .o files.  It will also compile the wrapper, it will create the linker scripts and create the .elf  and then use this blueprint makefile I mentioned earlier to copy out the relevant parts.  Then we can start reassembling the whole firmware file via compressing it, adding some padding  bytes and then basically creating the firmware file how it's expected by the Wi-Fi chip.  So here it's called firmware-5.bin and I will now SSH into this board I've shown earlier  and in order to access the debug.fs we first need to set up the WLAN0 interface.  Now we can go into the debug.fs and look at the memory location basically without any  modifications.  So for this I can just use dd to copy out the relevant section and use hexdump to look  at it.  So here we see that the first few bytes are not 1, 2, 3, 4.  And now we install our modified firmware.  Just copy it over via SCP and we will need to remove and add the PCIe driver to apply  the changes.  Now we can SSH back into the router, just checking the kernel lock real quick.  So here it's already setting up the regulatory domain so this looks quite good so it did  not crash on us.  And we can go back into the debug.fs, set up the WLAN0 first, go back to the debug.fs.  Then basically we can copy out the same memory location which we just did a second ago and  have a look at it via the hexdump.  And as you can see here, we modified two whole bytes, thanks.  So with this quick summary and future work, I was able to modify the next one framework  to make it work with the Qualcomm based firmwares.  I have a demo patch I have shown, you can get it via this location.  There's also the patches for Binary Ninja and GDraw in there, the GCC plugin and already  a pre-compiled version of GCC and also the patched Benutils file for the assembler.  So this shows that it's possible to make modifications on Qualcomm based firmwares, there are still  some improvements which I need to do to make modifications even more easier.  I also want to look at this production software driver thing I mentioned at the very beginning,  so this QCAC-LD, so this is basically using this QDAL software on a PC to communicate  and you can get it from some sketchy Chinese servers and then I want to see if I can maybe  enable some features which are used in this production software also by myself and I also  want to look into this code swap feature a little bit more.  So I want to thank Martin Kort aka Problem Kaput, so he did some awesome Game Boy Advance  reverse engineering and he helped me out in some cases, so it turns out that he did not  even reverse engineer the main processor of the Game Boy Advance but also the Wi-Fi chip  used there and this turns out to be an older Atheros based Wi-Fi chip which he also reverse  engineered and Atheros later on was also bought by Qualcomm and I think lots of knowledge  from back there still went into their products.  And I also want to thank Roku aka RKNibble, I used her script to do the unzipping of  the firmware file.  That's it from my side, thanks everybody, if you want to reach out, here's my email address.  If you have any questions, shoot.  I have also one more note, I have found a firmware file which contains all debug symbols  for this IPQ4019 chip, I have no easy way to distribute this without the original repository  being taken down, but if you want to poke at it yourself, reach out to me, I will send  you the link.  Any questions?  No?  Okay then, thanks everybody.  APPLAUSE


