---
source: src/feeds/atom.rs
expression: result
---
[
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/06/23/a-few-debugging-resources/",
        link: "https://jvns.ca/blog/2019/06/23/a-few-debugging-resources/",
        title: "What does debugging a program look like?",
        date: 2019-06-23T18:48:35Z,
        summary: None,
        content: Some(
            "\n\n<p>I was debugging with a friend who‚Äôs a relatively new programmer yesterday, and showed them a few\ndebugging tips. Then I was thinking about how to teach debugging this morning, and <a href=\"https://twitter.com/b0rk/status/1142825259546140673\" rel=\"noopener noreferrer\">mentioned on\nTwitter</a> that I‚Äôd never seen a really good\nguide to debugging your code.  (there are a ton of really great replies by Anne Ogborn to that tweet\nif you are interested in debugging tips)</p>\n\n<p>As usual, I got a lot of helpful answers and now I have a few ideas about how to teach debugging\nskills / describe the process of debugging.</p>\n\n<h3>a couple of debugging resources</h3>\n\n<p>I was hoping for more links to debugging books/guides, but here are the 2 recommendations I\ngot:</p>\n\n<p><strong>‚ÄúDebugging‚Äù by David Agans</strong>: Several people recommended the book\n<a href=\"http://debuggingrules.com/\" rel=\"noopener noreferrer\">Debugging</a>, which looks like a nice and fairly short book that explains\na debugging strategy. I haven‚Äôt read it yet (though I ordered it to see if I should be recommending\nit) and the rules laid out in the book (‚Äúunderstand the system‚Äù, ‚Äúmake it fail‚Äù, ‚Äúquit thinking and\nlook‚Äù, ‚Äúdivide and conquer‚Äù, ‚Äúchange one thing at a time‚Äù, ‚Äúkeep an audit trail‚Äù, ‚Äúcheck the plug‚Äù,\n‚Äúget a fresh view‚Äù, and ‚Äúif you didn‚Äôt fix it, it ain‚Äôt fixed‚Äù) seem extremely resaonable :).  He\nalso has a charming <a href=\"http://debuggingrules.com/?page_id=40\" rel=\"noopener noreferrer\">debugging poster</a>.</p>\n\n<p><strong>‚ÄúHow to debug‚Äù by John Regehr</strong>: <a href=\"https://blog.regehr.org/archives/199\" rel=\"noopener noreferrer\">How to Debug</a> is a very\ngood blog post based on Regehr‚Äôs experience teaching a university embedded systems course. Lots of\ngood advice.  He also has a <a href=\"https://blog.regehr.org/archives/849\" rel=\"noopener noreferrer\">blog post reviewing 4 books about debugging</a>, including Agans‚Äô book.</p>\n\n<h3>reproduce your bug (but how do you do that?)</h3>\n\n<p>The rest of this post is going to be an attempt to aggregate different ideas about debugging\npeople tweeted at me.</p>\n\n<p>Somewhat obviously, everybody agrees that being able to consistently reproduce a bug is important if\nyou want to figure out what‚Äôs going on. I have an intuitive sense for how to do this but I‚Äôm not\nsure how to <strong>explain</strong> how to go from ‚ÄúI saw this bug twice‚Äù to ‚ÄúI can consistently reproduce this\nbug on demand on my laptop‚Äù, and I wonder whether the techniques you use to do this depend on the\ndomain (backend web dev, frontend,  mobile, games, C++ programs, embedded etc).</p>\n\n<h3>reproduce your bug <em>quickly</em></h3>\n\n<p>Everybody also agrees that it‚Äôs extremely useful be able to reproduce the bug quickly (if it takes\nyou 3 minutes to check if every change helped, iterating is VERY SLOW).</p>\n\n<p>A few suggested approaches:</p>\n\n<ul>\n<li>for something that requires clicking on a bunch of things in a browser to reproduce, recording\nwhat you clicked on with <a href=\"https://www.seleniumhq.org/\" rel=\"noopener noreferrer\">Selenium</a> and getting Selenium to replay\nthe UI interactions (suggested <a href=\"https://twitter.com/AnnieTheObscure/status/1142843984642899968\" rel=\"noopener noreferrer\">here</a>)</li>\n<li>writing a unit test that reproduces the bug (if you can). bonus: you can add this to your test\nsuite later if it makes sense</li>\n<li>writing a script / finding a command line incantation that does it (like <code>curl MY_APP.local/whatever</code>)</li>\n</ul>\n\n<h3>accept that it‚Äôs probably your code‚Äôs fault</h3>\n\n<p>Sometimes I see a problem and I‚Äôm like ‚Äúoh, library X has a bug‚Äù, ‚Äúoh, it‚Äôs DNS‚Äù, ‚Äúoh, SOME OTHER\nTHING THAT IS NOT MY CODE is broken‚Äù. And sometimes it‚Äôs not my code! But in general between an\nestablished library and my code that I wrote last month, usually it‚Äôs my code that I wrote last\nmonth that‚Äôs the problem :).</p>\n\n<h3>start doing experiments</h3>\n\n<p>@act_gardner gave a <a href=\"https://twitter.com/act_gardner/status/1142838587437830144\" rel=\"noopener noreferrer\">nice, short explanation of what you have to do after you reproduce your\nbug</a></p>\n\n<blockquote>\n<p>I try to encourage people to first fully understand the bug - What‚Äôs happening? What do you expect\nto happen? When does it happen? When does it not happen? Then apply their mental model of the\nsystem to guess at what could be breaking and come up with experiments.</p>\n\n<p>Experiments could be changing or removing code, making API calls from a REPL, trying new inputs,\npoking at memory values with a debugger or print statements.</p>\n</blockquote>\n\n<p>I think the loop here may be:</p>\n\n<ul>\n<li>make guess about one aspect about what might be happening (‚Äúthis variable is set to X where it\nshould be Y‚Äù, ‚Äúthe server is being sent the wrong request‚Äù, ‚Äúthis code is never running at all‚Äù)</li>\n<li>do experiment to check that guess</li>\n<li>repeat until you understand what‚Äôs going on</li>\n</ul>\n\n<h3>change one thing at a time</h3>\n\n<p>Everybody definitely agrees that it is important to change one thing a time when doing an\nexperiment to verify an assumption.</p>\n\n<h3>check your assumptions</h3>\n\n<p>A lot of debugging is realizing that something you were <strong>sure</strong> was true (‚Äúwait this request is\ngoing to the new server, right, not the old one???‚Äú) is actually‚Ä¶ not true. I made an attempt to\n<a href=\"https://twitter.com/b0rk/status/1142812831420768257\" rel=\"noopener noreferrer\">list some common incorrect assumptions</a>. Here\nare some examples:</p>\n\n<ul>\n<li>this variable is set to X (‚Äúthat filename is definitely right‚Äù)</li>\n<li>that variable‚Äôs value can‚Äôt possibly have changed between X and Y</li>\n<li>this code was doing the right thing before</li>\n<li>this function does X</li>\n<li>I‚Äôm editing the right file</li>\n<li>there can‚Äôt be any typos in that line I wrote it is just 1 line of code</li>\n<li>the documentation is correct</li>\n<li>the code I‚Äôm looking at is being executed at some point</li>\n<li>these two pieces of code execute sequentially and not in parallel</li>\n<li>the code does the same thing when compiled in debug / release mode (or with -O2 and without, or‚Ä¶)</li>\n<li>the compiler is not buggy (though this is last on purpose, the compiler is only very rarely to blame :))</li>\n</ul>\n\n<h3>weird methods to get information</h3>\n\n<p>There are a lot of normal ways to do experiments to check your assumptions / guesses about what the\ncode is doing (print out variable values, use a debugger, etc). Sometimes, though, you‚Äôre in a more\ndifficult environment where you can‚Äôt print things out and don‚Äôt have access to a debugger (or it‚Äôs\ninconvenient to do those things, maybe because there are too many events). Some ways to cope:</p>\n\n<ul>\n<li><a href=\"https://twitter.com/cocoaphony/status/1142847665690030080\" rel=\"noopener noreferrer\">adding sounds on mobile</a>: ‚ÄúIn the\nmobile world, I live on this advice. Xcode can play a sound when you hit a breakpoint (and\ncontinue without stopping). I place them certain places in the code, and listen for buzzing Tink\nto indicate tight loops or Morse/Pop pairs to catch unbalanced events‚Äù (also <a href=\"https://twitter.com/AnnieTheObscure/status/1142842421954244608\" rel=\"noopener noreferrer\">this tweet</a>)</li>\n<li>there‚Äôs a very cool talk about <a href=\"https://qnoid.com/2013/06/08/Sound-Debugging.html\" rel=\"noopener noreferrer\">using XCode to play sound for iOS debugging here</a></li>\n<li><a href=\"https://twitter.com/wombatnation/status/1142887843963867136\" rel=\"noopener noreferrer\">adding LEDs</a>: ‚ÄúWhen I did embedded\ndev ages ago on grids of transputers, we wired up an LED to an unused pin on each chip. It was\nsurprisingly effective for diagnosing parallelism issues.‚Äù</li>\n<li><a href=\"https://twitter.com/irvingreid/status/1142887472441040896\" rel=\"noopener noreferrer\">string</a>: ‚ÄúMy networks prof told me\nabout a hack he saw at Xerox in the early days of Ethernet: a tap in the coax with an amp and\nmotor and piece of string. The busier the network was, the faster the string twirled.‚Äù</li>\n<li><a href=\"http://peep.sourceforge.net/intro.html\" rel=\"noopener noreferrer\">peep</a> is a ‚Äúnetwork auralizer‚Äù that translates what‚Äôs\nhappening on your system into sounds. I spent 10 minutes trying to get it to compile and failed so\nfar but it looks very fun and I want to try it!!</li>\n</ul>\n\n<p>The point here is that information is the most important thing and you need to do whatever‚Äôs\nnecessary to get information.</p>\n\n<h3>write your code so it‚Äôs easier to debug</h3>\n\n<p>Another point a few people brought up is that you can improve your program to make it\neasier to debug. tef has a nice post about this: <a href=\"https://programmingisterrible.com/post/173883533613/code-to-debug\" rel=\"noopener noreferrer\">Write code that‚Äôs easy to delete, and easy to debug too.</a> here. I thought this\nwas very true:</p>\n\n<blockquote>\n<p>Debuggable code isn‚Äôt necessarily clean, and code that‚Äôs littered with checks or error handling\nrarely makes for pleasant reading.</p>\n</blockquote>\n\n<p>I think one interpretation of ‚Äúeasy to debug‚Äù is ‚Äúevery single time there‚Äôs an error, the program\nreports to you exactly what happened in an easy to understand way‚Äù. Whenever my program has a\nproblem and says sometihng ‚Äúerror: failure to connect to SOME_IP port 443: connection timeout‚Äù\nI‚Äôm like THANK YOU THAT IS THE KIND OF THING I WANTED TO KNOW and I can check if I need to fix a\nfirewall thing or if I got the wrong IP for some reason or what.</p>\n\n<p>One simple example of this recently: I was making a request to a server I wrote and the\nreponse I got was ‚Äúupstream connect error or disconnect/reset before headers‚Äù. This is an nginx\nerror which basically in this case boiled down to ‚Äúyour program crashed before it sent anything in\nresponse to the request‚Äù. Figuring out the cause of the crash was pretty easy, but having better\nerror handling (returning an error instead of crashing) would have saved me a little time\nbecause instead of having to go check the cause of the crash, I could have just read the error\nmessage and figured out what was going on right away.</p>\n\n<h3>error messages are better than silently failing</h3>\n\n<p>To get closer to the dream of ‚Äúevery single time there‚Äôs an error, the program reports\nto you exactly what happened in an easy to understand way‚Äù you also need to be disciplined about\nimmediately returning an error message instead of silently writing incorrect data / passing a\nnonsense value to another function which will do WHO KNOWS WHAT with it and cause you a gigantic\nheadache. This means adding code like this:</p>\n\n<pre><code>if UNEXPECTED_THING:\n    raise \"oh no THING happened\"\n</code></pre>\n\n<p>This isn‚Äôt easy to get right (it‚Äôs not always obvious where you should be raising errors!‚Äú) but it\nreally helps a lot.</p>\n\n<h3>failure: print out a stack of errors, not just one error.</h3>\n\n<p>Related to returning helpful errors that make it easy to debug: Rust has a really incredible error\nhandling library <a href=\"https://github.com/rust-lang-nursery/failure\" rel=\"noopener noreferrer\">called failure</a> which basicaly lets\nyou return a chain of errors instead of just one error, so you can print out a stack of errors like:</p>\n\n<pre><code>\"error starting server process\" caused by\n\"error initializing logging backend\" caused by\n\"connection failure: timeout connecting to 1.2.3.4 port 1234\".\n</code></pre>\n\n<p>This is SO MUCH MORE useful than just <code>connection failure: timeout connecting to 1.2.3.4 port 1234</code>\nby itself because it tells you the significance of 1.2.3.4 (it‚Äôs something to do with the logging\nbackend!). And I think it‚Äôs also more useful than <code>connection failure: timeout connecting to 1.2.3.4 port 1234</code>\nwith a stack trace, because it summarizes at a high level the parts that went wrong instead of\nmaking you read all the lines in the stack trace (some of which might not be relevant!).</p>\n\n<p>tools like this in other languages:</p>\n\n<ul>\n<li>Go: the idiom to do this seems to be to just concatenate your stack of errors together as a\nbig string so you get ‚Äúerror: thing one: error: thing two : error: thing three‚Äù which works okay but\nis definitely a lot less structured than <code>failure</code>‚Äôs system</li>\n<li>Java: I hear you can give exceptions causes but haven‚Äôt used that myself</li>\n<li>Python 3: you can use <code>raise ... from</code> which sets the <code>__cause__</code> attribute on the exception and then\nyour exceptions will be separated by <code>The above exception was the direct cause of the following\nexception:..</code></li>\n</ul>\n\n<p>If you know how to do this in other languages I‚Äôd be interested to hear!</p>\n\n<h3>understand what the error messages mean</h3>\n\n<p>One sub debugging skill that I take for granted a lot of the time is understanding what error\nmessages mean! I came across this nice graphic explaining <a href=\"https://pythonforbiologists.com/29-common-beginner-errors-on-one-page/\" rel=\"noopener noreferrer\">common Python errors and what they\nmean</a>, which breaks down\nthings like <code>NameError</code>, <code>IOError</code>, etc.</p>\n\n<p>I think a reason interpreting error messages is hard is that understanding a new error message might\nmean learning a new concept ‚Äì <code>NameError</code> can mean ‚ÄúYour code uses a variable outside the scope\nwhere it‚Äôs defined‚Äù, but to really understand that you need to understand what variable scope is! I\nran into this a lot when learning Rust ‚Äì the Rust compiler would be like ‚Äúyou have a weird lifetime\nerror‚Äù and I‚Äôd like be ‚Äúugh ok Rust I get it I will go actually learn about how lifetimes work\nnow!‚Äú.</p>\n\n<p>And a lot of the time error messages are caused by a problem very different from the text of the\nmessage, like how ‚Äúupstream connect error or disconnect/reset before headers‚Äù might mean ‚Äújulia,\nyour server crashed!‚Äú. The skill of understanding what error messages mean is often not transferable\nwhen you switch to a new area (if I started writing a lot of React or something tomorrow, I would\nprobably have no idea what any of the error messages meant!). So this definitely isn‚Äôt just an issue\nfor beginner programmers.</p>\n\n<h3>that‚Äôs all for now!</h3>\n\n<p>I feel like the big thing I‚Äôm missing when talking about debugging skills is a stronger\nunderstanding of where people get stuck with debugging ‚Äì it‚Äôs easy to say ‚Äúwell, you need to\nreproduce the problem, then make a more minimal reproduction, then start coming up with guesses and\nverifying them, and improve your mental model of the system, and then figure it out, then fix the\nproblem and hopefully write a test to make it not come back‚Äù, but ‚Äì where are people actually\ngetting stuck in practice? What are the hardest parts? I have some sense of what the hardest parts\nusually are for me but I‚Äôm still not sure what the hardest parts usually are for someone newer to\ndebugging their code.</p>\n",
        ),
        media_type: None,
        media_link: None,
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/brag-documents/",
        link: "https://jvns.ca/blog/brag-documents/",
        title: "Get your work recognized: write a brag document",
        date: 2019-06-28T18:46:02Z,
        summary: None,
        content: Some(
            "\n\n<p>There‚Äôs this idea that, if you do great work at your job, people will (or should!) automatically\nrecognize that work and reward you for it with promotions / increased pay. In practice, it‚Äôs often\nmore complicated than that ‚Äì some kinds of important work are more visible/memorable than others.\nIt‚Äôs frustrating to have done something really important and later realize that you didn‚Äôt get\nrewarded for it just because the people making the decision didn‚Äôt understand or remember what you\ndid. So I want to talk about a tactic that I and lots of people I work with have used!</p>\n\n<p>This blog post isn‚Äôt just about being promoted or getting raises though. The ideas here have\nactually been more useful to me to help me reflect on themes in my work, what‚Äôs important to me,\nwhat I‚Äôm learning, and what I‚Äôd like to be doing differently. But they‚Äôve definitely helped with\npromotions!</p>\n\n<p>You can also <a href=\"https://jvns.ca/blog/brag-documents/#template\" rel=\"noopener noreferrer\">skip to the brag document template at the end</a>.</p>\n\n<h3>you don‚Äôt remember everything you did</h3>\n\n<p>One thing I‚Äôm always struck by when it comes to performance review time is a feeling of ‚Äúwait, what\n<em>did</em> I do in the last 6 months?‚Äú. This is a kind of demoralizing feeling and it‚Äôs usually not based\nin reality, more in ‚ÄúI forgot what cool stuff I actually did‚Äù.</p>\n\n<p>I invariably end up having to spend a bunch of time looking through my pull requests, tickets,\nlaunch emails, design documents, and more. I always end up finding small (and sometimes\nnot-so-small) things that I completely forgot  I did, like:</p>\n\n<ul>\n<li>mentored an intern 5 months ago</li>\n<li>did a small-but-important security project</li>\n<li>spent a few weeks helping get an important migration over the line</li>\n<li>helped X put together this design doc</li>\n<li>etcetera!</li>\n</ul>\n\n<h3>your manager doesn‚Äôt remember everything you did</h3>\n\n<p>And if you don‚Äôt remember everything important you did, your manager (no matter how great they are!)\nprobably doesn‚Äôt either. And they need to explain to other people why you should be promoted or\ngiven an evaluation like ‚Äúexceeds expectations‚Äù (‚ÄúX‚Äôs work is so awesome!!!!‚Äù doesn‚Äôt fly).</p>\n\n<p>So if your manager is going to effectively advocate for you, they need help.</p>\n\n<h3>here‚Äôs the tactic: write a document listing your accomplishments</h3>\n\n<p>The tactic is pretty simple! Instead of trying to remember everything you did with your brain,\nmaintain a ‚Äúbrag document‚Äù that lists everything so you can refer to it when you get to performance\nreview season! This is a pretty common tactic ‚Äì when I started doing this I mentioned it to more\nexperienced people and they were like ‚Äúoh yeah, I‚Äôve been doing that for a long time, it really\nhelps‚Äù.</p>\n\n<p>Where I work we call this a ‚Äúbrag document‚Äù but I‚Äôve heard other names for the same concept like\n‚Äúhype document‚Äù or ‚Äúlist of stuff I did‚Äù :).</p>\n\n<p>There‚Äôs a basic template for a brag document at the end of this post.</p>\n\n<h3>share your brag document with your manager</h3>\n\n<p>When I first wrote a brag document I was kind of nervous about sharing it with my manager. It felt\nweird to be like ‚Äúhey, uh, look at all the awesome stuff I did this year, I wrote a long document\nlisting everything‚Äù. But my manager was really thankful for it ‚Äì I think his perspective was ‚Äúthis\nmakes my job way easier, now I can look at the document when writing your perf review instead of\ntrying to remember what happened‚Äù.</p>\n\n<p>Giving them a document that explains your accomplishments will really help your manager advocate for\nyou in discussions about your performance and come to any meetings they need to have prepared.</p>\n\n<p>Brag documents also <strong>really</strong> help with manager transitions ‚Äì if you get a new manager 3 months\nbefore an important performance review that you want to do well on, giving them a brag document\noutlining your most important work &amp; its impact will help them understand what you‚Äôve been doing\neven though they may not have been aware of any of your work before.</p>\n\n<h3>share it with your peer reviewers</h3>\n\n<p>Similarly, if your company does peer feedback as part of the promotion/perf process  ‚Äì share your\nbrag document with your peer reviewers!! Every time someone shares their doc with me I find it SO\nHELPFUL with writing their review for much the same reasons it‚Äôs helpful to share it with your\nmanager ‚Äì it reminds me of all the amazing things they did, and when they list their goals in their\nbrag document it also helps me see what areas they might be most interested in feedback on.</p>\n\n<p>On some teams at work it‚Äôs a team norm to share a brag document with peer reviewers to make it\neasier for them.</p>\n\n<h3>explain the big picture</h3>\n\n<p>In addition to just listing accomplishments, in your brag document you can write the narrative explaining the big picture of your work. Have you been really focused on security? On building your product skills &amp; having really good relationships with your users? On building a strong culture of code review on the team?</p>\n\n<p>In my brag document, I like to do this by making a section for areas that I‚Äôve been focused on (like ‚Äúsecurity‚Äù) and listing all the work I‚Äôve done in that area there. This is especially good if you‚Äôre working on something fuzzy like ‚Äúbuilding a stronger culture of code review‚Äù where all the individual actions you do towards that might be relatively small and there isn‚Äôt a big shiny ship.</p>\n\n<h3>use your brag document to notice patterns</h3>\n\n<p>In the past I‚Äôve found the brag document useful not just to hype my accomplishments, but also to reflect on the work I‚Äôve done. Some questions it‚Äôs helped me with:</p>\n\n<ul>\n<li>What work do I feel most proud of?</li>\n<li>Are there themes in these projects I should be thinking about? What‚Äôs the big picture of what I‚Äôm working on?  (am I working a lot on security? localization?).</li>\n<li>What do I wish I was doing more / less of?</li>\n<li>Which of my projects had the effect I wanted, and which didn‚Äôt? Why might that have been?</li>\n<li>What could have gone better with project X? What might I want to do differently next time?</li>\n</ul>\n\n<h3>you can write it all at once or update it every 2 weeks</h3>\n\n<p>Many people have told me that it works best for them if they take a few minutes to update their brag document every 2 weeks ago. For me it actually works better to do a single marathon session every 6 months or every year where I look through everything I did and reflect on it all at once. Try out different approaches and see what works for you!</p>\n\n<h3>don‚Äôt forget to include the fuzzy work</h3>\n\n<p>A lot of us work on fuzzy projects that can feel hard to quantify, like:</p>\n\n<ul>\n<li>improving code quality on the team / making code reviews a little more in depth</li>\n<li>making on call easier</li>\n<li>building a more fair interview process / performance review system</li>\n<li>refactoring / driving down technical debt</li>\n</ul>\n\n<p>A lot of people will leave this kind of work out because they don‚Äôt know how to explain why it‚Äôs important. But I think this kind of work is especially important to put into your brag document because it‚Äôs the most likely to fall under the radar! One way to approach this is to, for each goal:</p>\n\n<ol>\n<li>explain your goal for the work (why do you think it‚Äôs important to refactor X piece of code?)</li>\n<li>list some things you‚Äôve done towards that goal</li>\n<li>list any effects you‚Äôve seen of the work, even if they‚Äôre a little indirect</li>\n</ol>\n\n<p>If you tell your coworkers this kind of work is important to you and tell them what you‚Äôve been\ndoing, maybe they can also give you ideas about how to do it more effectively or make the effects of\nthat work more obvious!</p>\n\n<h3>encourage each other to celebrate accomplishments</h3>\n\n<p>One nice side effect of having a shared idea that it‚Äôs normal/good to maintain a brag document at\nwork is that I sometimes see people encouraging each other to record &amp; celebrate their\naccomplishments (‚Äúhey, you should put that in your brag doc, that was really good!‚Äù). It can be hard\nto see the value of your work sometimes, especially when you‚Äôre working on something hard, and an\noutside perspective from a friend or colleague can really help you see why what you‚Äôre doing is\nimportant.</p>\n\n<p>Brag documents are good when you use them on your own to advocate for yourself, but I think they‚Äôre\nbetter as a collaborative effort to recognize where people are excelling.</p>\n\n<p>Next, I want to talk about a couple of structures that we‚Äôve used to help people recognize their accomplishments.</p>\n\n<h3>the brag workshop: help people list their accomplishments</h3>\n\n<p>The way this ‚Äúbrag document‚Äù practice started in the first place is that my coworker <a href=\"https://karla.io/\" rel=\"noopener noreferrer\">Karla</a> and I wanted to help other women in engineering advocate for themselves more in the performance review process. The idea is that some people undersell their accomplishments more than they should, so we wanted to encourage those people to ‚Äúbrag‚Äù a little bit and write down what they did that was important.</p>\n\n<p>We did this by running a ‚Äúbrag workshop‚Äù just before performance review season. The format of the workshop is like this:</p>\n\n<p><strong>Part 1: write the document: 1-2 hours</strong>. Everybody sits down with their laptop, starts looking\nthrough their pull requests, tickets they resolved, design docs, etc, and puts together a list of\nimportant things they did in the last 6 months.</p>\n\n<p><strong>Part 2: pair up and make the impact of your work clearer: 1 hour</strong>. The goal of this part is to\npair up, review each other‚Äôs documents, and identify places where people haven‚Äôt bragged ‚Äúenough‚Äù ‚Äì\nmaybe they worked on an extremely critical project to the company but didn‚Äôt highlight how important\nit was, maybe they improved test performance but didn‚Äôt say that they made the tests 3 times faster\nand that it improved everyone‚Äôs developer experience. It‚Äôs easy to accidentally write ‚ÄúI shipped\n$feature‚Äù and miss the follow up (‚Äú‚Ä¶ which caused $thing to happen‚Äù). Another person reading\nthrough your document can help you catch the places where you need to clarify the impact.</p>\n\n<h3>biweekly brag document writing session</h3>\n\n<p>Another approach to helping people remember their accomplishments: my friend Dave gets some friends\ntogether every couple of weeks or so for everyone to update their brag documents. It‚Äôs a nice way\nfor people to talk about work that they‚Äôre happy about &amp; celebrate it a little bit, and updating\nyour brag document as you go can be easier than trying to remember everything you did all at once at\nthe end of the year.</p>\n\n<p>These don‚Äôt have to be people in the same company or even in the same city ‚Äì that group meets over\nvideo chat and has people from many different companies doing this together from Portland, Toronto,\nNew York, and Montreal.</p>\n\n<p>In general, especially if you‚Äôre someone who really cares about your work, I think it‚Äôs really\npositive to share your goals &amp; accomplishments (and the things that haven‚Äôt gone so well too!) with\nyour friends and coworkers. It makes it feel less like you‚Äôre working alone and more like\neveryone is supporting each other in helping them accomplish what they want.</p>\n\n<h3>thanks</h3>\n\n<p>Thanks to Karla Burnett who I worked with on spreading this idea at work, to Dave Vasilevsky for\nrunning brag doc writing sessions, to Will Larson who encouraged me to start one <a href=\"https://lethain.com/career-narratives/\" rel=\"noopener noreferrer\">of these</a> in the\nfirst place, to my manager Jay Shirley for always being encouraging &amp; showing me that this is a\nuseful way to work with a manager, and to Allie, Dan, Laura, Julian, Kamal, Stanley, and Vaibhav for\nreading a draft of this.</p>\n\n<p>I‚Äôd also recommend the blog post <a href=\"http://blog.aashni.me/2019/01/hype-yourself-youre-worth-it/\" rel=\"noopener noreferrer\">Hype Yourself! You‚Äôre Worth It!</a> by Aashni Shah which talks about a similar approach.</p>\n\n<h2><a rel=\"noopener noreferrer\">Appendix: brag document template</a></h2>\n\n<p>Here‚Äôs a template for a brag document! Usually I make one brag document per year. (‚ÄúJulia‚Äôs\n2017 brag document‚Äù). I think it‚Äôs okay to make it quite long / comprehensive ‚Äì 5-10 pages or more\nfor a year of work doesn‚Äôt seem like too much to me, especially if you‚Äôre including some\ngraphs/charts / screenshots to show the effects of what you did.</p>\n\n<p>One thing I want to emphasize, for people who don‚Äôt like to brag, is ‚Äì <strong>you don‚Äôt have to try to\nmake your work sound better than it is</strong>. Just make it sound <strong>exactly as good as it is</strong>! For\nexample ‚Äúwas the primary contributor to X new feature that‚Äôs now used by 60% of our customers and\nhas gotten Y positive feedback‚Äù.</p>\n\n<h3>Goals for this year:</h3>\n\n<ul>\n<li>List your major goals here! Sharing your goals with your manager &amp; coworkers is really nice because it helps them see how they can support you in accomplishing those goals!</li>\n</ul>\n\n<h3>Goals for next year</h3>\n\n<ul>\n<li>If it‚Äôs getting towards the end of the year, maybe start writing down what you think your goals for next year might be.</li>\n</ul>\n\n<h3>Projects</h3>\n\n<p>For each one, go through:</p>\n\n<ul>\n<li>What your contributions were (did you come up with the design? Which components did you build? Was there some useful insight like ‚Äúwait, we can cut scope and do what we want by doing way less work‚Äù that you came up with?)</li>\n<li>The impact of the project ‚Äì who was it for? Are there numbers you can attach to it? (saved X dollars? shipped new feature that has helped sell Y big deals? Improved performance by X%? Used by X internal users every day?). Did it support some important non-numeric company goal (required to pass an audit? helped retain an important user?)</li>\n</ul>\n\n<p>Remember: don‚Äôt forget to explain what the results of you work actually were! It‚Äôs often important to go back a few months later and fill in what actually happened after you launched the project.</p>\n\n<h3>Collaboration &amp; mentorship</h3>\n\n<p>Examples of things in this category:</p>\n\n<ul>\n<li>Helping others in an area you‚Äôre an expert in (like ‚Äúother engineers regularly ask me for one-off help\nsolving weird bugs in their CSS‚Äù or ‚Äúquoting from the C standard at just the right moment‚Äù)</li>\n<li>Mentoring interns / helping new team members get started</li>\n<li>Writing really clear emails/meeting notes</li>\n<li>Foundational code that other people built on top of</li>\n<li>Improving monitoring / dashboards / on call</li>\n<li>Any code review that you spent a particularly long time on / that you think was especially important</li>\n<li>Important questions you answered (‚Äúhelped Risha from OTHER_TEAM with a lot of questions related to Y‚Äù)</li>\n<li>Mentoring someone on a project (‚Äúgave Ben advice from time to time on leading his first big project‚Äù)</li>\n<li>Giving an internal talk or workshop</li>\n</ul>\n\n<h3>Design &amp; documentation</h3>\n\n<p>List design docs &amp; documentation that you worked on</p>\n\n<ul>\n<li>Design docs: I usually just say ‚Äúwrote design for X‚Äù or ‚Äúreviewed design for X‚Äù</li>\n<li>Documentation: maybe briefly explain the goal behind this documentation (for example ‚Äúwe were getting a lot of questions about X, so I documented it and now we can answer the questions more quickly‚Äù)</li>\n</ul>\n\n<h3>Company building</h3>\n\n<p>This is a category we have at work ‚Äì it basically means ‚Äúthings you did to help the company overall, not just your project / team‚Äù. Some things that go in here:</p>\n\n<ul>\n<li>Going above &amp; beyond with interviewing or recruiting (doing campus recruiting, etc)</li>\n<li>Improving important processes, like the interview process or writing better onboarding materials</li>\n</ul>\n\n<h3>What you learned</h3>\n\n<p>My friend Julian suggested this section and I think it‚Äôs a great idea ‚Äì try listing important\nthings you learned or skills you‚Äôve acquired recently! Some examples of skills you might be\nlearning or improving:</p>\n\n<ul>\n<li>how to do performance analysis &amp; make code run faster</li>\n<li>internals of an important piece of software (like the JVM or Postgres or Linux)</li>\n<li>how to use a library (like React)</li>\n<li>how to use an important tool (like the command line or Firefox dev tools)</li>\n<li>about a specific area of programming (like localization or timezones)</li>\n<li>an area like product management / UX design</li>\n<li>how to write a clear design doc</li>\n<li>a new programming language</li>\n</ul>\n\n<p>It‚Äôs really easy to lose track of what skills you‚Äôre learning, and usually when I reflect on this I\nrealize I learned a lot more than I thought and also notice things that I‚Äôm <em>not</em> learning that I\nwish I was.</p>\n\n<h3>Outside of work</h3>\n\n<p>It‚Äôs also often useful to track accomplishments outside of work, like:</p>\n\n<ul>\n<li>blog posts</li>\n<li>talks/panels</li>\n<li>open source work</li>\n<li>Industry recognition</li>\n</ul>\n\n<p>I think this can be a nice way to highlight how you‚Äôre thinking about your career outside of strictly what you‚Äôre doing at work.</p>\n\n<p>This can also include other non-career-related things you‚Äôre proud of, if that feels good to you! Some people like to keep a combined personal + work brag document.</p>\n\n<h3>General prompts</h3>\n\n<p>If you‚Äôre feeling stuck for things to mention, try:</p>\n\n<ul>\n<li>If you were trying to convince a friend to come join your company/team, what would you tell them about your work?</li>\n<li>Did anybody tell you you did something well recently?</li>\n</ul>\n",
        ),
        media_type: None,
        media_link: None,
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/08/27/curl-exercises/",
        link: "https://jvns.ca/blog/2019/08/27/curl-exercises/",
        title: "curl exercises",
        date: 2019-08-27T09:29:38Z,
        summary: None,
        content: Some(
            "\n\n<p>Recently I‚Äôve been interested in how people learn things. I was reading Kathy Sierra‚Äôs great book <a href=\"https://www.amazon.com/Badass-Making-Awesome-Kathy-Sierra/dp/1491919019\" rel=\"noopener noreferrer\">Badass: Making Users Awesome</a>. It talks about the idea of <em>deliberate practice</em>.</p>\n\n<p>The idea is that you find a small micro-skill that can be learned in maybe 3 sessions of 45 minutes,\nand focus on learning that micro-skill. So, as an exercise, I was trying to think of a computer\nskill that I thought could be learned in 3 45-minute sessions.</p>\n\n<p>I thought that making HTTP requests with <code>curl</code> might be a skill like that, so here are some curl\nexercises as an experiment!</p>\n\n<h3>what‚Äôs curl?</h3>\n\n<p>curl is a command line tool for making HTTP requests. I like it because it‚Äôs an easy way to test\nthat servers or APIs are doing what I think, but it‚Äôs a little confusing at first!</p>\n\n<p>Here‚Äôs a drawing explaining curl‚Äôs most important command line arguments (which is page 6 of my <a href=\"https://wizardzines.com/zines/bite-size-networking\" rel=\"noopener noreferrer\">Bite Size Networking</a> zine). You can click to make it bigger.</p>\n\n<p><a href=\"https://jvns.ca/images/curl.jpeg\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/curl.jpeg\" width=\"600px\"></a></p>\n\n<h3>fluency is valuable</h3>\n\n<p>With any command line tool, I think having fluency is really helpful. It‚Äôs really nice to be able to\njust type in the thing you need. For example recently I was testing out the Gumroad API and I was\nable to just type in:</p>\n\n<pre><code>curl https://api.gumroad.com/v2/sales \\\n                         -d \"access_token=&lt;SECRET&gt;\" \\\n                         -X GET  -d \"before=2016-09-03\"\n</code></pre>\n\n<p>and get things working from the command line.</p>\n\n<h3>21 curl exercises</h3>\n\n<p>These exercises are about understanding how to make different kinds of HTTP requests with curl.\nThey‚Äôre a little repetitive on purpose. They exercise basically everything I do with curl.</p>\n\n<p>To keep it simple, we‚Äôre going to make a lot of our requests to the same website:\n<a href=\"https://httpbin.org/\" rel=\"noopener noreferrer\">https://httpbin.org</a>. httpbin is a service that accepts HTTP requests and then tells you what request\nyou made.</p>\n\n<ol>\n<li>Request <a href=\"https://httpbin.org/\" rel=\"noopener noreferrer\">https://httpbin.org</a></li>\n<li>Request <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a>. httpbin.org/anything will look at the request you made,\nparse it, and echo back to you what you requested. curl‚Äôs default is to make a GET request.</li>\n<li>Make a POST request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a></li>\n<li>Make a GET request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a>, but this time add some query parameters (set\n<code>value=panda</code>).</li>\n<li>Request google‚Äôs robots.txt file (www.google.com/robots.txt)</li>\n<li>Make a GET request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a> and set the header <code>User-Agent: elephant</code>.</li>\n<li>Make a DELETE request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a></li>\n<li>Request <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a> and also get the response headers</li>\n<li>Make a POST request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a> with the JSON body <code>{\"value\": \"panda\"}</code></li>\n<li>Make the same POST request as the previous exercise, but set the Content-Type header to\n<code>application/json</code> (because POST requests need to have a content type that matches their body).\nLook at the <code>json</code> field in the response to see the difference from the previous one.</li>\n<li>Make a GET request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a> and set the header <code>Accept-Encoding: gzip</code> (what happens? why?)</li>\n<li>Put a bunch of a JSON in a file and then make a POST request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a> with\nthe JSON in that file as the body</li>\n<li>Make a request to <a href=\"https://httpbin.org/image\" rel=\"noopener noreferrer\">https://httpbin.org/image</a> and set the header ‚ÄòAccept: image/png‚Äô. Save the\noutput to a PNG file and open the file in an image viewer. Try the same thing with with different\n<code>Accept:</code> headers.</li>\n<li>Make a PUT request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a></li>\n<li>Request <a href=\"https://httpbin.org/image/jpeg\" rel=\"noopener noreferrer\">https://httpbin.org/image/jpeg</a>, save it to a file, and open that file in your image\neditor.</li>\n<li>Request <a href=\"https://www.twitter.com/\" rel=\"noopener noreferrer\">https://www.twitter.com</a>. You‚Äôll get an empty response. Get curl to show you the response\nheaders too, and try to figure out why the response was empty.</li>\n<li>Make any request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a> and just set some nonsense headers (like <code>panda: elephant</code>)</li>\n<li>Request <a href=\"https://httpbin.org/status/404\" rel=\"noopener noreferrer\">https://httpbin.org/status/404</a> and <a href=\"https://httpbin.org/status/200\" rel=\"noopener noreferrer\">https://httpbin.org/status/200</a>. Request them again and\nget curl to show the response headers.</li>\n<li>Request <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a> and set a username and password (with <code>-u username:password</code>)</li>\n<li>Download the Twitter homepage (<a href=\"https://twitter.com/\" rel=\"noopener noreferrer\">https://twitter.com</a>) in Spanish by setting the <code>Accept-Language: es-ES</code> header.</li>\n<li>Make a request to the Stripe API with curl. (see <a href=\"https://stripe.com/docs/development\" rel=\"noopener noreferrer\">https://stripe.com/docs/development</a> for how,\nthey give you a test API key). Try making exactly the same request to <a href=\"https://httpbin.org/anything\" rel=\"noopener noreferrer\">https://httpbin.org/anything</a>.</li>\n</ol>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/curl.jpeg",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/08/30/git-exercises--navigate-a-repository/",
        link: "https://jvns.ca/blog/2019/08/30/git-exercises--navigate-a-repository/",
        title: "git exercises: navigate a repository",
        date: 2019-08-30T09:25:15Z,
        summary: None,
        content: Some(
            "\n\n\n\n<p>I think the <a href=\"https://jvns.ca/blog/2019/08/27/curl-exercises/\" rel=\"noopener noreferrer\">curl exercises</a> the other day went\nwell, so today I woke up and wanted to try writing some Git exercises. Git is a big thing to learn,\nprobably too big to learn in a few hours, so my first idea for how to break it down was by starting\nby <strong>navigating</strong> a repository.</p>\n\n<p>I was originally going to use a toy test repository, but then I thought ‚Äì why not a real\nrepository? That‚Äôs way more fun! So we‚Äôre going to navigate the repository for the Ruby programming\nlanguage. You don‚Äôt need to know any C to do this exercise, it‚Äôs just about getting comfortable with\nlooking at how files in a repository change over time.</p>\n\n<h3>clone the repository</h3>\n\n<p>To get started, clone the repository:</p>\n\n<pre><code>git clone https://github.com/ruby/ruby\n</code></pre>\n\n<p>The big different thing about this repository (as compared to most of the repositories you‚Äôll work\nwith in real life) is that it doesn‚Äôt have branches, but it DOES have lots of tags, which are\nsimilar to branches in that they‚Äôre both just pointers to a commit. So we‚Äôll do exercises with tags\ninstead of branches. The way you <em>change</em> tags and branches are very different, but the way you\n<em>look at</em> tags and branches is exactly the same.</p>\n\n<h3>a git SHA always refers to the same code</h3>\n\n<p>The most important thing to keep in mind while doing these exercises is that a git SHA like\n<code>9e3d9a2a009d2a0281802a84e1c5cc1c887edc71</code> always refers to the same code, as explained in this\npage. This page is from a zine I wrote with Katie Sylor-Miller called <a href=\"https://wizardzines.com/zines/oh-shit-git/\" rel=\"noopener noreferrer\">Oh shit, git!</a>. (She also has a great site called\n<a href=\"https://ohshitgit.com/\" rel=\"noopener noreferrer\">https://ohshitgit.com/</a> that inspired the zine).</p>\n\n<p><a href=\"https://wizardzines.com/zines/oh-shit-git/samples/ohshit-commit.png\" rel=\"noopener noreferrer\">\n<img src=\"https://wizardzines.com/zines/oh-shit-git/samples/ohshit-commit.png\" width=\"100%\">\n</a></p>\n\n<p>We‚Äôll be using git SHAs really heavily in the exercises to get you used to working with them and\nto help understand how they correspond to tags and branches.</p>\n\n<h3>git subcommands we‚Äôll be using</h3>\n\n<p>All of these exercises only use 5 git subcommands:</p>\n\n<pre><code>git checkout\ngit log (--oneline, --author, and -S will be useful)\ngit diff (--stat will be useful)\ngit show\ngit status\n</code></pre>\n\n<h3>exercises</h3>\n\n<ol>\n<li>Check out matz‚Äôs commit of Ruby from 1998. The commit ID is <code>3db12e8b236ac8f88db8eb4690d10e4a3b8dbcd4</code>. Find out how many lines of code Ruby was at that time.</li>\n<li>Check out the current master branch</li>\n<li>Look at the history for the file <code>hash.c</code>. What was the last commit ID that changed that file?</li>\n<li>Get a diff of how <code>hash.c</code> has changed in the last 20ish years: compare that file on the master\nbranch to the file at commit <code>3db12e8b236ac8f88db8eb4690d10e4a3b8dbcd4</code>.</li>\n<li>Find a recent commit that changed <code>hash.c</code> and look at the diff for that commit</li>\n<li>This repository has a bunch of <strong>tags</strong> for every Ruby release. Get a list of all the tags.</li>\n<li>Find out how many files changed between tag <code>v1_8_6_187</code> and tag <code>v1_8_6_188</code></li>\n<li>Find a commit (any commit) from 2015 and check it out, look at the files very briefly, then go back to the master branch.</li>\n<li>Find out what commit the tag <code>v1_8_6_187</code> corresponds to.</li>\n<li>List the directory <code>.git/refs/tags</code>. Run <code>cat .git/refs/tags/v1_8_6_187</code> to see the contents\nof one of those files.</li>\n<li>Find out what commit ID <code>HEAD</code> corresponds to right now.</li>\n<li>Find out how many commits have been made to the <code>test/</code> directory</li>\n<li>Get a diff of <code>lib/telnet.rb</code> between the commits <code>65a5162550f58047974793cdc8067a970b2435c0</code> and\n<code>9e3d9a2a009d2a0281802a84e1c5cc1c887edc71</code>. How many lines of that file were changed?</li>\n<li>How many commits were made between Ruby 2.5.1 and 2.5.2 (tags <code>v2_5_1</code> and <code>v2_5_3</code>)</li>\n<li>How many commits were authored by <code>matz</code> (Ruby‚Äôs creator)?</li>\n<li>What‚Äôs the most recent commit that included the word <code>tkutil</code>?</li>\n<li>Check out the commit <code>e51dca2596db9567bd4d698b18b4d300575d3881</code> and create a new branch that\npoints at that commit.</li>\n<li>Run <code>git reflog</code> to see all the navigating of the repository you‚Äôve done so far</li>\n</ol>\n\n<div>\n\nSolutions\n<div>\n<div>\n<p>\n</p><p>\n<strong>Question #1: </strong>Check out matz\'s commit of Ruby from 1998. The commit ID is `3db12e8b236ac8f88db8eb4690d10e4a3b8dbcd4`. Find out how many lines of code Ruby was at that time.\n</p>\n<p><strong>Solution #1: </strong>\n</p><pre>git checkout 3db12e8b236ac8f88db8eb4690d10e4a3b8dbcd4\nfind . -name \'*.c\' | xargs wc -l\n</pre>\n<p></p>\n<p>\n<strong>Question #2:</strong> Check out the current master branch\n</p>\n<p><strong>Solution #2:</strong>\n</p><pre>git checkout master\n</pre>\n<p></p>\n\n<p>\n<strong>Question #3:</strong> Look at the history for the file `hash.c`. What was the last commit ID that changed that file?\n\n</p><p><strong>Solution #3:</strong>\n</p><pre>git log hash.c\n# look at the first line to get the commit ID. \n# I got 3df37259d81d9fc71f8b4f0b8d45dc9d0af81ab4.\n</pre>\n<p></p>\n<p></p>\n\n<p>\n<strong>Question #4:</strong> Get a diff of how `hash.c` has changed in the last 20ish years: compare that file on the master branch to the file at commit `3db12e8b236ac8f88db8eb4690d10e4a3b8dbcd4`.\n</p><p><strong>Solution #4:</strong>\n</p><pre>git diff 3db12e8b236ac8f88db8eb4690d10e4a3b8dbcd4 hash.c\n</pre>\n<p></p>\n\n<p>\n<strong>Question #5:</strong> Find a recent commit that changed `hash.c` and look at the diff for that commit\n</p><p><strong>Solution #5:</strong>\n</p><pre>git log hash.c\n# look at the first line to get the commit ID. \n# I got 3df37259d81d9fc71f8b4f0b8d45dc9d0af81ab4.\ngit show 3df37259d81d9fc71f8b4f0b8d45dc9d0af81ab4\n</pre>\n<p></p>\n<p>\n<strong>Question #6:</strong> This repository has a bunch of **tags** for every Ruby release. Get a list of all the tags.\n</p><p><strong>Solution #6:</strong>\n</p><pre>git tags\n</pre>\n<p></p>\n<p>\n<strong>Question #7:</strong> Find out how many files changed between tag `v1_8_6_187` and tag `v1_8_6_188`\n</p><p><strong>Solution #7:</strong>\n</p><pre>git diff v1_8_6_187 v1_8_6_188 --stat\n# 5 files!\n</pre>\n<p></p>\n<p>\n<strong>Question #8:</strong> Find a commit (any commit) from 2015 and check it out, look at the files very briefly, then go back to the master branch.\n</p><p><strong>Solution #8:</strong>\n</p><pre>git log | grep -C 2 \' 2015 \' | head\ngit checkout bd5d443a56ee4bcb59a0a08776c07dea3ee60121\nls\ngit checkout master\n</pre>\n<p></p>\n<p>\n<strong>Question #9:</strong> Find out what commit the tag `v1_8_6_187` corresponds to.\n</p><p><strong>Solution #9:</strong>\n</p><pre>git show v1_8_6_187\n</pre>\n<p></p>\n<p>\n<strong>Question #10:</strong> List the directory `.git/refs/tags`. Run `cat .git/refs/tags/v1_8_6_187` to see the contents of one of those files.\n</p><p><strong>Solution #10:</strong>\n</p><pre>$ cat .git/refs/tags/v1_8_6_187\n928e6916b25aee5b2b379999a3fa8816d40db714\n</pre>\n<p></p>\n<p>\n<strong>Question #11:</strong> Find out what commit ID `HEAD` corresponds to right now.\n</p><p><strong>Solution #11:</strong>\n</p><pre>git show HEAD\n</pre>\n<p></p>\n<p>\n<strong>Question #12:</strong> Find out how many commits have been made to the `test/` directory\n</p><p><strong>Solution #12:</strong>\n</p><pre>git log --oneline test/ | wc\n</pre>\n<p></p>\n<p>\n<strong>Question #13:</strong> Get a diff of `lib/telnet.rb` between the commits `f2a91397fd7f9ca5bb3d296ec6df2de6f9cfc7cb` and `e44c9b11475d0be2f63286c1332a48da1b4d8626 `. How many lines of that file were changed?\n</p><p><strong>Solution #13:</strong>\n</p><pre>git diff f2a91397fd7f9..e44c9b11475d0 lib/tempfile.rb\n</pre>\n<p></p>\n<p>\n<strong>Question #14:</strong> How many commits were made between Ruby 2.5.1 and 2.5.2 (tags `v2_5_1` and `v2_5_3`) \n</p><p><strong>Solution #14:</strong>\n</p><pre>git log v2_5_1..v2_5_3 --oneline | wc\n</pre>\n<p></p>\n<p>\n<strong>Question #15:</strong> How many commits were authored by `matz` (Ruby\'s creator)?\n</p><p><strong>Solution #15:</strong>\n</p><pre>git log --oneline --author matz | wc -l\n</pre>\n<p></p>\n<p>\n<strong>Question #16:</strong> What\'s the most recent commit that included the word `tkutil`?\n</p><p><strong>Solution #16:</strong>\n</p><pre>git log -S tkutil\n# result is 6c5f5233db596c2c7708d5807d9a925a3a0ee73a\n</pre>\n<p></p>\n<p>\n<strong>Question #17:</strong> Check out the commit `e51dca2596db9567bd4d698b18b4d300575d3881` and create a new branch that points at that commit. \n</p><p><strong>Solution #17:</strong>\n</p><pre>git checkout e51dca2596db9567bd4d698b18b4d300575d3881\ngit branch my-branch\n</pre>\n<p></p>\n<p>\n<strong>Question #18:</strong> Run `git reflog` to see all the navigating of the repository you\'ve done so far\n</p><p><strong>Solution #18:</strong>\n</p><pre>git reflog\n</pre>\n<p></p>\n</div>\n</div>\n</div>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://wizardzines.com/zines/oh-shit-git/samples/ohshit-commit.png",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/09/01/ways-to-write-zines-without-fancy-tools/",
        link: "https://jvns.ca/blog/2019/09/01/ways-to-write-zines-without-fancy-tools/",
        title: "How to write zines with simple tools",
        date: 2019-09-01T09:02:43Z,
        summary: None,
        content: Some(
            "\n\n<p>People often ask me what tools I use to write my zines (<a href=\"https://twitter.com/b0rk/status/1160171769833185280\" rel=\"noopener noreferrer\">the answer is\nhere</a>). Answering this question as written has\nalways felt slightly off to me, though, and I couldn‚Äôt figure out why for a long time.</p>\n\n<p>I finally realized last week that instead of ‚Äúwhat tools do you use to write zines?‚Äù some people may\nhave actually wanted to know ‚Äúhow can I do this myself?‚Äù! And ‚Äúbuy a $500 iPad‚Äù is not a terribly\nuseful answer to that question ‚Äì it‚Äôs not how I got started, iPads are kind of a weird fancy way to\nwrite zines, and most people don‚Äôt have them.</p>\n\n<p>So this blog post is about more traditional (and easier to get started with) ways to write zines.</p>\n\n<p>We‚Äôre going to start out by talking about the mechanics of how to write the zine, and then talk\nabout how to assemble it into a booklet.</p>\n\n<h3>Way 1: Write it on paper</h3>\n\n<p>This is how I made my first zine (spying on your programs with strace) which you can see here: <a href=\"https://jvns.ca/strace-zine-unfolded.pdf\" rel=\"noopener noreferrer\">https://jvns.ca/strace-zine-unfolded.pdf</a>.</p>\n\n<p>Here‚Äôs an example of a page I drew on paper this morning pretty quickly. It looks kind of bad\nbecause I scanned it with my phone, but if you use a real scanner (like I did with the strace PDF\nabove), the scanned version comes out better.</p>\n\n<p><a href=\"https://jvns.ca/images/drawing-status-codes.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/drawing-status-codes.png\" width=\"500px\"></a></p>\n\n<h3>Way 2: Use a Google doc</h3>\n\n<p>The next option is to use a Google doc (or whatever other word processor you prefer). <a href=\"https://docs.google.com/document/d/1byzfXC0h6hNFlWXaV9peJpX-GamJOrJ70x9nu1dZ-m0/edit?usp=sharing\" rel=\"noopener noreferrer\">Here‚Äôs the Google doc I wrote for the below image</a>, and here‚Äôs what it looks like:</p>\n\n<p><a href=\"https://jvns.ca/images/docs-status-codes.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/docs-status-codes.png\" width=\"500px\"></a></p>\n\n<p>They key thing about this Google doc approach is to apply some ‚Äúless is more‚Äù. It‚Äôs intended to be\nprinted as part of a booklet on <strong>half</strong> a sheet of letter paper, which means everything needs to be\ntwice as big for it to look good.</p>\n\n<h3>Way 3: Use an iPad</h3>\n\n<p>This is what I do (use the Notability app on iPad). I‚Äôm not going to talk about this method much\nbecause this post is about using more readily available tools.</p>\n\n<p><a href=\"https://jvns.ca/images/ipad-status-codes.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/ipad-status-codes.png\" width=\"500px\"></a></p>\n\n<h3>Way 4: Use a single sheet of paper</h3>\n\n<p>This is a subset of ‚ÄúWrite it on paper‚Äù ‚Äì the <a href=\"https://en.m.wikibooks.org/wiki/Zine_Making/Putting_pages_together\" rel=\"noopener noreferrer\">Wikibooks page on zine making</a> has a great guide that shows how to write out a tiny zine on 1 piece of paper and then fold it up to make a little booklet. Here are the pictures of the steps from the Wikibooks page:</p>\n\n<p><a href=\"https://jvns.ca/images/Zinemaking-folding-8cut-plan.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/Zinemaking-folding-8cut-plan.png\"></a>\n<a href=\"https://jvns.ca/images/Zinemaking-folding-8cut-1.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/Zinemaking-folding-8cut-1.png\"></a>\n<a href=\"https://jvns.ca/images/Zinemaking-folding-8cut-2.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/Zinemaking-folding-8cut-2.png\"></a>\n<a href=\"https://jvns.ca/images/Zinemaking-folding-8cut-3.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/Zinemaking-folding-8cut-3.png\"></a>\n<a href=\"https://jvns.ca/images/Zinemaking-folding-8cut-4.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/Zinemaking-folding-8cut-4.png\"></a>\n<a href=\"https://jvns.ca/images/Zinemaking-folding-8cut-5.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/Zinemaking-folding-8cut-5.png\"></a>\n<a href=\"https://jvns.ca/images/Zinemaking-folding-8cut-6.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/Zinemaking-folding-8cut-6.png\"></a>\n<a href=\"https://jvns.ca/images/Zinemaking-folding-8cut-7.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/Zinemaking-folding-8cut-7.png\"></a></p>\n\n<p>Sumana Harihareswara‚Äôs <a href=\"https://www.harihareswara.net/pix/playing-with-python-zine/playing-with-python-zine.pdf\" rel=\"noopener noreferrer\">Playing with\npython</a>\nzine is a nice example of a zine that‚Äôs intended to be folded up in that way.</p>\n\n<h3>Way 5: Adobe Illustrator</h3>\n\n<p>I‚Äôve never used Adobe Illustrator so I‚Äôm not going to pretend that I know anything about it or put\ntogether an example using it, but I hear it‚Äôs a way people do book layout.</p>\n\n<h3>booklets: the photocopier method</h3>\n\n<p>So you‚Äôve written a bunch of pages and want to assemble them into a booklet. One way to do this (and\nwhat I did for my first zine about strace!) is the photocopier method. There‚Äôs a great guide by Julia Gfr√∂rer in\n<a href=\"https://twitter.com/thorazos/status/1158556879485906944\" rel=\"noopener noreferrer\">this tweet</a>, which I‚Äôm going to reproduce\nhere:</p>\n\n<p><img src=\"https://pbs.twimg.com/media/EBQFUC0X4AAPTU1?format=jpg&amp;name=small\"><br>\n<img src=\"https://pbs.twimg.com/media/EBQFUC0XsAEBhHf?format=jpg&amp;name=small\"><br>\n<img src=\"https://pbs.twimg.com/media/EBQFUC1XUAAKDIB?format=jpg&amp;name=small\"><br>\n<img src=\"https://pbs.twimg.com/media/EBQFUDRX4AMkIAr?format=jpg&amp;name=small\"></p>\n\n<p>That explanation is excellent and I don‚Äôt have anything to add. I did it that way and it worked\ngreat.</p>\n\n<p>If you want to buy a print copy of that how-to-make-zines zine from Thruban Press, you can <a href=\"https://www.etsy.com/thorazos/listing/693692176/thuban-press-guide-to-analog-self?utm_source=Copy&amp;utm_medium=ListingManager&amp;utm_campaign=Share&amp;utm_term=so.lmsm&amp;share_time=1565113962419\" rel=\"noopener noreferrer\">get it\nhere on Etsy</a>.</p>\n\n<h3>booklets: the computer method</h3>\n\n<p>If you‚Äôve made your zine in Google Docs or in another computery way, you probably want a more\ncomputery way of assembling the pages into a booklet.</p>\n\n<p><strong>what I use: pdflatex</strong></p>\n\n<p>I do this using the <code>pdfpages</code> LaTeX extension. This sounds complicated but it‚Äôs not really, you don‚Äôt\nneed to learn latex or anything. You just need to have pdflatex on your system, which is a <code>sudo apt\ninstall texlive-base</code> away on Ubuntu. The steps are:</p>\n\n<ol>\n<li>Get a PDF with the pages from your zine (pages need to be a multiple of 4)</li>\n<li>Get the latex file from <a href=\"https://gist.github.com/jvns/b3de1d658e2b44aebb485c35fb1a7a0f\" rel=\"noopener noreferrer\">this gist</a></li>\n<li>Replace <code>/home/bork/http-zine.pdf</code> with the path to your PDF  and <code>1-28</code> with <code>1-however many\npages are in your zine</code>.</li>\n<li>run <code>pdflatex formatted-zine.tex</code></li>\n<li>Tweak the parameters until it looks the way you want. The <a href=\"http://texdoc.net/texmf-dist/doc/latex/pdfpages/pdfpages.pdf\" rel=\"noopener noreferrer\">documentation for the pdfpages package is here</a></li>\n</ol>\n\n<p>I like using this relatively complicated method because there are always small tweaks I want to make\nlike ‚Äúoh, the right margin is too big, crop it a little bit‚Äù and the pdfpages package has tons of\noptions that let me make those tweaks.</p>\n\n<p><strong>other methods</strong></p>\n\n<ol>\n<li>On Linux you can use the <code>pdfjam</code> bash script, which is just a wrapper around the pdfpages latex\npackage. This is what I used to do but today I find it simpler to use the pdfpages latex package\ndirectly.</li>\n<li>There‚Äôs a program called <a href=\"https://www.bookletcreator.com/\" rel=\"noopener noreferrer\">Booklet Creator</a> for Mac and Windows\nthat <a href=\"https://twitter.com/mrfb/status/1159478532545888258\" rel=\"noopener noreferrer\">@mrfb uses</a>. It looks pretty simple to\nuse.</li>\n<li>If you convert your PDF to a ps file (with <code>pdf2ps</code> for instance), <code>psnup</code> can do this. I tried\n<code>cat file.ps | psbook | psnup -2 &gt; booklet.ps</code> and it worked, though the resulting\nPDFs are a little slow to load in my PDF viewer for some reason.</li>\n<li>there are probably a ton more ways to do this, if you know more let me know</li>\n</ol>\n\n<h3>making zines is easy and low tech</h3>\n\n<p>That‚Äôs all! I mostly wanted to explain that zines are an easy low tech thing to do and if you think\nmaking them sounds fun, you definitely 100% do not need to use any fancy expensive tools to do it,\nyou can literally use some sheets of paper, a Sharpie, a pen, and spend $3 at your local print shop\nto use the photocopier.</p>\n\n<h3>resources</h3>\n\n<p>summary of the resources I linked to:</p>\n\n<ul>\n<li>Guide to putting together zines with a photocopier by Julia Gfr√∂rer: <a href=\"https://twitter.com/thorazos/status/1158556879485906944\" rel=\"noopener noreferrer\">this tweet</a>, <a href=\"https://www.etsy.com/thorazos/listing/693692176/thuban-press-guide-to-analog-self?utm_source=Copy&amp;utm_medium=ListingManager&amp;utm_campaign=Share&amp;utm_term=so.lmsm&amp;share_time=1565113962419\" rel=\"noopener noreferrer\">get it on Etsy</a></li>\n<li><a href=\"https://en.m.wikibooks.org/wiki/Zine_Making/Putting_pages_together\" rel=\"noopener noreferrer\">Wikibooks page on zine making</a></li>\n<li>Notes on making zines using Google Docs: <a href=\"https://twitter.com/mrfb/status/1159478532545888258\" rel=\"noopener noreferrer\">this twitter thread</a></li>\n<li><a href=\"http://www.stolensharpierevolution.org/\" rel=\"noopener noreferrer\">Stolen Sharpie Revolution</a> (the first book I read about\nmaking zines). You can also get it on Amazon if you want but it‚Äôs probably better to buy directly\nfrom their site.</li>\n<li><a href=\"https://www.bookletcreator.com/\" rel=\"noopener noreferrer\">Booklet Creator</a></li>\n</ul>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/drawing-status-codes.png",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/09/06/how-to-put-an-html-page-on-the-internet/",
        link: "https://jvns.ca/blog/2019/09/06/how-to-put-an-html-page-on-the-internet/",
        title: "How to put an HTML page on the internet",
        date: 2019-09-06T15:56:37Z,
        summary: None,
        content: Some(
            "\n\n<p>One thing I love about the internet is that it‚Äôs SO EASY to put static HTML websites on the\ninternet.  Someone asked me today how to do it, so I thought I‚Äôd write down how really quickly!</p>\n\n<h3>just an HTML page</h3>\n\n<p>All of my sites are just static HTML and CSS. My web design skills are relatively minimal\n(<a href=\"https://wizardzines.com/\" rel=\"noopener noreferrer\">https://wizardzines.com</a> is the most complicated site I‚Äôve developed on my own), so keeping all my\ninternet sites relatively simple means that I have some hope of being able to make changes / fix\nthings without spending a billion hours on it.</p>\n\n<p>So we‚Äôre going to take as minimal of an approach as possible in this blog post ‚Äì just one HTML page.</p>\n\n<h3>the HTML page</h3>\n\n<p>The website we‚Äôre going to put on the internet is just one file, called <code>index.html</code>. You can find it\nat <a href=\"https://github.com/jvns/website-example\" rel=\"noopener noreferrer\">https://github.com/jvns/website-example</a>, which is a\nGithub repository with exactly one file in it.</p>\n\n<p>The HTML file has some CSS in it to make it look a little less boring, which is partly copied from\n<a href=\"https://example.com/\" rel=\"noopener noreferrer\">https://example.com</a>.</p>\n\n<h3>how to put the HTML page on the internet</h3>\n\n<p>Here are the steps:</p>\n\n<ol>\n<li>sign up for a <a href=\"https://neocities.org/\" rel=\"noopener noreferrer\">Neocities</a> account</li>\n<li>copy the index.html into the index.html in your neocities site</li>\n<li>done</li>\n</ol>\n\n<p>The index.html page above is on the internet at\n<a href=\"https://julia-example-website.neocities.org/\" rel=\"noopener noreferrer\">julia-example-website.neocities.com</a>, if you view source you‚Äôll see that it‚Äôs\nthe same HTML as in the github repo.</p>\n\n<p>I think this is probably the simplest way to put an HTML page on the internet (and it‚Äôs a throwback\nto Geocities, which is how I made my first website in 2003) :). I also like that Neocities (like\n<a href=\"https://glitch.com/\" rel=\"noopener noreferrer\">glitch</a>, which I also love) is about experimentation and learning and having\nfun..</p>\n\n<h3>other options</h3>\n\n<p>This is definitely not the only easy way ‚Äì Github pages and Gitlab pages and Netlify will all automatically\npublish a site when you push to a Git repository, and they‚Äôre all very easy to use (just connect\nthem to your github repository and you‚Äôre done). I personally use the Git repository approach\nbecause not having things in Git makes me nervous ‚Äì I like to know what changes to my website I‚Äôm\nactually pushing. But I think if you just want to put an HTML site on the internet for the first\ntime and play around with HTML/CSS, Neocities is a really nice way to do it.</p>\n\n<p>If you want to actually use your website for a Real Thing and not just to play around you probably\nwant to buy a domain and link it to your website so that you can change hosting providers in the\nfuture, but that is a bit less simple.</p>\n\n<h3>this is a good possible jumping off point for learning HTML</h3>\n\n<p>If you are a person who is comfortable editing files in a Git repository but wants to practice\nHTML/CSS, I think this is a fun way to put a website on the internet and play around! I really like\nthe simplicity of it ‚Äì there‚Äôs literally just one file, so there‚Äôs no fancy extra magic to get in the\nway of understanding what‚Äôs going on.</p>\n\n<p>There are also a bunch of ways to complicate/extend this, like this blog is actually generated with\n<a href=\"https://gohugo.io/\" rel=\"noopener noreferrer\">Hugo</a> which generates a bunch of HTML files which then go on the internet, but\nit‚Äôs always nice to start with the basics.</p>\n",
        ),
        media_type: None,
        media_link: None,
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/09/12/new-zine-on-http/",
        link: "https://jvns.ca/blog/2019/09/12/new-zine-on-http/",
        title: "New zine: HTTP: Learn your browser\'s language!",
        date: 2019-09-12T12:13:27Z,
        summary: None,
        content: Some(
            "\n\n<p>Hello! I‚Äôve released a new zine! It‚Äôs called ‚ÄúHTTP: Learn your browsers language!‚Äù</p>\n\n<p>You can get it for $12 at <a href=\"https://gum.co/http-zine\" rel=\"noopener noreferrer\">https://gum.co/http-zine</a>. If you buy it, you‚Äôll get a PDF that you can\neither read on your computer or print out.</p>\n\n<p>Here‚Äôs the cover and table of contents:</p>\n\n<div>\n<a href=\"https://gum.co/http-zine\" rel=\"noopener noreferrer\"><img width=\"250px\" src=\"https://jvns.ca/images/http-zine-cover.png\"></a>\n<a href=\"https://jvns.ca/images/http-zine-toc.png\" rel=\"noopener noreferrer\"><img width=\"250px\" src=\"https://jvns.ca/images/http-zine-toc.png\"></a>\n</div>\n\n<h3>why http?</h3>\n\n<p>I got the idea for this zine from talking to <a href=\"https://twitter.com/polotek\" rel=\"noopener noreferrer\">Marco Rogers</a> ‚Äì he\nmentioned that he thought that new web developers / mobile developers would really benefit from\nunderstanding the fundamentals of HTTP better, I thought ‚ÄúOOH I LOVE TALKING ABOUT HTTP‚Äù, wrote a\nfew pages about HTTP, saw they were helping people, and decided to write a whole zine about HTTP.</p>\n\n<p>HTTP is important to understand because it runs the entire web ‚Äì if you understand how HTTP\nrequests and responses work, then it makes it WAY EASIER to debug why your web application isn‚Äôt\nworking properly. Caching, cookies, and a lot of web security are implemented using HTTP headers, so\nif you don‚Äôt understand HTTP headers those things seem kind of like impenetrable magic. But actually\nthe HTTP protocol is fundamentally pretty simple ‚Äì there are a lot of complicated details but the\nbasics are pretty easy to understand.</p>\n\n<p>So the goal of this zine is to teach you the basics so you can easily look up and understand the\ndetails when you need them.</p>\n\n<h3>what it looks like printed out</h3>\n\n<p>All of my zines are best printed out (though you get a PDF you can read on your computer too!), so\nhere are a couple of pictures of what it looks like when printed. I always ask my illustrator to\nmake both a black and white version and a colour version of the cover so that it looks great when\nprinted on a black and white printer.</p>\n\n<p><a href=\"https://gum.co/http-zine\" rel=\"noopener noreferrer\"><img width=\"75%\" src=\"https://jvns.ca/images/http-zine-cover.jpeg\"></a>\n<a href=\"https://jvns.ca/images/same-origin-policy.jpeg\" rel=\"noopener noreferrer\"><img width=\"75%\" src=\"https://jvns.ca/images/same-origin-policy.jpeg\"></a></p>\n\n<p>(if you click on that ‚Äúsame origin policy‚Äù image, you can make it bigger)</p>\n\n<p>The zine comes with 4 print PDFs in addition to a PDF you can just read on your\ncomputer/phone:</p>\n\n<ul>\n<li>letter / colour</li>\n<li>letter / b&amp;w</li>\n<li>a4 / colour</li>\n<li>a4 / b&amp;w</li>\n</ul>\n\n<h3>zines for your team</h3>\n\n<p>You can also buy this zine for your team members at work to help them learn HTTP!</p>\n\n<p>I‚Äôve been trying to get the pricing right for this for a while ‚Äì I used to do it based on size of\ncompany, but that didn‚Äôt seem quite right because sometimes people would want to buy the zine for a\nsmall team at a big company. So I‚Äôve switched to pricing based on the number of copies you want to\ndistribute at your company.</p>\n\n<p>Here‚Äôs the link: <a href=\"https://wizardzines.com/zines-team/\" rel=\"noopener noreferrer\">zines for your team!</a>.</p>\n\n<h3>the tweets</h3>\n\n<p>When I started writing zines, I would just sit down, write down the things I thought were important,\nand be done with it.</p>\n\n<p>In the last year and a half or so I‚Äôve taken a different approach ‚Äì instead of writing everything\nand then releasing it, instead I write a page at a time, post the page to Twitter, and then improve it and\ndecide what page to write next based on the questions/comments I get on Twitter. If someone replies\nto the tweet and asks a question that shows that what I wrote is unclear, I can improve it! (I love\ngetting replies on twitter asking clarifiying questions!).</p>\n\n<p>Here are all the initial drafts of the pages I wrote and posted on twitter, in chronological order.\nSome of the pages didn‚Äôt make it into the zine at all, and I needed to do a lot of editing at the\nend to figure out the right order and make them all work coherently together in a zine instead of\nbeing a bunch of independent tweets.</p>\n\n<ul>\n<li>Jul 1: <a href=\"https://twitter.com/b0rk/status/1145824140462608387\" rel=\"noopener noreferrer\">http status codes </a><br></li>\n<li>Jul 2: <a href=\"https://twitter.com/b0rk/status/1145896193077256197\" rel=\"noopener noreferrer\">anatomy of a HTTP response </a><br></li>\n<li>Jul 2: <a href=\"https://twitter.com/b0rk/status/1146054159214567424\" rel=\"noopener noreferrer\">POST requests </a><br></li>\n<li>Jul 2: <a href=\"https://twitter.com/b0rk/status/1146065212560179202\" rel=\"noopener noreferrer\">an example POST request </a><br></li>\n<li>Jul 28: <a href=\"https://twitter.com/b0rk/status/1155493682885341184\" rel=\"noopener noreferrer\">the same origin policy </a><br></li>\n<li>Jul 28: <a href=\"https://twitter.com/b0rk/status/1155318552129396736\" rel=\"noopener noreferrer\">what‚Äôs HTTP? </a><br></li>\n<li>Jul 30: <a href=\"https://twitter.com/b0rk/status/1156048630220017665\" rel=\"noopener noreferrer\">the most important HTTP request headers </a><br></li>\n<li>Jun 30: <a href=\"https://twitter.com/b0rk/status/1145362860136177664\" rel=\"noopener noreferrer\">anatomy of a HTTP request </a><br></li>\n<li>Aug 4: <a href=\"https://twitter.com/b0rk/status/1158012032651862017\" rel=\"noopener noreferrer\">content delivery networks </a><br></li>\n<li>Aug 6: <a href=\"https://twitter.com/b0rk/status/1158726129508868097\" rel=\"noopener noreferrer\">caching headers </a><br></li>\n<li>Aug 6: <a href=\"https://twitter.com/b0rk/status/1158848054142873603\" rel=\"noopener noreferrer\">how cookies work </a><br></li>\n<li>Aug 7: <a href=\"https://twitter.com/b0rk/status/1159163613938167808\" rel=\"noopener noreferrer\">redirects </a><br></li>\n<li>Aug 8: <a href=\"https://twitter.com/b0rk/status/1159492669384658944\" rel=\"noopener noreferrer\">45 seconds on the Accept-Language HTTP header </a><br></li>\n<li>Aug 9: <a href=\"https://twitter.com/b0rk/status/1159812119099060224\" rel=\"noopener noreferrer\">HTTPS: HTTP + security </a><br></li>\n<li>Aug 9: <a href=\"https://twitter.com/b0rk/status/1159829608595804160\" rel=\"noopener noreferrer\">today in 45 second video experiments: the Range header </a><br></li>\n<li>Aug 9: <a href=\"https://twitter.com/b0rk/status/1159839824594915335\" rel=\"noopener noreferrer\">some HTTP exercises to try </a><br></li>\n<li>Aug 10: <a href=\"https://twitter.com/b0rk/status/1160185182323970050\" rel=\"noopener noreferrer\">some security headers </a><br></li>\n<li>Aug 12: <a href=\"https://twitter.com/b0rk/status/1160933788949655552\" rel=\"noopener noreferrer\">using HTTP APIs </a><br></li>\n<li>Aug 13: <a href=\"https://twitter.com/b0rk/status/1161283690925834241\" rel=\"noopener noreferrer\">what‚Äôs with those headers that start with x-? </a><br></li>\n<li>Aug 13: <a href=\"https://twitter.com/b0rk/status/1161262574031265793\" rel=\"noopener noreferrer\">important HTTP response headers </a><br></li>\n<li>Aug 14: <a href=\"https://twitter.com/b0rk/status/1161679906415218690\" rel=\"noopener noreferrer\">HTTP request methods (part 1) </a><br></li>\n<li>Aug 14: <a href=\"https://twitter.com/b0rk/status/1161680137865367553\" rel=\"noopener noreferrer\">HTTP request methods (part 2) </a><br></li>\n<li>Aug 15: <a href=\"https://twitter.com/b0rk/status/1161997141876903936\" rel=\"noopener noreferrer\">how URLs work </a><br></li>\n<li>Aug 16: <a href=\"https://twitter.com/b0rk/status/1162392625057583104\" rel=\"noopener noreferrer\">CORS </a><br></li>\n<li>Aug 19: <a href=\"https://twitter.com/b0rk/status/1163460967067541504\" rel=\"noopener noreferrer\">why the same origin policy matters </a><br></li>\n<li>Aug 21: <a href=\"https://twitter.com/b0rk/status/1164181027469832196\" rel=\"noopener noreferrer\">HTTP headers </a><br></li>\n<li>Aug 24: <a href=\"https://twitter.com/b0rk/status/1165277002791829510\" rel=\"noopener noreferrer\">how to learn more about HTTP </a><br></li>\n<li>Aug 25: <a href=\"https://twitter.com/b0rk/status/1165623594917007362\" rel=\"noopener noreferrer\">HTTP/2 </a><br></li>\n<li>Aug 27: <a href=\"https://twitter.com/b0rk/status/1166466933912494081\" rel=\"noopener noreferrer\">certificates </a></li>\n</ul>\n\n<p>Writing zines one tweet at a time has been really fun. I think it improves the quality a lot,\nbecause I get a ton of feedback along the way that I can use to make the zine better. There are also\nsome experimental 45 second tiny videos in that list, which are definitely not part of the zine, but\nwhich were fun to make and which I might expand on in the future.</p>\n\n<h3>examplecat.com</h3>\n\n<p>One tiny easter egg in the zine: I have a lot of examples of HTTP requests, and I wasn‚Äôt sure for a\nlong time what domain I should use for the examples. I used example.com a bunch, and google.com and\ntwitter.com sometimes, but none of those felt quite right.</p>\n\n<p>A couple of days before publishing the zine I finally had an epiphany ‚Äì my example on the cover was\nrequesting a picture of a cat, so I registered <a href=\"https://examplecat.com/\" rel=\"noopener noreferrer\">https://examplecat.com</a> which just has a single\npicture of a cat. It also has an ASCII cat if you‚Äôre browsing in your terminal.</p>\n\n<pre><code>$ curl https://examplecat.com/cat.txt  -i\nHTTP/2 200 \naccept-ranges: bytes\ncache-control: public, max-age=0, must-revalidate\ncontent-length: 33\ncontent-type: text/plain; charset=UTF-8\ndate: Thu, 12 Sep 2019 16:48:16 GMT\netag: \"ac5affa59f554a1440043537ae973790-ssl\"\nstrict-transport-security: max-age=31536000\nage: 5\nserver: Netlify\nx-nf-request-id: c5060abc-0399-4b44-94bf-c481e22c2b50-1772748\n\n\\    /\\\n )  ( \')\n(  /  )\n \\(__)|\n</code></pre>\n\n<h3>more zines at wizardzines.com</h3>\n\n<p>If you‚Äôre interested in the idea of programming zines and haven‚Äôt seen my zines before, I have a\nbunch more at <a href=\"https://wizardzines.com/\" rel=\"noopener noreferrer\">https://wizardzines.com</a>. There are 6 free zines there:</p>\n\n<ul>\n<li><a href=\"https://wizardzines.com/zines/wizard/\" rel=\"noopener noreferrer\">so you want to be a wizard</a></li>\n<li><a href=\"https://wizardzines.com/zines/tcpdump/\" rel=\"noopener noreferrer\">let‚Äôs learn tcpdump!</a></li>\n<li><a href=\"https://wizardzines.com/zines/strace/\" rel=\"noopener noreferrer\">spying on your programs with strace</a></li>\n<li><a href=\"https://wizardzines.com/zines/networking/\" rel=\"noopener noreferrer\">networking! ACK!</a></li>\n<li><a href=\"https://wizardzines.com/zines/debugging/\" rel=\"noopener noreferrer\">linux debugging tools you‚Äôll love</a></li>\n<li><a href=\"https://wizardzines.com/zines/perf/\" rel=\"noopener noreferrer\">profiling and tracing with perf</a></li>\n</ul>\n\n<h3>next zine: not sure yet!</h3>\n\n<p>Some things I‚Äôm considering for the next zine:</p>\n\n<ul>\n<li>debugging skills (I started writing a bunch of pages about debugging but switched gears to the\nHTTP zine because I got really excited about that. but debugging is my favourite thing so I‚Äôd like\nto get this done at some point)</li>\n<li>gdb (a short zine in the spirit of <a href=\"https://wizardzines.com/zines/tcpdump/\" rel=\"noopener noreferrer\">let‚Äôs learn tcpdump</a>)</li>\n<li>relational databases (what‚Äôs up with transactions?)</li>\n</ul>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/http-zine-cover.png",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/09/13/a-year-explaining-computer-things/",
        link: "https://jvns.ca/blog/2019/09/13/a-year-explaining-computer-things/",
        title: "Taking a year to explain computer things",
        date: 2019-09-13T11:05:15Z,
        summary: None,
        content: Some(
            "\n\n<p>I‚Äôve been working on explaining computer things I‚Äôm learning on this blog for 6 years.\nI wrote one of my first posts, <a href=\"https://jvns.ca/blog/2013/09/30/hacker-school-day-2-what-does-a-shell-even-do/\" rel=\"noopener noreferrer\">what does a shell even do?</a> on\nSept 30, 2013. Since then, I‚Äôve written 11 zines, 370,000 words on this blog, and\ngiven 20 or so talks. So it seems like I like explaining things a lot.</p>\n\n<h3>tl;dr: I‚Äôm going to work on explaining computer things for a year</h3>\n\n<p>Here‚Äôs the exciting news: I left my job a month ago and my plan is to spend the next year working on\nexplaining computer things!</p>\n\n<p>As for why I‚Äôm doing this ‚Äì I was talking through some reasons with my friend Mat last night and he\nsaid ‚Äúwell, sometimes there are things you just feel compelled to do‚Äù. I think that‚Äôs all there is\nto it :)</p>\n\n<h3>what does ‚Äúexplain computer things‚Äù mean?</h3>\n\n<p>I‚Äôm planning to:</p>\n\n<ol>\n<li>write some more zines (maybe I can write 10 zines in a year? we‚Äôll see! I want to tackle both\ngeneral-interest and slightly more niche topics, we‚Äôll see what happens).</li>\n<li>work on some more interactive ways to learn things. I learn things best by trying things out and\nbreaking them, so I want to see if I can facilitate that a little bit for other people. I started\na project around this in May which has been on the backburner for a bit but which I‚Äôm excited\nabout. Hopefully I‚Äôll release it soon and then you can try it out and tell me what you think!</li>\n</ol>\n\n<p>I say ‚Äúa year‚Äù because I think I have at least a year‚Äôs worth of ideas and I can‚Äôt predict how I‚Äôll\nfeel after doing this for a year.</p>\n\n<h3>how: run a business</h3>\n\n<p>I started a corporation almost exactly a year ago, and I‚Äôm planning to keep running my\nexplaining-things efforts as a business. This business has been making more than I made in my first\nprogramming job (that is, definitely enough money to live on!), which has been really surprising\nand great (thank you!).</p>\n\n<p>some parameters of the business:</p>\n\n<ul>\n<li>I‚Äôm not planning to hire employees or anything, it‚Äôll just be me and some (awesome) freelancers.\nThe biggest change I have in mind is that I‚Äôm hoping to find a freelance editor to help me with editing.</li>\n<li>I also don‚Äôt have any specific plans for world domination or to work 80-hour weeks.  I‚Äôm just\ngoing to make zines &amp; things that explain computer concepts and sell them on the internet, like\nI‚Äôve been doing.</li>\n<li>No commissions or consulting work, just building ideas I have</li>\n</ul>\n\n<p>It‚Äôs been pretty interesting to learn more about running a small business and so far I like it more\nthan I thought I would. (except for taxes, which I like exactly as much as I thought I would)</p>\n\n<h3>that‚Äôs all!</h3>\n\n<p>I‚Äôm excited to keep making explanations of computer things and to have more time to do it. This blog\nmight change a bit away from ‚Äúhere‚Äôs what I‚Äôm learning at work these days‚Äù and towards ‚Äúhere are\nattempts at explaining things that I mostly already know‚Äù. It‚Äôll be different!  We‚Äôll see how it\ngoes!</p>\n",
        ),
        media_type: None,
        media_link: None,
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/09/30/notes-on-building-sql-exercises/",
        link: "https://jvns.ca/blog/2019/09/30/notes-on-building-sql-exercises/",
        title: "Notes on building SQL exercises",
        date: 2019-09-30T12:35:51Z,
        summary: None,
        content: Some(
            "\n\n<p>In the last couple of weeks I‚Äôve been working on some interactive SQL exercises\nto help people get better at writing SQL queries. This is a pretty new thing\nfor me so I thought I‚Äôd write a few notes about my process so far!</p>\n\n<h3>why SQL is exciting: distributed SQL engines</h3>\n\n<p>To me the reason why SQL is exciting is that a lot of companies are storing\ntheir data in distributed SQL databases (Google BigQuery, Amazon Redshift,\nSpark SQL, Presto, etc) that let you run a complicated query across a billion\nrows pretty quickly! They‚Äôre fast partly because they‚Äôre designed to run your\nquery across possibly tens or hundreds of computers.</p>\n\n<p>At my last job I wrote thousands of SQL queries to do data analysis while I was\nworking on the machine learning team, mostly ad hoc queries to answer questions\nI had about our data. I learned a lot of fun tricks to make them faster /\neasier to write and I‚Äôve never really talked about it!</p>\n\n<p>So I think SQL is a really nice way to go from ‚ÄúI have this sort of complicated\nquestion about billions of rows of data‚Äù to ‚Äúok, that‚Äôs the answer, great, I\ncan move on‚Äù.</p>\n\n<h3>why write exercises: knowledge != skills</h3>\n\n<p>This is the first time I‚Äôm really trying in earnest to write exercises to teach\nsomething, instead of just explanations of the thing. The reason I‚Äôm doing this\nis that I read <a href=\"https://www.amazon.com/Design-People-Learn-Voices-Matter/dp/0321768434\" rel=\"noopener noreferrer\">Design for how people learn</a>\nby Julie Dirksen and she makes the point that <strong>knowledge</strong> is different from\n<strong>skills</strong>.</p>\n\n<p>She defines a ‚Äúskill‚Äù as ‚Äúsomething you have to practice‚Äù. And SQL is\ndefinitely something that you have to practice if you want to learn it! So I\nthought ‚Äì SQL is a relatively simple skill (as\nprogramming/programming-adjacent skills go!), maybe I can make something\ninteractive and relatively simple to help people improve their SQL skills!</p>\n\n<p>It‚Äôs also, well, a challenge, and I like trying things I haven‚Äôt tried before.</p>\n\n<h3>how I‚Äôm doing it: start with a challenge</h3>\n\n<p>I started out doing these SQL exercises in kind of the obvious way: start out\nwith easy exercises, and then make them harder and harder over time to\nintroduce new concepts. But when I watched people trying it out, I noticed a\nproblem ‚Äì a lot of people already <strong>know</strong> some SQL, and sometimes they would go\nthrough all the exercises without learning anything at all! That‚Äôs no fun!</p>\n\n<p>So I came up with a different structure for each section of the SQL exercises:</p>\n\n<ol>\n<li>Start with a ‚Äúchallenge‚Äù that tests the skill the section is trying to teach.</li>\n<li>If the challenge is too hard, move on to a bunch of easier exercises that teach you the skills you need to solve the challenge.</li>\n</ol>\n\n<p>Since showing is easier than explaining: <a href=\"https://joins-238123.netlify.com/aggregations/\" rel=\"noopener noreferrer\">here‚Äôs a draft of a page teaching\nGROUP BY</a>. Here‚Äôs a screenshot of what the initial ‚Äúchallenge‚Äù for basic group by looks like:</p>\n\n<p><a href=\"https://jvns.ca/images/aggregation-challenge.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/aggregation-challenge.png\" width=\"500px\"></a></p>\n\n<p>I think that challenge in particular isn‚Äôt very good yet (I have a lot of work to do!) but that‚Äôs the idea.</p>\n\n<h3>how I‚Äôm getting feedback: anonymously track responses</h3>\n\n<p>Early on I also realized that I needed to get feedback about which challenges people were finding hard / easy. Every time someone runs a query, I track</p>\n\n<ul>\n<li>a randomly generated UUID for the person doing the challenge (like <code>f139a44c-ef09-43d2-8d7d-cabba9c28aa1</code>)</li>\n<li>the query they ran</li>\n<li>the puzzle they were solving</li>\n</ul>\n\n<p>I‚Äôve already learned a lot from this, for example:</p>\n\n<ul>\n<li>at first I required that the column names match, but it resulted in a lot of people getting the ‚Äúwrong‚Äù answer for no good reason, so I removed that requirement</li>\n<li>when I‚Äôm not clear about how the results should be ordered, often people end up with the right answer except in the ‚Äúwrong‚Äù order. Still need to do something about that.</li>\n<li>if I don‚Äôt explain the syntax for <code>COUNT(DISTINCT col)</code>, some people end up using the wrong syntax and getting stuck</li>\n<li>in the joins exercise, some people get stuck on join order. (they do <code>x LEFT join y</code> instead of <code>y LEFT JOIN x</code> and then don‚Äôt get the right answer)</li>\n</ul>\n\n<p>So basically (in addition to making more exercises) I think I need to spend more time cataloguing where/how people are getting stuck in practice and helping make sure fewer people get stuck.</p>\n\n<h3>the tech stack</h3>\n\n<p>To build this, I‚Äôm using:</p>\n\n<ul>\n<li><a href=\"https://github.com/kripken/sql.js\" rel=\"noopener noreferrer\">sql.js</a> to run the SQL queries in the browser with SQLite (there‚Äôs no server, it‚Äôs all done in the browser, so nobody can take down the server by running really expensive queries :) )</li>\n<li><a href=\"https://firebase.google.com/docs/firestore\" rel=\"noopener noreferrer\">firestore</a> to store the queries people are trying out</li>\n<li><a href=\"https://vuejs.org/\" rel=\"noopener noreferrer\">vue.js</a> to manage Javascript components</li>\n<li><a href=\"https://www.typescriptlang.org/\" rel=\"noopener noreferrer\">typescript</a> to compensate a bit for my lack of Javascript experience and help me refactor my code more easily</li>\n<li><a href=\"https://tailwindcss.com/\" rel=\"noopener noreferrer\">tailwind css</a> for CSS</li>\n</ul>\n\n<p>I also bought the <a href=\"https://refactoringui.com/\" rel=\"noopener noreferrer\">Refactoring UI</a> book to try to improve my web design skills a tiny bit. I think it‚Äôs helped a little so far.</p>\n\n<p>Vue components let me really easily add new challenges/exercises to a page like this:</p>\n\n<pre><code>&lt;Puzzle\n   id=\"count-the-owners\"\n   title=\'Count the number of different cat owners\'\n   description=\"\n   You can use &lt;code&gt;COUNT(DISTINCT column)&lt;/code&gt; to count distinct values of a column. (you can also do &lt;code&gt;SUM(DISTINCT column)&lt;/code&gt; or &lt;code&gt;AVG(DISTINCT column)&lt;/code&gt; but I\'m not sure why that would be useful.\n   \"\n   answer= \"\n   SELECT count(distinct(owner)) AS num_owners\n   from cats\n   \"\n   v-bind:table_names=\'[\"cats\"]\'\n   &gt;\n&lt;/Puzzle&gt;\n</code></pre>\n\n<h3>the goal: make something that‚Äôs worth $100 or so</h3>\n\n<p>What I‚Äôm working towards is making exercises &amp; challenges that would help\nsomeone with beginner/intermediate SQL skills improve their SQL fluency enough\nthat it‚Äôd easily be worth $100 to them. We‚Äôll see if I can get there! I don‚Äôt\nknow whether I‚Äôll price it at $100, but that‚Äôs my goal for how useful it should\nbe.</p>\n\n<p>The person I have in mind is sort of (as usual) myself 6 years ago, when I‚Äôd\n<em>heard</em> of SQL and could write a basic query but if you gave me a table of VERY\nINTERESTING DATA I couldn‚Äôt really effectively use SQL to answer the questions\nI had about it.</p>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/aggregation-challenge.png",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/10/01/zine-revenue-2019/",
        link: "https://jvns.ca/blog/2019/10/01/zine-revenue-2019/",
        title: "Zine revenue for 2019",
        date: 2019-10-01T12:53:58Z,
        summary: None,
        content: Some(
            "\n\n<p>I occasionally get questions like ‚ÄúCan you share what you‚Äôve learned about\nrunning a business?‚Äù The most surprising thing I‚Äôve learned is that it‚Äôs\npossible to make money by teaching people computer things on the internet, so I\nwant to make that a little more concrete by sharing the revenue from the <a href=\"https://wizardzines.com/\" rel=\"noopener noreferrer\">zine\nbusiness</a> so far in 2019. Here‚Äôs a graph of revenue by\nmonth (the last month is September 2019):</p>\n\n<p><img src=\"https://jvns.ca/images/revenue-by-month.png\" width=\"500px\"></p>\n\n<p>This adds up to $87,858 USD for 2019 so far, which (depending on what I release in the\nrest of this year) is on track to be similar to revenue for 2018 ($101,558).</p>\n\n<p>Until quite recently I‚Äôd been writing zines in my spare time, and now I‚Äôm <a href=\"https://jvns.ca/blog/2019/09/13/a-year-explaining-computer-things/\" rel=\"noopener noreferrer\">taking a year to focus on it</a>.</p>\n\n<h3>how $30,000 for September breaks down</h3>\n\n<p>The most obvious thing in that monthly revenue graph above is that 2 months\n(September and March) have way more revenue than all the others. This is\nbecause I released new zines (Bite Size Networking and HTTP: Learn your browser‚Äôs\nlanguage) in those months.</p>\n\n<p>Here‚Äôs how the $30,000 for September breaks down:</p>\n\n<ul>\n<li>it‚Äôs 85% sales to individuals, 15% <a href=\"https://wizardzines.com/zines-team/\" rel=\"noopener noreferrer\">corporate licenses</a></li>\n<li>it‚Äôs approximately:\n\n<ul>\n<li>$18,000 the new HTTP zine</li>\n<li>$10,000 the various zine packs (the 6 pack and the 3 pack)</li>\n<li>$2,000 other individual zines</li>\n</ul></li>\n</ul>\n\n<p>This September was the month with the most sales ever, which is mostly because\nof individual humans who find the zines useful (thank you!!).</p>\n\n<h3>expenses</h3>\n\n<p>The main expenses are paying illustrators and an accountant, a mailing list,\nand various books I buy to learn how to do things better. They probably come\nout to about 10% of revenue or so, and then there are taxes after that.</p>\n\n<h3>giving away free copies has been great</h3>\n\n<p>With the HTTP zine, like many of my previous zines, I‚Äôve been giving away one\nfree copy for every copy that people buy, so that people can get it even if $12\nis hard for them to afford. (if you can‚Äôt afford $12, <a href=\"https://gumroad.com/l/http-zine/buy-one-give-one\" rel=\"noopener noreferrer\">here‚Äôs the\nlink</a>, there are about 70\navailable as I‚Äôm writing this). I‚Äôm pretty happy with this setup ‚Äì we‚Äôve given\naway 1358 copies so far. (I think of this as kind of a ‚Äúsales‚Äù statistic too)</p>\n\n<p>I think I want to automate the system to give away free copies a bit more soon\n(like by automatically updating the number of free zines available using the\nGumroad API instead of periodically doing it manually).</p>\n\n<h3>hopefully this is a useful data point!</h3>\n\n<p>Writing about money on the internet is weird, so this will probably be the\nfirst and last zine revenue post, but I‚Äôm writing it down in the hopes that\nit‚Äôs a useful data point for others. I thought for a long time that you could\nonly really make money from writing on the internet with ads or sponsorships,\nbut it‚Äôs not true!</p>\n\n<p>The goal of this isn‚Äôt to say ‚Äúyou should run a business‚Äù or anything, just\nthat this is a thing that‚Äôs possible in the world and that many developers\ndo really value good educational materials and are happy to pay for them (if\nyou‚Äôre one of those people, thank you!)</p>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/revenue-by-month.png",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/10/03/sql-queries-don-t-start-with-select/",
        link: "https://jvns.ca/blog/2019/10/03/sql-queries-don-t-start-with-select/",
        title: "SQL queries don\'t start with SELECT",
        date: 2019-10-03T10:24:53Z,
        summary: None,
        content: Some(
            "\n\n<p>Okay, obviously many SQL queries do start with <code>SELECT</code> (and actually this post is only about <code>SELECT</code> queries, not <code>INSERT</code>s or anything).</p>\n\n<p>But! Yesterday I was working on an <a href=\"https://twitter.com/b0rk/status/1179419244808851462?s=20\" rel=\"noopener noreferrer\">explanation of window\nfunctions</a>, and I\nfound myself googling ‚Äúcan you filter based on the result of a window\nfunction‚Äù. As in ‚Äì can you filter the result of a window function in\na WHERE or HAVING or something?</p>\n\n<p>Eventually I concluded ‚Äúwindow functions must run after WHERE and GROUP BY\nhappen, so you can‚Äôt do it‚Äù. But this led me to a bigger question ‚Äì <strong>what\norder do SQL queries actually run in?</strong>.</p>\n\n<p>This was something that I felt like I knew intuitively (‚ÄúI‚Äôve written at least\n10,000 SQL queries, some of them were really complicated! I must know this!‚Äú)\nbut I struggled to actually articulate what the order was.</p>\n\n<h3>SQL queries happen in this order</h3>\n\n<p>I looked up the order, and here it is! (SELECT isn‚Äôt the first thing, it‚Äôs like the 5th thing!) (<a href=\"https://twitter.com/b0rk/status/1179449535938076673\" rel=\"noopener noreferrer\">here it is in a tweet</a>).</p>\n\n<p><small>\n(I really want to find a more accurate way of phrasing this than ‚Äúsql queries\nhappen/run in this order‚Äù but I haven‚Äôt figured it out yet)\n</small></p>\n\n<div>\n<a href=\"https://jvns.ca/images/sql-queries.jpeg\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/sql-queries.jpeg\" width=\"400px\"></a>\n</div>\n\n<p>In a non-image format, the order is:</p>\n\n<ul>\n<li><code>FROM/JOIN</code> and all the <code>ON</code> conditions</li>\n<li><code>WHERE</code></li>\n<li><code>GROUP BY</code></li>\n<li><code>HAVING</code></li>\n<li><code>SELECT</code> (including window functions)</li>\n<li><code>ORDER BY</code></li>\n<li><code>LIMIT</code></li>\n</ul>\n\n<h3>questions this diagram helps you answer</h3>\n\n<p>This diagram is about the <em>semantics</em> of SQL queries ‚Äì it lets you reason through what a given query will return and answers questions like:</p>\n\n<ul>\n<li>Can I do <code>WHERE</code> on something that came from a <code>GROUP BY</code>? (no! WHERE happens before GROUP BY!)</li>\n<li>Can I filter based on the results of a window function? (no! window functions happen in <code>SELECT</code>, which happens after both <code>WHERE</code> and <code>GROUP BY</code>)</li>\n<li>Can I <code>ORDER BY</code> based on something I did in GROUP BY? (yes! <code>ORDER BY</code> is basically the last thing, you can <code>ORDER BY</code> based on anything!)</li>\n<li>When does <code>LIMIT</code> happen? (at the very end!)</li>\n</ul>\n\n<p><strong>Database engines don‚Äôt actually literally run queries in this order</strong> because they\nimplement a bunch of optimizations to make queries run faster ‚Äì we‚Äôll get to\nthat a little later in the post.</p>\n\n<p>So:</p>\n\n<ul>\n<li>you can use this diagram when you just want to understand which queries are valid and how to reason about what results of a given query will be</li>\n<li>you <em>shouldn‚Äôt</em> use this diagram to reason about query performance or anything involving indexes, that‚Äôs a much more complicated thing with a lot more variables</li>\n</ul>\n\n<h3>confounding factor: column aliases</h3>\n\n<p>Someone on Twitter pointed out that many SQL implementations let you use the syntax:</p>\n\n<pre><code>SELECT CONCAT(first_name, \' \', last_name) AS full_name, count(*)\nFROM table\nGROUP BY full_name\n</code></pre>\n\n<p>This query makes it <em>look</em> like GROUP BY happens after SELECT even though GROUP BY is first, because the\nGROUP BY references an alias from the SELECT. But it‚Äôs not actually necessary\nfor the GROUP BY to run after the SELECT for this to work ‚Äì the database engine can\njust rewrite the query as</p>\n\n<pre><code>SELECT CONCAT(first_name, \' \', last_name) AS full_name, count(*)\nFROM table\nGROUP BY CONCAT(first_name, \' \', last_name)\n</code></pre>\n\n<p>and run the GROUP BY first.</p>\n\n<p>Your database engine also definitely does a bunch of checks to make sure that what you\nput in SELECT and GROUP BY makes sense together before it even starts to run\nthe query, so it has to look at the query as a whole anyway before it starts to\ncome up with an execution plan.</p>\n\n<h3>queries aren‚Äôt actually run in this order (optimizations!)</h3>\n\n<p>Database engines in practice don‚Äôt actually run queries by joining, and then\nfiltering, and then grouping, because they implement a bunch of optimizations\nreorder things to make the query run faster  as long as reordering things won‚Äôt\nchange the results of the query.</p>\n\n<p>One simple example of a reason why need to run queries in a different order to\nmake them fast is that in this query:</p>\n\n<pre><code>SELECT * FROM\nowners LEFT JOIN cats ON owners.id = cats.owner\nWHERE cats.name = \'mr darcy\'\n</code></pre>\n\n<p>it would be silly to do the whole left join and match up all the rows in the 2\ntables if you just need to look up the 3 cats named ‚Äòmr darcy‚Äô ‚Äì it‚Äôs way\nfaster to do some filtering first for cats named ‚Äòmr darcy‚Äô. And in this case\nfiltering first doesn‚Äôt change the results of the query!</p>\n\n<p>There are lots of other optimizations that database engines implement in\npractice that might make them run queries in a different order but there‚Äôs no\nroom for that and honestly it‚Äôs not something I‚Äôm an expert on.</p>\n\n<h3>LINQ starts queries with <code>FROM</code></h3>\n\n<p>LINQ (a querying syntax in C# and VB.NET) uses the order <code>FROM ... WHERE ... SELECT</code>. Here‚Äôs an example of a LINQ query:</p>\n\n<pre><code>var teenAgerStudent = from s in studentList\n                      where s.Age &gt; 12 &amp;&amp; s.Age &lt; 20\n                      select s;\n</code></pre>\n\n<p>pandas (my <a href=\"https://github.com/jvns/pandas-cookbook\" rel=\"noopener noreferrer\">favourite data wrangling\ntool</a>) also basically works like this,\nthough you don‚Äôt need to use this exact order ‚Äì I‚Äôll often write pandas code\nlike this:</p>\n\n<pre><code>df = thing1.join(thing2)      # like a JOIN\ndf = df[df.created_at &gt; 1000] # like a WHERE\ndf = df.groupby(\'something\', num_yes = (\'yes\', \'sum\')) # like a GROUP BY\ndf = df[df.num_yes &gt; 2]       # like a HAVING, filtering on the result of a GROUP BY\ndf = df[[\'num_yes\', \'something1\', \'something\']] # pick the columns I want to display, like a SELECT\ndf.sort_values(\'sometthing\', ascending=True)[:30] # ORDER BY and LIMIT\ndf[:30]\n</code></pre>\n\n<p>This isn‚Äôt because pandas is imposing any specific rule on how you have to\nwrite your code, though. It‚Äôs just that it often makes sense to write code in\nthe order JOIN / WHERE / GROUP BY / HAVING. (I‚Äôll often put a <code>WHERE</code> first to\nimprove performance though, and I think most database engines will also do a\nWHERE first in practice)</p>\n\n<p><code>dplyr</code> in R also lets you use a different syntax for querying SQL databases\nlike Postgres, MySQL and SQLite, which is also in a more logical order.</p>\n\n<h3>I was really surprised that I didn‚Äôt know this</h3>\n\n<p>I‚Äôm writing a blog post about this because when I found out the order I was SO\nSURPRISED that I‚Äôd never seen it written down that way before ‚Äì it explains\nbasically everything that I knew intuitively about why some queries are allowed\nand others aren‚Äôt. So I wanted to write it down in the hopes that it will help\nother people also understand how to write SQL queries.</p>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/sql-queries.jpeg",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/10/21/print-collection-of-my-first-7-zines/",
        link: "https://jvns.ca/blog/2019/10/21/print-collection-of-my-first-7-zines/",
        title: "Your Linux Toolbox: a box set of my free zines",
        date: 2019-10-21T16:15:19Z,
        summary: None,
        content: Some(
            "\n\n<p>About a year and a half ago, <a href=\"https://nostarch.com/\" rel=\"noopener noreferrer\">No Starch Press</a> got in\ntouch with me about publishing a print box set of my zines. I have two kinds of\n<a href=\"https://wizardzines.com/\" rel=\"noopener noreferrer\">zines</a> right now:</p>\n\n<ul>\n<li>6 free zines (which are the first ones I wrote)</li>\n<li>6 zines that cost $10-$12 each (the newer ones)</li>\n</ul>\n\n<p>This set is basically a really lovely box set of all of the free zines, plus\nBite Size Linux :). Here‚Äôs what‚Äôs in the box:</p>\n\n<ul>\n<li><a href=\"https://wizardzines.com/zines/strace\" rel=\"noopener noreferrer\">Spying on your programs with strace</a></li>\n<li><a href=\"https://wizardzines.com/zines/wizard\" rel=\"noopener noreferrer\">So you want to be a wizard</a></li>\n<li><a href=\"https://wizardzines.com/zines/networking\" rel=\"noopener noreferrer\">Networking! ACK!</a></li>\n<li><a href=\"https://wizardzines.com/zines/tcpdump\" rel=\"noopener noreferrer\">Let‚Äôs learn tcpdump!</a></li>\n<li><a href=\"https://wizardzines.com/zines/perf\" rel=\"noopener noreferrer\">Profiling and tracing with perf</a></li>\n<li><a href=\"https://wizardzines.com/zines/debugging\" rel=\"noopener noreferrer\">Linux debugging tools you‚Äôll love</a></li>\n<li><a href=\"https://wizardzines.com/zines/bite-size-linux\" rel=\"noopener noreferrer\">Bite Size Linux</a></li>\n</ul>\n\n<div>\n<a href=\"https://jvns.ca/images/toolbox-inbox.jpeg\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/toolbox-inbox.jpeg\" width=\"300px\"></a>\n</div>\n\n<p>I‚Äôm really happy to get these zines into print, and that I can still give away\nall of the zines in the box away for free on my website ‚Äì I asked them to\nwrite it into my publishing contract that I could still give them away, and\nthey did :)</p>\n\n<h3>what it looks like</h3>\n\n<p>Here are the front covers of the zines in the box. We got colour covers\nillustrated for all of them, done by Vladimir Ka≈°ikoviƒá.</p>\n\n<p><a href=\"https://jvns.ca/images/toolbox-front.jpeg\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/toolbox-front.jpeg\" width=\"500px\"></a></p>\n\n<p>We had the idea to make the back covers a rainbow and I‚Äôm delighted about it:</p>\n\n<p><a href=\"https://jvns.ca/images/toolbox-back.jpeg\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/toolbox-back.jpeg\" width=\"500px\"></a></p>\n\n<p>There‚Äôs this fun ‚Äúthis toolbox belongs to:‚Äù detail on the bottom:</p>\n\n<p><a href=\"https://jvns.ca/images/toolbox-belongsto.jpeg\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/toolbox-belongsto.jpeg\" width=\"500px\"></a></p>\n\n<h3>where to get it</h3>\n\n<p>It‚Äôs in a bunch of physical bookstores, and online! Here are a bunch of links to places you could get it:</p>\n\n<p>North America:</p>\n\n<ul>\n<li><a href=\"https://nostarch.com/linuxtoolbox\" rel=\"noopener noreferrer\">No Starch Press‚Äôs website</a> (you can use code WIZARD for 25% off)</li>\n<li>Amazon: <a href=\"https://amzn.to/2oS6W94\" rel=\"noopener noreferrer\">USA</a>, <a href=\"https://www.amazon.ca/Your-Linux-Toolbox-Julia-Evans/dp/1593279779\" rel=\"noopener noreferrer\">Canada</a>, <a href=\"https://www.amazon.com.mx/Your-Linux-Toolbox-Julia-Evans/dp/1593279779\" rel=\"noopener noreferrer\">Mexico</a></li>\n<li><a href=\"https://www.powells.com/book/-9781593279776\" rel=\"noopener noreferrer\">Powell‚Äôs</a></li>\n<li><a href=\"https://www.barnesandnoble.com/w/your-linux-toolbox-julia-evans/1130017024#/\" rel=\"noopener noreferrer\">Barnes &amp; Noble</a></li>\n<li><a href=\"https://www.chapters.indigo.ca/en-ca/books/your-linux-toolbox-a-zine/9781593279776-item.html\" rel=\"noopener noreferrer\">Chapters</a> (Canada)</li>\n<li><a href=\"https://www.alibris.com/Your-Linux-Toolbox-A-Zine-Boxset-Julia-Evans/book/42519152?matches=17\" rel=\"noopener noreferrer\">Alibris</a></li>\n</ul>\n\n<p>Europe:</p>\n\n<ul>\n<li>Amazon: <a href=\"https://www.amazon.co.uk/Your-Linux-Toolbox-Julia-Evans/dp/1593279779\" rel=\"noopener noreferrer\">UK</a>, <a href=\"https://www.amazon.de/Become-Linux-Wizard-Zine-Boxset/dp/1593279779\" rel=\"noopener noreferrer\">Germany</a>, <a href=\"https://www.amazon.es/Become-Linux-Wizard-Zine-Boxset/dp/1593279779\" rel=\"noopener noreferrer\">Spain</a>, <a href=\"https://www.amazon.it/Become-Linux-Wizard-Zine-Boxset/dp/1593279779\" rel=\"noopener noreferrer\">Italy</a>, <a href=\"https://www.amazon.fr/Become-Linux-Wizard-Zine-Boxset/dp/1593279779\" rel=\"noopener noreferrer\">France</a>, etc</li>\n<li><a href=\"https://www.waterstones.com/book/the-linux-toolbox/julia-evans/9781593279776\" rel=\"noopener noreferrer\">Waterstones</a></li>\n</ul>\n\n<p>Asia/Pacific:</p>\n\n<ul>\n<li><a href=\"https://www.fishpond.com.au/Books/Your-Linux-Toolbox-Julia-Evans/9781593279776\" rel=\"noopener noreferrer\">Fishpond</a> (Australia)</li>\n<li><a href=\"https://www.angusrobertson.com.au/books/your-linux-toolbox-julia-evans/p/9781593279776\" rel=\"noopener noreferrer\">Angus &amp; Robertson</a> (Australia)</li>\n<li>Amazon: <a href=\"https://www.amazon.in/Your-Linux-Toolbox-Julia-Evans/dp/1593279779\" rel=\"noopener noreferrer\">India</a></li>\n</ul>\n\n<p>More:</p>\n\n<ul>\n<li><a href=\"https://wordery.com/your-linux-toolbox-julia-evans-9781593279776?cTrk=MTYzMjUwNDI3fDVkYWU0MjM3ZWU4OWU6MToxOjVkYWU0MjJlZWY0YzMyLjkwMDI5NzA5OmQxMWZhZGRi\" rel=\"noopener noreferrer\">Wordery</a> (free worldwide shipping)</li>\n<li><a href=\"https://www.bookdepository.com/Your-Linux-Toolbox/9781593279776\" rel=\"noopener noreferrer\">Book Depository</a> (free worldwide shipping)</li>\n</ul>\n\n<h3>why I‚Äôm doing this: to learn about print!</h3>\n\n<p>I don‚Äôt necessarily expect to make a lot of money from this box set\n(I get 10% or less of each sale, vs 97% for sales of my <a href=\"https://wizardzines.com/\" rel=\"noopener noreferrer\">other zines</a> online) but that‚Äôs not my\npriority with this project ‚Äì I did it because I love the free zines I\nwrote, I wanted to make a really nice print version of them, and I wanted to\nlearn about how print works and how traditional publishing works! I‚Äôve already\nlearned a lot about how publishing works and it‚Äôs been super interesting.</p>\n\n<p>People have been very excited about this print project so far which has been\nreally nice to see! Next I want to make it possible for people to order print\ncopies of my newer zines, and I‚Äôm trying to figure out how to do that now. (if\nyou have a print company that you‚Äôve really loved using, let me know!)</p>\n\n<p>I‚Äôm super happy about the print quality and if you get the box set I really\nhope you like it!</p>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/toolbox-inbox.jpeg",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/10/28/sqlite-is-really-easy-to-compile/",
        link: "https://jvns.ca/blog/2019/10/28/sqlite-is-really-easy-to-compile/",
        title: "SQLite is really easy to compile",
        date: 2019-10-28T11:19:19Z,
        summary: None,
        content: Some(
            "\n\n<p>In the last week I‚Äôve been working on another SQL website\n(<a href=\"https://sql-steps.wizardzines.com/\" rel=\"noopener noreferrer\">https://sql-steps.wizardzines.com/</a>, a list of SQL examples). I‚Äôm running all\nthe queries on that site with sqlite, and I wanted to use window functions in\none of the examples (<a href=\"https://sql-steps.wizardzines.com/lag.html\" rel=\"noopener noreferrer\">this one</a>).</p>\n\n<p>But I‚Äôm using the version of sqlite from Ubuntu 18.04, and that version is too\nold and doesn‚Äôt support window functions. So I needed to upgrade sqlite!</p>\n\n<p>This turned to out be surprisingly annoying (as usual), but in a pretty\ninteresting way! I was reminded of some things about how executables and shared\nlibraries work and it had a very satisfying conclusion. So I wanted to write it up here.</p>\n\n<p>(spoiler: the summary is that <a href=\"https://www.sqlite.org/howtocompile.html\" rel=\"noopener noreferrer\">https://www.sqlite.org/howtocompile.html</a> explains\nhow to compile SQLite and it takes like 5 seconds to do and it‚Äôs 20x easier\nthan my usual experiences compiling software from source)</p>\n\n<h3>attempt 1: download a SQLite binary from their website</h3>\n\n<p>The <a href=\"https://www.sqlite.org/download.html\" rel=\"noopener noreferrer\">SQLite download page</a> has a link to\na Linux binary for the SQLite command line tool. I downloaded it, it worked on\nmy laptop, and I thought I was done.</p>\n\n<p>But then I tried to run it on a build server I was using (Netlify), and I got\nthis extremely strange error message: ‚ÄúFile not found‚Äù. I straced it, and sure\nenough <code>execve</code> was returning the error code ENOENT, which means ‚ÄúFile not\nfound‚Äù. This was kind of maddening because the file was DEFINITELY there and it\nhad the correct permissions and everything.</p>\n\n<p>I googled this problem (by searching ‚Äúexecve enoent‚Äù), found <a href=\"https://stackoverflow.com/questions/5234088/execve-file-not-found-when-stracing-the-very-same-file\" rel=\"noopener noreferrer\">this stack overflow answer</a>,\nwhich pointed out that to run a binary, you don‚Äôt just need the binary to exist! You also need its <strong>loader</strong> to exist. (the path to the loader is inside the binary)</p>\n\n<p>To see the path for the loader you can use <code>ldd</code>, like this:</p>\n\n<pre><code>$ ldd sqlite3\n\tlinux-gate.so.1 (0xf7f9d000)\n\tlibdl.so.2 =&gt; /lib/i386-linux-gnu/libdl.so.2 (0xf7f70000)\n\tlibm.so.6 =&gt; /lib/i386-linux-gnu/libm.so.6 (0xf7e6e000)\n\tlibz.so.1 =&gt; /lib/i386-linux-gnu/libz.so.1 (0xf7e4f000)\n\tlibc.so.6 =&gt; /lib/i386-linux-gnu/libc.so.6 (0xf7c73000)\n\t/lib/ld-linux.so.2\n</code></pre>\n\n<p>So <code>/lib/ld-linux.so.2</code> is the loader,and that file doesn‚Äôt exist on the build\nserver, probably because that Xenial installation didn‚Äôt have support for\n32-bit binaries (?), and I needed to try something different.</p>\n\n<h3>attempt 2: install the Debian sqlite3 package</h3>\n\n<p>Okay, I thought, maybe I can install the <a href=\"https://packages.debian.org/bullseye/amd64/sqlite3/download\" rel=\"noopener noreferrer\">sqlite package from debian\ntesting</a>. Trying\nto install a package from a different Debian version that I‚Äôm not using is\nliterally never a good idea, but for some reason I decided to try it anyway.</p>\n\n<p>Doing this completely unsurprisingly broke the sqlite installation on my\ncomputer (which also broke git), but I managed to recover from that with a\nbunch of <code>sudo dpkg --purge --force-all libsqlite3-0</code> and make everything that\ndepended on sqlite work again.</p>\n\n<h3>attempt 3: extract the Debian sqlite3 package</h3>\n\n<p>I also briefly tried to just extract the sqlite3 binary from the Debian sqlite\npackage and run it. Unsurprisingly, this also didn‚Äôt work, but in a more\nunderstandable way: I had an older version of libreadline (.so.7) and it wanted\n.so.8.</p>\n\n<pre><code>$ ./usr/bin/sqlite3\n./usr/bin/sqlite3: error while loading shared libraries: libreadline.so.8: cannot open shared object file: No such file or directory\n</code></pre>\n\n<h3>attempt 4: compile it from source</h3>\n\n<p>The whole reason I spent all this time trying to download sqlite binaries is\nthat I assumed it would be annoying or time consuming to compile sqlite from\nsource. But obviously downloading random sqlite binaries was not working for me\nat all, so I finally decided to try to compile it myself.</p>\n\n<p>Here are the directions: <a href=\"https://www.sqlite.org/howtocompile.html\" rel=\"noopener noreferrer\">How to compile\nSQLite</a>. And they‚Äôre the EASIEST\nTHING IN THE UNIVERSE. Often compiling things feels like this:</p>\n\n<ul>\n<li>run <code>./configure</code></li>\n<li>realize i‚Äôm missing a dependency</li>\n<li>run <code>./configure</code> again</li>\n<li>run <code>make</code></li>\n<li>the compiler fails because actually i have the wrong version of some dependency</li>\n<li>go do something else and try to find a binary</li>\n</ul>\n\n<p>Compiling SQLite works like this:</p>\n\n<ul>\n<li>download an <a href=\"https://www.sqlite.org/download.html\" rel=\"noopener noreferrer\">amalgamation tarball from the download page</a></li>\n<li>run <code>gcc shell.c sqlite3.c -lpthread -ldl</code></li>\n<li>that‚Äôs it!!!</li>\n</ul>\n\n<p>All the code is in one file (<code>sqlite.c</code>), and there are no weird dependencies! It‚Äôs amazing.</p>\n\n<p>For my specific use case I didn‚Äôt actually need threading support or readline\nsupport or anything, so I used the instructions on the compile page to create a\nvery simple binary that only used libc and no other shared libraries.</p>\n\n<pre><code>$ ldd sqlite3\n\tlinux-vdso.so.1 (0x00007ffe8e7e9000)\n\tlibc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbea4988000)\n\t/lib64/ld-linux-x86-64.so.2 (0x00007fbea4d79000)\n</code></pre>\n\n<h3>this is nice because it makes it easy to experiment with sqlite</h3>\n\n<p>I think it‚Äôs cool that SQLite‚Äôs build process is so simple because in the past\nI‚Äôve had fun <a href=\"https://jvns.ca/blog/2014/10/02/how-does-sqlite-work-part-2-btrees/\" rel=\"noopener noreferrer\">editing sqlite‚Äôs source code</a> to\nunderstand how its btree implementation works.</p>\n\n<p>This isn‚Äôt really super surprising given what I know about SQLite (it‚Äôs made to\nwork really well in restricted / embedded contexts, so it makes sense that it\nwould be possible to compile it in a really simple/minimal way). But it is\nsuper nice!</p>\n",
        ),
        media_type: None,
        media_link: None,
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/10/28/some-research-on-shipping-print-zines/",
        link: "https://jvns.ca/blog/2019/10/28/some-research-on-shipping-print-zines/",
        title: "Some research on shipping print zines",
        date: 2019-10-29T10:51:14Z,
        summary: None,
        content: Some(
            "\n\n<p>I‚Äôve been doing some preliminary research on shipping printed zines, since\n<a href=\"https://jvns.ca/blog/2019/10/21/print-collection-of-my-first-7-zines/\" rel=\"noopener noreferrer\">Your Linux Toolbox</a>\nis out now and a bunch more people have been asking about print copies of my\nother zines. I thought I‚Äôd write down what I‚Äôve learned so far because it turns\nout shipping is pretty complicated!</p>\n\n<p>My original question I was trying to answer was ‚Äúcan I ship a single zine\nanywhere in the world for less than $6 or so?‚Äú, so let‚Äôs start there.</p>\n\n<p>Surprisingly the best single resource I found was this\nvery extensive <a href=\"https://www.pcmag.com/roundup/360934/the-best-e-commerce-fulfillment-services\" rel=\"noopener noreferrer\">PCMag article on e-commerce fulfillment services</a>.</p>\n\n<h3>why not use letter mail?</h3>\n\n<p>The most obvious way to send a zine inexpensively in the mail is with letter\nmail ‚Äì letters smaller than 10‚Äù x 6‚Äù and under 60g or so can be sent by letter\nmail. My zines definitely fit that criteria, even when printed on nice paper.\nThis would be really good because international package postage is EXPENSIVE,\nbut sending a letter to Belgium only costs <a href=\"https://postcalc.usps.com/Calculator/MailServices?country=10033&amp;ccode=BE&amp;oz=90219&amp;omil=False&amp;dmil=False&amp;mdt=10%2F28%2F2019&amp;mdz=11%3A29&amp;m=1&amp;dvi=0&amp;o=2\" rel=\"noopener noreferrer\">$2.13 according to USPS‚Äôs\nwebsite</a>.</p>\n\n<p>The issue with this is the small print on that USPS page:</p>\n\n<blockquote>\n<p>Value of contents can not exceed $0.00</p>\n</blockquote>\n\n<p>So it seems like you‚Äôre not actually allowed to send things worth money via\nletter mail. Probably that‚Äôs related to customs or something. Or maybe letter\nmail is subsidized by the government? Not sure why.</p>\n\n<h3>Option 0: Ship zines myself</h3>\n\n<p>I‚Äôve done this before and it was actually really fun to do once but I think this is pretty unlikely to be a good idea because:</p>\n\n<p>a. cost: I live in Canada, almost everyone I sell zines to is outside of Canada, and\nb. availability: I‚Äôd like for people to be able to get shipments when I‚Äôm out of town / on vacation</p>\n\n<h3>Option 1: Amazon</h3>\n\n<p>One obvious answer to how to sell book-like things is ‚Äúsell them on Amazon!‚Äù.\nAmazon actually has at least 3 different programs that you can use to sell\nbooks online (Amazon Advantage, Fulfilled By Amazon, Kindle Direct Publishing),\nand since Amazon is such a big thing I looked into all of them a little bit.</p>\n\n<p>In general the forums on <a href=\"https://sellercentral.amazon.com/\" rel=\"noopener noreferrer\">https://sellercentral.amazon.com</a> seem to be a good way\nto understand how the various Amazon options work for people.</p>\n\n<p>I wrote a lot about Amazon here but overall it doesn‚Äôt seem that great of an\noption ‚Äì it‚Äôs really complicated, selling on Amazon‚Äôs website isn‚Äôt very\nappealing, and I think there would be a lot of additional fees.</p>\n\n<p><strong>Kindle Direct Publishing</strong></p>\n\n<p>Kindle Direct Publishing is a service where Amazon will take care of everything\nfrom printing to shipping. (It has ‚ÄúKindle‚Äù in the same but they actually do\nprinting as well). <a href=\"https://www.cs.princeton.edu/~bwk/\" rel=\"noopener noreferrer\">Brian Kernighan‚Äôs new Unix\nbook</a> is an example of a book published\nwith KDP.</p>\n\n<p>KDP won‚Äôt work for this project because they don‚Äôt support saddle stitching\n(stapling the zine), so I didn‚Äôt look into it too much. Here‚Äôs a link to their\n<a href=\"https://kdp.amazon.com/en_US/help/topic/G201834180\" rel=\"noopener noreferrer\">paper and cover options</a>\nthough.</p>\n\n<p><strong>Amazon Advantage</strong></p>\n\n<p>Amazon Advantage doesn‚Äôt do printing ‚Äì you ship them books, and then they take\ncare of shipping them to people. This seems great on its surface (‚Äúamazon just\ntakes care of it!‚Äú).</p>\n\n<p>advantages:</p>\n\n<ul>\n<li>the book on Amazon looks the same as a book from a major publisher, it shows\nup as ‚Äúships and sold by Amazon‚Äù</li>\n</ul>\n\n<p>disadvantages:</p>\n\n<ul>\n<li>Amazon Advantage takes 55% and <strong>they</strong> decide how to price your book.</li>\n<li>It costs $99/year</li>\n<li>‚ÄúYou have to ship them your books every week or two depending on how many\ncopies they want, you can‚Äôt just ship them a big shipment to keep in a\nwarehouse‚Äù</li>\n</ul>\n\n<p>The biggest issue for me here seems to be ‚Äúyou have to ship them books every\nweek or two‚Äù, which seems like it could get very expensive if Amazon Advantage\nkeeps asking you to ship them small quantities of zines.</p>\n\n<p><strong>Fulfilled By Amazon</strong></p>\n\n<p>Fulfilled By Amazon seems like the Amazon option that involves the least Amazon\nmagic. Basically I‚Äôd ship books to their warehouses and then they ship the things from those warehouses.</p>\n\n<p>how it works:</p>\n\n<ul>\n<li>you just pay them for shipping and warehousing fees (no percentage cost)</li>\n<li>you can ship them a big batch of things and they‚Äôll store them until they‚Äôre sold</li>\n<li>they have <a href=\"https://services.amazon.com/global-selling/global-selling-guide.html\" rel=\"noopener noreferrer\">FBA global selling</a>\nto sell in countries other than the US (by shipping to Amazon warehouses in\nthose countries) or <a href=\"https://sellercentral.amazon.fr/gp/help/external/200149570?language=en_GB&amp;ref=efph_200149570_cont_201101640\" rel=\"noopener noreferrer\">FBA export</a>\n(to sell to other countries from the US store). There are a bunch of scary words on those pages about international taxes.</li>\n</ul>\n\n<h3>Option 2: Blackbox</h3>\n\n<p>Blackbox is a shipping company by the Cards Against Humanity folks. This is\ntheir <a href=\"https://www.blackbox.cool/downloads/Blackbox_2019_Pricing.pdf\" rel=\"noopener noreferrer\">Pricing PDF</a>. I‚Äôm not\n100% sure if I can work with them ‚Äì the first time I filled out the form on\ntheir website saying I was interested they said they weren‚Äôt accepting new\ncustomers, but I think now they may be?</p>\n\n<p>Here‚Äôs the summary:</p>\n\n<ul>\n<li>$6.64 for domestic US shipping</li>\n<li>$11ish for domestic Canadian shipping</li>\n<li>$10ish for domestic Australia shipping</li>\n<li>$7ish for domestic UK shipping</li>\n<li>$10ish for international shipping from their UK warehouse to anywhere in the world</li>\n</ul>\n\n<h3>Option 3: Shipbob</h3>\n\n<p>Shipbob is an shipping company for smaller ecommerce companies. Here‚Äôs the <a href=\"https://shipbobcdn.azureedge.net/webassets/dashboard/files/ShipBob_GrowthPlan_Quote_2019.pdf\" rel=\"noopener noreferrer\">pricing PDF I found</a>.</p>\n\n<ul>\n<li>$6ish for domestic US shipping</li>\n<li>$10ish for shipping to other countries</li>\n</ul>\n\n<p>The main difference as far as I can tell between Shipbob and Blackbox is that\nShipbox lets you include up to 5 items per order for free and Blackbox charges\n$0.70 per additional item in an order.</p>\n\n<p>Shipbob advertises a 99.8% fulfillment accuracy rate which is pretty\ninteresting ‚Äì it means they expect 2 in 1000 orders to have a problem. That\nseems pretty good!</p>\n\n<h3>Option 4: Whiplash</h3>\n\n<p>Similar to the other two above. <a href=\"https://sales.getwhiplash.com/pricing\" rel=\"noopener noreferrer\">Their pricing\npage</a>. Shipbob and Blackbox both include\neverything (shipping, packaging materials, and someone packing your order) in\ntheir fee, and Whiplash seems to charge separately for shipping.</p>\n\n<h3>Shipping 1 thing costs the same as shipping 5 things</h3>\n\n<p>Zines are pretty light ‚Äì I just weighed some of my zines printed on high\nquality paper on my kitchen scale and they‚Äôre 40g each on average. Most of\nthese shipping services seem to charge in increments of half a pound, so\nshipping 5 zines (about 200g) costs about the same as shipping 1 zine (about\n40g).</p>\n\n<p>This makes me think it would be more reasonable to focus on shipping packages\nof many zines ‚Äì right now the <a href=\"https://wizardzines.com/zines/all-the-zines/\" rel=\"noopener noreferrer\">6 pack of all my zines</a> costs $58 for a PDF\ncollection, and $6-$12 shipping for something around that price seems\nsuper reasonable (and I could probably even do ‚Äúfree‚Äù shipping, aka pay the\nshipping costs myself).</p>\n\n<p>The other thing that I think could be work well is shipping packages of 5 or 10\nof the same zine so that a group of people can each get a zine and save on\nshipping costs.</p>\n\n<h3>this seems like it could work!</h3>\n\n<p>I still have no plan for how to print zines, but writing all this down makes me\nfeel pretty optimistic about being able to ship zines to people. Even though\nshipping individual zines doesn‚Äôt seem that practical, I think shipping packs\nof 5-10 zines could be really reasonable!</p>\n\n<p>Speaking of print ‚Äì I printed a zine with Lulu last week and just got it in\nthe mail yesterday. I didn‚Äôt think Lulu would be able to print it in the way I\nwanted, and they didn‚Äôt, so I‚Äôm really happy to know that and be able to move\non to trying other non-print-on-demand printers.</p>\n",
        ),
        media_type: None,
        media_link: None,
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/11/18/some-notes-on-vector-drawing-apps/",
        link: "https://jvns.ca/blog/2019/11/18/some-notes-on-vector-drawing-apps/",
        title: "Some notes on vector drawing apps",
        date: 2019-11-18T10:47:41Z,
        summary: None,
        content: Some(
            "\n\n<p>For the last year and a half I‚Äôve been using the iPad Notability app to draw my\n<a href=\"https://wizardzines.com/\" rel=\"noopener noreferrer\">zines</a>. Last week I decided I wanted more features, did a bit of research, and\ndecided to switch to Affinity Designer (a much more complicated program). So\nhere are a few quick notes about it.</p>\n\n<p>The main difference between them is that Notability is a simple note taking app\n(aimed at regular people), and Affinity Designer is a vector graphics app\n(aimed at illustrators / graphic designers), like Adobe Illustrator.</p>\n\n<p>I‚Äôve never used a serious vector graphics program before, so it‚Äôs been cool to\nlearn what kinds of features are available!</p>\n\n<h3>Notability is super simple</h3>\n\n<p>This is what the Notability UI looks like. There‚Äôs a pencil, an eraser, a text\ntool, and a selection tool. That‚Äôs basically it. I LOVED this simplicity when I\nstarted using Notability, and I made 4 zines using it (help! i have a manager!,\noh shit, git!, bite size networking!, and http: use your browser‚Äôs language).</p>\n\n<p><a href=\"https://jvns.ca/images/notability.jpg\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/notability.jpg\" width=\"400px\"></a></p>\n\n<p>Recently though, I‚Äôve had a couple of problems with it, the main one being that\ntext boxes and things drawn with the pencil tool don‚Äôt mix well. (In general\nNotability has been GREAT though and their support team has always been\nincredibly helpful when I‚Äôve had questions.)</p>\n\n<h3>Affinity Designer is really complicated</h3>\n\n<p>Affinity Designer, by comparison, is WAY more complicated. Here‚Äôs what the UI looks like:</p>\n\n<p><a href=\"https://jvns.ca/images/affinity-designer.jpg\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/affinity-designer.jpg\" width=\"400px\"></a></p>\n\n<p>There are</p>\n\n<ul>\n<li>14 tools on the left</li>\n<li>14 more panels on the right that alter what the tools do</li>\n<li>a bottom toolbar which has different options for each tool</li>\n<li>2 menus which together have another 25 things or so that you can do</li>\n</ul>\n\n<p>I still don‚Äôt understand what all the tools do (what‚Äôs the\ndifference between Pencil and Vector Brush? I don‚Äôt know!). But I‚Äôm pretty\nexcited about this because (unlike with Notability) there are so many options\nthat if I‚Äôm frustrated about something, 90% of the time there‚Äôs a way to do the\nthing I want!</p>\n\n<h3>switching from Notability to Affinity Designer is really easy</h3>\n\n<p>Switching to Notability wasn‚Äôt the best: I <a href=\"https://jvns.ca/blog/2018/03/31/reverse-engineering-notability-format/\" rel=\"noopener noreferrer\">reverse engineered the file\nformat</a>\nto transfer some files over but the quality was never the best (probably\nbecause of problems with my script) and I ended up having to redraw a lot of\nthem in practice.</p>\n\n<p>With Affinity Designer, I can just</p>\n\n<ul>\n<li>export a PDF with Notability (or anything else)</li>\n<li>import the PDF with Affinity Designer</li>\n<li>and then I can easily edit it and that‚Äôs it?!?</li>\n</ul>\n\n<p>It‚Äôs not perfect ‚Äì the vector paths it comes up with are kind of weird, probably\nbecause of the way the PDF is ‚Äì but it‚Äôs very good! It makes me feel confident\nthat if I need to make a small edit to something I made in the past I can just\nimport the PDF!</p>\n\n<h3>what can a vector drawing app do?</h3>\n\n<p>here are a few things Affinity Designer can do that Notability can‚Äôt:</p>\n\n<ul>\n<li><strong>custom paper sizes</strong>: In Notability every page is 8.5x11, but usually I\nwant something more like 5.5x8.5 which is a different aspect ratio (this is\ntechnically sort of possible in Notability by importing a PDF of the correct\nsize but it‚Äôs a pain and it means you can‚Äôt use a grid)</li>\n<li><strong>custom colour palettes</strong>: I can have the 10 colours I like to use in my\ncomics all in one place</li>\n<li><strong>grouping objects together</strong> ‚Äì I can ‚Äúgroup‚Äù a bunch of objects together so\nthat I can easily resize and move them all together</li>\n<li><strong>two kinds of text box</strong>. This is the kind of thing that I wouldn‚Äôt have\nunderstood 2 years ago but that now I LOVE ‚Äì you can either have ‚Äúart text‚Äù\nthat acts like an image (no word wrap, gets bigger when you resize it) or\n‚Äúframe text‚Äù that has word wrap and doesn‚Äôt get bigger when you resize it.</li>\n<li><strong>really great import</strong> ‚Äì I can import a PDF or SVG and individually edit /\nmove around parts of the PDF. Notability doesn‚Äôt have any import tools that\nlet you edit after importing.</li>\n<li><strong>custom export options for printing</strong>. I don‚Äôt understand what it <strong>does</strong>\nyet but there are export presets for print PDFs and it fixes some printing\nproblems I was having!</li>\n</ul>\n\n<p>There are also a LOT more features that I‚Äôm not interested in but I‚Äôm pretty\nexcited about those 6 things and it feels like an app that I won‚Äôt grow out of.</p>\n\n<h3>iPad apps are great</h3>\n\n<p>I‚Äôve been exclusively using Linux for the last 15 years where the image\nediting/media tools aren‚Äôt always great (though I really like Inkscape and I hear good things about Krita!), so\nit‚Äôs really cool to have access to all these great iPad apps. And the prices seem pretty reasonable:</p>\n\n<ul>\n<li>Notability is $12</li>\n<li>Affinity Designer is $20</li>\n<li>LumaFusion (a nice video editor I‚Äôve been using a little) is $30</li>\n</ul>\n\n<p>It doesn‚Äôt make me want a Mac (I like the Linux desktop experience!), but it‚Äôs\nnice to have access to a bunch of these great tools. And I think a lot of\nthese art tools work better on an iPad than on a computer anyway since you can\njust draw on the screen :)</p>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/notability.jpg",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/11/18/how-containers-work--overlayfs/",
        link: "https://jvns.ca/blog/2019/11/18/how-containers-work--overlayfs/",
        title: "How containers work: overlayfs",
        date: 2019-11-18T13:34:29Z,
        summary: None,
        content: Some(
            "\n\n<p>I wrote a comic about overlay filesystems for a potential future container <a href=\"https://wizardzines.com/\" rel=\"noopener noreferrer\">zine</a>\nthis morning, and then I got excited about the topic and wanted to write a blog\npost with more details. Here‚Äôs the comic, to start out:</p>\n\n<p><a href=\"https://jvns.ca/images/overlay.jpeg\" rel=\"noopener noreferrer\">\n<img src=\"https://jvns.ca/images/overlay.jpeg\">\n</a></p>\n\n<h3>container images are big</h3>\n\n<p>Container images can be pretty big (though some are really small, like <a href=\"https://hub.docker.com/_/alpine?tab=tags\" rel=\"noopener noreferrer\">alpine\nlinux is 2.5MB</a>). Ubuntu 16.04 is\nabout 27MB, and <a href=\"https://hub.docker.com/r/continuumio/anaconda3/tags\" rel=\"noopener noreferrer\">the Anaconda Python distribution is 800MB to\n1.5GB</a>.</p>\n\n<p>Every container you start with an image starts out with the same blank slate,\nas if it made a copy of the image just for that container to use. But for big\ncontainer images, like that 800MB Anaconda image, making a copy would be both a\nwaste of disk space and pretty slow. So Docker doesn‚Äôt make copies ‚Äì instead\nit uses an <strong>overlay</strong>.</p>\n\n<h3>how overlays work</h3>\n\n<p>Overlay filesystems, also known as ‚Äúunion filesystems‚Äù or ‚Äúunion mounts‚Äù let you mount a filesystem using 2 directories: a ‚Äúlower‚Äù directory, and an ‚Äúupper‚Äù directory.</p>\n\n<p>Basically:</p>\n\n<ul>\n<li>the <strong>lower</strong> directory of the filesystem is read-only</li>\n<li>the <strong>upper</strong> directory of the filesystem can be both read to and written from</li>\n</ul>\n\n<p>When a process <strong>reads</strong> a file, the overlayfs filesystem driver looks in the upper\ndirectory and reads the file from there if it‚Äôs present. Otherwise, it looks in\nthe lower directory.</p>\n\n<p>When a process <strong>writes</strong> a file, overlayfs will just write it to the upper directory.</p>\n\n<h3>let‚Äôs make an overlay with <code>mount</code>!</h3>\n\n<p>That was all a little abstract, so let‚Äôs make an overlay filesystem and try\nit out! This is just going to have a few files in it: I‚Äôll make upper and lower\ndirectories, and a <code>merged</code> directory to mount the combined filesystem into:</p>\n\n<pre><code>$ mkdir upper lower merged work\n$ echo \"I\'m from lower!\" &gt; lower/in_lower.txt \n$ echo \"I\'m from upper!\" &gt; upper/in_upper.txt\n$ # `in_both` is in both directories\n$ echo \"I\'m from lower!\" &gt; lower/in_both.txt \n$ echo \"I\'m from upper!\" &gt; upper/in_both.txt \n</code></pre>\n\n<p>Combining the upper and lower directories is pretty easy: we can just do it with <code>mount!</code></p>\n\n<pre><code>$ sudo mount -t overlay overlay \n    -o lowerdir=/home/bork/test/lower,upperdir=/home/bork/test/upper,workdir=/home/bork/test/work \n    /home/bork/test/merged\n</code></pre>\n\n<p>There‚Äôs was an extremely annoying error message I kept getting while doing\nthis, that said <code>mount: /home/bork/test/merged: special device overlay does not\nexist.</code>. This message is a lie, and actually just means that one of the\ndirectories I specified was missing (I‚Äôd written <code>~/test/merged</code> but it wasn‚Äôt being expanded).</p>\n\n<p>Okay, let‚Äôs try to read one of the files from the overlay filesystem! The file <code>in_both.txt</code> exists in both <code>lower/</code> and <code>upper/</code>, so it should read the file from the <code>upper/</code> directory.</p>\n\n<pre><code>$ cat merged/in_both.txt \n\"I\'m from upper!\n</code></pre>\n\n<p>It worked!</p>\n\n<p>And the contents of our directories are what we‚Äôd expect:</p>\n\n<pre><code>find lower/ upper/ merged/\nlower/\nlower/in_lower.txt\nlower/in_both.txt\nupper/\nupper/in_upper.txt\nupper/in_both.txt\nmerged/\nmerged/in_lower.txt\nmerged/in_both.txt\nmerged/in_upper.txt\n</code></pre>\n\n<h3>what happens when you create a new file?</h3>\n\n<pre><code>$ echo \'new file\' &gt; merged/new_file\n$ ls -l */new_file \n-rw-r--r-- 1 bork bork 9 Nov 18 14:24 merged/new_file\n-rw-r--r-- 1 bork bork 9 Nov 18 14:24 upper/new_file\n</code></pre>\n\n<p>That makes sense, the new file gets created in the <code>upper</code> directory.</p>\n\n<h3>what happens when you delete a file?</h3>\n\n<p>Reads and writes seem pretty straightforward. But what happens with deletes? Let‚Äôs do it!</p>\n\n<pre><code>$ rm merged/in_both.txt\n</code></pre>\n\n<p>What happened? Let‚Äôs look with <code>ls</code>:</p>\n\n<pre><code>ls -l upper/in_both.txt  lower/lower1.txt  merged/lower1.txt\nls: cannot access \'merged/in_both.txt\': No such file or directory\n-rw-r--r-- 1 bork bork    6 Nov 18 14:09 lower/in_both.txt\nc--------- 1 root root 0, 0 Nov 18 14:19 upper/in_both.txt\n</code></pre>\n\n<p>So:</p>\n\n<ul>\n<li><code>in_both.txt</code> is still in the <code>lower</code> directory, and it‚Äôs unchanged</li>\n<li>it‚Äôs not in the <code>merged</code> directory. So far this is all what we expected.</li>\n<li>But what happened in <code>upper</code> is a little strange: there‚Äôs a file called\n<code>upper/in_both.txt</code>, but it‚Äôs a‚Ä¶ character device? I guess this is how the\noverlayfs driver represents a file being deleted.</li>\n</ul>\n\n<p>What happens if we try to copy this weird character device file?</p>\n\n<pre><code>$ sudo cp upper/in_both.txt upper/in_lower.txt\ncp: cannot open \'upper/in_both.txt\' for reading: No such device or address\n</code></pre>\n\n<p>Okay, that seems reasonable, being able to copy this weird deletion signal file doesn‚Äôt really make sense.</p>\n\n<h3>you can mount multiple ‚Äúlower‚Äù directories</h3>\n\n<p>Docker images are often composed of like 25 ‚Äúlayers‚Äù. Overlayfs supports having\nmultiple lower directories, so you can run</p>\n\n<pre><code>mount -t overlay overlay\n      -o lowerdir:/dir1:/dir2:/dir3:...:/dir25,upperdir=...\n</code></pre>\n\n<p>So I assume that‚Äôs how containers with many Docker layers work, it just unpacks\neach layer into a separate directory and then asks overlayfs to combine them\nall together together with an empty upper directory that the container will write its changes to it.</p>\n\n<h3>docker can also use btrfs snapshots</h3>\n\n<p>Right now I‚Äôm using ext4, and Docker uses overlayfs snapshots to run\ncontainers. But I used to use btrfs, and then Docker would use btrfs copy-on-write snapshots\ninstead. (Here‚Äôs a list of when Docker uses which <a href=\"https://docs.docker.com/storage/storagedriver/select-storage-driver/\" rel=\"noopener noreferrer\">storage drivers</a>)</p>\n\n<p>Using btrfs snapshots this way had some interesting consequences ‚Äì at some\npoint last year I was running hundreds of short-lived Docker containers on my\nlaptop, and this resulted in me running out of btrfs metadata space (like <a href=\"https://www.reddit.com/r/archlinux/comments/5jrmfe/btrfs_metadata_and_docker/\" rel=\"noopener noreferrer\">this person</a>).\nThis was really confusing because I‚Äôd never heard of btrfs metadata before and it\nwas tricky to figure out how to clean up my filesystem so I could run Docker\ncontainers again. (<a href=\"https://github.com/moby/moby/issues/27653\" rel=\"noopener noreferrer\">this docker github issue</a> describes a similar problem\nwith Docker and btrfs)</p>\n\n<h3>it‚Äôs fun to try out container features in a simple way!</h3>\n\n<p>I think containers often seem like they‚Äôre doing ‚Äúcomplicated‚Äù things and I\nthink it‚Äôs fun to break them down like this ‚Äì you can just run one <code>mount</code>\nincantation without actually doing anything else related to containers at all\nand see how overlays work!</p>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/overlay.jpeg",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/11/20/what-makes-a-programming-exercise-good/",
        link: "https://jvns.ca/blog/2019/11/20/what-makes-a-programming-exercise-good/",
        title: "What makes a programming exercise good?",
        date: 2019-11-20T20:33:20Z,
        summary: None,
        content: Some(
            "\n\n<p>I‚Äôve been thinking about programming exercises lately, because I want to move\ninto teaching people skills. But what makes a good programming exercise? I\n<a href=\"https://twitter.com/b0rk/status/1197282185230860288\" rel=\"noopener noreferrer\">asked about this on Twitter today</a> and got some useful\nresponses so here are some criteria:</p>\n\n<h3>it‚Äôs fun</h3>\n\n<p>This one is sort of self-explanatory and I think it‚Äôs really important.\nProgramming is fun and learning is fun so I can‚Äôt see why programming exercises\nwould need to be boring.</p>\n\n<h3>it teaches you something you care about</h3>\n\n<p>I don‚Äôt think this has to strictly mean ‚Äúrelevant to your job right this\nsecond‚Äù ‚Äì people don‚Äôt just have jobs, we also want to make art and games and\nfun personal projects and sometimes just understand the world around us. But\nit‚Äôs important to know what goals the exercise can help you with and what it‚Äôs related to!</p>\n\n<p>Some arbitrary examples:</p>\n\n<ul>\n<li>take an image of something from a website and reproduce it from scratch with\nCSS (towards using CSS to make your own websites that look awesome)</li>\n<li>write a webserver from scratch without any frameworks (to learn the HTTP protocol,\nso that you can debug issues with a real webserver more easily)</li>\n<li>write a small raytracer (so you can make cool art with raytracing techniques on shaderhub!)</li>\n<li>write a tiny bit of assembly (as a very initial step towards understanding of\nwhat Spectre and Meltdown are even about and why we need to make all our\ncomputers run slower to prevent them)</li>\n</ul>\n\n<h3>it‚Äôs a challenge</h3>\n\n<p>I don‚Äôt know if this is everyone‚Äôs experience but I often start programming\nexercises and get bored quickly (‚Äúoh, I know how to do this, this is boring‚Äù).\nFor me it‚Äôs really important for the exercise to teach me something I really\ndon‚Äôt know how to do and that‚Äôs a little bit hard for me.</p>\n\n<p>My favourite set of programming exercises is the <a href=\"https://cryptopals.com/\" rel=\"noopener noreferrer\">cryptopals crypto\nchallenges</a> because they get harder pretty fast ‚Äì by\nexercise #6, you‚Äôre already breaking toy encryption protocols, and by #12\nyou‚Äôre breaking an Actual Encryption Protocol (AES in ECB mode)!</p>\n\n<h3>you can tell if you succeeded</h3>\n\n<p>It‚Äôs easy to write exercises that are too vaguely specified (‚Äúwrite a toy tcp\nstack!‚Äú). But what does that mean? How much of a TCP stack am I supposed to\nwrite? Having test cases and clear criteria for ‚Äúyay! you did it!\ncongratulations!‚Äù is really important.</p>\n\n<h3>you can do it quickly</h3>\n\n<p>In less than 2-3 hours (an evening after work), say. It‚Äôs hard to find time to\nspend like 8 hours on an exercise unless it‚Äôs REALLY exciting.</p>\n\n<p>I also think that giving some specific real-world benchmark data seems nice (‚ÄúI\ndid this from scratch in 97 minutes‚Äù).</p>\n\n<h3>the author believes in you</h3>\n\n<p>This is a bit fuzzier but very lovely ‚Äì <a href=\"https://twitter.com/mojavelinux/status/1197323090427953152\" rel=\"noopener noreferrer\">this person on Twitter wrote</a>:</p>\n\n<blockquote>\n<p>Similar to that, the writing is patient and gives me the impression that it\nbelieves in my ability to accomplish the task. ‚Ä¶ I learned a ton in the early\ndays from Linux HOWTOs. Some gave me the sense that it was impossible to\nfail. Just follow the steps. It‚Äôs all there.</p>\n</blockquote>\n\n<p>Especially if you‚Äôre doing a somewhat challenging exercise like we talked\nabout above, I think it‚Äôs nice for the author to believe in your! (and of course it‚Äôs\ncrucial that they‚Äôve actually written the exercises so that they‚Äôre <em>right</em> and\nyou can likely do the thing!)</p>\n\n<h3>it‚Äôs been tested</h3>\n\n<p>I read the (great) biography <a href=\"http://www.amazon.com/exec/obidos/ASIN/0307473414/metafilter-20/ref=nosim/\" rel=\"noopener noreferrer\">Dearie: The Remarkable Life of Julia Child</a>\nrecently and one thing that stood out to me is that she <em>tested</em> all of the\nrecipes in Mastering the Art Of French Cooking. It took her <em>years</em> to write\nthe book and test the recipes and make sure that American home cooks actually\nhad access to all the ingredients and had the.</p>\n\n<p>I don‚Äôt think all cookbook authors test their recipes, but I think testing\nreally improves cookbooks.</p>\n\n<p>I started writing some SQL exercises (like <a href=\"https://joins-238123.netlify.com/aggregations/\" rel=\"noopener noreferrer\">this prototype of one on GROUP\nBY</a>) a while back, and at some point I\nrealized the big thing holding me back was that I didn‚Äôt have testers! I\ncouldn‚Äôt find out if people were actually learning from them or not!</p>\n\n<p>This is a new thing for me because when I write blog posts I don‚Äôt test them (I\nbarely even proofread them!). I just write them and publish and people often\nlike them and that‚Äôs it! I said to <a href=\"https://stackingthebricks.com/\" rel=\"noopener noreferrer\">Amy Hoy</a>\n(who is amazing) on Twitter that I didn‚Äôt understand why you have to test\nexercises if you don‚Äôt have to test blog posts and she <a href=\"https://twitter.com/amyhoy/status/1197291805449940993\" rel=\"noopener noreferrer\">pointed out</a> that people\nhave much higher expectations for exercises than for blog posts ‚Äì with the\nblog posts you maybe expect to learn 1-2 new facts, but with exercises you\nexpect to actually develop a new skill!</p>\n\n<p>Also, people are often investing a lot more time in exercises (especially if\nthey have to set up a dev environment or something!), so it‚Äôs extra important\nto make sure that they actually work.</p>\n\n<h3>you won‚Äôt get stuck</h3>\n\n<p>It‚Äôs SO EASY to get stuck on some random irrelevant point in a programming\nexercise that‚Äôs totally unrelated to the skill you‚Äôre trying to learn. For\nexample there might be an easily-avoidable mistake that you can make with the\nexercise and spend a lot of time debugging but it doesn‚Äôt actually teach you a\nlot.</p>\n\n<h3>it‚Äôs easy to get help</h3>\n\n<p>If you‚Äôre doing a challenging exercise, you might want to get help from your\nfriends / colleagues / the internet!</p>\n\n<p>Some things that can go wrong:</p>\n\n<ul>\n<li>None of your friends have ever heard of the thing the exercise is teaching so\nyou can‚Äôt talk about it with them</li>\n<li>The exercise expects you to be using the newest version of some software, but\nactually all the examples on the internet are for some older version so it‚Äôs\ndifficult to search for help even though the exercise is technically correct</li>\n<li>The community around the tech used in the exercise is hostile/unhelpful</li>\n</ul>\n\n<p>One obvious way to accomplish this is by letting people use the programming\nlanguage they‚Äôre most comfortable in, because they probably already know how to\nGoogle for help in that environment.</p>\n\n<h3>no time-consuming setup required</h3>\n\n<p>Installing software is boring, and a lot of programming projects require\ninstalling software! A few things that can go wrong with this (though there are a lot\nmore than this!)</p>\n\n<ul>\n<li>I get a compiler error when I try to install this package on my computer</li>\n<li>The example actually requires some very specific package versions to work\nproperly and if you don‚Äôt have those exact versions installed you get a bunch\nof cryptic errors and need to google for 3 hours to fix them</li>\n</ul>\n\n<p>This kind of thing is a huge waste of time and super demoralizing. And it‚Äôs not\ntrivial to avoid! If you‚Äôre trying to teach someone a specific piece of\nsoftware, often that software</p>\n\n<p>A few options I‚Äôve seen or used to manage this:</p>\n\n<ul>\n<li>tell people what you know works (‚ÄúI‚Äôve tested this in Mac/Linux but not Windows‚Äù)</li>\n<li>avoid requiring any software to be installed (‚Äújust use python‚Äù)</li>\n<li>use Docker to run everything</li>\n<li>run all the code in the person‚Äôs browser (because browsers usually do about the same thing)</li>\n<li>use a cloud system (so everything runs on someone else‚Äôs computer). This is what I do for my <a href=\"https://github.com/jvns/pandas-cookbook\" rel=\"noopener noreferrer\">pandas cookbook</a>, which lets you run it in Binder, this really great free service for hosting Jupyter notebooks.</li>\n</ul>\n\n<h3>it‚Äôs easy to extend</h3>\n\n<p>@tef has this great talk on Scratch <a href=\"https://www.youtube.com/watch?v=vU9myNJI9l4\" rel=\"noopener noreferrer\">A million things to do with a computer!</a> which explains the 3 ideas of Scratch:</p>\n\n<ul>\n<li>low floors</li>\n<li>wide walls</li>\n<li>high ceilings</li>\n</ul>\n\n<p>It sucks when you start learning something and then learn that what you can do\nwith the Thing is very limited! It‚Äôs exciting when you learn something and see\n‚Äúoh, wow, there are SO MANY POSSIBILITIES, what if I did X instead?‚Äù</p>\n\n<h3>that‚Äôs a lot of things!</h3>\n\n<p>The criteria we arrived at:</p>\n\n<ul>\n<li>fun exercises</li>\n<li>that teach you something you care about</li>\n<li>that are challenging</li>\n<li>with clear success criteria</li>\n<li>that can be done quickly</li>\n<li>with no complicated setup</li>\n<li>and few hidden gotchas</li>\n<li>using a tech stack that‚Äôs easy for you to get help with</li>\n<li>where there‚Äôs a lot of room to grow</li>\n</ul>\n\n<p>That seems pretty hard, but it seems like a good goal to aspire to! I‚Äôm going\nto keep very slowly working on exercises!</p>\n",
        ),
        media_type: None,
        media_link: None,
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/11/25/challenge--make-a-bouncy-window-manager/",
        link: "https://jvns.ca/blog/2019/11/25/challenge--make-a-bouncy-window-manager/",
        title: "Challenge: Write a bouncy window manager",
        date: 2019-11-25T18:03:40Z,
        summary: None,
        content: Some(
            "\n\n<p>Hello! I‚Äôm writing a short series of programming challenges with\n<a href=\"http://www.cipht.net/2017/10/03/are-jump-tables-always-fastest.html\" rel=\"noopener noreferrer\">Julian</a>,\nand this is the first one!</p>\n\n<h3>the challenge</h3>\n\n<p><img src=\"https://jvns.ca/images/bouncewm.gif\"></p>\n\n<p><strong>requirements</strong></p>\n\n<p>The goal here is to make a very silly Linux window manager that bounces its windows\naround the screen, like in the gif above.</p>\n\n<p><strong>anti-requirements</strong></p>\n\n<p>The window manager doesn‚Äôt need to do anything else! It doesn‚Äôt need to support:</p>\n\n<ul>\n<li>moving or resizing windows</li>\n<li>switching between windows</li>\n<li>minimizing windows</li>\n<li>literally any of the other things you might normally expect a window manager to do</li>\n</ul>\n\n<p>It turns out implementing this kind of toy window manager is surprisingly approachable!</p>\n\n<h3>the setup: start with tinywm</h3>\n\n<p>All the instructions here only work on Linux (since this is about writing a Linux window manager).</p>\n\n<p><strong>starter kit: tinywm</strong></p>\n\n<p>Writing a window manager from scratch seems intimidating (at first I didn‚Äôt\neven know how to start!). But then I found\n<strong><a href=\"http://incise.org/tinywm.html\" rel=\"noopener noreferrer\">tinywm</a></strong>, which is a tiny window manager\nwritten in only <strong>50 lines of C</strong>. This is a GREAT starting point and there‚Äôs\nan annotated version of the source code which explains a lot of the details.\nThere‚Äôs a Python version of tinywm too, but I wasn‚Äôt able to get it to work.</p>\n\n<p>I did this challenge by modifying <a href=\"http://incise.org/tinywm.html\" rel=\"noopener noreferrer\">tinywm</a> and it worked really well.</p>\n\n<p><strong>tools</strong></p>\n\n<ul>\n<li><strong>Xephyr</strong> lets you embed an X session in a window in your regular desktop, so that you can develop your toy window manager without breaking your usual desktop. I ran it like this: <code>Xephyr -ac -screen 1280x1024 -br -reset -terminate 2&gt; /dev/null :1 &amp;</code></li>\n<li>You can start an xterm in the Xephyr desktop with <code>xterm -display :1</code></li>\n<li>I compiled my window manager with <code>gcc bouncewm.c -g -o bouncewm -lX11</code> and ran it with <code>env DISPLAY=:1 ./bouncewm</code></li>\n<li><strong>xtrace</strong> lets you trace all requests to the X windows system that your window manager is making. I found it really helpful when debugging. (run it like <code>xtrace ./bouncewm</code>)</li>\n</ul>\n\n<p><strong>documentation</strong></p>\n\n<p>Some useful references:</p>\n\n<ul>\n<li>the <a href=\"https://git.suckless.org/dwm/file/dwm.c.html\" rel=\"noopener noreferrer\">dwm source code</a> (dwm is a 2000-line-of-C window manager)</li>\n<li>the <a href=\"https://tronche.com/gui/x/xlib/\" rel=\"noopener noreferrer\">Xlib programming manual</a></li>\n</ul>\n\n<p>If you‚Äôre not comfortable writing C, there are also libraries that let you work\nwith X in other languages. I personally found C easier to use because a lot of\nthe window manager documentation and examples I found were for the Xlib C library.</p>\n\n<h3>my experience: 5 hours, 50 lines of code</h3>\n\n<p>To give you a very rough idea of the difficulty of this exercise: I did this in\n4 or 5 hours this morning and last night, producing the window manager you see\nin the gif at the top of the blog post (which is 50 lines of code). I‚Äôd never\nlooked at the source code for a window manager before yesterday.</p>\n\n<p>As usual when working with a new library I spent most of that time being\nconfused about various basic things about how X works. (and as a result I\nlearned several new things about X!)</p>\n\n<p>For me this challenge was a fun way to:</p>\n\n<ul>\n<li>learn some basics about the X window system protocol (I‚Äôve been using window managers for 15 years, today I got to write one!)</li>\n<li>research an unfamiliar library (‚Äúooh, what does this function do?‚Äù)</li>\n<li>use a C library, since I don‚Äôt usually write C</li>\n</ul>\n\n<h3>send me your solution if you do this!</h3>\n\n<p>I‚Äôll post the solution I came up in a week. If you think this window manager\nchallenge sounds fun and end up doing it, I‚Äôd love it if you sent me your\nsolution (to julia@jvns.ca)!</p>\n\n<p>I‚Äôd be delighted to post any solutions you send me in the solutions blog post.</p>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/bouncewm.gif",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/2019/12/03/solutions-to-the-tiny-window-manager-challenge/",
        link: "https://jvns.ca/blog/2019/12/03/solutions-to-the-tiny-window-manager-challenge/",
        title: "Solutions to the tiny window manager challenge",
        date: 2019-12-03T14:57:10Z,
        summary: None,
        content: Some(
            "\n\n<p>Hello! Last week I posted a small <a href=\"https://jvns.ca/blog/2019/11/25/challenge--make-a-bouncy-window-manager/\" rel=\"noopener noreferrer\">programming challenge to write a tiny window manager that bounces windows around the screen</a>.</p>\n\n<p><img src=\"https://jvns.ca/images/bouncewm.gif\"></p>\n\n<p>I‚Äôll write a bit about my experience of solving the challenge, or you can just\nskip to the end to see the solutions.</p>\n\n<h3>what‚Äôs a window manager?</h3>\n\n<p>An X window manager is a program that sends messages to the X server (which is\nin charge of drawing your windows) to tell it which windows to display and\nwhere.</p>\n\n<p>I found out that you can trace those events with <code>xtrace</code>. Here‚Äôs some example\noutput from xtrace (for the toy window manager which is just moving windows\nabout)</p>\n\n<pre><code>000:&lt;:02d8: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=560 y=8}\n000:&lt;:02da: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=554 y=12}\n000:&lt;:02dc: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=548 y=16}\n000:&lt;:02de: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=542 y=20}\n000:&lt;:02e0: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=536 y=24}\n000:&lt;:02e2: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=530 y=28}\n000:&lt;:02e4: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=524 y=32}\n</code></pre>\n\n<h3>you can run programs without a window manager</h3>\n\n<p>You technically don‚Äôt <em>need</em> a window manager to run graphical programs ‚Äì if\nyou want to start an xterm in a window-manager-less X session you can just run</p>\n\n<pre><code>xterm -display :1\n</code></pre>\n\n<p>and it‚Äôll start the xterm. Here‚Äôs a screenshot of an X session with no window\nmanager open. I even have 2 windows open! (chrome and an xterm). It has some\nmajor usability problems, for example I don‚Äôt think you can resize or move or\nswitch between windows. Which is where the window manager comes in!</p>\n\n<div>\n<a href=\"https://jvns.ca/images/no-wm.png\" rel=\"noopener noreferrer\"><img src=\"https://jvns.ca/images/no-wm.png\" width=\"300px\"></a>\n</div>\n\n<h3>move a window with XMoveWindow</h3>\n\n<p>The challenge was to make the window bounce around the screen.</p>\n\n<p>In the <a href=\"http://incise.org/tinywm.html\" rel=\"noopener noreferrer\">tinywm source</a> they use <code>XMoveResizeWindow</code> to move and resize windows,\nbut I found in the <a href=\"https://tronche.com/gui/x/xlib/window/XMoveWindow.html\" rel=\"noopener noreferrer\">docs</a> that there‚Äôs\nalso a function called <code>XMoveWindow</code>. Perfect!</p>\n\n<p>Here‚Äôs what it looks like. What could be simpler, right? And it works just the way I‚Äôd expect!</p>\n\n<pre><code>XMoveWindow(display, windowID, x, y)\n</code></pre>\n\n<p>Except‚Ä¶</p>\n\n<h3>problem: multiple <code>XMoveWindow</code>s don‚Äôt work</h3>\n\n<p>I ran into a problem (which I got stuck on for a couple of hours) where when I\nran XMoveWindow twice, it would only apply the last move.</p>\n\n<pre><code>XMoveWindow(display, windowID, 100, 200)\nusleep(2000 * 1000); # sleep for 2 seconds\nXMoveWindow(display, windowID, 300, 400)\n</code></pre>\n\n<p>I‚Äôd expect this to move the window once, wait 2 seconds, and them move it\nagain. But that was not what happened! Instead, it would pause for 2 seconds\nand then move the window once (to the second location).</p>\n\n<h3>use xtrace to trace window manager events</h3>\n\n<p>I used xtrace to trace the events and found out that my <code>ConfigureWindow</code>\nevents that <code>XMoveWindow</code> was sending were all being sent at the same time. So\nit seemed like X was batching the events. But why?</p>\n\n<h3>XSync forces X to process events</h3>\n\n<p>I didn‚Äôt know why this was happening, but I emailed Julian about it and he\npointed me in the direction of\n<a href=\"https://tronche.com/gui/x/xlib/event-handling/XSync.html\" rel=\"noopener noreferrer\">XSync</a>, which forces\nX to process all the events you‚Äôve sent it. Sure enough, I used XSync and\neverything worked beautifully.</p>\n\n<h3>solutions</h3>\n\n<p>I asked people to email me if they completed the challenge, and 4 people did!\nHere are their solutions. All the solutions I got implemented more features\nthan I did, so I‚Äôd encourage you to look at all the solutions if you‚Äôre\ninterested in how to solve this problem!</p>\n\n<ul>\n<li><a href=\"https://gist.github.com/jvns/d5a0a4daf300f3dd7fa76d13b5aa2d53\" rel=\"noopener noreferrer\">Kacper S≈Çomi≈Ñski‚Äôs solution</a> (which uses <code>XQueryTree</code> to find the windows to bounce, which is nice)</li>\n<li><a href=\"https://github.com/whichxjy/bounce-wm/blob/master/bounce-wm.c\" rel=\"noopener noreferrer\">@whichxyj‚Äôs solution</a></li>\n<li><a href=\"https://github.com/ayzenquwe/stressfulwm\" rel=\"noopener noreferrer\">Alexsey Lagoshin‚Äôs stressfulwm</a>, which allows bouncing multiple windows:</li>\n<li><a href=\"https://github.com/aldrinmartoq/bouncywm-ruby\" rel=\"noopener noreferrer\">Aldrin Martoq Ahumada‚Äôs bouncywm-ruby</a>, which is the only solution in a language other than C I got! It uses an Xlib Ruby library that looks pretty straightforward to use.</li>\n<li>one really nice one with fancier bouncing effects which I‚Äôll post here later if the person sends me the source</li>\n<li><a href=\"https://gist.github.com/jvns/c7a297fc4e17e797fd7b76b68860e55c\" rel=\"noopener noreferrer\">my solution</a></li>\n</ul>\n\n<p>Here‚Äôs a gif of Alexsey‚Äôs solution. Apparently <code>XQuartz</code> on a Mac performs better than Xephyr!</p>\n\n<p><img src=\"https://raw.githubusercontent.com/ayzenquwe/stressfulwm/d06531d286a5f00424bf12f7c77b18e11437ff20/gif/example.gif\"></p>\n\n<p>And Aldrin‚Äôs solution, with a great use of <code>xeyes</code>:</p>\n\n<p><img src=\"https://raw.githubusercontent.com/aldrinmartoq/bouncywm-ruby/f6d424b6107c1349c8ee338b6a46c7116c6d1ea7/demo/demo.gif\"></p>\n\n<h3>thanks!</h3>\n\n<p>Thanks to everyone who emailed me a solution, and if you write your own\nimplementation I‚Äôd love to post it here too, especially if you write one that\nisn‚Äôt in C or Ruby! I‚Äôm julia@jvns.ca. (and if you write a solution but don‚Äôt want me to post it I‚Äôd still love to see it!)</p>\n",
        ),
        media_type: Some(
            MediaType {
                mime: "image/*",
            },
        ),
        media_link: Some(
            "https://jvns.ca/images/bouncewm.gif",
        ),
        comments_link: None,
    },
    Post {
        id: -1,
        unique_id: "https://jvns.ca/blog/twitter-memes-challenge/",
        link: "https://jvns.ca/blog/twitter-memes-challenge/",
        title: "Challenge: find Twitter memes with suffix arrays",
        date: 2019-12-05T10:41:54Z,
        summary: None,
        content: Some(
            "\n\n<p>This challenge is a mix of data analysis and using fun algorithms! It‚Äôs the second challenge in a a short series of programming challenge I‚Äôm writing with Julian. (the <a href=\"https://jvns.ca/blog/2019/11/25/challenge--make-a-bouncy-window-manager/\" rel=\"noopener noreferrer\">first one was to write a tiny fun window manager</a>)</p>\n\n<p>Twitter has a lot of memes. For example, if you search Twitter for <a href=\"https://twitter.com/search?q=%22attendant%3A%20is%20there%20a%20doctor%20on%20%22&amp;src=typed_query&amp;f=live\" rel=\"noopener noreferrer\">Flight attendant: is there a doctor on this flight?</a>,\nyou‚Äôll find a bunch of tweets making jokes like this:</p>\n\n<pre><code>Flight Attendant: is there a doctor on board?\nParent: *nudging* That should\'ve been you\nMe: Not now, this is serious\nParent: Not asking for a hacker to help, are they?\nMe: AAAAAAAA\\x00\\xd0X?\\xfc\\x7fBBBBj\\x0bX\\x99Rfh-p\\x89\\xe1Rjhh/bash/bin\\x89\\xe3RQS\\x89\\xe1\\xcd\\x80\nParent:~#\n</code></pre>\n\n<p>or if you search <a href=\"https://twitter.com/search?q=%22as%20a%20kpop%20fan%22&amp;src=typed_query\" rel=\"noopener noreferrer\">as a kpop fan</a> there are thousands of these:</p>\n\n<pre><code>me as a kpop fan \n\n- kpop fan age: 10 years\n- first group ever stan: super junior\n- current ult groups: iKON, X1, Day6\n- number of albums: &gt;20 \n- concerts attended: 6\n- lightsticks owned: 2\n</code></pre>\n\n<p>So! Suppose you have a million tweets from the last 2 days. How do you find\nthe jokes / quizzes / memes people are playing with on Twitter?</p>\n\n<h3>Challenge: find the twitter memes in 1 million tweets</h3>\n\n<p>This is a pretty open ended challenge and you can do it any way you want.\nHere‚Äôs a <a href=\"https://www.dropbox.com/s/5xnaf1mbdr1424p/twitter.db.gz?dl=0\" rel=\"noopener noreferrer\">SQLite database with 1.2 million tweets</a>, collected\nfrom the <a href=\"https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data\" rel=\"noopener noreferrer\">twitter streaming api</a> over 2 days.\nIt‚Äôs 250MB (70MB compressed), it only has English tweets. It excludes retweets and many tweets that are generated by bots.</p>\n\n<p>The challenge: find at least 5 Twitter memes using that dataset.</p>\n\n<h3>memes as common substrings</h3>\n\n<p>The idea here is that memes are substrings like ‚Äúme as a kpop fan‚Äù that many\ndifferent people are using. The tricky thing is that you don‚Äôt really know how\nlong those substrings will be, and maybe you‚Äôre interested in phrases of\ndifferent lengths.</p>\n\n<p>You can probably do this challenge without using anything fancy (with a hashmap\nof phrases or something) but I think it‚Äôs a nice opportunity to play with a fun\ndata structure: suffix arrays! So let‚Äôs talk about what those are.</p>\n\n<h3>suffix arrays: sort all suffixes</h3>\n\n<p>Suffix arrays sort all suffixes of a string. For example, here‚Äôs the suffix array for ‚Äúplantain‚Äù which has the suffixes plantain, lantain, antain, ntain, tain, ain, in, n.</p>\n\n<pre><code>ain\nantain\nin\nlantain\nn\nntain\nplantain\ntain\n</code></pre>\n\n<p>Representing this as a list of strings would be very inefficient (quadratic space), so instead we\nreplace each suffix with the index of its first character in the original\nstring ‚Äì <code>[5,2,6,1,7,3,0,4]</code>.</p>\n\n<pre><code>5 (ain)\n2 (antain)\n6 (in)\n1 (lantain)\n7 (n)\n3 (ntain)\n0 (plantain)\n4 (tain)\n</code></pre>\n\n<p>Here‚Äôs a real example of what a suffix array of 1 million tweets concatenated\nlooks like. This is an excerpt from the middle of the suffix array, with\nsome of the suffixes that start with <code>A little</code>.</p>\n\n<pre><code>...\n A little distracted for a bit ...what do i do w my life hon.........\n A little exercise I did this afternoon.  #comics #art #clip.........\n A little extra Christmas Cash on me! Good Luck to everyone!.........\n A little girl in Savannah, Ga., appears to be the 38th huma.........\n A little heavy on the smut t‚Ä¶ https://t.co/nvoxE7SNjTI wa.........\n A little in state battle tonight. #nova vs #penn. two very .........\n A little kiss...‚Äù one more time I‚Äôm going to vomit. #TT.........\n A little late catching up on last nights @GoodDoctorABC. On.........\n A little less bling never hurt anyone! Next project...üéÑ .........\n A little more intensity to augment their talent and a coupl.........\n A little more time, because I have never lived really  - Os.........\n A little mor‚Ä¶ https://t.co/kcq3zf9jgeWe love MX ‚ù§\u{fe0f}&lt;F0&gt;&lt;9F&gt;&lt;A7&gt;.........\n A little over 50k! Can We Guess How Much Is In Your Account.........\n A little ray of joy &amp;amp; light in the midst of these very .........\n A little refreshment‚Ä¶ https://t.co/HgX8PmYwPIThank you @L.........\n A little respect goes a long way. .........\n A little salt in d country\'s troubled legal system‚ÄúGrant................\n A little snow &amp;amp; people lose all common senseromantic st...............\n A little sun for the soul @realfreewebcams https://t.co/3CB...............\n A little sunkissed moment for y‚Äôall. ...............\n ....\n</code></pre>\n\n<p>Again, this is actually represented by a bunch of integer indexes into a\nconcatenated string of all the tweets, like <code>[18238223, 1921812, ...]</code> so it‚Äôs a\nLOT more memory efficient than actually repeating all those strings.</p>\n\n<h3>suffix arrays let you find common substrings!</h3>\n\n<p>So what does this have to do with Twitter memes? Well, we can basically</p>\n\n<ol>\n<li>concatenate all tweets into a big string</li>\n<li>make a suffix array of that string</li>\n<li>iterate through the suffix array and notice when you see a lot of repeated substrings, like here:</li>\n</ol>\n\n<pre><code>me as a kpop fan ‚ú®kpop fan age: 15 y/o ‚ú®first group ever stan: blackpink ‚ú®current ult groups: btxt ‚ú®number of albu‚Ä¶ https://t.co/24diHX9sLm\nme as a kpop fan ‚≠êk-pop fan age: 12 y/o ‚≠êfirst group ever stan: bts ‚≠êcurrent ult gps: bts and txt ‚≠ênumber of albu‚Ä¶ https://t.co/8R95roQXoE\nme as a kpop fan ‚≠êk-pop fan age: 14 y/o ‚≠êfirst group ever stan: girls generation ‚≠êcurrent ult gp: txt ‚≠ênumber of a‚Ä¶ https://t.co/010hLuJscF\nme as a kpop fan ‚≠êk-pop fan age: 14-16 y/o ‚≠êfirst group ever stan: bts ‚≠êcurrent ult gps: bts txt ‚≠ênumber of albums‚Ä¶ https://t.co/0fDcxZGRrh\nme as a kpop fan ‚≠êk-pop fan age: 15 y/o ‚≠êfirst group ever stan: blackpink ‚≠êcurrent ult gps: txt ‚≠ênumber of albums‚Ä¶ https://t.co/d8zZL83TvV\nme as a kpop fan üå∏ k-pop fan age: 12 years old üå∏ first group ever stan: bts üå∏ current ult gps: bts &amp;amp; wanna one üå∏ n‚Ä¶ https://t.co/22R1nJpwNX\nme as a kpop fan üå∏k-pop fan age: 10 üå∏first group ever stan: 2pm üå∏current ult gps: skz,got7,itzy,twice, üå∏number of‚Ä¶ https://t.co/mAluaP2yxH\nme as a kpop fan üå∏k-pop fan age: 11 yo üå∏first group ever stan: beast üå∏current ult gps: ateez üå∏number of albums:  1‚Ä¶ https://t.co/qxtFHG9HDg\nme as a kpop fan üå∏k-pop fan age: 11 üå∏first group ever stan: bts üå∏current ult gps: bts and ateez üå∏number of albums:‚Ä¶ https://t.co/mKXlkrBBtC\nme as a kpop fan üå∏k-pop fan age: 13 (now im 19) üå∏first group ever stan: snsd üå∏current ult gps: nct day6 aoa mamam‚Ä¶ https://t.co/8XyQ3r5hwz\nme as a kpop fan üå∏k-pop fan age: 13 years üå∏first group ever stan: 2pm,suju,bigbang üå∏current ult gps: bts,tbz,ateez‚Ä¶ https://t.co/Zs1nQQz6Lt\nme as a kpop fan üå∏k-pop fan age: 14 (2005) üå∏first group ever stan: super junior üå∏current ult gps: exo, gfriend, rv‚Ä¶ https://t.co/vgmhe2vFMY\nme as a kpop fan üå∏k-pop fan age: 14 y/o üå∏first group ever stan: nct dream üå∏current ult gps: svt and,,*insert stan‚Ä¶ https://t.co/I38Ui69PvL\nme as a kpop fan üå∏k-pop fan age: 15 y/o üå∏first group ever stan: 5sos üå∏current ult gps: bts and 5sos also some ggs‚Ä¶ https://t.co/61ZmRkzmdl\nme as a kpop fan üå∏k-pop fan age: 15 y/o üå∏first group ever stan: bts üå∏current ult gps: SVT, GOT7, Day6 üå∏number of‚Ä¶ https://t.co/16SWb3mSPg\nme as a kpop fan üå∏k-pop fan age: 18 üå∏first group ever stan: suju &amp;amp; soshi üå∏current ult gps: snsd &amp;amp; izone üå∏number of‚Ä¶ https://t.co/SmSBFqJnGk\nme as a kpop fan üå∏k-pop fan age: 19 y/o marupok üå∏first group ever stan: APINK üå∏current ult gps: SEVENTEEN üå∏number‚Ä¶ https://t.co/StYjxr6uq9\nme as a kpop fan üå∏k-pop fan age: 19 üå∏first group ever stan: SuJu üå∏current ult gps: SuJu, SF9, SKZ, VIXX, ONEUS, NO‚Ä¶ https://t.co/2o2DulCY5b\n</code></pre>\n\n<h3>suffix arrays also enable really fast search</h3>\n\n<p>As an aside, the reason I got interested in suffix arrays in the first place\nwas actually not for finding Twitter memes at all but for search.</p>\n\n<p>I‚Äôve spent a lot of time using Nelson Elhage‚Äôs <a href=\"https://livegrep.com/search/linux\" rel=\"noopener noreferrer\">livegrep</a> at work to search\ncode. It creates a suffix array using the divsufsort library. He has a blog post\n<a href=\"https://blog.nelhage.com/2015/02/regular-expression-search-with-suffix-arrays/\" rel=\"noopener noreferrer\">Regular Expression Search with Suffix Arrays</a>\nwhere he talks about some of the implementation details.</p>\n\n<p>The reason suffix arrays work for fast search is basically that if you‚Äôre\nlooking for the string <code>A little</code>, you can do a binary search over the suffix array to\nfind every instance of <code>A little</code> in your dataset.  Binary searches are\nextremely fast so every search is guaranteed to run very quickly (in less than\na microsecond I believe). What livegrep does is more complicated than that\nbecause it does a regular expression search, but that‚Äôs the idea to start.</p>\n\n<p>There‚Äôs another blog post <a href=\"https://blog.griddynamics.com/using-suffix-arrays-to-fix-limitations-of-full-text-search/\" rel=\"noopener noreferrer\">How to use suffix arrays to combat common limitations of full-text search</a> applying suffix arrays to searching through a patent database. In that example, like with code search, the patent officers want to search patents for exact strings.</p>\n\n<h3>How do you make a suffix array?</h3>\n\n<p>You can use an existing suffix array library, for example <a href=\"https://golang.org/pkg/index/suffixarray/\" rel=\"noopener noreferrer\">index/suffixarray in\nGo</a>, which is what I used, or\n<a href=\"https://github.com/y-256/libdivsufsort\" rel=\"noopener noreferrer\">divsufsort</a>. There are <a href=\"https://github.com/debatem1/pydivsufsort\" rel=\"noopener noreferrer\">Python\nbindings for divsufsort</a>.</p>\n\n<p>If you‚Äôre more excited about the data structures/algorithms aspect of suffix\narrays you can also implement a suffix array-building algorithm yourself! I did\nnot do this but you can see an implementation of <a href=\"https://github.com/golang/go/blob/release-branch.go1.12/src/index/suffixarray/qsufsort.go\" rel=\"noopener noreferrer\">qsufsort here in Go</a>.\nThat implementation links to a paper. There are lots of algorithms for constructing suffix arrays ‚Äì<code>sais</code> and\n<code>divsufsort</code> are a couple of others.</p>\n\n<h3>5 or so hours, 100 lines of Go</h3>\n\n<p>As always with these challenges, I did this one to make sure that it‚Äôs both\ndoable in a reasonable amount of time and fun for at least one person (me).</p>\n\n<p>I did this one in about 5 hours and 100 lines of Go using the suffixarray\nimplementation in the Go standard library, with a bit of bash shell scripting\nto postprocess the results. This is a messy data analysis challenge ‚Äì as an\nexample of a messy thing, Spotify released their end-of-2019 results while I\nwas building the dataset and so there are a lot of tweets generated by the\nSpotify app.</p>\n\n<p>My results ended up looking something like this:</p>\n\n<pre><code>5  an Aries and that‚Äôs why I gotta \n5  an Aries and that‚Äôs why I am so \n5  an Aquarius and that‚Äôs why I \n5  AM SO PROUD OF YOU \n5  am not a fan of \n5  am I the only one who \n5  am going to have to \n</code></pre>\n\n<p>Then I sifted through them pretty manually to find the Twitter memes.</p>\n\n<h3>suffix arrays are used in bioinformatics</h3>\n\n<p>This ‚Äúfind twitter memes using suffix arrays‚Äù approach is a silly thing but it\ndoes have some relationship to reality ‚Äì DNA sequences are basically really\nlong strings, and biologists need to find patterns in them, and they sometimes\nuse suffix arrays to do it.</p>\n\n<p>I looked up <a href=\"https://codesearch.debian.net/search?q=divsufsort\" rel=\"noopener noreferrer\">packages in Debian that use libdivsufsort</a> and I found <a href=\"http://eddylab.org/infernal/\" rel=\"noopener noreferrer\">infernal</a>:</p>\n\n<blockquote>\n<p>Infernal (‚ÄúINFERence of RNA ALignment‚Äù) is for searching DNA sequence\n databases for RNA structure and sequence similarities. It is an\n implementation of a special case of profile stochastic context-free grammars\n called covariance models (CMs). A CM is like a sequence profile, but it\n scores a combination of sequence consensus and RNA secondary structure\n consensus, so in many cases, it is more capable of identifying RNA homologs\n that conserve their secondary structure more than their primary sequence.</p>\n</blockquote>\n\n<h3>email me the twitter memes you find if you do this!</h3>\n\n<p>If you do this exercise, I‚Äôd love it if you emailed me (julia@jvns.ca) with the twitter memes\nyou found and/or your code! I found about 8 but I‚Äôm sure there are more.</p>\n\n<p>I‚Äôll publish any solutions I get (unless you don‚Äôt want me to publish your solution ‚Äì just let me know!).</p>\n\n<p>Thanks to Julian for discussing suffix arrays and suffix trees and trigram indexes with me at length, and to Kamal who had the idea of using suffix arrays to find Twitter memes.</p>\n",
        ),
        media_type: None,
        media_link: None,
        comments_link: None,
    },
]
