The TTY demystified - <AUTHOR> http://www.linusakesson.net/programming/tty/index.php <comment><USER2> Even after reading this, TTYs will surprise you. I've done my share of TTY programming in Linux and here be dragons.

From modes to tty types to control chars and signals, the TTY system is itself a bloated relic whose stink becomes apparent when you try to connect it to the rest of the unix philosophy, where everything is supposed to be a stateless text stream.

This stuff is really interesting for historical reasons and made sense 20 years ago, but I long for the day where the TTY system is gone and character drivers work on some standard JSON stream or something.

<comment><USER3> meh. TTY programming was hard in the 90's, and then it was done. If you find it hard now, you're probably still just learning all about it. Its an old, well and truly solved problem. The slayers of yesterdays TTY dragons are todays WebSocket hackers, I surmise.

Still, I think the TTY - and UARTS in general - have a lot of utility in this day and age, warty dragons and all. I've got a terminal server in my lab just for the case of wiring up the small, light, cheap, efficient embedded systems which still depend on the dragons to deliver the fire. Nothing quite like having a functional 9600-baud term you can tap into with an Oscilloscope and work out what the bits mean .. try doing that with a JSON-based interface. ;)

<comment><USER4> I disagree, it is still hard: [https://bugs.freedesktop.org/show_bug.cgi?id=70290](https://bugs.freedesktop.org/show_bug.cgi?id=70290)

Further, there are still things for which there is not a right way to do it. If you look at the man page for "unbuffer" (part of expect), there's a "CAVEATS" section that describes a bug... that can't be fixed while PTYs are in their current state. I wouldn't call it "well and truly solved".

</comment></comment></comment><comment><USER5> Now I feel like I want to read a similar article describing how the Plan 9 terminal interface works, just to cleanse my palate.

</comment><comment><USER6> I learned something new from this: I was always under the impression that you could not do anything when your process is stopped via ^Z. But apparently there are two separate signals involved here: SIGTSTP (sent when ^Z is pressed and blockable) and SIGSTOP (also suspending the process but NOT blockable) (This reminds of the time SIGCLD / SIGCHLD were different).

I have a system where shell users run commands that can grab some locks, and they like to press ^Z to suspend them which isn't compatible with a simple fcntl-based lock.

<comment><USER7> vim and emacs both trap Ctrl-Z in the terminal and implement custom behavior. If they didn't, I think, you would not be dropped gracefully into a shell prompt after pressing Ctrl-Z; instead, you'd be left with the terminal in non- canonical mode with the vim or emacs process suspended, and there would be nothing graceful about that.

In your case, there should be a way to trap attempted Ctrl-Z suspends in your process and break the locks (or do whatever you need to do).

</comment></comment><comment><USER8> I have read a great book covering TTY program on ux machines, it is also great for getting an understanding on how it all works. The book title is : "Advanced Programming in the Unix Environment" By W. Richard Stevens and Stephen A. Rago

It gives a thorough walk through of the ttys and pty's, IMHO it is a must read for anybody wanting do to TTY programming, and it is also a great source for understanding how the streams works in connection with the tty's/pty's.

</comment><comment><USER9> > Line editing...a backspace key is often useful. This could of course be > implemented by the applications themselves, but in accordance with the UNIX > design philosophy, applications should be kept as simple as possible. So as > a convenience, the operating system provides an editing buffer

Actually this is backwards.

Line discipline is a historical artifact. Typically mainframes had IO front ends (AKA channel controllers) than managed IO so that the CPU (which BTW was what was charged for) could be kept busy and not wait around for users (by contrast CTSS, ITS and later machines like the Alto expressed an alternative, radical idea which was that the computer was there to support the user). Basically once a line was ready it was submitted by pressing the enter key and then its process was unblocked and processed the data. (Nowadays we just queue an event on the main thread...essentially the same thing!).

Likewise, a number of early network protocols like X.25 would charge per "line" (more about that later).

Unix was originally written for the PDP-7 (& later 11) which were minicomputers that didn't have channel controllers (in fact the earliest arpanet interfaces that weren't IMPS were PDP-11s acting as networking channel controllers for PDP-10 mainframes). Since IO was done by the kernel, you didn't want to to an expensive context switch for the common case of just reading a character and putting it into an input buffer.

So despite the comment quoted above, the true Unix model would have been for the kernel to have been agnostic and for each program to decide how it wanted to handle character input. But that would have been too expensive, so a more pragmatic approach prevailed.

By the way there are all sorts of other fossils in the TTY system. Multics used # to delete a character and @ to rub out the whole line -- a legacy of printing characters that persisted into Unix for years ###### decades. This seems weird today, but originally Multics (and later Unix) didn't have traditional line disciplines, or even control characters, much less select(). So it was handy to be able to essentially do immediate mode editing right in the input buffer.

Another fossil is that vi is just a visual mode grafted onto ed (clone of Multics qed IIRC), which was an editor with commands based on line-at-a-time IO.

ITS got proper IO virtualization first (and the predecessor to select, and extended it to the network with SUPDUP, later also implemented by Multics and eventually Unix, which made using Emacs possible) and actually had a more general version of select() before Unix was written, but at the time it was considered exotic.

Oh and the X.25 comment? Well in 1982 I was working in France and the only way for me and my colleagues who had come from MIT to get back to our mail at the MIT AI lab was a connect across the ocean on an international X.25 network hooked into MIT Multics and thence connect to the AI lab. After using fully interactive systems, it was painful to use a line-at-a-time system, so I figured out how to reset the X.25 front end to ship a whole packet for every character typed (packets had a maximum length so if you typed a really long line it would send a packet's worth, so I hacked it into thinking that a packet length was 1). Which was great, until shortly after the end of the month when someone "upstairs" had a heart attack at the size of the networking bill...

</comment><comment><USER7> I recently wanted to give a command-line program the ability to wait for and respond to keypresses instantly, without the user having to type Enter. It sounded simple enough. Here's what you have to do to set this up:

      // headers for terminal control
      #include <termios.h>
      #include <unistd.h>

      // global variable
      struct termios saved_attributes;

      // prototype of atexit callback
      void reset_input_mode (void);

      // --- inside main()

      // local variable
      struct termios termios_data;

      // tell stdio not to buffer output
      setvbuf(stdout, NULL, _IONBF, 0);

      // incantation to the terminal
      tcgetattr(0, &termios_data);
      saved_attributes = termios_data;
      termios_data.c_lflag &= ~(ECHO | ECHONL | ICANON | IEXTEN);
      termios_data.c_cc[VMIN] = 1;
      tcsetattr(0, TCSANOW, &termios_data);

      // set atexit callback so we can restore the prior terminal mode upon exit
      atexit(reset_input_mode);

      // --- outside main()

      // implementation of atexit callback -- restores terminal to prior mode
      void reset_input_mode (void)
      {
        tcsetattr (STDIN_FILENO, TCSANOW, &saved_attributes);
      }

Now, you can read input one byte at a time (with, e.g. getchar(3)) and output appears instantly.

</comment><comment><USER10> Uh! It makes me remember some obscure and mystic stuff for the IBM 3151 terminal [1]. This terminal has an auxiliary port where you can connect the printer, so it was a (not so) dumb terminal where you can connect other peripherals. There was a government project to attach an individual printer (instead of connecting it to the server) to some terminals but nobody knew the escape codes to control that port. At some point the IBM guys sent a difficult to find IBM 3151 with these details. Probably this manual is now already available on Internet.

[1] [http://ps-2.kev009.com/tl/techlib/manuals/adoclib/aixasync/a...](http://ps-2.kev009.com/tl/techlib/manuals/adoclib/aixasync/asycomgd/term3151.htm)

</comment><comment><USER11> Terminal.app and other terminal emulators are the prototypical case where a TTY -- more precisely, a TTY/PTY pair -- are required these days.

Every virtual terminal in the Linux text-mode console also requires a TTY/PTY pair.

Shell mode (and more generally any mode derived from Comint mode) in Emacs also requires a TTY/PTY pair: although the Emacs documentation claims that Comint-derived modes can be configured to use a pipe instead, a comment in comint.el confesses that no one has gotten that to actually work.

And of course, SSHing into a remote host requires at least one TTY/PTY pair (on the remote host).

Are there other place in the software on, e.g., a Mac reliant on a TTY? If there are, please tell me what they are!

<comment><USER4> Both `ssh` and `sudo` will only ask you for a password from a TTY; they'll refuse to work if they can't find a TTY (unless, of course you have a graphical prompt configured).

The `expect` suite of tools uses a TTY/PTY pair to script programs that might otherwise be unhappy with not having a PTY. Same with the `script` program, which on my system is part of the 'util-linux' package.

On GNU/Linux, `systemd-nspawn` (a chroot-like tool) uses a TTY/PTY pair to isolate the child process from the outside world.

</comment></comment><comment><USER12> This guy's other articles are gold too.

</comment><comment><USER4> It may be worth noting that both of the kernel source files mentioned have moved from `drivers/char/` to `drivers/tty/`. I _think_ that all of the other details are up to date, though.

</comment><comment><USER13> This was a great read, I hope it stays on the front page until morning so more people see it. It filled some gaps left from my Operating Systems class about TTY, especially the history with Teletypes.

</comment><comment><USER14> I wish he had said something about'^?' vs '^H'. To this day I still struggle sometimes when logged into a Unix machine and run into this problem.

<comment><USER15> ASCII DEL (^?) is natively a 'forward delete' operation; it erases the character under the cursor while advancing to the next character. That's why its value is 0x7F: on paper tape it's all holes, so it can punch over any other character.

The problem arose from two later, non-Unix (and arguably deliberately anti- Unix¹) influences.

One, as the Debian notes linked in a sibling comment mentions, was GNU Emacs, whose bindings came from custom non-ASCII, bucky-bit² devices, and gleefully stepped on ASCII control characters like Backspace (^H) and Device Control 3 (^S).

The other was DEC's new VT2x0 series of terminals, with the VMS-oriented LK201 keyboard³, that hid Backspace and Escape on unlabelled function keys (F12 and F11 respectively) and made the key in the typist's backspace position send DEL. (The VT100⁴ had had both Backspace, in the typist's location, and DEL, where US PC keyboards have \|.) This was the big one, as BSD 4.2 for the VAX spread and dragged the LK201 along with it.

¹ [http://richard.esplins.org/static/downloads/unix-haters- hand...](http://richard.esplins.org/static/downloads/unix-haters-handbook.pdf)

² [http://en.wikipedia.org/wiki/Bucky_bit](http://en.wikipedia.org/wiki/Bucky_bit)

³ [http://deskthority.net/wiki/DEC_LK201](http://deskthority.net/wiki/DEC_LK201)

⁴ [http://terminals.classiccmp.org/wiki/index.php/DEC_VT100](http://terminals.classiccmp.org/wiki/index.php/DEC_VT100)

</comment></comment><comment><USER16> All these years later and under Linux I still can't unambiguously read a UART line break condition....</comment>


Mobile Web is & will be slow - <AUTHOR> http://tapiphany.org/posts/mobile-web-is-will-be-slow

<comment><USER2> Where's your argument? Where is the analysis? Tests? Predictions? The future? Who was your audience and their device demographics?

Gartner gave a decent report and predictions for 2013. The W3C is now opening classes on Mobile Web.

You're being terribly disingenuous by preaching your faith about the mobile Web which implies foresight no further than next quarter.

People are paying money to learn these skills, and you're just echoing blatantly reactionary non-points.</comment>


Ask HN: Would a self-aware AI be able to spot a buffer overflow in itself? - <AUTHOR> <comment><USER2> Simple answer: no.

Unless you have some radical new processor design plans up your sleeve, your AI will run on a register machine. RMs can be shown to be equivalent to the Turing machine, the Turing machine cannot decide the halting problem (which your question is an instance of).</comment>


Creditor moves to dismantle copyright troll Righthaven - <AUTHOR> http://arstechnica.com/tech-policy/news/2011/10/creditor-moves-to-dismantle-copyright-troll-righthaven.ars <comment><USER2> Quoting:

A filing in the case shows that Randazza wants the court to "authorize the U.S. Marshalls to execute Hoehn's judgment through seizure of Righthaven's bank accounts, real and personal property, _and intangible intellectual property rights_ [my emphasis] for levy, lien, auction or other treatment appropriate for satisfaction of Hoehn's judgment."

I hope this isn't a spurious question, but could this end with _Hoehn_ owning copyrights entrusted to Righthaven?

<comment><USER3> Righthaven never had any copyrights to begin with. Which is why they keep getting absolutely slaughtered in cases (no standing).

So unfortunately Randazza won't get those copyrights.

<comment><USER4> I believe the rights being referred to are things like Righthaven's own name, trademarks, copyrights and so forth, not just what they may have acquired from other parties.

<comment><USER3> Probably.

Still, I can't imagine the trademark "Righthaven" has a lot of value any more.

</comment></comment></comment></comment><comment><USER5> Hilarious, but also sad. What's going to happen to the $120,000 payment handed down to the other folks, plus all other Righthaven suits still in court?

Does it end up being just $120,000 out of pocket to defend against a frivolous lawsuit, or can people go after the people behind Righthaven for this as well?

</comment><comment><USER6> "Righthaven had asked U.S. District Judge Philip Pro of Nevada to stay the fee award, saying it might slip into bankruptcy if forced to pay."

So fucking what?!

This must be about the dumbest, most ludicrous reasoning I have read in a long, long time.

So you frivolously try to screw people and companies with dodgy copyright lawsuits. After the bullets of your double barrel shotgun forcefully blasted into your feet you argue that you can't continue with your "business model" when forced to pay for the consequences?

Either something here is very much out of context or those Righthaven dudes are really not the smartest cookies in the jar.</comment>


Raspberry Pi-Alikes (2013) - <AUTHOR> http://pi.gate.ac.uk/pages/hardware.html

<comment><USER2> Note that this is a bit outdated. I was looking for the cubietruck, which is linked at the very bottom of the page - and is from 2013.

In either event, I have both the pi and the cubietruck. The pi is cheaper, smaller, and has much much(did I say much?) better documentation. The Cubietruck blows the pi away in features and processing power...but at the cost of really bad documentation.

<comment><USER3> Yeah I've noticed with a lot of the recent Pi competitors they're pushing out powerful stuff but failing (or choosing not) to realize the Pi is successful because of the huge community of support available. The Pi is definitely my suggestion for anyone new to small embedded Linux systems.</comment></comment>


Ask HN: What software/app do you wish would exist? - <AUTHOR> Looking for some ideas about some stuff that might already kind of exist, but could be improved on in terms of features/UX. Also, it could be something completely new. <comment><USER2> Something like [https://anichart.net/Fall-2019](https://anichart.net/Fall-2019) for western movies and TV series. What is coming out this year? What has come out in the past year? What was good in 2008. Far more titles, so filters and maybe recommendation engine important. Drive people to amazon, netflix signups etc for money.

A better way of discovering what book to read next. Amazon recommendations not browsable enough. Goodreads not assistive enough. Amazon recommendation dollars.

<comment><USER3> Does metacritic not work for this purpose for movies for you? It is what I use to do the things you list.

</comment></comment><comment><USER4> A tool that lets me do code reviews, but like really good and integrated. I use gitlab on a daily basis, but it's painstakingly slow and basically just lets me stare at text. Some of the below features may be available, but not satisfactory in my opinion:

\- I want to see the changes in a directory tree that I can navigate through!

\- I want to comment on any piece of code, not only changes and their surroundings!

\- I want to hide things, or mark them as "seen", to be able to look at things in the order I want without any risk of losing progress!

\- I want to be able to filter/sort by the "change type", e.g. Imports, renames, indentation changes etc.!

\- I want the diff to be aware of programming semantics, e.g. know when a function just moved from one file to another, or I add a method overload and don't get a diff telling me I added a closing bracket and another whole method signature in the middle of the old method!

\- Let me type code! I want to try out and propose changes!

\- I want freedom editing my review before submitting it the way I have freedom editing my branches before committing it!

\- I want to have single comments refer to ranges of changes, which aren't necessarily connected or in the same file!

\- I want to easily see what happened to the changes since I reviewed them! (Not a clickable link "changed since then bla bla", show me right there!).

\- And probably most important: It has to be fast! Hyperbole: Having to wait for the page to load for a minute for large diffs, and then having to wait another minute for all changes to _actually_ be visible is a no-go.

The available tooling is already very powerful, but it still frequently fails me and I believe having this be extremely polished would be a real productivity game changer. Maybe there are already some great tools? Last time I had to look over a huge amount of changes I resorted to a patch file; grepping away unnecessary changes like package renames etc and just deleting text lines I considered sufficiently reviewed.

<comment><USER5> delta-force.net does this I believe. Not free though!

</comment></comment><comment><USER6> Look for 1-star reviews of existing software, and make something that fixes those pain points.

<comment><USER7> I scraped Capterra awhile back with this intention. What I learned is that every business category (there are hundreds) had at least 3 market leaders with 4 or more stars.

That's partially due to Capterra being incentive on behalf on the ads they run to have positive reviews up, but it could indicate also that we've reached a saturation point where the CRM for X niche has been largely filled.

</comment></comment><comment><USER8> Simple to use command line utility to search for files and directories with matching names and/or content defined by strings or regex patterns.

All linux utils that do that require anal effort, comes with obscene syntax and demand obscure combination of multiple things to accomplish single task.

I always wanted to write my own one on python but never found time.

<comment><USER9> Have you tried fd yet? it's more intuitive than find.

[https://github.com/sharkdp/fd](https://github.com/sharkdp/fd)

</comment></comment><comment><USER10> A decent online diagram creator. One that handles the layout in an automatic way, making it easy to insert new intermediate nodes without dragging everything around manually. Creating a good architecture diagram is currently an exercise in tedium. Many tools don't even snap / route arrows well.

Would be great if it allows data entry via either text or graphical UI. Basically, a modern version of DOT and DOTTY.

<comment><USER11> Have you seen PlantUML [1]? You create your diagrams in text in their DSL which is fairly intuitive.

There's a hosted version of the rendering engine here [2] though it's easy enough to run locally.

[1] [http://plantuml.com/](http://plantuml.com/)

[2] [https://www.planttext.com/](https://www.planttext.com/)

</comment></comment><comment><USER12> Every modern office application as they currently are, only they save to an easy to parse text file.

I'm talking about AutoCAD, Ms Office, etc.

</comment><comment><USER13> A PDF spell checker for last-second resume sendoffs.

</comment><comment><USER14> A NLP text interface basically for everything I interact with either on a computer or my mobile. I doubt I'll ever use a voice based tool like Siri or its' equivalents, but I'd really like to type in commands I'd like to issue rather than click around an UI.

It feels error prone and isn't always very accessible as I'm motorically a bit challenged although definitely not disabled.

I simply adore applications that let me invoke commands through text like the command palette in vscode or Sketch Runner in Sketch. This is also why I'm a heavy CLI user and spend a lot of time customising my aliases and work flows. I'd like to extend this outside of the terminal.

<comment><USER7> You can set Siri on a Mac to accept text and turn off voice responses, then trigger it with a keyboard shortcut. Works pretty well and much faster than the voice controls.

</comment></comment><comment><USER5> An enhanced Activity Monitor for MacOS. Basically annotate all the processes so I can click on them and see where they are from, what they do, and whether or not I can kill them. WTH is "Dock" for and why is using so much CPU?

<comment><USER7> I would buy this.

<comment><USER5> Me too.

</comment></comment></comment><comment><USER15> Generator for answers to the question "What software/app do you wish would exist"

<comment><USER16> You will need a recursive name for it :-)

</comment></comment><comment><USER2> Oh, I got another one - Design your new kitchen and use AR to preview it.

Step 1: Measure your room by pointing the phone at the corners and triangulating the room. Can do iregularly shaped room. More pictures = more angles = floor plan through the power of trig.

Step 2: On your generated 2d floorplan place the kitchen units you want

Step 3: hold up your phone, point it at a corner again to triangulate location, generate 3D perspective view of the kitchen units / new floor / painted walls / lighting and put it ontop of the picture in the right place.

Make a rough prototype and then convince a major retailer to pay you to finish it without owning the IP.

</comment><comment><USER17> A cross-platform note taking app, with the ability to jot down an idea and later associate resources with it (links, media, other notes, etc.).

Idea + resources = Project (like an article or a book), and tags can help allocate the same resource to multiple projects.

It should allow the note taking to be very quick and fast, like writing a message.

And yeah, I've been trying a bunch of apps over the years. Evernote, the closest one, doesn't fit yet.

<comment><USER18> Have you tried EMACS orgmode ([https://orgmode.org](https://orgmode.org))? With all the goodies, such as org-capture, linking features and so on you can get great solution basically for free in terms of money. On the other hand it is quite time consuming investment, but hey, note taking system is for life if it's good, right?

Although the interface is text-based (I believe it's a pro not con) it is therefore quite clunky in terms of media integration (photos are good, but I don't know about the videos) the rest (links, associating other resources, tagging system, todos system, exporting, projects etc) is quite superior to everything I've ever seen. Also it's FOSS and therefore extremely expendable IF you are willing to learn a bit of elisp (although I don't know it and I'm fine using just pieces of code from the internet and Spacemacs).

</comment></comment><comment><USER19> Ms Access in a modern form

<comment><USER20> 1) That's far beyond the scope of a single person's side project

2) What isn't modern about Access? What do you want to change that isn't just removing features?

<comment><USER21> > What isn't modern about Access?

Almost everything.

Access does not have granular security, fault tolerance, auditing, or a couple dozen other features that one would expect in a modern database.

<comment><USER19> I wouldn't argue if anyone said access wasn't designed to do those things. Microsoft always has the next solution you should move to.

I have seen Access app developers run their access app on top of a MS Sql server to bring that kind of functionality in, though.

Access was less of a database and more "let's work with our data using forms and reports".

It was powerful in the hands of people who understood the data and details they work with day in and day out and let them form solutions for themselves when there was no other way.

In many enterprises access is just there like excel, but custom software is not permitted.

</comment></comment></comment></comment><comment><USER22> I would love if Python was built into Outlook and would let me write custom rules in Python to process mail.

<comment><USER23> What are some custom rules you would like to write in python ?

<comment><USER22> The options in Outlook are very limited.

Some that I encountered recently were:

You cannot use regular expressions.

There is not a way to create a rule to delete or move messages after a N days.

</comment></comment></comment><comment><USER24> I wish there were something like Visual Basic for the development of web applications.

<comment><USER25> We built one!

[https://anvil.works](https://anvil.works) is the tool. It is quite explicitly based on VB and other systems like it: it's got a drag-and-drop UI designer, backed by Python. Double-click a button, and edit the Python code that runs when it's clicked. You write Python on the front end (we transpile to JS), Python on the back end, and we host it for you with a click. Check it out!

<comment><USER26> Nice solution, but there's no way I would touch it unless it's open source. Any plan for that?

</comment></comment></comment><comment><USER27> A Wireshark plugin that tells you process name/id instead of just address/port. This would only work for local server or clients, not remote ones.

</comment><comment><USER28> An ToDo app which could save and sync with single text-like file. Yes, what I want is an cross platform, modern UI Emacs and `the` org-mode :)

<comment><USER29> What about Notable? [https://github.com/notable/notable](https://github.com/notable/notable)

</comment></comment><comment><USER30> I would like a mobile app that allows me to drag-and-drop a task from my daily todo list onto my daily schedule.

</comment><comment><USER27> Vim keybindings for file explorer(s).</comment>


Don't Slam The Door - <AUTHOR> http://blog.lexilewtan.com/dont-slam-the-door

<comment><USER2> As adults, do we really need to be reminded not to slam doors? It is essentially a tantrum, and while feeling frustrated or ridiculously annoyed is perfectly normal, it should be expected of us as adults to respond to these situations without blowing up. Whether in relationships or a job, 'slamming the door' is nearly guaranteed to make one look like the immature one in the conflict.

</comment><comment><AUTHOR> i'm talking to people who aren't quite adults yet, but who are beginning to craft their adult life through internships, relationships, and mentors.

<comment><USER3> I thought this was a great post. It's giving me pause to reconsider how I'm making my next career moves.</comment></comment>


Samsung Galaxy Z Fold 2 - <AUTHOR> https://en.wikipedia.org/wiki/Samsung_Galaxy_Z_Fold_2 <comment><USER2> I absolutely love my Flip and it's clamshell design (the Wikipedia seems to hint that the fold will be a clamshell design too)

The battery life is terrible. The camera is a joke. I've broken the screen in three separate places because it's super fragile.

But I genuinely love the thing - the aesthetics are gorgeous, and it gets out my way when I'm not using it, far less of a distraction than any other phone I've used.

I really hope to see more folding smart phones on the market - I also really hope Samsung make another phone like the Flip without having a massive screen on the front!</comment>


The Fearless Invention of One of L.A.'S Greatest Poets - <AUTHOR> https://www.newyorker.com/magazine/2020/05/18/the-fearless-invention-of-one-of-las-greatest-poets <comment><USER2> Some more on Wanda Coleman. I'm not very familiar with her work, but this post has caused me to look closer. Ty!

[https://www.poetryfoundation.org/poets/wanda- coleman](https://www.poetryfoundation.org/poets/wanda-coleman)</comment>


A Map of Wireless Passwords from Airports Around the World (Updated Regularly) - <AUTHOR> https://foxnomad.com/2016/04/26/map-wireless-passwords-airports-lounges-around-world-updated-regularly/ <comment><USER2> Just a thought: I wonder if using one of these networks without being a club member or ticketed in first class could get you in trouble via the Computer Fraud and Abuse Act? With all of the stories you hear of the CFAA being stretched, it makes me think.

</comment><comment><USER3> Direct link to the mentioned map: [https://www.google.com/maps/d/viewer?mid=1Z1dI8hoBZSJNWFx2xr...](https://www.google.com/maps/d/viewer?mid=1Z1dI8hoBZSJNWFx2xr_MMxSxSxY)

</comment><comment><USER4> Site seems to be down.</comment>


Give the Gift of Github - <AUTHOR> https://github.com/blog/763-give-the-gift-of-github

<comment><USER2> Just sent out the mail, I _love_ this idea. Really smart on the part of github: they understand that mixing free (for exposure) & paid/private (e.g. apps) projects is the future for lots of developers.

</comment><comment><USER3> I assumed this was going to be a way to pay for another users account. This is way better, though. Surprised there's no advertisement involved from the companies funding this.</comment>


Chan Zuckerberg Initiative commits to investing $3B to cure diseases - <AUTHOR> http://www.theverge.com/2016/9/21/13003174/chan-zuckerberg-initiative-commits-to-investing-3-billion-to-cure <comment><USER2> Live video from Priscilla & Mark: [https://www.facebook.com/zuck/videos/vb.4/10103120865133051/](https://www.facebook.com/zuck/videos/vb.4/10103120865133051/)

<comment><AUTHOR> Do these live feeds get archived, and if so are the archived versions available at the same URL?</comment></comment>


Ask HN: Apache cordova or what for mobile apps - <AUTHOR>

I want to build an app to target android first, then as many other platforms as quick to do thereafter.<p>So what tooling should I download to get a mobile app running by sunday afternoon? The app should be pretty, talk to a backend service and run on my android and if possible an ipad? <comment><USER2> I have recently started using Xamarin for a client who wanted apps for the major mobile platforms. It was a little confusing at first to get going and understand how to structure my app. It took about a whole day to understand what I needed to know. But once it clicked, it was very easy to write my code for the platforms in a small amount of time. The code I wrote is 100% reusable for each of the platforms.

</comment><comment><USER3> It depends upon what you know currently and your requirements. If you are familiar with HTML, java script and jQuery, Yes you can use cordova.

if you want to use smartphone functionality like GPS, Camera cordova is providing plug in's.</comment>


Snowden says he might face U.S. retaliation for his disclosures - <AUTHOR> http://www.washingtonpost.com/world/snowden-says-he-may-face-us-retaliation-for-his-disclosures-but-he-is-not-worried/2014/01/26/b7d71558-86e7-11e3-833c-33098f9e5267_story.html

<comment><USER2> Snowden just can't stop talking about himself.</comment>


A Call for Better Vulnerability Response - <AUTHOR> http://blog.erratasec.com/2015/01/a-call-for-better-vulnerability-response.html#.VLUqW81boXc

<comment><USER2> Dupe of [https://news.ycombinator.com/item?id=8874047](https://news.ycombinator.com/item?id=8874047).</comment>


Dialog Semiconductor to acquire Atmel for $4.6B - <AUTHOR> http://www.dialog-semiconductor.com/content/dialog-semiconductor-acquire-atmel-46-billion <comment><USER3> Atmel saw huge success with the AVR at the end of the 90s, and they had the ARM microcontroller market nearly to themselves with their SAM7 and SAM9 series of controllers. Unfortunately they dropped the ball when the Cortex cores came (and ARM really sprung into popularity) which ST gladly picked up and ran away with (around 2008, when the STM32F1 was introduced). What happened to Atmel is that they chose to invest their resources in the AVR32 series (riding on the success of the 8-bit AVR and the strong political position of the AVR group inside Atmel; IIRC the ARM group was in France while the AVR group was based in Norway), but they didn't read the market correctly. Everyone was going away from proprietary architectures and into the Cortex ecosystem (not always rationally) - customers bought into the pseudo- standardization ARM was selling them. This standardization was only perceived, as once you get into the 32-bit domain, the code is mostly C that can easily compile between cores - its the peripheral drivers that are non-standard and where most of the porting work exists and going from an ST Cortex to an Atmel Cortex is a big headache. Anyway, this shift in the market caught Atmel (and others, like Microchip and Renesas) off guard and pushed ST into the Cortex dominance it has today.

Around that time, Atmel changed management and refocused the company (selling off their memory lines, other fringe products and their fabs) to be a primarily microcontroller company, but it was a bit too late and they found themselves behind. It's been a while since I worked with 8 bit micros, but the word is that their AVR series is not really innovative anymore.

Seems like Atmel is back on its feet though with great new Cortex-F7 processors (I think they have the industry speed record at 300mhz and some other benefits).

I haven't looked at the graphs, but $4.6bn is kind of a valuation low point considering their last 10 years or so. They probably understood that they're too small to survive by themselves, especially as how complicated microcontrollers are becoming. I don't think microcontrollers are such a fantastic business - talking to my friends at component distribution like Avnet and Arrow, they make a lot more money on some passive components (like power) than on micros, and for a long time some companies have been allegedly selling microcontrollers at a loss, just to get the designs.

<comment><USER4> Nice summary of Atmel. I had no idea that Atmel was not doing well. Atmel has the basic arduino business, although a lot of other architectures, especially newer and more powerful ones, are using the arduino framework for their CPUs. Even the newest official arduino platform is ARM based.

I'm constantly reading about how the 8-bit market is still doing well, despite the emergence of ARM. Even though the AVR hasn't really innovated much, it has picked up a lot of peripherals and low power modes which has kept the AVRs from being strictly in the bargain basement MCU market. I don't think there has been much innovation in the 8-bit market, but the AVRs have a really good programming model, especially compared to the 8-bit PICs. I don't know if that's enough to give AVRs a shot at new designs, but they're one of the best 8-bit options today, IMHO.

<comment><USER3> Arduino has been huge for Atmel in terms of brand recognition and popularity of AVRs, but overall, its not a huge revenue source (even if 1m Arduino boards are sold yearly, that's at best $3m-$4m in sales). Of course that there are indirect benefits.

Most of the designs I know, migrate away from 8 bits and into 32 bit. The Cortex M0 is great. While designing with 8 bit is somewhat of an upgrade dead- end, if you design a product with an ST Cortex-M0, you can pretty easily go up their Cortex family ladder while retaining a lot of compatibility on both the board and firmware levels (ST is particularly good at that, which is why its my default recommendation when designing a new product - I can explain why if anyone is interested). The denser 8 bit parts are really expensive anyway - <USER2> suspect its because manufacturers know its mostly "locked in" customers that need more flash or I/O for an existing design and need a more capable, near drop-in replacement.

I also vastly prefer AVRs to PICs, but AVRs were kinda where I started my embedded programming career so I guess I just reflect on them fondly.

<comment><USER5> Please do. I can easily see the benefits of being able to upgrade a chip while retaining software/hardware compatibility but I'd love to hear any other selling points. Are they reasonably competitive with equivalent AVRs in terms of cost and power? Do they have a PicoPower equivalent?

<comment><USER3> PicoPower - I can't tell you the specifics, as comparing power consumption is a bit difficult (you need to figure out which peripherals you need running, whats the system clock, input voltage, sleep/wakeup regime etc and calculate from there), but they do have STM32L0 series of ultra-low power based on Cortex-M0+.

As to why do I like ST:

1\. They are very easy to design boards for. Each IO pin on the Cortex micros maps to up to 16 different internal functions. That gives amazing flexibility from the board design perspective.

2\. They have fantastic long term migration paths. You can directly go from Cortex-M3 to Cortex-M4 to Cortex-M7, while, in many cases, retaining package compatibility.

3\. They have very good design validation. Their errata sheets are quite shorter than what I remember working with other manufacturers.

4\. They have a huge amount of useful IO, such as timers (I have one design using over 15 timer channels) and a quite flexible and powerful ADC architecture.

5\. They probably have the widest selection of Cortex microcontrollers in the industry.

There are downsides, too; I generally find their peripherals less advanced than Atmel's (regular CAN instead of CAN-FD, USB inferior to Atmel's, I2C that's really sucky and prone to errors, slower clock and no dual precision FPU on the new F7 family). There was also some story with ST's 2MB flash parts, being not really usable - but I'm not sure what was that about.

Overall you can't go wrong with ST. The long term migration ability and board design flexibility are well worth the downsides.

Coming from an AVR, you're immediately going to get way more memory, much higher clock frequencies, ARM CMSIS libraries (fun if you have any math/DSP going on) and more industry standard tools.

<comment><USER6> Unfortunately STM uses a lot of external silicon IP (all MCU designers have to) from different firms that don't really work well together or are straight up inferior as you mentioned.

Their eratta sheets are shorter but I've run into a number of problems that I've worked around with help from their engineers and a year later I still couodnt find any notes on them. I'm guessing they underreport edge case silicon bugs like almost all silicon manufacturers.

My last project used a STM32F437 with 100 mbit ethernet, just about saturating the connection at 80mbits while maxing out the processor to interleave the bits of all incoming data and bit banging every GPIO unused by peripherals to control 30k individually adressable LEDs at 45 fps which is just incredible for a $7-10 microcontroller. Then after a few weeks of field testing we identified half a dozen problems all resulting in nondeterministic failures of the DMA or MAC peripheral requiring a hard reset. I think the most problematic one, DMA1 or DMA2 stalling when they tick within 10-20 cycles of each other, is the only one that was added to the errata and still plagues the STM32F4x7 parts.

That said if you care about part swapability, developer productivity, and need a very powerful ARM that you're not pushing to the edge, the STM32 line of chips is a great choice.

<comment><USER3> That sounds nasty. I've had my share of problem with Atmel parts though when operating at high rates - on their Cortex-M3 and SAM9 there were problems of AHB overruns and data loss when doing fast Ethernet or SPI transfers. No hard lockups though.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> The problem is that the ARM Cortex microcontroller space doesn't have any differentiation left.

As a microcontroller user, I care about two features: RAM and price.

Really. That's it. I'll eat the engineering to switch lines if I can hold my price but double my RAM.

I don't care about frequency--everything is fast enough. I care a bit about flash, but I'm almost never flash limited. If you don't have _enough_ of the peripherals I need I won't buy, but everybody has a ton of peripherals since transistors are cheap. And everybody's tool chains suck equally so there is no advantage to be gathered there.

<comment><USER8> There are people who care about power as well, right? E.g. the security system where I'm living now has remote wireless units with fairly big lithium batteries that last roughly 10 years....

<comment><USER7> > There are people who care about power as well, right?

Not as much as you would think.

Battery life in the embedded space is mostly defined by how much you can stay off and what your leakages are rather than what your active current is--since you can quite often just stay asleep a little longer.

Most of these microcontrollers are in the same range for leakage and are at the point where you have to start considering other leakage sources on the board (capacitors for example).

It's Amdahl's Law in action: the big contributors have all been smacked down pretty much as far as they can (sure, you might get 10% here or there ... but it's going to be real work now) so now getting overall performance isn't so easy.

If someone _really_ wanted to get my attention with their microcontroller, they should produce a combined BLE/WiFi chip where the WiFi can be run off of a coin cell battery like BLE (that would mean something like 2mW transmit power rather than 200mW).

<comment><USER5> In particular I suspect there's very few gains that can be made for wireless communications. My contention is that there exists a information-theoretic lower bound on the amount of energy it takes to transmit a given amount of information a given distance. Like a heat engine, at some point you can no longer make generational improvements in efficiency. You get asymptotically closer to the bound, you can't ever beat it. And wireless power tends to dominate the power budget of anything that uses it.

High-bandwidth signals require more power to travel a given distance, low- bandwidth signals go farther on less power but transmit less information. Intuitively this is why you can go worldwide with CW (Morse code) on just a few watts of power, while a SSB signal on the same frequency doesn't go far at all. But I think you can mathematically support this by playing with the bandwidth+power terms of the Shannon-Hartley theorem.

If we want to transmit a given amount of data per time unit, there are two natural extremes. We can either transmit it at the average data rate necessary, or we can race-to-sleep by blasting it out as fast as possible. I'm not sure which is more efficient given that SNR is inside the log2 of the S-H theorem, it depends on the relationship between SNR and bandwidth as you spread a given quantity of RF power thinner and thinner. Based on the log2 I would think this is an inverse-square kind of thing, making it an equivalent tradeoff.

Showerthoughts: because of the Inverse Square Law, it's probably more efficient to make a mesh of multiple small hops than a single long shot. As the spacing of the mesh decreases to zero, you end up with a wire.

Also, check out the ESP8266 chip, it doesn't have a BTLE onboard but it does handle the "SOC with low-power wifi" thing.

<comment><USER7> > Also, check out the ESP8266 chip, it doesn't have a BTLE onboard but it does > handle the "SOC with low-power wifi" thing.

I know all about it. Still won't run on a coin cell.

Nobody will build a 2mW WiFi chip until Apple forces somebody to build it. Then _EVERYBODY_ will rush to build it. Then _EVERYBODY_ -1 will go bankrupt when Apple doesn't use their chip.

It's the Zen of Hardware.

</comment></comment></comment></comment></comment><comment><USER9> Here's why Dialog is buying Atmel for $4.6 billion [http://fortune.com/2015/09/21/dialog-buying- atmel/?xid=soc_s...](http://fortune.com/2015/09/21/dialog-buying- atmel/?xid=soc_socialflow_twitter_FORTUNE)

</comment><comment><USER10> As an Atmel customer, what does it mean to me?

Will nothing change? Will Atmel's (or is it Dialog's ?) micro-controllers will become even more power efficient?

<comment><USER11> Hell, what happens to Arduino?

<comment><USER12> Even if they stopped making AVRs together, there are plenty of other (arguably better) MCUs on the market anyway.

At the company I work for, we've replaced all the Atmega chips in our products with the STM32F0 series chips for lower-end requirements and with STM32F4 series for higher end stuff, because they have more grunt and generally have more peripherals, and are similar prices.

<comment><USER13> How many MCUs have the same strong, libre toolchain?

<comment><USER14> Free GCC tool chains are present for the TI MSP430, TI Tiva Series C (Cortex M4) and also TI Hercules (Cortex M3) series. Apart from that I have tried STM32F4 series with GCC tool chains.

I have personally developed Bluetooth stack using the GCC tools for the MSP430 using the CC246x Bluetooth modules and it was a blast.

So you have a lot of options with free tool chains. I personally use only free toolchains on linux. It is the best development setup and you have a lot of options for debugging and tool development.</comment></comment></comment></comment></comment>


What Is a Backdoor? - <AUTHOR> https://twitter.com/Huawei/status/1235128718869164032 <comment><AUTHOR> Note that submission doesn't equal agreement.

Personally, I think that ordering 5G equipment is mostly a choice of _which_ country spies on you, so I do not find Huawei's argument to be persuasive - but I still thought this warranted a discussion.</comment>


Twitter co-founder: We'll have made it when you shut up about us - <AUTHOR> http://news.cnet.com/8301-13577_3-10256113-36.html?part=rss&tag=feed&subj=TheSocial <comment><USER2> Microblogging will have made it when we dont have to depend on a single centralized provider to work. The system needs to be naturally distributed and handle partial failure, everything similar to email.

<comment><USER3> Google wave.

</comment></comment><comment><USER4> That's one possibility. The other possibility: when we all shut up about Twitter, it won't because it's become an unconscious tool. It'll be because it's a long forgotten fad.

</comment><comment><USER5> The article title is a bit misleading: because it suggests a naive thought process (because Twitter _depends_ on buzz for it's value).

But the actual point he makes is very valid: which is that when Twitter becomes an unconcious tool then it will have "made it" (w/e that means).</comment>


Someone stole my identity on Upwork and all I got was this lousy blog post - <AUTHOR> https://medium.com/@_chelleshock/someone-stole-my-identity-on-upwork-and-all-i-got-was-this-lousy-blog-post-d63aab2b4c90#.3vw78p3ah <comment><USER2> Dupe: [https://news.ycombinator.com/item?id=12787894](https://news.ycombinator.com/item?id=12787894)

</comment><comment><USER3> Related: [https://news.ycombinator.com/item?id=12773282](https://news.ycombinator.com/item?id=12773282)</comment>


Genetic algorithms for training deep neural networks (2017) - <AUTHOR> https://eng.uber.com/deep-neuroevolution/ <comment><USER2> The article is from last year but it's still extremely valuable and interesting.

Exploring this topic is currently my primary hobby. Specifically, I've been using OpenAI's retro (Sonic, Contra, Mario, Donkey Kong and, more recently FZero) and comparing the ancient NEAT with more fashionable stuff like DQN, PPO, A3C and DDPG.

With my extremely limited experience, NEAT seems to outperform all of these other algorithms. I believe the advantage is the potential for strange/novel network structure.

And the best part is that NEAT doesn't require a powerful GPU.

Apologies for the shameless plug but here's a link to a series on youtube I made about using Retro and NEAT together to play Sonic. [https://www.youtube.com/watch?v=pClGmU1JEsM&list=PLTWFMbPFsv...](https://www.youtube.com/watch?v=pClGmU1JEsM&list=PLTWFMbPFsvz3CeozHfeuJIXWAJMkPtAdS)

<comment><USER3> You are evolving the topology, but using regular gradient descent/backprop for any given network, correct?

<comment><USER4> No, in NEAT both the weights and topology are evolved. It is totally gradient- free.

</comment></comment></comment><comment><USER5> This is a fun book (published in 2001) about how a professor and his graduate assistant developed a world-class checkers-playing algorithm using neuroevolution:

[https://en.wikipedia.org/wiki/Blondie24](https://en.wikipedia.org/wiki/Blondie24)

[https://www.amazon.com/Blondie24-Playing-Kaufmann- Artificial...](https://www.amazon.com/Blondie24-Playing-Kaufmann-Artificial- Intelligence/dp/1558607838)

(Edit) One of the funniest parts I remember is that they had to leave it running on a Pentium III for like a month or something.

</comment><comment><USER6> For those interested, i built a python app on top of tensorflow/keras to do neural networks architecture & hyper parameters search with genetic algorithms.

[https://github.com/guybedo/minos](https://github.com/guybedo/minos)

<comment><USER7> Note that here the weights are evolved, not just the parameters. Gradient descent is not used at all AFAIK.

</comment></comment><comment><USER8> I lost it at "emerging revolution".

My PhD thesis in 2000 already used genetic algorithms for seeds and it was hardly new then.

<comment> ur-whale Genetic algorithms are (and were already back in 2000) a pretty decent and -more importantly- generic solution to the problem of global optimization (as opposed to local optimization) when the problem to optimize has some sort of (maybe not-so-smooth) structure.

Many of the recent "AI" development very often boil down to finding a local extremum using some sort of ski down the slope optimization program (aka "training"). These techniques very rarely tackles global optimization, or when they do, bundle it up under the moniker "hyper-parameter tuning".

A good example of something that falls under "global optimization" and isn't often tackled in deep learning would be finding the correct deep net architecture for a given problem.

The problem doesn't lend itself very well to local optimization, but might yield to GA-type optimization.

<comment><USER9> How are genetic algorithms guaranteed in any way to give you a global optimum? I thought it might just work better when your search space is non smooth.

<comment><USER10> They're not, as far as I know. In fact, that was one of the big selling points of reinforcement learning - that it tends to reach a better minimum than GA.

</comment></comment></comment></comment><comment><USER11> I tried training small RNN models using GA around 1990. I had lunch with John Koza (the genetic programming pioneer) and he suggested that it was an interesting idea but would not scale. The Uber team, my controlling mutation, got it to scale - good for them. I did a few years later use this as an example in my book "C++ Power Paradigms", McGraw-Hill 1994 (Genetic Algorithms, Neural Networks, and Constraint Programming).

</comment><comment><USER12> Too many ideas, too little time! I am already thinking for a while about how Deep Learning and Genetic Algorithms could benefit each other.

GAs allow optimization of parameters without a differentiable loss function - a major problem with evaluating behavior of a neural model, for example.

But also GAs could benefit from ML/DL. Predicting loss functions from a chromosome representation (to save computing time), learning to select promising pairs and even learning cross-over operators.

</comment><comment><USER13> This is the not-so-secret sauce when training neural nets and backtesting for algorithmic trading. It dramatically reduces the time taken.

<comment><USER14> Sounds interesting, are you able to go into more detail?

</comment></comment><comment><USER15> step 1: hook it up to a car</comment>


How to build a JavaScript package manager - <AUTHOR> https://yarnpkg.com/blog/2017/07/11/lets-dev-a-package-manager/ <comment><USER2> It was making a lot of sense until the part about recursive dependencies. To my mind, once this happens it becomes a dependency graph instead of a tree, and instead of trying to satisfy dependencies "locally" I would work out all the constraints and then solve them. The "choose highest, check parent" algorithm seems to me like it would make results depend on which package in a cycle you tried to resolve first, and apparently with peerdependencies it sometimes is incorrect ([https://github.com/yarnpkg/yarn/issues/422](https://github.com/yarnpkg/yarn/issues/422)). Any insight as to why it was designed this way? Just the NP-completeness?

<comment><USER3> I've made some early work on using constraints when I started working on Yarn, but it ended up unpractical because of the sheer execution time it required. A huge number of constraints were required to satisfy the requirements, which was made even worse when you consider that a dependency version might have different sub-dependencies than its other versions.

That being said, I'm not an expert in SAT logic, and it's quite possible a better solution is possible! Maybe some kind of hybrid algorithm, possibly? In fact, one thing I'd really like to do in medium term would be to externalize the resolver part of Yarn - this way, it would be much easier to experiment with it to try to find algorithms that fits various requirements. If you want correctness you would use a slow but comprehensive algorithm, if you want speed you would use a naive algorithm like the one in the article, etc.

<comment><USER2> Interesting, which packages did you try running it on for which it took too much execution time? Where was most of the time spent?

<comment><USER3> I unfortunately don't remember - possibly a combo of Babel + Webpack and their plugins, tho, they're usually my go-to choices to test install perfs.

Now that I think about it, it's quite possible that the process was apparently hanging on the same issue I describe in my article, where babel-core depends and babel-cli and vice versa. Maybe it would work better if I were to run the process a first time with a simple algorithm like the one exposed in the article, that would clear up any dependency loop, then a second more complex pass that wouldn't have to deal with this.

</comment></comment></comment></comment><comment><USER4> Love this kind of format as a way to explain how Yarn works. Awesome article!

</comment><comment><USER5> Oh no. Not just another package manager ... a package manager generator... _runs_ :-)

</comment><comment><USER6> First impression: fantastic section titles.

<comment><USER7> I chuckled as well.

</comment></comment><comment><USER8> Yet another package manager except it teaches you how to write one. Nice demo with explaining titles. Too bad I don't know JS and honestly I don't think the world needs another package manager just like it doesn't need another single page web app to read Hello World. But the post definitely gives a bird's eye view of things.

<comment><USER9> I don't understand these sorts of comments. The world probably doesn't need a lot of different things--another video game, another search engine, another social network--but strict _need_ is not why we do things.

Why was it so important for you to leave this sort of dismissive comment?

<comment><USER10> Agree with the parent. Regarding yarn, it was created because npm started showing weaknesses in many things, with large and complex dependency trees: install was slow, sometimes unreliable and not cross-platform portable, and the codebase was difficult to refactor due to legacy code and backward compatibility, so it made more sense to create a new package manager without that burden and try to iteratively align with npm to become a drop-in replacement.

It's easy to fall into the fatigue due to new languages, tools etc. coming out each week but please think twice before posting dismissive comments like that.

I observe a lot of "I don't read HN anymore" sentiment lately among top tech Twitter users, partially due to that.

There's a lot of awesome stuff being given out for free and people are whining. (Yes, I'm a bit tired of constantly following and evaluating things, but I keep it for myself.)

The situation came to the point where some guys are thinking about _not opensourcing their tools_ anymore due to constant criticism and maintenance fatigue. Read the two links below (accidentally, the first one is from yarn co-author!):

[https://twitter.com/sebmck/status/879282797915119616](https://twitter.com/sebmck/status/879282797915119616)

[https://nolanlawson.com/2017/03/05/what-it-feels-like-to- be-...](https://nolanlawson.com/2017/03/05/what-it-feels-like-to-be-an-open- source-maintainer/)

<comment><USER9> I've seriously thought about it myself, shutting off and going back to closed software. I've had my software praised one week as innovative and awesome, to the next week being "yet another example of what is wrong with these young developers and their JavaScript" (I'm 35, I've been programming for 20 years). All here on HN. I won't even touch Reddit anymore.

<comment><USER11> Have fun writing it and using it and/or make money from it.

Nothing else really matters...</comment></comment></comment></comment></comment>


Show HN: Only NBA Tweets from ESPN Stats & Info - <AUTHOR> https://github.com/assafmo/nba-espn-stats-and-info <comment><AUTHOR> I love [https://twitter.com/ESPNStatsInfo](https://twitter.com/ESPNStatsInfo)!

But I'm only interested in NBA tweets, so I created this over the last couple of days.

Enjoy.</comment>


Dynimize: Speed Up MySQL with CPU Performance Virtualization - <AUTHOR> https://dynimize.com/ <comment><USER2> Does the presence of a HDD or SSD across the three architectures make a difference, or is this purely a CPU problem to be solved? I see the testing made sure to fit all tables in memory with a large enough innodb buffer pool, so was storage even a factor?

[https://dynimize.com/blog/discussions/dynimizer-mysql- cross-...](https://dynimize.com/blog/discussions/dynimizer-mysql-cross- microarchitecture-analysis/)

It seems to me the only system with SSDs (Ivy Bridge) sees higher transactions per second improvement than the other systems.

[https://dynimize.com/performanceSpeedup](https://dynimize.com/performanceSpeedup)

<comment><USER3> These specific tests we did were read only where the working set fits into memory, so SSD vs HDD doesn't matter because they were CPU bound tests to highlight the performance improvements. So storage isn't a factor here. If that wasn't the case then faster storage would help make it more CPU bound and Dynimizer would make a bigger impact when using SSD vs HDD.

</comment></comment><comment><USER4> This doesn't seem to be JITing actual queries (a la [https://www.pgcon.org/2017/schedule/events/1092.en.html](https://www.pgcon.org/2017/schedule/events/1092.en.html))

It seems to be JITting the generic binary code.

Would these same gains be seen from compiling MySQL for the actual target architecture gentoo-style e.g. -farch=native ?

<comment><USER5> It looks like it's taking measurements from live process, so probably not arch=native. But most likely the runtime profile guided optimisation would do a similar job.

</comment></comment><comment><USER6> Nice website, and very cool looking product!

Has this been run in production anywhere? Were there any instances of corrupted writes? Inaccurate reads?

What's your plan for monetization down the road?

<comment><USER3> Yes it's definitely being used in production. We're starting to collect production use cases and will provide some on our website soon. Here's an example website with growing traffic using MariaDB + Dynimizer with Wordpress, and they found Dynimizer very helpful: [https://www.cgmagonline.com](https://www.cgmagonline.com)

In terms of innacurate reads or corrupted writes, that would be a bug if it ever happens. That would not be part of normal operation and would not be expected. That said, all software including MySQL, gcc, and Linux are full of bugs and Dynimizer is not immune to that of course. However it has been stress tested thoroughly with MySQL, MariaDB, and Percona Server up to MySQL 5.7, MariaDB 10.2.

<comment><USER6> Understood, and thank you for the reply. I look forward to seeing some use cases published. Definitely bookmarking the site now.

Best of luck with this. Exciting stuff!

</comment></comment></comment><comment><USER7> Very, very cool. Section 7 of the manual[0] gives some hints on how this black magic works:

> 7\. Workload Requirements

> To obtain benefit from the current version of Dynimizer, all of the > following workload conditions must be met:

> A small number of CPU intensive processes - On a given OS host where the > workload is running, the workload must be comprised of one or a few CPU > intensive processes. Optimizing a large number of processes at once is not > recommended.

> Long running programs - The processes being optimized have long lifetimes, > and their workloads are long running in order to amortize the warmup time > associated with optimization.

> x86-64 - Optimized processes must be 64-bit, derived from x86-64 executables > and shared libraries, which must comply with the x86-64 ABI and ELF-64 > formats. Most statically compiled applications on Linux meet this > requirement.

> Dynamically Linked - Target processes must be dynamically linked to its > shared libraries. Statically linked processes are not yet supported. Most > Linux programs are dynamically linked.

> No self modifying code - The target application must not be running its own > Just-In-Time compiler such as those found in Java virtual machines. This > therefore excludes Java Applications.

> Front-end CPU stalls - The workload wastes a lot of time in CPU instruction > cache misses, instruction TLB misses, and to a lesser extent branch > mispredictions.

> User mode execution - Much of that wasted time is spent in user mode > execution (as opposed to kernel mode), as Dynimizer only optimizes user mode > machine code.

> Because of these requirements, Dynimizer takes a whitelist approach when > determining if programs are allowed to be optimized, with MySQL and its > variants being the currently supported optimization targets on that list for > this early beta release. Other programs are not currently supported, and > while they can be used with Dynimizer, they should be very thoroughly tested > by the user or system administrator before being deployed in a production > environment.

> Future versions of Dynimizer may eliminate many of these workload > requirements, broadening the variety of applicable scenarios as well as > further increasing the performance delivered in previously beneficial cases.

[0] [https://dynimize.com/manual](https://dynimize.com/manual)

<comment><USER8> The real important bits: "Front-end CPU stalls - The workload wastes a lot of time in CPU instruction cache misses, instruction TLB misses, and to a lesser extent branch mispredictions".

My educated guess is that it relocates the hot path of the text segment to better pack into the instruction cache. Cool.

<comment><USER3> Sure does.

</comment></comment></comment><comment><USER9> Cool, but a bit grandiose and historically arrogant:

"[T]he industry's first CPU performance virtualization software"

"A New Frontier For JIT Compilers… JIT compilers use as input a virtual machine code format… Dynimizer [uses] real machine code as input instead"

Just 20 years after Mojo, Dynamo, DynamoRIO, etc:

[http://program-transformation.org/Transform/BinaryOptimisers](http://program- transformation.org/Transform/BinaryOptimisers) [http://www.dynamorio.org](http://www.dynamorio.org) [https://www.complang.tuwien.ac.at/andi/bala.pdf](https://www.complang.tuwien.ac.at/andi/bala.pdf) [http://cseweb.ucsd.edu/~lerner/mojo.ps](http://cseweb.ucsd.edu/~lerner/mojo.ps) …

<comment><USER3> Of course these projects were a major source of inspiration for Dynimizer. However they are not JIT compilers. They are more like virtual machines or binary translators. Today DynamoRIO and Mojo (which ended up as Intel PIN) are used for program introspection and analysis, not for application acceleration.

<comment><USER8> "Dynamic binary translation" is the term of art. Which of course VMWare and VirtualPC were doing 20 years ago in dynamically translating x86 ring-0 code to ring-3 code.

Dynamizer is translating x86-64 to faster x86-64, but the concept is the same.

DynamoRIO was actually talked about for application acceleration. There was at least a PoC that did dynamic function inlining.

</comment></comment></comment><comment><USER10> This is _very_ cool stuff.

I would be very interested to hear a sampling of the war stories that came out of building this. I had a friend working on the IBM zPDT JIT at one point, and while I unfortunately can't remember many of the details at the moment, I remember boggling (in that sort of emergently satisfying way) at some of the 'oh shit' moments that came up.

<comment><USER3> Lots of 'oh shit' moments. Lots.

</comment></comment><comment><USER3> Here are the slides from Percona Live 2018:

[https://www.percona.com/live/18/sites/default/files/slides/A...](https://www.percona.com/live/18/sites/default/files/slides/Accelerating%20MySQL%20with%20JIT%20Compilers%20-%20FileId%20-%20129518.pdf)

<comment><USER11> Mentioned as the installation method in those slides:

      sudo bash c 'bash <(wget O  https://dynimize.com/install) default'

Come on, please don't teach people horrendous security practices... :(

<comment><USER12> It's depressing how commonplace `curl|(ba)sh` has become.

This will sound clichéd but I blame the rise of "poor mans devops" whereby management fires all the ops, and lets developers manage infrastructure.

<comment><USER13> I agree, it may be cliche, but I think the exact same thing whenever I see this kind of practice, too. Or that the developer that has never had to manage a live system with users that know his phone number and his boss's phone number.

"Oh, this is just for a test mock up. Nobody is supposed to actually use this to install it for real."

Well, to experienced people it makes you look moderately stupid, and to inexperienced people it looks like an elegant solution. It's actively hostile to secure system planning.

It reminds me of the NPM left-pad debacle[0] and some of the criticism[1] that came up from that.

0: [https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/](https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/)

1: [http://www.haneycodes.net/npm-left-pad-have-we-forgotten- how...](http://www.haneycodes.net/npm-left-pad-have-we-forgotten-how-to- program/)

<comment><USER12> I've given up waiting for nodejs to become a reliable environment. Just recently the `is-even` package came to light and highlighted that things aren't getting any better than when leftpad was a thing.

I can't wait to see tc39's response to the `is-even` shit show after they decided to just add leftpad to the stdlib.

<comment><USER13> Wow, I hadn't heard about the is-odd/is-even/is-number thing. That's hilarious and awful.

Reminds me of: [https://github.com/jezen/is- thirteen](https://github.com/jezen/is-thirteen)

<comment><USER12> The "best" part of it all is that apparently js engines have an internal optimisation for `foo % 2 === 0`, because it's such a common thing.

This clown was using a bit wise operation in `is-even` "because everyone already knows about % 2 === 0`, and thus was hurting performance (on top of whatever extra memory is used for the module, function call overhead etc)

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> This looks like seriously impressive technology, and yields impressive results, but: I don't think I'd be comfortable with the idea of something rewriting my database in production. What I -would- probably be OK with is having dynamize analyze my workload in a staging/load test environment, and then producing a new binary for me which I could then run through its paces.

Beyond actual errors being produced, I'm wondering what'd happen in weird scenarios such as one where by primary gets heavily optimized for its write load and creeps up to, say, 80% CPU or so at peak.. What then happens then if my replica which has been heavily optimized for its read load gets promoted in a failure scenario and gets pegged at high CPU?

Final thought here is if this tech really is solid, when is AWS going to start shipping it with my VM?

<comment><USER3> This does not rewrite your database. It optimizes the live in-memory machine code of the mysqld (MySQL Server) process. It must run on the same OS host as the mysqld (MySQL Server) process being optimized. So if you are using this on the master and not on the replica, the replica won't be touched. Hope that makes sense.

<comment><USER15> I think the point was that profile guided optimization relies on the workload staying relatively fixed. If the workload suddenly changes (like promoting a readonly slave to be the writable master) the assumptions made by PGO may not be valid, and performance could be worse than if no modifications were made in the first place.

I think you'd just have to measure scenarios before using it in production.

<comment><USER3> Dynimizer can react to drastic changes in workload and reoptimize depending on how you configure it.

</comment></comment></comment></comment><comment><USER16> Very cool product.

From the /product page:

> ...It profiles applications using the Linux perf_events subsystem and > interfaces with a target application's machine code through the Linux ptrace > system call. When optimizing a program, it loads a code cache into the > target program's address space...

</comment><comment><USER17> @davidyeager: In the legend of the graph in Dynimizer System Overhead in [https://dynimize.com/product](https://dynimize.com/product), both series are labelled as "Without Dynimizer".

Couple of questions. Since this seems to be a very general technology, why the emphasis on MySQL (and DBs in general)? Marketing? Also, I found Dynimize vs Dynimizer confusing - is that company name vs product name?

<comment><USER3> Yes we will correct the legend in that graph, thanks for reporting it.

Dynimize is the company, Dynimizer the product. We may ditch the name Dynimizer and just go with Dynimize to avoid confusion. Thoughts?

It is a general purpose approach to optimization and MySQL is just a starting point. It was chosen first because it has a broad user base and is relatively easier to support compared to many other Linux programs: single process architecture, long process lifetimes, OLTP workloads are known to spend much of their time in front-end CPU stalls on the CPU side which are effectively targeted with profile guided compiler optimizations, and it's statically compiled. We've tried it on MongoDB and seen similar benefit but not supported yet. Coming soon. Windows will probably require some driver development for effective sample based profiling and will happen later on. We will improve the effectiveness of our other optimizations that don't target front-end CPU stalls and better support multiprocess workloads with short process lifetimes, which will allow us to target many other types of programs in the future.

Hope that clarifies things a bit.

</comment></comment><comment><USER18> This looks really interesting!

Wonder if it could be integrated into the whole OS/kernel, and if it can help with typical ML workloads like running a randomforest in scikit learn.

</comment><comment><USER19> Very simple-to-use product. There's almost no friction and you get 10% extra TPS. They'll make a lot of money.

<comment><USER20> How are they making money?

</comment></comment><comment><USER21> I'm very skeptical running a script from a random web page that promises to make my program faster.

<comment><USER22> Indeed, it is literal insanity to run this on anything production that you care about.

<comment><USER8> It isn't any different than running your production in a container, a VM, or the cloud, all of which can significantly affect what's actually going on.

</comment></comment></comment><comment><USER23> Does it makes a difference on VM / containers / VPS ? Or does-it works only on server CPUs ?

<comment><USER3> Works with VMs or on a VPS. We have done a lot of testing on KVM, Xen, and a bit on VMware. Still need to do a bit of work to properly support containers.

</comment></comment><comment><USER24> I wonder, do the authors have a reverse engineering background? It seems like a lot of concepts from reverse engineering were applied to the JIT compiler, which I find incredibly cool.

</comment><comment><USER25> The main infographic shows an impressive improvement in TPS, but what about single query execution time? I often see the CPU maxed out by complex queries against large tables.

</comment><comment><USER26> So neat. I'd love to read a paper about how this works in more detail. Does it exist?

</comment><comment><USER27> What I saw of this looked pretty neat... until the web page crashed Safari (on iPad).

</comment><comment><USER28> Heyo, workloads that fit in RAM are not that interesting. I hope they have identified their market ahead of time, for their sake, because I don't think it's valuable.

<comment><USER3> That was used to highlight the maximum improvement expected. When the working set doesn't fit into RAM then you will get some combination of a smaller amount of performance improvement plus a reduction in CPU usage. The faster the storage, the greater the increase in tps that you'll see. Note that replication is often CPU bound. We will be applying this to non-database workloads as well in the near future.</comment></comment>