The Dactyl-ManuForm Keyboard - <AUTHOR> https://github.com/abstracthat/dactyl-manuform <comment><USER2> I'm currently in the process of building one of these Manuform keyboards. If anybody else wants to build one, here are a few helpful things that took me hours to find:

1\. By far the most readable closure script. Has more features than any other script I found (killer feature for me: wide pinky buttons): [https://github.com/ibnuda/dactyl- keyboard/tree/refaktor](https://github.com/ibnuda/dactyl- keyboard/tree/refaktor)

2\. The same script as a web-form without exposing every parameter. Generates openSCAD 3D files: [http://dactyl.siskam.link/manuform](http://dactyl.siskam.link/manuform)

3\. Nice assembly guide 1: [https://nickgreen.info/dactyl-manuform-build- log/](https://nickgreen.info/dactyl-manuform-build-log/)

4\. Nice assembly guide 2: [https://medium.com/swlh/complete-idiot-guide-for- building-a-...](https://medium.com/swlh/complete-idiot-guide-for-building-a- dactyl-manuform-keyboard-53454845b065)

The original Dactyl and Manuform scripts lack commenting, are hard to understand and lack some cool features.

</comment><comment><USER3> This is pretty awesome.

For those who haven't explored ergonomic keyboards, 1) you should! take care of your hands! and 2) shit gets preeeeetty wild out there.

For further reading, check out ortholinear keyboards like the Planck [1], or crazier split designs like the ErgoDox EZ [2], or even "keywell" (or just "pit") designs like the Kinesis Advantage2 [3]. All require a learning curve (obviously!), but they offer some pretty immediately noticeable (in my experience) advantages for your hand health.

One of my favorite design items are thumb clusters like you see on the last two links here: why should your thumb be limited to just spacebar, and why should both of your thumbs do the same thing?

[1] [https://ergodox-ez.com/pages/planck](https://ergodox-ez.com/pages/planck) [2] [https://ergodox-ez.com/](https://ergodox-ez.com/) [3] [https://kinesis- ergo.com/shop/advantage2/](https://kinesis-ergo.com/shop/advantage2/)

<comment><USER4> I have not heard good things about the ergodox's thumb keys. A lot of people say you need to stretch your thumbs pretty far to use them (even if you have big hands), which kind of works against the goal of not putting stress on your hands.

<comment><USER5> It seems it's very polarising.

I personally hate it and I find only a single key on the cluster is comfortable for me. I do have to do big stretches or move the whole hand for the other keys.

There are many better alternatives out there.

<comment><USER6> >There are many better alternatives out there.

Its just like buying an office chair or a pair of shoes. Its not that there are better alternatives, is more like everyone should find a keyboard that's comfortable and optimal for them as an individual. The Ergodox-EZ is a great keyboard, just not a great one for you.

BTW do you have really small hands? Based on the size of it and my experience I would assume that most people could comfortable hit the two larger keys on the thumb cluster, they are right next to each other.

<comment><USER5> You're right that it's subjective. I do think the Ergodox is much better than standard keyboards, I just find the large number of (for me) unused keys detrimental.

Yeah I have slightly smaller hands. I also have problems with my right thumb that makes me extra sensitive.

Btw if someone wants to compare the key positions of popular keyboards see this site: [https://jhelvy.shinyapps.io/splitkbcompare/](https://jhelvy.shinyapps.io/splitkbcompare/)

</comment></comment></comment></comment></comment><comment><USER7> More split keyboards:

\- Corne: You usually have to solder it yourself, and source the switches/caps. A few sites will build it for you. [0][1[[2]

\- LilyPro: Same as above. [3][4]

\- Georgo/GeorgoPlex: [7]

Trackball: Someone mentioned a vertical mouse, those are really great, but I have been pushing trackballs on a few colleagues. A good in between is the newer logitech, that has a tilt mechanism, but its a thumbball. [5] You can find more options in at r/trackballs [6]. I can recommend the "Kensington Expert Wireless Trackball Mouse" and "CST L-Trac Blue Trackball", although the latter, the original company shutdown and are being manufactured by xkeys. There's also an open source trackball, ploopy [8].

P.s. buying keyboards and soldering them are great for a quick fix, but this hobby gets expensive quick. I am waiting on a few group buys; another 65% keyboard, another split keyboard and a few more key cap sets.

P.s.s. group buys can be a hit or miss (mostly hit), treat them like kickstarters, and maybe you'll get your parts in a year or so. At least in r/mechanicalkeyboards, a few users have recently been complaining a tad more about not seeing the final product v.s. renders.

[0] [https://github.com/foostan/crkbd](https://github.com/foostan/crkbd)

[1] [https://keyhive.xyz/shop/hotswap-corne- helidox](https://keyhive.xyz/shop/hotswap-corne-helidox)

[2] [https://www.littlekeyboards.com/products/corne-mx-hotswap- ki...](https://www.littlekeyboards.com/products/corne-mx-hotswap-kit)

[3] [https://keyhive.xyz/shop/lily58](https://keyhive.xyz/shop/lily58)

[4] [https://github.com/kata0510/Lily58](https://github.com/kata0510/Lily58)

[6] [https://www.reddit.com/r/Trackballs/](https://www.reddit.com/r/Trackballs/)

[7] [https://www.gboards.ca/](https://www.gboards.ca/)

[8] [https://www.ploopy.co/](https://www.ploopy.co/)

<comment><USER8> Ok, gboards are something straight out of Snow Crash. I didn't know I needed this, thanks a lot :)

<comment><USER7> We've been joking around about getting long trrs cables, wrapping around the back side of our office chairs and attaching the keyboards to the arm rests, go all Picard.

QMK supports mouse/trackball support, I think gboard and a few others support adding a blackberry track ball and we'd be aces.

</comment></comment></comment><comment><USER4> A while back I got diagnosed with carpal and cubital tunnel. After many, many hours of research, I eventually bought a Quefrency from Keebio ([https://keeb.io/pages/assembly-service](https://keeb.io/pages/assembly- service) has the premade ones) to see if it would help. I can now confidently say, if you are worried about getting CT, or you have it and want something easier on your hands, you _need_ to check this out. Seriously, for most people on here an injury like CT can make you lose your job. The split form factor _really_ helps alleviate the pain. Not bending your wrists or elbows helps way more than you can imagine. Even now my hands will still hurt if I use a regular keyboard but I can type on my split board all day long. The Quefrency also has the advantage of not being a different form factor (like the Dactyl), it's just a standard 60% split staggered layout (the smaller form factor helps reduce finger travel as well, but imo it's mostly the split). Plus, it just runs off a pro micro so you can flash your own firmware very easily (it runs on QMK which is open source)

I've also heard good things about the levinson and nyquist but I haven't personally tested them. I have also used an Iris for a while and like it a lot, but it looks like they're backordered for now. The viterbi looks a little big imho, but if you want a big split board it would definitely be a good option.

I actually am somewhat in the keyboard business right now (I got inspired by how well it worked, and it's really fun!), if anyone has any questions or needs some recommendations feel free to AMA.

BTW, if you end up going with a different form factor, be ready to type at 10-20WPM for at least a week or two. When I got my first ortho board, I typed at 13WPM and gained back roughly 5-10WPM a week. You can't use a standard layout while you learn, either, or you lose a ton of the muscle memory.

<comment><USER9> About a year ago I bought half of a Nyquist (6x5 - 5 is standard height minus the F-row, which is not a big deal since you have a second layer) and a Vieterbi (7x5). That gives enough space to fit a full 60% keyboard layout, but unlike the Quefrency the keys are in a grid instead of staggered. Getting used to it didn't take long for me.

I can use the computer _faster_ and more efficiently than with a full-size keyboard, since you have a second layer to put buttons onto and so your fingers don't need to move (it's completely programmable; add 30 layers if you want). I had two buttons, one where the Fn key is on laptops (lower left), and one where the enter key normally is (mid right), to switch to the second layer, where I had WASD,QE,RF = arrow keys, home/end, pgup/dn, and hjkl = arrow keys. This is very useful. And stuff like printscreen, media controls, etc. no longer require stretching your fingers or moving your hands. I had space in the corners on both thumbs, and backspace/enter right next to those on the left/right hands. Also useful and saves your pinky from long stretches to backspace. Positioning it naturally aligned with your arms & wrists is also great.

Overall, highly recommend the Viterbi - which would give you an extra column beyond what I had. But not sure how it would work in half paired with the new Nyquist revision; the Viterbi hasn't been updated yet iirc.

I built that one with loud clicky Cherry Blue switches; recently I switched to the newer revision Nyquist with quiet Matias/Alps tactile switches. The switches are great but very few keycap options; the official ones make it not that quiet since they're not a perfect fit with the keys and thus rattle. The Blues were great but eventually the loudness starts to become just slightly painful; now I use the `bucklespring` program on my computer to simulate it. The new Nyquist has the microcontroller built into the PCB, which simplifies assembly, and adds support for key backlighting, which is useful in the dark. The lack of the 13th column really hurts; with the Nyquist+Viterbi you get a full keyboard with stuff like the F-keys and arrow keys on the second layer. With this I need to put things like +=-_\ and []{} on the second layer. This effectively makes it a triple-layer setup for typing, since you need to use both shift and Fn to write characters, whereas previously Fn was for moving around and doing actions. Even after a long while to get used to it my brain often gets confused and I have to pause for a few seconds to try to figure out how to insert the character. But at the same time lighting is essential (to me) for working at night, and even if that wasn't an issue I don't want to spend another $120 on making a new Viterbi or Nyquist/Viterbi hybrid with quieter keyswitches.

So overall the Nyquist seems like the sort of thing hardcore keyboard enthusiasts would buy, and less suitable for programmers (maybe OK for other people). You can head over to /r/mechanicalkeyboards to find plenty of weirdos who would spend $50 on a single keycap and keep a collection of keyboards with 30 keys total.

If you don't mind sharing, what do you do in the kb business?

</comment></comment><comment><USER10> This is an amazing keyboard, and the best thing: it's really easy to modify to suit your needs. I got to try a friend's Dactyl so I had a general idea of what doesn't feel ergonomic to me about it. Then I took some measurements of my fingers, moved the columns around accordingly, somewhat flattened the whole structure and printed it. And got the most ergonomic keyboard I could imagine. (Gee, I really should solder the other side one so I could use it... :s)

Of course this is a lot of work and not for most people. But the point is that 3D modeling _anything_ from scratch is near impossible for a first timer, yet the dactyl family is easy to modify for anyone who can code.

</comment><comment><USER11> Here's a question I've been pondering, and I'm curious about the community's thoughts:

 _To what extent should the distance between keys be customized to the length of the user's fingers?_

For example, being 6'1", I have larger than average hands, with wider than average distance between the tips of my fingers, and if I pay attention, I can feel how my fingers feel cramped on the standard 104-key layout I've been using my whole life.

Similarly, I see comments here about the differences between people, and the sniff test makes me think it's almost all about the different lengths of user's fingers.

</comment><comment><USER12> I must take this moment to extol the virtues of vertical mice. I had severe pain in my forearm and wrist. I switched to a vertical mouse and that pain went away immediately.

I personally use a now 8? year old Evoluent vertical mouse.

Tell your friends and family. It's totally unnatural for you to twist your arm to hold a normal flat mouse. The vertical mouse is natural and like shaking someone's hand. Try it right now: Hold your hand out like you're going to shake someone's hand, and then slowly turn your hand to hold a flat mouse. Feel that tension? It's terrible for you.

<comment><USER13> I recently bought an Evoluent vertical mouse.

I'm not using it all the time, but one of the best features for a Linux user is the separate middle button. I find mouse-wheel buttons are often stiff, which is uncomfortable when copy-pasting a lot of text using the middle button. The Evoluent has a separate middle button, and a click on the wheel.

</comment></comment><comment><USER14> I'm sure I'm one of many people here searching for the ultimate keyboard.

I missed the kick starter for the Keyboardio Atreus.

I'm looking forward to the launch so I can order one retail.

In short it's a split layout ortholinear similar to the Planck.

Two killer features for me are Bluetooth (and wired) and QMK firmware that allows layers.

[https://shop.keyboard.io/](https://shop.keyboard.io/)

<comment><USER15> Atreus is a split kb but on a single piece, so... not quite? For me the nice thing of a split keyboard is to have each half line up with my shoulders, feels really comfy to type like this (this way my arms rest on my chair's armrests most of the time).

I looked for a while for a nice split mechanical keyboard and I found all sort of crazy things, including the ones from [1]. In the end, I decided to try the kinesis gaming kb [2] and I really like it! (so much so I even got one for my brother :-) It has some cool macro recording keys and extra buttons to map and other features. Caveat: it is wired.

1: [http://reddit.com/r/mechanicalkeyboards](http://reddit.com/r/mechanicalkeyboards) 2: [https://gaming.kinesis-ergo.com/](https://gaming.kinesis-ergo.com/)

<comment><USER16> The Atreus is by far my favorite keyboard, I have a Keyboardio that I never use because the Atreus is so great.

The key is to pull it off your desk and put it in your lap. You end up typing with your hands on your thighs, which feels quite natural and relaxed.

</comment></comment></comment><comment><USER17> I have 3 pairs of the "skeletons" 3D printed, but have never gotten around to the hand-wiring part... It's hassle and one mistake leading to a shorted microcontroller will mean A LOT of desoldering.

<comment><USER18> It's unlikely that you'd make a mistake that would damage the microcontroller. The most likely failure modes are a key that never activates, or a key that is stuck activated, or a row/column that never activates, or a row/column that is stuck activated.

In the case of a broken row/column, if you can't find the problem visually, then you're stuck binary-searching your connections until you find where it is, but at least that's only O(log N) desoldering work.

</comment></comment><comment><USER19> Does anyone have any recommendations for one handed keyboards that are practical?

<comment><USER20> The only one handed keyboard I know of is the one made by matias.

<comment><USER21> Which imo currently makes the best tactile dampened mechanical switches as well. Their quiet click switch is silent and tactile, with actually solid tactility (as opposed to just about every other tactile switch out there).

People have had some reliability issues though, so it's a mixed bag. They have recently gotten some new tooling, so hopefully they will start making keyboards that look somewhat not like a shit 90s keyboard.

<comment><USER20> Yeah, I love their switches but blew three of them in two years which is pretty unacceptable. I've used cherries far longer and never had that problem.

<comment><USER21> I still hope they can fix their issues. I am not paying 150 bucks for a keyboard that might stop working properly without reason. Then I much rather get a high quality rubber dome board. The tactility in those is generally fine (at least better than cherry MX brown), even though I am no fan of having to bottom out.

I am more interested in the newest addition to the unicomp portfolio. The "new model m" and their upcoming TKL boards seem like something I will like :)

</comment></comment></comment></comment></comment><comment><USER22> The Dactyl/Manuform keyboard is a shining totem of one of the deeper rabbit holes out there.

I know; I made it back out alive. My aspirational keyboard is a GergoPlex. I own other mechanical keyboards, such as a Planck EZ and an ErgoDox EZ. My actual daily driver is a Leopold FC660C with Topre Silent 45g rubber dome switches, modified with a Hasu FC660C Controller which can be programmed using QMK (or TMK) firmware. My layout is no longer taking most of my time. I use "one finger each hand" chording and "tap/hold" keys to access many custom layers for controlling my computer and editing in VSCode. My fingers rarely move more than one keycap off home row; my thumbs don't get much work.

When I'm evaluating a new cookbook I compare recipes I know cold. For comparison, "old school" shaving is another rabbit hole that I survived. What habit stayed with me? A brush and Mitchell´s Wool Fat Shaving Soap. If this has the ring of truth for you, you might draw my same conclusions about mechanical keyboards.

Years ago I owned three Kinesis Advantage keyboards. In a way, I loved them. I got rid of them because of too many systematic issues (stuck modifier keys, for example) one could trace to primitive USB hardware they could't afford to redesign. Their new internals might be better.

Every so often I'd try a mechanical keyboard to see what I was missing. I wouldn't spend enough, or get wrist rests to adjust to the higher keycaps. Only the better keyboards and switches are worth the trouble; I got hooked once I finally spent enough.

Our species wants to meld with machines. Many of us wish we could immerse our hands in nutrients and grow physical connections to our machines. I've met computer scientists working on brain implants; they talk about Alzheimer's to get grants but their eyes are clearly on the horizon. The mind urge to do this instead in software and custom hardware is irrepressible. One needs to appreciate this to understand the levels of obsession in the keyboard rabbit hole. It's a realm where we all can be Thomas Edison.

Your results may vary, but many people conclude that the feel of Topre rubber dome switches is vastly superior to any of the hundreds of mechanical switches now available. This is an obsession killer. Why? You can't buy Topre switches separately. No one makes replacement keycaps, and the adaptors for standard keycaps impair the keyboard feel. The stock keycaps are fine but dull-looking, but what touch typist ever looks at their keyboard? Still, the majority of keyboard forum posts proudly show off custom keycaps. If one were to cannabilize Topre keycaps from an existing keyboard, the printed circuit boards are much harder to design. So one can't experiment with novel designs, such as the Dactyl/Manuform.

This leaves customizing the layout, which the Hasu FC660C Controller makes possible. Getting out of that rabbit is a separate problem, for which I don't have good advice. I had been fighting a basic urge to site my consciousness in my hands, a defining identity for our species. I wanted to be outside woodworking, rather than inside programming. Customizing my keyboard was a trick, to get my nose back in the kibble bowl. One day, I was satisfied with my layout, and I moved on. Now, I'm struggling with the urge to write a new programming language, rather than just coding the project I want to finish. I don't have hand issues requiring a more radical ergonomic design, but that may be partly that I don't move my fingers far off home row, even for modifier keys.

<comment><USER23> I built a GergoPlex a couple months ago and really enjoy it. I don't use it 100%, but switch to it for a few hours a day to get practice. I really like the chording features; "sd" for backspace, "cv" for tab, and others that I've customized using QMK to my particular tastes.

Originally the switches were Choc browns, but for this board they're a bit too heavy, and I went with gChoc switches that are super light and work better for chords and layers.

One issue I have is how low profile it is. I don't have a case or anything to raise it up and my wrists are either floating on the desk and it takes getting used to. There are some custom cases (like the gHeavy) but they're either expensive or back-ordered due to COVID.

</comment></comment><comment><USER24> NKRO?</comment>


Ask HN: IRC channels you frequent? - <AUTHOR> What are the best IRC channels for programming, reverse engineering, networking? I'm specifically interested in networking/hosting related channels, the channels I've joined are dead. <comment><USER2> Just check out popular IRC servers (e.g. Freenode), search for keywords and start joining some channels. Lurk for a while to see the tone and then get to know people.

If you want particular recommendations from HN, it would seem beneficial to search through prior posts since this topic has come up before. If you want the best options for your specific domain though, it's likely going to involve some investigation from your end to see which channels and which individuals mesh well with what you're looking for.

</comment><comment><USER3> IRC has, for all intents and purposes, been replaced by Discord. In other words, a large number of communities I frequented have re-formed in Discord, and are much more active.

There's still a few holdouts, but it's usually just 3 or 4 good-ol-boys who haven't left the server for 12 years.

It's interesting to watch contenders like gitter, slack, mattermost, riot/matrix, etc, falling to the rise of Discord, which was for gaming communities but now it's like the reddit of chat servers.

<comment><USER4> Discord is proprietary; and developers like IRC

<comment><USER5> I'm a developer. IRC is _still_ the best to me, after all these years.

Social networks come and go. IRC is _FOREVER_. ;)

I've been on IRC for just shy of 20 years. It is still the network that provides the most value and help, BY FAR.

<comment><USER6> Relevant XKCD: [https://xkcd.com/1782/](https://xkcd.com/1782/)

</comment></comment></comment></comment><comment><USER7> [https://news.ycombinator.com/item?id=17874061](https://news.ycombinator.com/item?id=17874061)

[https://news.ycombinator.com/item?id=17479863](https://news.ycombinator.com/item?id=17479863)

[https://news.ycombinator.com/item?id=17454009](https://news.ycombinator.com/item?id=17454009)

</comment><comment><USER8> Freenode has a service that lets you search channel names:

        /msg alis help

for details. I used this to find all sorts of gems:

* ##binutils for linker/loader stuff,

* ##math usually has meaty content and knowlegdeable users,

* #ircdocs is great for irc spec and implementation stuff,

* #spacex and #nasa, just because, and

* ##kernel, ##physics, ##prolog along with many others.

Happy IRCing!

</comment><comment><USER9> #LRH on irc.efnet.org is easily the best IRC channel around

</comment><comment><USER10> Stopped using IRC as a place to hangout when I was 17. I'm 42 now.

</comment><comment><USER11> On tangential note, this post reminds me of a bizarre project by London Trust Media[0]

[0][http://irc.com/](http://irc.com/)

</comment><comment><USER12> I hang out in a ton of Freenode channels...

</comment><comment><USER13> This should be an "Ask HN:" post.

</comment><comment><USER14> #python #pocoo #startups</comment>


The Chicago School of Free Speech - <AUTHOR> http://www.wsj.com/articles/the-chicago-school-of-free-speech-1472168075 <comment><USER2> Previous discussion: [https://news.ycombinator.com/item?id=12360736](https://news.ycombinator.com/item?id=12360736)

</comment><comment><USER3> Before claiming free speech, one should define what it means. Because nowadays free speech is often used as an excuse for openly discriminatory positions. Speech, just like any other action, is impactful, so just like any other action, it should be limited, and its limits should be drawn according to ethical and practical principles. Or maybe does he mean that you're allowed to write papers without citing the works of your peers, and write bullshit and still be published/get a good grade as well ? And that one could shout hateful speech in a megaphone in the amphitheatre without being bothered ?</comment>


Intel Processor L1TF Vulnerabilities CVE-2018-3615, CVE-2018-3620, CVE-2018-3646 - <AUTHOR> https://blogs.oracle.com/oraclesecurity/intel-l1tf <comment><USER2> _Disabling Intel Hyper-Threading technology in some situations._

Okay, but which situations?

Edit:

[https://www.intel.com/content/www/us/en/architecture-and- tec...](https://www.intel.com/content/www/us/en/architecture-and- technology/l1tf.html)

 _Is Intel recommending HT is turned off?

For a portion of the datacenter market – specifically a subset of those running traditional virtualization technology, and primarily in the datacenter – it may be advisable that customers or partners take additional steps to protect their systems. This is principally to safeguard against situations where the IT admin or cloud provider cannot guarantee that all virtualized operating systems are protected. Deciding to take additional steps depends on the system software in use, the workload, and the customer's assessment of the security threat model for their environment._

I wonder how many devs should think about this? Is it a risk from Vagrant, Docker, etc?</comment>


Stitcher's plan to outshine terrestrial radio - <AUTHOR> http://www.xconomy.com/san-francisco/2013/01/02/stitcher-the-pandora-for-talk-works-to-make-internet-radio-easier/

<comment><USER2> This is one of the better profiles of Stitcher that I've seen. It really contextualizes how we're trying to disrupt terrestrial talk radio (I work for Stitcher).</comment>


Six Apart Japan and Movable Type will be acquired by Infocom - <AUTHOR> http://www.sixapart.com/blog/2011/01/sixapart-japan-to-be-acquired.html <comment><USER2> Infocom?

Will we get feelies with our blog platforms now?</comment>


Activitystrea.ms the next evolution of ATOM RSS? - <AUTHOR> http://www.slideshare.net/factoryjoe/activitystreams-is-it-getting-streamy-in-here

<comment><AUTHOR> the actual site - <http://activitystrea.ms/></comment>


Gittip stats - <AUTHOR> https://www.gittip.com/about/stats.html <comment><USER2> First: I really like the idea of Gittip.com. Congratulations on getting this far.

But I strongly object to the tip amounts. Without going into too much detail, $0.25/week is not something I'd really consider a tip for a barista making my coffee, my dry cleaner, or any of the other myriad service folks I interact with on a regular basis. $0.25 will get you about 10 minutes of on-street parking in many cities; it might take longer to get your coffee. I'd guess that my attorney, physician, or my lawn guy would not consider $0.25/week a genuine "thank you" gesture as much as "hey, I have this quarter in my pocket." Personally, I think programmers are worth a lot more than most programmers think they themselves are worth.

Every other day, there's a HN post about some clueless business guy looking for technical co-founders. The HN community rages about how business people always undervalue their technical contributions. And then here's a project where programmers will tell you that a good project is worth $0.25/week. It that's true, maybe the "clueless" business guys are right to devalue technical work.

So while I really like your idea, I really do wish you'd reconsider the worth of programmers who contribute to these projects. I'm trying to be as constructive as possible here, because I think good things can come from what you're doing if you don't advance the notion that good programming in any meaningful quantity is worth $0.25.

<comment><USER3>
        $0.25/week is not something I'd really consider a tip

Should weekly tips only be given to people whose work I use weekly? If so, then yes, I agree that $0.25/week is not a good tip.

But what about the person whose Emacs config files I read at GitHub and found somewhat informative? And what about the person whose personal wiki (e.g., Org Mode files) I read at GitHub and found somewhat informative? Should I give these people good one-time tips instead of small weekly tips? Maybe I should, but I think I'd like to have the option to give them small weekly tips. Or is there some reason I shouldn't have this option?

<comment><AUTHOR> I'd like to see someone address this.

<comment><AUTHOR> Preferably on the (newly updated) GitHub ticket:

<https://github.com/whit537/www.gittip.com/issues/180>

:-)

</comment></comment></comment></comment><comment><USER4> Taking a cut of a transaction is a hard business to be in. Taking a cut of a microtransaction is an astronomically harder business to be in. Taking a cut of a microtransaction that has no reason to exist is even harder than that.

I sympathize that devs want ways to get paid for developing software, but they exist, and they start with _charging money for developing software_ and not futzing about with sub-latte level payments. Since you cannot meaningfully improve a dev's life with sub-latte payments, you will not successfully create a business on top of them.

<comment><USER5> I cannot disagree with you, and I'm not about to say "but this time it's different, it's all about the long tail!" I just want to point out, I don't think whit537 is in the business of taking cuts of transactions[0]. He's very much dogfooding here[1], and I would hazard a guess that whether gittip survives or not will depend on whether people like it enough to make it viable for whit537 to pay himself.

0: [https://github.com/whit537/www.gittip.com/issues/95#issuecom...](https://github.com/whit537/www.gittip.com/issues/95#issuecomment-6743996)

1: <https://www.gittip.com/whit537/>

<comment><AUTHOR> Amen amen.

</comment></comment></comment><comment><USER6> I know a guy working on a related idea. Instead of funding people, his system works by funding issues. The link is <http://www.freedomsponsors.org>

The workflow, as I understand it, goes something like that:

1) There's an issue on a project you use that's getting no love.

2) You go to freedomsponsors register that you'd like some issue solved, so much that you'd be willing to put your money where your mouth is. Also, you'll probably want to put a link to freedomsponsors on the project's issue tracker.

3) When one or more people solve the issue, the sponsor pays them (if more than one person has worked on the issue, the sponsor can distribute the money in anyways).

<comment><USER7> so basically sort of a reverse kickstarter for open source. That's brilliant. I love it.

I would sponsor someone fixing moving selections in gimp. I cannot figure out how to do it. There has to be a more intuitive method.

<comment><USER6> I'll tell him to put a giant Sponsor button on every screen.

First you'll have to click the tiny "Login with google" button in the upper right corner, then the "Sponsor new issue" button. Then you paste the link to the original issue. After that, the site will guide you a little bit better.

<comment><USER7> is there a way to sign up for an account, or is that planned? I don't like signing into things with google/whatever

<comment><USER8> We will add more ways to signup: facebook, twitter, yahoo and github, at least.

I'm not a big fan of owning users' passwords, so unless people really really ask for it, we probably won't add a "create login/pass" functionality. Also, there is the concept of reputation - Sponsors that pay their offers as promised will have good reputation. For reputation to mean something, we need to know something about the user, and I believe linking with a social account is our best option for that.

Anyway, talking about FreedomSponsors on GitTip's thread is probably not good for MY reputation here on HackerNews (right? I don't really know, I just signed up). So I'm gonna create a new topic and answer questions there...

Thanks for your insterest!

<comment><USER9> What about using Mozilla's Persona (formally BrowserID), instead of those commercial services you mentioned?

<comment><USER8> I didn't know about that untill now. I'll look more into it. But the main idea behind the way accounts work on FS is that the user should have some sort of online presence before he can sign up.

So if Mozilla's Persona doesn't offer a way for the user to publish some sort of online profile, then we probably won't add it as a primary form of signing up on FS. Linking an existing account to authenticate later is ok. But we do want to make it more difficult to create completely "anonymous" profiles. At least that's our thinking today anyway.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> This is going to sound pretty lame, but I've been suffering from a severe case of Signup Fatigue recently. My business has been doing pretty well so I'd be OK with committing to a weekly debit to my business card, but I don't have a GitHub account and Signup Fatigue is preventing me from setting one up just to give money to people.

Am I being excessively whiny? Are there any plans to not require a GitHub account to give people a little money?

Really neat effort, I hope it continues to take off.

<comment><USER11> Why not Flattr?

<comment><USER12> <http://news.ycombinator.com/item?id=4239336>

</comment></comment></comment><comment><USER13> Those 319 people (74%) are contributing only $79/month. If minimum contribution was raised to US$1 it would generate more donations even if >60% leave.

<comment><USER14> I'm kind in agreement about this. $0.25 a week really is tiny.

<comment><AUTHOR> Hmmm ... for the record, here's the thinking that led to the current tip structure:

[http://blog.gittip.com/post/26505682007/is-personal- funding-...](http://blog.gittip.com/post/26505682007/is-personal-funding- viable)

<comment><USER15> FWIW, I'd love to see a $1.00 option added to the mix. 25¢ is too small, $3 is too much, $1 is just right. :-)

<comment><USER16> Agreed, 25c is too small, min should be $1. Also I think the sweet spot is between $1-6, IMO there should be multiple options in the middle here.

<comment><USER13> Yeah, the current pricing jumps from $1 to $12 a month, $1 to $4 is $4-$24, quite a gap. Honestly, I'd also prefer monthly contributions, thinking in week increments is an american thing. Credit card statements and salaries come in monthly for most of the world, I think.

</comment></comment></comment></comment></comment></comment><comment><USER17> All the people telling you to increase the price are right, and then some. So some advice:

\- dont focus so much on the individual developers, focus more on getting money in the system. You want to know how best to help an individual developer? It's by having lots of users, and lots of money.

\- first way you can help do this is by having a minimum monthly charge. I pay $150 a month on a database. I pay 4 digits worth of EC2 bills. I can afford a $10 minimum.

\- 2nd way is to strongly focus on enterprise plans. For $1000 a month, you get a big giant banner add on gittip, that everyone else sees every time they add a tip or receive one, or play with their settings.

\- dont make people spend the whole thing - distribute the excess to the developers that others have chosen, in proportion to how popular they are. Or use it to highlight projects you like, or projects for people who contribute money, or something like that.

Stop focussing on the low end - bump up that price, and who cares if it's too expensive for some people. Giving $0.25 isn't charity - its actually costing you money.

<comment><AUTHOR> A +1 added to <https://github.com/whit537/www.gittip.com/issues/180>

> minimum monthly charge

Newly ticketed: <https://github.com/whit537/www.gittip.com/issues/185>

> strongly focus on enterprise plans

Existing ticket: <https://github.com/whit537/www.gittip.com/issues/106>

> distribute the excess

Similar existing ticket:

<https://github.com/whit537/www.gittip.com/issues/93>

<comment><AUTHOR> Thanks for the feedback. :-)

<comment><USER17> No problem. You're doing something cool, great job!

<comment><AUTHOR> :-)

</comment></comment></comment></comment></comment><comment><USER18> The stats page should have permalinks for each week so that if someone clicks on this HN submission in 1 month they can see the intended stats, not the latest.

<comment><AUTHOR> Agreed. Just in case, here are some quick numbers so far, for the record:

Week 1: $24.80 / 12 people

Week 2: $4.24 / 7 *

Week 3: $30.08 / 19

Week 4: $110.64 / 44

Week 5: $379.56 / 71

Week 6: $615.66 / 98

Week 7: $600.99 / 110

As you can see, we slowed this past week, to be honest. Looking for advice on getting back to ~doubling each week. :-)

* In week 2 our payment processor died. We were expecting $55 / 26. See also:

[http://blog.gittip.com/post/25565694263/how-not-to-get- acqui...](http://blog.gittip.com/post/25565694263/how-not-to-get-acquired)

<comment><USER15> _As you can see, we slowed this past week, to be honest. Looking for advice on getting back to ~doubling each week. :-)_

I'm afraid you're probably going to be looking at a plateau until you can get the payments thing straightened out permanently so that recipients can depend on getting an ACH every Friday (or Monday, whatever). Closing that loop will instantly motivate every open source hacker to put a "Help fund my work @ Gittip" button everywhere. Bummer the BalancedPayments option didn't pan out.

<comment><USER19> _Bummer the BalancedPayments option didn't pan out._

It's panning out; we're in the process of helping him with the integration.

Also, the "once a week" payout period is unique to Gittip's model (maybe because Stripe requires a 7-day wait?). With Balanced, a marketplace can pay out as frequently as it's earning: even daily. So that's an option Gittip can consider in the future.

<comment><USER15> Ah, sorry; last I saw, he was heading towards Braintree with a a 6-month horizon for the various merchant account approvals, etc. I hope the current path works out. :-)

<comment><AUTHOR> Yes, new cards are going into Balanced as of this morning. Hoping to implement payouts for people in the US in the next week or two:

<https://github.com/whit537/www.gittip.com/issues/22>

</comment></comment></comment></comment></comment></comment><comment><USER20> This is really cool. I look forwarding to support people in the Django community who make invaluable apps through this.

I would prefer if I could reward people for specific repos, though. Otherwise people, myself included, will probably be clueless why I am receiving money, and as a result it won't create an incentive to improve a particular product.

The text could read.

        Hey, I just met you,
        And this is crazy,
        But here's my money,
        So keep working on django-registration maybe.

<comment><USER21> I like the fact that the donations are per individual, rather than per project. It gives the recipients the freedom to explore other opportunities, rather than tying them down to their one popular project.

<comment><USER20> I just want it there as an optional addition. People should totally be able to donate to the general GitHub account.

</comment></comment></comment><comment><USER22> What exactly are the economics of sub-dollar charges? I'd imagine the CC charges exceed donation amount

<comment> a-priori Gittip charges fees of $0.30 + 3.9%. They say that they use Balanced Payments, which charges $0.30 + 2.9%, so Gittip is taking a 1% commission on all money that enters the system.

For the sub-dollar charges, keep in mind that the fees are levied for the _sum_ of your tips (assuming no one tips you). So if you only tip one person $0.25 then it's highly inefficient, mostly a 'tip' for the credit card companies.

At 1x$0.25 tips, this structure has an effective overhead of 124.9%; at 10 tips its overhead is 15.9%; at 100 tips, it's 5.1%; and at 1000 tips it's 4.0%.

<comment> jack-r-abbit One solution would be for Gittip to implement something like an account balance. Say I charge my Gittip Wallet with $25 seed money. My CC gets charged $25 + $.30 + 3.9% ($26.07). I set up $1.75/week of tips to various people. My Gittip Wallet gets dinged $1.75 per week for 14 weeks ($24.50). On week 15 when it comes time to give my $1.75, my Gittip Wallet is too low. So it gets another $25 bump (my CC gets hit with another $26.07) and we're good to go for a bit. Wash. Rinse. Repeat. Make the auto-bump amount a user selection in case someone is gifting $30 a week, they could auto-bump at say $100 a pop.

Edit: Also, if I understand correctly, if I am getting tips, that would also go into the same wallet. So in theory the tips I receive would extend the amount of time between auto-bumps. Or not at all if I'm getting more than I am giving.

<comment><USER23> That's exactly how the electronic tolling plans I am familiar with work, which is what makes them cost effective.

<comment> jack-r-abbit yes. In SF Bay Area we have FasTrak for bridge/road tolls. Clipper card for public transit. For each, you keep some balance in your account that gets depleted as used and is recharged as needed.

<comment><AUTHOR> E-ZPass here in PA.

<comment><AUTHOR> Ticket is: <https://github.com/whit537/www.gittip.com/issues/167>.

</comment></comment></comment></comment></comment></comment></comment><comment><USER24> This is good to hear. I'm cautiously optimistic about this project and hope we see numbers grow a few orders of magnitude in the coming year.

<comment><AUTHOR> :-)

</comment></comment><comment><USER25> As the creator of FuckItJS, I have unfortunately fallen short of my "$1,000,000.00 per week" goal

<comment><AUTHOR> A little love, people! :-)

<https://www.gittip.com/mattdiamond/>

</comment></comment><comment><USER26> I suspect I'm not alone in saying this: If someone paid me, I'd be able to contribute a lot more to open source projects. As it stands, I am inclined to worry that sharing my source code will undermine my ability to make money while writing it, and this might not be an ideal situation.

</comment><comment><AUTHOR> Gah! Title changing feels gross. :-(

Original: "After 7 weeks, 100+ people are funding open source devs $600/wk on Gittip.com"

Problem: <http://news.ycombinator.com/item?id=4102013>

</comment><comment><USER27> Awesome, I signed up, but how do I change my username? I don't want to use the one supplied to me by logging in via Github.

<comment><AUTHOR> Not supported atm, sorry. :-(

Ticketed: <https://github.com/whit537/www.gittip.com/issues/181>

<comment><USER27> Chop chop! :P I'll keep an eye out. Thanks.

</comment></comment></comment><comment><USER28> Thanks for this tool, I prefer this over Paypal donation! (thanks for all the tips in advance :) <https://www.gittip.com/suyashjoshi/>

</comment><comment><USER11> Excuse me, but I don't understand why anyone would use this instead of flattr? Sorry for sounding blatant, but havent they got this micro-donation thing going quite well?

<comment><AUTHOR> Here's my take: <http://news.ycombinator.com/item?id=4239336>

</comment></comment><comment><USER29> I like this, and I just signed up! Do you think tipping is something that GitHub might want to add as a core feature of their web site?

<comment><AUTHOR> Can't speak for GitHub, of course. They did try an experiment with Pledgie a few years back, but that fizzled:

<https://github.com/blog/57-getting-paid-the-open-source-way/>

</comment></comment><comment><USER30> Signed up for fun! I still don't see the tip buttons once I show a widget to a viewer.

<comment><AUTHOR> Hmmm ... example I can look at?

<comment><AUTHOR> P.S. Started a ticket to track this: <https://github.com/whit537/www.gittip.com/issues/177>

</comment></comment></comment><comment><USER31> Do you support organizations?

<comment><AUTHOR> Short answer: No.

Long answer: Trying to figure out the best way to do that. See:

<https://github.com/whit537/www.gittip.com/issues/27>

<comment><USER16> I would also like to see Organization support. i.e. 1 payment destination to fund the whole core team as they see fit.

<comment><AUTHOR> I've added +2 to the ticket.</comment></comment></comment></comment>


OpenNews: 24 Hours to Choose Your Own Adventure. - <AUTHOR> http://sinker.tumblr.com/post/29130883993/opennews-24-hours-to-choose-your-own-adventure

<comment><USER2> Hi there. I'm the director of the Knight-Mozilla OpenNews program. We're looking for developers, hackers, and the like who want to spend 10 months traveling the world and building new tools for journalism. It's an incredible opportunity.

This link is for the final pitch, as the application window closes at midnight Eastern tomorrow night.

Happy to answer any questions here. Also, you can jump straight to the application if you want: <http://mozillaopennews.org/fellowships/apply.html></comment>


How Is Critical Life or Death Software Tested? - <AUTHOR> http://motherboard.vice.com/en_uk/read/how-is-critical-life-or-death-software-tested <comment><USER2> I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception).

For external hardware inputs, we had software simulating the messages when we couldn't get the real equipment in the lab. We were writing software for a radar that didn't yet exist.

And it was tested and tested and tested and tested.....

The when the pieces came together in integration. They tested and tested. An when they built the physical radar, the software worked. A few configuration things here and there needed adjusting.

Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running.

Was it bug free? Probably not. You can't test every possible scenario. But we took the work seriously. It took longer to get code done than any other environment I've been in, but the code ended up very solid quality.

<comment><USER3> > Actually testing was built into the software. When it came up it would talk > to the physical parts to make sure everything was communicating ok before it > could start running.

I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are performed. At most, there's some sort of metrics and server monitoring, but nothing in the actual software.

It's a work of fiction, but Star Trek has these "diagnostics" that are run, with several levels of correctness checks, for pretty much everything. In your typical app, it could be useful to ask it to "run diagnostics", to see if everything was still performing as expected.

<comment><USER4> Early in my career I worked on military flight data recorders, including the development of the software for the F-22's "black box". Those systems have SBIT, IBIT, PBIT and MBIT sub-systems were BIT is "built in test" and S = startup, I = initiated, P = periodic and M = maintenance. I remember making the Star Trek diagnostic joke myself when I was assigned the SBIT work.

Each BIT does varying level of testing based on it's runtime budget but there are a lot of very basic tests that don't make much sense until you see your first field report of some register bit "sticking". Its much better to ground a plane that can't add 1+1 than to find that out in mid-flight.

<comment><USER5> What types of errors would cause one of the tests to fail? Is it mostly testing for hardware errors, or are there any software logic errors that could make it to production, but be caught by one of the tests several months down the road? The only software related items I can think of are edge cases where a built in test is based on real time input. Kind of like running the calculations through multiple independent implementations of the software.

<comment><USER6> There's actually a decent probability of memory corruption in space applications due to radiation. So in addition to checking communication across busses, application checksums are typically run continuously.

</comment></comment></comment></comment></comment><comment><USER7> My experience with testing avionics controllers was: Everyone seemed to have the correct idea (that bugs were basically Not Allowed.) The company set up enough testing so that those bugs were eventually eliminated. However, the difference between good projects and bad projects was mostly the amount of time and money that this took.

My favorite was one where we had an entire test harness written in Python that could completely control the operation of the device being tested in a way that resembled human input. Code was first written in Ada against a monolithic requirements document, while testers wrote their standard test cases against the same document. After the exhaustive amount of testing that took place by developers, testers themselves had the freedom to create contrived test cases that might have escaped the attention of devs (What if we just turn the machine on and off 10 times, because why not?).

This had the advantages of a formal software process as well as the ability to exploit human creativity. It also led to me losing a bet that a doppler radar can't be fooled with an empty potato chip bag.

<comment><USER8> Care to elaborate on the potato chip bag story? Sounds intriguing.

<comment><USER9> My wild guess loosely based on my time spent writing software to analyze observed radar signals: I believe the parent probably opened the bag along the seams to get one flat piece of reflective (on the inside of the bag) material. I know weather balloons are often reflective so they can be tracked by radar. I'm guessing a full-size potato chip bag opened flat would be big enough for the radar to see, couldn't tell you if that's what the parent meant by fooling the radar or if something else was done to the bag to make the radar interpret it as an aircraft.

<comment><USER7> This is pretty much it. My boss was definitely a level 99 duct tape programmer, and this was one of the many examples of his ingenuity.

</comment></comment></comment></comment><comment><USER10> Worked on anesthesia machines for a few years. Since both hardware and software is involved, the testing was quite extensive.

* Lots of manual testing. While we did unit testing and some automated integration testing, most defects were found using exhaustive manual testing by trained engineers.

* Randomized UI testing. Used UI automation to exercise the UI with various physical configurations of the system. Would often run this overnight on many systems and analyze failures every day.

* Extensive hazard analysis. Basically, we wrote down everything that could possibly go wrong with the system (including things like gamma radiation), estimated the likelihood and harm, and then listed mitigations. The entire system could run safely even if there was full power failure. "Fail safe"

* Detailed software specifications, each of which was linked to manual test cases. Test cases were signed off when executed.

* Animal testing for validation. We went to a vet school and put a bunch of dogs under and brushed their teeth.

* Limited release for production. We would launch the system at one or two hospitals and monitor it for a few weeks before broader release.

<comment><USER11> You just have to look at Therac 25 for the risks of relying on software interlocks alone. One of the prevailing pieces of feedback was the lack of _hardware_ interlocks - whether that's possible on anesthesia machines I don't know…but the prevailing wisdom is to use/include hardware interlocks wherever that's feasible, for any critical life-supporting equipment.

<comment><USER10> Not sure what you mean by "interlocks", but the hardware was quite distributed. Each critical component had its own board and industrial microcontroller. And we had various levels of watchdogs keeping track of system health at all times.

<comment> david-given Interlocks are usually fairly crude safety measures, normally in hardware, to make sure that particular combinations of events cannot happen.

The Therac-25 is a famous comp.risks cautionary tale. Among the many, many design misfeatures (if you haven't come across it, it's worth a read) was the one that killed people:

It was capable of providing two kinds of radiation therapy; electron beam radiation and X-ray radiation. It worked by having an electron beam generator which could be operated at either high power or lower power. Low power was used directly. High power was only used to irradiate a tungsten target which produced X-rays. (I'm simplifying here.)

You can probably guess what went wrong; people were directly exposed to the high power electron beam. Several of them died.

The obvious interlock here (which apparently previous versions had) was to have a mechanical switch which would only enable the high-power beam when the tungten target was rotated into place. No target, no high power. Simple and relatively foolproof (although it's possible for interlocks to go wrong too).

<comment><USER12> Really, every engineer should read the report[1] from the Therac-25 investigation. I would hope anybody that is working on anything that could be potentially dangerous has _already_ read it.

There problems in the Therac-25 went a lot further than just the bad design of the target-selection, which _had_ an (badly designed) interlock. It checked that the rotating beam target was in the correct position to match the high/low power setting (and NOT the 3rd "light window" position without _any_ beam attenuator).

While many design choices contributed to the machine's problems, you could probably say that two big design failures lead to the deaths associated with Therac-25. One was this interlock, which _failed_ if you didn't put it in place (there was no locking mechanism, either, just a friction stopper). If the target was turned slightly, the 3 micro-switches would sense the wrong pattern (bit shift)... which was pattern for one of the OTHER positions.

There was also a race condition in the software that would turn on the beam at a power _MUCH_ higher than it is ever used. This race was only triggered when you typed in the treatment settings _very quickly_ , which is why the manufacturer denied there was a problem: when they tried to recreate the bug by carefully - that is, very _slowly_ \- following the reported conditions, it never failed.

Therac-25 is an incredibly powerful lesson in what we mean by "Fail Safe", and why it is absolutely necessary to have _defense in depth_. Fixing the target wouldn't have fixed the race condition power-level bug. Fixing any of the software wouldn't have fixed the bad target design that could be turned out of alignment. Oh, and they had a radiation sensor on the target (which could shut off the machine as _another_ independent layer of defense... but they mounted it on the turnable target, so the micro-switch problem allowed the sensor to be moved away from the beam path.

The really telling thing, though, is how the _previous_ model acted. It was not software controlled, and was an old-style electromechanical device. It turns out the micro-switch problem existed there as well (among other problems)... and it would _blow fuses_ regularly. Which was yet another layer of safety. It turns out that when they upgraded it to a software-based control system, they got cheap and took out all those "unnecessary" hardware interlocks and "redundant" features. There is a lot of blame to go around, but this is where I put most of the responsibility. You never assume one (or even a few) safety feature will work - the good engineer assumes it will all break at any moment, and makes sure that it will _still Fail Safe._

> (although it's possible for interlocks to go wrong too)

If there is one lesson to learn from the Therac-25, this was it. Things break, mistakes happen, and when you're building a device that shoots high-energy x-rays at people, you need to assume that everything _did_ go wrong, and make sure the rest of the device can safely handle that situation.

[1] [http://sunnyday.mit.edu/papers/therac.pdf](http://sunnyday.mit.edu/papers/therac.pdf)

<comment><USER11> _" The really telling thing, though, is how the previous model acted…it would blow fuses regularly"_

Good. When a fuse blows, it shows something is wrong, and needs fixing. Replacing the fuse with a nail or something else that doesn't blow is a sure- fire way to set the thing on fire. Bad enough for a desk-lamp, a little worse for radiotherapy machine.

Sounds like people were irritated by fuses blowing, and decided to simply short-circuit the fuses instead.

<comment><USER12> The people _using_ the machine a the hospital would replace the (expensive) fuses when they blew. It was the _manufacturer_ that made the later model (the Therac-25) that didn't have the fuses (and other "old" hardware features).

Obviously, something was still very wrong. User error (or other bugs? I'm not sure) in the older hardware and the infamous race condition in the software- controlled Therac-25 was causing the beam to turn on some shockingly high amount of power. The better design of the older models _saved people 's lives_ by simply blowing fuses when the power went too high.

You could, perhaps, blame the poor communication between the hospitals and the manufacturer, because the fuse problem _should_ have cause a bit of a panic among the engineer who designed the machine.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> > As for the code itself, its perfection came as the result of basically the > opposite of every trope normally assigned to "coder." Creativity in the > shuttle group was discouraged; shifts were nine-to-five; code hotshots and > superstars were not tolerated; over half of the team consisted of women; > debugging barely existed because mistakes were the rarest of occurrences. > Programming was the product not of coders and engineers, but of the Process.

Serious question: where do I get a job like this? It's my dream way of programming professionally.

<comment><USER2> Large corporations with government contracts have this kind of work. Probably financial institutions too.

The coding proces is slower than most people are used too and can become frustrating.

<comment><USER14> Financial institutions do not necessarily do things this way. Some parts might, but I don't have any experience with them. The parts I _do_ have experience with it is utterly a miracle that anything works.

<comment><USER15> > The parts I do have experience with it is utterly a miracle that anything > works.

Is my feeling about everything industry I've worked in.

The stuff that runs telecoms (mostly billing side) particularly is the stuff of nightmares.

<comment><USER16> Seriously? _Billing_ code? I would have imagined that code would be so subject to customer complaint that it would be forced into quality.

<comment><USER17> I've worked in a support team for a telecom billing system and I was tasked with interacting with our development team to investigate bugs in production (and eventually moved to the development team). These systems are created just like any other commercial system, without any formal proofs and minimum requirement docs. To make things worse, they have to be flexible enough to support all and any billing plans that the business might come up with, so there is a lot of moving parts.

As other people have said here, nobody wants to touch it. Developers would often limit themselves to fix just a small portion of the code even though they thought the overall system could be improved in many ways, for fear of breaking something, causing a few million dollars of damage and getting fired. There was no assurance that any part of the systems should work like this or that.. only some vague expectations.

You're right, that would be a systems that should be built from scratch with that kind of concern but unfortunately it's not.

<comment><USER18> It is a nice challenge how to rewrite such a system. One way would be to build a parallel system, identify all inputs and duplicate them to the parallel solution and then compare the outputs; in case of discrepancies fix the erroneous system. Once the systems produce same results (or once the new system produces better results than the old one) you just switch the systems.

The rewrite doesn't have to be complete; it can (should) be done in pieces of course.

<comment><USER15> "Replace in place" is fairly common.

The big issues is duplicating the system while it is still morphing in production for all the edge cases, it often feels like trying to paint a moving bus.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> I'm currently writing software that is non-critical for satellites. It's non- critical in the manner that if we get things wrong our company will lose millions of dollars but the satellite won't burn any resources that it can't get back.

We are currently porting code over from C++ to a C# system with parallel computation. The current system has been flying for a long time but has no testing and is tied to a bad UI. So we are re-writing.

That said, accuracy is number one. We have a pretty solid method for testing so far. We know have some robust input scenarios and we know that we want to get a specific output. So we are able to do fairly robust automated "regression" testing. If the numbers don't match then we have an issue, and we have to fix that before moving on.

After every validation that the new code gives us acceptable margins of error we wrap it up with unit tests so that we can then modify the code to try to optimize. Our testing is integrated from the highest level of the code down (I know backwards) but that's how we know we can validate the input.

We have a lot of testing and a long schedule. If this weren't critical software we'd have a much shorter turn around on what we are writing. We also work very closely with subject matter experts on every change we make. We have a guy who's been working with this software (and the subsequent theory) for 20 years. He's open to change, but he also validates everything so we don't accidentally change the output when we're optimizing.

<comment> david-given C# is an odd choice for avionics software, surely? It's got a really heavyweight runtime and nondeterministic behaviour due to the garbage collector. I assume this is the satellite's application layer and isn't real time?

...and what are you running it on? I wasn't aware of any embedded operating systems which supported C#!

<comment><USER20> I'm guessing its actually running on a ground station, providing commands or analysing data from satellites, hence why its non-critical and the need has arisen for parallel computation, presumably to speed it up.

Many ARM embedded systems can support C# with the open source .net micro framework ([http://www.netmf.com](http://www.netmf.com)), which doesnt require any OS, and was originally developed for Microsoft's SPOT watch. I haven't used it myself, and I agree with you that it doesn't sound, ideal at first for realtime apps, but none or soft realtime embedded applications are common too.

<comment><USER19> Yeah it's a ground station application. So it isn't running on any embedded hardware. We'd have done something different for that.

Performance isn't the only thing we are considering. We want to get improved performance, but our old analysis code was extreemly hard to maintain, so that went into the decision as well. Honestly, I'd probably pick another language, but I wasn't on the project when it started.

I personally would have liked to do this with F# because of how functional it is at it's core, but that's cuz we have a lot of Microsoft expertise in house.

Also one thing with the engineering apps is that anything where engineers (not software engieners) don't have to learn a new language is going to be an easier sell.

</comment></comment></comment></comment><comment><USER21> The story of the Boeing engineers flying on the test flights is a perfect example of "skin in the game" (from Antifragile by Nassim Nicholas Taleb).

Here's what I wrote about that in a blog post on Antifragility and SW development:

At the end of the book, there is a chapter on ethics that Taleb calls "skin in the game". To have skin in the game, you should share both in the upside and downside. Taleb quotes the 3,800 year old Hammurabi's code: "If a builder builds a house and the house collapses and causes the death of the owner – the builder shall be put to death". It is interesting to view this from a software development perspective. I have never worked on software where people's lives were in danger if the software failed, but I would not be willing to submit to Hammurabi's code if I did. But I think a little less extreme form of skin in the game is actually very good. Being on call for example. If the software you wrote fails, you may get called in the middle of the night to help fix it. I have been on call at most of the places I have worked in the past, and I think it has a lot of benefits. It gives you an incentive to be very thorough, both in development and testing. It also forces you to make the software debuggable – otherwise you yourself will suffer. Another way of introducing skin in the game is dog-fooding – using the software you are developing in your daily work. I have never worked on software that we have been able to dog-food, but I think that is another great practice.

[http://henrikwarne.com/2014/06/08/antifragility-and- software...](http://henrikwarne.com/2014/06/08/antifragility-and-software- development/)

<comment><USER22> The examples you are giving are all about sharing the downside. What are the options for the upside?

</comment></comment><comment><USER23> It's essentially an analog of the concept of tolerance in the physical world of manufacturing and assembly. The less your tolerance for error the more formal and carefully controlled the process, and the more money spent in testing, verification, feedback, and improvement.

And yet you can still measure one value in the metric system and another in English units and drill a smoking hole in Mars. It was sort of striking to read Charles Fishman's statement about the software being bug free, followed immediately by the supporting fact that the last three versions had one bug each. If they had one bug, how are you 100% sure they didn't have two?

<comment><USER24> > It was sort of striking to read Charles Fishman's statement about the > software being bug free, followed immediately by the supporting fact that > the last three versions had one bug each. If they had one bug, how are you > 100% sure they didn't have two?

I think you're being overcritical here. It's really really really difficult to reach 100% in any real form of measurement. So I think when they mean "bug free", they probably mean the chance of a bug is below some threshold of probability. The famous six sigma rule comes in mind.

I do grant you that this is not specified explicitly in the article, but they do say: "each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors."

<comment><USER23> >> So I think when they mean "bug free", they probably mean the chance of a bug is below some threshold of probability. The famous six sigma rule comes in mind.

No doubt, and that was in fact my point. I think most of us would be very reluctant to use the phrase "bug free," and in this case his statement was obviously meant to be in stark contrast to that reluctance.

<comment><USER24> I think this is more of a technicality that's very subjective. I feel that "no" depends on the probability of something happening as well as how catastrophic it is.

For example, after buying a lottery ticket, I can pretty comfortably say that I'm not going to win. Sure, there is a chance, but is it really going to happen? No.

As another example, if I put down a waterbottle on my bike seat, there's a 1% chance of the water spilling onto the sidewalk. At this point, I'm also comfortable saying that it's not going to happen. However, if the waterbottle has a 1% chance of exploding when I put it down, I don't think I'll be comfortable saying that it won't happen. The chance of that must be much much lower before I can accept it.

I would like to give the person who said that the benefit of the doubt. I'm sure they understand the implication of bug free, but it's likely just easier to say that rather than explaining stats/tradeoffs to a journalist.

</comment></comment></comment></comment><comment><USER25> For people interested in the idea of formal verification, you may want to look at TLA+ ([https://en.wikipedia.org/wiki/TLA%2B](https://en.wikipedia.org/wiki/TLA%2B)) and PlusCal ([https://en.wikipedia.org/wiki/PlusCal](https://en.wikipedia.org/wiki/PlusCal)), which have been mentioned on HN before.

They're a specialty system for writing code (and mathematical proofs) where every possible system behavior for a given range of inputs can be examined for safety (outputs within allowed ranges with no unexpected behavior) and for liveliness (the expected progression from one output to another).

</comment><comment><USER26> It's all just a question of cost.

We know how to write software that comes arbitrary close to perfection. But as defects asymptotically approach zero, cost skyrockets.

The interesting question is what technologies can bend that cost/quality curve.

<comment><USER27> This is why this discussion sometimes frustrates me. A lot of the defects we have are because you aren't willing to pay for the sort of software that wouldn't have defects. It's natural to read that as a sort of cynical accusation, but instead, I mean it straight... you really aren't willing to pay what it would take, and you shouldn't be. A $1000 Facebook-access app for your phone (that still somehow has some sort of economies of scale going for it, but that's another discussion) might not crash on you and might take a lot fewer phone resources, but there's no way it's going to be $1000 better than what we get today for free for the vast bulk of users.

On the flip side, the cavalier attitude developers who are on the very, very low side of the curve, where huge quality improvements can be obtained for very cheap, towards those cheap practices also frustrates me. What do you mean you can't be bothered to run the automated test suite that I have handed you on a silver platter on your build server? Are you serious? How can you pass up something so cheap, yet so powerful? And I don't just mean, how can you not be bothered to set it up, etc... I mean, as a professional engineer, how can you justify that?

<comment><USER11> "What do you mean you can't be bothered to run the automated test suite that I have handed you on a silver platter on your build server?"

As devil's advocate, why not just run this for me (e.g. on every commit/every push)? Much like the web usability ethos "Why make me think" \- why make me work? The lower the barrier to testing - ideally zero, it just happens without the dev having to do anything - the more testing will happen.

I don't often get the chance to set things up this way, but when I do, each dev works in their own git branch, and sends a pull-request with their changes. The test server(s) then run the complete test-suite on the branch, and either note the PR with "Tests passed" or emails the dev with "Tests failed" and the reasons. Devs don't need to think about running tests, reviewers/release managers don't need to even consider PRs until the "Tests passed" message shows up…saves time and effort for everyone, and improves code quality. The cost is simply the initial setup time.

<comment><USER27> "As devil's advocate, why not just run this for me (e.g. on every commit/every push)?"

In my real-life experience with a multiple-team environment, which is where the question came from, _my_ running the tests doesn't do any good if you're going to consider it "my" server and simply _ignore the results_.

The key point here isn't a technical one. The key point here was, as an _professional engineer_ , how can you justify not taking such a great bang/buck quality option that will far, far more than pay for itself? Explaining how I can be a professional engineer on your behalf more than misses the point.

</comment></comment></comment></comment><comment><USER28> No discussion of mission critical software is complete without mentioning Margaret Hamilton:

[https://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist...](https://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%29)

</comment><comment><USER29> The article bit misleading, almost suggests that software correctness is achieved by testing, while this is definitely not true. Just for the record, none of the other technical fields using testing for safety critical engineering products as the main solution to ensure safety. This is not too different for software components in mission critical use cases. There are several ways to build a reliable system out of unreliable parts (like combining 3 different units by the 2 out of 3 principle, etc.) but testing is just not the way. It is a nice to have thing.

</comment><comment><USER30> I think the article might be mistaken about one point: "For one thing, the Boeing approach is going out of style or has mostly gone out of style, according to SE poster Uri Dekel (handle: Uri), a Google software engineer."

It absolutely has not gone out of style in avionics software engineering. As a person who writes software for avionics, I can say that extensive design reviews at every step combined with rigorous testing is exactly how we build software. That's how its done at every avionics software company I've ever worked at (3 so far). Formal methods are generally still too cutting-edge and complicated for many people in this industry.

So maybe Google doesn't bother with design reviews, but those of us writing life-or-death software definitely do.

<comment><USER31> He meant the practice of sending the software engineers up on the first flight.

</comment></comment><comment><USER32> I've been wondering about this lately. My stepdad has heart failure and they put a heart pump in his chest. It regulates blood flow and settings can be changed, etc. Leave it to a dev to think "How much testing has gone into this thing?" Even one minor slip-up in his blood flow, either too high or too low, could mean a stroke and possibly death. Or, God forbid, the thing crashes somehow and stops working.

That machine is much easier to code than a space shuttle obviously, but I still wondered about it. The tech has to be rock solid. Even one malfunction could cause so much despair in a family, and could also cost your company millions.

<comment><USER33> I don't know much about heart pumps, but I do have an implanted pacer/defibrillator, and I'm currently not too happy with the people who made it.

I've been told by my cardiologist (and engineers working for the manufacturer) that it doesn't "fail safe" if the battery level drops too low to keep the device running (which is inevitable if it isn't replaced after 7-8 years, but it can and sometimes does happen prematurely and without warning).

In that situation, not only can it suddenly become unable to correct an arrhythmia (as expected), it could actually _cause one_ all by itself, or pace above 200BPM for no reason.

No one I've talked to in the healthcare industry seems at all surprised about this for some reason. They just started monitoring it more often the closer it got to the "replace me now" indicator level.

</comment></comment><comment><USER34> I think in the "normal" software industry we have a skewed picture of quality, simply because it's not the primary focus. For everyday software, it's fine if it works to 95%. Even bugs you _have identified_ must be weighed against new features, and features often win. For the customer it's better to have a piece of software that has all the features they need, but calculates a bad result every 1/1000 times, or crashes a couple of times per day, than a program that doesn't have all the features. It's also the "release early release often" thing where the cheapest testers are your end users. That is far from the "release once patch never" of rockets. So feature bloat and suffering quality isn't really due to bad practices, it's an active choice.

</comment><comment><USER35> Could have at least mentioned DO-178C

[http://en.wikipedia.org/wiki/DO-178C](http://en.wikipedia.org/wiki/DO-178C)

</comment><comment><USER36> I worked on an emr.

Each developer got between 1/2 to 3/2 qa people, in addition to dedicated qa engineers. You had to submit detailed test plans for features. Surprisingly, the company didn't expend much effort on unit tests -- they were there, but not heavily emphasized.

</comment><comment><USER37> If Amazon can do it, then surely an airplane manufacturer can do it [1]

[1] [http://cacm.acm.org/magazines/2015/4/184701-how-amazon- web-s...](http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-services- uses-formal-methods/fulltext)

<comment><USER38> that Amazon paper is talking about design/algorithm verification using formal methods. This does NOT verify that the design or algorithm in question is actually implemented correctly and matches the design perfectly.

<comment><USER39> And as I recall, TLA+ doesn't tie into code generation which doesn't help with the verification process.

</comment></comment></comment><comment><USER40> It depends on the software budget and managers. When I worked a company one the CEO's had an idea about adding a camera to road semaphores so that cars wouldn't stop is the road is empty. I asked: What happens if the sun light hits the camera too much? The guy laughed in my face and told I was being ridiculous. I left the company some months later for other reasons but It was pretty scary to me to hear such words at that time. If you're writing software that might cost someone else's life, the budget shouldn't be a limitation nor time. I'm not aware of any software security regulation, If I'm going to release critical software, Does some entity exists to validates my code? This kind of stuff should exist to block companies that only see the profit of getting a contract.

<comment><USER11> Genuinely, in road safety, the company typically considers and weighs the cost of fixing an issue, vs the cost of lawsuits in the event of death/injury.

One paper on the topic talks about the Ford Pinto fuel system design: [http://users.wfu.edu/palmitar/Law&Valuation/Papers/1999/Legg...](http://users.wfu.edu/palmitar/Law&Valuation/Papers/1999/Leggett- pinto.html)

The GM ignition-switch recall also sparked a similar debate: [http://en.wikipedia.org/wiki/2014_General_Motors_recall](http://en.wikipedia.org/wiki/2014_General_Motors_recall)

So it's not uncommon that economics outweighs risk-to-life in a lot of businesses.

<comment><USER41> Which is one reason people should be a lot more suspicious of "tort reform" than they are.

</comment></comment></comment><comment><USER42> Additional reading material: some of NASA's own rules for safe code (with explanations of each! love the stack one, free memory management :P).

[http://spinroot.com/gerard/pdf/P10.pdf](http://spinroot.com/gerard/pdf/P10.pdf)

<comment><USER43> JPL also published a C coding standard, which details language constructs that one should and shouldn't use in a mission critical embedded system. Some of the rules make a reappearance there (the "Power of Ten" article is mentioned in the introduction).

[http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf](http://lars- lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf)

</comment></comment><comment><USER44> The kind of process needed to achieve high quality is just not that fun. Most of the programmers I know would run screaming complaining about the bureaucracy and red tape.

</comment><comment><USER45> Not sure if true, but I heard a couple of researchers in my uni wrote some code for Boeing. They wrote the code in Prolog. I assume because it's easier to formal test it. And they didn't trust the compiler, so they had to check the generated machine code line by line. Apparently.

<comment><USER46> If it was DO-178B/C Level A, then yeah, they'd have to inspect the machine code. Regardless of trusting the compiler.

And the compiler would have to be inspected too.

</comment></comment><comment><USER47> "There's a serious move towards formal verification rather than random functional testing," he writes. "Government agencies like NASA and some defense organizations are spending more and more on these technologies. They're still a PITA [pain in the ass] for the average programmer, but they're often more effective at testing critical systems."

this part is kinda cool because there's also always been a movement for toward this for opensource/commercial security software, at least for the kernel. Of course, we have no such thing today (as in i doubt anyone here runs such a system in production), but the interest is there.

</comment><comment><USER48> It's true that writing software requires only computers, and that's it would be too expensive to test it in real situations, when the stakes are high maybe it's also important to do live testing ?

<comment><USER49> Hardware in the loop (HIL) is a sort of middle ground, where all the expensive stuff that breaks are replaced by ordinary computers, but the actual control hardware is not.

Corporate video by people who do this, explaining it: [https://www.youtube.com/watch?t=116&v=YpxPAuHNpdM](https://www.youtube.com/watch?t=116&v=YpxPAuHNpdM)

<comment><USER50> The mission systems avionics on a F/A-18F costs around $1-2m per aircraft. This includes Displays, PowerPC based AMC boards, power supplies etc. Hardware in the Loop testing quite practical and was ran as an engineering flight simulator. Missions were flown in the simulator, and problems located without costing $20k/hr per aircraft for an actual flight test.

SpaceX do HIL tests with the Falcon 9 & Dragon spacecraft. [http://www.spaceflightnow.com/falcon9/003/120424date/](http://www.spaceflightnow.com/falcon9/003/120424date/)

</comment></comment></comment><comment><USER51> Okay, the LightSail spacecraft is not a "life or death" thing, but, as it was in the news yesterday[1] seriously logging into CSV file onboard can crash the system and have to wait until it reboots itself (don't react to soft reboot either)?

Feels sad, that a lot of lessons learned are getting lost along the way...

[1]: [http://www.planetary.org/blogs/jason- davis/2015/20150526-sof...](http://www.planetary.org/blogs/jason- davis/2015/20150526-software-glitch-pauses-ls-test.html)

</comment><comment><USER52> I think the right answer here is to develop languages or libraries that enforce the constraints you want.

I think we should beware of adding layers of "silver bullet" technologies that promise to fix the last crisis, but increase the friction of development.

To illustrate what I mean, PLCs are normally programmed in ladder logic. From that level, I don't think you can crash the machine or corrupt memory. So the risks are limited to the "operating system" if you will, which can be more mature and tested than the "application".

</comment><comment><USER39> Testing is certainly very important, but I think it should be emphasized that testing is but one part of the _software development life cycle_ that produces an acceptably safe product.

</comment><comment><USER53> I always thought that the way to ensure zero failures at a level is to ensure accuracy at the next level.

For example, if all trains runs with a minute accuracy at every points, then there won't be any collision by mistake.

</comment><comment><USER54> As a professional tester, I find this pretty interesting. Always wondered how these ultra critical programs were tested. (I work in the creative website space)

</comment><comment><USER55> Have languages been written that are designed to be maximally orthangonal to each other in regards introducing bugs? If so what combinations are used?</comment>