Processor Design: An Introduction (1998) - <AUTHOR> http://www.gamezero.com/team-0/articles/math_magic/micro/index.html <comment><USER2> I've taught this stuff before and I find it best to start with boolean logic and combinatorial circuits, then sequential ones, but don't just jump into building a CPU right away; instead, after discussing clocked/sequential logic and registers/memory ("storage loops"), I discuss a counter, then connect the output of the counter to the address inputs of a memory to create a machine that just reads memory. Adding an "accumulator" and adder turns that into a rudimentary adding machine, and I note that such a machine was already able to save --- or eliminate, depending on your viewpoint (although I rarely will go on an ethical tangent) --- countless hours of human labour at the beginning of the last century. I then slowly evolve the adding machine into something closer to a CPU by making it capable of other operations than adding (and this is also a good time to show the elegance of two's complement), gradually turning the numbers stored in memory into not just data to be added, but operation codes, and eventually, with the addition of control flow and indirection instructions, memory addresses too.

This "bottom-up" approach tends to bore some students, but in my experience also leads to far deeper understanding than more superficial ones that skip over a lot of the intermediate steps. It helps to keep the interest by showing these machines actually working, either with actual logic chips and a breadboard, or a simulator; and of course give the students hands-on experience if possible.

<comment><USER3> As a student, I found the bottom-up approach quite interesting in a lab format. (Though it was an FPGA lab - learning about CPU architecture was a side benefit. And it was at M.Sc. level, so we were somewhat familiar with the basics.)

We built some fundamental blocks in VHDL (e.g. buffers, mux, adder, memory, decoder, ALU), combined them into a basic CPU with fetch/decode/execute stages, added a load/store unit and control flow instructions to make it turing complete, then added optimizations like pipelining. We wrote some code to run on it. Then we switched to a LEON3 off the shelve softcore and implemented our own AMBA peripherals to interface with some real sensors/actors.

The ability to view the NAND gates your code synthesizes was great. Tracking signal propagation delays motivated certain basic design decisions very well. Writing code that runs on a CPU that you also wrote is so incredibly rewarding. (And it's interesting when a bug can be either in your program or your CPU.) And finally integrating a real-world softcore really showed how practical this can actually be even outside of Intel / AMD.

<comment><USER4> I find the gigatron very interesting for this. It bypasses all the vhdl/tooling complexity. [https://gigatron.io/](https://gigatron.io/)

</comment></comment></comment><comment><USER5> In a similar vein, recently released on the MIT OpenCourseWare Youtube channel, the material from 6.004, Computation Structures:

[https://www.youtube.com/playlist?list=PLUl4u3cNGP62WVs95MNq3...](https://www.youtube.com/playlist?list=PLUl4u3cNGP62WVs95MNq3dQBqY2vGOtQ2)

<comment><USER6> This looks great. I hadn't seen this before. Thanks for sharing.

</comment></comment><comment><USER7> I'm reminded of nandgame - Teaches you to build a CPU/ALU bitwise from scratch [http://nandgame.com/](http://nandgame.com/)

It's good fun :-)

<comment><USER8> MHRD is a similar game: [https://store.steampowered.com/app/576030/MHRD/](https://store.steampowered.com/app/576030/MHRD/)

Also cf. (somewhat different, but related spirit) Silicon Zeroes ([https://store.steampowered.com/app/684270/Silicon_Zeroes/](https://store.steampowered.com/app/684270/Silicon_Zeroes/)) from the makers of Manufactoria.

</comment></comment><comment><AUTHOR> there is a light treatment of instruction decoding here, [about 2/3 way through] targeted to similar level of pre exposure :

[https://hackernoon.com/how-does-a-cpu-work- af3488d182a2](https://hackernoon.com/how-does-a-cpu-work-af3488d182a2)

the primary post is cut short but i like it for the level of detail presented to someone with toe in the door digital hardware skills.

<comment><USER9> Ben Eater has an excellent step by step tutorial on creating a complete (simple) computer on a bunch of breadboards. It's fascinating to watch, and he does a great job explaining all the details. Highly recommended.

[https://eater.net/8bit](https://eater.net/8bit)

</comment></comment><comment><USER10> I have to mention "But How Do It Know?" here as well. I came across it from a previous HN discussion and picked up a copy. It's really, really good. And it's a really easy read. [http://www.buthowdoitknow.com/](http://www.buthowdoitknow.com/)

</comment><comment><USER11> This body of knowledge taught to us from early 1960's onwards is remarkable, they accomplished so much in the world. Clever 'graphical' algorithms like Karnaugh Maps that subsume more theoretical structures such as Boolean Lattices are a delight</comment>


CCP: players' attempt to destroy Eve Online economy is "f***ing brilliant" - <AUTHOR> http://www.eurogamer.net/articles/2012-04-27-ccp-players-attempt-to-destroy-eve-online-economy-is-f-ing-brilliant

<comment><USER2> That's intriguing, but I know nothing about the game...how are these players going to destroy its economy?

<comment><USER3> Will need a follow up from someone who's active, or at least spent more time in-game than I did.

Jita's the busiest economic hub in the game, so they're effectively shutting down a sizable chunk of the game's trade. Though it's in a secure area, inasmuch as the in-game police will kill you for attacking other players, so many ships are involved that they can just suicide attack even the larger vessels before being stopped.

I doubt this will actually "break" the economy though. I imagine it will be disrupted for a time until the campaign ends or trade moves elsewhere.</comment></comment>


Leonardo Da Vinci's wacky piano is heard for the first time, after 500 years - <AUTHOR> http://www.smh.com.au/entertainment/music/leonardo-da-vincis-wacky-piano-is-heard-for-the-first-time-after-500-years-20131118-2xpqs.html

<comment><AUTHOR> Really quite neat. The sound of this thing is incredible.</comment>


Ask HN: What was the worst bug you've ever solved ? - <AUTHOR> What is the worst bug (the software kind) that you've encountered in your hacking career ? <comment><USER2> Back when text messaging capability was a rarity on mobile phones, which were themselves rare, I was testing an SMS-based weather forecast service that I had written on behalf of one of the mobile network operators.

The testing worked well on the emulator so I decided to test it over the public network to an actual handset. Only I forgot to advance a recordset through which I was looping, so the code never hit the end of recordset condition. It took me some time to notice there was a problem...

The fact that I crippled a national SMS network for a few hours was bad.

The fact that my company had to pay for each SMS, wiping out out profit for that month was worse.

The fact the handset was mine and on my first date with a girl later that evening (whom I later married) my handset kept beeping with incoming text messages (about 96,000 if I remember) was the ultimate.

The handset didn't have a silent-no vibrate function (either it beeped or it vibrated or it did both) and the SMS inbox filled up after 200 or so messages meant it took days for the inbox to fill up, me to clear it message by message, then fill up again ad nauseam.

Still, I laugh about it now...

<comment><USER3> One of my colleagues did this with our automated notification system. After his phone received about $60 worth of text messages, he panicked and shut down the server!

Then again $60 is only about 1200 messages.

<comment><USER4> > Then again $60 is only about 1200 messages.

You must not be in the US..

<comment><USER5> Yeah, that's like $10,000 to $20,000 in text messages.

<comment><USER3> What? Are you saying it costs nearly $10 per message or more? How can 1200 messages cost $10K - $20k?

<comment><USER4> That's what we call "speaking in hyperbole"

</comment></comment></comment></comment></comment></comment><comment><USER6> I inherited a giant hideous stock-management system. It did a certain amount of automated ordering without manual intervention.

Long story short: a nasty race condition meant that it was over-ordering duplicate products from the suppliers to the tune of tens of thousands of dollars per day.

On the general theme, my most frightening software experience was when I met a guy who was the star programmer for a company doing controllers for elevators. I got talking to him, and he showed me some code. It took me about 15 seconds to identify an edge case that would engage the motor while the elevator was at the top floor (thereby attempting to pull the elevator into the roof?). It took me a further hour to explain what the edge case was. The bug wasn't that scary, as I'm sure there are hardware failsafes, but the general dimness of the guy writing software to control lifts was scary.

I started taking the stairs after that.

</comment><comment><USER7> In high school I got a job for my local Department of Public works in the Power division. I lived in a small New England town that did their own power, much like most towns do their own water and sewer.

My job was as an assistant to the inventory guy, a 70 year old feisty man with one hand named Al.

I was often bored when Al would tell me to 'go hide somewhere' so I wrote some software to help him manage the inventory system. The power engineers in charge saw this and after a few small programming assignments had me work on updating the newly installed SCADA control system. This was a specialized programming environment that controlled all the power in the town.

We were setting it up to buy power from the local college during the yearly 'peaks' in August, thus reducing our yearly electrical bill by potentially millions of dollars.

After a month of working on it and incrementally adding my changes I screwed up. I knew this when I submitted a change and all the alarms went off at the substation.

Half the towns power was out. I got it back on after an hour, and nobody called with ventilator issues, so I think there was no real harm done.

The engineer in charge of the department laughed it off when he saw my apprehension about the situation. He said in the grand scheme of things they have made far bigger mistakes than that, probably referring to the blown transformer a couple months earlier.

<comment><USER8> That must have been one scary hour.

</comment></comment><comment><USER9> A batch run of only a few thousand items was running all night long, rarely finishing and causing all kinds of problems when people logged in in the morning. The users had been complaining about this for years.

I was given the ticket and found a "SLEEP 10" (causing a 10 second pause for each item) in the 10 year old BASIC code, put in by the original programmer for debugging purposes, and never removed before it was promoted.

I removed the "SLEEP 10" and run time went from 12 hours to 23 seconds.

The users loved me, but my boss was not pleased. He said, "You should have changed it to a SLEEP 5 so we had something else to give them the next time they complained."

<comment><USER10> Perhaps it was intentional on the original programmer's part... but he never got round to reducing the sleep time - <http://thedailywtf.com/Articles/The- Speedup-Loop.aspx>

</comment></comment><comment><USER11> Not so much a bug, but this incident cut my lifespan by about 5 years or so:

I was fixing an account balance on a customer's master database. Any change that happens on the master gets replicated to 30 branches, usually at 10 minute intervals.

I wrote the UPDATE statement, highlighted it, and pressed "Execute". Unfortunately, I didn't select the WHERE clause, so I basically gave all their customers (85000) a 50c credit balance. The IDE I was using also had a bug that caused it to ignore the auto-commit setting (which was turned off), so it basically committed the transaction. First I tried a ROLLBACK, which failed obviously. Realizing I had to act quickly, I disconnected the network cable (I was working on the DBMS server) to stop replication. I extracted the transaction log from the current database into a textfile (a few hundred MB) and restored the database from the most recent backup. Then I basically ran the extracted transaction log as SQL scripts against the database, hoping that it wouldn't fall over. I didn't want to do a normal restore because I was afraid of it showing up in the logs.

Within a very stressful 60 minutes I had everything back to normal. I never told a soul IRL about it.

<comment><AUTHOR> Which is one of my pet gripes with SQL, that the default is to affect all records on update, that should have been:

UPDATE tablename SET xxx='newvalue' WHERE ALLRECORDS;

Or something to that effect.

<comment><USER12> MySQL has --safe-updates (aka --i-am-a-dummy), which will refuse to execute UPDATE and DELETE statements without a key in the WHERE clause.

<comment><AUTHOR> Would you feel the same if in unix the default for 'rm' was 'rm -rf *' ?

</comment></comment></comment></comment><comment><USER13> For a different definition of worst:

I started a job recently at an ecommerce company. There was a long-standing bug with the cart display in the upper right of the page always saying that the cart was empty. People would report it all the time, and the quite smart lead programmer said it was something really complicated that he hadn't had time to investigate.

But eventually, after I sort of knew my way around the code, and when I finished up all the tasks on my to-do list, he handed me that as a why-not- investigate sort of thing. He didn't really have any idea, just that the previous long-gone coder had said it was complicated and in the depths of the way the front end code interacted with the order system.

So, I reproduce it, and look at the template code. These two lines, right next to each other:

[% cart_summ = ourdb.cart_summary %] [% IF cart_cumm.qty > 0 %]

Note that the two variables don't match. And this was broken on the site for _FOUR YEARS_. And nobody looked because someone said he had and it was hard, and nobody had time for a hard problem.

 _facepalm_

<comment><USER14> hehe. reasons to develop YOUR_LANGUAGE-lint?

</comment></comment><comment><USER15> Some users of a (shipped, fairly heavily used) web app we had deployed were getting kicked back to the login screen at random. Sometimes, very frequently.

Looking in the logs we could see that these users were somehow losing their authentication cookie and the application was correctly bouncing them to login. So how were they losing their cookie? Assuming it was a bug in the code we searched and searched to no avail.

Finally I discovered that the hardware load-balancer our CTO/'IT' guy' had insisted on was the culprit. The load balancer would buffer fragmented requests and re-assemble them before sending them on to the server. Unfortunatly the load balancer had a huge bug in its firmare.

If a user was using firefox, on windows, and their request was fragmented such that the first packet contained data up to the end of a header line including the \r but not the \n, so the next packet would start with a \n and then a header name, the load balancer would insert a \n\r between the two packets, thus effectively truncating the HTTP headers, usually before the cookie lines.

When I found this bug I couldn't believe that this was actually happening, I thought I was taking crazy pills, but you could run a sniff on the front and back side of the load balancer and see the request go in well formed and come out busted. We ditched the hardware load balancer and all was well.

</comment><comment><USER16> Was converting an avionics subsystem from Ada to C. It was a client application that had to talk to an Ada server, sending and receiving rather huge chunks of data, large, deeply nested, intricate structure types. The C structure type had to match the Ada type exactly, or else it wouldn't work.

I got it working fine on our desktop simulation, but running on the actual hardware it was consistently off. After extensive testing, I realized that it was a bug in the compiler for the target hardware, such that a very particular type of structure (something like, {int, char, float}) was being packed incorrectly, resulting in a 2-byte pad that shouldn't be there. If I reordered the structure elements, it was fine, but that particular grouping and order refused to work correctly.

It was GCC, so we could fix the compiler ourselves, right? Not really, as, for avionics systems the compiler has to be thoroughly qualified for avionics use, and changes equal requalification. I "fixed" it by storing the float as an array of characters, converting it to and from a real float type as we needed to use the data value.

Trivial, perhaps, but I was very excited to resolve the problem, after spending days barking up wrong trees. One usually expects that the problem is not in the compiler... :-)

<comment><USER17> Were you using GCC's __attribute__((__packed__))?

Anyway, the standard way (to handle protocols) is to parse the thing not making any assumption of the struct layout.

<comment><USER16> Yes. The other structures all packed correctly.

If I am understanding what you are saying, we really couldn't do that, as the server sent and expected to receive binary blobs of data; the only way to know what was what was to have a map of where the data elements were.

<comment><USER18> I like to use macros (in C) to spit out the structure size and offsets of each structure member over a serial port, in a format I can then cut-n-paste back into the source. This output consists of a bunch of (compile-time, when possible) assertions so that any changes to a structure break the build. These assertions go on both the embedded side and the PC server side, so any weird packing issues show up at compile time.

</comment></comment></comment></comment><comment><USER19> Bug with the most spectacular results:

As a (former) hardware engineer, I've worked on many projects where bugs have physical effects. This can range anywhere from amusing to seriously dangerous.

One such bug involved a mistake in the assembly diagram and silkscreen for a circuit board. The result was that a tantalum capacitor was installed backwards on a 12V supply rail.

Tantalum capacitors are polarized, and they fail in a spectacular way when reverse-biased. In this case, the supply rail could source upwards of 20A, so the fireworks were loud and impressive. Luckily the cap was easily replaced and the only permanent damage was cosmetic.

Hardest-to-troubleshoot bug:

In my subsequent return to the world of software, I worked on device drivers for network interfaces (among other things).

NICs frequently operate through a circularly-linked list of packet descriptors, which contain pointers to buffers in RAM where the NIC can DMA packet data. The hardware fills the DMA buffers and marks the descriptor as "used," and the driver chases the NIC around the ring, processing the packet data and marking the descriptors as free.

In testing, we discovered that under long periods (hours, usually) of heavy load, the system would occasionally freak out and stop processing packets. Sometime later, various software modules would crash.

Working backwards through the post-mortem data, I saw that the NIC would get "lost" and dump packet data all over system memory. I dumped the descriptor ring (tens-of-thousands of entries) and wrote some scripts to check it for consistency.

To make a very long story short, when the NIC was stormed with lots of 64b packets with no gaps, it would eventually screw up a DMA transfer and corrupt the "next" pointer in the descriptor ring. On the subsequent trip through the ring, the NIC would chase an errant pointer off into system memory and corrupt other system data structures.

Since hardware can DMA anywhere in RAM, the OS is powerless to stop it. The resulting errors can be ridiculously hard to track down and fix.

</comment><comment><USER20> Had an obscure picking id wrap around because a table wasn't getting cleared for debugging purposes which resulted in excessive amounts of beer being delivered to unsuspecting customers at an automated gas station.

Here's a video of part of the result: <http://www.youtube.com/watch?v=RUhLDtPnSuQ>

<comment><USER3> A bug that gives out free beer. Talk about a bug with a silver lining :)

</comment></comment><comment><USER21> Runaway robots at Anybots have caused:

      - 2 holes in drywall
      - 1 bent bookshelf
      - 1 dent in concrete floor
      - 1 frightened Jessica
      - http://www.youtube.com/watch?v=qkenIInV9rI

The last one was fun because I have logs showing packets from the PC/104 computer stack (running FreeBSD) connected to the robot while it was in midair.

<comment><AUTHOR> How do you dent a concrete floor with a robot ?

<comment><USER21> Monty weighs 160 lbs, and at the time had an over-designed metal piece for its neck. A sensor failed and it faceplanted, taking a 1 cm deep chunk out of the concrete. It's still there, under some carpet.

<comment><AUTHOR> Sure doesn't look that heavy, maybe I'm misjudging the scale here, I'll have another look at that video.

Nice balancing by the way, smooth dampening.

Edit: Ah yes, now I see it, that's a small scope and a lab powersupply in the background, I estimated the size off by a lot.

I figured the whole thing was about a foot tall or so, sorry!

</comment></comment></comment></comment><comment><USER22> The most memorable bugs are the ones that cause physical damage. This was mine:

<http://www.youtube.com/watch?v=b7i2KkYYulI>

Damage: Blown tire, dented rim, looking like fools in front of our peers.

<comment><USER23> Um.. what was that? Some autonomous vehicle navigation system?

<comment><AUTHOR> Darpa Urban Challenge, top right of the video.

<comment><USER24> What was the bug?

<comment><USER22> Depends on your point of view. :-) Either:

(a) The CAN bus (which reports wheel velocity and position of the steering wheel among other things) micro-controller hardware stopped sending interrupts which caused the main computer to think we were stopped. As far as I can tell this is just a straight up hardware bug with the Philips ARM chip we were using. This caused the accelerator controller to floor it because it was only seeing the last CAN message we ever got which happened to be zero velocity. Same thing with the steering (hence the big swerve).

or

(b) I failed to consider the contingency of not getting any CAN interrupts (either because the of the [very intermittent] hardware bug or because the connector got disconnected--turns out the symptoms are the same) and didn't have any code written to deal with it. Guess what I wrote that night. :-) Luckily I had all day to think about it while a new tire was fetched and by the time I figured out what was going on it took about 10 minutes to code the fix: a watchdog timer that shuts the world down if there are no inputs from the sensors for any reason.

It seems obvious in retrospect but when things are working you sometimes forget about weird failure cases.

<comment><AUTHOR> Any chance of getting 'second opinion' style sensors in there that can provide you with sanity checks ? Such as 'GPS reports movement, but wheel sensors do not, we have a problem ?'

That way you can avoid a paralysis of the control software until the vehicle has really come to a halt.

<comment><USER25> I read somewhere about realtime applications that do something like this but redundant sensors holding a consensus polling algorithm. Have three sensors reporting the same thing and if they are not all in agreement within some kind of delta then go into some kind of limp mode or have the two sensors in agreement be the ones that the system uses for it's algorithms. I cannot recall where I read it though.

<comment><USER26> Me too. The back of my brain is telling me it was for some sort of plane control software? Maybe? An interesting tidbit that I recall was that they used different manufacturers to hedge their bets against bugs.

<comment><USER25> That sounds about right. I seem to recall that it was for aerospace too. Maybe NASA? Something about zero defect software. I cannot for the life of me find the article right now though. I also vaguely remember that it was a HN submission too.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER27> Two immediately jump to mind. One that had a massively bad impact to the company, another that might have..

First, using perl a (later-fired) co-worker added a hardcoded check like the following:

if ($client_id = "specific_id") { #email reports }

Needless to say, we emailed reports for all of our clients to a specific client, didn't go over too well considering that many of them were competitors.. It was particularly bad because he had previously been talked to about flipping the constants to avoid the = vs == bug.

Second, possibly abused but not known for sure, was found a few _years_ after initially being put out. Our webapp created a session ID for each user, MD5 hash.

Except it started like: StringBuffer md5HashedBuffer = new StringBuffer(userId);

Which, because the userId was an int, simply creates a string buffer of size userId, not a string buffer initially populated with userId.

The rest of the hash was added afterwards, then the one-time created, with the result that everybody's session id was the same. Changing your user id in the GET or POST would allow you to be logged in as a different user.

<comment><USER28> Why weren't both of these bugs caught during code reviews?

</comment></comment><comment><USER29> After having launched our product I was spending some time reviewing commits together with the senior tech lead. Still to this day I can recall the commit number, the filename and write down the code from memory responsible for what turned out to be the source of a bug completely wiping out our users computers. Someone had mixed uncommenting a piece of code together with fixing a bug which hid the fact that some horrible code was active in the product. It took us 5 minutes to produce a fix and push it out to the update servers. Did we end up wiping someones computer? Yup, about a dozen known cases including a couple one in-house. I don't even want to think about how many actual cases there were, considering that we had about 2 million downloads of our product before the bug was fixed.

</comment><comment><USER30> A while back, I developed a program to generate invoices for about a dozen busy warehouses. During testing, for convenience sake, I hard-coded in my local printer.

Unfortunately, I forgot to return the printer name to a variable when promoting into production. Hilarity ensued.

<comment><AUTHOR> Hehe, that one had me laughing here. Ouch. Hope you put enough paper in it ;)

<comment><USER31> Better one.

Guy I knew - awesomely good - hex edited a DOS boot sector on a in house machine to use FUCK.SYS instead of whatever.sys it normally is (I forget). He renamed that file to fuck.sys, rebooted and the machine ran. Cool!

We laughed and reinstalled DOS and two days later the boss come charging in yelling 'I have a client on the phone who says his new machine can't find FUCK.SYS!'

The awesome guy goes 'uh oh'. I laughed.

<comment><USER32> config.sys

</comment></comment></comment></comment><comment><USER33> The worst bug I encountered was due to IBM MVS (or COBOL--I was never sure which was at fault) losing addressability of part of a variable length record. Now you see it, now you don't. The solution at our shop was to move the whole record to itself before attempting to look into the record. I was a newbie. If the old guard hadn't told me that workaround, I NEVER would have thought of it. This problem eventually went away, but 25 years later we still occasionally ask each other "did you try moving it to itself" when dealing with new problems. We chuckle, while today's newbies shake their heads at our Old Fart humor.

The worst one I ever caused was when Visa started carrying two amount fields in their credit card records. One was the amount in original currency, the other was converted to the receiving system's local currency. I used the original amount. Our hand-made test data used the same currency for both amounts, so no problem in test. Imagine my surprise when we went live and our system started posting original currency amounts to cardholder accounts, which at the time only supported US dollars. Luckily, we caught it early and senior management and cardholders were all good sports about it. I think those credit card statements with massive amounts became collector's items.

<comment><USER34> Since you mention COBOL records were variable-length I'd guess that they probably contained ODOs (Occurs-Depending-On, variable length arrays for those of you who aren't COBOL literate). In order for a group or record MOVE to work properly you had to move the subordinate ODO values first, otherwise the runtime system would miscalculate the target record length, possibly truncating the MOVE.

This also meant you couldn't use READ INTO for variable length records (which is equivalent to a READ followed by a MOVE) without taking some care.

As you say: "newbies shake their heads..."

</comment></comment><comment><USER35> I was working on a C++ daemon process that communicated over a TCP socket. At the time, we were using the Poco library's facilities to do the standard daemon startup stuff (get rid of the controlling pty, point standard fds to /dev/null, etc). Anyway, one of our field installations wasn't working, so I took a look. It turned out that the communications over the TCP socket weren't working -- where the client process expected a few header bytes containing the message length, it was getting wacky values. I tried a bunch of stuff, and in the end, I displayed the header as ASCII, and it showed up as "SQL: INS". This blew me away; this looked like some debugging output that normally goes to standard output when the process wasn't running in daemon mode.

As it turns out, the Poco library didn't read Steven's UNIX book all that closely, and they _closed_ all of the file descriptors when turning a process into a daemon, instead of reopening them to point to /dev/null. So, standard output was closed, and its file descriptor was reused for the TCP socket. Of course, things like "cout" always assume that standard output is at a particular descriptor, so all the standard output from the program was getting written to the TCP socket.

Boy, that was confusing.

</comment><comment><USER36> I worked on a piece of arcade equipment (manufacturer and model shall remain nameless) that used a bunch of solenoids to control the works under glass.

A little race condition in the code allowed one of the smaller solenoids to stay in a duty-cycled state, effectively turning the coil into a small space heater. Given the right play conditions and length of play, the coil could catch fire, and a couple of times it did. Lots of wood and plastic under glass made for a fun little display.

I heard one story about a unit in Paris being dragged out of a cafe and into the street, then put out with axes and buckets of water. Wish I had been there to see that.

</comment><comment><USER37> How do you define worst?

How about most widespread? Once while trying to debug a CPAN module I figured out that if $condition was false then Perl had a bug causing

my $foo = $bar if $condition;

to leave $foo with whatever value it had on the _previous_ function call. (The exact behavior is more complex than that, but that's a good first approximation.) I then made the mistake of reporting this in a way that made it clear that

my $foo if 0;

was a static variable. Cue years of people like me trying to get the bug fixed and other people trying to keep it around. In the meantime _EVERY_ large Perl codebase that I've looked in has had this idiom somewhere and it has caused hard to notice and reproduce bugs.

How about worst damage to a system? Due to a typo I once caused my employer to send the Bloomberg's ftp system every large file that it had ever sent. Since it sent a large file every day, this crashed their ftp server, meaning that a number of feed providers for Bloomberg didn't have their feeds update that day. I implemented no less than 3 fixes that day, any one of which would keep the same mistake from causing a problem in the future.

How about most (initially) bizarre? Debugging a webpage that a co-worker produced where, depending on the browser window size, you couldn't type into the form. The bug turned out to be a div that had been made invisible but left on top of the page. At the wrong window size it was in front of the form, so you couldn't click on the form elements behind it. (I figured this out by first recreating it with static HTML, then using binary search to figure out what parts of the page I could whack and still reproduce it until I had a minimal example. Then it was trivial to solve.)

<comment><AUTHOR> That second one reminds me of this: An ISP called planet internet changed their homepage, only to find that they reliably crashed Explorer (3 at the time).

Took a while before the phone rang if I wanted to have a look.

It turned out they had a little animated gif in there with the inter-frame interval set to 0, causing a divide by 0 in Explorer.

That gif was pretty much the last suspect on the list.

Divide & conquer until you are simply staring at the solution and still you don't see it...

</comment></comment><comment><USER38> The store locator function on a national pizza chain's web site would completely hang their web server whenever an international search was done. Many, many hours and days of testing and debugging led us to conclude, and build a proof that it was a reproducible bug within IBM's Domino platform, only on AIX boxes, only when: 1) A script using LotusScript, their proprietary language was kicked off, and... 2) A Java agent was then kicked off before the original script completed.

At the time, Java was a new feature within that platform, so there weren't many apps that mized both languages.

After getting to this point, IBM joined in the fix effort, and we had daily conference calls, on which we always had IBM execs lurking because their 6.0 release of the platform was imminent, and this bug had the potential to wreak major havoc if not fixed before launch.

So I cannot personally claim to have done the actual bugfix - the IBM programmer did that. But it was a great learning experience to work together with IBM to find and fix it.

<comment><AUTHOR> Dominos Pizza :)

</comment></comment><comment><USER39> Non-strict comparator with the STL. Nearly impossible to identify when it's happening. It's happened once to me and once or twice to a coworker over the last couple years, and takes 3-5 days to debug every time.

Example:

      struct FooSort {
        bool operator()(Foo const& a, Foo const& b)  const {
      -    return not a < b ;
      +    return b < a;
         }
      };

<comment><USER40> Try a worse version of this - comparing the results of a floating-point function call where the left operand gets moved into the 80bit fp unit during computation, and the right operand stays in register.

Obviously there is a precision difference between the two numbers, enough to make a < b and b < a return true in a surprising number of cases. The way I ended up fixing it was by putting the result of the function call in a member variable in every struct, pre-computing all of the results, and comparing based on that value.

<comment><USER41> I hit this, too. Regression tests were failing when I changed code that obviously shouldn't change the output of the program at all. This happened on a regular basis when we changed numeric code, because of the normal limitations of floating-point arithmetic; we just made sure the numerical results were accurate and updated the regression tests. (The regression tests were quite handy for finding logic errors; they weren't really used to test numerical accuracy.)

But in this case I was just adding some error checks, which weren't even being triggered. Clearly this shouldn't affect the results of our numerical calculations. Since my code shouldn't affect the calculations, I was convinced that our existing numerical code had a subtle memory or timing bug. (I knew that floating-point code was tricky, but clearly I was doing _exactly the same operations_ on _exactly the same values_.) I spent days staring at code, and then my boss told me to stop working on it since the results were clearly correct in both versions, even if they weren't identical.

A few weeks later I read about how values change when they're copied out of the x87 stack into registers. And I thought, naw, we couldn't possibly be using x87 arithmetic. But we were. Which was horrifying, since floating-point calculations could be a bottleneck under some workloads. But we had been running that way since before I started working on it, so at least it wasn't my fault. I added a compiler option to request sse2 floating point instead of x87 floating point. Voila, predictable floating-point results, plus measurably faster performance on a few tests.

</comment></comment></comment><comment><USER42> This bug was solved _after_ the title went gold and shipped in US :/ We did not find out about it until a few users emailed us. The bug got through SONY and our internal testing...

-It was a first gen PS2 game that had to get pushed out since it was considered a launch title. The dungeons in the game were seeded and randomly generated - In some instances, a lever to proceed to the next dungeon was made inaccessible due to the random seed that enclosed the lever with 'wall' tiles - the player either had to start the game over and hope the dungeon was seeded correctly, OR play through the game without finishing that particular dungeon. It was not total killer in the sense that the game could still be completed, but it was tough to know that such a glaring bug got through...

-Fix was made for the Korean version :) Basically a check to make sure that wall polys did not enclose any levers prior to generating the dungeon, if the lever was bounded, re-seed, generate and check again...

</comment><comment><USER43> The nastiest was an interaction between two libraries. If you simultaneously imported xml.dom and matplotlib (both python libraries), and then called functions in one of the libraries (I think matplotlib), the program would segfault.

My incomplete set of unit tests didn't catch it because they were imported by seperate submodules of mine.

I only managed to find it by writing a unit test, going back in version control, searching for the first revision where the problem occurred, and going line by line through the changeset (luckily I commit early and often).

</comment><comment><USER44> While I was working at Lucent, they had an builtin ftp client which was used to transfer images to the router from a server.

Right after I checked in my code (a completely unrelated feature), this ftp facility broke such that you couldnt ftp an image to any router on any platform. So I was assigned this bug which basically stalled the release on every platform that we supported. (The CTO himself called me.) Now you have to realise this BigCo supplies every cell phone carrier in the nation - Sprint, Verizon etc.

What I found was that the ftp system had a bug such that if the image being ftp-ed was an exact multiple of 8k (or something like that) then it would fail. My checkin made the image file an exact multiple of 8K. (Story of my life!)

I found the bug, emailed the CTO and he assigned someone from the core team to fix it. That guy calls me up and in the end I fixed it myself using vi and kibitz.

</comment><comment><USER45> Nameless BigCo came in for PCI compliance on Struts 1.something they coded the credit card number as a member variable of a struts action (struts 1 used a singleton pattern) the last submitted person got to pay for all of the simultaneous transactions going on in the system. Fortunately my team caught it in acceptance review.

</comment><comment> rm-rf Worst software bugs, in increasing order of severity:

An ASP.NET app that didn't consume it's record sets before closing it's database connections, thereby breaking IIS's connection pooling and causing hundreds SQL server login/logouts per second. Lots of interesting side effects on that bug.

An obscure page latching problem on a high volume SQL server (hundreds of transactions & thousands of queries per second). The SQL server would spend all of it's time waiting on page latches, response time would grow exponentially. An eleven hour phone call with MS support finally identified the problem.

Oracle 10.2.0.[1234]

<comment><USER46> _You_ are responsible for Oracle 10?

We need to have a word.

<comment><USER47> I don't know anything about Oracle 10, can you elaborate?

</comment></comment></comment><comment><USER48> I'm still a little new to the game, so I've only had one hacking job thus far. It spanned 4 years and taught me a lot about what to do and what not to do. I could type a small book about all the crazy things they did in that company and the nightmare of spaghetti code, the needlessly complex 3000+ table database, but I'll try to focus on just a few things that went wrong over the years or that needed fixed.

Two of the previous employees got into a debate on whether or not you could directly connect to the database through JavaScript and fetch recordsets, etc. To prove that you could, one to the developers did just that...and left it in place...in production...with the full connection string and user credentials in plain site in the javascript code on the page.

Countless sql injection holes were plugged over the years, but not before we got hit with an attack that plugged javascript ad code into 50 or more tables and a few hundred thousand rows.

The initial developers used ".inc" as the file extensions for include files like headers, footers and database access. These files sometimes had html, javascript, asp or all 3. You never knew. The problem was IIS treated these like text files and would gleefully serve them up if accessed them directly - revealing server side sourcecode, more connection strings, etc, etc.

Our accounting "system" was woefully inadequate. Our sales people always said "yes" and we eventually ran into some clients that wanted their invoices formatted and calculated in a way our system couldn't handle. We were always far too busy to take the time needed to properly engineer the new solution, so this one clients bill was done manually - through SQL Management Studio - every month, for close to two years before I left. Reconciliation of the bill, was also done manually. The kicker is, billing was done based on dates that assignments were completed. These dates were not only not locked down once billing had begun, but new assignments could be injected into the billing period long after the invoice was generated, because multiple factors. Needless to say, it took me half a day, once a month to reconcile their invoices and on several occasions, due to the very manual nature, payments were applied to the wrong assignments, transactions, etc. Once to the tune of $200,000. That was fun to fix. :)

I could go on and on and on and on....

</comment><comment><USER49> I worked on a taxi booking and dispatch system, written in c, and running on dos with custom networking via RS232. This system was installed at around 300 locations around the UK, and on one fateful day, every installation crashed.

It came down to me to find and fix the problem, and it was subtle. The clue lay in the fact that all of the sites that crashed did so within about a minute of one another.

Turns out that some of the old, old sections of the software had been written by the MD, who, despite referring to himself as 'the emperor of c', was in fact an atrocious programmer.

The actual trigger was the comms system looking at a byte that determined as to whether a message had been received. This byte was set to the character 'A' if a message was received. It just so happened that the first byte of the current value of the number of seconds since 1970 evaluated to 'A', and had been written into that memory location a negative index into an array that hadn't been initialised.

This negative index into an array that _shouldn't_ have been empty caused a section of memory to be overwritten that made the comms system think that it had received a packet. This snowballed quickly, and took down the system within about five seconds of boot.

Took the best part of two days to track down, and, of course, it was everyone else's fault but the emperors.

<comment><USER12> Let me guess: The crash occurred slightly before 6 PM on July 22, 2004?

</comment></comment><comment><USER21> At Viaweb, my careless use of $_ in Perl led to the name of every credit card shown in the merchant UI being replaced by a secret auth key. I didn't see it because that particular auth check was bypassed for admin users like me. It wasn't a serious security hole, and we changed the auth key afterwards. But as it happened, I was in a cranky mood when I created the auth key and it had some bad words in it, so it was a little embarrassing.

</comment><comment><USER41> I worked on a program that ran large batch jobs, sometimes taking more than twenty-four hours. This was actually spectacular performance, since we used custom hardware to do most of the computation. I wrote the code that interfaced with the hardware. When the code timed out trying to talk to the hardware, the only sensible thing to do was report the error and abort the program.

Unfortunately, this seemed to happen quite often. Jobs would abort randomly, after about eight hours, sometimes much less and sometimes not at all. Overheating was the obvious first suspect, so that was investigated and ruled out. The hardware was running cool and was in perfect working order. The customer started splitting jobs into shorter batches and combining the results by hand. We wrote code to help them automate this workaround. But batches were still randomly aborting. And I couldn't replicate the bug, despite having identical hardware to the customer. Something in the customer's environment was essential.

Eventually, somebody at the customer figured out the problem. My boss called me up and said, "The customer suspects your code is not time-travel compliant." It was true. My code assumed that time always goes forwards. If time ever went backwards while my code was waiting for hardware, it would immediately time out and abort the batch. And our customer encountered a bug where time _did_ appear to go backwards occasionally. I was too stressed out over other tasks to ask for details of the bug. I just sent them a fix and breathed a sigh of relief when they accepted it.

A Google search now reveals that there was an issue with time going backwards under Xen on dual-core Opterons, which is what the target platform was. They never told us they were using Xen. Maybe that's why they were much nicer to us after the problem was diagnosed!

</comment><comment><AUTHOR> Ok, here's one of mine, it's only fair.

Jasper L. systems administrator of an early web hosting company calls up one evening, there is a problem with the paging system.

A certain host is being paged as 'offline' but when checking the machine works fine.

So I go there bit by bit we check out the software, everything works fine, but sure enough every 10 minutes or so the machine (called 'chopper', I'll remember that for the rest of my life) gets reported as 'down' again.

But there is absolutely nothing wrong with it.

After ruling out all the software bits we figure it must be hardware somehow. The way the supervisor works is it sends a ping packet to the machine, and if the machine responds it is deemed to be up. But chopper misses one ping out of every 5 or so, and sometimes several in a row.

We swap out the computer, move the scsi drive to another box and boot it. Chopper registers as off-line, but works just fine.

More confusion, finally, out of desperation we start messing with the network. This is all 10Base T, coaxial cables with little T connectors on the machines and a terminator at the end of the line on the last 'T' to make sure the impedance is right.

The terminator was on, so that wasn't it.

We went for a break at that point, we'd been at it for hours. Finally, by elimination we figure the _only_ things we haven't changed yet are the drive and the 'T' connector, but surely that can't be it.

And it was... somehow that T connector was a pretty good filter for some bits in the incoming or outgoing ping packet changing one of the bits, causing the IP checksum to fail. No returned ping... we replaced it back and forth 3 more times just to make sure we weren't seeing things.

<comment><USER50> That's not 10BaseT, it's 10Base2 (or thinnet): <http://en.wikipedia.org/wiki/10BASE2>

And--even though I've strung cables for both kinds, and of course 10Base2 takes far less cable, the maintenance issues and troubleshooting headaches of the bus topology make me _so_ glad we've progressed to more robust network topologies.

<comment><AUTHOR> Ah, yes, of course, you're right. If I could vote I would vote you up but my voting seems to be on the blink.

<comment><USER51> Define "irony" :)

</comment></comment></comment></comment><comment><USER52> I worked on the communication middleware for an early Windows tablet. My part sat between a VB6 GUI app (that we customized for each customer) and a RS-232 device and handled all the communications.

One customer had intermittent communication failures that we couldn't identify for weeks. I finally went on-site and hooked up a serial port logger between the tablet and device. After an hour of testing, I finally captured what caused the communication failure. It was some debug message from the GUI app, somehow coming across the serial port. I called back to the office and had a guy from our group go over the the VB app team and ask which genius was logging stuff to the serial port. Sure enough, some guy had been using VB's remote debugging feature with a null modem cable and left it turned on in the shipped app.

</comment><comment><USER53> Every since time there is a memory corruption on a games console it takes hours and hours of tedious detective work to find.... shudder. Glad I don't do that any more.

<comment><USER54> Yeah I had 'fun' with an embedded bootloader that would corrupt a single byte in the image (at offset 2^20 iirc)

</comment></comment><comment><USER55> We set up a new network segment with multiple VLANs trunked over a fiber line. The entire thing worked flawlessly except for one VLAN. It turned out to be a conflict between a very strict media converter and bad ARP packet-generating code in the device acting as the DHCP server on that VLAN. The ARP packets it generated were too short for the media converter, so it dropped them thinking they were damaged and the computers on the VLAN were unable to find the gateway. The end solution was to go with a dumber media converter that didn't work at the packet level. The DHCP device to this day still has the bad code.

</comment><comment><USER56> The bug itself wasn't very interesting, just a brainfart. But it wasn't doing what it should, so I added debugging output - one line per pixel. (I was generating a png from a custom image format.)

Because there was so much output, I did a `| head` to keep it manageable. Saw what I was doing wrong, fixed it, reran the command, it seemed fine - but the output image looked exactly the same as it had before.

It took me about an hour to realise that once head had exited, the pipeline sent a signal to my program, killing it. The image wasn't getting written until after all the pixels had been processed, so it never got to that stage.

</comment><comment><USER57> I once worked an issue with a crashing Exchange server.

It had to do with a connection being reset in the space of time between the server checking if the connection was good, and actually using it.

The client could reproduce it at will, and after a little bit of code at home, I could, too. Back at the office, no repro.

It took a lot of back and forth (to put it mildly), but the problem at work was that the network was too slow - the server got the reset with plenty of time to "notice" it, and the crash didn't occur.

Once we set up the repro on its own switch at work, everything failed in the expected manner.

</comment><comment><USER58> Peter Seibel ask this question in all his interviews in Coders at Work: <http://www.codersatwork.com> It seemed like most were concurrency related.

</comment><comment><USER59> most embrarrasing:

" #define INTERVAL 10 * 86400 "

read the above 'C' statement. Yeah, there are no brackets surrounding the '10*86400', INTERVAL was being used in some calculations.i wasted a lot of time debugging this crap.

</comment><comment><USER60> Embedded OS for a consumer product; the units were freezing (very occasionally) in the field. Usually the units were in store kiosks (= disappointed and unimpressed could-have-been-customers).

Turned out to be a race condition in an interrupt handler, where the OS would say "wake me up when something interesting happens" but something else would sneak in, clobber the wake-up trigger, which meant "... never."

Two weeks to find it, fixed by swapping two instructions in some assembly glue.

The harder a bug is to find, the simpler the fix is.

<comment><USER50> "Simpler" depends if you count the troubleshooting time. To rip off an old anecdote:

      Change one line of code: $3.00
      Find which line of code to change: $3,436.88

</comment></comment><comment><USER61> I once did a crawler that searched the content of a couple of large websites for phone numbers.

I did this daemon in Perl, using fork() to search multiple websites in parallel. When a new search was initiated, the daemon broke the search in multiple packages (of 5000 items to be searched). After initiating 5000 searches (keeping the maximum number of active processes at 30), I did a wait for all the children to finish, so that I can mark the package as "done".

The trouble with fork is that opened sockets are sometimes not forked very well. And I forked my database connection (a DBD::Pg). Forking a DBD::Pg connection went well in my tests, but once in a few thousand forked processes, a process would stall. Sometimes it recovered after a few hours, sometimes it didn't.

I tried setting an ALARM that would make the process auto-kill itself after some time. Didn't work. The final workaround was to monitor the child processes from the parent, and kill -9 the ones that hanged. But the whole processing became too slow.

Finally I gave up on the idea of packages and on marking the search as "done". So I just marked the search as "done" after the last child was forked.

It was a difficult bug because the client had no idea it was a bug, and waited patiently for the results (sometimes as much as 2 days). It was also difficult because at first I had no idea why my processes hanged.

</comment><comment><USER62> Worst bug monetarily:

I was consulting for a multinational on a commodities trading app. I broke the stuff sending trade data to the Risk Management system that does automatic hedge trades. My manager comes down the hall and tells me I've caused $4 million in exposure in the past two hours. At the end, her comment is: "A million here, a million there, business as usual!"

I told that story to a former coworker, and he pointed out, "Well, your boss was calm about it. I bet those traders were _livid_!"

<comment><USER31> so it's your fault...

<comment><USER62> Yes, it was my fault. And we fixed it and rolled that out to production in minutes. And those trades probably got their hedge trades done manually. But at the moment, it seems pretty bad.

</comment></comment></comment><comment><USER63> A neat piece of software calculating the prices for customers Had a hierarchical pricing model for different contracts with large corporations. But the hierarchies were not separated. You started in one hierarchy and ended up in another one. While trying to read the tree from the bottom up you could end up reading the whole database for calculating a price with conditions from every customer.

I must not state the accuracy of that price...

</comment><comment><USER64> Probably the worst bug I ever encountered was a documentation bug.

Back in 1970, I was at my first programming job, working for a very small consulting company. The contract was to convert a large system from 1401 Autocoder to System/360 Cobol. However, instead of the source code for the current system, we were given system specs, which were almost flawless.

However, in final acceptance testing, one program, for one particular accounting line, kept producing results that differed from the current system. The customer would not accept this, even though my code implemented the specified calculation precisely.

Eventually, after a few rounds of back and forth, I demanded to see the source code of the current system, and after much more back and forth, the source listing was produced. Looking at the source, it seemed to be doing exactly what my code was doing.

There was one fact not taken into account - the present system, rather than running on the 1410 for which it was written, was running in emulation mode on their new 360. And they hadn't bought a license for the compiler. So the small accounting change they had made a few months prior had been made to the object code - the card deck itself. Luckily the programmer who had made the change had written a note to that effect on the listing - in the object code part, where he eventually sheepishly showed it to me. We then got paid.

(I also once had a bug that almost got national distribution via Time magazine. Luckily they had only printed a few thousand copies before someone noticed it. That one would have gotten me fired, but for the fact that the same release also included a halftone compression routine that enabled them to push back their photo deadline by a day. The reason the bug escaped my testing was that it only occurred in pictures where the line count was of the form 4n+3 - I had tested with both even and odd line counts, but never managed to use one that produced the off by one error.)

</comment><comment><USER65> We were doing some early testing on this distributed system, and process A kept backing up well under it's nominal load. A had no allowance for shedding excess load (it was broadcasting high frequency safety-critical data), and the network buffer backed up under the shitty messaging layer. It turned out that process B[0..n] couldn't pull the messages off the wire quickly enough because process C was blasting some other data to B at about 1000 times the nominal loading, filling up B's VM and kicking off the (improperly-tuned) garbage collector for 2-second intervals -- it ate the processor time needed to handle the load. Total death spiral.

Needless to say we ended up with more robust load management code and tuned the output of some processes.

</comment><comment><USER66> One bug in Daylife's API tester's javascript. It just never worked in IE for some stupid reason. The solution was surprisingly simple.

After some analysis with firebug, I just figured out that the variable's value wasn't being preserved after a particular point. So I just had to take it's value and assign it back to itself.

It's in this file: <http://demos.daylife.com/api_tester/js/daylife.tester.js> (lines prefixed with comments "IE Fix")

[Full disclosure: I was not and I'm not a part of Daylife. I only participated in a contest they once conducted and was asked to try solving their problem by one of their staff.]

<comment><USER67> Is this relevant to following by any chance?

<http://karma.nucleuscms.org/item/101>

<comment><USER66> exactly. thanks for the blog post link.

</comment></comment></comment><comment><USER47> This isn't a particularly bad one, but it's the first one that popped into my head, and is somewhat unexpected unless you're very familiar with how JavaScript RegExp's "exec" method works:

        > r = /^x$/g
        /^x$/g
        > r.exec("x")
        x
        > r.exec("x")
        null
        > r.exec("x")
        x
        > r.exec("x")
        null

(When exec is used on a regex with the global flag it will remember the position of the end of the last match and perform the next match beginning at that offset. Obviously will cause very bizarre behavior if you expect it to be idempotent...)

</comment><comment><USER68> Worst: c/c++ pointer memory errors, duh. Especially when there are thousands of pointers and don't know which one is overwriting memory that it shouldn't be overwriting.

Second worst: linking c programs when some of the symbols are duplicated in more than one library yet are defined differently, Nobody mentioning this to the developer made it all the more exciting.

<comment><USER69> Thanks to the Boost.org developers the C++ pointer issues are mostly a thing of the past. Using shared_ptr et. al. for all but the most time-critical code has made memory-smashing, memory leaks, etc. so easy to avoid!

</comment></comment><comment><USER21> Calling the least mean square fit function in the linpack library used to go into an infinite loop occasionally. I tracked it down to some very dusty FORTRAN code in the linpack kernel, which gfortran42 compiles incorrectly. Adding -ffloat-store to the compile flags for that library fixed it.

</comment><comment><USER70> ExternalInterface hell (Flash-JS bridge)

<comment><USER47> Like how the moment you move a Flash object ExternalInterface breaks completely in IE?

</comment></comment><comment><USER71> Define "worst". The most costly? The hardest to diagnose? The one with the most stupid cause?

<comment><AUTHOR> Take your pick. Whatever works for you. For me the definition would probably be the hardest to diagnose, but I'm sure there are better ones that lead to more interesting stories and lessons to be learned.

Bugs are a great learning experience.

</comment></comment><comment><USER72> The most expensive bug that I can talk about publicly that I've encountered:

I used to work on software for a very expensive (started at 70k) DNA/protein analysis hardware solution. This was in the late 90s, and our GUI was a couple million lines of MFC code. I was responsible for an analysis package written in cross platform (linux and windows) C++, but the main UI and all the complex interface was MFC.

My then boss was a unix idiot who hated Windows. Which is fine and all, but a Windows product was paying our salaries. So one day he decided to rewrite my (well tested, though not unit tested) file handling code written using the win32 API and port it to posix. Not to accomplish anything different, mind you, and not to add it to the cross platform bits since it was useless in isolation and, oh yeah, we had multiple millions of lines of MFC so nothing was ever getting ported.

In any case, during this "upgrade", he found a function called DirTouch, which was intended to make sure certain directories existed during the normal course of saving data. Well, this gentleman subtly changed the semantics from "create the directory if it doesn't exist" to "create the directory if it doesn't exist, but if it does already, then silently delete whatever is there". This wasn't the whole bug, but this change of semantics to a destructive function was the root cause.

This change got shipped. One of our customers killed literally more than $1.3MM worth of data, since each machine run might cost $100K given lab time, reagents, prep, etc.

<comment><USER73> What happened to him? And did he try to blame you? In his mind I'm sure all he did was port your code, which must after all have been buggy.

<comment><USER72> Well... it was a shitshow. We were just told by a livid CEO that a major university (the bio world is surprisingly small and it's pretty easy to get a terrible rep) had a major data loss, etc. We were on the phone with them while simultaneously overnighting the computer to data recovery trying to figure out what happened. It took two frantic days before I figured out what the problem is, patched it, and started the emergency release process for all the other customers.

But yeah, I initially took responsibility for the mess, since it was most likely my code that did it. After figuring out precisely what happened, and having to demonstrate to my boss that the old code did not delete stuff out of directories, that responsibility got walked back a bit. Still... I got "laid off" within 5 months of that.

Basically, there was blame to go around: the university should have had a comprehensive backup solution in place, particularly given the expense involved in creating the data; I should have been more careful to not needlessly call this directory touch operation while creating cached bits in the data files (the "data file" that the user thought of was actually a directory because we were running into fat16 and fat32 file size limits), but at the end of the day, my boss took code that used to work and turned it into code that not only didn't work but was broken in the worst way possible for no other reason than to fuck about with posix idiocy. In a giant win32/mfc app.

</comment></comment></comment><comment><USER74> I've had a lot of frustrating experiences, but the one that made me slap my face the hardest due to the shear simplicity of it was the time I was working on a windows service. I was opening a socket and listening, but for whatever reason I could not get the client to connect! After a half hour or so of playing around i finally minimized the million windows i had open to see Norton sitting there asking me to unblock the application _face palm_

</comment><comment><USER75> A multi threading system and a queue: it will be hard to explain all the algorithm here but sometimes you think that one thread can't affect others and it is not true I had to find an exception that kills no just the thread but doesn't allows the queue consuming to continue, I remember I went home something like 5am.</comment>


Show HN: SoundBite – A time traveling note taking app - <AUTHOR> https://www.mysoundbite.co?hn <comment><USER2> Very smart idea. I use RecUp (was DropVox) a ton - mostly to capture nice moments or sounds (not meetings). I record speculatively - I think/hope nice audio is coming, so I press record. Sometimes I'm right, often I'm wrong. When I listen later it often takes a few listens to recall what I was trying to record. Looking forward to trying this.

<comment><USER2> Having now fiddled with it a bit, here a few comments:

Double-tap didn't seem to work for me.

In the "Saving..." alert I recommend indicating how much time is being saved (e.g., "Saving last 10 secs")

I recommend including the past ten or twenty seconds in with the real-time recording. Usually there will be some trigger that makes one want to start recording in real time, so would be good for that to be included.

My particular use-case is what I described above: audio mementos. For example, recording a speech the principal gave to new students on my daughter's first day of elementary school, or a ridiculously funny impression my son does. I want those as mp3 files in Dropbox. That's a feature I'm willing to pay for, as I did w/ RecUp.

I'm curious about the "sign in" option on your site. Is the idea to sync notes?

</comment></comment><comment><USER3> Nice landing page</comment>


Show HN: calcl – Programmatic Calculator - <AUTHOR> https://github.com/codr7/calcl <comment><USER2> Yet another bc? [https://www.gnu.org/software/bc/](https://www.gnu.org/software/bc/)

<comment><AUTHOR> There is definitely some overlap from what I can see, fix point numbers is one notable difference.

I would say calcl is more flexible; since it's designed to be used standalone, as an opcode in any cidk [0] program or as a C++ library.

It also serves as a minimal example of how to design an interpreter around cidk.

[0] [https://github.com/codr7/cidk](https://github.com/codr7/cidk)</comment></comment>


DNSCrypt – how expired certificates became a thing of the past - <AUTHOR> https://00f.net/2019/05/04/fixing-expired-certificates/ <comment><USER2> So this solves a problem with intermediate certificates expiring, but the scope is pretty narrow.

There are two types of key here, a "provider public key" and a "resolver public key". The provider key signs the resolver key which signs the requests. This is a system to keep the resolver key from having problems. If you're a website, this is basically irrelevant, because you do the equivalent of having the provider key sign requests. You have no intermediate to worry about at all.

But what if you're Mozilla, and you _do_ have a problem with intermediates expiring?

Well if you put this directly into place everything would get much worse. Extensions would expire after a few hours.

You would have to change your validation logic, to accept signatures that were made _while_ the intermediate was valid, even though the intermediate has expired. (You could set a 12 month limit to make this no less secure than the old system.)

But... if you did this validation change, that would have prevented the extension problem in the first place, all by itself!

This system of short-lived intermediates is _nice_ , but it's way more important in its original home. For extension signing it would almost only be a convenience.

<comment><USER3> IMO there should be a system for soft-expire. For example browser should issue alerts when website uses a certificate that is about to expire soon (less than 10% of its life, for example). Of course it's expected that website owner replaces certificate before that happens, but even in that case site still works with a slightly degraded UI, so visitors could reach site owner and tell him that something wrong. Similar system could be used with Firefox extension system as well. Probably any user would prefer one alert rather than having all his extensions disabled.

<comment><USER4> What is my grandmother going to do with that information? She doesn't know what a certificate is, only that her bank is giving her some warning. Users don't need this warning, developers do.

<comment><USER3> She might call her bank and ask. Now bank will know that something is not quite right. There's no way to distinguish developer from user. But yeah if user opened developer console in his life, you might treat him differently. That's one possible approach I guess.

</comment></comment></comment></comment><comment><USER5> I've used DNSCrypt for some time and think it's great software, however I disagree with the author's position it makes certificate expiration a thing of the past. Setting a short expiry and automation for renewal is _optional_ \- and I strongly believe optional security is irrelevant. If certificates have a good reason to expire after 24 hours, or sooner, the software should guarantee it AND provide the automation tooling required to update it. Relying on the user to be informed and act accordingly is not a valid strategy.

<comment><USER6> The specification says that resolvers MUST rotate the short-term key pair every 24 hours at most. This is not optional.

Automation tooling to update the certificates is provided. The Docker image that a lot of people use to run servers does this automatically. Keys are rotated every 8 hour.

The client software issues warnings, but tolerates certificates having a time- to-live longer than 24 hours. Not all DNS operators have shifted to short- lived certificates yet, so being tolerant is a reasonable thing to do.

<comment><USER5> I don't use Docker; I run dnscrypt-wrapper on my own server and clients connect to it using your dnscrypt-proxy software. Please pardon my ignorance, but how would you propose automating key rotation in this circumstance (assuming the key is "sdns://...")? Right now it's an involved process of interactively logging in weekly to generate the keys and copy-paste the value into the dnscrypt-proxy.toml configuration file - for each client! I'm probably missing something obvious. Thank you for your work and reply.

<comment><USER6> The DNS stamp (the sdns: address that clients use) doesn't have to change. That wouldn't be very practical :)

See the documentation of dnscrypt-wrapper: [https://github.com/cofyc/dnscrypt- wrapper](https://github.com/cofyc/dnscrypt-wrapper)

Two kind of keys are involved: a long-term key ("provider key", used for signatures), and ephemeral keys (used for encryption, and signed by the long- term key).

Clients only need the public long-term key, which is included in the DNS stamp.

What you are rotating is just the ephemeral key pair (`--gen-crypt-keypair`).

This is similar to what happens to TLS certificates: you can update the certificate for a domain name as frequently as needed. But you don't need to install a new version of the operating system every time somebody adds or changes a certificate for their website.

dnscrypt-wrapper can serve multiple certificates simultaneously, so you can serve the next certificate even if the current one hasn't expired yet.

</comment></comment></comment></comment><comment><USER7> The real rule, it seems to me, is that certificates need to expire _while the original person deploying the system is still working on it_. So that's a measure in terms of the speed of businesses, not any fundamental constant. A year is far too long: people transfer, reorgs happen, etc. Let's Encrypt's three months is a bit too long, I think; it's exactly one quarter, and things tend to change at the quarterly level. You _could_ deploy something and be gone from the project by the time it expires. But anywhere from hours to weeks should be fine.

<comment><USER8> > _certificates need to expire while the original person deploying the system > is still working on it._

This seems really arbitrary. Elaborate?

<comment><USER7> First, it's a lot more robust for the system to be initially designed to support certificate renewal than to figure it out later, usually because "later" means "after it expires." That might either be going through the exercise of renewing certificates by hand or setting up automation. If the system is already in maintenance mode / "legacy", and you don't have a process for renewing certificates, you're at higher danger of missing it.

Also, the person designing the system is evaluated on it working. If it breaks while it's on their watch, it counts against them; if it breaks while on someone else's watch, standard performance review practices are generally unreliable at making it count against them even if it really ought to. Therefore, if you assume standard performance review practices (which is the right way to model incentives for present-day developers/deployers working for for-profit companies, and because of the massive influence of for-profit companies, not a wrong way to model social incentives for everyone else), you want to incentivize them to make renewal part of the system, again either via automation or documentation.</comment></comment></comment>


Differences between open science and reproducability - <AUTHOR> https://doi.org/10.1073/pnas.1921320117 <comment><USER2> This paper is about differences between the academic community discussing open science and reproducibility. One main takeaway is perhaps unsurprising: the open science community is apparently more collaborative, with a more connected co-authorship network and more use of "prosocial" language in paper abstracts.

The more interesting finding is that there are more women in high status (first or last) author positions in the open science community (when the number of authors is smaller) and there is an increasing trend in this community of women occupying such positions. This would be predicted by theories that STEM is an individualistic enterprise less likely to attract people with communal values. Women are more likely to have communal values and this is often provided as a cultural explanation for gender gaps in STEM. But the open science community is a part of STEM that sees communal practices (specifically, the publication of data and code along with information about findings) as key to improving science. This is in contrast to the reproducibility community which has legitimate criticisms of established scientific practices but does not emphasize pro-social practices in the same way as the open science community.

In sum I think the paper is useful by

1\. Showing that while "open science" and "reproducibility" have some superficial similarities they are distinct communities with interesting differences.

2\. Showing ways that the open science community seems more collaborative and communal and thus it seems attractive to women (and likely may be this way because women are helping to drive it).

The paper also has some shortcomings. Names are not gender and gender isn't binary. There's a lot of discussion about diversity and team science which honestly doesn't seem to have much to do with the empirical contributions of the paper.

<comment><USER3> > there are more women in high status (first or last) author positions in the > open science community

Is this true even when excluding fields and papers where names go in alphabetic order?

<comment><USER4> Which fields put names in alphabetical order? In medicine, at least, deciding who goes in the first and last name spots is a really big deal, and fights over this sometimes ruin professional relationships.

<comment><USER5> Author names in alphabetical order is pretty ubiquitous in pure mathematics. There are also usually few (1-5) authors on a paper, and some people who contribute a small amount to a paper (say, a nicer proof of a lemma) are not listed as an author, but thanked elsewhere in the paper.

I think the reason this works for pure maths and not some other fields is that pure maths is quite "slow", in that people don't release papers nearly as frequently as other fields, and the papers tend to be longer.

</comment></comment></comment></comment><comment><USER6> There is no science without reproducability. Merely stories, or formalized stories people call models.

<comment><USER7> Not all people share your point of view. :(

From time to time, I review scientific papers for Machine Learning journals.

In the modern machine learning theoretical papers of the type: We looked in problem A, here is a pattern B and it is the case because <math.>

Nearly all papers that I get for a review are applied papers: We took dataset A, model B, tweaked knob C and our result is better than paper D, but we do not know why. I am fine with this approach. Many good ML ideas were introduced in this way.

But I give reject if authors did not provide the dataset and code that reproduces results.

It is not an issue for other reviewers and my reject is often overridden by editors and papers get accepted, even if there is not way that anyone ever will be able to reproduce it.

</comment></comment><comment><USER8> My take-away from the abstract is that this is not about open science and reproducibility _per se_ but about academic disciplines that have emerged to study those things.

<comment><USER9> Some academic fields have serious problems with reproducibility, such as social psychology. The preference for social interaction over reproducibility can in part be explained by subjectivity of academic field.

<comment><USER10> I just want to add that almost all fields have serious problems with reproducibility. In my case: distributed machine learning... You may think otherwise because subjectivity is not involved, but many "top-tier" papers publish code and release parameters that, _if runnable_ , do not reproduce the paper's results.)

<comment><USER11> Err.. well social psych and ML don't constitute "almost all fields".

If someone asked me for a list of where to find pseudo-science, it would be fairly small set: social psych, nutrition, AI.

<comment><USER12> Biological fields (cancer drugs, early stage drug research, etc) often don't perform much better. In many other hard science fields, like say graphene production via CVP (as a friend worked in), researchers intentionally make their work almost impossible to replicate. The papers look fine but the authors quickly learn which parameters to exclude to make it almost impossible to reproduce. Otherwise they'd loose competitive advantage, and potential money from chip manufacturers, etc. Pretty much all people from toddlers to phd researchers are masters of game theory, be it conscious or not. Luckily there are also a decent number of scientists who try and counterbalance those urges, but I believe the current socioeconomics are pushing them out. Hopefully I'm wrong there.

<comment><USER11> RCTs are category-changing

if your field uses RCTs, it's a science. Whether the results are occasionally, due to incompetence or stats, wrong.

The issue with social psych et al. is that they are methodologically /incapable/ of producing reproducible results. It's association modelling of extremely complex domain's. Essentially superstitous.

<comment><USER12> Agreed, the amount of superstitious in many fields is mind boggling..

Even then in many fields like comp. sci. the role of RCT's are pretty fuzzy (and many results like OO, FP are more superstitious than scientific). So I do consider much of C.S. to be more akin to philosophy than science. Still in say Physics the concept of RCT's don't apply directly, but the effect is the same underlying hypothesis -> experiment -> re-hypothesize -> experiment cycle. That cycle requires predictions of novel events which fulfills the same role, otherwise it's just data fitting.

One "technology" that could strongly improve our ability to do proper science without RCT's or even improve RCT or hypothesis formulation would be properly integrating causation into statistics (particularly via Bayesian methods). See the work of Judea Pearl [1] for a digest of current work on causal maths. Then scientific fields can clearly define the degree of effect or causation for various results.

Even in medicine/biology RCT's don't always confirm a theory as you can have overlapping effects where the tested variable does work but not for reasons in a given hypothesis. For example see Statins & cholesterol which work to a degree but largely because cholesterol is a necessary but not only ingredient to plaque formation and heart disease. RCT's proved early results of Statins, but new more powerful ones failed to materialize benefits. It turns out it's much more complicated (one random google result plucked for context [2]).

I imagine those cases in medicine as similar to "microbenchmarking" in C.S. where it's easy to see that small scale microbenchmarks are true but fail to improve the overall system performance. Sometimes they make the overall system worse. Medicine in particular doesn't do enough real science. RCT's are only performed at a "microbenchmark" level but rarely performed for the overall system, partly due to cost and difficulty. Hence why I think something like causal maths could _really_ help society formally and explicitly estimate causal properties from groups of RCT's.

Sorry for the semi-rant, but it's been bouncing around my head for a few weeks now!

1: [http://cdar.berkeley.edu/wp-content/uploads/2017/04/Lisa- Gol...](http://cdar.berkeley.edu/wp-content/uploads/2017/04/Lisa-Goldberg- reviews-The-Book-of-Why.pdf) 2: [https://www.statnews.com/2019/04/03/statin- heart-disease-pre...](https://www.statnews.com/2019/04/03/statin-heart- disease-prevention-more-than-medicine/)</comment></comment></comment></comment></comment></comment></comment>


Show HN: Sslhash: SSL without a certificate authority - <AUTHOR> https://github.com/jD91mZM2/sslhash <comment><USER2> Rather pointless to compare the SSL certificate to a hash ofcourse. Why not just compare it to a copy of the certificate. No need to obfuscate and allow user to be fooled with changed hash.

<comment><AUTHOR> The hash is a lot smaller and less likely to be subject of a copy-paste error ¯\\_(ツ)_/¯

</comment></comment><comment><USER3> How does expiration work? What happens if an TLS/SSL cert is leaked/compromised? Change the clients as well?

<comment><USER4> You would probably need some sort of "authority" to redistribute the hashes.

<comment><USER3> One of the main reasons for the library is no need for cert authority. Why not just create a regular cert Authority certificate and put the trusted authority cert on the clients. That makes it so you almost never have to change the clients certs. Also, it's supported by standard TLS libraries and clients.

<comment><AUTHOR> True, didn't think of that. The server would still have to generate it using a command though... I like having things automated.

</comment></comment></comment></comment><comment><USER5> I'm confused, domains are not a barrier to entry (I'd argue that the developer account is a much larger barrier). Why is this a needed thing?

<comment><USER6> Not sure what you mean by developer account...

Anyway, if I understand correctly, this is basically just a user friendly version of certificate pinning.

<comment><USER5> Oops! For some reason I thought it was Swift at first. My bad, I understand it a bit better now then :).</comment></comment></comment>


Canada's fossil fuel subsidies amount to $1,650 per Canadian - <AUTHOR> https://thenarwhal.ca/canadas-fossil-fuel-subsidies-amount-to-1650-per-canadian-its-got-to-stop/ <comment><USER2> To be clear, this is treating externalities like carbon emissions as a subsidy - IE we should have a massive carbon tax to internalize it. Which from an economics standpoint makes perfect sense, but is not what most readers would consider a "subsidy" and so is a bit of a dishonest presentation.

I agree with the conclusions, but I think presenting it that way tends to only embolden the do-nothing anti-climate-action critics.

<comment><USER3> It's a very dishonest way of framing it. The entire article makes it sound like they're talking about actual payment to wealthy fossil fuel companies that could be paid to actual Canadians instead and would make them $1,650 a year better off.

<comment><USER4> Not exactly. Framed the way you do it doesn't mean i'm $1650 better off, it means government is hiding the real cost of my consumption and encouraging me to use fossil fuels.

I'd rather bare the cost, and be able to realize the financial benefits of reducing my carbon footprint.

</comment></comment></comment><comment><USER5> > _Finally, there is the $60 billion subsidy that the IMF focused on — the > "social costs" of carbon that governments pay, instead of fuel producers._

So it's not a direct financial contribution by the government.

This viewpoint was hotly debated in another recent thread:

[https://news.ycombinator.com/item?id=19905104](https://news.ycombinator.com/item?id=19905104)

</comment><comment><USER6> I wonder how the IMF would classify the amount of money the USA spends on military protection of the shipping lanes (to protect oil shipments) around the Middle East...

</comment><comment><USER7> From the study cited:

> The paper updates estimates of fossil fuel subsidies, defined as fuel > consumption times the gap between existing and efficient prices (i.e., > prices warranted by supply costs, environmental costs, and revenue > considerations),

It's not necessarily a direct transfer subsidy or even lower tax rate for certain industries. So you can take that for what its worth

</comment><comment><USER8> If you want to talk about actual dollars in tax reductions or subsidies for fossil fuels in Canada perhaps this link would be of better use:

[https://www.iisd.org/faq/unpacking-canadas-fossil-fuel- subsi...](https://www.iisd.org/faq/unpacking-canadas-fossil-fuel-subsidies/)

</comment><comment><USER9> Interestingly there has been proposals to build Nuclear to deal with SAGD drilling in the north. I've heard (no specific source) that 1 barrel is used in extraction for every 3 produced. Electricity could enter the grid and the waste heat/steam could assist Tar sand extraction and refinement processes.

But environmentalists protest Nuclear too.

<comment><USER10> Nuclear has a PR problem just like airplanes because of how high profile the failures are (and to be honest, most nuclear plants are using 1960s tech). People are often irrational, like worrying about their flight while _driving_ to the airport.

TerraPower is pretty interesting... but political setbacks. [https://www.popularmechanics.com/science/energy/a25728221/te...](https://www.popularmechanics.com/science/energy/a25728221/terrapower- china-bill-gates-trump/)

<comment><USER9> These guys are more interesting to me, but it's not really my area of expertise:

[https://flibe-energy.com/news/](https://flibe-energy.com/news/)

[https://www.youtube.com/watch?v=0BybPPIMuQQ](https://www.youtube.com/watch?v=0BybPPIMuQQ)

</comment></comment></comment><comment><USER11> The estimate is from 2015 - Trudeau got into Power in Nov. 2015...

<comment><USER2> At its planned maximum, the carbon tax on Canadians will be $50/tonne. Canada emits 20 tonnes per capita per year, Assuming a naive calculation, that's $1000 per Canadian. So under that model, half the Canadian subsidy will be gone by 2023.

However, this simplification I'm ignores that, to protect businesses Canada has implemented output-based allocations, where the first n-tonnes per unit product of a large business are exempted, meaning that the cleanest and greenest businesses within an industry pay _no_ carbon tax because they're below their output allocation. That prevents them from getting stomped by competitors without carbon pricing, but means a tremendous amount of GHGs in Canada will be going untaxed.

Yeah, we need to get $100 per tonne ASAP, and start forming coalitions with other countries so that we can use tariffs instead of output-based allocations to protect carbon taxed businesses from untaxed competitors.

<comment><USER12> This is why I think H.R. 763 (in the US) is a well-designed bill. It taxes everything, and has a built-in tariff to serve as incentive to other countries to implement similar carbon taxes. I hope it passes. I believe it starts out at $15 per ton, then goes up by $10 per ton each year, with adjustments based on whether we're falling short of our greenhouse gas reduction goals, or greatly exceeding them.

<comment><USER2> For reference, Canada's is $20CAD/tonne currently, going up $10/year up to a cap of 50. So the US $15 is about the same to start, but ramping up a bit faster when you figure exchange rate.

</comment></comment></comment></comment><comment><USER13> Again, it's socialism for the rich and capitalism for the poor.</comment>


The AI Revolution: The Road to Superintelligence - <AUTHOR> http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html

<comment><USER2> When he mentions a theoretical IQ of 12,952 I thought he might be making a reference to Red Dwarf - but Holly's IQ became 12,368. Does anyone know where this number comes from or is it simply random?</comment>