What every compiler writer should know about programmers [pdf] - <AUTHOR> http://www.complang.tuwien.ac.at/kps2015/proceedings/KPS_2015_submission_29.pdf <comment><USER2> One of the things this person misses is a lot of these are undefined _with the explicit goal to let compiler authors take advantage of them_.

So his real argument is with the C committee for not defining these things the way he wants them. His claim that the behavior of what should happen is obvious is pretty strange (IE "it should do exactly whatever the compiler is trying to prove it doesn't")

It's essentially a complaint that compiler authors should follow a mystical spirit of C that he seems to have in his head.

(Plus, you know, he's kind of a dick about it all)

I read things like "We had plans to turn this optimization into a production feature, but eventually realized that the GCC maintainers only care about certain benchmarks and treat production programmers as adversaries that deserve getting their code broken. ... The current GCC and Clang maintainers suggest that we should convert programs to "C". How would that turn out for Gforth?"

I was intrigued, so i went looking for the mailing list discussion where the gcc or clang people suggested this or where any of this happened.

I can't find it. On either set of mailing lists. I searched for gforth, i searched for anton, etc Nothing.

<comment><USER3> > _lot of these are undefined with the explicit goal to let compiler authors > take advantage of them._

Constructs should be undefined because the current landscape of C implementations is surveyed, and those constructs are found to be nonportable to such an extent that their behavior cannot be even "implementation-defined" or "unspecified".

That's the beginning and end of it, period.

That's why certain thing were undefined or unspecified in the first ANSI C standard. For instance, different compilers had different evaluation order for function arguments, without documenting which, or guaranteeing that it would always be the way it is.

The committee didn't sit there and decide "let's make evaluation orders undefined for the sake of blazing speed, even though all C compilers today are doing it left to right. Let's have it so that after 1989, they can use whatever order they want!"

It was, "Ooops, we can't codify a specific evaluation order, because looking at what's out there, there isn't one. If we were to codify an order, implementations not having the order that we codify would have to be modified to conform with our standardized language."

As time goes on, these stupid things should be _tightened_. Over the course of thirty years, it's okay to codify some _gratuitous_ lack of definedness in a language area and make everyone straighten it out in their implementations.

<comment> loup-vaillant I believe the order of evaluation of function arguments is _unspecified_ (though it might lead to undefined behaviour in some cases, I'm not sure).

\---

Signed integer overflow is more interesting. In most platforms, it is very well defined: it either wraps around or it saturates. On one's complement machines I don't know, though I'd expect a deterministic (yet mostly nonsensical) result.

So it might sound reasonable to have signed integer overflow be _implementation defined_. Except that another reasonable behaviour is to trigger a trap, just like divide by zero. So they put in into the _" anything can happen"_ bucket, that is, undefined. And now we have compilers taking advantage of this, by assuming signed overflow simply _never happens_. Which sometimes lets them deduce interesting things such as `k<k+1` evaluates to `true` no matter what.

I'm not sure the standard bodies anticipated that. I bet they were thinking something along the lines of _" the runtime can assume it never happens, and fail to perform the relevant checks in the name of performance"_. I'm not sure they originally interpreted this as _" the compiler can assume it never happens, and treat the alternative as dead code"_.

Overall, I see 3 kinds of undefined behaviour: the most innocuous one is when the runtime is allowed to trigger a trap, and crash the program right away. The second one is when the runtime is allowed to not perform the relevant check, and go bananas because of mangled memory or something. This one is more serious, and the source of many security vulnerabilities. The third one is when the _compiler_ can assume the relevant undefined behaviour doesn't even happen, and make lots of interesting deductions from this assumption. This one is mostly insane, though I can understand why it would be nice in some limited cases.

We need 3 buckets for undefined behaviour. "Anything can happen" is too coarse grained.

</comment></comment></comment><comment><USER4> I'm a compiler writer. I'm not actually unsympathetic to these points, and agree that compilers intended for practical use had better have strong justification for surprising behavior.

But the pervasively nasty tone of the paper is not going to make many friends among the people whose work the authors are trying to influence.

<comment><USER5> I agree about the unpleasant tone of the paper, putting "facts" and "optimizations" in quotes, implying they are not real, and talking about "claims" of GCC maintainers, and cracks about psychic powers.

But there is a good underlying point to the paper, and one I tend to agree with - a compiler's behavior should give considerable weight to how the language is used more than taking a literal view of what the words mean in the spec.

Taking things to an absurd extreme to make the point, "undefined behavior" implies that a compiler writer can insert code to reformat the hard disk if it encountered such code. Granted, the point of a spec is to define things so that compiler writers don't have to divine intent, but perfection is never attained and hence compiler writers have to use their judgment.

I know in my compiler work, I have backed out some optimizations that were allowed by the spec, but broke too many programs. The inconsequential speed improvement was just not worth it.

<comment><USER6> It may be unpleasant, but the reality is that many programmers do feel that way. I can say that as a kernel programmer, I approach using a new version of gcc with a particular version of dread. What sort of random bugs that might end up causing kernel crashes or data corruption will arise when I try using a new compiler?

As a result, I'm quite sympathetic to the attitude of "any sufficiently advanced compiler is indistinguishable from a malicious adversary"....

<comment><USER7> Just use a more defined language than C..

<comment><USER8> Yes. I think, if I had to do embedded work, I would use Ada.

<comment><USER9> I immediately thought about the same language. It's clean, safe, fast and explicit, even if a bit verbose. With a quite nice compiler, last time I checked (some 10 years ago), and without "undefined behaviour".

</comment></comment></comment></comment></comment></comment><comment><USER10>
        int d[16];
        int SATD (void)
        {
            int satd = 0, dd, k;
            for (dd=d[k=0]; k<16; dd=d[++k]) {
                satd += (dd < 0 ? -dd : dd);
            }
            return satd;
        }

This gets optimized into an infinite loop, and the paper says that this is incorrect. I'm on the side of the compiler. This code is broken, if you do an out-of-bounds access then an infinite loop is the least of your worries.

<comment><USER11> What's wrong with this code? I don't see it.

(disclaimer: I am a programmer, not a compiler)

<comment><USER12> This:

        k<16; dd=d[++k]

, whenever the code reached the end of the loop, effectively becomes

        dd=d[++k]
        if(k<16) goto startOfLoop;

That, in turn is equivalent to:

        k+=1
        dd=d[k]
        if(k<16) goto startOfLoop

In the second line, k is used to index into array d. Conforming programs do not access memory outside of arrays, so the compiler can infer that

      0 <= k < 16

Given that, it is clear that the _goto startOfLoop_ branch will always be taken. Inside the loop, the program accesses some memory, but that data never is visible to the outside world, so the compiler can optimize away the memory reads and writes.

The compiler is that aggressive because it has to be in real world programs. For example, compilers using C as their target instruction set may generate bounds checks for every array access, trusting the C compiler to remove as many as possible of them. Macros are another common cause of superfluous source statements that programmers will expect the compiler to optimize away.

I think it will be very tricky to detect cases where compilers should be wary of applying 'obvious' optimizations.

<comment><USER13> I would be pretty miffed if I wrote a, say, Pascal compilers with array bounds checks, and the compiler "optimized" the checks away just because accessing a[i] would be undefined behaviour. OTOH, in a loop like

for (i=0; i<16; i++) { if (i>=0 && i<16) do something with a[i] else report an error;

you can certainly optimize the if to if(1). That's an optimization*.

<comment><USER12> No, it would not. Compilers can remove boundary checks that provably happen after accessing an item of an array, not those before the array is accessed.

For your example, the Pascal code (at least, I hope this is Pascal; it has been a while since I wrote any):

      if (i>=0 and i<16) then
      begin
        x := a[i]
      end
      else
      begin
        ReportError;
      end

is equivalent to the C code:

      if (i>=0 && i<16)
      {
        if(i>=0 && i<16)
        {
           x = a[i]
        } else {
          RuntimeAbort("Array access outside bounds");
        }
      } else {
        ReportError();
      }

A good Pascal compiler, like a good C compiler, would optimise away that second boundary check and the call to RuntimeAbort. Neither compiler is allowed to optimise away the first check.

<comment><USER13> The point of my posting was that you don't need "optimizations" to optimize away redundant bounds checks, optimization* is able to do it just fine. Sorry if I did not get that across clearly. What does your "No, it would not" refer to?

</comment></comment></comment></comment></comment></comment><comment><USER14> The author doesn't understand the market demands for a compiler:

1) Customers who want their text-size to be as small as possible so they can shave $0.0015 off their BOM by using smaller ROM parts

2) Chip vendors who have 7 figure sales that will fall-through if they can't show a 0.7% higher Dhrystone MIPS (yes Dhrystone, the benchmark that was obsolete 20 years ago).

3) Managers who make buying decisions purely based on a set of their own pet benchmarks, with no regard to anything in this paper.

Damn right a compiler will take advantage of every optimization opportunity afforded by the standard. Good compilers will have ways of being much less aggressive about optimizations; I bet with gcc and clang each you have some -f flags that will disable certain optimizations (not to mention just good old -O0); I know most commercial compilers do.

[edit]

I would like to see some less-optimizable C standard catch on, so that there could be a mode common to multiple compilers that makes programmers happy. Unfortunately attempts to create such a dialect have failed to get off the ground.

[edit2]

Part of this is because there is less of a consensus on what are optimizations* and "optimizations" than the author would believe.

<comment><USER13> As for the market demands on compilers, I hope that this paper influences that in more reasonable directions. In the meantime, I have nothing against having an option (or more) for "optimizations". Enabling new "optimizations" by default however has the effect of miscompiling existing programs and is not such a good idea. At the very least there should be one flag for disabling "optimizations" without disabling optimizations _.

Concerning what are optimizations_: If the observable behaviour does not change (even when there is undefined behaviour), it's an optimization _. Also, register allocation and rearranging stuff in memory is an optimization_ (because programs that rely on that not changing break on maintenance). Other transformations are, at best, "optimizations". If you have any questions about specific transformations, please ask.

</comment></comment><comment><USER3> Optimizations based on undefined behavior are actually based on the belief that the code in question doesn't have undefined behavior (that the programmer is taking care of any possibility of undefined behavior, where that possibility exists).

This assumption should be made only if the programmer who wrote that code is a machine: a machine translating something which itself doesn't have undefined behavior, into C that doesn't have undefined behavior.

That assumption should not be made when the programmer is a person. Or, if the assumption is used, it should be accompanied by a diagnostic. "I'm expressing what appears to be your intent using a more efficient approach, which breaks in a surprising way if you made a mistake."

For instance, consider

       int compare(int x, unsigned char y)
       {
         return x + y > x;
       }

This is always true, or else the behavior is undefined. So it can be replaced with:

       int compare(int x, unsigned char y)
       {
         (void) x; /* somewhat accepted de facto idiom for ignoring arguments */
         (void) y;

         return 1;
       }

And GCC does turn it into "return 1". But why the hell would a _human_ write a complicated function in place of just constant 1?

If a machine generated this due to following a template of code, that is understandable, and making it return 1 is helpful --- assuming that the generator is carefully written to avoid relying on undefined behavior.

Even if the generator actually relies on undefined behavior, the issue can be fixed in the generator, which can be run to regenerate all the code; but there is no such easy band-aid for human-written code. (A code generator can even have a catalog of what version of what compiler brand supports what undefined behavior in what way, and tune its output accordingly to what compiler is used in the subsequent pass.)

What is unhelpful on the part of compiler writers is supporting some undefined behavior as a _de facto_ extension for many years, and then backpedaling on it.

One day your code that depends on signed integers having two's complement "odometer wraparound semantics" stops working after years or decades because of some compiler cowboy who was in millennial kindergarten when that code was written.

<comment><USER15> (x + 0 is not greater than x.)

<comment><USER3> Thanks for that observation; it should be >=.

I will leave that as it is so that your comment makes sense; let this be the erratum.

</comment></comment></comment><comment><USER16> Out of all ridiculous claims in that .pdf, many pointed out by others I find this especially entertaining:

 _But do "optimizations" actually produce speedups for benchmarks? Despite frequent claims by compiler maintainers that they do, they rarely present numbers to support these claims. E.g., Chris Lattner (from Clang) wrote a three-part blog posting8 about undefined behavior, with most of the first part devoted to "optimizations", yet does not provide any speedup numbers. On the GCC side, when asked for numbers, one developer presented numbers he had from some unnamed source from IBM's XLC compiler, not GCC; these numbers show a speedup factor 1.24 for SPECint from assuming that signed overflow does not happen (i.e., corresponding to the difference between -fwrapv and the default on GCC and Clang). Fortunately, Wang et al. [WCC+12] performed their own experiments compiling SPECint 2006 for AMD64 with both gcc-4.7 and clang-3.1 with default "optimizations" and with those "optimizations" disabled that they could identify, and running the results on a on a Core i7-980. They found speed differences on two out of the twelve SPECint benchmarks: 456.hmmer exhibits a speedup by 1.072 (GCC) or 1.09 (Clang) from assuming that signed overflow does not happen. For 462.libquantum there is a speedup by 1.063 (GCC) or 1.118 (Clang) from assuming that pointers to different types don't alias. If the other benchmarks don't show a speed difference, this is an overall SPECint improvement by a factor 1.011 (GCC) or 1.017 (Clang) from "optimizations"._

I mean, the speed-up of 7.2%, 9%, 6.3%, 11.8% for specific cases is huge. It will make real time and money difference for people. He is advocating against having that just because he thinks what signed integer overflow should do is obvious. Entitlement, ignorance, incompetence. I am not sure which one is it. It takes 1 minute to figure out that signed integer overflow is undefined and act accordingly.

<comment><USER17> Developer time is a zero sum game. If one of these optimizations bites me and I spend one day figuring out what happened, I don't spend one day doing optimization somewhere else, where a much greater speedup can be achieved. Or, to out it in your money difference terms, I could be adding value to the program somewhere else, adding or polishing features.

It's a balancing game. Looking at it through the lens of % speedup, ignoring everything else is just stupid. In a perfect world we'd make decisions based on numbers. He provided his. Where are yours?

</comment></comment><comment><USER18> I half expected the document to present statistical evidence about how programmers tend to behave, and use that to generate advice for compiler writers.

The article is a nice read (and doesn't present that kind of statistical evidence), but I'm still wondering, _has_ anyone done a significant study of how programmers work? I would be really fascinated to read an analysis of, say, data taken from full-time software engineers at a given company where all of their actions are recorded (key-strokes, compiler runs, error messages, etc.). Similar to how online competitive gamers have their actions recorded to the microsecond to identify weak points.

It would even be interesting to know, e.g., what is the net average number of lines of code created/deleted per day?

<comment><USER19> The average doesn't matter, specific problems are reported on the mailinglists, for example. You wouldn't sway everyone's writing style with some statistical evidence, not in a giant ecosystem like C/unix and it's not a democracy either where 51% would just ignore the other 49% existence.

<comment><USER18> What? No, I mean that's how to allocate your resources well, you prioritize the problems that will affect most of the people most of the time.

</comment></comment></comment><comment><USER3> What? "tested and production" programs might be conforming according to the C standard, but that's only because it's a largely useless term: _[a] conforming program is one that is acceptable to a conforming implementation._ [C99, 4]

If the implementation is conforming (which might not be the case) and has accepted the program (which could be chock full of cruft), then the program is conforming. The "conforming" designation simply doesn't assure us of any measure of quality, other than that the program isn't so bad that it doesn't compile and link (with that one toolchain which produced its executable).

If the program is tested and in production using a compiler that is operated in a nonconforming mode (often the default on many compilers), then acceptance of the program doesn't speak to its conformance at all.

This kind of twaddle in the abstract is a very bad way to start out a paper. (And I'm one of the "converted" people, you don't have to work hard to convince me about the dangers of optimizing based on "no UB" ssumptions, without a single diagnostic squawk.)

<comment><USER13> "Conforming program" may be a largely useless term, but it's one of the two conformance levels for programs that the C standard defines. The other is "strictly conforming program", and it does not include any terminating C program, so it's also a largely useless term.

Now C compiler maintainers justify "optimizations" with the C standard, and indeed, a compiler that behaves like that is a "conforming implementation" (the only conformance level that the C standard defines for compilers). Is that a useful term?

Yes, we should be exploring the fringes of the C standard (in particular, "optimizations") less, because that is not useful territory; instead, we should orient ourselves by looking at the C programs that are out there.

But anyway, my thinking when I wrote that was: if somebody comes at you with the frequent stance of "your program is non-standard crap full of bugs", when it worked as intended with all previous versions of the compiler, you can shoot back at them with "it's a conforming program". Of course, if the other side is a language lawyer, that approach will not help, as you demonstrate, but if not, maybe you can get the other side to take a more productive stance.

<comment><USER3> "Strictly conforming program" is a set that certainly includes terminating programs; I think you mean that it has a shortcoming because it doesn't exclude non-terminating programs? But a program that keeps running forever, servicing commands entered by a user, is nonterminating, yet possibly correct and useful. Nontermination doesn't constitute misuse of the language as such.

> _you can shoot back at them with "it's a conforming program_

Not really; retorting with a useless term from ISO C doesn't help in any way. You can only effectively shoot back by demonstrating that all the claims that the program is buggy are rooted in constructs and uses which are in fact defined on all the platforms which the program explicitly supports. (Just not ISO-C-defined.)

If you're doing something that isn't defined by ISO C, and isn't documented by your compiler or library either, then you do in fact may have a real problem.

But portability to platforms which are not specified for the program is a specification issue, not a coding issue. If the specification is broadened to encompass those platforms, then it becomes a coding issue.

I wouldn't listen to anyone who complains that, say, some program of mine only works on two's complement machines, not ones with sign-magnitude integers. My response would not even be "patches welcome" (a patch to "fix" that would definitely not be welcome).

On the other hand, merely calling a function which isn't in the C program and isn't in ISO C is undefined behavior, as is including a nonstandard header file like #include <unistd.h>. We can make a conforming implementation in which #include <unistd.h> and a call to fork() reformats the disk.

Simply accusing a program of undefined behavior isn't damning in an of itself; hardly any useful program can escape that blame.

Basically, I can out-language-lawyer anyone who criticizes my code from that perspective, so I'm safe. :)

<comment><USER13> "Strictly conforming program" excludes implementation-defined behaviour, and AFAICS all ways to terminate a C program are implementation-defined behaviour, so terminating C programs are not strictly conforming.

"C" is actually a little bit wider than "strictly conforming", because it includes implementation-defined behaviour (I did not know that when I wrote the paper).

So "C" does not actually correspond to a conformance level defined in the C standard. So while the "C" advocates like to support their stance with language lawyering, they actually pick those pieces of the C standard that suit them and ignore the others. That is fine with me, but if the standard is not the guideline, what is? For me it is the corpus of existing working programs; "C" advocates seem to be more interested in benchmarks.

</comment></comment></comment></comment><comment><USER19> The diagram on page 13 has linear interpolation plotted between the datapoints. That's one step away from fitting an arbitrary polynomial for the points. Don't do it, implying measurements between point releases isn't very sensible.

<comment><USER20> It's easier to follow the data points with nice visible connections.

</comment></comment><comment><USER21> Click-bait.</comment>


New York Auto Dealers' Lobby Prepping New Anti-Tesla Legislation - <AUTHOR> http://www.slate.com/blogs/moneybox/2014/02/26/new_york_vs_tesla.html

<comment><USER2> "If you allow someone to come into the market with no overhead, that's an unfair advantage" hmmm isn't that the whole point of markets? time for some creative destruction in car dealer land.

</comment><comment><USER3> Texas S owner here, people are still getting their cars here despite these restrictions. Here is a pic from May of last year when I picked up mine. Most of these cars are being prepped for their owners. [https://dl.dropboxusercontent.com/u/13941904/cars/teslas.jpg](https://dl.dropboxusercontent.com/u/13941904/cars/teslas.jpg)

It makes buying more cumbersome and we paid 20% down on something we never seen or even driven a test model. I also was not able to put TX sales tax into my loan and had to pay the lump sum when registering the vehicle here.

Keep in mind you can now see the S at the service center and take it for a test drive but you need to make an appointment. The problem is they can't (or not supposed to) help you with buying or financing there, you have to do it over the phone and on the website.

</comment><comment><USER4> That is a similar argument used as when the Amazon tax was first proposed, with the retailers complaining about Amazon's competitive advantage due to not having shipping facilities in New York and people not paying sales tax.

A slightly less onerous solution would be to institute a sales tax for out of state car sellers selling to NY buyers.

<comment><USER5> You are required to pay sales tax on a newly purchased vehicle when you register it in NY state. If you buy it from someone out of state, you still have to pay NY state sales tax:

[http://dmv.ny.gov/node/2086](http://dmv.ny.gov/node/2086)

</comment></comment><comment><USER6> Can someone explain why you can't take a Tesla for a test drive in Texas? If the "showroom" doesn't actually sell any cars, then shouldn't it be considered a non-retail facility (no different from the Tesla charging stations in Texas)?

<comment><USER3> You can still take a test drive in TX, I've done it. Just make an appointment. I'm not sure how they can do it legally. My guess is local contractors are hired to do it and not Tesla themselves.

</comment></comment><comment><USER7> Wonder if this is what the talks between Apple and Tesla were about?</comment>


Fighting Cybercrime – What Happens to the Law When the Law Cannot Be Enforced? - <AUTHOR> https://www.weforum.org/agenda/2019/02/fighting-cybercrime-what-happens-to-the-law-when-the-law-cannot-be-enforced/ <comment><USER2> If it is impossible to enforce laws against cybercrimes, one option would be to bring back the concept of outlawery: [https://en.wikipedia.org/wiki/Outlaw](https://en.wikipedia.org/wiki/Outlaw)

Once a jury finds that a given domain was responsible for a crime but it was impossible to actually go after them, the government would declare them to be a fair target for any hackers.

Note: I don't know if this would actually be a good idea. I just think its an interesting idea.

<comment><USER3> A principle that is discussed is hacking back. It actually almost passed in Georgia, USA. [1] The idea would be that if something was attacking you, you could go on the offense to disrupt it. Personally, I think that would be a lot of fun.

1\. [https://threatpost.com/georgia-governor-vetoes- controversial...](https://threatpost.com/georgia-governor-vetoes- controversial-hack-back-bill/131822/)

<comment><USER4> It almost passed in Georgia because our legislators are idiots. They also almost passed a law criminalizing security research altogether.

The problem with allowing open season is that unlike the wild west, outlaws did not literally hijack the bodies of victims to masquerade as someone else before engaging in criminal activity. The "hacker" you're attacking is usually another victim of the operator serving as a proxy. Attribution in the cybersecurity space is very difficult and prone to error. You'd just be institutionalizing lynch mobs.

</comment></comment></comment><comment><USER5> There are certainly cybercrimes that could be much better enforced. Running DDoS botnets of insecure network cameras, distributing ransomware, making robocalls, revenge porn, and so on.

But there's another set of cybercrimes where I don't really care about enforcement. Running Sci-Hub, bypassing DRM to refill inkjet printer cartridges, violating superinjunctions, and suchlike.

Sadly, the latter set tends to have much better lobbyists than the former, so I'm always nervous about the outcome of legislators attempting to improve the law.

<comment><USER4> > revenge porn

I've always wondered why USC 2257 is never pulled out to deal with this. It seems like the sort of law enacted to allow arbitrary punishment of porn producers-- just find some fault in their paperwork to wreck their day. An angry ex with your honeymoon tape has no such paperwork at all.

</comment></comment><comment><USER6> so much bullshit to force new abusive legislation. at least they avoided "think of the children"itis

nothing on this topic is new. laws for wire fraud and money laundry already lay a foundation for prosecuting international crimes. the problem is one of three groups (us-eu-jp-au, Russia, China) selectively enforcing prosecution for whatever reasons. new legislation will change nothing.

</comment><comment><USER7> I always thought the way we talk about "unsecured" services was wrong. I leave my door unlocked all the time and have no expectation someone will come by and test it. Were they to see it unlocked they'd have no expectation they were allowed in. If someone does break in nobody suggests it was because their choice of lock was "weak".

I don't expect I'd ever see it in my lifetime, but I would love to live in a world where people don't try random ssh passwords thousands of times a day against my servers. Being on the internet is like living in a bad neighborhood.

</comment><comment><USER8> Law can't be enforced because some powers don't want that. What about money laundering that could get hidden within seemingly legit transactions for multi billon dollars companies? Wouldn't be abusive if they try to regulate the behaviour of people buying online? How do they know the way to conduct that kind of investigation? Doesn't seem realistic.

</comment><comment><USER9> It erodes the authority of the lawmaker.</comment>


Start-Up Slowdown – How the U.S.
Can Regain Its Entrepreneurial Edge - <AUTHOR> http://www.foreignaffairs.com/articles/142498/robert-litan/start-up-slowdown

<comment><USER2> This article confuses the term "startup" and "new business".

By their definition a new hot dog stand, a new independent pharmacy, or a new plumbing business is a startup.

They are not.

A startup is a business that can potentially have very rapid growth.

I actually agree with some of the recommendations of the article but I can't take the authors seriously when their analysis throws everything into the same bucket.</comment>


Why context is so important - <AUTHOR> http://joel.is/post/31109939037/why-context-is-so-important

<comment><USER2> I prefer the word "structure" in place of "context" because people seem to have a better understanding of what that means and why it's important.

Whether we're talking about words (the context of a politician's remark) or experiences (the context of a startup founder's dilemma), these things do not exist on their own. They have a history. They relate to other words and other experiences. There are sometimes complex interactions between different pieces.

The key to understanding context is _doing good research_. Ask the right questions, look for the right pieces of information, uncover the important players. Read or listen carefully, taking notes if you have to. Follow where your research leads even if it's unexpected. Once you've got a solid grasp on the subject (and not before), you can make a plan for going forward.</comment>


How I came to write D (2014) - <AUTHOR> http://www.drdobbs.com/architecture-and-design/how-i-came-to-write-d/240165322 <comment><USER2> Discussed at the time: [https://news.ycombinator.com/item?id=7555426](https://news.ycombinator.com/item?id=7555426).</comment>


Introducing Infinithree: Against Wikipedia Deletionism - <AUTHOR> http://infinithree.org/post/2579843779/introducing-infinithree <comment><USER2> Hopefully this takes off.

After years of complaints and issues with rampant deletionism and abusive editors, Wikimedia has _finally_ started to collect statistics on how this has negatively effected the quality, depth and breadth of the project.

I can't find the Wikimedia Foundation report on this (it coincided with the great programming deletion incident), but it was almost universally negative. Some of the more recent data collection tells a similar story, it shows that new users, while attempting to make good faith contributions, are quickly drummed out of the place by established wikipedians via negative comments and rampant deletionism.

Despite this, and years of evidence that this is a serious problem (to the point of being blindingly _obvious_ ) with the project, the foundation hasn't taken any particular move to rectify it. I fear they will never get around to doing anything, or the response will be tepid and pointless.

With bits being basically free these days, there's just no real reason to limit Wikipedia to just being a duplicate of a Paper-and-Ink encyclopedia.

<comment><USER3> > With bits being basically free these days

The work required to maintain quality is not free though. The larger Wikipedia becomes, the more resources are needed to ensure a high quality.

There is already a system completely free of "deletionism" with limitless scope which anyone can publish anything on. Its called the web. Why do people want their favorite topic published on Wikipedia rather than on their own page? Because they want the additional authority which comes from being on Wikipedia. But this authority comes from quality control.

I think some of the complaints against deletionism could be resolved with independent domain-specific wikipedias with different rules. For example a programming language-pedia could allow original content written by the language designers, something that is not appropriate for wikipedia.

<comment><USER2> _Why do people want their favorite topic published on Wikipedia rather than on their own page? Because they want the additional authority which comes from being on Wikipedia. But this authority comes from quality control._

I actually find wikipedia's aggregation and summarization of information just as, if not more, valuable than the authority angle.

There actually are some quite interesting domain localized wikis around, on top of the random wiki for a game or fan wiki (or encyclopedia dramatica or conservapedia for that matter)

<http://www.wikia.com/Wikia>

<http://www.wikispaces.com/>

</comment></comment></comment><comment><USER4> I see the most promising line as actually being something considerably different from Wikipedia, rather than an opposition to it, which despite the top-line rhetoric does seem to be what the proposal is hinting at.

Wikipedia aims to be a summary of existing sources (books, journal articles, newspapers, etc.), with citation to where it got its material, but originating nothing; basically, the ultimate tertiary source. I think that's a useful thing to have, but it's not the only possible thing you could imagine building.

"True and useful" is a rather different set of criteria, and there is a whole space of websites that could pursue that, ranging from some that already exist (music and literary fansites, some academic wikis, knowyourmeme.com, Everything2) to many that don't yet.

<comment><USER5> For one thing, it implies the ability to work on primary research right there in the site. It would need sufficient justification to be accepted as "true" and it would need a "use" to somebody even if only in the same field, but that covers basically all well conducted academic research.

</comment></comment><comment><USER6> My longstanding beef with deletionism is that it goes against the Wiki Way.

If Ward's wiki had operated in the same way, it wouldn't have been anywhere near as influential and useful.

<comment><USER7> But Ward's wiki is topic-focused, which is itself a good layer of defense. It's full of important (to us) but abstruse subjects like Liskov- substitutability, which simply aren't going to hold the interest of the people who want to maintain exhaustive lists of Pokémon characters.

<comment><USER6> Hmm, that would suggest a decentralized network of domain-specific wikis would be a more robust model than Wikipedia's centralized one.

Back in the day, Lion Kimbro was heavily advocating for something like that using the notion of WikiNodes to interconnect the wikis: [http://meatballwiki.org/wiki/action=browse&id=WikiNode&#...</a>

</comment></comment></comment><comment><USER8> As a "deletionist" (at least in theory if not in practice) I welcome this project.

I'm glad that somebody is finally doing this, instead of just sitting around complaining that wikipedia isn't what they want it to be. Complaining that wikipedia isn't an indiscriminate collection of information is like complaining that Target isn't a fine lobster restaurant -- it was never intended to be and it's fine the way it is. There's enough "inclusionists" out there that I'm sure they can build a parallel encyclopaedia with a slightly different philosophy, and if (as I suspect is true) it turns out to be swamped with useless and unverifiable information (not to mention a huge number of vanity pages) then at least we'll know.

However:

 _[it] will discard 'notability' and other 'encyclopedic' standards in favor of 'true and useful'_

I'm not sure how "useful" differs from "notable" and I suspect they're just going to wind up having the same arguments about different words. For instance, is a ten thousand word biography of my cat both true and useful? It's certainly true, and it _might_ be useful to someone at some point, so why delete it?

<comment><USER9> "Complaining that wikipedia isn't an indiscriminate collection of information"

Nobody's making that complaint, so that's a classic straw man. The complaint is the legitimate information is being kept off the project by people who don't think it's important enough for wikipedia. I recently argued against the proposed deletion of an article on a charter school at its AfD. Despite the school being mentioned in many articles, people were arguing that elementary schools simply aren't worthy of mention on Wikipedia. It is this attitude that we oppose, not that "that wikipedia isn't an indiscriminate collection of information".

<comment><USER8> _Nobody's making that complaint, so that's a classic straw man._

Well, I've met at least one person who advocated the idea that wikipedia should contain any information that was true. When I asked him if I then should be able to create a many-terabyte article entitled "List of random numbers that hugh3's computer generated at [such and such a date and time]" then he said it should. So there are certainly _some_ folks taking up that extreme position.

Otherwise you're just arguing at the margin. Maybe the notability criteria for schools _are_ too strict. Maybe they're not. I can't think of anything about which I give less of a damn. (For what it's worth I checked, and my high school is certainly listed while one of my two primary schools is and the other [no less intrinsically notable] redirects to the particular suburb in which it is located].)

<comment><USER9> The person you cite is in no way representative of inclusionists. The analogous position would be to say that deletionists believe Wikipedia should only have articles on subjects that Britannia has articles on. You might be able to find a person who holds such a view, and it might be right to call them a deletionist. It would be disingenuous, however, to say that deletionists hold those views.

<comment><USER10> The sad part is how often the type of pattern you identified is used, especially on the 24 hour "news" networks by pundits of all sorts. Take any group and find the extreme or hangers on then use those people to define the group as a whole.

</comment></comment></comment></comment></comment><comment><USER11> Quite ambitious. I like it.

This post is a few months old, though, and no public developments can be seen. I hope it goes on.

</comment><comment><USER12> Isn't Google the answer to deletionism?

Why should and and all content be stored in a special place called "wikipedia" or some other name, when you can write anything you want about any subject on your blog and have it indexed by a central search engine?

Wikipedia offers two things that lone blog posts don't: collaborative editing and "findability" (SEO).

Findability should be the problem of search engines; the fact that we need to go to Wikipedia (or IMDB...) to find specific information is a testament to Google's imperfection.

If you want to publish "a 10,000 words biography of [your] cat", you should do so on your blog, and that should be easy to find when searching for "hugh3's cat" (but not when searching for "cats", unless you're some modern-day Montaigne who can teach us eternal truths about cats just by speaking about his own).

I would even argue that Wikipedia hurts findability to some extent, since it relies on anonymous editing, and reputation builds on identity. Knowing who wrote what is a very useful piece of information (something Quora is addressing).

What about collaborative editing? Successful collaborative editing is made of two parts: software and a community. The software problem is largely solved; a working community of people who know and care, is the hard part. But it's also in the process of being solved by what StackExchange is doing: building sites around existing communities.

So here's what I think will eventually happen:

\- human-machine interface / findability engine: Google

\- clusters of (constantly updated) information: StackExchange sites (and others)

\- niche content: your blog

In this future I'm not sure there's much room for Wikipedia; Wikipedia was built around the idea that there is such a thing as "static" knowledge or "eternal truth". There isn't: knowledge is a process and a conversation.

(Wikipedia is in fact a kind of cache for the current state of knowledge, but that problem could probably be solved differently and more efficiently.)

<comment><USER3> Wikipedia is "the mainstream consensus", a compilation of knowledge that is generally agreed upon by reasonably sane and well-informed people, in a format accessible by the layman. I think that will continue to be valuable.

If you are a layman who wants to know "what is a computer" or "what is programming", then Wikipedia is a much better starting point than Stackoverflow or Don Knuths personal blog.

</comment></comment><comment><USER13> Hey! This is my project, and I almost missed this thread entirely! @$&*!# offline time!

The ∞³ blog has been thin on updates but I'm still aiming for a launch of core functionality soon and I'm very interested in feedback/collaborators.

</comment><comment><USER14> My biggest complaint with Wikipedia isn't deletionism, but epistemology. Stating that the sky was blue in London on Monday May 30, 2011 demands less rigorous verification than claiming that the rulers of some state did or did not commit genocide. One is superficially inconsequential while the other is likely to lead to a lot of strife.

To give a concrete example, a prominent US geneticist states in the biography on his personal website that his biological father was a highly decorated American pilot. This is a fairly uncontentious claim, yet fails to meet the Wikipedia standard of verifiability.

<comment><USER15> This is the difference between saying "the US geneticist's father was a decorated pilot" and " the US geneticist claims his father was a decorated pilot". The latter would be acceptable in Wikipedia, wouldn't it?

</comment></comment><comment><USER16> There are some major problems with Wikipedia but I don't think deletionism is big enough to sustain a new effort.

My biggest gripes are that many articles are just too long. I'd like to see a word limit or something.

Also, I really don't like the emphasis on citations. It leads to articles comprised no more of disorganized sets of citable sentences. Somewhat counter- intuitively, the citation emphasis leads to less popular but citable statements dominating in many places.

</comment><comment><USER17> Jason Scott proposes a different set-up[1], where he compares an evolution of Wikipedia to how newsgroups got set up, with different servers choosing to retain articles based on whatever rules they are run by. He, of course, envisions himself subscribing to one that would retain _everything_ , notability be damned.

[1] <http://ascii.textfiles.com/archives/2646>

<comment><USER8> Doesn't the article say that this (forking wikipedia) has been tried several times and never worked?

</comment></comment><comment><AUTHOR> I was ranting about this issue for long time, really like that someone is working on that.

</comment><comment><USER18> How is this different than everything2?

<comment><USER2> It'd be great if everything2 was converted into a wiki instead of the individual submission model it follows now.

<comment><USER19> It depends - for purely factual content, it would work well. However, it would also discourage the random articles, essays and writings which give E2 its particularly personal voice. Wikipedia is very much about stating the facts about each subject in an encyclopedic way; E2 often ends up being more about several peoples' (possibly differing) opinions on a subject. They're both very valid approaches, and I'm glad both can exist at the same time.

</comment></comment></comment><comment><USER20> Smells like vaporware. I never really trust it when people first make a blog and a domain and a fancy headline and a fancy logo, and write prose with lots of "will be's" and "will have's", all without showing anything substantial.

<comment><USER13> I'm delighted that two unicode characters smushed together ('∞³') strikes you as a fancy logo. If you're patient enough, you'll probably like the other design choices that are coming, too.

<comment><USER21> Don't take his criticism too hard. It's just part of the startup culture: Get _something_ out there and in front of real users, as early as possible, so you can begin the virtuous cycle of success.

<comment><USER13> No worries; this isn't my first spin around the roller rink.

I've struggled with the question of whether it's better to talk in advance of something to show, or just wait until launch and let the site speak for itself.

I've decided to talk about the vision and general themes before tangible launch for two reasons:

(1) With any such community-built project, who shows up in the early days will have a giant influence on how it develops... and I'm hoping a little bit of advance discussion will mean more of Infinithree's natural userbase can find it and participate as soon as it's ready.

(2) We'll need plenty of help. So if the general outline strongly resonates with potential deep (even eventually full-time) collaborators, I'd love to start working with them even before public launch.

So I'm taking the risk of both the immediate 'vaporware' and eventual 'underwhelmed' reactions with full awareness of how these things can go.</comment></comment></comment></comment>


Never Write Your Own Database (2017) - <AUTHOR> https://medium.com/@terrycrowley/never-write-your-own-database-736f704c780 <comment><USER3> Akin's rules for spacecraft design [0] include: "Any exploration program which "just happens" to include a new launch vehicle is, de facto, a launch vehicle program."

By analogy, any software project that includes writing a database is, de facto, a database project.

[0] [http://spacecraft.ssl.umd.edu/akins_laws.html](http://spacecraft.ssl.umd.edu/akins_laws.html)

<comment><USER4> Another common version of this is: video game project turning into a video game engine project, usually not even reaching the actual content phase.

<comment><USER5> But weren't such projects rather common?

<comment><USER6> 20 years ago, when much less things were expected from engines and graphics alike.

It's like a web framework. Coding one when CGI came out was way less work than now. If you want to match what django or rails does now, it takes years.

I read somewhere it would cost millions if you tried to hire dev to recode django at today's market rate.

<comment><USER7> While these tools such as django and rails or Unreal and Unity are excellent achievements, we'll never get even better tooling if really smart people aren't working on these sorts of projects.

I follow a project called Handmade Hero[0] by a longtime game and game tooling programmer where he codes a complete game from scratch with absolutely zero dependencies except Windows. It's an excellent educational resource and I've learned a lot about how games work, including game engines such as Unreal of Unity. This has enhanced my knowledge of how I might better use those engines and I feel that it's been a valuable use of my time.

[0] [https://handmadehero.org/](https://handmadehero.org/)

</comment></comment></comment></comment></comment><comment><USER8> Sometimes it's a huge advantage. I wrote a network search engine. On a single 1TB spinning disk, I could handle 5TB of traffic, stored and indexed, per day. That's around 2 billion packets indexed. The key was having an log/merge system with only a couple bits of overhead per entry, and compressed storage of chunks of packets for the actual data. (This was before LevelDB and Elasticsearch.)

In practice the index overhead per packet was only 2-3 bits. This was accomplished by lossy indexes, using hashes of just the right size to minimise false hits. The trade-off being that an occasional extra lookup is worth the vastly reduced size of compressed indexes.

To this day, I'm not sure of general purpose, lossy, write-once hashtables that get close to such little overhead.

Competitors would use MySQL and insert per packet. The row overhead was more than my entire index. But it worked out: just toss 50k of hardware at it.

But... It does take over a lot of engineering time writing such bespoke software. Just compressing the hashes (a common info retrieval problem) is a huge area, now with SIMD optimised algorithms and everything.

<comment><USER9> I'm going to go out on a limb and guess that it would have been cheaper to upgrade the hardware.

<comment><USER8> Depends on scale. At higher end, it was near impossible to scale when you're e.g. inserting a MySQL row per packet. But maybe good enough for a viable business. I would probably try to take it as far as possible on Elastic if I were to write it today.

Same thing if you read the Dremel paper. Worrying about bits helps when scaling.

<comment><USER10> Because Lucene wasn't good at near-realtime in 2009 or so, Twitter's original (acquired via summize) search was written in mysql. It might have even been a row for every token, not quite sure.

IIRC, when we moved to a highly-customized-Lucene-based system in 2011, we dropped the server count on the cluster from around 400 nodes to around 15.

</comment></comment></comment></comment><comment><USER11> I had to roll my own transactional database for an embedded product working in a very constrained environment (2MHz ARM, 2MB total memory). I did very extensive search and I found no product with the combination of characteristics I was looking for:

\- working in very small but more importantly, constant memory (predictability is a must for reliable embedded app),

\- provide provable transactional characteristics with guarantee to roll back to previous state in case transaction is interrupted and still be consistent and able to accept new transactions,

\- minimum storage overhead -- the amount of overhead was directly translating to competitiveness of the device at the time as multiple vendors tried to provide their own solutions for the same preexisting platform

\- storage write pattern that would provide wear leveling for the underlying flash

I ended up writing my own database (less than 1k LOC total with encoding/decoding data using BER-TLV) that would meet all characteristics, take few tens of bytes of stack, take few bytes of overhead per write. The database would vacuum itself and coalesce record state automatically. It had some O(n^3) algorithms but THAT'S OK since the amount of data could never be so large that it could pose any problems.

The project took 2 years to complete. I spent maybe a month designing, implementing and perfecting the database. I wouldn't say that the Akin's law of spacecraft design applies here. I would probably spend more than that if I had to integrate existing product and end up with inferior product anyway.

<comment><USER12> The title is misleading - it's actually about how and why they did end up writing their own db. From the article:

> There's an old adage in software engineering — "never write your own > database". So why did the OneNote team go ahead and write one as part of > upgrading OneNote's local cache implementation?

<comment><USER11> What I am really objecting to is those hard "rules". "Never optimize early". "Never roll your own database", etc.

All those rules work for most but not all projects. It's the same as saying "You shall allways obey traffic rules". Maybe I should, but sometimes I may not want to brake on yellow light when I have clearly impatient driver tailgating me.

As we gain experience we learn the world is not black and white. Akin's law #8 says:

"In nature, the optimum is almost always in the middle somewhere. Distrust assertions that the optimum is at an extreme point."

<comment><USER13> But your objection is rather silly!! Those rules are meant for beginners. If you're not a beginner you already know when/if/how you can break the rules. It is correct to teach kids that they should tell the truth in every scenario, even if as adults we don't.

<comment><USER14> "It is correct to teach kids that they should tell the truth in every scenario, even if as adults we don't."

Not correct. Example, a stranger: "are you alone a home?" or "what is the keycode for the door" etc.

<comment><USER13> I think its still correct. Sorry, I don't find your examples to be a reason to teach children that truth is optional.

<comment><USER15> It would be much better to teach children that the truth doesn't exist, that everything we call "reality" is a lie on some level, and that the best you can do is to stay honest to yourself and your intentions. Probably much harder to teach, but much closer to what they will fight with for the rest of their lives. I think it's the lack of such preparation in childhood which makes people give up, either never trying to figure out what the "truth" even is or relying on an imaginary friend in the sky as a source of one and only truth.

Some simplifications are useful in some contexts, but the world is an unimaginably complex thing and trying to dumb it down can only take you so far.

<comment><USER16> "the truth doesn't exist, that everything we call "reality" is a lie on some level"

Wait so does this truth exist, or is it also a lie on some level?

<comment><USER15> > does this truth

It's not truth at all. Just an observation. It happens to fit with my perceptions. That's it.

You can artificially define truth by tying it to a particular frame of reference, but that's not "the truth", as that frame of reference is not 100% transferable to others anyway. The idea of umwelt, as I understand it, seems to work here. Still, it's just an observation, an impression.

I'm not really saying anything new here. Descartes was saying something similar quite a long time ago. Then again, he could have meant something else entirely and there's no way for me to know which is it. I just assume my understanding is close enough to the intended meaning. I may well be completely wrong on this.

Basically, there's nothing you can be really sure about, including the fact that you can't be really sure about anything. There are only things that appear to work for you and, possibly, others. You can use them. Just don't believe them unquestioningly.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> Possibly relevant are these 2 great papers co-authored by Michael Stonebraker[1]:

 _"One Size Fits All": An Idea Whose Time Has Come and Gone_ [2]

 _The End of an Architectural Era (It's Time for a Complete Rewrite)_ [3]

They basically show that classical RDBMs are inefficient by around a factor of at least 10 in every conceivable application. I tend to trade in a little of that for the kinds of dramatically simpler mechanisms discussed in TFA.

[1] [https://en.wikipedia.org/wiki/Michael_Stonebraker](https://en.wikipedia.org/wiki/Michael_Stonebraker)

[2] [https://cs.brown.edu/~ugur/fits_all.pdf](https://cs.brown.edu/~ugur/fits_all.pdf)

[3] [http://cs-www.cs.yale.edu/homes/dna/papers/vldb07hstore.pdf](http://cs- www.cs.yale.edu/homes/dna/papers/vldb07hstore.pdf)

<comment><USER18> Do they identify any cases for which classical RDBMS are inefficient by a super-constant factor?

<comment><USER19> ACID semantics subtley and indirectly requires locks. Any scenario where lock contention might be a problem would be a problem for an RDBMS (assuming the traditional definition of RDBMS as a ACID compliant). Those scenarios can easily be "super-constant", especially if you have to distribute across multiple nodes. That requires locks that are held for the duration of an "external call" over the network which is A Bad Idea and could definitely be "super-constant".

</comment></comment></comment><comment><USER20> "Never write/roll your own" is the counterpoint to the NIH "Not Invented Here" syndrome... however, the older I get, the more reasonable NIH cases I see, such as the one in the article. People will say, for example, "why would you write your own ORM when there are so many available for free? Why reinvent the wheel?" Yet when I start looking at the wheels that are available, they take up a lot of space, a lot of time, take quite a while to learn (while the boss is asking me "is it done yet? is it done yet?"), have their own bugs, and need to be regularly upgraded and maintained against. This isn't necessarily a flaw - these "wheels" (ORMs, file storage solutions, web application frameworks and, dare I say it? security solutions!) are designed to solve generic problems rather than the specific targeted problem that I'm focused on.

<comment><USER21> If your problem is that your boss is asking "is it done yet?" writing your own new ORM does not seem like the most productive antidote to learning an existing hard-to-understand ORM....

<comment><USER20> Not in my experience. If I'm coding, and I can demonstrate that I'm checking stuff into Git (or SVN or CVS or Clearcase), and I have something to demo every couple of days, they're happy, even if there's theoretically a faster way to get from point A to point B. On the other hand, if my only progress (for a whole day!) is "researching" some software, they go into full-blown panic mode. I've found this to be absolutely consistent over a career of now 25 years and 10 different employers.

</comment></comment></comment><comment><USER22> If there are use cases even _now_ where a roll-your-own solution is best, imagine the 1980s.

My first programming roles were in a mainframe shop with a major defense contractor, which had some _brilliant_ mainframe system programmers over the years. Their major unclassified systems (payroll, shop order control, etc.) were internetworked with a homegrown real-time system, and they all had a home-grown disk access system (random and sequential) that was surreal in its speed and reliability, all coded in 370 assembler. On the business applications side, they had a thorough API that was callable from even COBOL programs.

By the mid-1980s, upper management decided they had to "standardize" and began developing replacement systems using IBM's IMS. Performance was _unusably_ bad. I left around that time to join the UNIX RDBMS world, so I don't know if they ever found a solution that could actually be rolled out.

Working with the older stuff was actually _fun_ because things just _worked_ , and the customers of our legacy systems were really happy.

</comment><comment><USER23> The major cost factors in software development are developer wages.

It's all about how you spend the time you have. Don't build stuff that you can get off shelf. You're not going to write a better database, even if you think you can. You may get great performance for the current system on day one (after spending a huge amount of time and effort on developing it, perhaps with zero value created) but over the lifetime of the database you'll incur huge costs that you probably can't even fully foresee. The details here are not clear enough, but working from first principles ("I want to build an Evernote-like app") I can't imagine an experienced developer suggesting you should write a database (cache/file system).

The best software development is mundane: glue together what's there, buy the resources you need to get sufficient performance. Switch a component if you need something to be faster, re-architect key parts, and buy the new components off the shelf too. It's fun to build new stuff with custom algorithms you work out, but instead of that you can go home at 5pm, try building a database in your spare time and learn why you shouldn't, and still have space to relax.

As Sam Altman wrote recently, the productivity tip that most people are astoundingly ignorant of is: choose what you do carefully.

It's telling that this article does not begin: "we had a bottleneck that could not be solved."

<comment><USER24> On the flip side - building just enough software to solve the problem at hand can sometimes run circles around assembling things using off the shelf components. This is especially true if something like speed / memory / correctness / optimality is important.

<comment><USER23> Could you give a concrete example?

<comment><USER24> a. High frequency trading

b. Embedded software

c. High performance computing

d. Robotics

e. Military / Aviation software

In each of the above fields there are categories of software where you can run circles around general solutions by writing something custom and yes some of those involve writing your own database.

<comment><USER25> and f. databases.

ha, just kidding.

<comment><USER24> As someone who works professionally on databases this made me chuckle.

<comment><USER25> I work on database software too, so I know it should be there. plus fuck databases, they are such a pain :-)

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> I have written my own database. In 3 months. Shipped it and installed in customers' sites, and it's been working great for many years now. So do write your own database, but know exactly how kernel, libc, language libraries, sockets work :-)

<comment><USER27> Alternatively, write your own kernel!

<comment><USER26> A coworker did but for an embedded device. It worked well but we did spend a lot of time debugging the networking.

<comment><USER28> Often such things "work" technically, but if you leave the project, some poor schmuck has to figure out what you did. It may be obvious to you, but not to others. Remember that maintenance is a bigger cost than development on average. Write systems for other developers, not for yourself.

<comment><USER26> Well it works and the poor schmuck is the person I hired and trained and they are still there. The db is still being sold and making money. Since code will be read more times than be written, I try to write it so that a junior version of me from 10 years ago could understand it, after waking up in the morning before the 1st cup of coffee :-)

The best advice is really to keep it simple. Build the system of out small, isolated units with clear APIs between them. The API and components should be isolated at runtime, so they can be restarted or fail separately, as well as in the code - separate modules / files / libraries. That way looking at it can be easier to understand how everything fits together.

The simple part wasn't just a platitude or a generalizing statement. A db is really a beast that can easily turn into a giant ball of spaghetti with a ton of features, settings and tweaks and never-ending list of bugs. A lot of work and thinking I did on it (including a rewrite after an initial prototype) was with the goal of making it simple by cutting unnecessary features.

</comment></comment></comment></comment></comment><comment><USER29> It seems to me that Microsoft frequently writes new embedded databases, leaving them poorly documented and accessible from weird tools. For example, the registry, Windows installer, structured storage.

<comment><USER30> Structured storage isn't embedded database tech. It's actually FAT (or close enough) inside a file, with a bunch of helper COM objects.

</comment></comment><comment><USER31> There are two old commonly used programs that have their own database - <USER2> and Sendmail. Both have problems because their database is crappy.

<comment><USER32> Doesn't OpenLDAP implement it's own database or use bdb? I remember it corrupting itself if the server lost power.

<comment><USER33> OpenLDAP uses LMDB now, one of the fastest and reliable key/value stores around, which they wrote precisely because berkeleydb sucked.

<comment><USER32> This was early 2016 from the centos 7 repos, so this must be relatively recent. It didn't even have to be writing to it. I just had to reboot and it was left in a dirty state.

<comment><USER34> CentOS repos often don't use current software, so not necessarily.

</comment></comment></comment></comment></comment><comment><USER35> The looser your requirements are, the easier it is to write your own so-called database. Nobody should be writing a full ACID/SQL/whatever database as part of another project. On the other hand, if you want to implement a "database" in the same sense that many implement a "filesystem" which is just an HTTP object layer on top of a real filesystem, knock yourself out. Just don't think you're doing the same thing as the people writing more featureful general- purpose databases.

<comment><USER36> If you're willing to accept that durability is a lie, and also to compromise (possibly a lot) on performance and trust that the CPU and RAM aren't actively helping the disk to sabatoge you, you can make a fairly full-featured ACI database in a weekend using Merkle trees and a single-threaded rendevous serializer. The hard part is getting those guarantees while _also_ having latency and throughput vaguely comparable to naive file writes. (I'd add something about a better API than assembly language, but database people decided to imitate COBOL, so I can't really criticise assembly with a straight face.)

<comment><USER19> Log structured merge is the last piece of your architecture. It solves the latency part. It can also solve your durability issues, if you fsync when necessary and you understand your hardware.

<comment><USER36> Yeah, sorry, I assumed log/journal plus reapply-on-recovery went without saying. You can't have guaranteed durability (short of blockchain-style distributed stores) because it's always possible for your database server to suffer a unrecoverable hardware failure between transmitting "transaction success" and taping out its weekly backup. If all else success, your datacenter catching fire and burning down would suffice. So it's only ever a matter of reducing (not eliminating) the _probability_ of having to roll back commited transactions. (Tldr: fsync can't help if your hard drive melted.)

<comment><USER19> Oh, I assumed we were talking distributed. Fsync to more than one disk. My biases :).

<comment><USER36> Fsync to more than one disk is great way to reduce the probability of forced rollback, but simutaneous failure is a still possible.

Then again, ACI does assume you aren't having CPU/memory/etc failures, so to the extent that you count anything as guaranteed, it is probably possible to 'guarantee' durablity. My original point was more that "minimally adequate" isn't that hard compared to going from there to a high performance, general purpose system (and also a offhand snipe at SQL).

</comment></comment></comment></comment></comment></comment><comment><USER37> I wrote my own database once. It's an "event sourcing" database, but I wrote it way before event sourcing was cool. It can handle over 1M events/sec on one core. I don't know any off-the-shelf product that could replace it.

I also helped write a log-structured OLTP-ish database that sits in front of MySQL.

I have only two regrets about these systems. First, MySQL was, in retrospect, the wrong backing store. Second, I used Thrift serialization. Thrift is not so great.

<comment><USER38> Did using MySQL allow you to ship a product on-time and under budget? Or, to put this the other way, if you had used The Right Backing Store would you still have delivered the product in a timely manner at the right price-point?

<comment><USER37> Unclear. I think the right backing store these days would be PostgreSQL. At the time, both were equally easy to get started with. PostgreSQL didn't have fully serializable transactions yet (IIRC), and I was more familiar with MySQL. OTOH, MySQL has sucked in myriad ways. Replication is very easy to screw up. The query optimizer and SQL advanced feature support is not so good. In general, there have been a lot of rough edges.

</comment></comment></comment><comment><USER39> There's also the (anti)pattern of making your own database in a lesser sense - implementing a set of tables on a SQL database for generic data. I think this is called the entity-attribute-value pattern. My impression is this is often considered deplorable, yet practically every application built on a database that I've encountered at work uses it to some extent.

<comment><USER40> As soon as you allow some kind of user-defined meta data, you basically end up with the entity-attribute-value pattern.

Every sufficiently advanced system does that, be it a CRM that allows custom fields, or a project management or ticket system, a workflow system etc. And basically all enterprise software contains elements of these systems.

<comment><USER41> A good alternative would be to use JSON types in RDBMS in combination with classic field types. EAV is really just free form data and JSON is easier to query in MySQL or Postgres.

<comment><USER40> Yes, but that's a fairly recent development, often without good ORM support, and I don't know how cross-database compatible the support is.

For a new application that only supports one database, it's probably a better choice.

</comment></comment></comment></comment><comment><USER42> "As alluded to above, this does introduce a separate challenge with multi-file consistency. The team took an interesting approach here. The index file points to the location of the root of the page graph and free space map in each page cache file. When persisting a multi-file transaction, all page cache files are flushed to disk before the index file."

That sounds familiar: [https://pthree.org/2012/12/14/zfs-administration-part- ix-cop...](https://pthree.org/2012/12/14/zfs-administration-part-ix-copy-on- write/)

</comment><comment><USER43> I used to work at a company where the "database" was literally a raw dump of the in-memory array of structs defined in a C program. Had some fun with that because struct padding depended on compiler flags, so different varieties of the program that were built with different compiler flags could not read each other's "database".

I never got over it, really. On the plus side, it was super fast, that I cannot deny. But it always seemed super gross to me.

<comment><USER44> Lots of software used to do this, back when computers were really slow and low on RAM. The old MS Office, Adobe Photoshop formats are notorious for it.

<comment><USER43> To be fair, the software had been first written in the early 1990s, so I can imagine it was a rational decision back then.

I was just always amazed that there never were any problems with corrupt files.

</comment></comment></comment><comment><USER45> I was not expecting this article to be about OneNote, but I'm glad that it is. OneNote is a surprisingly good product, but it takes so excruciatingly long to sync what must be 10KB of text. I look forward to what these changes enable.

</comment><comment><USER46> If you're writing your own database and you are serious about making it a business, reach out to me via my profile here. Ive been there done that, happy to help.

</comment><comment><USER47> We wrote our own database 15 years ago - needed something embeddable into the application on both windows and linux, i386 and alpha, and the ability to do transactional schema modifications without global locking. If I recall correctly, postgresql was at 7.3 at the time and would happily dump core if you tried something like 'drop table x;rollback'.

And we didn't need full sql semantics, because we partially implemented it into our scripting language anyway and used only cursor APIs. postgresql's mvcc was still a great inspiration for how to design the actual row storage.

Served us well, but 15 years later, a bit too much of custom features make it hard to switch to a standard database.

</comment><comment><USER48> If everybody never writes a database, who does it?

<comment><USER49> Domain experts for whom the database is the main project rather than a tacked on requirement.

</comment></comment><comment><USER50> I wrote a database for my app and it was the best decision we made.

<comment><USER51> Without a comparison of how good the other decisions were, this means nothing :)

<comment> zupa-hu that comment made my day xD

</comment></comment></comment><comment><USER52> _The advantage of rolling your own is that you get to optimize and control things at a level that is just not possible when working through a complex external component_

Then you stick it on a filesystem that does not work the way you think it does or even worse you stick it on a remote filesystem (eg CIFS/SMB) that does not work the way you think it does and is now backed by an FS that also does not work the way you think it does.

Your DB _may_ work fine but make sure you onboard the lessons that all the others have learnt through bitter experience.

</comment><comment><USER53> OneNote is fantastic. Of all the Microsoft things I have installed on my Mac, it's light years ahead of the rest in terms of stability, usability, speed. It doesn't get enough recognition!

</comment><comment><USER17> TL;DR: Don't build your own database -- here's how and why we (Microsoft OneNote) did anyhow and it worked out really well for us.

<comment><USER54> Did it ? last i've tried oneNote, it was a buggy piece of crap, coulnd't even finish the import(on win 7). so IDK , databases ?

<comment><USER55> I'd say that since this article was written well after Win7 that it's in response to your experience and other experiences like yours.

<comment><USER54> I tried the import recently.

</comment></comment></comment></comment><comment><USER56> At SunSed we have created our own DB system for the exact same reason:

WordPress average page generation: 1 sec

SunSed CMS: 0.03 sec

Database is almost always the reason for slow applications.

Edit: if you do caching right, WordPress becomes as fast.

</comment><comment><USER57> It took me a long time to feel comfortable with the idea of building our own database. For Stream it worked out well though: [https://stackshare.io/stream/stream-and-go-news-feeds-for- ov...](https://stackshare.io/stream/stream-and-go-news-feeds-for- over-300-million-end-users) Tools like RocksDB and Raft give you really flexible building blocks.

</comment><comment><USER58> We would have never had NoSQL ...

<comment><USER59> Yeah you would. IBM was doing it in the 1960s pre-SQL.

</comment></comment><comment><USER60> This article is timely. I have just written a database to fill a gaping hole in the ecosystem. It even understands SQL.

</comment><comment><USER61> Thing with rules is that you can't know when is right to break them if you don't know them.

</comment><comment><USER62> Well depends on the case, I needed a fast ip to country resolving, really fast one. I did my database in C for it and it was 120 times (yes!) faster than fully optimized postgres table, not to mention memory, disk and cpu footprint.

Custom made database for just a specific problem can be a lot faster than generic one. But depends on your knowlidge.

<comment><USER63> That's just a lookup table, not a database.

<comment><USER62> "A database is an organized collection of data."

By definition. First sentence in wiki. I don't know what are your definitions.

<comment><USER63> If you want to call a lookup table a database I'm ok with that.

<comment><USER64> But that means I'm writing my own databases at least a couple times a year. I'm not sure I'm okay with that.

</comment></comment></comment></comment></comment><comment><USER65> I'm pretty sure the mantra is "Never Write Your Own Crypto", not "never write your own database".

I know tons of people who roll their own flat-file storage engine and are perfectly happy with it, even scaling up to their moderate couple of tens-of- thousands of users. Nothing fancy.

Personally, I had to write my own database (just like what the author of the article wound up doing), and had a delightful time learning all sorts of things, and now it is one of the most popular databases out there ([https://github.com/amark/gun](https://github.com/amark/gun)) and I encourage others to try (if they have time) building one themselves!

<comment><USER66> The mantra tends to be "just use SQL" or "just use SQL, unless...". Similar to the advice "just use an int, unless...".

The wisdom in that is usually pretty sage. Reach for a SQL database unless there are reasonable and concrete reasons not to. Then, evaluate whether there's a need to roll your own, or if there's something you can grab right off the shelf.

I'd argue there are plenty of perfectly valid reasons to roll your own, and it's not _particularly_ difficult if you build on an already-proven storage foundation (file system, LMDB, etc.)</comment></comment>


Server and Client RCE in Git version 2.7.1 and below - <AUTHOR> http://seclists.org/oss-sec/2016/q1/645 <comment><USER2> A couple of thoughts on the potential impact: [https://ma.ttias.be/remote- code-execution-git-versions-clien...](https://ma.ttias.be/remote-code- execution-git-versions-client- server-2-7-1-cve-2016-2324-cve-2016%E2%80%912315/)

Server-side: github & bitbucket will get patched quickly, if they're even still vulnerable. Self-hosted installations like Gitlab will be more difficult, as it requires sysadmins to patch themselves. History has thought us this takes too long.

Client-side: possibly the biggest impact, as nearly every Linux distribution ships vulnerable versions. Any kind of local system user activity could trigger the RCE. Technically, that includes any PHP, Ruby or Python site that allows shell commands to be executed - which, by default, they nearly all do.

It has all the potential to be huge.

<comment><USER3> At GitLab this was fixed with [https://gitlab.com/gitlab-org/gitlab- ce/merge_requests/3240](https://gitlab.com/gitlab-org/gitlab- ce/merge_requests/3240) which we plan to release tomorrow in 8.5.7 [https://gitlab.com/gitlab-org/gitlab- ce/issues/14308](https://gitlab.com/gitlab-org/gitlab-ce/issues/14308)

<comment><USER3> You can already download the fixed packages of 8.5.7 right now.

<comment><USER3> And the blog post is out [https://about.gitlab.com/2016/03/16/gitlab-8-dot-5-dot-7-rel...](https://about.gitlab.com/2016/03/16/gitlab-8-dot-5-dot-7-released/)

<comment><USER4> Nice, thanks for the fast response!

<comment><USER3> You're very welcome.

</comment></comment></comment></comment></comment></comment><comment><USER5> Note: if you're using Ubuntu, there is a semi-official PPA that has a non- vulnerable version (2.7.3): [https://launchpad.net/~git- core/+archive/ubuntu/ppa](https://launchpad.net/~git-core/+archive/ubuntu/ppa)

<comment><USER6> But a fix should come via the normal update channel soon? I'm on wily, should I expect to add this PPA or risk vulnerability?

<comment><USER7> Ubuntu should announce the fix at [https://www.ubuntu.com/usn/](https://www.ubuntu.com/usn/) but I can't load the page right now.

(removed DSA link as per advice below)

<comment><USER8> That Debian advisory is a different, older vulnerability. Looks like they know about it but haven't released anything yet:

[https://security-tracker.debian.org/tracker/source- package/g...](https://security-tracker.debian.org/tracker/source-package/git)

<comment><USER7> Oops, thanks.

</comment></comment></comment></comment></comment><comment><USER9> Times like this i'm glad i'm still on Mercurial (no `strcpy` overflows in Python). Is anyone planning on writing a DVCS in Rust?

<comment><USER10> It wouldn't have to be a new DVCS. It could be a Rust implementation of Git. e.g. Java Git implementation (JGit) used by Eclipse, Gerrit, etc.

<comment><USER11> Pijul, linked in a sibling comment, appears to actually be a Rust implementation of Darcs (or at least inspired by it).

<comment><USER12> Pijul is not a reimplementation of darcs, it's based on a new theory, without the performance drawbacks of darcs.

Btw, rewriting darcs would really feel like reinventing the wheel. Haskell is safe and great already.

</comment></comment></comment></comment><comment><USER13> [https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2016-2315](https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2016-2315)

</comment><comment><USER8> Sounds like this could be a big deal for bitbucket.org and gitlab.com? Esp. considering private repositories there.

<comment><USER3> For sure a big deal, we're deploying GitLab.com as we speak [https://twitter.com/gitlabstatus/status/709888872549847040](https://twitter.com/gitlabstatus/status/709888872549847040)

<comment><USER3> And GitLab.com us updated. If someone finds anything please contact us [https://about.gitlab.com/disclosure/](https://about.gitlab.com/disclosure/)

</comment></comment></comment><comment><USER14> BitBucket Cloud is currently on Git 220.127.116.11.g1fb337f (Version Info link in the footer [https://bitbucket.org/support](https://bitbucket.org/support)). Anyone know what about GitHub?

</comment><comment><USER2> Public mirror here if the official ones go down: [https://marc.ttias.be/oss- security/2016-03/msg00180.php](https://marc.ttias.be/oss- security/2016-03/msg00180.php)

</comment><comment><USER15> Git kind of implies that you're going to execute something from the remote end anyway so it's not something like hartbleed....

<comment><USER16> Remote code execution is worse than Heartbleed.

<comment><USER15> Oops, my bad. I didn't realize this was a server side issue, that is pretty bad.</comment></comment></comment>


Show HN: Lifedigger – Learnable personal knowledge base - <AUTHOR> https://www.lifedigger.io <comment><USER2> I'm not sure I understand the USP from it. I'd love to have one place for my personal knowledge base. But how does this one actually work? I don't get it from the landing page

<comment><AUTHOR> IMO, a video would definitely explain our USP better than our current landing page, we will put one in place in a few days.

Think of it as an Evernote, with cross references and a review system.

Basically you create logs, small pieces of knowledge, and you put tags on it. The tagging system mimics a folder-based structure with less restriction.

LifeDigger was born out of a frustration of mine : All the note apps I have used, or even some PKB, were desperately static. You end up with a lot of notes/articles with no easy way to review them.

In LifeDigger, you can instantly search into them, by title, tags, priority ... But more than all, you can select some logs (a whole tag e.g.) and review them in a kind of slideshow.

</comment></comment><comment><USER3> Great app ! ;)</comment>


Tech and Liberty - <AUTHOR> https://stratechery.com/ <comment><USER2> Comments moved to [https://news.ycombinator.com/item?id=21453782](https://news.ycombinator.com/item?id=21453782).</comment>


Ask HN: What's a simple app/software that made a lot of money? - <AUTHOR> What are some apps or software that made decent money (6-figures+), but aren't hard to make at all?
Think something that a programmer could make in a weekend or two.
Does anything come to mind? <comment><USER2> Check out my site [https://IndieHackers.com](https://IndieHackers.com) and sort by highest revenue. The "simplest" apps on there (from a technical perspective) making the most money are probably Semantria ($150k/mo), ConferenceBadge.com ($45k/mo), Nomad List ($33k/mo), Storemapper ($21k/mo), and FormCraft ($13.4k/mo).

Of course, if the app is simple to build and makes a ton of money, that invariably means that the other components (marketing, sales, distribution, etc) are tougher to crack.

<comment><AUTHOR> This is an awesome looking site, but I doubt those apps were that simple to create. I could be wrong. I kind of meant like a "flashlight" app or something similar like that.

<comment><USER2> Ah, yeah there's nothing like that on the site doing six figures. A lot of these products did start off as small MVPs that could be built in a weekend or two, however. They just got more complex over time, especially as they scaled to handle additional customers. I suppose that's not a concern with a flashlight app :)

Can I ask what your goal is? Are you looking to hire a developer for cheap and then do the marketing yourself?

</comment></comment></comment><comment><USER3> How about Whatsapp? A plain old chat application, with some bells and whistles like attaching map locations and contacts, but more importantly removing the usual barriers of "Invite your friend to talk to them!", ended up worth 18 billion dollars (and counting).

<comment><USER4> Whatsapp was not simple.

Angry Birds?

<comment><AUTHOR> I'm pretty sure Angry Birds took a long time to make.

<comment><USER4> Flappy bird I meant to say

</comment></comment></comment></comment><comment><USER5> I am not sure! That the programmer could make the app with no problems. I mean that If it could be then it should be available for every county. And also for every currency as well. And The money can also be transfered in Bank accounts only.</comment>


A Sneak Preview of Wolfram Alpha: Computational Knowledge Engine - <AUTHOR> http://cyber.law.harvard.edu/events/2009/04/wolfram

<comment><USER2> Is this really going to be that amazing, or is Stephen Wolfram just flapping his gums?

<comment><USER3> I hope it isn't as game changing as the Segway was...

<comment><USER4> It is a New Kind of Search until proven otherwise ..</comment></comment></comment>


Ask HN: How bad is it to send Tax ID via plain text email? - <AUTHOR>

I recently received an email from BrainTree Payments to verify my merchant information. They need to get the correct information for tax purposes, I get it.
I was dismayed to see my Tax ID and Legal Name next to each other in a plain text email.
I am a developer, but I do not have a lot of knowledge in the area of computer networking. So I am not 100% sure of the implications of sending sensitive information via plain text email.
How bad is this? <comment><USER2> It's bad but not tragic. If your mailserver and theirs are both set up to use TLS connections by default, it's likely that your data was not available in plaintext on a public network. However the problem with opportunistic encryption is that while most service providers do set up their systems like that, it's not guaranteed.

Still, I'm mildly surprised that Braintree is not following the 'secure administrative login' pattern that most banks use for that type of account management information; since that reduces the risk exposure.

TL;DR they should make you log in to their https website.</comment>