How can C Programs be so Reliable? (2008) - <AUTHOR> http://tratt.net/laurie/blog/entries/how_can_c_programs_be_so_reliable <comment><USER2> The author makes a good point about the discipline imposed by not having exceptions. Programmers tend to write code in one of two modes: the quick-and- dirty mode where you consistently don't check return values, or the built-to- last mode where you consistently do. If you start in the first mode and have to fix a bug, you often have to add error checking all up and down the call chain from where it occurs to where it's handle, so you _very quickly_ migrate toward the second mode. By contrast, programmers in exception-based languages often get to a state where they catch some exceptions in some places where their tests have found bugs, but large sections remain in quick-and-dirty mode. The code tends to keep running longer after the actual error, leaving latent bugs that are harder to debug when they actually surface. The kinds of errors that are characteristic to C, such as segfaults from bad pointer handling, have obvious effects and more often have obvious causes as well.

I'm not totally anti-exception, and I think manual memory management is only appropriate in an increasingly small set of situations. I'd rather program in Python than C when I have the choice, but I acknowledge that the Python code I write is probably a bit less robust than the C code I write. I'm probably not alone. Theory aside, _in practice_ the greater prevalence of sloppy error handling in languages other than C seems to leave the field pretty level when it comes to robustness.

<comment><USER3> It depends on the problem you're trying to solve, I think.

Let's consider a command line application that fetches a URL, like wget. Without exceptions, you would check the return code of all the system functions you call (dns, sockets, etc), and if any of them fail you can't really recover. You just write out an error message and exit. With exceptions, you could wrap the whole thing and if any system function throws an exception, you print out the exception and exit. So it's much the same in this case.

Now put the url fetch code into a larger application. With exceptions this is pretty easy - you can catch any exception thrown by your url fetch code and then retry it a few times until it succeeds, or you inform the user of the app. Without exceptions this is tougher as you need to unroll everything manually in order to retry.

Finally we have something complicated like running a simulation. If something gives an error, we probably don't want to stop the whole simulation. We'd want to check the specific error result and then fix the simulation and continue. In this case exceptions would add a lot of boiler plate code and would need to be handled just as carefully as error results (eg, wrapping each call in its own try and handling each error).

So it's pretty clear that _it depends_ is very applicable to when you'd rather use exceptions or error results. In general I think exceptions are better as long as they're only used for actual error cases as it gives the user of the functions more control over where and at what level to handle exceptions.

<comment><USER4> But what if the exception was in your code and not the system call? This is incredibly dangerous.

Monads are perfect for this. Really combines the best of both worlds without all the unspecified/undefined behavior.

<comment><USER5> You could always write Monads in C... [http://blog.sigfpe.com/2007/02/monads- in-c-pt-ii.html](http://blog.sigfpe.com/2007/02/monads-in-c-pt-ii.html)

</comment></comment></comment></comment><comment><USER6> Popular C software is so reliable because of _enormous_ amount of effort spent developing, testing, and polishing it over many years - not because error handling in C is superior to that in higher-level languages.

If your python script sticks around for 30 years constantly being used by millions of people - I bet it will be rock solid as well. All other things [1] being equal [2], a program written in a higher-level language will likely be more reliable - simply because it will likely be shorter. C has many properties conductive to great longevity of software written in it, but error handling is not one of them.

Now, to the rant about not knowing the exact number of exceptions a function can throw. Most sane languages [3] give you exception hierarchy. You know that every function can throw an Exception (or Throwable, or whatever the root of all things evil is called in your language), and you either handle it or let in propagate. The important bit is _you catch exceptions based on what you can handle in a current context, not on what the functions you call can throw_ [4]. As long as you realize that every call can potentially throw, it is not important what exactly is thrown - letting you abstract away the exact nature of an error is a feature, not a bug, that's what exceptions are there for! Let's say some function deep below changed and now throws a new exception type. So what? It didn't change what you can handle, so there's no reason to modify calling function - just let the exception propagate. What if nothing in your program can handle that exception? You let it crash (hopefully to the debugger, and no, it's not ironic that I suggest it in a topic about reliable programs - crashing is a reliability feature invented to expose problems that would be hidden and hard to debug otherwise). Of course you may also put a few catch-all clauses in a very limited number of strategic places.

If you approach exceptions this way they are vastly superior to error codes.

[1] Scope of the problem, amount of effort spent on development, programmer skills

[2] Or at least on the same order of magnitude

[3] With a notable exception of C++ - I'm not sure what they were smoking when they decided that throwing arbitrary values is a good idea

[4] And this, by the way, is what's wrong with checked exceptions in Java

PS. Didn't mean that having no exceptions is insane - but that if a language has exceptions, the sane thing to do is to organize them in a hierarchy with a single root.

<comment><USER7> Yes!

Most software I've seen can be dramatically improved by removing exception handlers. (And then fixing the underlying problems).

On the server side I prefer one exception handler which is the OS, it will kill your process, release and free all memory, file handlers, etc. Then I wrap it in a bash script / other monitoring tool, and have it immediately restart whenever it fails.

As for the server itself I'll also add an automatic reboot to the server to ensure nothing out of the ordinary persists for very long (like that PATH modification that someone forgot to add in the appropriate place to get set on restart). If it's virtualized I prefer to destroy the entire server if possible.

<comment><USER8> Are the servers you're building serving concurrent clients? An exception could take out multiple in-flight requests.

(not against your idea, just curious how you handle it)

<comment><USER9> One advantage of forking servers; kill and reboot the parent, and you don't loose in-flight connections.

That said, I do this as well, even the best behaved daemon can get... funky... after a few months. Planned outages for a daemon restart are ok in my experience, particularly if you can fail over to other nodes as part of a rolling restart.

Of course, this refers to planned restarts, though forking servers helps with unplanned exceptions as well.

<comment><USER10> Don't you find performance suffers? AIUI this approach means you can only handle as many concurrent requests as you have processes, and the OS scheduler has less information to work with than if you were using threads.

<comment><USER9> Not typically: using forking daemons does not mean that you can't also use threads. The ideal model probably uses both - so they can be stuck to a processor, but still use threads per request. It's nice to have a library to abstract the implementation details away for you, but not necessary.

<comment><USER10> Right, but doesn't the error handling approach you describe mean allowing a whole process to fail whenever an error condition occurs, which would cause any requests that were being handled by other threads of that process to fail even though they were perfectly valid?

<comment><USER7> Can you really tell me that you know after an exception that the other threads are really in a well-defined state let alone 'perfectly valid'?

Look at something like ZeroMQ that is being rewritten specifically to avoid the non-determinism inherent in throwing an exception.

Once you're using threads it's pretty much anyone's guess as to what state the system is in at any point, add exceptions and it just gets worse.

<comment><USER10> An exception in one thread shouldn't affect others - why would it?

I agree that unstructured use of threading primitives leaves you with an unpredictable system, but it's possible to build safer, higher-level abstractions and use those.

<comment><USER7> Because fundamentally the only reason to use threads rather than fork is to share memory. If an exception leaves shared memory in an undefined state then all threads that share that memory are in undefined states.

<comment><USER10> True, but why would an exception ever leave memory in an undefined state? I can imagine it corrupting a thread's own stack in some languages, but you wouldn't use a thread's stack for shared memory (at least, not without some construct that told other threads when it was safe to access it)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> The author's point is weakened by the fact that the Converge VM is now (5 years after this article) written in python: [https://github.com/ltratt/converge/search?l=c](https://github.com/ltratt/converge/search?l=c) (as of [https://github.com/ltratt/converge/tree/08dadda29b/vm](https://github.com/ltratt/converge/tree/08dadda29b/vm)). Compare [https://github.com/ltratt/converge/tree/converge-1.x/vm](https://github.com/ltratt/converge/tree/converge-1.x/vm).

 _" there are some obvious reasons as to why it might be so reliable: it's used by (relatively) large numbers of people, who help shake out bugs; the software has been developed over a long period of time, so previous generations bore the brunt of the bugs; and, if we're being brutally honest, only fairly competent programmers tend to use C in the first place. But still, the fundamental question remained: why is so much of the software I use in C so reliable?"_

Why could the fundamental question not be explained by all the points he just mentioned?

I'm a C programmer, and this guy is hitting all my biases, but this isn't very well thought-out.

<comment><USER12> I believe he explains this switch here: [http://tratt.net/laurie/blog/entries/fast_enough_vms_in_fast...](http://tratt.net/laurie/blog/entries/fast_enough_vms_in_fast_enough_time)

It's a _very_ good read. IIRC, the reasons aren't really rooted in the language that is used. It's more about the tooling provided by PyPy for constructing a compiler.

</comment></comment><comment><USER13> Speaking as someone who has been a long time coder in C and Java, I think the point he makes about checking errors is BS. You have to handle exception in Java and you're perfectly free to consider all the possible failure paths if you want though typically you'll handle the ones you can sensibly do something about and ignore the rest (i.e.: log or just fail outright depending on context). I do NOT miss the C world where you have to handle the results of every single function call if you want to be safe. The exception handling approach IMO is generally superior - the normal code executes simply and the exception handling code gathers all the exception cases together. Much better than interspersing both and especially better than handling the same error 10 times in a row rather than just doing it once.

That said, I love C. It's a very simple language and that's a virtue.

</comment><comment><USER14> _" [Pointers are] arguably the trickiest concept in low-level languages, having no simple real-world analogy[.]"_

I'm not sure how "address" is not a good, "simple real-world analogy" for pointers. It may be that I've just internalized enough of the behavior of pointers that I'm glossing over important differences, though... I struggled with pointers long enough ago that I don't remember struggling with pointers.

<comment><USER15> For a lot of people I think it takes a while to really internalize the ramification of the fact that you can have a pointer to memory you're not allowed to use. (That, plus the way they're used in C as a cheap answer to templates/generics.) I understood that a pointer was an address from the start, but it was still some time before I figured out how to use them correctly.

<comment><USER16> You've never arrived at a physical address to find a shuttered business or gaping crater? I can understand the syntax of pointers being confusing, but the concept has distinct real world analogies. (That being said, I find the syntax of physical addresses in other countries to sometimes be confusing.)

<comment><USER17> I've definitely never arrived at an address to find a gaping crater myself.

<comment><USER14> I have - after demolition following an earthquake. I wasn't lead there _by_ the address, though - just passing by.

</comment></comment></comment></comment></comment><comment><USER10> Walking across a tightrope is a lot easier than you would think. When you're walking on a tightrope you really concentrate on your walking; it's stressful, but not actually hard. Still, I don't think many people would choose to walk across a tightrope when an easier option was available.

Exceptions, meanwhile, allow a counterintuitive but effective strategy for greater reliability. My boss likes to say that the cloud is great because machines fail more often - the point being that rather than the impossible task of making individual machines failure-proof, you instead design systems that can handle the failure of any given machine. Often the best way to make a resilient system is to break the problem into quite coarse-grained blocks, and then define appropriate responses (such as failing a single message and processing others, or retrying the whole block) whenever a given block fails.

</comment><comment><USER18> "Once one has understood a concept such as pointers (arguably the trickiest concept in low-level languages, having no simple real-world analogy) "

huh?

So when you write a letter to someone, you actually attach their house onto the outside of the envelope? No of course you don't. You write their ADDRESS on the envelope.

All this lore about how difficult pointers are, is bunk. The syntax of pointers vs addresses vs dereferenced pointer values, can be difficult at first ... and can be very difficult if you read and/or write unecessarily obfuscated code ... but there is no need.

<comment><USER19> Sometimes I just write the contents of their house on the outside of the envelope.

You have to admit it is one of the trickier concepts people learn when programming. Once you understand it, it's definitely a "Duh, how else would it work?" moment but getting to that point can take some time.

<comment><USER20> I agree. The concept may be simple, but when you're just starting out as a C++ newbie and you've got pointers-to-pointers-to-pointers flying all over the place in this ugly syntax trying to satisfy some contrived homework assignment on a short deadline, things can get confusing rather quickly.

</comment></comment></comment><comment><USER21> C also has "better" I'd argue that average person programming in C is more matured, and understand nitty gritty of programming, computer science, performance and how things work at low level far better than, say, Visual Basic or even Java programmer.

I think we have been putting too much faith in to languages. After certain level of sophistication, the weight of quality shifts to person programming in it instead of language itself. However most language designers would tell you that their holly grail is to shift this threshold so programmers can be as lower in skills and ignorant as possible while quality of program as high as possible.

Lot of these things can be also said for JavaScript. I've now coded fair bit in plain JS constantly avoiding temptation to use some other language that compiles to JS (and even avoid TypeScript for that matter). Just 100% pure JavaScript. And I can tell you that my habits and flow is very different when I code in JS. Back then I though coding in pure JS would blow everything up just because of lack of types or silly thing like missed semicolons or as simple stuff like forgetting to pass 10 in parseInt. The fact is that none of these bothers me at all anymore. I rarely find bugs that can be attributed to these deficiency in JavaScript. Strongly typed languages now feels like some kind of over hyped hoax.

</comment><comment><USER22> > it's reliable in the sense that it doesn't crash, and also reliable in the > sense that it handles minor failures gracefully.

This is an incomplete definition of reliability. It seems that the author is only considering programs which work as advertised when used as intended.

Most software is written optimistically, i.e., the "good path" is tested most. As soon as you start feeding the program with invalid input, it is no longer reliable, as witnessed by the myriad of low-level security bugs (stemming from the nature of C) in virtually every nontrivial C program ever written.

Now, would you rather have your program crash (be "unreliable" by the author's definition) or end up running arbitrary code injected by an attacker?

(Security exploits are the most prominent example; many programs will do something weird when fed unexpected input.)

</comment><comment><USER23> There are idioms and methods for writing reliable C programs, which you learn with experience and reading better code than yours. If you learned languages in the order higher-to-lower, you eventually understand that you cannot write a program in C at the same pace as in a higher level language.

What has worked for me is: always have a second terminal open for the manual pages and the standards; and to regularly read quality C source code. Opengroup.org has thorough references for the standard C library, including its relationship to POSIX and the SUS; and the various kernels and core Unix utilities are always a good read.

</comment><comment><USER24> I've never in my life written as careful and premeditated code as when I was writing C on my Amiga when I was young.

One miss and you would tilt the whole system.

There was no memory protection and everything was system wide. If stuff failed, you _had to_ deal with it because if you didn't you probably left something unlocked or left a resource open which hogged it and prevented other applications from using it. And anything left open would leak memory as well. And there was only system memory so other programs would eventually fail memory allocations because _your_ program didn't close some handle somewhere.

Surprisingly, most of the programs I wrote worked correctly early from the beginning. There was no "I'll just code this thing up and fix bugs later" phase because there were absolutely no safety nets. And fixing bugs in retrospect usually meant rebooting your machine, starting up the editor and compilers again for another round.

I'm pretty pedantic on Unix too but I'm still a lazy bastard compared to those times. And my C programs on Unix do take a whole lot more iterations to "stabilize" than those I wrote on my Amiga.

In some perverse masochistic way I somehow miss that.

</comment><comment><USER25> It is an interesting observation. I have found that scripted languages are unreliable as a function of loadable modules (which is to say they depend on some module that gets loaded, that at some point changed, and then broke the script). C has that problem too with shared libraries but there are generally fail stop policies in place (if you can't get the version you were linked against exit rather than try to valiantly proceed)

What is perhaps most important is that software does not "rot" in the sense of building materials becoming weaker after exposure to the elements. But it does fair poorly against changes in things it assumes to be true. Some languages, like Ada or Mesa, which have very detailed configuration specifications, achieve high reliability at the cost of high configuration specificity. That makes them brittle against even modest changes in their environment.

The art is striking the balance. Take something like thttpd, 20 year old code still compiles, still does exactly what it always did (simple HTTP web server) very reliable. But the barest minimum of reliance on the ambient environment it runs in.

<comment><USER26> > _I have found that scripted languages are unreliable as a function of > loadable modules (which is to say they depend on some module that gets > loaded, that at some point changed, and then broke the script)._

That's why I'm thankful for a lot of these newfangled virtual environments and machines along with their respective package managers (and why more people should use them!).

</comment></comment><comment><USER27> C programs are so damn reliable I spent last three weeks mostly trying to debug others' code (from multiple software projects - I was lucky enough they decided they'd all start hitting obscure bugs at approximately same time) and waiting for it to crash again.

Just imagine my feelings when a critical server that ran just fine for 3 years gets a bit more load and starts getting assertion failures. That was a known and already fixed bug, but after version bump to next stable those had changed to SIGSEGVs - and it's not trivial NULL pointer dereferencing but a weird concurrency issue.

C has neither [relatively] strong guarantees of OCaml and Haskell (so I could be at least sure there're no trivial bugs with that weird macros doing crazy pointer juggling), nor dynamic flexibility of Python and Erlang (so I could hack working runtime live, letting it fail and hot-swapping code at will).

</comment><comment><USER28> Adjusted for effort I don't know of any evidence that they are...

<comment><USER29> How does effort affect reliability? Even if you consider one as consequence of the other ("it requires more effort to make a reliable C program than the equivalent program in $language") that doesn't make the final product any less reliable.

<comment><USER30> It does when you only have a set amount of effort that you can dedicate on a project. Which is usually the case.

<comment><USER31> I am not sure that's a useful criterion in practice, though. People are fond of saying that languages will not make you orders of magnitude more productive, and I think the corollary is that on average they will not make you orders of magnitude _less_ productive either. (Controlling of course for characteristics of problems and such.)

The problem is that there's not really a great way to reason about this, either. If you're talking about how long it takes to write a program in C vs Java, how do you measure time spent writing code vs debugging vs bug fixing, et al?

All in all it's certainly possible it takes longer to write code in C, but for all anyone knows, writing more reliable code up front means fewer bugs later on. Code which is more performant by default may lead to fewer performance regressions when someone induces pathological GC behavior. Contrariwise, there's stuff like memory corruption as the OP describes, which is probably way more difficult in a native environment instead of a hosted one.

It's hard not to reach the conclusion that we're all just talking out of our hats anyway (myself included).

</comment></comment></comment></comment><comment><USER32> Also, compile-time checks and the tendency of lower-level languages to fail catastrophically rather than subtly or silently plays a part. The author also has a lot of experience to draw on, which might just make his work more solid and error free in the first place.

</comment><comment><USER33> Some good points in the essay but I have to admit that for me the reason I like C is that it's the first language I ever really did any serious programming in and the first language that made me think hard. Because of that it just feels like home when I come back to it and I can never really evaluate it without that bias. Its the simplest model of the machine that you can get without going down to assembly. Its the same reason I like riding and fixing bicycles. In both cases sparse resources have kept the design to the bare minimum.

</comment><comment><USER34> There are multiple factors that contribute, but here's a few that I haven't (to the best of my recollection) seen mentioned so far: tooling (crucial), "do the simplest thing that could possibly work" attitude brought about by (lack of) a built-in collections library and simple syntax, and lack of rapidly changing requirements during development.

Tools for C are powerful, mature, and available on near every platform. Every single C programmer that I know uses multiple memory debuggers, for example: e.g., valgrind for leak checks and more on smaller code segments, LLVM address sanitizer on unit test runs, jemalloc or tcmalloc/google-perftools, and more.

"Do the simplest thing that could possibly work" is somewhat close to the author's point: simply put, while there are widely used cross-platform hash tables in C, it takes more effort to use one; hence, they aren't used e.g., when N is known to be 100 (or if maximum size is bound, there's likewise no pressure to use the built in hash table/red-black tree and one can use perfect hashing instead...). In another big distinction I see between code I wrote in C vs. higher-level languages is that one can simply use a stack allocated variable length array -- or (if random access doesn't happen) a linked list in place of a heap allocated resizable array.

These exammples, I think demonstrated how C discourages you from interpreting "the simplest thing that could possible work" as "the thing that is fastest to code"; that's not to say development speed is not an issue -- it's a _huge_ issue! -- but it often leads to code where the unneeded complexity (such as using a red-black tree or a skip list instead of a sorted stack allocated array!) is hidden from the programmer but is nonetheless present.

Onto the last point, I agree sentiment expressed by others questions the premise. I think the examples of C programmers where we explicitly know of something as being a C program are heavily biased towards well known and open source software; open source software, language runtimes, operating systems, browsers (usually in a clean and limited subset of C++ like, e.g., Chromium and likely Firefox as well -- I only mention Chromium as I've re-used base libraries from its codebase), and so on. We don't, however, think of parking meters, vending machines, and many other usually embedded (but not life critical) systems that -- unlike open source systems software -- are built from unclear, confused, and ever changing business requirements ("well, this city introduced a new red-zone, so now we have to make sure to charge extra 0.33 cents between hours 8 and 13 except on all Holidays, but not July 4th").

Yet when people bring up Java, it's easy to forget about gmail or Minecraft, but instead to think of the insurance claim management system that you had to use after your rental car got rear-ended near 4th and Harrison, that crashes with a user-visible stack trace when you click the wrong button and (even in 2013) doesn't use Ajax and requires opening a new page (slowly, as they don't use Java's built in thread-pools but spawn new threads, because the system has to run at a customer site that uses Enron's custom implementation of Java 1.3 on an AS/400) to make an changes.

<comment><USER35> >Tools for C are powerful, mature, and available on near every platform.

If only this was true for embedded platforms! I have no tools at my disposal aside from a vendor supplied debugger. No pre-existing test infrastructure exists, what we have we have had to build up ourselves.

One of C's last remaining strongholds is in embedded, and the tooling in this world is atrocious! (Though some of the instruction and performance profiling tools are beyond amazing!)

Rigorous coding standards and a heavy reliance upon code reviews keep a code base clean and sane, sort of like on any other project written in any other language!

<comment><USER36> Last remaining strongholds?

C is everywhere.

I'm in a similar situation with embedded stuff at the moment. We're basically stuck with debug-by-printf over a serial port, far from ideal!

I have worked on other embedded projects that were better, for instance you can make a gdb server that operates over serial or tcp/ip and allows you to use gdb (or even graphical debuggers like ddd) on a host machine to step through code as it runs on your device. This does rely on there being resources available to do this though I guess...

</comment></comment></comment><comment><USER36> Because those of us who have been doing it for 15 years or more know we're tangling with tigers and have figured out how to use their strength against them...

</comment><comment><USER31> These pieces are the reason why I still read HN today. Thanks for submitting it and thanks for upvoting it, everyone.

</comment><comment><USER37> I should note that less experienced programmers tend to start with higher level languages, as a rule, so the average experience of a C programmer is likely higher than that of the HLL programmer. This general experience level alone gives the resulting code a reliability boost.

<comment><USER38> With notable exceptions, I think this is true. I've been coding seriously for 2 years and am starting to move down the stack. I see others in my cohort doing the same thing.

</comment></comment><comment><USER39> > "Surprisingly few people can really program in C and yet many of us have > quite strong opinions about it."

Maybe they have strong opinions precisely because they don't find C to be a language which is easy to "really" program in?

</comment><comment><USER40> C programmers have better culture. Than say Java programmers. If I was a swordsman and had a rusty sword and you had a new one, the sword does not decide the fight.

<comment><USER41> Yes, mainly because new sword has an auto-grinder welded to it that might start running at any given moment, even if you are in a deathly fight :-)

</comment></comment><comment><USER42> Kent Pitman's idea of "languages as political parties" [1] seems to me to offer the best perspective on the surprising reliability of C. The core community shaping C over the years has been:

1\. Unix-centric, so the community has a coherent view on the preferred sort of semantics offered to users of code;

2\. Applies Kernighan--Ritchie--Pike coding style that gives a clear "code smell" [2] for C code; and

3\. Is the natural home of "Worse is Better", which favours source-code simplicity over specification simplicity.

For example, the "C party" favours procedures "succeeding" with non-zero exit codes on failure where other language parties have language constructs to represent failure. This can be better from the point of view of source-code simplicity, both in code and compiler, than exceptions, and worse in terms of semantics. The LK coding style, among many other C coding styles, prefers the use of goto (or sometimes setjmp/longjmp) to handle exit status [3], which is frowned upon in nearly all non-C programming language communities; the restriction of goto/longjmp to handling errors in this way avoids the problems Dijkstra pointed out [4,5], and correct usage of goto/longjmp in properly checking exit status has good code smell if generally accepted coding conventions for the use of goto to handle failure are followed.

There's a nice fragment of code illustrating the idea on SO [6].

[1]: Pitman 1994, Lambda: The ultimate political party, [http://www.nhplace.com/kent/PS/Lambda.html](http://www.nhplace.com/kent/PS/Lambda.html)

[2]: [https://en.wikipedia.org/wiki/Code_smell](https://en.wikipedia.org/wiki/Code_smell)

[3]: From [http://www.tux.org/lkml/](http://www.tux.org/lkml/): _So now we come to the suggestion for replacing the goto 's with C exception handlers. There are two main problems with this. The first is that C exceptions, like any other powerful abstraction, hide the costs of what is being done. They may save lines of source code, but can easily generate much more object code. Object code size is the true measure of bloat. A second problem is the difficulty in implementing C exceptions in kernel-space. This is convered in more detail below._

[4]: Dijkstra 1968, A case against the goto statement, [http://www.cs.utexas.edu/users/EWD/transcriptions/EWD02xx/EW...](http://www.cs.utexas.edu/users/EWD/transcriptions/EWD02xx/EWD215.html)

[5]: Knuth, Structured programming with go-to statements, [http://pic.plover.com/knuth-GOTO.pdf](http://pic.plover.com/knuth-GOTO.pdf)

[6]: [http://stackoverflow.com/a/741517/222815](http://stackoverflow.com/a/741517/222815)

</comment><comment><USER43> Good Resource!

</comment><comment><USER44> Tarsnap is the classic example of a C program whose reliability almost defies belief. Its codebase is large; it solves a very hard problem; and it manages to handle almost every cornercase flawlessly. (As of the latest version, it seems to handle every cornercase flawlessly, as far as I know.)

But I've always wondered: if Colin were as experienced with Python as he is with C, would Tarsnap be better served by writing it in Python? It seems like the codebase would be half as large.

In other words, is C the reason why Tarsnap is good, or is it because of Colin? Or is it both?

Example: Lisp, along with PG, is the reason why HN is good. Many of HN's features simply wouldn't be easily accomplished without it, so they wouldn't ever have been implemented. But I'm not sure the same could be said of Tarsnap.

 _But let 's be honest - in how many languages can one retrospectively add a garbage collector?_

Many. Lisp in particular. HN's links like /x?fnid=KxPuWC6qnkl4nxcUTDh4xs are an example of this.

<comment><USER45> Wait, HN is good? The community is interesting, but I've never been particularly impressed by the implementation. In particular, the "feature" where "More" links and such randomly expire after a while just screams "shoddy implementation" to me.

<comment><USER46> The thing is, the "more" links are specific to each user and generated page. They're not just "page 3 at the time you click on the link", but the actual next 30 links following the 30 it's previously shown you. As a result, it needs to store state for each of these links it's created (IIRC, the fnid is a reference to the closure containing that state). For obvious memory management purposes, these need to be expired after some time.

I agree that it's an odd implementation choice though. I guess it made sense when HN was smaller and it could grant more ressources to each user. Nowadays, Reddit's simpler system (with risks of duplicates or missing rising links as you go deeper) would probably make more sense.

<comment><USER45> I understand that it's a tough problem and can totally understand solving it by just making the links quit working like that. But that sort of thing doesn't strike me as "good", even if it is highly pragmatic.

</comment></comment></comment></comment><comment><USER47> Since everyone has access to decent c toolchain these days you tend to see this over and over again. The "wisdom tradition" of UNIX? There is some kind of need for this kind of counter factual "the ancients had powers we can't believe" thing... the thing is that languages and frameworks surface sets of techniques for solving problems that if not present end up needing to be reinvented... this goes for C and UNIX, mainframes, os400, erlang...whatever. Maybe I'm just being contrary, but it seems to me that the lessons I learn in one language apply to all the others at least in some facet.

I worked my way through SICP last year and it really changed the way that I think about code...while also not making me a lisp nut. It seemed to me that lisp could have been anything else and the techniques would still be there.</comment>


Your top productivity web apps? - <AUTHOR>

Everyone keeps talking about web apps and how great they are. But I keep wondering: how many people use them, which web apps do they use, which ones have actually replaced their desktop alternative? How close are we to using productivity web apps in a mainstream fashion?
So, what do you think? What web apps do you use in your work routine? Which web apps have beaten their desktop alternatives?
Deyan <comment><USER2> Firefox + leechblock + <http://nowdothis.com>

jottit.com (better than Vim, OpenOffice, MovableType or Google Docs for compiling a set of notes about a subject over time. See <http://nixcommands.jottit.com> for an example)

<http://tmedweb.tulane.edu>, sort of a corner case, but I think there might be a lot of room for this sort home-spun productivity site for a few hundred people.

</comment><comment><USER3> I use adsotrans.com for Chinese-English translation support. Collaborative translation support tools that don't screw around with grammar are great.

</comment><comment><AUTHOR> just wanted to add that in the past two days numerous announcements have been made about developments in the saas marketplace - especially from Microsoft with their new OS as well as browser-based office apps.</comment>


Mt. Gox now lets you check your balance - <AUTHOR> https://www.mtgox.com/?check-balance <comment><USER2> WARNING: Looks like a fishing(fake?) page!

if works even if i enter bogus account details!</comment>


SQLite Is Serverless - <AUTHOR> https://www.sqlite.org/serverless.html <comment><USER2> I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are just files that are easy to copy, store, version, etc and don't require a centralized service).

I'm not saying ALWAYS use SQLite for these cases, but in the right scenario it can simplify things significantly.

Another similar use case would be AI/ML models that require a bunch of data to operate (e.g. large random forests). If you store that data in Postgres, Mongo or Redis, it becomes hard to ship your model alongside with updated data sets. If you store the data in memory (e.g. if you just serialize your model after training it), it can be too large to fit in memory. SQLite (or other embedded database, like BerkleyDB) can give the best of both worlds-- fast random access, low memory usage, and easy to ship.

<comment><USER3> I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline.

With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON.

I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

<comment><USER4> > I only wish there was a way to open an http-fetched SQLite database from > memory so I don't have to write it to disk first.

The sqlite3_deserialize() interface was created for this very purpose. [https://www.sqlite.org/c3ref/deserialize.html](https://www.sqlite.org/c3ref/deserialize.html)

</comment></comment></comment><comment><USER5> > Of those that are serverless, SQLite is the only one known to this author > that allows multiple applications to access the same database at the same > time.

IIRC, MS Access allowed that, which explained a lot of its popularity.

<comment><USER6> MS Access "allowed" it but depending on the version it could be quite problematic. We had a use case where Tableau connected to an Access file read only (but which another program used as a data store and wrote to it often) on an Windows file share, and once in a while the lock files would get screwy and we would have to manually delete the lock files to get things working again. Deleting lock files could be a huge chore because you have to figure out stuck processes and kill them. Task Manager wasn't up to the task, so had to use SysInternal tools to help with that.

Access is really meant for single-user scenarios I feel. Maybe the locking mechanism has gotten better but for multiuser access I tell people to use a real SQL database.

<comment><USER7> It was ok pre-windows 2000. I had access 97 it on 9 workstations on NT with never a problem! Moment office 2000 and windows 2000 came out it went to hell. Moved to SQL Server then. I am not sure what changed.

It was quite frankly the most productive custom business software package I have ever used. Literally would take 10 people a week to do something custom you could do in an afternoon with 1 person in Access. I suspect the same is true now.

<comment><USER8> We had a much heavier multiuser setup on Access 97, and it worked fine. We had the same breakdown as you did, and had to do a registry edit on all users' machines to keep it limping along until we could move to SQL Server, which was the right thing to do anyway.

<comment><USER7> Interesting. Nice to know it wasn't just me. There was no one to talk to about it back then so you had to think on your toes.

SQL Server was a crap load easier to back up reliably

</comment></comment></comment></comment></comment><comment><USER9> > It is important to understand these two different definitions for > "serverless". When a database claims to be "serverless", be sure to discern > whether they mean "classic serverless" or "neo-serverless".

It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

<comment><USER10> This such an air-headed comment to make. You must realize that page describing how SQLite is "serverless" has been up probably longer than your entire adult life. It is important in its context, they are not trying to "redefine" (lol) the term.

<comment><USER9> It doesn't matter. It is not the generally accepted definition of serverless. The meaning of a word is based on how it is generally used, and almost nobody means this when they use the word "serverless".

At this point trying to use the word in this way just creates a bunch of unnecessary confusion. Call it something else so we can move on to more important (and clear) discussions.

(Also there are a lot of assumptions and snarky comments in these responses about my age, which is quite rude and pretty elucidating.)

<comment><USER11> Their usage of serverless predates all of these cloud application offerings. Just because a lot of people in this particular echo chamber like to think of serverless as meaning one thing, there are plenty of others who have been using the same word to mean something different for quite a bit longer.

Moreover, the cloud as a service definition isn't even accurate. There still is a server — it's just not one the developer has to worry about.

</comment></comment></comment></comment><comment><USER12> In case SQLite is not enough and you need redundant servers or clustering, there's also database servers that use SQLite as storage engine: [http://rqlite.com/](http://rqlite.com/) [https://dqlite.io/](https://dqlite.io/)

<comment><USER13> So you run an app with sqlite on one server and sync to sqlite on another server? What would be the benefit to using a separate db server like in the 'neo-serveless' setup?

<comment><USER12> No, in this case you always have to use rqlite/dqlite because they manage the network synchronization. They use SQLite as storage engine (one SQLite database per server instance).

<comment><USER13> I understand that in those cases rqlite/dqlite is used. But that it just a technical detail. My point is that I am running two servers: one with the app and Xqlite and another one with Xqlite.

In case of a neo-serveless setup, I also have two servers: one with the app, the other one with the db server.

So what are the benefits of the Xqlite setup? I looked into that before and for one thing, Xqlite is slower (obviously) then just sqlite. So speed is not a key benefit. I also will have to manage both servers myself.

At least for a seperate db server I have the benefit that I can buy that as a service, incl management, backups and such.

<comment><USER14> Not the author, or knowing of all the technical details... simplistic replication structure and redundancy/failover without an expensive or more complex RDBMS solution while still self-hosting the service.

There are still a lot of instances where you cannot use a cloud provider for your app or database.

To be honest, I'd probably lean more towards a nosql database that has in the box, relatively easy replication strategy, though that might mean 3+ db servers for good performance. (RethinkDB, MongoDB, Cassandra/ScyllaDB, Cockroach). Just depends on the budget and resources.

<comment><USER15> Well, ScyllaDB is free and open source, so that should help the budget. (Though we do have an enterprise version, base price is FREE!)

<comment><USER14> I'd probably reach for Scylla wherever I would consider Cassandra (or similar), though the similar bigtable/columnstore-like services in most cloud hosts is generally still going to be easier.

The performance over C* is surprisingly good.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> It does allow multiple applications to access the same database at the same time, but when you do so it really hurts performance. I noticed this when i wrote a web crawler in go and used sqlite as the backend.

As soon as i connect using the command line interface, it slows down significantly.

Just something to bear in mind if you want to use it with multiple processes!

<comment><USER17> Even with WAL enabled?

<comment><USER18> Let's say there are concurrent writes to a single table with few indexes. Process of updating said table and indices eventually boils down to update of a single B+ tree from a multiple threads. In general until _serialized_ it is physically impossible without corrupting the tree. Sure there might be other specially crafted implementation of table more friendly to concurrent writes but there will be trade-offs. There are no miracles in the world.

As for particular case with WAL, all it does in this particular scenario is act as a queue. If your database load is spiky then it can even out load and make an impression of faster response. Under constant load it will not speed up the things and will internally serialize all actual updates

<comment><USER19> If you _need_ concurrent/parallel writes, SQLite is not the right tool for you. You may as well lament that hammers are no good for driving screws.

<comment><USER18> Can you please point me to where did I say anything about SQLite being right tool for concurrent write? My point was that true concurrent writes to a generic database table with indexes are are physically impossible in a general case disregarding what database it is.

</comment></comment></comment></comment></comment><comment><USER20> What I really want: SQLite storage backend driver for s3/gcs. No need for disks then. I haven't been able to find such a solution though; and am not technically proficient enough in C (the Lang SQLite is written in) to do so myself.

<comment><USER21> I am working on RediSQL[1] and I am about to launch a managed version.

The interface will be either HTTP or Redis protocol, you create your database, and daily I will back it up on S3.

(If interested you can subscribe for updated here: [https://simplesql.carrd.co/](https://simplesql.carrd.co/)

[1]: [https://redisql.com/](https://redisql.com/)

<comment><USER20> Redisql looks pretty sweet! To be frank, and this is only my personal opinion, I don't think I would want to pay for an api, but rather run my own. The business model of providing everything OSS but sending telemetry seems rather intriguing; as a hobbyist user I am ok with such telemetry being collected. If I were to run it in production for a business app though, I wouldn't even bother considering the unpaid version for the following reasons:

1\. I do not want my production instance to shut down for WHATEVER reason. This is just not an acceptable risk for most businesses. The only time a DB can go down is something goes wrong.

2\. As an engineer, I understand that 3 counters that are not accurate aren't a big deal. I can even look into the source and see that they really do as you say. Justifying this to a security org will be a complete nightmare, as most security orgs in enterprises are staffed with barely technical folks masquerading as "security".

So, it seems like a pretty good way to coerce enterprises to pay up while letting hobbyists continue using it. Very smart, I wish you the very best!

<comment><USER21> Thanks! That was exactly my reasoning.

Make it available and simple for hobbyist and small companies, ask money to who can afford it to sustain the development.

</comment></comment></comment></comment><comment><USER22> a bit tangential, but when do you move form using in-application data structures (maps, trees, vector/arrays) to using a database? Is it basically when the data doesn't fit in memory? I've been programming for almost a decade and I've never come across needing a database... (for context, it's ten years without anything web related) I'm interested in them and I'd love to learn SQL but I can't even think of a use case outside of managing user in some web based application

<comment><USER23> SQL queries tend to be much smaller than their equivalent data-structure traversal procedures. This can be beneficial even when you still want the data in your process's address space, hence embedded database engines like sqlite. Libraries like Linq can also provide the same expressive power over a programming language's native objects and collections.

As to why you'd want a separate DB server process: long lived mutable structures tend to drift into unexpected states, which is why the standard computer troubleshooting procedure since forever is to restart. Database management systems specialize in not having that problem. You deploy one that's widely used and hardened over many years, change it infrequently, and operate it with great care. Or pay a cloud provider to do that.

Then _everything else_ gets to outsource the burden of persistence to it, and the vast majority of the workloads you develop and operate are stateless. These are drastically more convenient and more resilient to mistakes. They're effectively "restarted" with each API request, and you can stand them up, tear them down, migrate them to new machines, etc. as much as you want without fear of data loss.

<comment><USER18> " _long lived mutable structures tend to drift into unexpected states_ "

Why are those "tend to drift" ? As an example I wrote sort of like game server for one of my applications. Internally it has those exact forever lived mutable structures. I've never observed it to drift into any unpredictable state. Works like a charm and running for many month. I only reboot it when I need to update it to a new version.

The only catch here is that all data fit into RAM. With the amount of RAM modern computers can be stuffed with I do not really see if my server would ever run out of it.

Of course it backed up by database but it merely serves as a persistence layer for this particular application.

<comment><USER24> Consider why 3rd normal form exists to begin with.

In the real world, let's say you have a shipping address and a billing address for a customer, and they are usually (but not always) the same.

Eventually, a customer moves, changing both of their addresses. But the user forgets to change their billing address with their delivery address.

A proper database would have a 'billing address is same as delivery address' logic, following the principle of DRY (don't repeat yourself).

\</comment></comment></comment></comment><comment>----

There are lots of examples here of what can go wrong when you repeat yourself in a database application. The user may have an error when repeating themselves over the dataset (delivery address is correct, but zip code on billing address has a typo).

Dealing with these issues at scale, with hundreds of thousands of customers, is certainly a problem. Normal forms can formalize these issues and help the business owner avoid the problems.

Where do you verify the existence of zip codes and cities? Where do you check for typos? How do you prevent contradictions on the submitted information?

Your human customers will make many mistakes. Your logic must hold up even in the presence of faulty data.

<comment><USER18> _" A proper database would have a 'billing address is same as delivery address logic"_

The database does not have logic like this. It has to be implemented by stored procedure. When I have application server all such logic (if applicable) is handled by code in much more performant way. No data goes to a database directly. Everything passes through the app server along with the validation data transformation etc, etc. As already said the database in this particular case is nothing more but persistence layer.

Again we can put all kind of theoretical speculations but as I already said, my particular server does not have data drifting to some faulty states.

<comment><USER24> >> "A proper database would have a 'billing address is same as delivery address logic"

> The database does not have logic like this.

Of course it wouldn't have 'logic', databases are just stores of data.

You'd have one 'address' table, with probably a int-primary/surrogate key. Then the 'delivery' and 'billing' address would be an int, pointing to the address table.

Furthermore, the billing and delivery address would be foreign keys, so the internal database logic would keep the tables in sync with no application code required.

With the data organized in this manner, the application code becomes logic free and braindead easy to write. Or at least, corner cases become easier to handle and more explicit. (Say two customers share the same address, do you allow repeats in the address table? Or do you allow customers to tie address information together? Either way, your decision rests on how you define the primary key)

> When I have application server all such logic (if applicable) is handled by > code in much more performant way.

The most performant way is no logic at all. A proper database removes a lot of checks, the storage format itself naturally creates logic free code.

Some application logic is necessary of course. But you can minimize the logic needed by thinking about data layout.

</comment></comment></comment><comment><USER25> SQLite is amazingly prevalent as well. Your own phone probably has hundreds of SQLite databases on it. One challenge with SQLite, though, is you have to download the whole thing to make use of it.

Shameless plug, but I made a fun side project that allows Amazon Athena to read SQLite databases from S3. [https://github.com/dacort/athena- sqlite](https://github.com/dacort/athena-sqlite)

</comment><comment><USER26> I love SQLite, but people approach it from a classic RDBMS angle which confuses them.

Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file.

That's it. Stop overthinking it.

Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/write methods. SQLite is not really any different, except the read/write work is already done, and you use SQL to format the data values and encode the read/write logic instead of the language you are working in.

The file format has some cool extensions like text indexing, geospatial, etc. But it's no more a RDBMS than reading and writing a JSON file is.

"But there's indexes!" Yes, just liked you might build an index on your JSON file and read that before reading the JSON file to know where things are faster -- and then you have to write all the code to do that. SQLite is just a file format, where you can also build indexes and all the code for that is already done for you.

It's just a file format with an API that's similar to the ones you'd use for any regular old RDBMS and uses SQL as the domain language to read/write data.

It's just a file format. Anything you can do/can't do with a file format you do with SQLite.

It's just a file format. A nice convenient one that you are probably better off using than most other things for most purposes.

It's just a file format.

 _edit_

I'm glad this comment is getting such a response. I'm not trying to be mean, just help clarify thinking.

Here's two thought experiments:

1) If SQLite didn't require you to use SQL as the read/write logic and was called "datalite", and instead just forced you to use C function calls, exactly like you would if you were working with literally any other file format on the planet, would you still be confused as to what it is?

2) Do you consider reading and writing to any other file format anywhere in the hierarchy of RDBMSs? Consider Python's csv module. Is that an RDBMS? Let's move away from tabular data, how about python-docx?

<comment><USER27> Consider: it's totally possible to strip down Postgres until all you have left is an embedded RDBMS of the style of SQLite. (I'm not sure why nobody has done this yet, actually.) Would you call the result "just a file format"?

Such an instance of "embedded Postgres" would still have a huge sprawling catalog (PG_DATA) directory attached to each use of it, so it wouldn't be contained to a single file. But neither is SQLite contained to a single file—SQLite maintains a journal and/or WAL in a second file.

And, yes, this "embedded Postgres" would require things like vacuuming. But... so does SQLite. Have you never maintained an application that maintains a long-lived "project" as a single SQLite file, where changes are written into this project file repeatedly over a long period? (Think: the "library" databases of music/photo library management software.) SQLite database files experience performance degradation from dead tuples too, and need all the same maintenance. Often "database version migrations" of such software is written to either rewrite the SQLite file into a clean state, or—if it has the possibility of being too big for that to be a quick task—to call regular VACUUM-like SQL commands to clean the database state up.

——

Now, I get what you're trying to say; the point that you're trying to make—that SQLite might be a relational _database_ , but it's not a relational database _management system_ in the sense of sitting around online+idle where it can do maintenance tasks like auto-vacuuming. Unlike an RDBMS, SQLite doesn't have its own "thread of execution": it is a library whose functioning only "happens" when something calls into it. By analogy, regular RDBMSes are like regular OS kernels, while SQLite is like a library kernel or exokernel.

But that doesn't mean that SQLite _is_ a file format! It can be _used_ as one, certainly, but what SQLite _is_ is exactly the analogy above: the kernel of an RDBMS, externalized to a library. As long as you "run" said kernel from your application, and your application _is_ a daemon with its own thread of execution, then your application _is_ an RDBMS.

This can be seen most directly in systems like ActorDB, that simply act as a "transaction server" routing requests to SQLite. ActorDB is, pretty obviously, an RDBMS; but it achieves that not due to its own features, but 99% due to embedding SQLite. All it does is call into SQLite, which already _has_ the "management system" part of an RDBMS built in, just not called unless you use it—just like exokernels often already have things like a scheduler, just not called into unless you as the application layer do so.

<comment><USER26> SQLite is a file format with a familiar API and uses SQL as the logic for searching/adding data to the file.

Approach it exactly the same way you'd approach using a CSV file and all the confusion and overthinking about it goes away. Approach it as a stripped down RDBMS and you end up with all kinds of questions about support for this or that RDBMS familiar service.

You can write your own SQLite file reader/writer. Here's the specs (includes the specs for the Journal and WAL files and semantics as well) [https://www.sqlite.org/fileformat.html](https://www.sqlite.org/fileformat.html)

Here's an example of somebody who's done this. [https://sqljet.com/](https://sqljet.com/) \- this is not a wrapper on the sqlite C code, this is a re-implementation of that code that is binary compatible with SQLite files.

The Journal file only exists as a temporary file until transactions complete. The .sqlite file you make is the entire atomic file that follows the SQLite file format. The Journal has its own file format. Same goes for the Write- Ahead log.

RDBMSs also manage connection queues, account management, rights and permissions, and so on. Many overcome various OS limitations by providing their own entire file management, fopen(), virtual memory and other subsystems that are tuned to their workloads.

SQLite is a file format. SQLite uses familiar relational paradigms to make it easy to read/write data to the format without having to learn yet another API and domain language. The API code is extraordinarily well tested, and it makes simple complex logic like transaction journaling, indexing and so on.

>Consider: it's totally possible to strip down Postgres until all you have left is an embedded RDBMS of the style of SQLite.

No! SQLite is not an embedded RDBMS. It's a file format.

If there was a library you could import, and it provided methods to read and write directly to files that PostgreSQL could read/write to and there was nothing else to install, no runtime, no daemons, no servers, etc., then we could pass around self-contained PostgreSQL files to each other. Then PostgreSQL files would be a file format as well.

Have you ever used a library to read/write from a CSV, JSON, JPEG? It's no different than doing so for a SQLite file!

SQLite is a file format.

<comment><USER28> File formats don't have an API.

> SQLite is a file format with a familiar API and uses SQL as the logic for > searching/adding data to the file.

That description is for a library, not a file format. SQLite is a library, that saves to a convenient format and allows you to query the file using the SQL syntax. File formats dont have "logic".

You are arguing the equivalent that Word is a file format. While there is a Word file format, Word itself is an application.

<comment><USER26> > File formats don't have an API.

So when you read/write to any other file format, you just read/write bytes directly to/from disk and re-implement the parsing and read/write logic in your own code every time?

> File formats don't have "logic".

Every file format has logic otherwise it's just random entropy in an arbitrarily long byte stream on a disk. How to read/parse and interact with that format depends entirely on the logic and scheme for that file. For example, many file formats have an index of some kind that you must read and parse before you can figure out where the other data lives, compressed file formats often store a dictionary, image formats often have a compression/decompression logic that must be followed for reading/writing to them.

> You are arguing the equivalent that Word is a file format. While there is a > Word file format, Word itself is an application.

.docx is the file format for Word documents. There are many APIs and programs that can read/write to/from .docx files.

</comment></comment></comment></comment></comment><comment><USER29> Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify?

I think not, but I wonder if some hack is available by virtue of it simply being a file that you can read (and somehow) write to.

The best I came up with: let's say you have a toy project, and you call the Github API and replace the file upon every write. Implementing a read is easier as you know where the file is located. This hack shows that you somehow need write access to get any form of performance out of it, because this hack is super slow.

<comment><USER30> > Does this mean that you can hack some database storage (w/ sqlite) together > on frontend only hosting platforms like Github or Netlify?

You can run SQLlite in the client via webassembly therefore open a SQLite file in the browser to query it yes, you just can't write anything in it and expect it to persist somehow on the static hosting service itself.

> The best I came up with: let's say you have a toy project, and you call the > Github API and replace the file upon every write. Implementing a read is > easier as you know where the file is located. This hack shows that you > somehow need write access to get any form of performance out of it, because > this hack is super slow.

No you'll need a server for that, you can't make random HTTP requests to any server in the browser, because of CORS/SAME ORIGIN policies.

"SQLite is serverless" is meaningless buzzword. It just means that SQLite is equivalent a flat file where you'd shove some data, just that you can use SQL to query that file instead of having to index data in it yourself.

<comment><USER31> "SQLite is serverless" is by no means a meaningless buzzword. The term had a clear meaning before it was coopted by the current web dev fad. SQLite does not operate on a client-server architecture the way e.g. MySQL or PostgreSQL do.

<comment><USER32> Just to back that up, it had a clear meaning because -less is a valid suffix to append to English words. When grandma runs out of cookies she is cookieless (when a website doesn't use cookies it is also cookieless). It doesn't have to be "in the dictionary" to make sense in conversation.

A quick search of usenet shows "serverless" being used in 1994. It wasn't a term or a buzzword, it wasn't common, it was just English: [https://groups.google.com/d/msg/comp.os.linux.misc/r76oNl98C...](https://groups.google.com/d/msg/comp.os.linux.misc/r76oNl98C1k/CkuR8EEuYV0J)

<comment><USER31> Yes, this is what I was trying to say. It wasn't a thing people would throw around like a buzzword but if someone used it in conversation and especially in context (like the SQLite page, which was written over a decade ago, does) people would understand what you meant.

</comment></comment></comment></comment></comment><comment><USER33> This document is golden, serverless means an application reading and writing to the database in the process directly

</comment><comment><USER34> I find this article valuable. I made a mental note to check if SQLite is enough every time I consider one of the "serverless" options.

</comment><comment><USER35> Choose your own definitions... but SQLite is not serverless by common parlance.

Author creates two definitions for serverless which don't match the common usage. Serverless is more about DevOps / deploy experience than how the program leverages OS processes internally.

Apparently MS and AWS are ISPs?

Maybe SQLite could be serverless if you defined it as incapable or running as a server on its own?

<comment><USER36> Serverless used to have a very clear definition: not having or using a server, so SQLite is a perfect example of serverlessness. I was extremely confused when the newfangled definition (called neo-serverless on TFA) showed up, or your definition for that matter. Who ever thought up theses confusing meanings for a word that used to be perfectly clear?

You don't need an ISP to have a server. Any computer or program that listens to a network port is a server.

<comment><USER37> Or is it any program that _responds_ to comms on a network port. The "serve" part of "server" ;)

</comment></comment></comment><comment><USER38> I don't think we fully appreciate just how pervasive SQLite is in our computing devices. Apple recognized its power and utility and embedded it in macOS and iOS, and Google apparently followed suit in Android. The universality of SQLite is due in no small measure to its stellar quality and the trustworthiness of its author, D. Richard Hipp.

</comment><comment><USER39> Serverless, like all embedded databases ;-)

In the sense you don't need to provision an additional piece of infrastructure to power your application :-)

</comment><comment><USER40> Yep SQLite is in fact serverless. We based our open source web based IDE on SQLite and it works fantastically. The best part is that you can even take a SQLite database and use sql.js and run it offline in the browser!

</comment><comment><USER41> _" Microsoft Azure Cosmo DB and Amazon S3 are examples of a neo-serverless databases."_

Can we back away a bit from the bandwagoning of misused terminology? Serverless literally means "running your apps on somebody else's server". S3 is not a server you run your apps on, it is SaaS that you manipulate through an API - you don't put your apps on it. If S3 is serverless, then literally every network service of any kind is serverless.

<comment><USER14> I always preferred XaaS nomenclature myself. "XaaS with autoscaling"... is probably better than "Serverless"

</comment></comment><comment><USER13> The main problem that I see for using sqlite is exactly for it being 'Classic Serverless'. Because how does one keep an up te date backup? Deploying to Heroku, Dokku, AWS Lambda and such means the sqlite file will be lost on a crash or new deploy. Even a VM can crash. Export to S3 on every write? Maybe if changes do not happen often, so only for specific use cases (and actually I think you should just generate static html in that case).

I use sqlite for local tests for cases where the live app has a 'neo- serverless' database. It is very, very fast so the tests run almost instantly.

<comment><USER42> > _Because how does one keep an up te date backup?_

[https://www.sqlite.org/backup.html](https://www.sqlite.org/backup.html)

There's a ".backup" command:

* [https://sqlite.org/cli.html#special_commands_to_sqlite3_dot_...](https://sqlite.org/cli.html#special_commands_to_sqlite3_dot_commands_)

Alternatively, given that it's ACID, you could just take a snapshot of the file system/volume in question, and do a recovery on restore.

Edit: SQLite also has WAL files, so presumably one could just use tar/rsync to create the backup, and only the last file would be 'corrupted', so you'd lose the last (few) transaction(s):

* [https://www.sqlite.org/wal.html](https://www.sqlite.org/wal.html)

<comment><USER43> Ding! correct answer. In addition, the backups can run in the background, simultaneous to active use in other threads. When the backup is initiated, it is a transaction, so it maintains knowledge of what portion of the database is new since the backup began and does not include that data. Which is what you want for backup integrity.

</comment></comment></comment><comment><USER44> I like the definition of classical- and neo- serverless. Does anyone knows more articles about serverless in terms of classical serverless?

</comment><comment><USER45> > Neo-Serverless: The database engine runs in a separate namespace from the > application, probably on a separate machine, but the database is provided as > a turn-key service by the hosting provider, requires no management or > administration by the application owners, and is so easy to use that the > developers can think of the database as being serverless even if it really > does use a server under the covers.

I'm having a failure of imagination here.

What would I use a database for that I could reasonably assert to my peers and superiors that _no_ maintenance whatsoever is required? Backups and restores count as maintenance. Multi region is now common, if not pervasive.

Are there public datasets that are so common that it would be worth it to provide it as a service? What other service would behave like S3 but look like SQLite?

I get that one might be safe to assume that "serverless" isn't just pure functions. It could reach out to other services that are not serverless and still not consume (further) resources on a set of machines while not in use.

But a severless database... I'd have to have something aggressively read- mostly, written to S3 at intervals and read from serverless processes. But is that a new thing or reading data from S3?

</comment><comment><USER46> I integrated my database with the app server, but I still open a socket internally to access it.

So my system is also "serverless" in that meaning.

Acronyms used to be confusing, this is just ridiculous.

[http://root.rupy.se](http://root.rupy.se)

</comment><comment><USER47> I use SQLite as a backend for my website and absolutely love it. Not having to setup a full dbms just for a small blog is a dream, and backups are as easy as copying a file.

</comment><comment><USER48> CSV is also serverless..

<comment><USER49> CSV doesn't have a write-ahead log.

<comment><USER50> I don't think sqlite had WAL when that page was created.

<comment><USER12> I think bob means transactions.

<comment><USER49> Transactions, durability and acceptable performance with concurrent access.

</comment></comment></comment></comment></comment><comment><USER51> I currently use it to build a read-only rest service for GeoNames gazetteer. Each instance contains a copy of the database which means its really linearly scalable.

</comment><comment><USER52> My understanding of serverless = easily scalable, managed service.

But somehow the word annoys people. Maybe we should find a better word?

<comment><USER53> > My understanding of serverless = easily scalable, managed service.

There is already a term for that concept: managed services. There is no such thing as a managed service that's designed not to be scalable. Some implementations may be better at scaling than others, but that's it.

The serverless buzzword is pure marketing.

<comment><USER54> On the one hand, the term serverless is really frustrating to me.

On the other hand, there's a certain amount of amusement I get from seeing "the cloud" become a marketing buzzword in the mid-late '00's, and now seeing the same thing happen with "serverless."

When you look at the implementation of the two "technologies", they're about 95% the same. Yet somehow they're pitched as these big revolutions.

In another decade when terminals or p2p become popular again we'll be hearing about some new buzzword like "Terran" computing, or "social" architecture or something.

</comment></comment></comment><comment><USER55> "There's no server process" vs "there's no server on your itemised bill".

</comment><comment><USER56> I don't get it, then any db can be classic serverless if they run in the same computer as the app?

<comment><USER57> _> any db can be classic serverless if they run in the _same computer_ as the app?_

Not just the _" same computer"_ ... it's the _same process id (PID)_.

Extract of relevant text from that webpage:

 _> Classic Serverless: The database engine runs within the same process, thread, and address space as the application. There is no message passing or network activity._

In other words, when you compile and link "sqlite.c" into your own executable, the same PID (process) that handles text input and paints pixels on the screen -- is the same PID that writes to the sqlite database file. It's all the same _process_. That's what they mean by "classic serverless".

In contrast, if you make a Go executable that writes to MySQL/PostgreSQL db and make them both run on the same physical computer, that's not "classic serverless". It's because when you enter _" ps -aux"_ to list all running processes, you see _separate PIDs_ for the Go executable and the MySQL db engine.

Other jargon used might be _" in-process"_ vs _" out-of-process"_ or _" embedded"_ vs _" external"_. SQLite is sometimes characterized as "in-process embedded database" but MySQL is an "out-of-process" db.

</comment></comment><comment><USER58> Isn't it easier to say "embedded" and "severless"

<comment><USER59> Not in this thread apparently. I guarantee nobody here as ever used the term "serverless" over "embedded" or "in-process" in their entire career but apparently the purity and nostalgia of SQLite overrides it.

<comment><USER31> This article was written in 2007, so yes, people have been using the word "serverless" before managed service providers decided to take it up as a buzzword.

<comment><USER59> Nobody used it before. Even the about page says: "SQLite is an embedded SQL database engine."

"serverless" is a marketing term, then and now. Not sure why there's such a big defense of it. If you have to argue this much over the provenance of a term that clearly isn't used the same way today then it's a good sign that it's not very useful.

<comment><USER19> > _Nobody used it before._

The archive.org links already provided to you prove that you're flat wrong about that. SQLite used the term many years before it was a webshit buzzword. They used it in the intuitive straight forward english sense; somebody who has no hat is hat _less_ , somebody who has no home is home _less_ , and a database system that has no server process is server _less_. In 2007 when it was written, nobody would have batted at eye at this term, the meaning would have been immediately clear to anybody who had any familiarity with databases.

<comment><USER59> Just because a single page on the internet used it does not mean the term was used in the industry. It also doesn't matter if it technically makes sense, although it's a stretch (because there is a server process, you just share it).

It's a marketing term, and a poor one at that. That's why SQLite even describes itself as a embedded, in-process database without client/server architecture. Because that's the common jargon.

I'm surprised at the endless defense of a marketing buzzword and the argument over which marketing definition is the "real" one. This is bikeshedding at its finest.

<comment><USER19> Nobody was saying it was a buzzword in the industry back then (actually...[1]). It was being used as a normal English word, not some misleading marketing drivel like the new use of the term. Somebody who has no shirt is shirt _less_. An MRE heater that doesn't catch fire is flame _less_. Somebody who isn't witty is wit _less_. Somebody who doesn't have a clue is clue _less_.

Are you starting to notice the pattern here? "less" is a suffix that can be applied to nearly any noun to describe the trait of lacking that noun. When this is done, the meaning is clear to native English speakers who've never heard that combination before. When this article was written in 2007, the meaning was clear. SQLite didn't muddy anything, didn't redefine anything.

[1] The term 'serverless' was in fact in use in the decade and a half prior to 2007: [https://books.google.com/ngrams/graph?content=serverless&yea...](https://books.google.com/ngrams/graph?content=serverless&year_start=1950&year_end=2008&corpus=15) I don't know if the term was being used _primarily_ in a technical context, but I have little doubt the term was being used to describe something that did not have a server. In fact I'm just about certain that spike was the term being used in a tech context. I've found you an example of the term being used in 1995 (DOI 10.1145/224057.224066):

> _A serverless network file system distributes storage, cache, and control > over cooperating workstations. This approach contrasts with traditional file > systems such as Netware, NFS, [...] where a central server machine provides > all file system services._

This is not precisely the same way that SQLite has used the term. Rather, they're using the term in an intuitive natural way that fits their context, just as SQLite used it in 2007.

Here is an example from 2007 (DOI: 10.1109/TNET.2006.886289):

> _Abstract—We explore exploits possible for cheating in real-time, > multiplayer games for both client-server and serverless architectures._

Here we have the term "serverless" clearly being contrasted with "client- server", which seems _very_ similar to the way in which that SQLite document used it.

</comment></comment></comment></comment></comment></comment></comment><comment><USER60> This is not new or unknown information, why is this on the front page?

<comment><USER61> Because other people deemed it interesting.

<comment><USER60> The fact that SQLite is serverless is common knowledge and not interesting unless you're brand new to databases

</comment></comment></comment><comment><USER62> I think better name (and coined since 1990ies) is __embedded db engine __. MySQL, Firebird (InterBase) etc can also be linked into binary and run in the same process, without any type of sockets.

</comment><comment><USER63> Two definitions of serverless?

I heard many, but what they're writing isn't one of them.

As far as I can tell SQLite was always called an embedded database, and never a serverless one.

<comment><USER50> The page is more that 10 years old. And it uses "serverless" in the sence of server-less aka not having a server. Which is pretty damn sensible terminology, as opposed to the newer "serverless" meaning" runs on some utility server outside of your care or control".

<comment><USER63> I know, but have you anyone calling "embedded databases" "serverless databases" in the last 10 years?

It's like someone found this page and felt very smart about it, because they stick it to the serverless crowd.

<comment><USER4> "embedded database" does not imply "serverless database". Other embedded RDBMSes run servers; they just run the server as a separate thread rather than a separate process. SQLite is different in that there is no separate thread of control. SQLite runs in the same thread as the application that calls it. There is no separate thread hanging around to clean up or handle background tasks after an SQLite function call returns.

</comment></comment></comment></comment><comment><USER64> > SQLite is an example of a classic serverless database engine. With SQLite, > there are no other processes, threads, machines, or other mechanisms (apart > from host computer OS and filesystem) to help provide database services or > implementation. There really is no server.

Well, it's bending the overall consensus defining serverless as a managed / and or stateless service.

I never saw anyone using the term "serverless" to mean "embeded".

Using this definition, anything and everything that is not requiring a specific server to be served / distributed can be described as serverless.

<comment><USER65> There's a section added in 2018 to deal with the apparent confusion.

It'd be a better idea to just delete the page. It may have been written long before the meaning changed, but it's pointless to fight a losing and completely insignificant battle over language.

<comment><USER66> Hard disagree. The "neo-serverless" version has always been extremely confusing to me. I expect serverless to mean the absence of a server.

In SQLite's particular case, it's subverting the expectation that has persisted since the beginning of time (of databases) that a database must be managed by a server.

</comment></comment></comment><comment><USER59> Why even write this? It's yet another convoluted use of the term and meaningless for SQLite of all things.

Re: downvotes - what are people disagreeing with? That the term is not convoluted? That it's actually useful? That it helps to have more sub- definitions in an industry known for overloaded terms? I guarantee not a single person here has used "classic serverless" over "in-process" or "embedded" in their entire career.

<comment><USER67> It describes an important architectural feature of SQLite.

Also, this page was first written in 2007 (or perhaps earlier) [1], long before 'serverless' was applied to things like Amazon Lambda.

[1]: [https://web.archive.org/web/20071115173112/https://www.sqlit...](https://web.archive.org/web/20071115173112/https://www.sqlite.org/serverless.html)

<comment><USER59> Adding an original timestamp to the page would've been more helpful than the updated section, or just deleting the page entirely. Or just title it as "SQLite is not client/server".

Nobody goes around talking about "classic serverless" instead of "in-process" which has been around for decades.</comment></comment></comment>


The Sorcerer of Jazz - <AUTHOR> http://www.nybooks.com/articles/2016/09/29/miles-davis-sorcerer-of-jazz/ <comment><USER2> Fantastic, legendary musician, but a monster as a person; a man who turned some of his female fans into prostitutes and beat them when they didn't make enough money to sustain his heroin habit.

This is common with a lot of figures from the past, of course, from Picasso the wifebeater to Wagner the antisemite, Washington the slave owner, to Gary Glitter the child abuser.

<comment><USER3> To be fair to Washington, he was also the only slave-holding founding father to free his slave in his will and spoke against slavery. And this in an era when owning slave was considered normal.

It's different from Picasso for example who abused women in an era when it was not considered ok to do so by society.

</comment></comment><comment><USER4> 'course, he beat women too...

[http://connectnyc.globat.com/documents/Mad%20At%20Miles%20by...](http://connectnyc.globat.com/documents/Mad%20At%20Miles%20by%20Pearl%20Cleage%20version%202.pdf)

<comment><USER5> It doesn't prevent me from enjoying his music though. I'm a proponent of decoupling the person and the ouvre. I can delight in his music and at the same time think he is a violent shitbag. It's not mutually exclusive.

<comment><USER6> If you appropriate the music you shouldn't call it his. And I agree to the extent that it was never his to begin with, standing on the shoulder of giants etc.

> It's not mutually exclusive.

Some would think it is. Death of the Author is a modern perspective. And again I agree, because concrete music is not as ephemeral as history, if that's the right word. I mean, history is unreliable, vague, abstracted, imaginary, indirect, something like that.

</comment></comment></comment><comment><USER7> There are some purists left who are equated with Miles Davis. Christian Scott comes to mind as one of the modern greats for sure:

[https://www.youtube.com/watch?v=I73T9yLdoH0](https://www.youtube.com/watch?v=I73T9yLdoH0)

[https://www.youtube.com/watch?v=m6u8UdpVsKM](https://www.youtube.com/watch?v=m6u8UdpVsKM)

</comment><comment><USER8> Anyone know of a reference for "The keyboardist Joe Zawinul was aghast at Davis's ruthless arrangement of "In a Silent Way,"... " part?

It is difficult to prove a statement like "..he pioneered hard bop.." wrong, but it is definitely a new view on the history of jazz in the post bop era.

<comment><USER9> _Anyone know of a reference for "The keyboardist Joe Zawinul was aghast at Davis's ruthless arrangement of "In a Silent Way,"... " part?_

NYRB might be sharpening the rhetoric a bit, but there's a good interview with the personnel where they go over this:

[http://www.miles-beyond.com/iaswbitchesbrew.htm](http://www.miles- beyond.com/iaswbitchesbrew.htm) ('bout halfway down the page)

</comment></comment><comment><USER10> I was drawn into _electric_ Miles, much like Brian Eno, in the 80's. But ultimately became a bigger fan of his _acoustic_ 1959-1963 sound.

Amazing career. Can't wait to see the movie.

</comment><comment><USER11> I was just listening to Milestone when I found this as it happens.</comment>


Give Snowden his due: He made a surveillance debate possible - <AUTHOR> http://www.latimes.com/news/opinion/opinion-la/la-ol-nsa-leaks-snowden-20130731,0,4611758.story <comment><USER2> We should more than give him his due. We should throw him a frickin ticker- tape parade.

<comment><USER3> I'll settle for a pardon.

<comment><AUTHOR> Under the current administration, I'm hoping we even get any sort of meaningful response to the pardon petition ([https://petitions.whitehouse.gov/petition/pardon-edward- snow...](https://petitions.whitehouse.gov/petition/pardon-edward- snowden/Dp03vGYD)).

I'm holding out a tiny glimmer of hope that Obama comes to his senses enough to pardon for the domestic pieces. It doesn't seem likely, but doesn't seem to directly contradict what he's said, and maybe Obama can pre-emptively compromise with _me_ for once...

<comment><USER4> The deadline's approaching. My money is on "We are unable to comment on an ongoing investigation."

<comment><AUTHOR> That does seem most likely, and I would not count it as meaningful. They've also said before that they don't comment on individual pardons, though I can't think of a good reason for that.

In this case, though, the pardon itself would be a substantial piece of policy. I think it would be disingenuous and disrespectful of the administration to refuse comment.

I expect to be disrespected, but hold out some small amount of hope that they will surprise me.

</comment></comment></comment></comment></comment><comment><USER5> Today was disheartening.

The Guardian writes an expose on what should be the greatest political scandal of our lifetime- the fact that low level contractors and government employees can access all of our chat history, emails and browsing history.

The response? No one on social media is really discussing it, CNN America only has a passing mention of it on their front page (buried near the bottom) and America goes on with its day.

What does it take to get American's outraged about something?

<comment><USER6> > What does it take to get American's outraged about something?

sex

<comment><USER7> When this story brokeout, i had the idea that the best way to publicize this is to find an example of a private sexual conversation on record, which they obviously have. Would have been interesting for Snowden to get one of those just in case.

That, or an inside trading potential example.

Money or sex is something everyone understands.

</comment></comment></comment><comment><USER8> NSA response to today's revelations: [http://www.nsa.gov/public_info/press_room/2013/30_July_2013....](http://www.nsa.gov/public_info/press_room/2013/30_July_2013.shtml)

<comment><USER9> Neat.

> Defending Our Nation. Securing Our Future.

...I read that as...

Defending Our Own Money. Securing Our Own Power.

<comment><USER10> NSA surveillance foiled this potential terrorist and he is now incarcerated:

[http://en.wikipedia.org/wiki/Najibullah_Zazi](http://en.wikipedia.org/wiki/Najibullah_Zazi)

He was going to bomb NYC subways.

Please, if you're going to participate in this so called "surveillance debate", stick to facts and refrain from hyperbole.

<comment><USER11> If you're going to participate in the "surveillance debate" don't be so stupidly myopic to argue only the ridiculous point that this keeps anyone safe from "terrorists".

If the NSA can literally look into any electronic conversation between anyone, then why haven't they stopped a single drug cartel? Any premeditated murder? Why hasn't a single banister been brought to justice over their internal emails detailing deliberate mortgage fraud? Why weren't they able to prove Bernie Madoff was swindling billions for years and there were many warnings? What about all the illegal insider trading of pretty much every congress person?

Oh that's right, because this system has absolutely nothing to do with bringing criminals to "justice" or keeping you safe.

This is digital enslavement and every single person who approves of this bullshit is complicit.

Trying to say that trillions of dollars in stolen money from people (if the funds taken from taxes are used to build a digital prison around you, those funds were stolen from you) to prevent some singular supposed attack is justified is farking insane!

<comment><USER12> The reason the NSA hasn't gone after anyone else is that it would probably be illegal to -- not that the program in general is legal, but it would be more blatantly illegal than simply focusing on groups affiliated with al-Qaeda.

The AUMF and constitutional case law give the executive branch a fair amount of discretion when it comes to national security. It's pretty hard to shoehorn drug cartels, bankers, and your run-of-the-mill murderers under national security. As such, prosecuting those folks using NSA-gathered evidence would force the NSA to reveal more about the program than it's comfortable doing.

<comment><USER11> I'm pretty sure you e missed my point. I also think you fail to understand what is really going on here.

</comment></comment></comment></comment></comment></comment><comment><USER13> The obsessive discussion about phone records is such an obvious straw man. Phone records are just the tip of the iceberg. It's all the other communications surveillance that we should be talking about. They'd be happy to give up phone record dragnets because they know that's not a useful repository of information anyway.

</comment><comment><USER14> Well well, a couple of people are starting to wake up. I was nearly despairing that the impact of what Snowden was saying would be lost in the snowstorm the government was trying to create to obscure his message by attacking the man.

It seems the message is getting through, and that is a great thing.

</comment><comment><USER15> Surveillance "debate"? I must have missed something.

<comment><USER16> In modern parlance the word debate is used place of "dialog" or "conversation", even if there is little actual debating going on.

</comment></comment><comment><USER17> We are winning but nothing is done and nothing will be done. The whistle blower won't be pardoned, and in a few month nobody will care about it. I so deeply wish that this is going to hurt at some point certain businesses so that some big player might use the little influence they may have to change things.

</comment><comment><USER18> Someone should suggest to the American public that this mass surveillance is a big threat to sports and will somehow kill Baseball and Football (the one played with hand) in the country.

</comment><comment><USER18> "I need to reduce your freedom to keep you safe", no one can argue with that.

</comment><comment><USER9> Also: What does everybody expect from "the debate"?

 _Is_ there a way we can get out of this mess? I currently don't see any, besides the one with massive loss.

Please, anybody, correct me.

<comment><AUTHOR> Call me naive, but I would be surprised to learn that these systems are being used to overtly control the electoral process. So the way out of this mess is to talk to your friends and family and make sure they understand the issues and what is at stake, and make sure they do the same. Elect politicians who will do better - Obama promised and didn't deliver, but the issue only seemed to matter to a sliver of what were basically his base anyway. In the wake of an election with this issue more front-and-center, because the people have demanded the issue remain more front-and-center, I'm not convinced you'll see the same behavior.

<comment><USER19> The electoral process has already been subverted through gerrymandering and a two-party system. All the good, honest, hard-working folks get stuck together with the parasites and psychopaths. In either party there is a very wide range of people from salt-of-the-earth types to scum-of-the-earth.

If you don't utilize gerrymandering then you lose, and if you're campaigning in a third-party then you lose.

<comment><AUTHOR> But that's not anything _new_ , and there's not any reason to expect that subversion to prevent fixing _these_ issues, and fundamentally _if enough people are convinced, things will change_ \- "enough" is higher than it probably should be, but I'm not convinced it's higher than it would need to be for a successful revolution or the like.</comment></comment></comment></comment>


The Mill CPU Architecture – The Compiler [video] - <AUTHOR> https://www.youtube.com/watch?v=D7GDTZ45TRw <comment><USER2> (Mill team)

Any questions, ask away, happy to explain or dodge as appropriate ;)

<comment><USER3> Where should I go to understand your work if I hate watching videos but like reading? Is the wiki the best resource?

<comment><USER4> Try watching the videos in 1.5x or 2x speed on youtube. This gives you the same information density as reading and you don't miss the nice animations travisb mentioned.

</comment></comment></comment><comment><USER5> The answer at 1:26:12 [1] about the belt being in fact a 'proxy' of the bypass network is very interesting.

[1] [https://youtu.be/D7GDTZ45TRw?t=5172](https://youtu.be/D7GDTZ45TRw?t=5172)

<comment><USER6> Yes, amazing. I've always wondered how the belt was implemented and it turns out its not a belt at all.

<comment><USER7> I think Godard actually explicitly said that in a Q&A of one of the very first lectures - although I have to admit at that point the finer details of the discussion were way over my head.

</comment></comment></comment><comment><USER8> He says that they are going to implement all instructions that are not supported by the hardware[1].

I wonder if they will use automatic synthesis of code for each instruction, and if so, how they are going to ensure good performance. Maybe superoptimization?

Another question I have is about their stance on free firmware. Will the mill family members require blobs?

[1] 13:50 in the linked video.

</comment><comment><USER9> Have they moved beyond vapor ware phase yet? Working silicon?

<comment><USER10> Nope, they don't even have FPGA prototypes yet.

<comment><USER11> It is a start-up.

They just got their patents up last year, demanding working hardware seems unreasonable and they have working machines simulated in software.

<comment><USER12> Plus, I think they mention somewhere their whole long-time plan is to only license the architecture to third-party hardware manufacturers.

<comment><USER13> I recall somebody asking Ivan about that in a previous talk, and he pointed out that Intel is a much larger company than ARM, so they'd rather manufacture and sell chips, and fall back on licensing the architecture if Plan A didn't work out.

<comment><USER8> Why is being a big company something to strive for?

<comment><USER14> Ivan said something in the aforementioned video to the effect of "Last year, Intel's dividend was as much as ARM Holding's entire revenue."</comment></comment></comment></comment></comment></comment></comment>