SQLite as an Application File Format (2014) - <AUTHOR> https://sqlite.org/appfileformat.html <comment><USER5> Previously: [https://news.ycombinator.com/item?id=18754634](https://news.ycombinator.com/item?id=18754634)

</comment><comment><USER6> The great thing about applications that use sqlite as their application file format is that you can write secondary apps and utilities to supplement the applications themselves.

For example, Adobe's Lightroom uses sqlite as their application file format, which means that it's almost trivial to write an application to help it do things it's either really bad/exceedingly slow at (like removing images from a catalogue that no longer exist on disk) or literally can't do (like generating a playlist of files-on-disk for a specific collection or tag), without being locked into whatever janky scripting solution exists in-app. If there even is one to begin with.

All you need is a programming language with a sqlite connector, and you're in the driving seat. And sure, you'll need to figure out the table schemas you need to care about, but sqlite _comes with sqldiff_ so it's really easy to figure out which operations hit which tables with only a few minutes of work.

Good luck reverse engineering proprietary file formats in the same way!

<comment><USER7> I have an application that uses sqlite as their file format but they've cleared the header so you can't open it directly.

I'd love to be able to make secondary applications like you've described but being enterprise software they don't want to make it too easy.

They obviously want to keep people locked in with their $40k per seat application!

I guess the first step is figuring out the page size and other bits the other meta data you set in the header [1].

I know I just have to sit down and understand the format better and I will eventually figure it out...

[1] [https://www.sqlite.org/fileformat.html](https://www.sqlite.org/fileformat.html)

<comment><USER6> That's mean, but you'll just have to live with that. If someone takes active steps towards preventing people from making secondary apps/utils, then they're (a) jerks, and (b) allowed to do that.

Might be worth actually contacting them to ask why, if you can make the case that secondary applications will increase the value of their app, not decrease it.

<comment><USER8> >That's mean, but you'll just have to live with that

I mean I know we're all on board with the idea of intellectual property actually being a thing now, but surely there are limits? I've seen people take the hard-line stance that if something is your property you should be able to dictate _exactly_ under what situation it can be used, but there have to be limits to IP holders rights on some level, and I feel like reverse engineering a file format is a pretty reasonable place to draw that line.

<comment><USER9> > we're all on board with the idea of intellectual property actually being a > thing now

We most certainly are not. I personally believe that intellectual property as a whole doesn't make sense in the 21st century and should be abolished.

> there have to be limits to IP holders rights on some level

There are. The laws generally recognize fair use and reverse engineering for interoperability.

> I feel like reverse engineering a file format is a pretty reasonable place > to draw that line

Absolutely. Unfortunately, in the US it seems corporations can force people to give up their rights by making them agree to it. Therefore, "you must not reverse engineer our software" is a standard clause in every contract and it's not negotiable.

<comment><USER10> I'd be more on board with it if it were taxed like other property. That patent's worth $1B, you say? Better let Accounting know. Conversely: you're suing for $1B for a patent violation but you only paid taxes on it being worth $300? $300 max it is, then.

<comment><USER11> > taxed like other property.

Most property owned by companies is not taxed.

<comment><USER12> Actually physical property is at least in my state both physical plant and inventory are taxed.

<comment><USER11> Interesting, I was not aware of those taxes.

[https://taxfoundation.org/does-your-state-tax-business- inven...](https://taxfoundation.org/does-your-state-tax-business-inventory/)

<comment><USER13> Nice introduction. Thank you.

I think taxes are very difficult. Personally, I think when Google or Facebook places an ad on its own (public) platform, they should have to pay a tax similar to a sales tax as they have sold themselves. Probably shouldn't apply to the ads on google.com homepage as they don't sell those as far as I understand.

The article mentions neutrality in taxes which is either silly or disingenuous though. Our tax code is not neutral. We openly use tax code as a way to motivate public behavior.

Ad space as inventory would be an interesting concept. It is tricky though. What is the value of something before it is sold? What is a fair tax rate? How often does this inventory adjust as you have more or fewer users? Messy.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> "Atomic transactions" is a feature needs _formal_ support in random file formats way more often than people realize. Simply writing to a file at all in an guaranteed-atomic way is much harder than it looks. That guarantee becomes important when your app gets widely distributed. If you have a million users of your free mobile app, 1 in a million events happen every day. For example: random hardware shutdown midway through a file write operation. How is your app going to react when it reads back that garbled file?

I've used SQLite on mobile apps to mitigate this problem. I've used LMDB on a cloud app where the server was recording a lot of data, but also rebooting unexpectedly. Would recommend. I've also gone through the process of crafting an atomic file write routine in C. [https://danluu.com/file- consistency/](https://danluu.com/file-consistency/) It was "fun" if your idea of fun is responding to the error code of fclose(), but I would not recommend...

<comment><USER15> Posix (and I think windows) guarantee this atomically and durably overwrites a file:

tempfile = mkstemp(filename-XXXX)

write(tempfile)

fsync(tempfile)

close(tempfile)

rename(tempfile, filename)

sync()

Assume the entire write failed if any of the above return an error.

In some systems (nfs and ext3 come to mind), you can skip the fsync and/or sync, but don't do that. It doesn't make things significantly faster on the systems where it's safe, but it definitely will lose data on other systems.

The only loophole I know of is that the final sync can fail, then return anyway. If that happens, the file system is probably hosed anyway.

<comment><USER16> You need a recovery step on startup to retry the rename if tempfile is complete, or delete it if it isn't.

That means you need a way to verify that tempfile is complete. I do that by removing filename after completing tempfile. And that requires a placeholder for filename if it didn't already exist (e.g. a symlink to nowhwere).

On crash, rename may leave both files in place.

This technique doesn't work if you have hardlinks to filename which should refer to the new file.

<comment><USER17> Regardless of whether the tempfile is complete, you can just ignore (or delete) it on startup. From the caller's perspective, the save operation doesn't succeed until the rename is done and written to disk.

</comment></comment></comment></comment><comment><USER18> I've been reading through many of the other pages on the SQLite website. I didn't realize that SQLite is developed by a company (Hwaci) and funded by selling support, licenses, and testing services. I had always assumed it was an open source project backed by a non-profit, similar to the Apache Software Foundation.

This is another very interesting example of an open-source business. I would be interested to learn more about how the Hwaci company operates (revenue, number of employees, etc.). I find this very interesting:

> We are a 100% engineering company. There is no sales staff. Our goal is to > provide outstanding service and honest advice without spin or sales-talk.

They list some "$8K-50K/year" and "$85k" price tags directly on their "Pro Support" webpage. These would usually be behind a "Schedule a Call" or "Get Quote" button. I've been thinking about doing something similar with my own on-premise licenses and support contracts. I'm not very good at sales and I don't really want to hire a sales team, so I'd be interested to know how this worked out for them.

I also liked this sentence, which is very similar to Basecamp's philosophy (and both companies were started around the same time - 1999 vs 2000):

> Hwaci intends to continue operating in its current form, and at roughly its > current size until at least the year 2050.

It's interesting to think that SQLite could have raised money and grown into a billion-dollar public company with thousands of employees.

I'm going to listen to this Changelog interview with Richard Hipp now [2], and also this talk on YouTube [3].

[1] [https://sqlite.org/prosupport.html](https://sqlite.org/prosupport.html)

[2] [https://changelog.com/201/](https://changelog.com/201/)

[3] [https://www.youtube.com/watch?v=Jib2AmRb_rk](https://www.youtube.com/watch?v=Jib2AmRb_rk)

<comment><USER6> Also important to note is that enterprise pricing is way higher than you'd think because when an enterprise needs a solution, something like $1000 is literally too cheap to justify the cost of procurement (which involves not just getting a license, but also internal documentation, which in turn needs to be entered into whatever process certification programme is in place, etc.)

<comment><USER19> That's why many companies have a policy where licenses below a certain value can just be expensed.

</comment></comment></comment><comment><USER20> From experience, I can also recommend using SQLite as an application file format. I landed on SQLite after looking for solutions for a file format for an educational app we made for simulating biological neural networks. The app is cross-platform, written in Qt and the simulations needed to be stored as a JSON describing the network, a thumbnail and some metadata. It was also intended to be extensible with more features and backwards compatible if new versions were released. I considered creating our own simple format, using ZIP files, HDF5, Qt resource files or SQLite.

I landed on SQLite for many of the reasons outlined in this article and in particular because of how easy it was to implement and maintain. SQLite is supported natively in QtSql, which made it extremely easy to write the save and load functions, and later extend these with more data fields. In addition, we did not have to worry about cross-platform support since this was covered by SQLite and Qt already.

<comment><USER21> Interesting, I used HDF5 in a similar situation because we needed to save a lot of same-sized rows of data (simulation time steps), so a matrix-oriented format seemed to make sense but it wasn't entirely without some need for cross-referencing between tables, so it does make me wonder now if sqlite would have been a comparable or better choice. Any reason for rejecting HDF5 in your case?

<comment><USER22> Is it cheap to insert/update/delete data in HDF5? If not, that should be the answer. I'm also curious if Parquet would fit your requirements.

</comment></comment></comment><comment><USER23> SQLite is amazing – trivial to embed in almost every context I've tried, rock solid, incredibly flexible, performs well enough for most use cases and for the others it probably still performs better than the solution I had in mind. I wouldn't dream of using anything else for application data, unless you _need_ something text based for human readers as well, and can't deal with having people use a SQLite client. And even then, really consider whether you _need_ this.

With SQLite embeddable in websites thanks to wasm, and the ability to create object URLs it's also pretty trivial to make full blown (read-only) SPAs delivered as a single HTML file. This last bit might seem crazy – and it kind of is – but if your clients are all on a LAN in a corporate network so bandwidth and latency aren't really an issue it makes a bit more sense.

I love SQLite, hands down one of my favorite tools.

</comment><comment><USER24> I think SQLite is underrated.

With WAL mode + increasing the page cache you can get some excellent concurrency, even if doing reads and writes at the same time.

With rqlite it's easy to make it a server database and have a cluster of SQLite databases ([https://github.com/rqlite/rqlite](https://github.com/rqlite/rqlite)).

I wouldn't try to create the new Instagram with it, but I think it'd be capable enough for many apps that are built on top of more complex DBs.

<comment><USER25> You are right! In our Delphi/Pascal community we have [https://github.com/synopse/mORMot](https://github.com/synopse/mORMot), which is a full client/server DB, remote ORM and RPC system based on top of SQLite.

</comment></comment><comment> trevor-e While we are on the topic of file formats, I stumbled across a cool project the other day that aims to make parsing well-known binary formats much easier: [http://kaitai.io/](http://kaitai.io/)

In high school I was obsessed with the game Starcraft. It came with its own map editor that would save to its own proprietary format. Some smart people came along and reverse engineered that format and allowed us to do all kinds of neat things we weren't supposed to. I see modding communities for games are more popular than ever, and finding this project brought back lots of great memories.

<comment><USER26> I'm super-on-board with declarative binary parsing schemes (i.e. not lua or python) but my own attempt to use kaitai was met with the same frustration that presumably caused PDF to be missing from the format gallery list ([http://formats.kaitai.io/](http://formats.kaitai.io/)): jump or offset-based structures

At the time, I tried using a stunt like defining one huge blob that "eats" the main file, then reaching back into it as we learned more, but it looks like somewhere along the way they acquired a "substream" behavior ([https://github.com/kaitai- io/kaitai_struct_doc/blob/c53060f7...](https://github.com/kaitai- io/kaitai_struct_doc/blob/c53060f773acd80d9a27f7a199d03256d9763713/user_guide.adoc#choosing- a-substream)) so maybe it's worth another look

<comment><USER27> Can you explain more? I deal with a shocking number of poorly supported binary formats, and this tool looks awesome. Buuuuttt... a lot of these formats are "headers with offsets to structured data". They're intermixed logs, so multiple incompatible bi art formats are freely mixed in the same file.

<comment><USER26> I can most easily speak to the PDF example, which is harder than the target I was going after due to it being a mixture of text and binary, with __a lot __of optional behavior (EOL is CR or LF or CRLF seemingly at random, for one painful example)

I took a few minutes just to kick the tires on the startxref of PDF, to get a feel for how the substream business plays out, and then stopped when I got to the part about how the offset position is written in a dynamically sized ascii string but represents an offset in the file

        meta:
          id: pdf
          file-extension: pdf
          endian: le
        seq:
        - id: magic_bytes
          contents: '%PDF-'
        instances:
          startxref_hack:
            type: startxref_hack0
            pos: _root._io.size - <USER2>
            # pick a reasonable guess to wind backward
            size: 24 - <USER4>
          eof_marker:
            # contents: '%%EOF'
            type: str
            encoding: ASCII
            size: 5
            # this isn't strictly accurate,
            # due to any optional CRLF trailing bytes
            pos: _root._io.size - <USER4>

        types:
          eat_until_lf:
            seq:
            - id: dummy
              type: u1
              repeat: until
              repeat-until: _ == 0xA

          startxref_hack0:
            seq:
            # this isn't accurate, since we may have jumped
            # into "endobj\n" or worse
            - id: junk
              type: eat_until_lf
            - id: startxref_kw
              contents: 'startxref'
            - id: startxref_crlf
              type: eat_until_lf
            - id: startxref_offset
              type: str
              encoding: ASCII
              terminator: 0xA

As best I can tell, the actual definition would involve a hypothetical `repeat: until-backward` where it starts at EOF (-5 in our case, due to the known EOF constant), reads _backward_ until it hits LF (and/or CR!), captures that as the startxref offset, reads backward eating CR/LF, skips backward `strlen("startxref")` bytes, and then is when the tomfoolery starts about reading the "xref" stanza, which, again, is a ascii description of more binary offsets, using zero-prefix padded numbers because of course it does

Don't get me wrong -- it's entirely possible that kaitai is targeting _strictly binary_ formats written by sane engineering teams, but the file format I was going after had a boatload of that jumping-around, repeating structs-of-offsets, too, so my holding up PDF as a worst-case example isn't ludicrous, either

</comment></comment></comment></comment><comment><USER28> Amongst all the praise we're seeing for this approach here, may we please just have a minute's silence for the various exploits sqlite has seen when loading arbitrary database files over the years?

My recent favourite being [https://media.ccc.de/v/36c3-10701-select_code_execution_from...](https://media.ccc.de/v/36c3-10701-select_code_execution_from_using_sqlite)

<comment><USER29> This is a security flaw of the system or application using SQLite, though. It is trusting data it should not trust and executing code provided with that data.

<comment><USER30> If I use libpng or libjpeg to decode an image and that causes a buffer overrun or another security vulnerability, that is very much a bug in libpng or libjpeg.

The same is true when I open a database in SQLite: if that causes a security problem, it's a bug in that library. I don't even see how you could validate a database file before you hand it over to SQLite.

<comment><USER31> The exploit mentioned above relies on third party functions being loaded into the database. eg application specific functions

Those functions have to be _explicitly_ loaded by the application after loading the database. eg: They're not stored in the database and loaded with it.

You'd validate the database (past the standard integrity checking), by loading it _and not_ adding any third party functions. Then check it's not doing anything dodgy with views, or outright disable views. Then load the third- party extensions (if needed).

If you wanted to go even further, you could also add an authorizer callback function:

[https://www.sqlite.org/c3ref/set_authorizer.html](https://www.sqlite.org/c3ref/set_authorizer.html)

That's called (multiple times) any time a SQL statement is prepared/executed, and catches things like functions being run, tables being accessed, etc. You can use that to only allow a whitelisted set of functions to run, which would help in some scenarios. eg:

[https://github.com/sqlitebrowser/dbhub.io/blob/e1c5042f857e9...](https://github.com/sqlitebrowser/dbhub.io/blob/e1c5042f857e9fadc13eb0ff193f813103b2eef6/common/sqlite.go#L196-L226)

The "Defense Against Dark Arts" page on the SQLite website has further good info about this:

[https://www.sqlite.org/security.html](https://www.sqlite.org/security.html)

As a data point, I implemented the above "Defense against Dark Arts" stuff recently for an online SQLite data publishing platform to let people run free form queries on databases (dbhub.io). It wasn't all that hard to implement. Much less so that I'd been expecting. :)

</comment></comment></comment></comment><comment><USER32> I was surprised, but also intrigued when I discovered that Manga Studio (Clip Studio these days) used sqlite for the native file format, including the pixel planes.

I wish my predecessor at Krita had made that choice, instead of choosing to re-use the KOffice xml-based file format that's basically a zip file with stuff in it. It would have made adding the animation feature so much easier.

<comment><USER33> What's wrong with xml + zip?

<comment><USER34> XML-based formats are export formats, not state-keeping formats. To use an XML-based format for storage, need to have a separate, canonical _in-memory_ representation of the data, which you then snapshot and serialize into XML upon request. You may or may not be able to get away with serializing less than your full in-memory object graph upon save, using techniques similar to DOM reconciliation. Either way, you'll still likely need your entire document/project _represented_ in memory.

If you're working with something analogous to a text document, this snapshot- and-serialize approach to saving works fine. If you're working with other types of data, though, this approach only works for trivial projects; once your document exceeds ~100MB, the overhead of snapshotting+serializing your object graph becomes bad enough that people stop saving very often (dangerous!), and it also makes the saving process itself more fragile (since the longer a save takes, the more likely it becomes that the process might be killed by some natural event like a power cut during it†.)

And, once your project size exceeds the average computer's memory capacity, an in-memory canonical representation quickly becomes untenable. You start to have to resort to hacks like forcing the user to "partition" their project, only allowing the user to work with one pieces at a time.

With an applicaton store-keeping format, you have none of these concerns; the store is itself the canonical data location. You don't have a canonical in- memory representation of the data; the in-memory representation is simply a write-through or write-back caching layer for the object graph on disk, and the cache can be flushed at any time. Or you may not have a cache at all; many systems that use SQLite as a file-format just do SQL queries directly whenever they want to know something, never instantiating any intermediate in-memory representation of the data itself, only retrieving "reports" built from it.

† You can fix fragile saving with a WAL log, but now the WAL log is your true application state-keeping format, with the XML format just being a convenient rollup representation of it.

<comment><USER35> I've never had my problem scale to the size that required a database/SQL, but I don't quite get the advantage of your solution. Having all your interactions with data have to go to disk though a cache muddles things b/c it makes it much harder to reason about performance (b/c when do you have a cache miss? and how do you configure a cache properly?) You introduce a lot more blackmagic variables to reason about.

If you're editing images I'd think it'd just makes more sense to have all of your stuff in RAM and then a saving-to-disk is done on a separate thread. I don't quite get why the users would stop saving in this example.

I'm not saying you're wrong - but more asking for some more details b/c I've never imagined using a DB on data that can fit in RAM

<comment><USER36> It's primarily a problem of inflexibility handicapping performance, not of "cache misses" and clever algorithms.

For example, imagine a word processing program opening a document and showing you the first page: you could load 50MB of kitchen sink XML and 250 embedded images from a zip file and then start doing something with the resulting canonical representation, or you could load the bare minimum of metadata (e.g. page size) from the appropriate tables and the content that goes in the first page from carefully indexed tables of objects. Which variant is likely to load faster? Which one is guaranteed to load useless data? Which one can save the document more quickly and efficiently (one paragraph instead of a whole document or a messy update log) when you edit text?

<comment><USER35> ah okay, incremental loading seems essential and I hadn't considered it. Thanks for explaining :)

</comment></comment></comment></comment></comment></comment><comment><USER37> Recently I've been using SQLite to store state for a data-syncing CLI utility in Python where I was considering using CSV or JSON at first. The Python 'Dataset' library, [https://dataset.readthedocs.io/en/latest/](https://dataset.readthedocs.io/en/latest/) has taken almost all of the friction out of using SQLite for trivial state persistence.

I don't have to manually manage schema or create tables - they're all lazily created from Dict keys on insertion, but indexes and SQL can still be used as desired.

This lends itself really well to Jupyter notebook assisted development, where functions can be quickly and interactively iterated without having to muck around with existing tables whenever data changes shape.

It's been a real productivity boost, and I've been looking around for something similar to use in Clojure.

<comment><USER38> Thanks for posting this, it's exactly what I was looking through this thread for.

</comment></comment><comment><USER39> How would SQLite go as for something like Audacity projects? Audacity projects are a pile of files: an XML file that declares the project metadata, tracks, channels, envelopes (an example of non-destructive editing), and references to the zillions of audio files that it drops in a directory beside the XML file (typically less than 1MB each).

I presume it splits all the audio up into small files so that most types of edits can only need to touch a small area.

If you directly ported that to SQLite, would it work fairly well, or would you want to restructure it somehow? Things like additions or deletions, would it need to write lots of extra data to the disk (would it be doing something like defragmenting, or would it grow larger than it should, or are there other tricks that I don't know about to delete a chunk from the middle of a file without needing to rewrite all of the file beyond that point)?

<comment><USER40> Adobe Lightroom seems comparable (non-destructive image editing.) It uses a sqlite database that points to image files on the filesystem. I think precisely the same approach could be taken with an audio editing application.

You could also consider putting the audio files into the sqlite db, which _might_ work alright. I've heard of image thumbnails being stored in sqlite dbs (maybe by Lightroom iirc?) though those are probably smaller than your audio clips I'm guessing.

</comment></comment><comment><USER41> Shouldn't this has 2014 inside the title? The story has also appeared several times in HN before.

Very interesting concept but now I think perhaps application file format using TileDB will be much better since it can support sparse data as well [1].

[1] [https://github.com/TileDB-Inc/TileDB](https://github.com/TileDB- Inc/TileDB)

<comment><USER36> TileDB uses a "pile of files", as the article calls it, it doesn't have SQL, and it has a rather embarrassing point of view on race conditions ([https://docs.tiledb.com/main/basic- concepts/consistency](https://docs.tiledb.com/main/basic- concepts/consistency)). It could be a good choice for applications that rely on large and simple multidimensional arrays, hardly the same "market" as application file formats.

Moreover, database tables are a _very_ good fit for sparse multidimensional arrays.

<comment><USER41> Perhaps SQlite or any SQL engine is not a good idea for application file format [1],[2], or is it just bad for web applications?

[1][https://news.ycombinator.com/item?id=23517663](https://news.ycombinator.com/item?id=23517663)

[2][https://support.google.com/chrome/thread/17537877?hl=en](https://support.google.com/chrome/thread/17537877?hl=en)

</comment></comment></comment><comment><USER42> We transitioned the file format for the images in our climbing-topo app from a pile of files to SQLite (the images are tiled).

Going from having high hundreds of tiny files on disk per topo photo to just one was an incredible boon to productivity for things like data backup and transferring files onto the device for testing.

<comment><USER43> Tell me more about your app

<comment><USER42> It acts as a digital window into our print guidebook catalogue. The books are mostly for the uk, with the rest spread across Europe.

I take the finished desktop-publishing documents and extract and package the data to put into the app, including tiling the images so we can have very high resolution photos on low-end devices.

There's a semi-technical article about the topo-view implementation here:

[https://www.ukclimbing.com/articles/features/rockfax_app_dee...](https://www.ukclimbing.com/articles/features/rockfax_app_deep_dives_part_0_topos-12679)

</comment></comment></comment><comment><USER44> Years pass, now decades, yet SQLite is still the most pleasant DB to work with.

<comment><USER45> Agree, but it also side-steps some complex topics like sharding, replication, high-availability, etc. I'm not suggesting those topics should be addressed by SQLite; that simplicity is an asset.

Has anyone used SQLite remotely over a network?

<comment><USER46> When Subversion changed their working copy format to rely on an SQLite database, I had a number of problems using working copies on network file systems.

No doubt this was an indication that the network file system was incorrectly configured, but I think the fact that it worked in practice with the file- based format and didn't with the SQLite format is a strike against the idea of using SQLite for what the user sees as saving a file.

<comment><USER47> SQLite recommends against sharing it on NFS for multiple access because locking is broken on NFS for all kinds of files, not just SQLite.

> But use caution: this locking mechanism might not work correctly if the > database file is kept on an NFS filesystem. This is because fcntl() file > locking is broken on many NFS implementations. You should avoid putting > SQLite database files on NFS if multiple processes might try to access the > file at the same time.

Source: [https://www.sqlite.org/faq.html#q5](https://www.sqlite.org/faq.html#q5)

<comment><USER46> Fair enough, but that means if someone's following the advice in the article ("It deserves your consideration as the standard file format on your next application design") they need to consider "will our users want to use our files on network filesystems?".

<comment><USER48> You could always have a lock file alongside the main to signal to other instances of the app that the file's in use. A bit janky, but workable.

<comment><USER46> That's no worse than what applications often already do when they use a simpler file format.

But you also have to worry about SQLite (reasonably) refusing to operate because it tries to make a locking call and gets an error response. I think there are options to turn that off, though.

</comment></comment></comment></comment></comment></comment></comment><comment><USER49> What happened to storing blobs in a database being giant rookie mistake? I'm currently dealing with this at the office, where now a SQLServer database has grown to unmanageable size from helpdesk software attachments.

<comment><USER50> Did you read the article? This isn't about storing blobs as data in SQLite, but rather using SQLite-formatted databases as the file format for an application.

As a simple example, Word documents are just zipped XML text files (try unzipping a .docx and looking inside). Instead of using this, you could a SQLite .db file (probably with a different extension), translating the XML files into tables, and folders into databases. The OpenOffice case study has more details: [https://sqlite.org/affcase1.html](https://sqlite.org/affcase1.html)

<comment><USER49> Apparently I'm misreading the following paragraph.

Any application state that can be recorded in a pile-of-files can also be recorded in an SQLite database with a simple key/value schema like this:

        CREATE TABLE files(filename TEXT PRIMARY KEY, content BLOB);

<comment><USER50> I agree that is unclear for an introduction, and that the article could have been written better. The "pile-of-files" they are referencing isn't a filesystem of arbitrary and unstructured binary data stored for humans to look at later (like your Helpdesk attachments), but rather structured and specific data for an application to read in a well-defined manner.

Your Helpdesk example used SQLServer problematically because a SQL database shouldn't be used as an arbitrary file store. But if you know what the file structure is and have a reasonable grasp for how it might scale (that each binary blob is small, that each user only adds one row to the database, etc), there are huge advantages to "a SQLite table with lots of binary and text columns" versus "a folder with lots of binary and text files." And if those text files are just small key-value pairs then maybe they should also go in SQLite.

</comment></comment></comment></comment><comment><USER51> I used to advocate for this, then tried opening a sqlite file I'd been using on another computer and it said it was corrupted. I took it back to the original computer and it opened just fine. So there was something going on that made it more complex and less portable than something simple like json, which is super frustrating because otherwise it's a really useful format for datasets, and now I can't really trust it.

<comment><USER52> If you use certain new features of SQLite, the database file gets "upgraded". At that time, you must use a version of SQLite that understands these new features.

So, the other computer you used was probably running an older version of SQLite. Just update it to make it work.

<comment><USER53> That's a good catch. I'd there any way to tell which version of SQLite created the file? Aside from trial and error, I mean?

<comment><USER7> The version is stored in byte 96 of the header of the sqlite file[1].

I haven't looked but there will be some sqlite command to query it and I'm sure some viewer tools will display it as well.

[1] [https://www.sqlite.org/fileformat.html](https://www.sqlite.org/fileformat.html)

<comment><USER54> You can print the version with this magical incantation:

        $ python -c 'import sys, struct; print(struct.unpack(">I", open(sys.argv[1], "rb").read()[96:100])[0])' foo.db

</comment></comment></comment></comment></comment><comment><USER55> Quickbooks for Mac uses sqlite for its file format and this allowed users to query the books with any SQL browser!

Alas, they encrypted the file starting two years ago.

</comment><comment><USER56> The article has great points.

Main reason for existence of lot of file formats is that enterprises don't want just about everyone access their files and modifying them. It greatly reduces the usage of their proprietary software hence their revenues.

I remember the days when open office trying to render doc file but formatting used to suck big time.

Open source softwares should leverage this kind of file formats for inter operating.

<comment><USER40> > _Main reason for existence of lot of file formats is that enterprises don 't > want just about everyone access their files and modifying them. It greatly > reduces the usage of their proprietary software hence their revenues._

I'm sure this does happen, but it seems more like MBA-paranoia than a legitimate concern. For instance I sincerely doubt Adobe has lost any revenue by using SQLite with lightroom, despite various open source tools being able to interact with their lrcat (sqlite) files.

</comment></comment><comment><USER16> SQLite is a fine single-file application file format, but if you want average users to be able to inspect and perhaps fix application data when things go wrong, a tree of JSON-format files is more friendly.

True, this requires care to ensure that such files are updated reliably, but that's not quite rocket science.

</comment><comment><USER57> Is anyone doing this in Javascript (via electron) and encrypting the file in such a way that the data is protected and not an obvious sqlite db?

I have some ideas on how to do this, but I'm curious if there's a "preferred" way to do it.

<comment><USER58> SQLite has encryption plugin support, or you could encrypt the file yourself at rest. However, no matter what you are doing, as the application needs to be able to decrypt the file, you have to ship the key and thus the protection will be able to be circumvented.

It's really not worth the effort IMHO

<comment><USER57> That's a good point... I don't want to have the game rely on "phoning home" to decrypt the db.

<comment><USER48> On top of that, even if you did have the game "phone home", that doesn't do much against someone inspecting the memory of a running game; one could just wait until the game does have the decryption key.

</comment></comment></comment></comment><comment><USER59> I used it with a different extension to save application for a product I wrote.

Any classes that need to be saved had serialize() and deserialize() functions. Serialize before saving to SQLite and if read in, deserialize after reading it from the DB.

</comment><comment><USER60> Maybe. I've used XML as a file format for all my recent projects. It's well supported in the .Net framework, I don't need any 3rd party assemblies. Using SQL to write out an object graph would involve more work.

<comment><USER61> It depends on how large and complicated your XML file is.

Once your XML file hits a certain size, minor updates incur a gigantic performance penalty because you have to write out the entire XML file every time.

Random writes in the middle of an XML file are impossible. For example, if you were to change an attribute so that it's one character longer, you still have to rewrite the remainder of the file in order to shift everything over by one character.

That's the main reason why SQLite is so popular for applications.

(I know that's from personal experience. I had to support an old version of an application that constantly wrote out XML while the new version that we hadn't shipped used SQLite. A customer that made heavy use of the old version basically hit the limits of XML but the version that used SQLite wasn't ready for them yet.)

But, I'm going to be honest here: We had some tables that only had a few rows, so I moved those to XML files. QE really liked it because it was easy to diagnose issues.

<comment><USER60> I think it depends on whether you need a file format or a database. All the benefits of a database are wiped out if you are implementing File->Save for a document based application.

<comment><USER61> File->Save is an extremely outdated workflow leftover from when disk drives took multiple seconds and you couldn't edit the document while you were writing to the disk.

Now your application should continuously save without any manual steps.

</comment></comment></comment></comment><comment><USER62> I just converted a 90's-vintage xBase app to SQLite. I just love it. Should have done this years ago.

The only thing I miss is the ability to jump X% into an index. Not a deal- breaker, though.

</comment><comment><USER63> A lot of well known apps are exactly doing this, using a sqlite db as the internal file format: Whatsapp, WeChat, Google Keep, Apple iMessage, Notes and Calendar...

Probably tons more if you care to dig around.

</comment><comment> gjsman-1000 I worked with an SQLite Database on a personal C#/WinForms app. It was a bit complicated to get started, but afterwards, what a handy trick in comparison to using something like the discontinued SQL Server Compact.

I did, however, "enhance" it a bit (or "proprietarize" it) by encrypting it with a short password and an AES algorithm with some uncommon settings. I never shipped it that way, but the output file looked like any other proprietary app's file - a mess of random symbols.

</comment><comment><USER64> Everything old is new again. I can recall a similar movement at the end of 90s regarding databases as application file formats. That time it was Microsoft Jet Database engine.

</comment><comment><USER65> Concurrent access to SQLite is the main reason I haven't really looked into Deno. For that matter, I really think that WebAssembly file system implementations should have enough flexibility to be able to work with a SQLite database file in the host OS.

The needs for concurrent SQLite pretty much covers the need for a robust file system.

</comment><comment><USER66> I keep switching between csv, parquet and sqlite. I dont need another article to encourage me to switch again! :)

</comment><comment><USER67> Does SQLLite have a more text-friendly storage format? This would enable Text editors to do edits/modifications, which is critical to debugging and testing and learning. Savefiles don't need the same ongoing access optimizations, or size optimizations for wire protocols, so a less efficient format that is text editor friendly would help this cause.

Also, does SQLLite have libraries in the native code for the major languages to read and write to them? XML (ugh), JSON, and YAML have managed to get decent implementations in almost all major languages.

<comment><USER36> Using a text editor to mess with a SQLite database doesn't make any sense. Do you use text editors to edit, say, MPEG media?

SQLite databases can be edited with with generic GUI and command line tools, both SQL-based and tabular editors, which are safer and more convenient than a text editor could ever be.

</comment></comment><comment><USER68> When I was reversing the XMDF E-Book format, I was surprised to find an embedded SQLite database... in an E-Book?? After reading this article, it makes more sense why someone would make that decision

</comment><comment><USER61> One minor point that needs to be raised: Using SQLite as your application file format locks you into the SQLite source code.

(And in most cases that's not a bad thing, it's free and open source.)

But, if you truly want an open file format, someone needs to be able to independently write a program that can read your file without relying on third party dependencies. This is why the browser vendors decided not to put SQLite into the HTML and JavaScript standards.

It is an awesome product! I've worked with it for over a decade and I'm a fan!

<comment><USER69> > it's free and open source

It's more than "open source" it's in the _public domain_.

The file format is fully documented and if someone like ISO or ANSI wanted to, they could make a standard out of it. It's also forward compatible since inception and versioned.

The browser vendors decided not to put SQLite into their browsers because "key/value good, SQL bad" and "not invented here".

IndexedDB is a clumsy reinvention with minimal ACID properties and isn't far advanced from ISAM. They could have used the SQLite file format and implemented IndexedDB as an API over the top. They could have allowed both standards to be implemented and then let reality take its course to choose which one was successful.

</comment></comment><comment><USER70> To my very limited knowledge it is very common for iOS and android apps to store data in SQLite.

Is it possible to extract or change these databases?

I have a few (offline) apps on my phone that I'd love to append data to

<comment><USER71> For unjailbroken iOS the only way to extract these databases is to make an unencrypted device backup in iTunes (Finder in Catalina) and then look at the backup file. The backed up files won't be conveniently named but you can still find it.

<comment><USER70> Can I push my edits back onto my iPhone or is that a pipe dream?

<comment><USER71> I suppose it's possible, but you need to study the way backups are organized and verified.

</comment></comment></comment></comment><comment><USER72> Would such an app hold the sqlite database open while the user has the document "open", and live-write user changes back to the database immediately? Or would it follow the traditional model of the user choosing to "Save"?

I worked on an app that did the former many years ago (to an Access database, not sqlite), and it did not go well because this broke user expectations on the usual "open/save/save as" model.

<comment><USER40> I think the former is preferable and most modern users will prefer it. However if your users are upset about it, you could do the later, or you could do the former and give them a dummy/placebo "save" option that does nothing, or maybe VACUUMs the DB.

<comment> mark-r The problem is not Save, the problem is when you close the app without using Save - the user will expect that to work like Cancel and discard all session changes.

</comment></comment></comment><comment><USER73> Where can you use SQLite?

Embedded Systems: Yes

Raspberry Pi : Yes

Mobile Apps. : Yes

Desktop Apps : Yes

Browsers : No

Servers : Yes

Supercomputers : Yes

<comment><USER61> You used to be able to use it in a browser with Google Gears.

The reason why it was never adopted was because the browser makers wanted to be able to independently implement their own database instead of everyone having to use the same source code.

</comment></comment><comment><USER74> IIRC one of the attempts to deliver on "Github but for Design files" used SQLite as an app file format.

</comment><comment><USER75> Is SQLite able to handle multiple processes that want to read or write to the file database at the same time? Or nearly at the same time.

They say that SQLite is more in competition with fopen, for file open, rather than a true RDMS system.

</comment><comment><USER76> Yeah you can use SQLite as a file format. We use it to store our project data at [https://github.com/zubairq/pilot](https://github.com/zubairq/pilot)

</comment><comment><USER77> They say that Microsoft Word, PowerPoint, etc are "fully custom formats", but actually those formats are "wrapped pile-of-files formats" (and 7-Zip can open them), although the contents of the files inside are custom formats (and they seem to have failed to consider that category).

The application ID number in the SQLite header can be used to identify application file formats. The application ID number is a 32-bit number, and there have been a few different ways to handle it; I have seen the use of hexadecimal and of ASCII; I used base thirty-six, and I have then later seen the suggestion to use RADIX-50. Additionally, there is a document about "defense against dark arts" in case you need to load untrusted files.

TeXnicard uses a SQLite database file (with application ID 1778603844) for the card database file. The version control file (optional, and not fully implemented yet) uses a custom format (which is fully documented), and it does support atomic transactions. It consists of a header followed by a sequence of frames, which are key frames and delta frames. The header of the version control file contains two pointers: one to the beginning of the most recently committed key frame, and the other one to the end of the most recently committed frame (whether a key frame or a delta frame; if all frames are fully committed, this will be equal to the length of the file). These pointers are written only after the rest of the file is written; if it gets interrupted, reads will ignore the partially written data, and further writes will overwrite the partially written data.

ZZ Zero uses a Hamster archive of custom (but documented) formats as its world file format. (A Hamster archive is zero or more "lumps" concatenated together. A lump consists of a null-terminated ASCII filename, 32-bit PDP-endian data size (measured in bytes), and then the data of that lump. The preceding text in these parentheses is the full definition of the Hamster archive format; you can use this to implement your own.)

Free Hero Mesh uses a "pile-of-wrapped-pile-of-files" format. A puzzle set consists of four files: .class (which stores class definitions), .xclass (which stores pictures and sounds to be used by the class definitions), .level (which stores levels), and .solution (which stores solutions). The .class file is a plain text file; the other three are Hamster archives. These are four logically distinct parts of a puzzle set; this allows you to split them apart, to create symlinks to share class definitions with puzzle sets, to substitute your own graphics, to work with multiple solution sets (e.g. per user), etc. If you need to do more than that, then you can of course extract the lumps if needed. For class definitions, you can just copy and paste the text.

MegaZeux used a fully custom format before, but now it uses a ZIP archive with the stuff inside being custom formats (one of which is the "MegaZeux Property List" format, which I have documented in Just Solve The File Format wiki; the authors of MegaZeux did not seem to document this format themself anywhere, so I figured it out and did it by myself).

For some cases, SQLite database is a good application file format; other times, I think other formats (such as text formats) may be better. It depends on the application. XML is too often used for stuff that isn't text markup stuff, and XML is especially bad for stuff that isn't text markup stuff, I think.

If you use SQLite though, you will get more than just the database access. It also gives you the string builder functions, the sqlite3_mprintf function, a page cache implementation, memory usage statistics, and a SQL interpreter; the SQL interpreter can be used as one way to allow user customization and user queries (including batch operations), without having to make an entirely new scripting language to embed.

They mention interfaces of SQLite are available for many other programming languages, although at least one that doesn't seem to have a interface to SQLite is PostScript (although you can use %pipe%, it doesn't work so well especially since it is only a one way pipe), and I am not sure if awk has it either.

<comment><USER39> .doc, .ppt, &c. were fully custom. .docx, .pptx, &c. are ZIP files. Remember that this was written in 2014, and many people were still using the old formats then, because many people didn't have Office 2007 or newer or software that could cope with this new format. (Now I think very few people still use the old Office formats.)

<comment><USER77> No, .doc, .ppt, etc are wrapped file of files, although not using a ZIP container format. I also believe Visual Pinball uses the same container format as Microsoft Word. 7-Zip just calls this format "Compound", although there are other names for it, such as "DocFile" and "POIFS". I just tried now such a file, and I got the following listing from 7-Zip:

         Date      Time    Attr         Size   Compressed  Name
      </comment></comment></comment><comment></comment><comment></comment><comment>- ----- </comment><comment></comment><comment> </comment><comment></comment><comment>  </comment><comment></comment><comment></comment><comment></comment><comment>
                          .....        35842        36352  WordDocument
                          .....          106          128  [1]CompObj
                          .....         4096         4096  1Table
                          .....         4096         4096  Data
      2007-12-25 22:33:00 D....                            ObjectPool
                          .....         4096         4096  [5]DocumentSummaryInformation
                          .....         4096         4096  [5]SummaryInformation
      </comment><comment></comment><comment></comment><comment>- ----- </comment><comment></comment><comment> </comment><comment></comment><comment>  </comment><comment></comment><comment></comment><comment></comment><comment>
                                       52332        52864  6 files, 1 folders

</comment><comment><USER78> GeoPKG is an open format that is just a SQLite schema. It's cool!

</comment><comment><USER40> Is there any good way to use SQLite as an application file format while also using it in WAL mode? They seem sort of mutually exclusive; near as I can figure the way to do this would be to execute `PRAGMA journal_mode=DELETE` when the user requests a save file, then copy the DB to the location the user specifies. I think this negates a few of the advantages of using sqlite as an application file format though.

<comment><USER79> I think the backup api or VACUUM INTO are the preferred ways, though you're gonna probably want to turn off wal on the exported copy still.

<comment><USER36> Shrinking the database is the equivalent of saving the file when the application shuts down or saving an extra copy to make a backup.

</comment></comment></comment><comment><USER80> are there any other "file format oriented" data storage system?? I see software tend to use zip file, sqlite or good old ole structured storage.

In my job we used a fat filesystem as a storage system and recently switched to sqlite, and while i love it i didn't really find any alternative.

</comment><comment><USER81> Where's a good place to find some useful public SQLite data sets?

</comment><comment><USER82> Are we talking about good old fashioned CRUD applications here or is this something different?

</comment><comment><USER83> Haha!! I thought it was only me using sqlite as my go-to file format in apps.

</comment><comment><USER84> What is an application file format?

<comment><USER85> The first section of the article explains it.

</comment></comment><comment><USER86> SQLite is a database that tries so hard to not be a database :)

</comment><comment><USER87> Please no. This is an incredibly complicated file format that is not particularly well suited for file saving. It's also very slow ((edit: correction) ~10 transactions per second) if you aim for integrity, or unsafe if you tune for performance (~50k transactions per second, but if your program or computer dies half way, your file is hosed). So, keeping files up to date in place makes your ui janky.

You can work around it by working in memory and writing out a whole new database from in memory structures on save and then do the atomic rename. But if you do that, you are probably better off with json, protobuf, or similar. The libraries around these formats are similarly battle tested, but they fit the needs better, supporting working in ram fully and then saving cleanly and easily.

<comment><USER88> > Please no. This is an incredibly complicated file format that is not > particularly well suited for file saving. It's also very slow (~100 > transactions per second) if you aim for integrity, or unsafe if you don't.

The kind of application files they are talking about (things like word processor documents, spreadsheets, drawings, source code control system data) would only be writing sporadically. During one of those sporadic writes they might need to update thousands of rows but those could all be done in one transaction.

<comment><USER87> I made a mistake -- it's an order of magnitude slower. That means a single transaction is within the threshold for human perception. If you block the UI thread on a transaction, you're now dropping 6 frames at 60fps. If you block other operations on the transaction completing, you end up with lag.

Reference: [https://www.sqlite.org/faq.html](https://www.sqlite.org/faq.html), question 19. (I've seen similar when testing on SSDs locally).

<comment><USER89> If you have an FPS or interactivity target that needs to be guaranteed to be better than about 0.1-1 second - depending on platform - you can't have any disk IO at all on the UI thread, or any thread that's supposed to react at interactive rates at all times.

It has generally become much better in the last decade or two, but one should still expect most OS's to _sometimes_ pause for excessive amounts of time on disk IO unless the API is specifically guaranteed to never pause. Even then one would be wise to measure/log deviations if it's critical for the application. OS guarantees might also be contingent on driver/subsystem guarantees, and bad drivers might sometimes affect what seems completely unrelated upstream systems.

<comment><USER87> > _If you have an FPS or interactivity target that needs to be guaranteed to > be better than about 0.1-1 second - depending on platform - you can 't have > any disk IO at all on the UI thread, or any thread that's supposed to react > at interactive rates at all times._

Yes, and other file formats encourage doing things in memory, so you don't have any disk i/o in the common path.

Using sqlite as a file format strongly discourages the simple, jank-free until you press save workflow of slurping your content, operating on it in memory, and then outputting it all in one operation as a response to an explicit user action. Instead, your whole application gets small but perceptible delays across all operations and interactions.

<comment><USER90> That conflates the use of the schema with the use of the format features. You can, in fact, use your custom in-memory structures and then "slurp it out" to BLOB when you're ready to save.

<comment><USER87> Yes. But if you go that way, though, sqlite is just a slow and clunky format compared to, eg, json or protobuf. I doubt that any program out there uses sqlite this way, other than those which realize they painted themselves into a performance corner and hacked around it for compatibility.

</comment></comment></comment></comment></comment></comment></comment><comment><USER91> I have been thinking about this as well lately. For my app ([http://ngrid.io](http://ngrid.io)) I'm using a custom built Entity- Component-System ([https://en.wikipedia.org/wiki/Entity_component_system](https://en.wikipedia.org/wiki/Entity_component_system)) based UI framework in Rust.

In the ECS paradigm, everything is stored as struct-of-arrays as opposed to array-of-structs ([https://en.wikipedia.org/wiki/AoS_and_SoA](https://en.wikipedia.org/wiki/AoS_and_SoA)) and as a result, your serialization becomes trivial. You are not chasing pointers to other objects when serializing. Apple's Core Data takes the object graph approach and it's a real clusterfuck.

ECS is very similar to databases in that your data is normalized and things are referenced by offsets rather than pointers [https://floooh.github.io/2018/06/17/handles-vs- pointers.html](https://floooh.github.io/2018/06/17/handles-vs-pointers.html)).

Why roll your own thing as opposed to use say SQLite? If you use SQLite, you will probably have some OOP on top of it which introduces a serious impedance mismatch. With ECS, you cut that whole layer out.

I'm guessing that I'm losing on some atomicity but for my use case that doesn't matter that much.

<comment><USER90> ECS is useful for real-time, but it means that you are rolling your own query logic. This is fine for the runtime of game engines because they generally aren't dealing with that many data types(for the very most complex AAA games, perhaps a few hundred components) and the queries used at runtime are simple and tend to demand optimization at the data structure level.

If your purpose is an editing tool you may want to reconsider. Editing changes the goal in a very substantial way and the complexity of your queries goes way up, which is where SQL syntax absolutely shines.

<comment><USER92> FWIW, a game I'm writing uses in-memory SQLite as its ECS, mostly because I wanted to see if I could. SQLite is surprisingly performant. My game is not realtime, but I tested it, and I could actually redraw the screen at 60+ FPS while querying for render data every frame, and I still had lots of frame time to spare.</comment></comment></comment>


Work on these things - <AUTHOR> https://marginalrevolution.com/marginalrevolution/2019/12/work-on-these-things.html <comment><USER2> I would add new ways of searching in Internet. If I searched for food recipes in the 2000s I would find independent blogs with some real local/family taste. Now I have a hundreds of results from click bait sites with the same commoditized recipes and the ugly blog with a good recipe deep in the long tail. We can say that we need improvements in the long tail when the tail deserves to move up (or to the left in a xy chart).

Internet search is a driver for the world economy, a tiny improvement would improve the life of entrepreneurs and their ecosystem beyond elite circles.

<comment><USER3> The problem with this is that black-hat SEO's have learned to mimic "independent blogs" and their linking/referencing networks (blogrolls, etc.) to the point that search engines are basically forced to discount all of these results. We need trusted, curated guides to the "independent" Internet, like the ODP/DMOZ of old. For both humans and search engines alike.

<comment><USER4> Ive got a half baked project for a blog search engine that won't index any pages with ads...

<comment><USER5> How do you define "ad"?

Keep in mind that as soon as you do define it, people are going to look for a workaround.

<comment><USER6> > How do you define "ad"?

Attempt to sell.

Whether that's for an agenda, a product, a way of life, service, access, or anything else with a direct (or indirect) transaction of value - it's a dead giveaway.

<comment><USER5> Is a recipe for my grandma's minestrone soup (which requires you to buy a specific brand of chicken stock) selling you something?

Is a blogpost highlighting a particularly efficient way to write a tight loop selling you something?

Is a plain white webpage selling you the benefits of that particular shade of white?

<comment><USER6> I'll take your questions at face value. You are likely being downvoted for your use of an aggressive tone, but that's another matter.

1) Recipe for a good minestrone soup, no. Actively endorsing a specific brand of an ingredient, yes.

2) No. (One might argue you're building a brand and advertising yourself. I don't, necessarily. If the efficient loop was possible to create using only a particular compiler, then yes.)

3) Yes. The page is selling a particular idea, and most likely lying by accident. After all, how many screens have been calibrated for colour correction accuracy? Are you, as a reader, getting the correct information?

<comment><USER5> Apologies for the brash tone.

I take the view that making a value judgment on an idea is essentially selling that idea. i.e. "A is better than B" is clearly advocating for A, as is "A is good"

I disagree with ads in general, but my problem is that blocking any form of endorsement as you suggested above would mean we can't transmit really any useful information. Obviously that means there's some acceptable level of endorsement... but that goes back to the problem of having a game-able filter.

<comment><USER6> Don't worry, I am well aware that my answer came off as snarky and brash so a similarly phrased response should not have been unexpected. And I know that my position is extreme. Even with the overall attitude change over the past 6-8 years about advertising[ß] my views are still outside the mainstream.

This is a personal opinion, but I believe you _can_ do endorsements without falling into advertising trap. It takes effort, because you need to quantify your reasons of choosing a particular option (or a set of options) over the remaining ones. You need to specify the reasons, be painfully aware of and outline your own biases, and be willing to acknowledge the tradeoffs in your stated choice.

Obviously, if you are externally incentivised to make a particular choice before you broadcast about it, that's an ad. As one may imagine, I absolutely detest the entire influencer phenomenon - to the point of treating any product or service using them as bad actors perpetuating a disgusting system.

ß: Back in 2011/2012 HN was, in general, _very_ pro-advertising. A lot of the [vocal] posters made their money on it or dealt with large advertising budgets on a daily basis. Ad block users were derided as parasites or thieves. Anyone questioning the common good was pushed away. Compare to the tone shift over the past couple of years and the difference is stark.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> >A comprehensive guide to the American healthcare system.

im afraid this might be a moving target at best. So much of American healthcare is arbitrary, clandestine, and opaque. Prices are rarely made public, and justification or schedule for increase in drug prices or premiums is almost never made except in the most dire circumstances (Insulin inflation for example) and even then, its almost condescendingly boilerplate. Even seemingly simple things like ACA exchange markets and Medicare are mind- numbingly complex institutions whos coverage varies widely from state to state. Healthcare coverage is also governed by religious code in many states, where simple services like abortion, contraceptives, and family planning are subject to strict Judeo-Christian moral laws. many service providers can refuse to even process drug prescriptions should the customers request cause the employee some religious moral unrest.

>Who are the actors and what are their incentives?

Pharmaceutically these are largely institutions borne of dynastic wealth seeking profit. They run advertisements and sponsor content in local, state, and national news in order to drive customers to consume drugs and treatments, but they also lobby and in some cases bribe physicians and caregivers to give preferential treatment to their products.

<comment><USER8> > simple services like abortion

This is dishonest. Philosophers call it "begging the question."

For someone who considers a fetus to be a human life, distinct from the mother's, with all its own dignity, this is not a "simple service".

<comment><USER9> Simple and moral are different. You're entitled to your opinion on the morality of abortion, but it is a simple medical procedure.

<comment><USER8> This is not the way the parent comment was using the word "simple". This is obvious from context.

</comment></comment></comment></comment><comment><USER10> "Summaries of the state of knowledge in different fields": I've been thinking a lot about this one lately. There's so much confusion (and willful disinformation) in the world when it comes to science, and so many journalists don't understand that a single article published in a journal does not mean that there is now a scientific consensus around this - that usually comes years later when a meta-study comes along. In most fields, as an outsider, you basically have to get lucky that one of these meta-studies has cropped up recently and you happen to talk to an expert that can point you at it. In medicine there is the Cochrane organization that does a pretty good job of communicating these findings in a clear way. But why isn't there something like this for all fields of science, constantly updated, so journalists, policy-makers and curious amateurs like myself have a single point of entry when we want to understand the state of the art in a given scientific field? A trustworthy source where I could check what's certain and what's still up in the air when it comes to CRISPR, which variants of string theory have we managed to disprove using colliders, which are still being considered, what do the currently best climate models look like and how certain are we of their accuracy?

It'd be a huge undertaking - continuously interviewing researchers, quantifying and communicating their answers in a way that non-experts can understand. But on the other hand, all the science we're already doing is a huge undertaking, and it's crazy that we let all the knowledge that these experts accumulate just languish in obscure journals targeted only at other experts, and letting the 'state of the art' just be this implicit thing in the heads of dozens of experts, instead of being stated explicitly somewhere accessible.

<comment><USER11> In the long run, this could also help combat the "Fake News" phenomenon.

Part of the Utopian mindset of the early Internet was the belief that making more content available to people would automatically lead to people being better educated and informed. But of course it also opened up people up to unprecedented amounts of disinformation, deliberate or otherwise.

Slowly establishing authorities on given topics and building their reputation could help over time to filter the wheat from the chaff on various topics.

</comment></comment><comment><USER12> Society failing some potentially super effective people is observable. If we employed half as many tricks to align people with challenging (skills- appropriate) and socially valuable work as we do to align potential buyers with products, I think a lot more would get done and a lot more people could live meaningful lives. I know several examples of people not doing things they're good at. Actively marketing yourself as a requirement for getting valuable work is an obstacle for many. Sometimes business culture is an obstacle.

Another thing is the chilling effect that our surveillance state has. Hacking away on tech in the US is not as appealing as it was (for some) before it was clear that your prospects are mostly being a cog in a system geared towards social control.

I think people need real privacy to learn, grow, and collaborate, or they just won't do that to the degree necessary to meet their potential.

Schools in the US are a big problem, and seem to be victims of broken politics and ideological battles on the regular. Profiteering in higher education also doesn't help.

</comment><comment><USER13> > Life expectancy in Hong Kong is 84.23 years, more than five years longer > than the US and the highest in the world. Hong Kong is not that wealthy > (median household income is $38,000 USD); it's somewhat polluted; people > don't obviously eat what seems like a healthy diet; and they don't seem to > exercise a great deal. What should we learn from this?

We should learn that Hong Kong is probably a smaller territory so any of its "average" metrics has a higher chance of being an outlier, out of sheer randomness.

<comment><USER14> It's over 7 million people. That seems like a robust dataset to me.

<comment><USER15> The fact that it's a smaller territory means that effects that would even out in larger territories don't apply. I.e. it's a relative effect.

Here's an example: suppose a large retirement community of Chinese immigrants exists in Hong Kong, raising the average age. In order for this kind of retirement community to shift the age distribution the same amount in a country 10x larger, that retirement community would have to be 10x or so larger as well. A 10x larger retirement community is way less likely.

This isn't to say that all small things lie on the extremes of the distribution, only that an extreme value is likelier to come from a smaller sample.

</comment></comment></comment><comment><USER16> IQ, life expectancy, top-tier universities, super effective, culture of excellence, higher GDP, worker productivity, state of knowledge. Given that a list of priorities betrays a view of the world, I find something eery about this one. Nothing said is obviously wrong, but there's something between the lines that I can't quite put my finger on. To retreat to imagery, I feel like a forest is being missed for the trees, but with a hint of violence.

<comment><USER17> I think what society is really missing is this sense of belonging. We're raised to think that wealth correlates with happiness but it clearly doesn't.

History has always formed these loops of prosperity, then spirituality. I mean it's nice not to starve, but I think people would be happier being part of a tight knit tribe who support one another, even more than having air- conditioning and dating apps.

<comment> blaser-waffle > We're raised to think that wealth correlates with happiness but it clearly > doesn't.

Studies show it does, to a degree. Once you hit something like $70-80k the satisfaction levels off.

It's one of those hygiene things -- not having it will make you (or others) miserable, but having it doesn't mean you're good or happy.

<comment><USER17> For the individual, yes.

For a society, I think the culture of making as much as possible has left everyone slightly worse off, instead of slightly better off as we were promised.

<comment><USER18> Do you think that trying to get everyone to the 70k level (or the equivalent in their region) has made society worse off?

<comment><USER17> If society was focused on getting everyone to 70k, everyone would be happier. But instead, modern society encourages people to make as much as possible, create as many jobs as possible, and pay as close as possible to 15k.

</comment></comment></comment></comment></comment></comment><comment><USER19> Is the American healthcare system the most influential?

Well, from the point of view that it sets a bad example, it is...

<comment><USER10> I mean they pay roughly 3-4 times as much for the same level of service as the single payer healthcare in my country. So it's a massive market with much larger profit margins, the lack of vertical integration adds not just one, but many layers of profit generation (and total cost increase) compared to a single-payer solution, and you can skew the perverse incentives in favor of your product in ways that are not legal or possible in other countries (e.g. lobbying/bribing doctors to favor your new drug). So yes, it's definitely influential, and it attracts companies, but not only in favorable or sensible ways if you're a consumer/patient.

</comment></comment><comment><USER20> "Bloomberg Terminal for everything. Figure out a way to build a growing corpus of structured data across the broadest variety of domains."

What domains do you suppose will have the biggest impact?

<comment><USER21> This already kind of exists with Wolfram language's built in knowledge-base.

</comment></comment><comment><USER22> Point of order: Based on the anti-linkbait guideline for titles, can this be changed to something less imperative and more informative?

"Underfunded research initiatives" seems like a reasonable start, but maybe someone can come upwith something better.

</comment><comment><USER23> You do not need a Bloomberg terminal for everything but one for each industry vertical. What Bloomberg did for capital markets ..for.. healthcare...retail etc.

</comment><comment><USER24> _Bloomberg Terminal for everything_

Isn't that Wolfram Alpha?

<comment><USER20> I interpreted this as building a terminal for each of all the various domains out there. Not just 1 Terminal with all the different data you need. That would close to impossible, and impractical to build.

Rather, focus on a specific domain, like startup funding for instance, organize/collect data on that, and allow people to draw correlations and insights from that data.

And to answer your question, does Wolfram Alpha tell me how much funding all food delivery startups had in 2019?

<comment><USER25> Would the ideal form of this look substantially different from crunchbase? I've never used a Bloomberg terminal myself, is this suggesting something about the user experience or just having all the data centralized in one place?

<comment><USER20> It would not. startup funding was an example of a domain where there are products like crunch base that just collect, aggregate and centralize all that data. The user experience is the easy part

</comment></comment></comment></comment><comment><USER26> > South Korea and Japan have developed much more rapidly than many Asian > countries, despite many others adopting relatively free "Washington > Consensus"-style trade policies.

On the contrary, these two countries didn't implement such policies at all; they used the opposite method, strict protectionism and state-managed capitalism.

See "kicking away the ladder" by Ha-Joon Chang.

<comment><USER27> I think that's exactly his point, actually, although I thought that was quite well understood. E.g. "How Asia works" explained the difference between taiwan/japan/s korea on one hand, and philipines/indonesia/malaysia on the other hand exactly around this axis.

Tyler Cowen tends to be one of the few famous economist who do admit that even though not well understood, there is such a thing as "culture" that influences economics and that is important, so I guess that's his angle. Another mystery under that angle is why France managed to modernize so well in XIXth, even though it had bad institutions. See e.g. Rodrik/Cowen talking about this in [https://medium.com/conversations-with- tyler/a-conversation-w...](https://medium.com/conversations-with- tyler/a-conversation-with-dani-rodrik-e02cf8784b9d) (search for "Barrington Moore" section). In How Asia Works, the author argued quite convincingly about the importance of land reform, and how splitting huge landlors domains into small pots of lands was critical. As each farmer would get incentivized to improve their production, skills and market mechanisms develop much more quickly.

<comment><USER28> Chinese emperors have done this type of land reform innumerable times and it has always ended up in famine and disaster/regime change. Land consolidation became a thing (as it inevitably does) and baron landlords became commonplace as the consolidation of fields occurred. The positive aspect of land redistribution via the equal field system is that it alters the power structure for the corrupt landlords who should not have power and prevent progressive changes that benefit society. The negative aspect is it's borderline impossible to vet who is "worthy" of being a landlord. The Chinese unfortunately would let "laissez-faire" take over property control.

Eventually someone who knew how to get a leg up on others would take advantage of others financial incompetence, likely mortgage the property, then acquire it, eventually leading up to owning basically a whole province thus becoming basically creating their own landed title and domain. Then suddenly you've got a feudalistic system that has always existed, but the head government now may even have a worse problem they created themselves. It's happened when the south basically broke off after the final rebellion during the Tang dynasty. These landholders eventually had enough power and cultural difference that they could break from the north and retain full autonomy.

<comment><USER27> Necessary but not sufficient and all that... In how Asia works, Studwell argued quite convincingly how land reforms in Japan, South Korea, etc. was a prelude to successful industrialization because it allowed creating financial surplus thanks to increase agricultural output across a significant part of the population.

And in China, it is well acknowledged that the reform from 1978 were a significant step toward China economic growth: [https://en.wikipedia.org/wiki/China%27s_Rural_Reform](https://en.wikipedia.org/wiki/China%27s_Rural_Reform)

<comment><USER28> But like I said about how these wide sweeping reforms, they create instability and also famine. The reason it ended up being good (as your link also shows) was due to land consolidation. Millions of independent farmers does not create a food surplus as much as a few consolidated landmasses that can maximize production. As I stated, the consolidation inevitably occurs and always have in the many times China has performed land redistribution.

The primary reason the growth occurred was because of the land consolidation. Instead of having millions of independent farmers, they consolidated the land and only required hundreds of thousands of laborers to work on the land to maximize production. Thus allowing for a massive labor force that could be utilized for industrial production.

</comment></comment></comment></comment></comment><comment><USER29> > "American healthcare system"

i would say it is more of a market than a system

<comment><USER30> In what sense? Surely a market involves some sort of comparative choice? Who is even in a position to do that? People get their insurance from their employers, and are treated at the nearest hospital (where the prices are an outright secret beforehand). I can't see how market economics apply even a little bit, as far as the person in need of care is concerned.

<comment><USER31> In my case, you have a choice between PPO and HSA. But i know people who opt to be on their spouses insurance.

<comment><USER32> The running joke in our office is, "If I get married again my spouse is going to work in government."

</comment></comment></comment></comment><comment> type-2 This.. doesn't make much sense actually.

</comment><comment><USER33> The dismal science indeed.

</comment><comment><USER34> The top post on HN, muses about capital allocation, has no mention of climate change. I guess we are doomed.

<comment><USER35> Tyler Cowen's whole gimmick is to play dumb and distract people from the actual issue. Back when unemployment was 10% he wrote about "zero marginal product" people. He literally called people useless. As if the financial crisis was caused by video games and dirty movies on the internet. When pressed on it he admitted he doesn't know much about macroeconomics. Change the subject, then play dumb, that's his only move.

<comment><USER36> While we're running through Tyler's greatest hits, he also argued in favour of high prices for drugs [0], since lower prices would lower innovation.

 _If the advocate of lower drug prices does not have clear quantitative evidence for a conclusion of "lowering drug prices will not harm innovation very much," commit the analysis to the flames, for it harbors nothing but sophistry and illusion._

Tyler Cowen is the opposite of Batman. I genuinely don't understand why he gets so much press.

[0] [https://marginalrevolution.com/marginalrevolution/2015/09/ar...](https://marginalrevolution.com/marginalrevolution/2015/09/are- pharmaceutical-drug-prices-too-high.html)

</comment></comment></comment><comment><USER37> > One of the single interventions that could do the most to improve global > welfare would be to improve the efficiency of the partner/marriage matching > ecosystem

hilarious, are they just throwing this out there or is there anything to back this up?

<comment><USER15> Agreed. This seems like a true 'first world problem'.

<comment><USER38> In what sense?</comment></comment></comment>