SQLite small blob storage: 35% Faster Than the Filesystem - <AUTHOR> https://www.sqlite.org/fasterthanfs.html <comment><USER2> I must point out Jim Gray's paper To Blob or Not To Blob[0]. His team considered NTFS vs. SQL Server, but most rationale applies to any filesystem vs. database decision.

The summary was "The study indicates that if objects are larger than one megabyte on average, NTFS has a clear advantage over SQL Server. If the objects are under 256 kilobytes, the database has a clear advantage. Inside this range, it depends on how write intensive the workload is," but keep in mind this is spinning media from 2006. Modern SSDs change the equation quite a bit, as they are much more friendly to random IO and benefit less from database write-ahead log and buffer pool behavior.

Also, when deciding between blob vs. filesystem, blobs bring transactional and recovery consistency. The DB is self contained, and all blobs are contained in it. A restore of the DB on a different system yields a consistent system, it won't have links to missing files, and there won't be orphaned files left over (files not referenced by records in DB).

Despite all this, my practical experience is that filesystem is better than blobs for things like uploaded content, images, pngs and jps etc. Blobs bring additional overhead, require bigger DB storage (more expensive usually, think AWS RDS) and the increased size cascades in operational overhead (bigger backups, slower restore etc).

[0] [https://www.microsoft.com/en-us/research/publication/to- blob...](https://www.microsoft.com/en-us/research/publication/to-blob-or-not- to-blob-large-object-storage-in-a-database-or-a-filesystem/)

<comment><USER3> My experience (for an application which had a working set of under 1 GB of files in the 50kb to N MB range, and approximately 50 GB persisted at any given time) was that preserving access to the toolchain which operates trivially with files was worth the additional performance overhead of working with the files and, separately, occasionally having to retool things to e.g. not have 10e7 files in a single folder, which is something that Linux has some opinions (none of them good) about.

Trivial example: it's easy to delete any file most recently accessed more than N months ago [+] with a trivial line in cron (the exact line escapes me -- it involves find), but doing that with a database requires that you roll your own access tracking logic. Incremental backups of a directory structure are easy (rsync or tarsnap); incremental backups of a database appeared to me to be highly non-trivial.

[+] Since we could re-generate PDFs or gifs from our source of truth at will (with 5~10 seconds of added latency), we deleted anything not accessed in N months to limit our hard disk usage.

<comment><USER4> ¿Porque no los dos?

Store the files in a database but expose them via FUSE.

<comment><USER5> That gives you the time cost of a filesystem plus the time cost of a database plus an extra round-trip in and out of kernel space.

<comment><USER4> > preserving access to the toolchain which operates trivially with files was > worth the additional performance overhead of working with the files

Sounds like they're OK with that.

In any case, couldn't you avoid the kernel trips with some dylib foolery (assuming the toolchain is dynamically linked)?

<comment><USER6> So: store the files in a database, expose them via FUSE, bypass FUSE and access the database?

<comment><USER4> Yep ;D

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> For web map tiles (millions of tiny PNGs), everyone who's anyone stores their tiles in sqlite rather than on disk: [https://www.mapbox.com/help/an-open- platform/#mbtiles](https://www.mapbox.com/help/an-open-platform/#mbtiles)

<comment><USER8> Interesting and very relevant to the discussion!

Hmm... Does SQLite have some geo data or 2d coordinate lookup capabilities as well?

<comment> shawn-butler Try SpatialLite[0] if you need advanced geometry capabilities. Has R-Tree spatial indexes if you use SQLite 3.6+, and rudimentary MBR index for prior versions. Has rudimentary support for curves as well.

[0]: [https://www.gaia-gis.it/fossil/libspatialite/index](https://www.gaia- gis.it/fossil/libspatialite/index)

<comment><USER9> As a data point, one of our users figured out how to load the SpatiaLite extension for DB Browser for SQLite (on Windows), then wrote up step by step instructions:

[https://github.com/sqlitebrowser/sqlitebrowser/wiki/SpatiaLi...](https://github.com/sqlitebrowser/sqlitebrowser/wiki/SpatiaLite- on-Windows)

(people had been asking previously, but we had no idea. ^\o/^)

</comment></comment></comment></comment><comment><USER10> This is weird benchmarketing. They are comparing reading/writing 100,000 individual files vs writing 100,000 entries into a single file(sqlite database). For comparison one could concatenate the same data into a single big file even faster than into sqlite.

They then do not offer logical analysis as to why things are faster.

My understanding is that reads are probably faster due to operating system readahead being able to predict reads better when they are within a single file. Writes are faster because they do one bulk COMMIT instead of many individual fsyncs.

<comment><USER11> > For comparison one could concatenate the same data into a single big file > even faster than into sqlite.

One could, but then one would have great difficulty retrieving the individual files back when needed.

The point is not about what has the greatest raw speed, the point is that that for applications that read lots of small files from the filesystem, they'll possibly get better performance and almost certainly use less disk space by sticking the files in SQLite instead.

<comment><USER12> > have great difficulty retrieving

Not that great, and the point is its comparing apples to oranges, pointless.

<comment><USER13> Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Using map tiles as the example, how do you easily retrieve just the tiles in a specific region? Ok, how about all tiles that have the "hasLand" attribute set? Or the "containsCoastline" attribute?

You'd end up rewriting your own version of a database.

<comment><USER14> > Writing your own methods to retrieve arbitrary chunks of data from a > monolithic file would be a lot of work.

Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database.

(it really depends on what language and/or library(ies) you use and for what purpose)

<comment><USER11> > Oh goodness have we come a long way if interacting with a file is harder > than interacting with a SQL database.

It's not interacting with a file, it's interacting with a file that contains other files of varying sizes that need to be accessed randomly with good performance.

<comment><USER15> If you only want to read then both zip and tar are good for this purpose. It's commonly used and known to work well. There are many formats that are really just a renamed zip file. Like jar, and epub. File entries don't have to be compressed so you can mmap.

If you want to do random writes then more consideration is needed.

<comment><USER11> > If you only want to read then both zip and tar are good for this purpose

And programmatically accessing them is significantly more complicated than using SQLite which involves dropping a single header in your project and about 10 lines of code.

> If you want to do random writes then more consideration is needed.

Which you almost certainly what you want to do for the sorts of use cases where SQLite is also under consideration.

<comment><USER16> _which involves dropping a single header in your project and about 10 lines of code._

10 lines of user code + the sqlite library. Using other formats like zip is hardly any more user code (+ some external library).

<comment><USER11> The sqlite 'library' is a single header file and a single c file that you can just drop in your project and use.

<comment><USER16> So are many libraries for other formats.

Btw I'm personally a big fan of SQLite, I just think the argument here is a bit of a straw man.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> _> So let your take-away be this: read/write latency for SQLite is competitive with read/write latency of individual files on disk. Often SQLite is faster. Sometimes SQLite is almost as fast. Either way, this article disproves the common assumption that a relational database must be slower than direct filesystem I/O._

This is under the _1.1 Caveats_ heading, which makes me feel the title is _a little_ misleading (but only in the way most benchmark headings probably are, I guess).

Incidentally, can someone more experienced with filesystem and database I/O confirm or contest the assertion here? Specifically, I'm not sure it's fair to generalize these results (even if valid) to categorical relational databases. But this is not a special area of expertise for me.

<comment><USER18> You're asking whether a database can read/write records faster than disk? Yes. The opposite is also true. I'm not an expert in either fs or db, but it's obvious that the design and use of one is not the same as the design and use of the other, and given the right circumstances one will do a particular job better than another.

An append-only transaction log is probably going to perform better on a spinning disk than random writes. An intelligent app sorting and committing writes in order is probably going to perform better than an fs on a disk with no queue ordering facility. As they said in the post, block-aligned writes in bulk are going to be more efficient than spreading out a bunch of writes not block-aligned. And latency for a db write is going to be lower if you don't close an fd, assuming that closing an fd would always trigger an fsync.

But a properly tuned filesystem, with the right filesystem, with the right kernel driver, with the right disk, with the right bus, with the right control plane, etc etc may work fantastically better than an unoptimized database, and a given workload may fit that fs perfectly.

It's important to remember that a benchmark is only meaningful to the person who is running the benchmark. If you want it to be meaningful to you, you have to try it yourself (and then presumably blog about it).

</comment></comment><comment><USER2> From OP: "SQLite is much faster than direct writes to disk on Windows when anti-virus protection is turned on. Since anti-virus software is and should be on by default in Windows, that means that SQLite is generally much faster than direct disk writes on Windows."

I don't get this. If scanning the content is important (as acknowledged by the author), then bypassing the scan via blob storage is a security issue and the application _should_ go through some extra hoops to scan the content before saving it to blob, and this should be measured and part of the comparison.

Also, if the SQLite files are exempt from AV scan, then the level field should also exempt the uploaded files folder in test. I mean, knowing the dice are loaded and then claiming it as an advantage does not seem professional.

<comment><USER19> Data chunks inside another file are effectively quarantined by default. There is no exempting or security issue, it's safe by design. It will get scanned occasionally but not on every write.

<comment><USER20> Except when they are not. Just because malcode is encapsulated doesn't mean it is neutralised. The issue is untrusted data, not how it is stored.

Word Docs with embedded content or can trigger code execution on parsing, data structures that break AV parsers, image or font data that causes system libraries to overflow/UAF/etc; There all could potentially be a security issue in a DB, the same as in a FS.

<comment><USER19> You don't need to worry about a single process writing data to a file and then reading it back and performing malicious actions. That process could just as easily have been exploited entirely in-memory. Scan the database when it exits and you're fine.

</comment></comment></comment></comment><comment><USER21> Interesting. We built a couple of systems many years ago using SQLite as the storage engine for binary files - one was a kind of mini version control utility for storing report files for a legacy ERP system.

I always thought that the SQLite system was far quicker to search and retrieve files from the dataset than it was doing so in our previous version, using hierarchical file folders.

Interestingly, we canned the project because the ERP vendor themselves release a similar tool using - wait for it - individual files in a folder system... (Addendum - much later on they switched to - wait for it - CVS for version control.... in 2010!!)

</comment><comment> Const-me SQLite is good when you're mostly reading.

For writing, the major drawback of SQLite is it doesn't support concurrent writes.

All filesystems do (at least when writing different files), all full-fledged RDBMS-es do, even some embedded databases do (like ESENT). Yet, in SQLite only a single thread can write.

Even embedded chips are multicore these days…

<comment><USER22> Isn't that because SQLLite's whole use case is "simple, small, fast database that competes with fopen() and not Postgres"? At least in my experience on NTFS, unless you know the section that 2 threads are writing to, and that section is different, you can't write to a file concurrently (N.B.: in C#)

I'd be happy to be wrong (seriously that would make my day if someone could explain a clean way to do the above) but if I want to get "big boy database features" then I'd use a heavy database like SQL Server. I think it's great that there's an option like SQLLite out there for rapid prototyping and "store all your music metadata" type applications!

<comment> Const-me > simple, small, fast database that competes with fopen() and not Postgres"

The hardware evolved in a way so even very small systems (phones, embedded, Raspberry Pi and other IoT devices) are now multi core. Multithreading is required to benefit from a multi-core CPU. For IO-heavy tasks, it's sometimes a good idea to implement multi-threaded IO as well.

> you can't write to a file concurrently

I can concurrently write different files. Or to different streams/blobs/records for most other IO methods, except SQLite.

</comment></comment></comment><comment><USER23> Consistently happy with SQLite, just a lovely little database. Performance is basically a cherry on top of the simplicity it provides.

</comment><comment><USER24> "To blob or not to blob" \- a paper by Microsoft Research from 2006. [https://www.microsoft.com/en-us/research/publication/to- blob...](https://www.microsoft.com/en-us/research/publication/to-blob-or-not- to-blob-large-object-storage-in-a-database-or-a-filesystem/)

</comment><comment><USER25> Storing many small files into few (compressed) bundle files is common since at least the 90's for storing game assets, but usually not with SQLite, but some sort of compressed format (simple zip archives are quite common but there are faster alternatives now). The bundling gives you better overall compression, _and_ faster read performance compared to individual files (at least on Windows) as long as you open the file only once, and from then on only seek and read.

SQLite is a _big_ code base, using a simple archive file format instead gives you most of the advantages, but without the bloat.

PS: This is mostly for read-only situations though. Using SQLite probably starts to make sense when the applications needs to write to and create new files in the archive.

<comment><USER2> But that would work only for static content, like the game assets. The discussion of BLOBs vs. filesystem comes up mostly in the context of content management and user/app uploaded content.

<comment><USER25> Yes you are right. I wrote the 'PS' before I saw your comment. Apologies :)

</comment></comment></comment><comment><USER26> Not surprised by the results, very surprised by the explanation given, the overhead of open/close calls.

As far as I know (from many measurements and talking to kernel people and researching the mechanism involved), the difference is due to the fact that buffers are shared between all pages of a single file, and not between files.

So for reads, the filesystem will do read-ahead of significantly more data than requested and keep that in buffers, and future reads will profit. Similar for buffers shared when writing.

The same effect will be reproducible with any format storing multiple objects in a single file, it has virtually nothing to do with "SQLite" or "Databases".

One tradeoff is the greater potential for inconsistencies, which despite all the measures taken is much greater when you modify a file rather than writing only completely new files. Another is the inconvenience and duplication of effort, because all your default file-system management tools aren't available.

It would be interesting to see if a pseudo-filesystem that is mapped to a single underlying file would show the same effects (preferably implemented in user space to avoid overheads of multiple kernel round trips).

<comment><USER27> > The same effect will be reproducible with any format storing multiple > objects in a single file

Given unbounded developer time, any advantage SQLite has here can of course be matched or beaten by custom code. It's just another C library, not magic.

The real issue is, how much work will it take you to do that?

SQLite is billed as competing with `fopen()`. That's the proper way to compare it: given equal development time spent talking to the C runtime library vs. integrating SQLite, how much speed and robustness can you achieve?

> the greater potential for inconsistencies

How? In this application, SQLite is filling a role more like a filesystem than a DBMS or `fopen()` call. You can corrupt a filesystem just as easily as a DBMS. SQLite protects itself in much the same ways that good filesystems do, and it can be defeated in much the same sort of ways.

> Another is the inconvenience and duplication of effort, because all your > default file-system management tools aren't available.

It's a classic tradeoff: do you need the speed this technique buys or not?

A better reason to avoid this technique is when the files you're considering storing as BLOBs need to be served by a web server that uses the `sendfile(2)` system call. There, the additional syscalls caused by the DBMS layer will probably eat up the speed advantage.

Software development is all about tradeoffs. No technique or technology is perfect for everything.

Now you know one more technique. Maybe it will be of some use to you someday.

> a pseudo-filesystem that is mapped to a single underlying file

That pretty much describes SQLite. Both SQLite and a good filesystem use tree- based structures to index data, both have ways to deal with fragmentation, both have ways to ensure consistency, both strive for durability, etc., etc.

> preferably implemented in user space to avoid overheads of multiple kernel > round trips

How are you going to avoid kernel round trips when I/O is involved?

If you think user-space filesystems are fast, go try FUSE.

<comment><USER26> > how much work will it take you to do that?

Dunno, use a zip library?

> [inconsistencies]

No, SQLite doesn't have the same tools available as filesystems. It is very good, but not quite as good.

> [Pseudo FS mapped onto a single file] That pretty much describes SQLite.

Not really. For one, it doesn't provide a filesystem-like interface.

>> avoid overheads of _multiple_ kernel round trips

> How are you going to avoid kernel round trips when I/O is involved?

Added emphasis.

> If you think user-space filesystems are fast, go try FUSE.

Exactly, because you tend to make _multiple_ round trips. If you can flatten that to just one, you avoid those overheads.

</comment></comment></comment><comment><USER28> _SQLite is much faster than direct writes to disk on Windows when anti-virus protection is turned on. Since anti-virus software is and should be on by default in Windows, that means that SQLite is generally much faster than direct disk writes on Windows._

Isn't this because writing to a SQLite database file bypasses most (or all) of the antivirus file scanning since it's can't see a complete file, and can only look at raw blocks of data?

So if you find value in having your antivirus scan all of your files, that's a disadvantage for using SQLite to store them?

</comment><comment><USER29> The reason I still use files rather then SQLite is that I don't know how SQLite handles concurrency.

For example I have a PHP app that is used by 10k users a day and it happily handles 100k tmp files in a single directory. On each request, it checks the file age via filemtime() and if new enough, includes the tmp file with a simple include(). (I write PHP arrays into the tmp files). If too old, it recalculates the data and writes it via fopen(), fputs() and fclose().

This migh be archaic but it has been working fine for years and never gave me any problems.

Somehow I would expect that if I simply replaced it with SQLite, I would run into concurrency problems.

<comment><USER27> > I don't know how SQLite handles concurrency.

Why remain in ignorance? There are several articles in the SQLite documentation that address concurrency:

[https://www.sqlite.org/wal.html](https://www.sqlite.org/wal.html) [https://www.sqlite.org/lockingv3.html](https://www.sqlite.org/lockingv3.html) [https://www.sqlite.org/whentouse.html](https://www.sqlite.org/whentouse.html)

> Somehow I would expect that if I simply replaced it with SQLite, I would run > into concurrency problems.

SQLite wouldn't really be ACID-compliant if multiple threads were enough to defeat the Durability guarantee, would it?

That's not to say that there are no concurrency problems in SQLite, but that they're more in the way of potential bottlenecks than data corruption risks. For a reader-heavy application like yours, I suspect SQLite will perform just as well or better than your existing solution.

If you're solely after speed, I'm not sure it would be worth rewriting your app to use SQLite. This technique's value is simply in the benefit it gives when you were already going to use SQLite for some other reason. If you just want a 35% speed boost, wait a few months or buy a faster SSD. Both are going to be easier and cheaper than rewriting the data storage layer of your application.

That said, maybe there are other things in SQLite that you could use. Easy schema changes, full-text searching, more advanced indexing than the filesystem allows, etc. If you go for one of those, then the extra speed is a nice bonus.

<comment><USER29>
        SQLite wouldn't really be ACID-compliant if
        multiple threads were enough to defeat the
        Durability guarantee, would it?

I'm not so much concerned about durability. More about SQLite not responding to "SELECT v FROM t WHERE id=123" with value v but instead with something like "Error: v is currently being written by another process. Try again later" or something.

No idea if that is a realistic scenario. I'm kind of surprised I don't have this kind of problem with my filebased solution. What happens if process A reads from a file while process B writes it? No clue.

<comment><USER27> > What happens if process A reads from a file while process B writes it?

If process B started first, the writer blocks access to the table being written to, so the reader waits for the writer to complete. There are timeout and retry behaviors, but within those configurable limits, that's what happens.

You can make SQLite behave as you worry about if you set the retries to 0 and timeout to 0, but that's not the default.

</comment></comment></comment></comment><comment><USER30> _35% faster than the same blobs can be read from or written to individual files on disk using fread() or fwrite()._

How about reading efficiently with unbuffered I/O directly on the file descriptor?

</comment><comment><USER31> I know that SQLite performs exceptionally on embedded devices including most phones, IoT devices, and more. However, for better or worse, it's a flat file. Does anyone know of a TCP/IP-speaking SQL database that would work well on an embedded device? PostgreSQL/MariaDB seem kinda heavy, and the net couplers for SQLite look pretty unsupported.

<comment><USER32> and what exactly do you think any database is at the end of the day? It's not magic, it all ends up on the disk as a file. I mean why is a page file called a FILE?

<comment><USER31> Sure, we can agree that (almost) everything's a file. However, through their related programs and daemons, we can ask files for more useful functionality such as indexes, versioning, and yes, socket communications.

</comment></comment></comment><comment><USER33> What about mixed delete/write/read workloads? How well does SQLite deal with fragmentation?

<comment><USER27> > What about mixed delete/write/read workloads?

The source code to the test program is part of the SQLite source tree. It should be fairly easy to modify it to test the scheme you have in mind.

> How well does SQLite deal with fragmentation?

SQLite operates a lot like a modern filesystem: tree-based indexing structures, writes go to empty spaces, deletes leave holes that may later be filled by inserts, etc.

SQLite has active mechanisms to reduce fragmentation on the fly: b-tree page defragmentation ([https://www.sqlite.org/fileformat.html](https://www.sqlite.org/fileformat.html)) and auto-vaccuum ([https://www.sqlite.org/pragma.html#pragma_auto_vacuum](https://www.sqlite.org/pragma.html#pragma_auto_vacuum)) being the main ones. If the DB file gets too fragmented, you can run a manual VACCUM ([https://www.sqlite.org/lang_vacuum.html](https://www.sqlite.org/lang_vacuum.html)) on it.

<comment><USER33> Sounds good. I might give it a test in our product to replace some file based storage scheme.

</comment></comment></comment><comment><USER34> > All filesystems do (at least when writing different files)

The article is about writing 100000 files, which will not scale well (since there will be massive contention at the directory inode).

</comment><comment><USER35> Can we draw a side conclusion from these measurements that open()/close() overhead on Win10 is significantly higher than on Win7? This seems doubtful.

<comment><USER27> See section 2.1 in the report: the Win10 and Win7 installations are on different hardware.

If you have a dual-boot Win10/7 box, it's easy to run the test yourself.

</comment></comment><comment><USER36> So anybody knows where I can get this blazing fast SQLiteFS? ;-)

</comment><comment><USER37> it's basically storing in memory vs storing on a disk

it's not sqlite vs filesystem it's memory storage vs disk storage

<comment><USER38> Are you stating that the SQLite database did not write the files to disk? Or simply that it's caching them in-memory? In the latter case, doesn't an operating system typically offer in-memory file buffering / caching? Otherwise, why would we need O_DIRECT or O_SYNC or fsync()?

<comment><USER37> I'm no expert in FS or sqlite, but as everyone knows, filesystems are not good processing small files, having an application that bundles in memory small blobs and then flush them to disk as a single file, is what make it faster.

ReiserFS 4 was (I believe still is) the only FS that improves small files performance by bundling small files and stored them on disk as a single blob.

as for that person who downvoted my original comment, I'd love to hear from you why you think I'm wrong. you don't just downvote comments because you disagree.

<comment><USER38> I do think your original claim is wrong. Your explanation in this response is not the same as your original claim. And what you wrote here is exactly in line with claims in the article:

"The size of the blobs in the test data affects performance. The filesystem will generally be faster for larger blobs, since the overhead of open() and close() is amortized over more bytes of I/O, whereas the database will be more efficient in both speed and space as the average blob size decreases."

Yes, of course SQLite is going to do some operations in-memory. Raw file writes using fwrite are also going to be initially written in-memory, unless O_DIRECT or some other mechanism is involved. And the article explicitly outlines that they made no effort to bypass file buffering, to the point of not even explicitly flushing to disk.

If both processes are writing the BLOBs to disk, then I don't see how your original claim applies. Writing to disk in a manner that is more efficient for small files is still writing to disk, and so the test is not in-memory vs on- disk as your original claim.

<comment><USER37> sqlite works in memory most of the time, on disk sometime when flushing, faster

filesystem, works on disk most of time, in memory some time (some buffering), slower

which makes my original comment true,

</comment></comment></comment></comment></comment><comment><USER39> This page does not live up to the standards of the other high quality technical articles on sqlite.org. I couldn't even find a reference to the file system used for comparison. But the whole setup is a farce anyway.

<comment><USER40> They tested it on 5 different OSes, which presumably included 3, probably 4 different filesystems (Windows, Mac, and Ubuntu would be using different ones, and I'm guessing the Ubuntu system has a different filesystem than Android too, though I don't actually know), so it doesn't really seem like it matters.

And why is the whole setup a farce anyway?

<comment><USER39> Of course it matters. Different file systems have different performance characteristics. Make sqlite into a file system, and I'm pretty sure you can't beat it by putting another sqlite on top.

<comment><USER40> When the author has tested 3 or 4 different filesystems against SQLite (and at least 2 of them you can be pretty sure as to what they are simply based on the OS, with a good guess at the other 2 as well), and SQLite beats them all, then it really doesn't matter.

As for turning SQLite into a filesystem, that's not really going to work. SQL isn't designed to support things like cheap "file" appends or reading only portions of a value or seeking or anything like that, so you'd end up having to read the entire value for any read, and write a new copy of the entire value for any write, and your performance would be _really really bad_. So yeah, putting SQLite inside of SQLite isn't going to work, because SQLite isn't a filesystem and wasn't designed to behave like one. Not to mention this entire article is about small blob storage, and embedding a SQLite database inside of SQLite isn't a small blob.

<comment><USER39> As for benchmarking multiple file systems with consistent result, point taken. To a degree. Some file systems optimize for directory lookups, and some don't.

As for turning SQLite into a filesystem, we could make it into a file system that would be fast in this particular benchmark. Right?

</comment></comment></comment></comment></comment><comment><USER41> I mean, is SQLite not also using the filesystem? Maybe I'm confused here. Seems like saying using C is 15% faster than ASM. But that includes user error, right?

<comment><USER42> The article is basically entirely an answer to that question.

The third sentence in particular is:

The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, whereas open() and close() are invoked once for each blob when using blobs stored in individual files. It appears that the overhead of calling open() and close() is greater than the overhead of using the database. The size reduction arises from the fact that individual files are padded out to the next multiple of the filesystem block size, whereas the blobs are packed more tightly into an SQLite database.

<comment><USER41> The advice to use a single open/close call instead of multiple is great, but that is advice on how to effectively _use_ the filesystem. It isn't in anyway, shape, or form beating the filesystem.

<comment><USER38> You can't use a single open/close call to write multiple BLOBs, unless you are concatenating them into one file. That's not typically what happens nor what people are advocating when they recommend not using database BLOBs and using the filesystem instead.

</comment></comment></comment></comment><comment><USER43> If I asserted: "It's faster to put 10,000 rows of csv data in single file instead of 10,000 individual files" even the most junior programmer would likely say "Well, duh, it's 1 file instead of 10,000".

Yet this benchmark is at the top of HN for some reason.

<comment><USER44> So you never think that using database to store file will be slower than using file to store file?

"Duh, I always knew that" is easy to say when you don't have to provide proof.

Also, are you sure that CSV will be faster as well? Do you have a benchmark for it?

<comment><USER39> "Measure before optimizing" has limited application. There are many things that need not be measured. Appending to a file is definitely not slower than doing memory management inside the file to allocate a new chunk, seeking to that position and then writing out the chunk. Period. (And the serialization overhead is negligible compared to the disk I/O).

<comment><USER44> It's not only "writing" though. It's "writing" _AND_ "reading".

Can you read the correct line in in CSV faster? Can you find the row with specific critiria like SQLite faster? Can you handle concurrency correctly like SQLite? Can you make sure your CSV is always in consistent state like SQLite?

The point of the article is that SQLite is still fast _even with all the benefit of database_.

The idea of SQLite is that you could default to using it, and move away when it hits its limit. Nobody should default to using CSV, you use CSV when you have to, despite all its limitation.

<comment><USER39> No. Read parent again.

</comment></comment></comment></comment></comment><comment><USER45> During my first week at American Express I did not have the credentials to install any applications. My manager wanted me to build a prototype and I couldn't install a database. Firebase and other third party was impossible because I had to work on an internal server. I thought SQLite would save my day, but I remembered it being really hard to install and set up. So I had to write my own DB that I called 'stupid-db'. It literally just reads/writes data into a file: [https://github.com/songz/stupid- db](https://github.com/songz/stupid-db)

The demo went well.

<comment><USER46> ???

SQLite is a library, not a database server. There's no install step - it's a single .C file (and a header, if you're into that). Most of the major scripting languages have rock-solid bindings that don't require any additional system libraries or software installs.

Python (in the stdlib!): [https://docs.python.org/2/library/sqlite3.html](https://docs.python.org/2/library/sqlite3.html)

Nodejs (prebuilt): [https://www.npmjs.com/search?q=sqlite](https://www.npmjs.com/search?q=sqlite)

Ruby (needs libs, which are preinstalled on macOS): [https://rubygems.org/search?utf8=&query=sqlite3](https://rubygems.org/search?utf8=&query=sqlite3)

<comment><USER45> you are right. this was also many years ago so my memory might fail me. Perhaps my internal server didn't have access to npmjs.</comment></comment></comment>


Engineering Team For Sale. Again. - <AUTHOR> http://acquihire.me

<comment> jack-r-abbit Interesting idea. Best of luck.

But in a world where sometimes even the smallest of details can make the difference, I'd like to point out that your front page has a right-rail blurb with a button inviting me to "Learn more"... which just links to your front page. While that did cause me to learn more (that you have a "bad" link)... I'm pretty sure that is not what you wanted me to learn.

</comment><comment><USER2> Nice idea, but I'm not sure "Our startup didn't work out" and "Proven track record" really go so well together...

<comment><AUTHOR> Startups don't end up working out for a great many reasons... this is a solid tech team.

</comment></comment><comment><USER3> I wish these folks all the best. Just one quick comment. My first question is what else have you built that I can see and play with? As with most things, show rather than tell.

Happy to interview your team on my podcast to get you some more exposure if you'd like.

</comment><comment><USER4> Why not just start a consultancy firm?

</comment><comment><USER5> Heh - Interesting way to market yourselves, I like it! Not sure if you've had success "selling" yourself as a package deal before, but I would expect it to be hard for most companies to afford bringing on a team of 6 all at once.

Best of luck!

<comment><AUTHOR> Thanks very much :-)

Yes, I helped another team in a similar fashion just a few weeks back, and it worked out well for everyone.

</comment></comment><comment><USER6> I wish best of luck for them. My experience after working for couple of startups taught me to keep working on what you love until the last minute. With perseverance, things will work out eventually.

</comment><comment><USER7> Some good design could have probably helped their failed startup, and the value proposition for this acquihire site.

<comment><AUTHOR> :-)

The value-prop on acquihire is purely the talent.

</comment></comment><comment><USER8> I am conflicted between admiring their forthrightness, and the feeling that this is an early sign of the Apocalypse.</comment>


Tashu Duset Sekar: Invented Languages and Aptitude Tests - <AUTHOR> http://www.lrb.co.uk/blog/2015/09/03/helen-dewitt/tashu-duset-sekar <comment><AUTHOR> This article was brought up on the CONLANG-L mailing list a couple of days ago, with the suggestion that the LCS (Language Creation Society) help hook up people who want to administer or take these kinds of tests with conlangers who are capable of constructing them.

If you're interested in invented language puzzles, you might also be interested to know that there are numerous, reasonably large online communities for fans and creators of artificial languages. The LCS maintains a list at [http://conlang.org/communities/](http://conlang.org/communities/)</comment>


Ask HN: How plausible is Bitcoin to be a Intelligence Agency operation? - <AUTHOR> Just think about it: with all the secrecy and complexity around it's origin, I wont be surprised one bit, if the whole thing is in fact a Government Intelligence Agency operation in order to combat terrorist (and criminal in some cases) financing with the illusion of border/government free and untraceable currency (which is a quite a contradiction to the block chain technology powering the whole thing)... <comment><USER2> not very likely at all...although FBIare meant to own quite a substantial amount of bitcoin from raids and seizures etc. ([https://www.wired.com/2013/12/fbi_wallet/](https://www.wired.com/2013/12/fbi_wallet/))

</comment><comment><USER3> Approximately zero.

<comment><AUTHOR> [http://www.zerohedge.com/news/2017-06-12/exposed-real- creato...](http://www.zerohedge.com/news/2017-06-12/exposed-real-creator- bitcoin-likely-nsa-one-world-currency)</comment></comment>


Avoiding Retpolines with Static Calls - <AUTHOR> https://lwn.net/SubscriberLink/815908/8fe01199eee0f77f/ <comment><USER2> > Changing the target of a static call requires patching the code of the > running kernel, which is an expensive operation.

Am I misunderstanding, or is this literally patching code pages at runtime? This sounds like it could be dangerous…

<comment><USER3> Seems like it could be done safely...

Remove X bit and add write bit.

Patch.

Remove write bit and add X bit.

Tricky part: If you get a protection fault trying to jump there, spin until the X bit is re-added.

So yeah. That is going to be a hefty performance cost to write.

<comment><USER4> Seems safer to me to park CPUs in known safe spots where they can spin until modification is complete. CPUs could check if there's a pending patch when they attempt to reschedule, and when all CPUs apart from the patching CPU are safe, then the patch can go ahead.

Trying to retry an operation (so it's transparent from the user's perspective) or reset state to the way it was at the time of the protection fault sounds harder to my mind.

PS: I looked at the ftrace doc linked above, and it's architecture dependent of course, but for ftrace's purposes, they use generally use breakpoints on the place being modified.</comment></comment></comment>


Apple makes all WWDC 2010 videos available for developers - <AUTHOR> http://developer.apple.com/itunes/

<comment><USER2> Kudos for making stuff accessible for free. Shame for requiring an Apple ID to just get to the Site and for not making it clear which type of account (free? paying?) you should create to have access to that content.

Contrast that with Microsoft that have been making videos of events available online for free without requiring an account of any sort for years... Most recent stuff: Tech.ed 2010: <http://www.msteched.com/> Mix 2010: <http://live.visitmix.com/> And the ever growing Channel9: <http://channel9.msdn.com/>

</comment><comment><USER3> If it wasnt for Google I/O completely being released on Youtube and Microsoft doing the same thing, Apple would still charge money for this.

I am writing this on my MBP, but I hate the fact that Apple always requires a competitor chasing them to be 'not that evil'.

</comment><comment><USER4> This is awesome, I've never ponied up the cash to listen to the WWDC lectures before, and I'm glad that Apple is following Microsoft and Google's way of releasing videos free after the event

<comment><USER5> Doesn't look free to me. Microsoft and Google don't require you to pay 99$ to watch their conference videos.

<comment><USER6> There is a free account. It's really hard to find. Here is the direct link: <http://developer.apple.com/programs/register/>

(– edit: argh! This is no sarcasm! It _is_ really hard to find.)

</comment></comment></comment><comment><USER7> Very nice. I remember them charging $500 for this last year. Does the WWDC NDA apply to these videos?

<comment><USER8> From iTunes: "The content presented within the WWDC 2010 session videos and slides in Apple Confidential Information and is subject to the Registered Apple Developer Agreement.".

So, yes.

<comment><USER7> Hmm, creating a new developer account is like a 15-second process; I don't see any way they could possible enforce the NDA.

</comment></comment></comment><comment><USER9> more discussion here: <http://news.ycombinator.com/item?id=1439949></comment>


Designing a safer intersection with geometry (2013) - <AUTHOR> http://www.theatlanticcities.com/commute/2013/01/could-these-crazy-intersections-make-us-safer/4467/ <comment><USER2> It's going to be really interesting to see what intersections look like once we no longer allow human drivers behind the wheel. We can only hope this will happen during our lifetime.

<comment><USER3> There's a link to that in the article.

[http://www.theatlanticcities.com/technology/2012/03/what- int...](http://www.theatlanticcities.com/technology/2012/03/what- intersections-would-look-world-driverless-cars/1377/)

</comment></comment><comment><USER4> "And we all know how that's gone in America with the simple roundabout."

Roundabouts are not unpopular, and exist in cities like DC with no one complaining. The reason why roundabouts are unpopular is because city meddlers often put them in places where their purpose is not to mitigate accidents but to attempt to limit speeding through side streets, with negligible (if not negative) effect on traffic safety.

<comment><USER5> Some terminology about circular traffic features:

Big things that you can travel quickly into, which are most commonly found on the U.S. East Coast (e.g. Washington, D.C.) are called rotaries, and are generally considered to not be particularly good or safe.

Mid-sized things with splitter islands preceding the entry, and that have an angle of entry that prohibits entering at speed are called roundabouts, and generally are considered a good thing from a speed / efficiency perspective.

Small things with no splitter islands preceding, that are usually put on quiet residential streets (common in Seattle among other places), are called traffic circles. Confusingly, rotaries are sometimes called traffic circles too (see Dupont Circle).

I disagree with your assessment about traffic circles being placed by city meddlers, as well as the assertion that they don't have a positive effect on safety. First, in Seattle and elsewhere, they are typically requested by residents of the street (not us meddling urban planners), and are built where a speeding problem is confirmed, as resources become available, and according to some geographic equity criteria. Second, though I haven't done an exhaustive review of traffic circle literature, this summary review article is bullish on traffic calming features generally, and specifically on traffic calming circles as well: [http://jpl.sagepub.com/content/23/4/347.short](http://jpl.sagepub.com/content/23/4/347.short)

<comment><USER4> Thanks for the terminology clarification. I grew up in the DC area, so to me a 'traffic circle' is all of the above, but thinking of places like dupont circle as the archetype of the 'working' circle.

I didn't say urban planners, I said city meddlers, i.e. select residents of the street. And all the complaints I have ever heard about the (small) traffic circles have been from the residents themselves. There could be a sample bias: I'm a lyft driver; the complaints come from passengers getting dropped off at their homes, usually from nights of drinking.

<comment><USER6> You are hearing complaints from the troublesome speeders and drunk drivers. Thank you for Lyfting them out of there own cars!

</comment></comment></comment></comment><comment><USER7> This reminds me of the great transformation of Drum Hill Rotary into Drum Hill Square: [https://email@example.com,-...](https://email@example.com,-71.3666283,17z/data=!4m2!3m1!1s0x89e3a35cf6c32ab7:0xfc19e146f5ee894e) The resulting design is ridiculously confusing and unintuitive. The first time I was driving to it I thought: "WTF were they thinking? How do I get into the right lane?" Then it changed to "Woah, I got lucky, I'm sure I will get confused the next time" and then "OK, somehow I keep getting it right" and "Wait, I am no longer stuck in traffic here?" "Not even at 5pm?" "...woah?"

Yes, designing the traffic lanes works.

</comment><comment><USER8> It's interesting to note that much of the gain in automobile safety over the past 50 years has been offset by an increase in miles driven due to increased sprawl [1].

It seems like this sort of intersection, while possibly being safer, would encourage even more sprawl due to its scale, and pedestrian-unfriendliness.

[1] [http://www.planetizen.com/node/68200](http://www.planetizen.com/node/68200)

</comment><comment><USER9> I also note the the "lights" in the simulation are cycling _WAY_ faster than I have ever seen a traffic light cycle in the US.

A whole lot of intersections in the US would likely improve with that simple "innovation".

People are less likely to crash a yellow-to-red light when they know that their turn is coming again in 30 seconds. Make that 5 minutes and streams of people are going to crash that light.

<comment><USER10> That sounds (to me) like an excellent idea, as long as the intersection doesn't have any pedestrian crossings.

</comment></comment><comment><USER11> I used to live near the Mixing Bowl [1] (not exactly an inspiration in city planning), so I just wonder how these would hold up under congestion. The mixing bowl proves you can do anything with ramps (& no lights!), but building flat intersections seems much harder.

Building freeway ramps isn't really that hard, though, so I don't see the point in restricting dimensions. It's good not to let it get out of hand like VDOT, but concrete is pretty cheap. Maybe if you've got a lot of 45mph roads connecting you'd want good lighted designs. Where I live now it's all 40s connecting to 55s connecting to the interstate, so it'd be pretty moot.

Planning for road speeds & traffic rates seem far more important than keeping things flat.

[1] [http://en.wikipedia.org/wiki/Mixing_Bowl_(Springfield)](http://en.wikipedia.org/wiki/Mixing_Bowl_\(Springfield\))

</comment><comment><USER12> I used to live near a road that was designed similar to the "Continuous flow" model. It wasn't quite that elaborate, but Palatine Rd. just north of Chicago has "local" and "express" lanes. (Or was it Dundee Rd.? I forget.) I have no doubt it flows more efficiently than other roads, but I consistently had people drive into the wrong lane directly at me. I've never had that on other streets. So the question is - do these trade efficiency for basic safety?

</comment><comment><USER9> That "diverging diamond" would terrify me.

Seeing a car moving at high speed on the right in a parallel trajectory is a deeply ingrained alarm bell that I'm in the wrong lane.

US Interstate 35 in Austin at 51st Street used to (may still have) have an overpass lane like this and it took a _LONG_ time to get used to without it causing "Danger Will Robinson" reactions.

</comment><comment><USER13> _Continuous Flow_ model reminds me of divide and conquer sorting algorithms.

Most elaborate roundabout( with traffic lights ):

[http://i193.photobucket.com/albums/z202/keber1/razno/tomaevo...](http://i193.photobucket.com/albums/z202/keber1/razno/tomaevo3-pzr.jpg)

<comment><USER14> That is far from the most elaborate roundabout. In the UK they have one that's fractal:

[http://en.wikipedia.org/wiki/Magic_Roundabout_(Swindon)](http://en.wikipedia.org/wiki/Magic_Roundabout_\(Swindon\))

</comment></comment><comment><USER6> In Continuous Flow, how do you ever get _in_ to the Left turn lane?! The video crops that part out, as if it is a prank design.

</comment><comment><USER4> amazing. I've always fantasized a "diverging diamond" for the I-5/Genessee exit in San Diego. There are space constraints (the freeway is in a canyon) that prevent more traditional designs, and the exit is a nightmare, causes atrocious traffic snarls.

I had no idea that someone had actually made one.</comment>


How To Recover the 97% Of Carts Abandoned In Mobile Shopping Apps - <AUTHOR> http://www.forbes.com/sites/ayoomojola/2013/09/05/how-to-recover-the-97-of-carts-abandoned-in-mobile-shopping-apps/

<comment><USER2> So will the % of abandoned carts _increase_ when people discover they can earn extra perks by always dropping the order?

<comment><USER3> Agreed, giving a discount or other reward for recovering an abandoned cart is a VERY dangerous road to go down IMO.

<comment><USER4> I am not sure I agree. Many major online retailers do this and are very successful. Promotions can be site-wide or category sales, individual items, certain brands, offering mail-in-rebates, etc.

Done right, it is wildly successful.

</comment></comment></comment><comment><USER5> Always nice to see a fresh term for "spamming your potential customers".

</comment><comment><USER6> Some sites force you to add an item to the cart in order to see the price -- I can understand how 97% are abandoned if these are included.

<comment><AUTHOR> Agreed - bad practice. Not advocating that in the least.

</comment></comment><comment><USER7> I'm sorry for such an awful (but accurate) pun, but this seems like a bit of a _pushy_ sales tactic. I personally run away from businesses that use tactics such as these.

Please, don't do what this article suggests.

</comment><comment><USER8> AKA How to get users to deny your application's request to send push notifications.

<comment><USER9> I installed the Gilt app mentioned in the article. Push notifications and emails almost everyday.

This is exactly how you get me to deny the request for push notifications.

This will be true until the end of time: be useful to your customer, not spammy.

<comment><AUTHOR> Understood - for flash sales though, a large portion of the demand is driven by the fact that the items are only available in limited quantities. As a result these pushes generate huge spikes of traffic - it's spammy if you don't care that the selection quickly shrinks, but if you're a compulsive shopper (true for a large swath of the shopping population) then you depend on it.

</comment></comment></comment><comment><USER10> Wait, people actually install apps to shop? Every time I've used a store app I've found it inferior to their mobile website which was, in turn, inferior to their full website.

<comment><AUTHOR> Agreed. The apps are getting better though (and have to - a lot of businesses are seeing 30 - 50% of usage/revenue coming via mobile). It's not a preference for businesses because they've spent a decade investing in the web and now mobile is coming along, but its driven by end users. If you don't adapt, your competitors will, and they'll eat your lunch as a result.

</comment></comment><comment><USER11> Oh HELL NO. If you push a notification to my device after I walk out of the sales process I am uninstalling your app immediately and rating you 1 star, you dirty fucker.

<comment><USER4> I believe you're missing the point. This is not another way to get spam, but to save some money on something you're interested in but did not yet purchase it.

</comment></comment><comment><USER12> "For recovering abandoned carts, many ecommerce sites use email. Apps, however, are just getting started. With half of emails now opened on a mobile device, email is much less effective (unless you use deep links)."

Can someone explain what the author means here? It seems that if you are using an app on your mobile device to shop, and emails sent to you are likely to be opened on that same device, then email would be a _good_ way to notify you.

<comment><USER13> The "unless you use deep links" parenthetical kills the entire article. Not providing a call to action that _allows your customer to make the purchase_ is just bothering the user to no effect!

Are people seriously sending out emails that say "This dress you might like is available in your size. Please open our app, find our search feature, remember the name of the dress we were telling you about, input your size that we apparently already know, add it to your cart, and checkout"?

<comment><AUTHOR> > The "unless you use deep links" parenthetical kills the entire article. Not > providing a call to action that allows your customer to make the purchase is > just bothering the user to no effect!

Not exactly - deeplinks done well take you right to the check out page, where all you have to do is confirm the purchase. Most push notifications for the moment open you into a random page into the app, which, I agree, doesn't really add value.

</comment></comment></comment><comment><USER14> How about coding sites such that folks don't need to 'add to cart' in order to find out what they'd actually be paying if they were to purchase? That's most often why I 'abandon' carts. They make me start a cart in order to see the bottom line.</comment>


A broken Windows 10 mail app ruined this guy's job search - <AUTHOR> https://www.reddit.com/r/tifu/comments/515qgc/tifu_by_applying_for_jobs_with_the_windows_10/ <comment><USER2> {Not suing Microsoft, guys. First, this is almost certainly my fault (I AM on TIFU, not techsupport). Second, I am English. We don't sue. We complain in the pettiest medium available to us.

LOL</comment>


UK police deny responsibility for poster urging to report kids using Kali Linux - <AUTHOR> https://www.zdnet.com/article/uk-police-distance-themselves-from-poster-warning-parents-to-report-kids-for-using-kali-linux/ <comment><USER2> if you read to the end of the article, it's been update to say that one of the police agencies with a logo on the poster IS actually admitting to producing the poster! And these are not rural bobbies but in theory an "elite" specialist force. So when you snitch on your child, these clowns will be the "experts" conducting the investigation. You have to hope that the kids caught in this dragnet encounter someone at the CPA who knows what a VM actually is otherwise they'll be taking a plea bargain which gives them a criminal record and prevents them ever using their technical skills for a productive job.

</comment><comment><USER3> Interesting that this isn't getting any tracking on HN. Too much of such nonsense from the UK already?

It's truly sad to see parents being encouraged to involve the police instead of informing themselves what their children, family members, are doing. This just shows by what kind of dystopian party this warning notice has been created.

<comment><USER4> [https://news.ycombinator.com/item?id=22320016](https://news.ycombinator.com/item?id=22320016) 371 points, 145 comments, was on the front page quite a bit

<comment><USER3> Ah, thanks! It seems I should've tried a few more search terms.</comment></comment></comment>


Show HN: Lookup anyone's email in seconds - <AUTHOR> I wanted to get some feedback about this chrome ext I made for gmail that looks up email addresses by a person's name: https://teragod.com/ please let me know your thoughts. <comment><USER2> Daniel,

It's hard to justify $100 for 100 searches when hunter.io gives you 100 for free.

Also, I can't really get it to work...

<comment><AUTHOR> The onboarding has been improved in the latest version. You should get a Teragod icon in your Gmail once you install the extension after that it'll show you a pop to create an account. Then you can use the search functionality to first search for a company and then someone who works there in order to find their email.

Please let me know if this works for you.</comment></comment>


Ask HN: What's the deal with executives having a "not" personal office? - <AUTHOR> I've noticed at two larger companies, that I worked at, with an open office plan the each executive would get a conference room just for the executive. They would decorate it, add a couch, remove the large table, and install a mini fridge while still having a desk on the floor, but never spend time there.<p>Why don't they call it their office instead of a conference room? We can all see it. <comment><USER2> The idea of there being separate spaces of all kinds for executives is pretty common in larger companies. How how and where they spend their time in those spaces is usually just up to them.

One company I worked at was acquired and the whole executive 4th floor was empty. Only a handful of people knew that floor had been left unlocked. For a good year it was a glorious place to take a lunch break and use the bathroom.</comment>


Stephen Wolfram Testifying at the Senate about A.I - <AUTHOR> https://writings.stephenwolfram.com/2019/06/testifying-at-the-senate-about-a-i-selected-content-on-the-internet/ <comment><USER2> Excellent! Besides maybe Geoffrey Hinton, I can't think of a better person to represent A.I. to laymen.

Say what you will about Wolfram and his personality, but the man is an icon of approachable and engaging science/math writing.

Heck, once I started reading _A New Kind of Science_ I could barely put it down -- I carried the damn thing on the subway even! Much better than Jane Austen anyway...</comment>


Show HN: Connecting Indiecontractors to Companies - <AUTHOR> https://indiecontractors.com/ <comment><AUTHOR> Indiecontractors is a website where contractors can find contract work to pay the bills, so that they can continue working on their side project until it becomes ramen profitable.

I've had this idea for some time now. First from shl (Gumroad) who tweeted that he likes to hire contractors who are supplementing their income while working on a side business. Then I heard Baird Hall's IH interview who also said he liked to hire indiehackers because they were reliable and competent.

In this economic downturn, there are many indiehackers who need to supplement their income. Please share with them!</comment>


Show HN: Kikagram - <AUTHOR> http://kikagram.rocks

<comment><AUTHOR> For those unfamiliar with Kik, Kik is a social messaging app similar to Whatsapp or Snapchat, but uses usernames instead of phone numbers.

The site/web app was designed for Kik, although will still work if you're using a browser, and prompt you to install Kik for the sharing functions.

NOTE: If you want to see it in Kik, Kik still doesn't detect the .rocks TLD in the URL sidebar in kik (but can access it just fine if someone else sends you it), please use kikagram.pw

</comment><comment><USER2> Pretty nice!</comment>


"Map Maker will be temporarily unavailable for editing starting May 12" - <AUTHOR> https://productforums.google.com/forum/#!category-topic/map-maker/news-and-announcements/crFEbGXJ-HI

<comment><USER2> What was the one "large scale prank" they mention that prompted this? I've seen a few HN posts with inappropriate map edits both funny and not, but I didn't think any of them qualified as "large scale".

<comment><USER3> A few weeks ago someone edited an area in Pakistan to show the Android logo robot urinating on an apple [1,2].

[1] [https://news.ycombinator.com/item?id=9432504](https://news.ycombinator.com/item?id=9432504) [2] [http://www.theregister.co.uk/2015/04/24/android_google_maps_...](http://www.theregister.co.uk/2015/04/24/android_google_maps_urinate_apple/)

</comment></comment><comment><USER4> Is it really necessary to call everything a gate? This is silly and boring.

</comment><comment><USER5> This would be a great time to change the HN title to match the link.

</comment><comment><USER6> This reeks of bad product management. Surely a good pm and eng team could come up with a more sensible solution than shutting all edits down. This is clearly a second rate product inside Google. Probably managed out of a remote office w bad oversight.

<comment><USER7> > Surely a good pm and eng team could come up with a more sensible solution > than shutting all edits down

Really? I don't know of a single company, person or team that has ever solved targeted spam. "Surely" is doing an awful lot of work there.</comment></comment>


How to Tell a Journalist from a Blogger - <AUTHOR> http://jolieodell.wordpress.com/2010/07/21/how-to-tell-a-journalist-from-a-blogger/ <comment><USER2> _"9. A journalist isn't a spy or a snitch. It's true that some of the wildest, most dramatic stories in the annals of this trade have revolved around the divulging of secrets. Watergate, for example — that required some first-class leaking and espionage. But the workaday journalist gets maybe a few of those stories in his lifetime. Journalism is not an exciting merry-go-round of overheard deals and eavesdropping in antechambers — or, to put a more modern spin on it, hacked accounts and leaked documents. If it were, journalists would be universally mistrusted and would never get invited to any parties, which make up a significant portion of our food-and-drink budgets as journalists' salaries are generally low. This is where bloggers have fucked over journalists more colossally than I can comfortably express. A couple bloggers posing as journalists spied, snitched — and did so in a way that benefited almost no one except the bloggers themselves — and now all producers of media are painted as untrustworthy vultures. The true journalist relies on deep knowledge of his beat, close relationships with industry experts, and dedication to his craft. He has the kind of skill that makes for a 20-year career in reporting, not the kind of childish sneakiness that makes for a one- time pageview blockbuster."_

I couldn't disagree more. Journalists are supposed to be hated. The fact that journalists want to be liked and invited to parties is how we get the country's most prominent "journalists" playing with squirt guns at Biden's house, buying into lies that lead to wars, and ignoring torture. What objectivity can we expect to see when the press merely wants to be close to the powerful. The press needs to be adversarial. If you aren't an "untrustworthy vulture", then you're not all that interested in sharing the truth. If you trade access for stories, you're a pretty poor excuse for a journalist (no matter how much you need the free meal).

 _"1. A journalist is trained in journalism. Whether in the hallowed halls of higher learning or in the less-hallowed halls of a professional newsroom, the journalist has been trained as such. The journalist's work has been pruned mercilessly by the red pens of professors, peers, and editors. A bachelor's degree in journalism, media studies, mass communication, or some similarly named program along with at least a few years under the tutelage of editors is the best preparation for calling oneself a journalist. A blogger might have a ton of general writing experience and even a degree in English or something along those lines, but — and this is a critical distinction — a writer per se is not a journalist. Not any more than a keyboardist is a concert pianist or a mechanic is a nuclear submarine technician. A journalist belongs to a specialized, technical subset of the writing professions that requires specific training. As one who has edited many a writer who attempts journalism, I can tell you the differences are vast — not simply niceties and nuances."_

Again I have to disagree. I say that professional journalism training does more harm than good. We have armies of people out there who know "how to write a story" and almost no one who knows anything at all about what they're supposed to be writing about. I'll take a historian writing about modern politics over a person who knows how to write for a large audience. I'll take a startup founder writing about business over someone who merely was assigned a story. I just prefer a scientist writing about science (That's why we have so many poorly researched "Cure for AIDS found" stories). Without knowledge and the ability to deliver context, you're just a stenographer. And I think that's why many journalists tend to be pretty poor at doing their jobs.

 _"4. A journalist attributes quotations and cites sources. One of the first lessons you learn in J-school is that "common knowledge" doesn't count as a source, and everything must have a source. Did it rain 5 inches yesterday? According to whom? Was the city budget cut? According to which documents? At what meeting? By which persons? Is a certain chemical bad for the environment? What experts say so, and what studies prove it? In keeping with the standards of objectivity, no fact can make it into print without having a firm attribution to some source outside the newsroom. Attribution along with objectivity are almost inviolable commandments, and the professional journalist is hard-pressed to cross them. Attribution in the digital age amounts to linking back to the source when a digital source is available. The blogger, on the other hand, can play fast and loose with "everybody knows" logic and refer to the omniscient "They" as a source of statistics or other knowledge. And linking back is seen as optional, since many bloggers would prefer to claim information as their own and silo pageviews and PageRank on their own domain."_

We have absolutely rampant anonymous citations from people we'd consider journalists today. We have sources close to an administration, off-the-record conversations, and lax citations. It's why even after prominent scientists said the BP oil spill was leaking upwards of 50,000 barrels of oil every day, prominent news outlets were still reporting company line of 5,000.

I could really go on all day but there's so much wrong with this.

There are real journalists out there. There are people that do really thankless work bringing important stories to light. Then there are people that get degrees in "Media Studies" and like going to parties with powerful people.

<comment><USER3> Dude, the "going to parties" part was, I thought quite clearly, a joke.

You can't write about information you can't get. You can't get information by being a dick/informant all the time. It's a delicate ethical balance, but it's a lot more nuanced than gadfly versus vulture.

<comment><USER2> We've seen several very high profile cases, specifically David Brooks of the New York Times, who wrote an article after Michael Hastings broke the General McChrystal story for Rolling Stone stating:

 _The most interesting part of my job is that I get to observe powerful people at close quarters. Most people in government, I find, are there because they sincerely want to do good. But they're also exhausted and frustrated much of the time. And at these moments they can't help letting you know that things would be much better if only there weren't so many morons all around._

[http://www.nytimes.com/2010/06/25/opinion/25brooks.html?_r=1...](http://www.nytimes.com/2010/06/25/opinion/25brooks.html?_r=1&ref=davidbrooks)

Then we have this example of the Biden Beach Party for journalists where people like Wolf Blitzer, David Sanger, Marc Ambinder, and Ed Henry get to play with waterguns with the White House Chief of Staff:

[http://link.brightcove.com/services/player/bcpid42950271001?...](http://link.brightcove.com/services/player/bcpid42950271001?bctid=90387058001)

I agree that you can't be a jerk all the time and break big stories. But you've got to acknowledge that at the moment, journalism isn't suffering because it's too adversarial. It's suffering because it's too deferential to powerful interests. Journalists like David Brooks enjoy going to cocktail parties. The most interesting part of his job is observing powerful people at close quarters - not reporting interesting stories.

As for your work-a-day journalist covering flower shows and local sports games, I think we can agree that bad might be good enough. But if you think about it, you don't need a real "journalist" at all to cover these stories. A lower standard "blogger" could probably do just fine.

As for covering a disaster, I would think that there's a huge world of difference. Disaster relief can be covered like the local flower show or it can be done right and it can be followed up on for weeks to make sure that it is being handled correctly by those in power, that corruption isn't rampant, and that those in need are getting the services they require (the aftermath of Hurricane Katrina comes to mind). No one has suggested that we move towards screaming heads and sensationalism. Research and fact-checking are indeed important.

</comment></comment></comment><comment><USER4> She lost me at

       1. A journalist is trained in journalism

I'm a designer and I'm trained in design, but I would never define a designer as someone who is "trained in design".

There are many ways to learn, and various paths to success. Traditional education is definitely not the only way. I know designers who aren't formally trained who are better designers than I ever will be.

I'm pretty sure the same thing can be said about journalists. We've all seen amateurs blow formally trained journalists out of the water, especially when reporting on highly specialized topics.

Besides, her only motivation for that statement is "a journalist requires specific training."

[edit: got gender of author wrong]

<comment><USER5> If you had of kept reading you would have seen them say that it doesn't matter if they were trained in higher education or in the newsroom as long as they were trained.

<comment><USER6> And then if you read a little more you'd see that this blogger hadn't structured it very well and seemingly contradicts one para with the following one:

 _A bachelor's degree in journalism, media studies, mass communication, or some similarly named program along with at least a few years under the tutelage of editors is the best preparation for calling oneself a journalist._

Suggesting that one needs a BA along with years of tutelage.

In any case I'd like to retitle the piece "No true journalist". She sounds like a cabinet maker bemoaning the Ikea-isation of furniture.

<comment><USER5> Best case not only case. Having the formal education then adding a few years experience. The alternative case she listed was just the experience in the newsroom.

<comment><USER6> Best case to prepare to call oneself a journalist sounds like it would be essential to actually doing it, rather than being preferential in preparing to do it. But yeah.

</comment></comment></comment></comment></comment><comment><USER7> A lot of newsroom editors and journalists would be better off if they read this article. It explains how journalism _should_ be, but unfortunately not how it is. There are exceptions of course - the Economist springs to mind.

Unfortunately a lot of journalists don't live by the points set forth in te article. _"A journalist is obsessed with the Truth."_ and _"A journalist is a skeptic (and often a critic)."_ for instance are a joke to anyone who's seen their press releases copied into an article more or less word by word.

I know a few journalists, and they do indeed strive towards high standards. The problem is that since most news corporations, as we all know, aren't doing very well they're under pressure to churn out articles at an ever higher rate. And the journalism suffers. There is just no way they can churn out five well researched, skeptic, truthful articles every day.

Unfortunately for news organisations this is a bad spiral: Worse reporting leads to fewer readers which lead to cutbacks which lead to worse reporting.

<comment><USER8> I up-voted this because I didn't see why someone down-voted you (to zero).

</comment></comment><comment><USER9> Is this article a troll or does it just have troll-like powers over me? Journalists care more about the truth than bloggers? A journalist serves the people? What planet is this person living on? The journalists whose work I have occasion to observe, with a few and dwindling number of honorable exceptions, serve _power_.

The reason journalism is dying is that most journalists -- by which I mean most people making a living under that job title; who on earth is the author referring to? -- long ago discarded these values. Luckily we have the internet and a significant body of courageous bloggers (among, it must be said, a lot of garbagey ones) who actually do care about them, producing real critique, analysis, and even reporting, that for once does not insult the intelligence.

I must be missing something. Surely it's not possible for anyone to have their head _that_ far up. Or maybe this is just what you get when the echo chamber succumbs to denial.

Edit: after surfing around a bit, I think it may be that the author is simply naive. I'd bet money that journalism "training" consists in good part of repeating such self-flattering bromides about journalists, and it sounds to me like the author is just repeating what she was taught. She probably doesn't deserve my rant, but the "journalists" I'm thinking of certainly do, so I'm leaving it.

</comment><comment><USER10> I broadly disagree; plenty of casual bloggers are _excellent_ journalists (in fact some of those people are easily amongst what I would consider the top echelons of journalism). That said; such people are just natural talents, so I also broadly agree with her critique of bloggers.

But, I think Jolie is muddling Journalists together as a whole, when in my mind there are two very distinct types. One is the type she describes; the meticulously neutral, fact checking, source citing reporter who deals in news and events and "things that happen". This is a very hard job. Then you have the opinion writers; and that is even tougher because you have to be critical but fair, opinionated but balanced - and then you have to write really well to pull it off. A really good blogger can easily fit into this crowd without the training Jolie suggests :)

I can tell you how to tell Bloggers from Journalists; bloggers will write a post in a couple of hours and post it. Journalists will spend a day (or more) writing the same post. And though the subject/conclusion might be identical you can always tell the difference.

<comment><USER3> I'm going to get into the more nuanced side of being a blogger AND a journalist in the very near future, but you're partially correct.

When I first started news blogging, I was directly told by my editor (Richard MacManus) that my writing was, while perfectly correct, too dry/boring to get any kind of significant pageviews. He and Marshall Kirkpatrick taught me how to _carefully_ interject analysis and heavily qualified opinion (which was always identified as such) into a blog post to make a kind of news/opinion hybrid.

It's a delicate art, indeed. And a topic for a different post. =)

<comment><USER11> Please: name some names. Your description of the ideal journalist is fairly good, but doesn't really tell us anything about the real world.

<comment><USER3> Naming names amounts to either hero-worship or gossip/slander. I plan to go into more real-world application of these principles for digital media journalists soon, though. I've already started posting in that series, called "Be a Better Journo."

Believe me, I would LOVE to call people out, and I can give a specific example for a good journo/blogger and a bad one for each of these 10 points.

</comment></comment></comment></comment><comment><USER12> A problematic dual-loyalty is embedded in this list.

"5. A journalist is obsessed with the Truth."

and

"6. A journalist serves the people." which was expanded on with "In telling all sides of a story _for the benefit of the proletariat alone_ "

Without getting too deeply into it, you can't serve the people _and_ the truth any more than a democratic vote will always arrive at the best public policy. A bias toward "the people with little or no power or influence in this world" is still a bias and will be a diversion from the truth.

</comment><comment><USER13> Short version: You're a journalist if you've been to j-school and if you pass all of Jolie O'Dell's subjective tests. Too bad there aren't hardly any journalists at all who can pass that test, including most likely Jolie herself.

For fun make the same list about who's a developer and who's not.

(I'm _not_ a blogger--at least not since maybe 4 years back)

</comment><comment><USER14> _A bachelor's degree in journalism, media studies, mass communication, or some similarly named program along with at least a few years under the tutelage of editors is the best preparation for calling oneself a journalist._

So much for all those journalists who got to where they are via cadetships/traineeships - you're nothing according to Jolie O'Dell.

Honestly, this point strikes me as someone trying to justify their own actions as the "correct" actions.

<comment><USER15> _... Whether in the hallowed halls of higher learning _or in the less-hallowed halls of a professional newsroom_, the journalist has been trained as such._

</comment></comment><comment><USER16> I think the blog post should rather be named "How to tell _journalism_ from _blogging_ ". Simply because you can be a journalist without devoting yourself to journalism (just like you can be a programmer without really doing any good programming).

As one that grew up knowing one of Sweden's most prominent journalists I can vouch for some extremely impressive skills that are needed in order to be a good journalist. It's very much about the amount of schooling and _continuous_ preparation you put into a column: a blogger may become successful merely venting his/her thoughts while a "true" journalist reads a book per hundred written words.

<comment><USER17> I agree that might be a better title. I thought it was interesting that she obviously considers herself a journalist but in that article, she was "ranting" with her blogger hat on.

You have to say she is a journalist and a blogger. But you can also make the distinction between when she is practicing journalism and when she is blogging. As she says, without an "op-ed" section, it is sometimes difficult to tell which is which!

</comment></comment><comment><USER18> In a way, the word "journalist" is similar to "hacker" - used with pride by members of the profession, and mostly with negative connotations by outsiders.

</comment><comment><USER19> By this arbitrary criteria I would say there are about 100 times as many blogger-journalists than there are television-news-journalist. Those guys on TV need to stop giving the bloggers a bad name.

<comment><USER20> I think a similar piece could be written with the title "How to tell a journalist from an anchor"

</comment></comment><comment><USER21> Excellent rant! However, she sets the standard too high for some mainstream "journalists" to qualify (no, not just MSNBC and FOX stooges.)

\--

Edit: s/he/she/g

<comment><USER3> My standards are set high on purpose; living up to them is a lifelong quest.

</comment></comment><comment><USER22> The part about sources and especially linking to them is something I wish bloggers would take to heart. I'm so tired of shitty blogs that go out of their way to link to more of their own garbage or search engine spam "tag pages" instead of companies, products, wikipedia, news sites etc to provide context.

</comment><comment><USER23> 3) and 5) are often contradictory, which is the key problem of "objective" journalism. If you don't use your own opinions/interpretations, whose do you use? Not interjecting opinion in practice means accepting the conventional wisdom, or accepting the framing of the subject of your article. Since both the conventional wisdom and the subjects of an article have a bias away from the truth, you cannot refrain from opinion and seek the truth.

</comment><comment><USER24> The difference between a professional journalist and an amateur blogger? Professional journalists have the benefit of being paid to lie to you.

</comment><comment><USER25> If you think about it:

      A journalist works for a news organization.
      A blogger writes via a blog.

Some are both. Most are not. Any good or bad connotations, evolving as they might (not the least because the spheres of connotations do come into contact with each other), can be understood as experiences relating to these two simple facts.

</comment><comment><USER26> Using this list of criteria, which I have no reason to disagree with, I am forced to conclude that the world needs more writers and fewer journalists.

Congratulations on the strong argument.

</comment><comment><USER27> How to tell a journalist from a blogger: A journalist is trained in journalism.

(This is a really poor argument)

</comment><comment><USER28> A great article. Bloggers who fancy themselves New Journalists would do well to read it and learn.

<comment><USER29> While I would agree that most bloggers should not be called journalists, I think that the author comes dangerously close to "no true Scotsman"-territory. Many, if not most of the people commonly referred to as "journalists" (e.g. people writing professionally for big newspapers) would not fit her definition of a journalist. It is more a description of a romantic ideal than of the people actually working in journalism.

<comment><USER30> Definitely. I laughed out loud when reading about how "true" journalists always provide attribution. "Real" journalism on anything of interest is cluttered with anonymous quotes and reeks of the journalist's efforts to protect his or her contacts and food budget. It's also true that journalists often suppress actual news (i.e. the insubordination of military commanders, to choose a recent example that one feels from sub-text Jolie actually may have had in mind) in favor of gossip and party line "reporting."

It was equally funny to read about journalists' unflinching devotion to questioning what they're told. A more uncritical type-what-you-hear crowd has seldom been seen. "Real" journalists are rather devoted to he-said-she-said kind of opinions-on-shape-of-earth-differ writing.

<comment><USER31> Yes, attribution is a funny subject.

Ever seen an article about science that linked to the actual paper that was the reason the article got written in the first place?

Ever seen a blog discussing science put that link in?</comment></comment></comment></comment>


Demystifying Magic Leap: What Is It and How Does It Work? - <AUTHOR> http://uploadvr.com/magic-leap-how-it-works/ <comment><USER2> Previously on Hacker News: [https://news.ycombinator.com/item?id=12182270](https://news.ycombinator.com/item?id=12182270)

</comment><comment><USER3> Anyone know when this will launch?</comment>


Why does Twitter limit to 140 characters? - <AUTHOR>

I wonder how they came up with that number. Is that the size of one TCP/UDP packet or what? <comment><USER2> SMS/text message length, minus the site identifier; lets people use the service via cellphone. As for why text messages are restricted to that length, word is that the guy who developed the first system felt 160 characters was plenty for most communication needs, based on the average length of a random sample of sentences and considering the tedium of having to enter the message via the 12 buttons of a then-typical telephone keypad.

So ultimately, it's arbitrary but practical.

<comment><AUTHOR> Thank you!

</comment></comment><comment><USER3> It's based on SMS [0], the standard that allows telephones to receive text messages. The message size in SMS is limited to 1120 bits, which works out to 140 8-bit characters (or 160 7-bit characters.) One of twitter's design goals was to be usable from the broadest set of mobile devices possible.

[0] <http://en.wikipedia.org/wiki/SMS>

<comment><AUTHOR> That sounds a good reason. Thank you!</comment></comment>