MongoDB queries don't always return all matching documents - <AUTHOR> https://engineering.meteor.com/mongodb-queries-dont-always-return-all-matching-documents-654b6594a827#.s3ko3vfnx <comment><USER2> Said it before, will say it again... "MongoDB is the core piece of architectural rot in every single teetering and broken data platform I've worked with."

The fundamental problem is that MongoDB provides almost no stable semantics to build something deterministic and reliable on top of it.

That said. It is really, really easy to use.

<comment><USER3> As a guy who works on ACID database internals, I'm appalled that people use MongoDB. You want a document store? Use Postgres. Why on earth would you use a database that makes so little in the way of guarantees about what results you get from it? I think most people have really low load and concurrency, so things seem to work. When things get busier you're in for a world of pain. Look I get that's it's easy to use and easy to get started with, but you're going to pay for all of that later.

<comment><USER4> > Why on earth would you use a database that makes so little in the way of > guarantees about what results you get from it?

Because some people can't stand having to work with SQL,migrations,schema and constraints, it's as simple as that ( That's not my opinion,that's just the rational behind MongoDB). Even if you use Postgres with the Json column type, you still need to write SQL queries and schemas.

In the context of analytics, it might make sense, I'm not a big data analyst, but I've seen MongoDB used to centralize logs.

<comment><USER5> > Because some people can't stand having to work with SQL,migrations,schema > and constraints

The thing is, if you actually try and write an app using MongoDB, you will rapidly find that you:

1) Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly).

2) Have schemas (except they'll be implicit and undocumented)

3) Constraints (except they'll be hidden inside your app logic, and violating them will cause data corruption).

The biggest lie about NoSQL databases is that they're schemaless. If you're EVER going to read the data back and do _anything_ with it, it has a schema.

<comment><USER6> This.

My company used mongo for years before we got our shit together.

Schmemas were always implicit (until we got our shit together and started defining and enforcing them with Python Schematics).

Migrations were crazy scripts you run in prod or hacks you stick into your code to "transition".

And yes, surprise constrains left and right causing awful anti-patterns. One- character key names to save disk. Hashed values for indexed keys to save memory. Awkward structuring to improve query performance.

The worst part is, we now have tons of important data in these databases and almost no one understands the legacy crazy app logic that makes them tick.

<comment><USER7> > used mongo for years before we got our shit together.

That's actually a legit use case. Use MongoDB while you get your shit together. I use global variables while I'm noodling around in code. Eventually I refactor.

<comment><USER8> The problem is, in a great portion of real world projects "eventually" never comes and there's just no time for any major refactoring or replacing technologies since you are too busy implementing the feature that was needed two weeks ago.

<comment><USER9> I've often dreamed of a specific type of software built and released as "prototypeware", where any app created using it will have certain built-in scaling limits—and going past them will _irrevocably_ force the app into a read-only mode. It would warn anyone monitoring it well in advance of hitting such a limit, of course. But there'd be no way to just slide the limit upward or otherwise tarry. It'd force the migration to something better just as if it were a Big Customer with Enterprise Compliance Demands.

If an _enforceable_ mechanism like that existed, I'd be a lot more confident in mocking things up. Stick SQLite in for the database, munge HTML and Javascript together, whatever—it's literally going to slap away the hand of anyone who tries to use it on a production workload, so why not?

(Going further, it'd be interesting to create some sort of quagmire of a software license, specifically for prototypeware, such that you'd be forced to rewrite all the prototype code instead of reusing even a hair of it in production. Maybe something like reassigning the IP to a trust, with the trust having an obligation to sue anyone and everyone who tries to create derivative works of the code they've been handed?)

<comment><USER10> This will not work. The whole "prototype" idea assumes once you grow out of the "prototype" phase you have the time, money, manpower, etc. to rewrite the whole thing based on solid, powerful technology and tools. That is, more of then than not, not the case.

The first problem is that every tool has demands, especially the limited ones, and you end up writing your application around those limits and demands, using platform-specific code that will have to be discarded and re-written come the migration.

The second problem is that these tools dictate design, and once you try migrating, you still have an application designed around the prototype tools, which make a lot of concessions and have design flaws because of that.

Finally, I've never understood the need for learning a specific tool, platform or language for "rapid prototyping". Use the tools you will use eventually, it's not that building something in, say, Java from scratch will take an order of magnitude more time and effort than building it on Node.js, despite all the hype, especially if you're a Java shop.

<comment><USER9> > it's not that building something in, say, Java from scratch will take an > order of magnitude more time and effort than building it on Node.js, despite > all the hype, especially if you're a Java shop.

I think we're picturing different things here. You're picturing having software engineers make the prototype, and then having the same engineers do the final implementation. Meanwhile, I'm picturing two different teams, with different competencies—one who knows a prototyping toolchain backward and forward and is extremely productive in it, and the other who knows a solidly- architected platform just as well.

The classical pipeline in the animation industry is to have two separate "teams" of artists. One team does _concept illustration_ and _storyboarding_ , and the other does _keyframe animation_ and _in-betweening_. The first of the two teams is essentially a team of prototypers. Their output is a product which stands on its own for internal evaluation purposes—but which isn't commercially viable "in production." (Nobody really wants to watch 1FPS sketches.) So, after the storyboarding is complete, the whole product is redone by the actual _animators_ into the more familiar product of 24FPS tweened vector-lines or CGI model-joint movements.

The more familiar case of this for web development is where the "prototype" is a PSD file. Professional capital-D Designers are usually Photoshop experts—they're very productive in it, and can mock up something that can be evaluated for being "what the customer wants" quickly, with rapid iteration if it's not right. Once they've got the customer's sign-off, their output product—their _prototype_ —can be tossed over to development staff to "make it work." (There are also an increasing number of interaction-design prototyping apps targeting the same set of designers, under the theory that they'll be able to become productive in quickly iterating the "feeling" of an app with a customer in the way they're already doing with the "look" of the app. I haven't met a designer that uses one of these professionally, but I think that's mostly because there aren't any of these yet well-known enough to be taught in art schools.)

But when it comes to _workflow_ and _use-case_ design, we don't really see the equivalent pipeline. Looking through the lens of separated "prototyper" and "engineer" roles, there are clearly tons of software-development tools that were _intended_ to be used purely by "prototypers": Rails' view scaffolding, for example. But since this role _isn 't_ separate, these things get used _by engineers_ , and sneered at, since, as you said, it's no more effort—when you're already an engineer—to just engineer the thing right from the beginning.

Interestingly, all of the true examples of workflow prototyping I can think of come from the specific domain of game development—but even there, nobody seems to realize that prototyping is the goal of these tools, and tries to misuse them as "production" tools. RPG Maker, seen as a tool for making a commercial RPG, is total crap. RPG Maker, seen as a tool for _prototyping_ an RPG, is an _excellent_ tool. Its output is effectively a _sketch_ , a _cartoon_ in the classical sense:

> The concept [of a cartoon] originated in the Middle Ages and first described > a preparatory drawing for a piece of art, such as a painting, fresco, > tapestry, or stained glass window.

A cartoon is a prototype used to communicate intent. Yes, you (as the producer of the finished piece) can cartoon together with a client to iterate on a proposal. But much more interestingly, a client can learn to cartoon on their own—and then, in place of a long design document, they can submit their cartoon to you. An RPG Maker game project is the best possible thing I could hope to receive as a design proposal from a client asking for me to make an RPG. It forces all the same _decisions_ to be made that making the actual commercial game does—and thus embeds the answers to those decisions in the product—but it doesn't require the same skillset to create that the commercial game does, so the client can do it themselves. The prototyping tool, here, is doing the "iterating on a design together" job of the designer for them.

We do have one common prototyping tool in the software world—Excel. A complex Excel spreadsheet is a cartoon of a business process, that nearly anyone can make. We as engineers might hate them, because people generally have no sense of project organization when making them—but every project to convert an Excel "app" will take far less time than one that involves collecting the business requirements yourself. The decisions have already been made, and codified, into the spreadsheet. You don't have to sit there forcing the client to make them. The process of cartooning has forced them to do it themselves.

\---

To summarize: software prototyping tools aren't _for_ engineers—if you have an engineer's mindset, you'll prototype at the speed of sound engineering practice, so prototype tools won't be any _help_ to you; and you'll be more familiar with the production-quality tools anyway, so you'll be _more_ productive in those than with the prototyping toolset.

But software prototyping tools definitely have uses: they can help designers to iterate on a "functional mock-up" to capture a client's intent; or they can even help clients to create those same mock-ups on their own. This is why "prototypeware" makes sense as software—but also why it should be self- limiting from being used in production. The prototype app wasn't created by someone with an engineering mindset—so there's no way it could end up well- engineered. Its purpose is to serve as a cartoon, a communication to an engineer; not to function in production on its own.

(Mind you, prototypeware _could_ be made to function as an MVP in closed-alpha test scenarios, in the same way that the MVPs of many startups are actually backed by manual human action in their early stages. The point there is to test the correctness of the _codified business process_ , rather than to support a production workload.)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> I've just migrated one project from mongo to postgresql and i advise you to do the same. It was my mistake to use mongo, after I've found memory leak in cursors first day I've used the db which I've reported and they fixed it. It was 2015.. If you have a lot of relations in your data don't use mongo, it's just hype. You will end up with collections without relations and then do joins in your code instead of having db do it for you.

<comment><USER12> > don't use mongo, it's just hype

I'm kind of curious as to where this hype is. I've almost never heard anybody say anything positive about mongodb. All I ever see is people saying it's terrible / hilarious for various reasons.

<comment><USER13> Like with any online community, Hacker News can be kind of an echo chamber where groupthink reigns and alternative points of view aren't encouraged. MongoDB hype has died down here, but there are still some people that are fans.

There are some things MongoDB does fairly well:

* MongoDB is really easy to use

* Document databases can be great and flexible solutions for some kinds of projects

* Documentation is fairly good so learning the basics isn't too hard even if you know nothing about it

* scales fairly well at the initial stages

* arguably quicker to get a project off of the ground with than traditions RDBMs, which might be the most important consideration for any startup even if a complete rewrite would eventually need to take place

That being said, I've used MongoDB significantly before and it wouldn't be my first choice for most types of new project: PostgreSQL probably would be

<comment><USER14> About the only thing I agree with is how great their docs are.

* Mongo is only easy to learn. Beyond simple demos, it gets harder and harder to use as projects evolve i.e. you have to do a lot of work yourself imo this is a common problem with nosql datastores that isn't exclusive to Mongo

* "Document databases can be great and flexible solutions for some kinds of projects": Postgresql has been able to work directly with JSON for some time now. There are also other document datastores that are more reliable than Mongo

* Scaling with Mongo is difficult, specifically the crazy setup. Even if you set it up properly, the results don't tend to match the marketing [https://aphyr.com/posts/322-jepsen-mongodb-stale-reads](https://aphyr.com/posts/322-jepsen-mongodb-stale-reads)

* "arguably quicker to get a project off of the ground with than traditions RDBMs" unless you're using Meteor, I'm also going to disagree here. Most frameworks target a relational database by default. Developing by convention tends to get you off the ground much faster than using something more specialized and niche

<comment><USER15> But they do make great mugs. Who here doesn't have at least a couple of MongoDB mugs. I don't use MongoDB and still have a bunch from random conferences over the last 3-4 years.

</comment></comment></comment></comment></comment><comment><USER16> If you're currently using MongoDB in your stack and are finding yourselves outgrowing it or worried that an issue like this might pop up, you owe it to yourself to check out RethinkDB:

[https://rethinkdb.com/](https://rethinkdb.com/)

It's quite possibly the best document store out right now. Many others in this thread have said good things about it, but give it a try and you'll see.

Here's a technical comparison of RethinkDB and Mongo: [https://rethinkdb.com/docs/comparison- tables/](https://rethinkdb.com/docs/comparison-tables/)

Here's the aphyr review of RethinkDB (based on 2.2.3): [https://aphyr.com/posts/330-jepsen- rethinkdb-2-2-3-reconfigu...](https://aphyr.com/posts/330-jepsen- rethinkdb-2-2-3-reconfiguration)

<comment><USER17> How does it compare to Couchbase? That seems to be lighting the world on fire in that space lately.

<comment><USER16> I'm not sure if lack of overbearing marketing speak counts for something, but RethinkDB definitely has that going for it.

I'm not an expert on couchbase (and neither on RethinkDB, to be frank, though I am a huge fan), but here's what RethinkDB has going for it:

\- Changefeeds - easily open a persistent connection to the server and get updates when the results of an almost arbitrary query changes.

\- Joins

\- Expressive query language that is pretty functionally minded, really shines in their clojure/haskell drivers

\- Excellent client libraries, well maintained

\- Geospatial queries/objects

\- Amazing admin interface (it has been amazing for a long time, too, not a recent change)

\- First class consideration of replication & sharding (it is not a bolt-on in any way shape or form)

\- API-driven cluster configuration

\- API driven permissions management (this is relatively new)

\- Excellent, easy to follow documentation

There are more things, but this is just what I can think of off the top of my head.

The team at RethinkDB is also just great -- I've met them in person and gotten help from them and they're straight shooters.

They've also got this great project coming up called Horizon: [https://www.youtube.com/watch?v=Sb1lH5mvYmU](https://www.youtube.com/watch?v=Sb1lH5mvYmU)

Also they have a video up with a member of the team building a realtime game with React Native: [https://www.youtube.com/watch?v=xRK0SYSgVF0](https://www.youtube.com/watch?v=xRK0SYSgVF0)

Maybe someone who is very familiar with couchbase can help make a list... I'll start it off:

\- Custom query language

\- First class consideration for scale -- replication and sharding

<comment><USER18> I'm ex-Couchbase, so I can probably give a reasonably informed but independent view on this.

Firstly, regarding the marketing, it may not have been to many people's tastes - but it definitely worked, and achieved a lot of what was set out in terms of raising the awareness of what was a decent product that wasn't as well known as its competitors. There may be cases where people avoid it because they don't like the marketing, but the reality, having seen its effect, is they are in the minority, and would probably serve themselves better by assessing products based on technology rather than spiel.

Now, on the actual technology!

What Couchbase has historically been good at is highly scalable Key-Value access, at very high performance and low latency. Performance is comparable to Redis, but CB much more mature sharding, clustering and HA. e.g. fully online growing/shrinking of cluster, protection from node failures, rack/zone failures and data center failures. Redis may be a good fit for single-machine caching situations, and also has its own advantages in terms its datastructures support, etc.

Quality of SDK's is pretty subjective, but I'd say the 2.x re-write of Couchbase SDK's makes them very solid. The Java SDK in particular is extremely good both in performance and by providing native RxJava interfaces.

In terms of query interface, there's geospatial and a new freetext capability on the way.

Couchbase chose down to go down the route of a SQL based interface as their main query language. This seems to be a bit love/hate with developers with some delighted and some perplexed. Maybe for devs it should really be about higher level interfaces like Spring are increasingly important anyway?

The native interface being SQL based is usually very popular with the BI / Reporting side of things.

Changefeeds (continuous query?) is a feature not in Couchbase which I would very much like to see in the future. One thing I would say is that it's something you have to be very careful in the design of to ensure scalability and performance. Consistency is something which would obviously need thought as well.

</comment></comment></comment></comment><comment><USER19> A lot of Mongo DB bashing on HA. We use it and I love it. Of course we have a dataset suited perfectly for Mongo - large documents with little relational data. We paid $0 and quickly and easily configured a 3 node HA cluster that is easy to maintain and performs great.

Remember, not all software needs to scale to millions of users so something affordable and easy to install, use, and maintain makes a lot of sense. Long story short, use the best tool for the job.

<comment><USER20> This has also been my experience. Millions of large documents on a single (beefy) node with a single user it's been fine. Although, the sysadmins had previously left me with flat file xml on shared storage so the bar was pretty low.

</comment></comment><comment><USER21> Oh, the fud of it.

The behavior is well documented here [https://jira.mongodb.org/browse/SERVER-14766](https://jira.mongodb.org/browse/SERVER-14766)

and in the linked issues. Seasoned users of mongodb know to structure their queries to avoid depending on a cursor if the collection may be concurrently updated by another process.

The usual pattern is to re-query the db in cases where your cursor may have gone stale. This tends to be habit due to the 10-minute cursor timeout default.

MongoDB may not be perfect, but like any tool, if you know its limitations it can be extremely useful, and it certainly is way more approachable for programmers who do not have the luxury of learning all the voodoo and lore that surrounds SQL-based relational DB's.

Look for some rational discussion at the bottom of this mongo hatefest!

<comment><USER22> I wouldn't call a JIRA ticket good documentation.

While I agree that it's good to know the limitations of the tools you chose those limitations should be clearly spelled out in the documentation.

I don't think most programmers have the luxury of learning all the voodoo and lore that surrounds MongoDB from JIRA tickets and blog posts.

<comment><USER21> > I don't think most programmers have the luxury of learning all the voodoo > and lore that surrounds MongoDB from JIRA tickets and blog posts.

That's how I learned everything I know about most FOSS products I have encountered - through the code pages and social media surrounding the project.

Pretty much everything about the mongodb hate derives from their marketing and sales. The truth is, they've obviously stumbled onto something the market wants, otherwise they would never have become so successful.

For me, as a long-time programmer with no database experience, the mental mapping of JSON constructs as both data and query language was far easier for me to absorb than the relational model, which didn't fit the paradigms that I was used to.

At my present gig, we've used Mongo DB for two years, scaling up to quite a large production setup. Like any technology it has strengths and weaknesses, but it has not been the utter failure that readers of Hacker News would be led to expect. We adopted it knowing quite a bit about its history, and it has turned out to be an excellent choice that has held up over time.

Periodically we've considered switching to postgres, and we may do so for part of our stack. But for the core jobs of data collection and batch processing data with fluid schema, I'm pretty sure we will stick with mongodb for the duration.

It's just a tool, folks.

</comment></comment></comment><comment><USER23> Strongly biased comment here, but hope its useful.

Have you tried ToroDB ([https://github.com/torodb/torodb](https://github.com/torodb/torodb))? It still has a lot of room for improvement, but it basically gives you what MongoDB does (even the same API at the wire level) while transforming data into a relational form. Completely automatically, no need to design the schema. It uses Postgres, but it is far better than JSONB alone, as it maps data to relational tables and offers a MongoDB-compatible API.

Needless to say, queries and cursors run under REPEATABLE READ isolation mode, which means that the problem stated by OP will never happen here. Problem solved.

Please give it a try and contribute to its development, even just with providing feedback.

P.S. ToroDB developer here :)

<comment><USER24> How does ToroDB handles sharding across multiple instances?

<comment><USER23> Right now ToroDB handles sharding at the backend (RDBMS) level, with those dbs that support that. There's currently a Greenplum-based backend on the works, that obviously handles sharding by itself. Also CitusDB is on the roadmap.

At a later release, we also plan to natively support MongoDB's sharding protocol.

</comment></comment></comment><comment><USER25> My general feeling is that MongoDb was designed by people who hadn't designed a database before, and marketed to people who didn't know how to use one.

Its marketing was pretty silly about all the various things it would do, when it didn't even have a reliable storage engine.

Its defaults at launch would consider a write stored when it was buffered for send on the client, which is nuts. There's lots of ways to solve the problems that people use MongoDB for, without all of the issues it brings.

<comment><USER26> I really agree with your sentiments, that first paragraph is a great quote. I grew quite an adverse to MongoDB after researching it. While I never found this specific caveat, I found other very worrying decisions.

> reliable storage engine

By "reliable" I assume you mean "consistent?" While MongoDB claims that it's CP (which it's not, as per the article) there's nothing wrong with inconsistent databases (AP, e.g. CouchDB). Mathematically there is no reason for MongoDB to behave like this. It's fundamentally broken; it's neither AP nor CP.

<comment><USER25> I actually mean reliable. Its probably different now, but at launch, the defaults were fsync'ing every 30 seconds or so. It would literally just apply the change to an memory mapped buffer and just fsync it once in a while.

They did that so they could look good in benchmarks, and it's why they recommended so strongly that your memory completely fit in RAM or else things would fall apart (pro-tip, any system that recommends that has a poorly designed storage engine).

They also screwed up the consistent side of things as well.

</comment></comment></comment><comment><USER27> I have moved from Mongo to Cassandra in a financial time series context, and it's what I should have done straight from the getgo. I don't see Cassandra as that much more difficult to setup than Mongo, certainly no harder than Postgres IMHO, even in a cluster, and what you get leaves _everything_ else in the dust if you can wrap your mind around its key-key-value store engine. It brings enormous benefits to a huge class of queries that are common in timeseries, logs, chats etc, and with it, no-single-point-of-failure robustness, and real-deal scalability. I literally saw a 20x performance improvement on range queries. Cannot recommend it more (and no, I have no affiliation to Datastax).

<comment><USER28> Genuinely curious: when you say "it brings enormous benefits to a huge class of queries that are common in timeseries", what are you referring to, exactly?

I run Cassandra in production and I love its operational simplicity, scale-out design, and write performance. But I think its support for time series is perhaps over-hyped. To me, it seems the only queries you can run in Cassandra is a key lookup (partition key row get) and a column slice (partition key row get filtered by an ordered range of columns). This allows for a certain time series use case e.g. where each row represents exactly one series, and where the only thing you want to do with a series is to get its raw values. But it doesn't allow for many of the things I personally think of when I think about "time series queries", e.g. resampling, aggregates, rollups, and the like.

<comment><USER27> I am referring to anything that resembles a range query, ie, where you require a bunch of contiguous information queried on a single key. Think "give me all of this person's chat entries from x time to y time", or indeed "give me all this topic's comment entries from x time to y time" (but not both - only one of the above would be efficiently stored - you decide which it would be).

Cassandra, as you know, forces a certain amount of "low level awareness" requirement on the programmer because to tap into its uniqueness, you need to know how you will query stuff, so that Cassandra will ensure that the most common range queries are contiguously stored in rows. All other databases hide the on-disk storage order from you in an abstraction, and you can find atomisation causing inefficiency. Cassandra forces you to think about it, and in return, guarantees contiguous storage order on disk along one of your keys so that along that key, retrieval is lightning fast as it requires only one pass.

Basically, both spinning disks but _also_ SSDs, are in essence, 1d media (ie, a lot in common with tape) in the sense that along one dimension you can read stuff massively fast, but as soon as you need to seek (ie start using dimension 2), even on an SSD, your performance dramatically declines. Cassandra forces you to think about your queries so that they will be "aligned" along the most efficient direction on disk.

Now agreed that if your queries cannot be aligned along said direction, then Cassandra drops to being no better than all the others, and penalises you with some complexity. That includes some examples of aggregrates, resampling etc (though I would argue that the order of magnitude contiguous read still helps these). Some of this can be mitigated with denormalisation ie: storing stuff more than once, in transposed or sub-sampled orders, something that relational DB purists will hate, with some justification (potential for inconsistency).

FWIW Riak TS sounds promising with automatic "blob" style storage etc and resampling capabilities which might take Cassandra on quite explicity and in a higher level, more convenient way. I am about to evaluate it because I agree with you that the resampling capability in particular could be better supported in Cassandra, though ultimately, both databases will still be limited by the underlying D1 v D2 "contiguous v seek" capabilities of the storage so I'm not expecting miracles from Riak.

By the way, I'm not even touching on Cassandra's scale-out ease. More perf needed? Literally just add boxes though it would be unfair not to comment on the cost of this, which is Cassandra's node-level consistency tradeoffs for very recently added data, and which is, if I recall correctly, why Facebook went to Hbase. You can force consistency at the query level, but performance can suffer.

<comment><USER29> After some truely horrific experiences with Riak K/V, especially combined with Riak Solr, I won't touch anything from Basho with a ten foot pole. Not sure what's going on over there, but the reality of Riak in production was miles away from what Basho's sales claimed was possible. And yes, we even spent about 4 months working with their tech support. It almost seems that "It's based on Erlang thus it scales" was the entirety of their design work.

I've also worked with Cassandra and have nothing but good to say about it, did what we asked it right out-of-the-box. Datastax was really helpful as well.

\--

And I have no affiliation with either Basho nor Datastax, just really happy with one product and completely blown away with the poor performance of the other.

</comment></comment></comment></comment><comment><USER30> Weird to see that Mongo is still around. We started to use them on a project ~4 years ago. Easy install, but that's where the problems started. Overall terrible experience. Low performance, Syntax a mess, unreadable documentation.

They seem to still have this outstanding marketing team.

</comment><comment><USER31> Should an infrastructure company be advertising the fact that it didn't research the technology it chose to use to build its own infrastructure?

All these people saying Mongo is garbage are all likely neckbeards sysadmins. Unless you're hiring database admin and sysadmins, Postgres (unless managed - then you have a different set of scaling problems) or any other tradition SQL store is not a viable alternative. This author uses Bigtable as a point of comparison. Stay tuned for his next blog post comparing IIS to Cloudflare.

Almost every blog post titled "why we're moving from Mongo to X" or "Top 10 reason to avoid Mongo" could have been prevented with a little bit of research. People have spent their entire life working with the SQL world so throw something new at them and they reject it like the plague. Postgres is only good now because they had to do some of the features in order to compete with Mongo. Postgres been around since 1996 and you're only now using it? Tell me more about how awesome it is.

<comment><USER32> My goal in writing this post was not to convince people to use or not use MongoDB, but to document an edge case that may affect people who happen to use it for whatever reason, which as far as I could tell was inadequately documented elsewhere.

<comment><USER31> Only the first line was directed at you - and it was more in jest. Everything else was directed more at the other commenters and Mongo detractors in general.

</comment></comment></comment><comment><USER33> While I love to hate on MongoDB as much as the next guy, this behavior is consistent with read-committed isolation. You'd have to be using Serializable isolation in an RDBMS to avoid this anomaly.

<comment><USER34> I think this is incorrect, but it's not as simple as the other replies are making it out to be.

Under read-committed isolation, _within a single operation_ , you must not be able to see inconsistent data. So if you do "SELECT <star>" on a table while rows are being updated, you're guaranteed to always see either the old value or the new value. But if you do two separate statements, "SELECT <star> WHERE value='new'" and "SELECT <star> WHERE value='old'" in the same transaction, you may not see the row because its value could have changed. Serializable isolation prevents this case, typically by holding locks until the transaction commits.

It gets messy because the ANSI SQL isolation levels are of course defined in terms of SQL statements, which don't map perfectly to the operations that a MongoDB client can do. Mongo apparently treats an "index scan" as a sequence of many individual operations, not as a single read. So you could argue that it _technically_ obeys read-committed isolation, but it definitely violates the spirit.

</comment></comment><comment><USER35> The real problem with Mongo is that it's so enjoyable to start a project with that it's easy to look for ways to continue using it even when Mongo's problems start surfacing. I'll never forget how many problems my team ended up facing with Mongo. Missing inserts, slow queries with only a few hundred records, document size limits. All while Mongo was paraded as web scale in talks.

</comment><comment><USER36> Does Meteor support a proper database system yet, a la. MySQL or Postgres?

<comment><AUTHOR> Yes - with Apollo/GraphQL (currently available as a technical preview): [http://docs.apollostack.com/apollo- client/meteor.html](http://docs.apollostack.com/apollo-client/meteor.html)

I recommend you check out the Apollo Meteor Starter Kit: [https://github.com/apollostack/meteor-starter- kit](https://github.com/apollostack/meteor-starter-kit)

<comment><USER36> Noticed how I referenced 2 proper RDBMS in my question? Then how you proceeded to introduce another flavour-of-the-month

<comment><USER37> I haven't looked at Apollo, but GP should've explained that GraphQL is not a database and can be hooked up to any backend, so with it I'm guessing you can use any kind of database in Apollo/Meteor apps. Still, kind of weird.

Another reason why I never bothered with Meteor.

<comment><USER36> I remember when Meteor was the JavaScript flavour-of-the-month and everyone was saying it will kill Rails. I wanted to believe so I looked into Meteor, then I saw its dependence on MongoDB... Nope!

<comment><USER37> If you study the backend JavaScript ecosystem you'll see no Rails-like all-in- one framework has ever succeeded. They're just not part of the culture. The only Rails-like thing in the JS ecosystem that's popular and great is Ember but that's only frontend.

</comment></comment></comment></comment></comment></comment><comment><USER38> MongoDB reminds me of an old saying that if you have a problem and you use a regex to solve it, you end up with two problems.

I have personally used MongoDB in production two times for fairly busy and loaded projects, and both times I ended up to be the person that encouraged migrating away from MongoDB to a SQL based storage solution. Even at my current job there's still evidence that MongoDB was used for our product, but eventually got migrated to PostgreSQL.

Most of the times I've thought that I chose the wrong tool for the right job, which may be true, but still leaves a lot of thought about the correct application. Right now I have a MongoDB anxiety - as soon as I start thinking about maybe using it(with an emphasis on maybe), I remember all the troubles I went through and just forget it.

It is certainly not a bad product, but it's a niche product in my opinion. Maybe I just haven't found the niche.

<comment><USER39> I literally brought up the regex joke in a meeting yesterday. A data warehouse was built on top of Mongo, and I get to help clean up the mess.

</comment></comment><comment><USER40> This single issue would make me not want to use MongoDB. I'm sure there are design considerations around it but I rather use something that has sane semantics around these edge cases.

</comment><comment><USER41> Not when they're changing rapidly, anyway. Well, that's relaxed consistency for you.

Does this guy have so many containers running that the status info can't be kept in RAM? I have a status table in MySQL that's kept by the MEMORY engine; it's thus in RAM. It doesn't have to survive reboots.

</comment><comment><USER42> CouchDB is simple and reliable. You can understand it from day one. I can't imagine why it isn't being used.

<comment><USER43> I really want an excuse to build something with CouchDB and PouchDB ([https://pouchdb.com/](https://pouchdb.com/)). Can you expand on your experiences with it?

<comment><USER44> I'm not the original poster, but I can give you some of my limited experience with CouchDB from an application I inherited. The original idea for the project still seems like a good idea to me. Basically they wanted to record events that came into the system and store them in a write only ledger. Then they wanted to version every change so that you have an audit trail. Finally they wanted to be able to create views of that ledger to create the kind of data that they would work with on a day to day basis. For this, CouchDB seems like a perfect fit.

Unfortunately, it didn't work out as well as one might hope because the people who implemented the idea didn't seem to be able to resist using the DB the way they would use a relational db. Instead of maintaining the concept of a write only ledger, they started to use it as a data store for things that were ephemeral. Also, instead of replicating the db, using a view to create a new db that was optimal for certain queries, they wrote a huge number of views in the main db. Finally they organised the views by relation rather than by use, so you would have 60-80 views in the same design document that would have to be reindexed if one of them changed.

The result was something with very poor performance and where the storage for the indexes was more than an order of magnitude more than the storage for the documents themselves.

CouchDB is also not super speedy at the best of times. There is a lot of latency involved in serializing the documents and farming them out to view servers, etc. So it takes a good 10 minutes to process a million documents, but you will find that your CPU is chugging along at 30-40% utilisation.

Having said all that, one of the things I want to try (but have only done some preliminary trials with) is to keep the concept of the write only ledger, but to replicate the db into several views of the data (some with severely restricted content). Then instead of building something like a rails application to farm out the data, make "couch applications" where you serve the HTML and JS directly from attachments on documents in the DB. In fact, I've written a React application to allow users to interact with portions of the data and it was quite simple. Then you can write a really small coordinating application to allow users to navigate to the parts of the system (really single page apps) that they want to use.

Again, the nice thing about this is that you have a write only data store with versioning and the ability to audit history. You have views that allow you to interact with a small subset of the overall data. You can easily write single page applications where deployment is as easy as pushing a document to the DB. Replication is relatively cheap and you can move expensive view creation to restricted versions of the DB. You can stick the whole thing behind a load balancer and scale it as cheaply as setting up a new replication (again just another document in your DB).

But, I will warn you. Don't use it like you would a relational DB, or else you will be in for a world of hurt. Especially you will see comments in this thread about migrations. If you are migrating your data, by definition you do not have a write-only-with-versioning application. Your application will have to deal with multiple versions of data or else you will not have the ability to audit history. If you do not care about this, then possibly there are better solutions than this.

<comment><USER42>
        one of the things I want to try (but have only done some
        preliminary trials with) is to keep the concept of the
        write only ledger, but to replicate the db into several
        views of the data (some with severely restricted content).

How is that different from the current concept of CouchDB views? You meant to replicate the DB to various different places and use the same CouchDB views from there? Or you meant something like a replication-view, in which some calculations are done with the documents in the source database and the target database receives the result of those calculations as their primary documents?

<comment><USER44> Yes, the latter. One of the main problems I've seen is that indexing things that you will never query is both expensive in time and space. Also it's amazing how many views tend to have exactly the same data, only sorted differently. And the reason to sort it differently is because you only want to work on a subset of the data, but you can only restrict the query in a contiguous section of keys.

An example of this might be that you have a large number of daily reports. They all need different aspects of the data, but you end up writing views that sort by date and then collate the result in the server. So you end up maintaining an index for data that you will never query again and you are doing lots of extra processing merging the data after the query. Much better to replicate one day's worth of data to a new db every evening (possibly setting up a continuous replication to keep it up to date) and then add views on _that_ db to do what you want. Like I said a full replication of a million documents takes about 10 minutes, so it's a reasonable thing to do.

<comment><USER42> I like the concept of views. I think it is the best part of CouchDB. Saving data and later defining views (that should be very fast). Once I wrote an app that stored massively enormous documents with lots of data, the views were used to turn that data into queriable information later.

I like what you suggested very much, because what CouchDB can currently do with views is very limited to what a powerful views implementation would, and the one you've suggested is a good suggestion of how to do it better.

Thank you. I'll give this a lot of thinking when I restart writing my [https://github.com/fiatjaf/summadb/](https://github.com/fiatjaf/summadb/)

</comment></comment></comment></comment></comment></comment><comment><USER45> I believe this is solved by Mongo's "snapshot" method on cursors: [https://docs.mongodb.com/v3.0/faq/developers/#faq- developers...](https://docs.mongodb.com/v3.0/faq/developers/#faq-developers- isolate-cursors)

<comment><USER32> If I understand correctly, this method says "only scan the built in _id index, not any other index". Which means that you will not hit this index-specific bad behavior, but also that you won't get the performance characteristics of using an index.

</comment></comment><comment><USER46> Mongo is hilarious. Ease of use is so important, we just don't much give a shit that it has all these gaping holes and flaws in it.

</comment><comment><USER47> Seriously, who looks at MongoDB and thinks "this is a sane way of doing things"?

To be fair, I've never been much of a fan of the whole NoSQL solution, so I may be biased, but what real benefits do you gain from using NoSQL over anything else?

<comment><USER48> Web scale!

</comment></comment><comment><USER49> I worked with MongoDB quite a lot in context of Rails applications. While it has performance issues and can generally become pain because of lack of relations features, it also allows for really fast prototyping (and I believe that Mongoid is much nicer to work with than Active Record).

When you're developing MVPs, work with ever changing designs and features, ability to cut off this whole migration part comes around really handy. I would however recommend to anybody to keep migration plan for the moment the product stabilizes. If you don't, you end up in the world of pain.

</comment><comment><USER50> Actually, if this lack of index update isolation is correct, you can get the matching document zero, one or _multiple times_!

</comment><comment><USER51> Mongo, in one word: sucks. Couchbase, does not.

<comment><USER52> I use RethinkDB in most of my production things. I recommend it.

<comment><USER53> Agreed, I've had nothing but positive experiences with it.

<comment><USER52> I started with Mongo too, had some performance issues and started used RethinkDB when they first released it. It does get better with every update.

</comment></comment></comment></comment><comment><USER54> It literally returns wrong answers for queries. I can't believe anyone this thread is defending it.

</comment><comment><USER55> What storage engine are you using? I wonder if the same issue comes in wiredtiger MVCC engine.

<comment><USER11> He wrote in comments that this issue is in mmap and in wiredtiger.

<comment><USER56> There are like 5+ storage engines available for Mongo, probably more, but those are just the ones I'm aware of, plus various forks, like TokuMX an Percona, etc... This is all FUD.

</comment></comment></comment><comment><USER57> So it's a bit weak in the design department, offers a bit less rigid semantics than one might hope, and from the start it's a technology that was almost a reaction to the rigid and enterprise-y of old.

Mongo reminds me a wee bit of JS...

</comment><comment><USER58> Unless you want to code every rdbms and enterprise feature in the application layer, don't use Minho, use Postgres or Use Marklogic. It is 'nosql', but it is acid compliant and uses MVCC so what the queries return is predictable.

</comment><comment><USER59> I hear a lot about MongoDB's reliability issues. How do CouchDB or other document store database compare in terms of reliability and consistency?

<comment><USER15> CouchDB is rock solid. Used it for 5 years now. Never got corrupted data. Has master-to-master replications. Really shines in sometimes offline operation mode (with re-sync on reconnect).

I use that extensively to build custom replication cluster topoligies (overlapping rings, star, hierarchy) etc.

Has HTTP interface so easy to build clients for.

Transactions are per document only. So have to design your application to accomodate it. Raw single document write speed is not as fast as Mongo or Postgres. But I noticed in a concurrent environment, multiple connections writing it scaled pretty well.

Moreover, CouchDB 2.0 will have built-in clustering from code donated by Cloudant. And it will also have a similar query language like MongoDB (instead of having to use Javascript / Python / Other map-reduce functions).

</comment></comment><comment><USER60> An interesting read into the development of a project that started using MongoDB and switched to PostgreSQL after eight months in production: [http://www.sarahmei.com/blog/2013/11/11/why-you-should- never...](http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use- mongodb/)

</comment><comment><USER61> This use-case is not something that you would use MongoDB for. Try Zookeeper.

This being said, I would feel embarrassed to post this on behalf of the engineering department of a company.

This post is just a very illustrated way of saying "we have no idea about what we are doing and our services are completely unreliable".

This is so bad that is more of an HR problem than it is an engineering problem.

<comment><USER34> Did you miss the part about how they're running a hosting platform that stores details about the status of containers for all their customers?

Zookeeper is fine for things like service discovery that deal with a bounded amount of data. You _don 't_ want to use it for something where the amount of data depends on, say, how many containers your customers decide to start. Every ZK server keeps all of its data on the Java heap, so if your data gets too big, _pow_. How big is too big? Don't worry, you'll find out the hard way sooner or later!

Plus, there's no sharding -- every write operation has to be acknowledged by a majority of nodes in your cluster. So for write-heavy workloads (which is what I would expect a service status dashboard to experience) your cluster actually gets _slower_ if you try to add more machines.

<comment><USER61> Zookeeper slows down when you add nodes since quorum/consensus is larger. You can mitigate some of this with non-voting nodes (observer nodes) but only up to certain extent. So yes, a single Zookeeper cluster won't scale horizontally.

But that doesn't limit the amount of independent clusters you can have.

The reason I suggested Zookeeper is because it offers you ephemeral nodes, which is convenient to mark stuff as unavailable.

</comment></comment></comment><comment><USER56> Y'all know other storage engines exist, right?

I searched the comments for "percona" and found nothing...

Figures.

Meanwhile, [https://github.com/percona/percona-server- mongodb/pull/17](https://github.com/percona/percona-server-mongodb/pull/17)

</comment><comment><USER62> Ahhh the Trough of Dissolutionment!

[1] [https://setandbma.wordpress.com/2012/05/28/technology- adopti...](https://setandbma.wordpress.com/2012/05/28/technology-adoption- shift/)

</comment><comment><USER63> Use of MongoDB at PlanGrid is probably the single worst technical decision the company ever made.

We've migrated our largest collections to Postgres tables and our happiness with that decision increases by the day.

</comment><comment><USER64> I am pretty excited about cockroachDb. Its still in beta so not suggested for production use yet, but its being designed pretty carefully and by a great team.. check them out cockroachlabs.com

</comment><comment><USER65> Is MongoDB really that bad?

I am someone just getting into Meteor Js and it seems like moving from MongoDB would make it Meteor trickier to learn.

Is it difficult to switch to an alternative? Thanks

<comment> nevi-me It's not, go ahead and use it, learn and gain experience. It's not a replacement for SQL databases. It doesn't have joins and the biggest issue academics and as sysadmins have is it's not fully ACID compliant, so no transactions for example.

If I was writing this 2 years ago, I would say horizontal scaling is much easier. Add a node to your replica, watch it catch up, and continue.

Have data stored in an array 4 levels deep? Mongo will find it for you. It's only difficult to switch to an alternative to the extent that you've convolved your schema in an unfriendly way. Most migration entails normalising your data into different SQL table and exporting it. Not rocket science as people make it seem to be.

I use SQL at work, Oracle, SAS excuse, a bit of MySQL and sometimes Postgres - I'm a consultant. I have tried some NoSQL DBs but always come back to Mongo, for personal projects.

I've done a few prototypes for clients using Mongo, but those are almost always for geospatial support.

<comment><USER65> Thanks

</comment></comment></comment><comment><USER66> I wonder how much data they are storing and in what pattern that they actually need a NoSQL database. I'm curious why someone would make that choice.

</comment><comment><USER67> If you were inserting changes in the status you'd have much better data and never incur in such issues.

</comment><comment><USER68> I'm prototyping in meteor using MongoDB and Compute Engine.

I have two VM instances in google cloud platform. One is a web app and the other is a MongoDB instance. They are in the same network. The connection I use is their internal IP.

Can other people eaves drop between my two instances?

</comment><comment><USER69> TL;DR During updates, Mongo moves a record from one position in the index to another position. It does this in-place without acquiring a lock. Thus during a read query, the index scan can miss the record being updated, even if the record matched the query before the update began.

<comment><USER70> _mongo developer 1_

Man, it's just taking too damn long to run this update. What should we do?

 _mongo developer 2_

Uh....remove all the locks?

 _mongo developer 1_

Oh, yeah, that makes sense, let's do that.

 _mongo developer 3_

Hey, what are you guys up to?

 _mongo developer 2_

Making it web-scale!

 _mongo developer 3_

Good job, keep it up!

<comment><USER71> Seriously. I was looking at DB usage statistics recently and was appalled MongoDB is still so popular. I thought it was done, nail in the coffin, when [https://www.youtube.com/watch?v=b2F-DItXtZs](https://www.youtube.com/watch?v=b2F-DItXtZs) came out 6 years ago, I haven't followed it much since then apart from the occasional post like this whose content is just "you thought it was bad already? haha it's worse."

</comment></comment></comment><comment><USER72> But it's web scale! </sarcasm>

</comment><comment><USER73> TLDR: He was reading the database while another process was writing to it.

Why all the Mongo hate? I'm sure this would happen with other databases.

<comment><USER74> No, this does not happen with any relational database I've worked with.

</comment></comment><comment><USER75> But is your database webscalwebscale? MongoDB is a web scale database.

<comment><USER62> You've seen THAT YouTube video then!

</comment></comment><comment><USER76> Everytime I hear arguments for going back to relational databases, I remember all the scalability problems I lived through for 15 years in relational hell before switching to Mongo.

The thing about relational databases is that they do everything for you. You just lay the schema out (with ancient E-R tools maybe) load your relational data, write the queries, indexes, that's it.

The problem was scalability, or any tough performance situation really. That's when you realized RDBMSs were huge lock-ins, in the sense that they would require an enormous amount of time to figure out how to optimize queries and db parameters so that they could do that magic outer join for you. I remember queries that would take 10x more time to finish just by changing the order of tables in a FROM. I recall spending days trying different Oracle hints just to see if that would make any difference. And the SQL-way, with PK constraints and things like triggers, just made matters worse by claiming the database was actually responsible for maintaining data consistency. SQL, with its naturalish language syntax, was designed so that businessman could inquire the database directly about their business, but somehow that became a programming interface, and finally things like ORMs where invented that actually translated code into English so that a query compiler could translate that back into code. Insane!

Mongo, like most NoSQL, forces you to denormalize and do data consistency in your code, moving data logic into solid models that are tested and versioned from day one. That's the way it's supposed to be done, it sorta screams take control over your data goddammit. So, yes, there's a long way to go with Mongo or any generalistic NoSQL database really, but RDBMS seems a step back even if your data is purely relational.

<comment><USER66> I've been in the opposite situation and I couldn't disagree more. But I will say this, it's always possible to take an RDBMS model and de-normalize it and use it like a NoSQL database (like reddit does, for example) but it's not possible to go the other way.

<comment> lloyd-christmas > but it's not possible to go the other way

Why not? We do exactly that. We prototype in mongo and then migrate to postgres when we're comfortable with where the app is headed.

<comment><USER66> I don't mean it's possible to use a different technology; I mean within the same technology (postgres, for example) you can use it both as a normalized relational database and/or as a de-normalized document store.

</comment></comment></comment></comment><comment><USER77> The article is interesting, but title is fud. Besides, all this is not unexpected:

> How does MongoDB ensure consistency?

> Applications can optionally read from secondary replicas, where data is > eventually consistent by default. Reads from secondaries can be useful in > scenarios where it is acceptable for data to be slightly out of date, such > as some reporting applications.

[https://www.mongodb.com/faq](https://www.mongodb.com/faq)

<comment><USER32> This is not related to reading from secondaries. This issue can occur in single node systems.

<comment><USER77> Yes, as I said, all this is NOT unexpected from them.</comment></comment></comment>


Tin Can Phone Modem - <AUTHOR> https://www.mikekohn.net/micro/tin_can_phone_modem.php <comment><USER2> Tangentially, a UK ISP managed to get DSL to work over wet string:

[https://www.revk.uk/2017/12/its-official-adsl-works-over- wet...](https://www.revk.uk/2017/12/its-official-adsl-works-over-wet- string.html)

<comment><USER3> Reminds me of gigabit Ethernet over rusty barbed wire: [https://news.ycombinator.com/item?id=15910263](https://news.ycombinator.com/item?id=15910263)

<comment><USER4> One thing of note is that 1000-base-T and G.991 (marketed as SHDSL/HDSL/SDSL depending on country/telco) is somewhat similar technology that is hard to classify as being baseband or broadband. One of my somewhat more absurd networking experiences was using SHDSL as "LAN with cheap wire" on anime convention. We had few hundred meters of flat phone "tinsel wire" left over from previous project and I had somehow acquired bunch of Cisco 17xxs and WIC-1SHDSL line cards. It worked well enough for our purposes and I learned one thing: plain G.991 works over random wire reasonably, while G.991bis with enabled channel bonding does not (probably because of crosstalk).

Edit: one notable thing was that we tested the whole setup beforehand with same cables, but coiled. And all the links were capable of at least 3Mbps. Once we unraveled the cables on the site most of the links were unable to train at more than 512kbps in two-pair mode, but worked reliably at 1.5Mbps in single-pair mode.

</comment></comment></comment><comment><USER5> This reminds me of an old magazine article (Acorn User?) that described how to use a pair of BBC Micros with wired in LDRs and flashlights to transmit data wirelessly.

<comment><USER6> And that reminded me of a childhood project where I hooked the user port (8 bit parallel) of one BBC up to another and wrote a chat program that sent data from one machine to another by POKEing ASCII values IIRC. Not a million miles from what you could do with an IBM PC printer port.

Not much use, the cable was about a foot long.

My child's generation won't be able to do that, but I'm glad we have arduinos and ESP32s to fill the same niche.

<comment><USER7> netcat on a lan can do it for kids today

<comment><USER6> it's less exciting just plugging an ethernet cable in

</comment></comment></comment></comment><comment><USER8> Very fun. Now for the next step, replace the cans with quantum entangled objects ;)

<comment><USER9> I don't know much about entangled particles or objects; that being said, the one thing I keep hearing is that quantum decoherence cannot violate causality or allow communication faster-than-light. Not that you were saying that, but I keep hearing that people think a kind of quantum walkie-talkie or one-time-pad is possible. I honestly don't know either way, as I'm not in this field.

It certainly is interesting, though!

[https://sci-hub.tw/https://doi.org/10.1038/nphoton.2007.22](https://sci- hub.tw/https://doi.org/10.1038/nphoton.2007.22)

</comment></comment><comment><USER10> I don't know why this hasn't been upvoted much so far. It's literally the definition of hacking (da, Hackernews). I'm always hesitant to use "literally" where it means figuratively, but this can modem literally _is_ hacking.

And to think that the concept is so simple and yet, so much fun, makes it yet more interesting.

</comment><comment><USER11> What do you think is the maximum transmit distance with error correction bits? Obviously it would be affected by string tension and string material, but do you think it would work strung between two nextdoor neighbors child bedrooms?

<comment><USER4> If you assume some kind of error correction the question stops being "would it work" and instead becomes "how slow/fast would it be"

</comment></comment><comment><USER12> We used to string talk back when we were kids, we built a poorman's network of those in my neighborhood and worked well until the next storm took them down. It is very rewarding to actually experience first hand how the vibration turns in to sinewaves and back into vibration. The problem with the current generation of kids is that most of their playtime is digitized, which may have some advantages but also I see quite a few disadvantages. Well, I don't want to take the conversation in different direction, this is about hacking modems over wire and is pretty cool project.</comment>


Redis latency spikes and the 99th percentile - <AUTHOR> http://antirez.com/news/83 <comment><USER2> A coworker pointed me to this interesting post noting that in a browser-facing web server (not the same as hits to a datastore), most users will experience the 99th percentile for at least one resource on the page.

[http://latencytipoftheday.blogspot.it/2014/06/latencytipofth...](http://latencytipoftheday.blogspot.it/2014/06/latencytipoftheday- most-page-loads.html)

<comment><USER3> This is only true if these latency outliers are evenly distributed over time, if they indeed happen once every 30 minutes than only users requesting a page at the same time will be affected.

<comment><USER2> Excellent point, yes.

</comment></comment></comment><comment><USER4> A nice follow-up and details digging into yesterday's "This is why I can't have conversations using Twitter" post:

[https://news.ycombinator.com/item?id=8526208](https://news.ycombinator.com/item?id=8526208)

<comment><USER5> Agree.

I really wanted to comment "just always make a blog post in reaction, twitter is not a good medium for explaining complex ideas" on that thread, but thought it would be unproductive.

Please just do this every time, people who care will read, people who don't care will probably not comment for fear of looking stupid.

</comment></comment><comment><USER6> Why is forking on xen slow? Most google hits for "xen forking slow" seem to point to some discussions about redis, but I guess other software would suffer from that too.

<comment><USER7> It's important to remember two things: First, in a more general way, fork() is kind of slow just by virtue of being a system call and involving several steps, so applications try to avoid making lots of calls to fork (e.g. webservers long ago stopped doing the naïve fork-for-every-request model).

Redis uses fork in a way nobody else seems to. Applications using tens of gigabytes of RAM -- databases, media editing, etc. -- just don't usually fork except during a startup process.

Redis uses fork for persistence. It's clever, and it works great in general, but it's weird! Unique, even. It seems much more likely to hit a fork() weakness in a noticeable way than almost any other application out there.

<comment><USER8> Fork has always been a mutant feature. Split the entire process into twins? So that the caller can immediately exec() and discard the tediously-cloned twin and become a different program? Its the most egregiously inefficient feature ever to grace an operating system. Page tables, stacks, heap allocations, file handles - all cloned and then discarded.

<comment><USER9> In this Regard, Redis is the first system using it the proper way I guess ;-) I mean, all this copying is not discarded but used to create a point-in-time snapshot.

<comment><USER10> No, every Unix command shell uses the copied program and file descriptors.

</comment></comment></comment></comment></comment><comment><USER11> Side note: you can set you Y axis format to "ms" in Grafana to make the values more descriptive and add the "Points" chart option under display styles to make the mean values visible, which are obscured by the 90th percentile bar in your chart. Also I assume label is wrong, it says 90ths percentile in the chart, but you speak of the 99th percentile.

<comment><USER9> Graphs are not mine, it's from Stripe engineers, so I've no control in how they are generated. About 90th vs 99th, I was talking about 99th but in the case of Redis latency spikes due to fork, you would get exactly the same graph as _all_ the requests are delayed in this moment.

<comment><USER11> Missed the "Stripe blog post" part, sorry for the misdirection.

<comment><USER9> np at all, thanks.

</comment></comment></comment></comment><comment><USER12> I'm kind of glad Redis did the fork approach first. It's the reason I went with a userspace COW implementation in my work instead of forking and that paid huge dividends. It's the difference between starting COW in 10-20 milliseconds versus seconds and most of that time is distributed coordination not flipping the COW boolean.

When you crank up the density per node to 256 or 512 gigabytes even bare metal is problematic and in some domains like telecommunications they don't care that the spikes are concentrated because they cause cascading failures.

I think a userspace COW implementation in Redis would be a big project because you would need a different strategy for every data structure. Being single threaded also makes it challenging to get other/more cores to do the serialization and IO. It's very doable just not within the current philosophy of thou shalt have one thread per process.

<comment><USER13> I think "userspace COW" is the wrong approach here.

The entire idea of getting a point-in-time snapshot of something subject to rapid change is problematic. Your options boil down to (1) make a "snapshot" and save that (Redis's current approach) or (2) accept that point-in-time consistency might be impossible, and work around it.

I wonder whether it'd be possible to have two persistence strategies in Redis: "consistent" and "low-latency". "Consistent" would use the current fork(2) COW behavior, "low-latency" would do some kind of one-chunk-at-a-time block copy and amortize the latency spike over the entire operation, while having the overall effect of less of a "cliff" to latency.

</comment></comment><comment><AUTHOR> fork test by redis labs (2012) [https://redislabs.com/blog/testing-fork-time- on-awsxen-infra...](https://redislabs.com/blog/testing-fork-time-on-awsxen- infrastructure#.VFJNvtaiPEI)

<comment><USER9> cc1.4xlarge is on pair with bare metal apparently...

<comment><USER14> That would make sense if the other EC2 Xen hosts were running PV guests (as was required for the smaller / older EC2 hosts IIRC? ) & the cc1.4xlarge was PVHVM, according to: [http://lists.xen.org/archives/html/xen- devel/2012-06/msg0102...](http://lists.xen.org/archives/html/xen- devel/2012-06/msg01021.html)

<comment><USER15> cc1.4xlarge was the first Linux HVM EC2 instance. All new families support HVM (and some are HVM only)

 _Should probably add a disclosure here. I ran the EC2 instance product management team for several years and continue to be involved._

<comment><USER14> Looks like a smoking gun then.

</comment></comment></comment></comment></comment><comment><USER16> Curious, why fork in the main thread? Forking traditionally is a pretty heavyweight operation. Perhaps versioning might be more performant?

<comment><USER13> Redis uses a reactive architecture using non-blocking I/O. They fork to get a point-in-time consistent snapshot that can be written to disk. The problem is that fork blocks, and while blocked, it stalls the event loop, subjecting incoming requests to stalls.

<comment><USER17> In Unix blocking means going into IO wait. In this case it's just slow when emulated ("paravirtualiuzed") by Xen.

</comment></comment></comment><comment><USER18> I know that redis really wants to be a persistent kvstore. I had a problem with a large website when I increased caching by 2 orders of magnitude (enough RAM to play with). When it came time to write a snapshot to disk, everything died for 5 minutes. Turned it off and haven't thought about it since. I'm not sure I'll ever shed my RDBMS predelections.

</comment><comment><USER13> This is a huge problem on wall street, where trades must have predictable latencies. Stop-the-world garbage collectors are another source of latency "catastrophes".

</comment><comment><USER19> Does fork not use copy-on-write? I would expect it to add overhead to all overlapped memory operations going forward, but am really surprised if it literally duplicated the entire memory contents.

<comment><USER20> Memory fragmentation seems to play a role too. Even if you have (barely) enough free RAM, machines can start swapping and a Redis server in that state is bound to wake you up at night... :(

<comment><USER21> If you run a server with anything but "swapoff -a" you're doing it wrong. (or even having any swap partition in /etc/fstab). If the server has to swap, get more RAM or scale your stuff out but never swap.</comment></comment></comment>


ReverseHttp - <AUTHOR> http://www.reversehttp.net/ <comment><USER2> There are two protocols described here. One is an extension to HTTP which allows the client and server to swap roles while still using the same connection. This would allow a server to "push" events to a client asynchronously. This will never work in a browser.

The other protocol tunnels an HTTP connection over another HTTP connection in the opposite direction. Tunneling asynchronous messages over HTTP is an old technique which can be implemented in a browser.

Neither protocol enables any kind of novel functionality. They merely add another layer of HTTP cruft.

<comment><USER3> Well, the intent is to design a systematic way of setting up a relay for HTTP requests from a public internet through a gateway to an application that otherwise wouldn't be publicly addressable. Without a protocol like the one I've drafted, setting up HTTP servers or CGI scripts stays ad-hoc, requiring local access to the gateway server and DNS and firewall configuration.

<comment><USER2> The problem could be solved much more generally with a protocol to request socket level forwarding of arbitrary network services. This could be used transparently to create a gateway for HTTP or any other protocol. Some existing protocols come close to doing this (e.g. SSH) but I don't know of any that handle public namespace allocation.

For the case of "works in a browser today" aka Comet, it is again better to solve the more general problem of bidirectional tunneling over HTTP (e.g. <http://xmpp.org/extensions/xep-0124.html>), through which you could make any sort of connection, including a reversed HTTP connection or the gateway request protocol above.

Imposing an extra HTTP layer and/or building on top of HTTP (in the first case) needlessly complicates and significantly restricts both of these protocols without deriving significant value from existing standards or infrastructure.

<comment><USER3> Check out <http://www.orbited.org/> for TCP-sockets in the browser (though I don't think it lets you act as a server yet). As you point out, nothing yet handles public namespace allocation, and this is key; for the specific case of TCP servers in the browser, port contention could become an issue fairly quickly, in which case lifting the level of abstraction to something like XMPP or HTTP (as I've done), where the addressing model is more flexible than TCP's, seems the sensible thing to do to avoid this.

XEP0124 ("BOSH") is very similar indeed to what I've defined; the differences are (1) BOSH is XML-specific and (2) it only provides a tunnel between the client (browser or not) and the server. What I've been experimenting with is content neutral, and, crucially, not only specifies the tunnel, but also specifies how the gateway server should expose the application at the end of the tunnel to the rest of the world. That's something that I have not seen before anywhere. (Except, as you mention, by SSH in a limited way.)

With regard to leveraging existing infrastructure: this is exactly why restricting ourselves to carrying HTTP over the transport is a good idea. We get to reuse _all_ existing infrastructure such as URLs, proxies, and of course the ubiquitous HTTP client libraries. Raw TCP sockets, even if the public namespace allocation issue were addressed, do not have an URL-like notion, and caching proxies do not exist; further, raw TCP access is in many environments not permitted or not available (e.g. within corporate firewalls, or running within a browser). Using HTTP rather than TCP is a deliberate choice to structure the network by providing not just a transport (packet- based, at that!) but a notion of addressing and a content model. HTTP out-of- the-box is a much richer protocol than TCP.

In conclusion, what I've proposed is in its transport aspect no more complicated than XEP0124, and in its name-registration aspect AFAIK not comparable to anything currently existing. The restriction to HTTP gives us an addressing model already widely supported and understood, and lets us reuse existing infrastructure and avoid needless reimplementation or reinvention.

<comment><USER2> URLs could be used with a generalized protocol. The client would specify the URL scheme, port and an arbitrary name and the server would generate and return a URL, or an error if it doesn't support the requested scheme (servers could support a very limited set of schemes and ports, perhaps just one). Raw socket endpoints would use "tcp://host:port" and "udp://host:port". Servers that provide raw sockets would probably want to create a subdomain for each endpoint to avoid port contention. Since the server knows the URL scheme, it can transparently do caching/filtering/mangling for particular protocols. Making a request with the "http" scheme would be functionally equivalent to your reverse HTTP.

This is just off the top of my head and there are surely better approaches but the point is that it's quite doable and probably as simple or simpler than something at the HTTP layer.

HTTP's "richness" is also what makes it a pain in the ass. It's a megalomaniacal protocol designed for a very specific purpose and when you are forced to use it for any other purpose, you have to carry a lot of baggage, and the baggage is full of rocks.

This gateway service is nearly always going to be used to create some sort of ad-hoc messaging endpoint, rather than a proper web server with web pages, so why force tunneling over TWO layers of HTTP while precluding the use of any existing wire-level level protocols?

It's time we buried the "use HTTP for everything" meme. We already have an everything protocol called TCP and if there's going to be a layer after that, it will be a carefully designed, flexible messaging protocol like AMQP. HTTP adds negative value as a general purpose transport and it's not even that great for serving web sites.

<comment><USER3> The generalized protocol/URL idea is a nice one. I'm not sure it'd be simpler, but it'd certainly be useful.

It's interesting you mention AMQP; this work I'm doing actually came out of some of the work I did as part of my work on RabbitMQ.

<comment><USER2> Ah, you work at LShift. A funny coincidence indeed.

I think the URL request protocol itself would be fairly simple but any use case would be application specific so coming up with a general purpose implementation might be tricky.

I'm just finishing off the Erlang book and for my first project, I was going to either build a general purpose Comet server (improving on Orbited, Meteor, cometd, etc) or flesh out the above protocol and implement it... or a combination of the two. If you want to offer input or be involved: jedediah at silencegreys dawt kom.

</comment></comment></comment></comment></comment></comment></comment><comment><USER4> The demo speaks for itself.

<http://www.reversehttp.net/demos/demo.html>

<comment><USER5> Case breaks things; entering a sub-domain name with uppercase letters causes the local server to fail. DNS names are supposed to be case-insensitive.

<http://www.rfc-archive.org/getrfc.php?rfc=4343>

<comment><AUTHOR> is that the fault of DNS or the actual server sitting behind it, expecting the host header?

<comment><USER3> It's the fault of the server reading the host header. I'll fix it now.

<comment><USER5> Cool. It normalizes to lowercase, but it's otherwise functional, even if the URL bar's case differs from the posted version.

</comment></comment></comment></comment></comment><comment><USER6> This is a different propopsal than the IETF draft by Donovon Preston, which is knows as "Reverse HTTP".

This is apparently "ReverseHTTP" and the specification looks about twenty times longer, needlessly. I'd love to hear any good reasons why I should use this "ReverseHTTP" instead of long-polling or Reverse HTTP... the IETF draft

<comment><USER3> It's very similar to Lentzcner & Preston's I-D, yes. We both seem to have independently invented the same general idea and chosen the same obvious name. The differences are the use of a ReSTful protocol, and more elaboration of the registration/name-management aspects. And of course that I haven't submitted it as an I-D :-)

My current draft is far too long, I agree; it describes not only the use of HTTP to retrieve requests (which is equivalent to Donovan Preston's idea), but also the interactions and headers needed to manage the tunnelled service. That latter is something that Lentzcner & Preston haven't addressed yet, I think.

</comment></comment><comment><USER7> Not sure I buy the premise "Polling for updates is bad." Certainly reverse http and/or web hooks do not cover, for example, all the same cases as http/atom/atompub. I'd like to see people's guidance on when to consider one or the other.

<comment><USER3> Perhaps more accurately, polling for updates in an event-based network is _suboptimal_ \-- especially since we have all this lovely packet-switching machinery available for use! -- but it's not completely wrong. Polling an RSS feed is equivalent to (a shitty form of) queue replication, and (slightly less closely) to TCP retransmissions, where event notification is equivalent to message delivery and to TCP segment transfer. The two approaches are in a sense dual. You can construct a message-streaming system from a state- replication system, and you can construct a state-replication system from a message-streaming system. Of course this still doesn't address when one or the other should be used: for that you have to get into the different scenarios for message replication. RSS/Atom etc are great when latency doesn't matter and you are multicasting, or when recipients desire (relative) anonymity. The cacheability of the polling approach can also be valuable.

Neither pull nor push solves the problem that SUP tries to address. For that, a layer on top is required -- essentially an embedded message broker with configurable private/shared queues and bindings. One very promising approach, once the transport is sorted out (which is what ReverseHttp is trying for), is to transplant the AMQP model (objects and operations) into the new setting.

</comment></comment><comment><USER8> Neat, almost a bit magical. But what kind of _common_ web development problems does this solve?

I'm sure there are a lot of contrived examples, but are there any _good_ ones? Facebook and Twitter.

Ok, but are there any _good_ and _common_ ones?

<comment><AUTHOR> Maybe there are no "common" uses just yet because we do not have an easily implemented way to do it. When Ajax first " came out", many said the same things. "great but what do we use it for." now, can you imagine the present day web without it?

<comment><USER9> I remember almost exactly the opposite about AJAX (when the paper came out, not when the XmlHttpRequest object was introduced) --- people went ape about what they could use it for. AJAX style genuinely made new things possible. This (supposedly) just makes them cleaner.

<comment><USER10> Are you questioning the relative merit of reverse http, or of the ability of the server to update the page?

I'm failing to see the advantage of reverse http over long polls, even though I am completely sold on the difference that having server push would make...

<comment><USER9> The relative merit.

</comment></comment></comment></comment></comment><comment><USER11> In safari, the loading spinner never stops. Not a good thing.

I don't really see what this is trying to solve either.

<comment><AUTHOR> I believe Comet works around this by using iframes, though I could be wrong. Either way, ReverseHTTP is another way to not have to poll for resources, which might be useful for instance in real time chat.

<comment><USER11> It's all in fuzzy definitions, but this is using 'comet'.

Comet is generally used to refer to any method that can emulate a raw socket - iframes, xhr, inserting script tags, etc etc

All this is doing is proxying http from the server, to the browser and back. It's an interesting thing to try, but I can't see any real life use for it.

<comment><USER12> This is much lower level and doesn't require javascript to work. (Ignoring the fact that the demo is an in-browser javascript http server). So, it's not comet.

The problem is that this isn't supported by any clients that are in wide- spread use, so you can't just load up a webpage and see a demo.

The advantage of this type of approach only becomes clear when you are using a lower-level http client library to access resources. It gives the server a chance to poll the client for information, without using Javascript. For browser approaches, this may not matter. However, for lower-infrastructure things, this approach is great.

I've used a very similar technique to link compute nodes to a job server where the compute nodes were behind a NAT. This eliminated any long polling required and still allowed the server to query the nodes for their status.

Again, not the type of thing where you're running anything in a browser, but I wanted to use HTTP as the protocol for simplicity, and needed a way for the server to talk to a client behind a NAT.

Now the down side is that you basically have to rewrite a web server in order for this to work. I'm not sure if this could be bolted on. You also need some sort of session management built in, so you can pair incoming (client->server) requests and outgoing (server->client) requests. And then you need a client library that can spin up it's own http server and handle it's own requests.

In my case, I was able to write everything from scratch. But I doubt my code would scale very well. I'm also not sure that in this case it isn't better to just make a new protocol. There is a lot of hackery required to get this to work, and I doubt you'll see web browsers support anything like this.

<comment><USER13> Don't get it. If you are interested in the low level communications, why wouldn't you simply use a socket and send your own application defined commands over port 80?

HTTP exists so that any browser can access any web server, it doesn't re- implement or otherwise allow the usage of TCP/IP.

As a corollary, I don't see why I need to know about your application's communication protocol, let alone adhere to it because it's now a standard.

<comment><USER12> We are talking about bi-directional communications between the client and server. Specifically, server initiated requests to the client. So the major issue that you can overcome with this would be overcoming some NAT/firewall issues.

This proposal would be to convert HTTP from being a client making requests to a server to (effectively) a server making / receiving requests from another server. So your browser would also be a (mini) server, handling requests from the main server.

This is largely for people that want to use HTTP as a message-passing protocol, but use it in a bi-directional manner between possibly NAT'd hosts.

<comment><USER3> "This is largely for people that want to use HTTP as a message-passing protocol, but use it in a bi-directional manner between possibly NAT'd hosts."

That is _exactly_ it. You've got it.

HTTP makes an almost ideal message passing protocol: it has a rich and battle- tested addressing model; it is asymmetric in a _helpful_ way (really! the response codes are similar to ICMP messages, where the requests are similar to IP datagrams); it is widely supported and deployed; it is content neutral.

It doesn't even have to be inefficient ;-) (<http://www.lshift.net/blog/2009/02/27/streamlining-http>)

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> How about just using hanging gets?

<comment><USER10> Maybe it makes some sense in server apis?

I don't know. I'm sold on using long polls in conjunction with actors...</comment></comment>


Show HN: Awsome – awesome one-liner deploy of static sites to AWS S3 and Route53 - <AUTHOR> https://github.com/dancali/awsome

<comment><USER2> Related: [http://stout.is/](http://stout.is/)

I don't plan to use any, but Awsome seems much easier.</comment>


Why Night Shift May Seem a Little Half-Hearted - <AUTHOR> http://mjtsai.com/blog/2016/08/15/why-night-shift-may-seem-a-little-half-hearted/ <comment><USER2> Federighi's response is nonsensical: LCDs don't use phosphors in the color filter plane.</comment>


The U.S. Lags in Buildings Made to Survive Quakes. Why? - <AUTHOR> https://www.nytimes.com/interactive/2019/06/03/us/earthquake-preparedness-usa-japan.html <comment><USER2> California has had dozens of earthquakes in the past few weeks... no damage at all.

This article is inappropriately titled. The US lags on _retrofitting_ older buildings to newer standards but that's only because we're don't tear our buildings down every 3 decades like they do in Japan.

</comment><comment><USER3> "US" is a bit of a broad term. I have never even experienced an earthquake, so I would say there is a decent portion of the US that has no need for such, likely expensive, renovations.

Almost wonder if this it a puff piece like the NYT's bit on 5g a few weeks ago.

<comment><USER4> There are so many earthquakes all the time. We probably were each hit by 100 this year, they just don't have a high magnitude, so we don't feel them. This has a nice map about hazard areas and other data: [https://earthquake.usgs.gov/monitoring/anss/](https://earthquake.usgs.gov/monitoring/anss/) Yearly overview: [https://earthquake.usgs.gov/hazards/hazmaps/conterminous/ind...](https://earthquake.usgs.gov/hazards/hazmaps/conterminous/index.php#2018) What the individual stations show: [https://earthquake.usgs.gov/monitoring/operations/heliplot.p...](https://earthquake.usgs.gov/monitoring/operations/heliplot.php?virtual_network=ANSS)</comment></comment>


Pricing a Simple, Useful Desktop App - <AUTHOR> I've grown tired of giving software away. So, I want to experiment with recouping some of the costs. I'm wrestling with pricing. I don't want to go too low as I'm concerned people will think it's cheap, and not too high as I want it to be easily affordable by average folks. I see value in micro-payments, but google check-out charges 2.9% + .30 cents a pop. So, I need the range (I think) to be $5.00 to $25.00. I feel both extremes are bad. Any suggestions? <comment><USER3> The single best advice I've ever heard for pricing: charge more.

Seriously. Whatever number you're thinking of, it is too low. Think of a number that makes you wince. That number is too low, too.

Many people here will suggest you charge like $5 or $10. These people are unwilling to buy your application at any price. They are not your customers. Their opinion on your price is irrelevant. (I mean this in the nicest possible way, guys.)

Charge for value. It isn't a "little" app or a "simple" app. Your customers are not programmers and do not know how many LOC it was (or, on the other extreme, how much loving care you put into it). They only see the value delivered to them. Price appropriate to the value.

People pay more money than you will ask, far more, for things which matter far less to them. Always remember that!

Charge more.

<comment><USER4> > _Charge more._

I understand that you meant that it's easy to underprice the app, but as a generic pricing guideline this advice is wrong.

Not considering running operational costs, the price should maximize (price x sales) figure, i.e. the revenue. It's true that in some cases doubling the price cuts the number of customers in less than a half, in which case the the "charge more" advice stands. But in other cases charging half of the current price may easily quadruple your paying userbase.

Finding the right price can be done only through trying different prices. At some point Amazon was giving random discounts to their users and trying to pinpoint the ideal price for the item. It didn't last long, of course, as people started gaming them, but it just goes on to show that the price validation _is_ a big deal.

Also keep in mind if you start with $200, see zero sales and then start gradually reducing it to $20, then you basically shoot your own credibility as a merchant as the $200 -> $20 drop makes you look greedy, detached from the reality and ultimately incompetent.

In the end it all very much depends on the application and the target customer base.

<comment><USER5> >Also keep in mind if you start with $200, see zero sales and then start gradually reducing it to $20, then you basically shoot your own credibility as a merchant as the $200 -> $20 drop makes you look greedy, detached from the reality and ultimately incompetent.

Are you suggesting that the other direction is preferable? I would think that it would be worse to keep jacking up the price.

<comment><USER6> I would say that you sell the program at several _different_ price ranges: call one Standard and one Enterprise, and another Home, and so on. They don't have to be any more different than the various versions of Windows are—that is, about 1% alteration, basically features flipped on or off in the shipped client using flags built into the reg. key.

Put them at drastically different price points. See which one(s) sell. Eliminate the price points/versions that don't, and roll their "benefits" upward (or downward, if you want to be nice.)

</comment></comment></comment></comment><comment><USER3> A domain name is $10 a year, and hosting at GoDaddy was about $4 a month back when I used it for selling downloadable software. "Recouping the costs" is setting your ambitions way, waaaaaay too low. (Indeed, if you're a professional programmer, the biggest cost _by far_ is the imputed value of your time.)

</comment><comment><USER7> Have you tried putting it online with a paypal donate button? That's worked wonders for our site and we make more than enough to break even. You'll find that firearms enthusiasts are generally a generous and thankful bunch and if you phrase a request for donation the right way they'll be more than happy to contribute towards your expenses.

I know of one who made a similar app for airgun users that is free but has nag screens but most everyone I know on the airgun forums have shot over (excuse the pun) the small fee to get rid of them out of gratitude for the app, which we all have found quite useful.

<http://www.chairgun.com/offset/chairgun2.htm>

</comment><comment><USER8> I would tend towards the higher price as long as your app is good value. You always have to compete the the "I could write that myself" viewpoint - <USER2> make it clear that the $25 will save the customer several hours work and meet their needs and you probably have a sale.

Try before you buy is the time honoured way of proving utility and value but that adds the cost of adding security to the app. Perhaps an unconditional money back guarantee?

<comment><USER9> If the software's any good, cracked warez versions will end up floating about anyway.

</comment></comment><comment><USER10> What does it do? What other software does the same thing and why is yours better?

<comment><AUTHOR> It calculates trajectory, drop, etc of a bullet depending upon caliber, grain, powder, etc. It's command line only. It's windows only. No GUI, but it's very simple to use and accurate. Very useful for folks who reload their own ammunition. Covers all calibers from smallest to largest.

<comment><USER11> What about making it an iPhone/Android/similar application? I envision people using it not just when reloading, but at the range, when buying ammo at the store, etc. I may be completely off in my opinion, but it seems the kind of thing that aficionados would drop a few bucks on. Some for making actual decisions; others to use as an adjunct/reference in conversation. And those phones are becoming a vast marketplace. For those using it for real work, other than reloaders, it would also place it at their disposal where they most need it: In the field.

Downside: Less of a percentage as your take. Upside: You're not hosting the store.

<comment><USER10> iPhone app; for some reason I don't think the intersection between gun owners and iphone owners is big enough to justify the cost of development.

<comment><USER12> I'd like to hear more on this cost of development. The biggest cost would be buying the developers license. His algorithms are already written, so it's just porting them to Objective C and slapping a Cocoa Touch interface to it (which shouldn't take more than a day, soup to nuts with no Cocoa experience). Plus you get the added benefit learning a useful SDK.

<comment><USER10> _I'd like to hear more on this cost of development._

The cost of picking up a new platform to learn. The cost of hours spent kluding an iphone development environment on whatever his current desktop is. The cost of writing said app. The cost of submitting it to Apple and waiting for how ever long it takes for approval. The cost of time spent polishing the site so it fits with the glossy, round-cornered aesthetic of the iPhone (possibly learning a graphics package or hiring a designer.) The cost of waiting for the pennies to trickle in from the _huge_ user base of gun-owning, bullet property measuring, badass iphone users who don't know of crackz sites.

</comment></comment></comment></comment></comment></comment><comment><USER4> I was going to say $19.95 (which is a de facto standard for moderately priced installable software), but then I read your description below and I think it just won't sell.

Expecting people to pay anything over $5 for a command line tool is unrealistic. And if it is priced at $5, then it won't sell because $5 is just not worth a hassle of paying for the users. Unless, of course, there is a streamlined purchasing mechanism. One click checkout that pre-fills the credit card info and such, but again that's not an option.

I would strongly suggest creating a GUI version of the tool and then it may have a chance of selling at $10 to $20 range .. though again .. know thy customers - people who make the ammunition at home typically do so because it cuts their costs. This means that you will need to somehow convince these cheap bastards that your app is worth paying for, and it's not an easy thing to do.

</comment><comment><USER13> I expect these kinds of apps - single purpose, but useful tools - TuneUp, or that Mac SSH filesystem thingy, etc, - to cost $15.

<comment><USER14> I would say $9.99 but $15 is good too. If I find something that genuinely helps me out, spending under $10 allows me to buy it without feeling guilty about spending money without asking the wife. Not that I need her permission, but in our relationship we clear most non-household purchases with each other if they cost more than $20. If I'm spending $10, that's like spending my lunch money. :)

</comment></comment><comment><USER12> I'd suggest $9.99 with a "trial". Give them a few calculations for free and then shut it down until they register.

I'd also suggest setting up a free web version but under a completely different name and domain. Slap on some relevant ads (Adsense, ebay, etc).

You're note going to destroy your market either way. You're not making anything right now, so experiment!

</comment><comment><USER15> This is a great indication of the potential of a desktop "appstore". If it works for the iPhone, why not desktop?

<comment><USER16> It always has? 99% of all software sold is for the desktop.

<comment><USER17> Mostly sold through brick-and-mortar retailers, though.

</comment></comment></comment><comment><USER18> Oldie but goodie: [http://www.joelonsoftware.com/articles/CamelsandRubberDuckie...](http://www.joelonsoftware.com/articles/CamelsandRubberDuckies.html)

In my startup one of the mistakes we made early was pricing things too low.

</comment><comment><USER19> $19.95 ? or $24.95 ??</comment>