A low-cost smartphone accessory that can detect disease markers - <AUTHOR> http://engineering.columbia.edu/smartphone-finger-prick-15-minutes-diagnosis%E2%80%94done-0

<comment><USER2> I work adjunct to this field and see this kind of gimmicky stuff all the time, a research group wants a second paper from an already existing technology. The cassettes will cost more than the phone, not the mention the labor involved in pushing a redundant diagnostic tool. Both HIV and syphilis have very cheap tests with quick turn around times [1]

[1] [http://www.medicinenet.com/hiv_testing/page3.htm](http://www.medicinenet.com/hiv_testing/page3.htm)

Results from rapid tests done in the doctor's office or at other points of care are usually available in 15-20 minutes.

<comment><USER3> From your link

> Results from rapid tests done in the doctor's office or at other points of > care are usually available in 15-20 minutes. If the rapid test is positive, > it is still necessary to send blood to a laboratory for a Western blot to be > sure the rapid test result is correct. Home tests are mailed to a > laboratory, and results return in one to two weeks. RNA testing results > usually take a few days to a week, depending on the lab.

From the article:

> Sia estimates the dongle will have a manufacturing cost of $34, much lower > than the $18,450 that typical [testing] equipment runs.

It's unclear from the article, but it appears that this avoids the need for the expensive re-testing? Would be good to have some clarification...

<comment><USER2> >>dongle will have a manufacturing cost of $34

Yeah but it uses disposables! Arnold O. Beckman made a killing selling salt water, imagine how much money we can make selling these things!</comment></comment></comment>


New: featurelist.org for upvoting and managing feature requests for any project. - <AUTHOR> http://featurelist.org

<comment><USER2> I like it. Thanks for making it.

<http://featurelist.org/projects/details/36>

<comment><AUTHOR> We didn't make this super-clear (other than a brief mention in the FAQ), but the basename in your project actually gives you a URL to use like this:

<http://featurelist.org/OurDoings>

For example, if you wanted to request features for the featurelist.org site yourself, you'd use:

<http://featurelist.org/featurelist>

<comment><USER2> I also noticed it in the help text for the form, but never saw the actual friendly URL because you redirect. I put in a feature request for you not to redirect. I hope others vote it up. :-)

<comment><AUTHOR> Heh, I voted it up myself. Likely will look into that.

</comment></comment></comment></comment><comment><USER3> I just made one(<http://featurelist.org/projects/details/35>) for my site Utility Mill(<http://utilitymill.com>)

Feel free to post some ideas for new utilities or improvements to Utility Mill itself.

<comment><AUTHOR> As I commented to you briefly on another site, the basename you choose when you create the project gives you a redirecting URL to share like this:

<http://featurelist.org/UtilityMill>

<comment><USER3> that is better, thanks.

</comment></comment></comment><comment><AUTHOR> This is a site by the makers of bug.gd (us) as a freebie for the open development community and hackers like those at news.YC.

</comment><comment><USER3> I made one for the iPhone too :-) <http://featurelist.org/iphone></comment>


Digg CEO cracks under Valleywag pressure - <AUTHOR> http://valleywag.com/359136/digg-ceo-cracks-under-valleywag-pressure <comment><USER2> If you've ever had the misfortune of having a run in with Valleywag. They're not exactly nice people.

I've actually stopped reading them because of it.</comment>


Google Design Resources - <AUTHOR> https://design.google/resources/ <comment><USER2> Looks like the brilliant minds at Google have never heard of <a> tags.

<comment><USER3> The sad part is that they _do_ use <a> tags! The site just does other stupid shit to break basic expectations about primary functionality, because "design! (handwave)". This is the kind of sewage that bubbles up when programmers learn about the web through single page app frameworks. There's a whole generation of front end developers now who seemingly don't know what a hyperlink is or why they might be worth using. Not every stupid website needs to be an app.

</comment></comment><comment><USER4> The state of design is in shambles. Every designer out there is heavily influenced by trends and frameworks. No one uses tables anymore even for legitimate uses. So much excess whitespace everywhere (Discourse forums vs. phpBB for example). Infinite scroll. Dark mode everywhere (hard to read text) and general copy-cat behavior from designers.

Every website today looks like Stripe, every startup has a magenta theme, gradients are back (even Apple embraced it).

What the fuck are designers doing without understanding the fundamentals of unobtrusive design? What happened to the principles laid out the Otl Aicher, Muller-brockmann, Anton Stankowski, Elliot Noyes and the likes - what we call the International Design movement?

I absolutely hate design of everything today. And the designers. And their ecosystems and frameworks. Javascript is the worst programming language of all. The whole thing reeks of amateurs running around making the world a noisier place.

Google's own design page doesn't meet their performance test. Last I checked, it scored 55/100\. Who is Google(and their designers) to tell the world how to design things? In science, we have to write papers, provide data and get the work peer reviewed. In design, you just publish and the louder one gets heard.

Sigh.

<comment><USER3> > _Last I checked, it scored 55 /100.

39.

> _Who is Google(and their designers) to tell the world how to design things?*

Indeed. They deserve giant middle fingers from all sides for this shit. I can't remember the last time I saw anything from design.google that wasn't a shitshow of bad practices and simultaneously pompous.

A flat site with a few links loads 400KB _after_ compression of javascript? Fuck all the way off, google.

</comment></comment><comment><USER5> And yet none of the 'cards' are links you cam middle click, ctrl click, right click on, but if you click on them they open in new tab.

</comment><comment><USER6> Anybody know what happened to Stage? Google acquired two great tools, Form and Pixate, and announced[0] that they are building a prototyping tool from both of these teams. That was the last we heard about Stage.

[0]: [https://design.google/library/design-never- done/](https://design.google/library/design-never-done/)

</comment><comment><USER7> The People + AI handbook is fantastic - full of well thought out, actionable advice on designing UIs on top of machine learning.

</comment><comment><USER8> Text entry fields with no border are the worst, if they are blank you can't even tell they are there.

The notes field on a contact in IOS is a good example of this unfriendly design. the amount of tapping I've done to try to figure out how to enter a note on a contact...

</comment><comment><USER9> I've heard a number of complaints that google itself did not adhere to the principles they are promoting, especially in material design.

</comment><comment><USER10> Should set up a redirect from google.design. If I remember this exists I'm going to randomly start typing variations of this.</comment>


The Python yield keyword explained - <AUTHOR> http://stackoverflow.com/questions/231767/the-python-yield-keyword-explained <comment><USER2> Thank you for linking this. It may not be hot news or a trendy topic but it was a very good read and a good refresher :)

<comment><AUTHOR> I'm glad you liked it. i've recently discovered the awesomeness of generators, and I'm starting to use them a lot in my code. I don't see them a lot in other people's code that I read, so I figure I'd spread the word a bit.

<comment><USER3> Although I agree it was an interesting read and I do like this language feature, I think generators should be used with caution since they might detract from the code's readability (especially generator expressions). Some async libraries like dieselweb just use them too much for my taste.

<comment><USER4> Generators actually allow you to decouple code in more ways than normal functions and classes let you. See the "Why functional programming matters" paper, or the more practical presentation here:

<http://www.dabeaz.com/generators/Generators.pdf>

<comment><USER3> Wow, excellent resource, definitely a must-read. However, I still stand by my opinion that there are more intuitive ways to control flow in python other than generator tricks.

</comment></comment></comment></comment></comment><comment><USER5> One of the most thorough explanations I've seen on any question.

</comment><comment><USER6> It's always nice when you finally understand "magic".

<comment><USER7> I found looking at the code that the C# compiler generates for its yield keyword to also be helpful for understanding this keyword.

Source of an iterator example: <http://gist.github.com/524972>

What the compiler generates for the above example: <http://gist.github.com/524974>

The meat of it is in the MoveNext() function, where the original function is split up along the yield keywords into cases in a switch statement.

</comment></comment><comment><USER8> His decorator explanation is also excellent.

<comment><USER9> Can I get a link to the decorator explanation? A quick browse through his history didn't reveal much.

<comment><USER10> [http://stackoverflow.com/questions/739654/understanding- pyth...](http://stackoverflow.com/questions/739654/understanding-python- decorators#1594484)

</comment></comment></comment><comment><USER11> I have it on my favorites, love it!, best explanation ever!

</comment><comment><USER12> loved the bank example !</comment>


Uber loses an average of 58 cents per ride – and says it's ready to go public - <AUTHOR> https://www.cbsnews.com/news/uber-ipo-uber-losing-an-average-of-58-cents-per-ride-and-says-its-ready-to-go-public/ <comment><USER2> The old line from the first .bomb:

"What we don't make in profit we'll make up for in volume!"</comment>


Happy feet: why a 61m-year-old penguin foot has researchers dancing for joy - <AUTHOR> https://www.theguardian.com/science/2017/apr/05/happy-feet-why-a-61m-year-old-penguin-foot-has-researchers-dancing-for-joy <comment><AUTHOR> My reason for posting this is not so much the article (which is interesting in itself) but the inclusion of 3 references at the bottom. This is something I would love to see expanded in newspapers in general, regardless of whether or not the references are available without subscriptions etc. When I was writing english and history essays in university the requirement to footnote most - and in the case of history, basically all - references, facts and third party opinions, forced me to make my writing tight, my arguments clear, and my conclusions reasonable.

Of course references for everything are not possible, and often the paper is its own source, but where it isn't I think we can make improvements here.</comment>


Pastor tries atheism, loses jobs, gains $19,000 - <AUTHOR> http://religion.blogs.cnn.com/2014/01/08/pastor-learns-the-price-of-atheism/

<comment><USER2> This is going to be interesting to watch. I wish him well.</comment>


Old box, dumb code, few thousand connections, no big deal - <AUTHOR> https://rachelbythebay.com/w/2020/05/07/serv/ <comment><USER2> Rachel presumably wrote her server in a reasonable language like C++ (though I don't see a link to her source), but when I wrote httpdito⁰ ¹ ² I wrote it in assembly, and it can handle 2048 concurrent connections on similarly outdated hardware despite spawning an OS _process_ per connection, more than one concurrent connection per byte of executable†. (It could handle more, but I had to set a limit somewhere.) It just serves files from the filesystem. It of course doesn't use epoll, but maybe it should — instead of Rachel's 50k requests per second, it can only handle about 20k or 30k on my old laptop. IIRC I wrote it in one night.

It might sound like I'm trying to steal her thunder, but mostly what I'm trying to say is _she is right. Listen to her. Here is further evidence that she is right._

As I wrote in [https://gitlab.com/kragen/derctuo/blob/master/vector- vm.md](https://gitlab.com/kragen/derctuo/blob/master/vector-vm.md), single- threaded nonvectorized C wastes on the order of 97% of your computer's computational power, and typical interpreted languages like Python waste about 99.9% of it. There's a huge amount of potential that's going untapped.

I feel like with modern technologies like LuaJIT, LevelDB, ØMQ, FlatBuffers, ISPC, seL4, and of course modern Linux, we ought to be able to do a lot of things that we couldn't even imagine doing in 2005, because they would have been far too inefficient. But our imaginations are still too limited, and industry is not doing a very good job of imagining things.

—

⁰ [http://canonical.org/~kragen/sw/dev3/server.s](http://canonical.org/~kragen/sw/dev3/server.s)

¹ [http://canonical.org/~kragen/sw/dev3/httpdito- readme](http://canonical.org/~kragen/sw/dev3/httpdito-readme)

² [https://news.ycombinator.com/item?id=6908064](https://news.ycombinator.com/item?id=6908064)

† It's actually bloated up to 2060 bytes now because I added PDF and CSS content-types to it, but you can git clone the .git subdirectory and check out the older versions that were under 2000 bytes.

<comment><USER3> > single-threaded nonvectorized C wastes on the order of 97% of your > computer's computational power

Can you elaborate on what this means exactly? For example, is there some reasonable C code that runs 33 times slower than some other ideal code? In what sense are we wasting 97% of our computer's computational power?

<comment><USER4> A good example of getting a ~3000x speedup from naive matrix multiplication in C here (slides 20 onward): [https://ocw.mit.edu/courses/electrical- engineering-and-compu...](https://ocw.mit.edu/courses/electrical-engineering- and-computer-science/6-172-performance-engineering-of-software-systems- fall-2018/lecture-slides/MIT6_172F18_lec1.pdf)

Includes a 9-level nested for loop, which is always great to see.

<comment><USER2> Thank you very much for posting this!

Roughly that 3000× is 18× from multithreading, 3× from SIMD instructions, 15× from tuning access patterns for locality of reference, and 3× for turning on compiler optimization options. This is a really great slide deck!

I was assuming "single-threaded nonvectorized C" already had compiler optimization turned on and locality of reference taken into account. As the slide deck notes, you can get some vectorization out of your compiler — but usually it requires thinking like a FORTRAN programmer.

So I think in this case reasonable C code runs about 54× slower than Leiserson's final code. However, you could probably get a bigger speedup in this particular case with GPGPU. Other cases may be more difficult to get a GPU speedup, but get a bigger SIMD speedup. So I think my 97% is generally in the ballpark.

A big problem is that we can't apply this level of human effort to optimizing every subroutine. We need better languages.

<comment><USER5> That's why you have people working on Halide, Taichi, DaCe, Tiramisu.

\- [https://halide-lang.org/](https://halide-lang.org/)

\- [http://taichi.graphics/](http://taichi.graphics/)

\- [http://spcl.inf.ethz.ch/Research/DAPP/](http://spcl.inf.ethz.ch/Research/DAPP/)

\- [http://tiramisu-compiler.org/](http://tiramisu-compiler.org/)

This way you can have a researcher implementing the algorithm (say bilinear filtering) and a HPC expert who tunes it with parallelism, SIMD, tiling.

I wrote an overview of most DSL for high performance or image processing in this issue: [https://github.com/mratsim/Arraymancer/issues/347#issuecomme...](https://github.com/mratsim/Arraymancer/issues/347#issuecomment-459351890)

<comment><USER2> This is great! Which of these do you think could be extended to general- purpose programming without the HPC expert? Taichi and DAPP seem to be aimed at that goal, but you seem to be implying they don't reach it yet?

<comment><USER5> You can use them without the HPC expert, Halide for example has a good autotuner and has been used by Google and Adobe to create image filters for mobile devices.

<comment><USER2> Thank you!

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> Whether intended or not, there's an undercurrent of "you're all so dumb for using Python" (or Ruby, or PHP, or other similarly performant language) here. I want to surface that and question it a bit.

It's totally reasonable for a company to choose the Python/Gunicorn option if they already have a bunch of people who know Python and they don't need to serve tons of requests per second.

Even if they do need to serve tons of requests per second, it's totally reasonable for them to still choose Python/Gunicorn if the cost of the additional servers is less than the cost of having to support multiple languages. Or if they get a lot of value from libraries that are unique to the Python ecosystem. Or if they care more about quickly iterating on features than driving down server costs.

I agree that there's a point where it stops making sense, and there are plenty of engineers who don't recognize when they're past that point because they keep doubling down on sunk costs and things they're familiar with. But let's not be too quick to assume people are in that camp when we don't know all the tradeoffs they're facing.

<comment><USER7> > Even if they do need to serve tons of requests per second, it's totally > reasonable for them to still choose Python/Gunicorn if the cost of the > additional servers is less than the cost of having to support multiple > languages.

How hard is it to get up to speed on any other tech stack? ASP.NET Core is extremely fast and the learning curve is close to none, for example.

If someone was able to wrap his head around backend development with Python I'm pretty sure they have the mental fortitude to onboard a tech stack that doesn't suffer from major performance problems.

<comment><USER8> 1) .NET APIs change very frequently.

2) C# is a very verbose language, that requires a lot of typing.

3) F#, the best language in .NET, is largely ignored by the .NET community.

<comment><USER7> > 1) .NET APIs change very frequently.

ASP.NET Core 2.1 was released in 2018 and will be supported until late 2021.

ASP.NET Core 3.1 was released a few months ago and there is no end of support in sight. Moreover, the changes between 2.1 and 3.1 were not that many. I've migrated a whole ASP.NET Core 2.1 web service to 3.1 in less than 1 hour.

> 2) C# is a very verbose language, that requires a lot of typing.

Nonsense. The only added verbosity to C# when compared with Python are the type declarations, which arguably are a problem plaguing Python. The first class support for events and async programming and properties in C# more than make up for it.

> 3) F#, the best language in .NET, is largely ignored by the .NET community.

I fail to see what point you were trying to make.

<comment><USER9> > ASP.NET Core 2.1 was released in 2018 and will be supported until late 2021.

Which is way too unstable, especially for the kind of corporate environment c# has typically been used in. Getting those places to upgrade to stable supported versions of the framework has always been a battle even when backwards compatibility was great, if they have to deal with breaking changes every few years they will never upgrade.

This is why so many companies stick with their ancient COBOL systems, most modern alternatives don't offer the stability they need.

<comment><USER7> > Which is way too unstable,

ASP.NET Core 2.1 is the LTS release of ASP.NET Core 2, which was released in 2017. I fail to see how a first class framework with a LTS that was released years ago can be described with a straight face as "way too unstable".

> Getting those places to upgrade to stable supported versions of the > framework has always been a battle

ASP.NET Core 2 is stable since at least 2 or 3 years ago, depending on how you decide to count.

> This is why so many companies stick with their ancient COBOL systems, most > modern alternatives don't offer the stability they need.

This assertion is simply wrong at so many levels. Don't confuse "why waste money maintaining working software" linesof reasoning as a sign of respect for stability.

More importantly, it's disingenuous to even think of the technical debt that keeps cobol on the map as relevant to the world of web services.

<comment><USER9> > I fail to see how a first class framework with a LTS that was released years > ago can be described with a straight face as "way too unstable".

I fail to see how you can call 2 years of support an LTS with a straight face, it's taking the piss out of the term. The LTS of the OS I'm likely to run it on is supported for 8 years. 2 years isn't even enough time to finish many projects on the same LTS it started on.

At work we've got 30 year old c/c++ code bases that still run, they'll probably run for another 20 at least, we've got 20 year old python code that still runs (for now) and we've got 20 year old c# projects that still run. That last one will never get rewritten in .net core, in part because they've pissed away the stability the framework had. It would be crazy to use tools with 2 years of support for any of those projects.

> Don't confuse "why waste money maintaining working software" linesof > reasoning as a sign of respect for stability.

Why should they waste money maintaining working software when there are stable options available? What does upgrading to asp.net core get them? Why should tens of thousands of companies waste money modifying working software just because someone on the core team thought the existing API was inelegant or too hard to maintain compatibility?

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> I think going back to basics would be a really good idea for a lot of people in software. It seems like modern developers are more disconnected than ever from the reality of the hardware situation sitting right next to them. I believe there was a post on the front page today detailing a certain 22ms Hello World execution...

</comment><comment><USER11> ```

First of all, it does not take "that much machine" to serve a fair number of clients. Ever since I wrote about the whole Python/Gunicorn/Gevent mess a couple of months back, people have been asking me "if not that, then what". It got me thinking about alternatives, and finally I just started writing code.

```

Another day, another questionable premise for a blog post. No, @rachelbythebay, the question is not "if not that, then what", and the answer is not reinventing the wheel. The question is just "what ___" \-- what do you plan on doing, what does your software need to do, what does it need to support? Use the right tool for the right job. If you have a language you're proficient in and with an ecosystem that supports you developing something rapidly, it's borderline malpractice not to start there. When you need to optimize, optimize then. Maybe that means you carve out a subcomponent into a new service, and you choose a language purpose built for speedily doing what you need. Maybe it means a lot of things, but it doesn't mean you throwing out the baby with the bathwater and setting out to recreate the baby, the bathtub, and the bathwater from scratch to answer the question of why your tub is overflowing.

I wish this blog post was about solving real engineering problems instead of writing code to provide mediocre answers to poor questions.

<comment><USER12> > If you have a language you're proficient in and with an ecosystem that > supports you developing something rapidly, it's borderline malpractice not > to start there.

I think the point of this post is that most new programmers do not _know_ things can be faster than their monstrous JS blob. The solution to slow requests is _more servers_ instead of fixing the code.

<comment><USER13> > I think the point of this post is that most new programmers do not know > things can be faster

Related to this: programmers who know an ORM or two but never learn SQL proper.

At my job I refactored a giant, slow, memory hungry reporting task into a single sql query. It used to take 10s of minutes to collate 1000s of datapoints. Now it takes 10s of _miliseconds_ to collate a factor 10 more data. Never mind after I added some indexes to speed the thing up.

Knowing about the layer below the abstraction you're working at can be rather useful at times.

<comment><USER11> What's telling here is that you refactored the individual task into a single query, and nothing outside of that. That's fantastic! You scoped your work into a small unit and derived a significant impact. You maximized your impact to effort ratio.

Notably, you did NOT decide to write your own dialect of SQL to "scratch an itch". Knowing the layer below the abstraction you're working at can be useful at times, but only in relation to understanding the context of how all the layers fit together and making effective, pragmatic decisions. Pointless rewrites are anything but. Good on you for avoiding that impulse and doing the right thing.

</comment></comment></comment></comment><comment><USER14> Compare this to the Enterprise platform I am dealing with on a current project, which has 4 x ec2 nodes with 8 CPUs and 32 Gb RAM.

I can DoS it with a single java client running 50 threads. If I use 100 the p95 shoots up to 30 - 40 seconds.

But the kicker is that no one (other than me) really cares. 50 concurrent threads is probably around the peak load it will get in prod, and various people involved think why bother trying to fix it?

It's driving me nuts.

Thanks for listening

<comment><USER12> Oh man, don't remind me. We have a bunch of GraphQL proxies in ECS that somehow cannot handle more than 5 connections each, so naturally the solution is to just spin up 19 more of them to get to 100 concurrent connections...

All of them sit at 1% cpu as well.

<comment><USER15> That actually made me laugh. The time savings can't possibly be worth it.

</comment></comment></comment><comment><USER16> The only endpoint to ever do anywhere near as little work as this example would be the heartbeat. Most of the cost comes from a combination of a bunch of things _other_ than simply fetching some data from a local device:

\- Web-anything is fetched off of a DB nowadays. That's another crapton of latency, because a) it's just easier to understand its characteristics if it runs on a separate system and b) most companies IME have either no DBAs at all or the DBAs have no time to look in depth at every system being built. So the DB resources are vastly underutilized, and the DB itself is badly understood.

\- Cache invalidation is _still_ the hardest problem, and every caching framework I've looked at seems to gloss over that part. Just cache all the things and hope people retry enough times to get the latest update. I would love someday to work on a system where things are aggressively cached at every level _and_ invalidated at every level and with perfect granularity.

\- Building for web scale from the beginning is premature optimization for the vast majority of companies. In the vanishingly unlikely scenario that the company actually grows 100-fold or more it makes sense to start investing heavily in performance. Of course, this also has the knock-on effect that the vast majority of software developers never get anywhere near a web scale system. OTOH it creates jobs for millions of developers, some of whom might end up building at scale someday.

Another elephant in the room is that building _anything_ at web scale is just not something anybody straight into the workforce is anywhere near qualified for. We desperately need more focused learning (mentoring, pairing, etc.) across the board to bring everybody up to speed faster.

</comment><comment><USER17> The typical response to these types of posts is "oh your /toy/ server doesn't account for x, y, z in my use case, like ddos, network issues, etc. But how many people _actually_ handle those cases in your production application? I can say for the majority of the applications I've written at large companies handling significant traffic API compatibility was far higher on the priority list than the cases that people often bring up.

IMO she's right, I wish we didn't mess wrap around gunicorn and gevent for some of our services. Certainly would've made my life easier and the services faster.

<comment><USER2> Also it sounds like Rachel's server does account for DDoS and network issues.

</comment></comment><comment><USER18> What WSGI do people recommend for python? I've been using gunicorn but this made me think of alternatives. Quick google search found this benchmark [0], is it really that bjoern is much quicker? It seems all other WSGI are ~ equivalent.

[0] - [https://www.appdynamics.com/blog/engineering/a-performance-a...](https://www.appdynamics.com/blog/engineering/a-performance- analysis-of-python-wsgi-servers-part-2/)

<comment><USER19> I'm using Waitress in production: [https://docs.pylonsproject.org/projects/waitress/en/stable/](https://docs.pylonsproject.org/projects/waitress/en/stable/) It's main thing is that it's really simple, worth a look if you're using Django.

<comment><USER20> going to look at this in depth tomorrow. Thanks

<comment> X-Istence The thing to note is that waitress is not built for the highest speed, or the fastest, or anything along those lines.

Its primary use case is that it is pure python, doesn't rely on any specific libraries or compilers to run/build, and is a threaded WSGI implementation so it uses Python threads to run a WSGI app.

It works well for what it needs to do, and hopefully it is fairly robust. I've personally ran waitress directly facing the internet, but will readily admit that in most cases running it behind a load balancer is a good idea, especially since it doesn't support SSL out of the box (yet, I should say, it's on my roadmap).

It won't win any speed contests and it won't win performance contests, but it holds its own.

If you have any issues, please drop by [https://github.com/pylons/waitress/issues](https://github.com/pylons/waitress/issues) and I'll see if I can help you out :-)

</comment></comment></comment></comment><comment><USER21> The general attitude here reminds a bit of the following post from the architect of the Varnish proxy. I think the attitude comes down to the fact that modern kernels and in general the foundations of network programming are pretty strong. We should trust them more.

[https://varnish-cache.org/docs/5.2/phk/notes.html](https://varnish- cache.org/docs/5.2/phk/notes.html)

<comment><USER22> I would argue you can say the same about foundations of rdbms systems as well. People build similar elaborate caches around those, for example rails has a "russian doll" cache layer built in, not realizing how much time has gone into developing well tuned caches within the database system itself, which simply needs to be allocated sufficiently large ram.

<comment><USER23> Russian doll caching also caches view rendering which can be non-trivial in Rails.

</comment></comment></comment><comment><USER24> Honest question: why go through the hassle of multiplexing waiting in a single thread only to dispatch to a thread per client anyway? Simply using blocking IO for the clients in those threads should be much simpler right?

<comment><USER25> If you get stuck in read(), you can't do neat things like waking up when it's time to kick a client for being idle, doing other housekeeping, or cleanly shutting down the whole thing in a timely fashion. When I ^C the server, it sends the same wake condvar-poke but it twiddles the flags so the worker shuts down instead.

<comment><USER26> Yes, using epoll with nonblocking I/O is better than blocking I/O on each worker thread. But that basically means that you are doing asynchronous programming--i.e., the exact same thing that the wonky Python/Gunicorn stack you described is doing! You're just doing it with better attention to important details.

Here, to me, is the key item:

 _The "listener" thread owns all of the file descriptors (listeners and clients both), and manages a single epoll set to watch over them._

This is exactly what any async server does: it centralizes all the file descriptor management and handling in one place, and only uses workers (whether they are threads or "green threads" or whatever) to read from/write to fd's that are marked as ready in the epoll set.

For your case, unless I'm misreading something, what the workers are doing in between the read/write is CPU intensive (or at least it's CPU work and not I/O work, even though it's not very "intensive" CPU work), so actual OS threads are a better choice for the workers since you can't rely on cooperative scheduling.

If what the workers were doing was I/O work (for example, sending a request to a remote database and waiting for a response), "green threads" would work fine (since their only real purpose would be to organize the I/O--the actual fd's are going to be managed by the central server that manages all the fd's and checks which ones are ready for read/write). And one definitely should not try to run "green threads" for the same server in multiple O/S threads (or worse still, multiple OS processes). For an I/O bound server, one shouldn't need to anyway.

<comment><USER27> Better attention to important details is what makes or breaks a library.

<comment><USER28> Exhibit A: Dropbox (attention to detail and executing it)

</comment></comment></comment></comment></comment><comment><USER29> > Ever since I wrote about the whole Python/Gunicorn/Gevent mess a couple of > months back, people have been asking me "if not that, then what". It got me > thinking about alternatives, and finally I just started writing code.

I actually want to know: then what? As a web developer who usually reaches for Django or Flask with Gunicorn because I just don't know any better, is there a better stack that doesn't face these problems? Or is this a 'call to action' for somebody to build a better web server that follows this advice?

<comment><USER30> > is there a better stack

Apparently, it's C, or ...assembly? According to the comment above yours.

Because developer time is apparently cheaper than CPU cycles. Weird.

<comment><USER31> Every time I read one her posts, I think, wow she must either work for literally dirt cheap or her code is expected to run on several thousand machines. Most projects just don't reach the scale or steadystate where developer time is cheaper than machine time. It's fun trying to squeeze the last drop of blood from stone, but it rarely makes economical sense.

<comment><USER32> To be fair, I believe she spent the guts of a decade at Google and Facebook as a Production Engineer, so this isn't really that surprising.

</comment></comment></comment></comment><comment><USER33> Actually, now that i'm re-reading this, it's a disaster. She claims to write this to help newcomers, yet everything is so cryptic.

`it kicks off a "serviceworker" thread to handle it.`

It doesn't even tell me how I should do this. I don't know what 'serviceworker' code looks like. I don't know what 'kicking off' means.

This post reads like it was made for maybe 3 or 4 people in the world who truly 'get it' and if you aren't in that elite club you're a terrible engineer, apparently. There's not even a lick of example code to get an idea of what is going on. Engineering is a big word that shouldn't be used in this blog post.

</comment><comment><USER34> The problem with this kind of "benchmark" is that it doesn't measure anything relevant for realistic situations where thousands of connections will behave in arbitrary unexpected ways, like just stalling, and real web applications have absymal worst-case behaviour in case of I/O, cron-jobs running in the background, network hiccups etc.. You don't provision your systems with only best-case situations in mind. But sure, if you want to serve useless random numbers without ever even hitting the disk, a Raspberry PI is able to saturate its network connection.

</comment><comment><USER35> The C10k problem was challenging around the turn of the century. I suppose it's now not. I wonder how much CPU would be saved using an event-based architecture.

<comment><USER36> Even in 2007 when I was starting to cut my teeth on larger web traffic there was a lot of discussion around serving 10k concurrent connections. I remember being blown away by a graph showing high throughput at 70k concurrent by a YAWS server, and started following Erlang as a result.

<comment><USER37> In 2004 we had eDonkey servers handling 1M concurrent connections.

Code, changelog and a bit of history: [https://lugdunum.shortypower.org/kiten.html](https://lugdunum.shortypower.org/kiten.html)

<comment><USER36> Oh, wow, I vaguely remember running across eDonkey at one point. I don't think I ever realized it could handle that kind of load. I was in a position to mostly stick with Apache for various non technical reason basically for long enough that eventually Apache got to the point that it could handle the traffic I needed to deal with especially with a CDN in front of it.

</comment></comment></comment></comment><comment><USER38> "at least one stupid chat client that's actually a giant piggy web browser running"

I wonder what app that could be :-)

<comment><USER39> I'll bite, is it slack?

<comment><USER40> In all honesty, there's a whole slew of platforms it could be:

\- Slack

\- Keybase

\- Mattermost

\- Discord

\- Microsoft Teams

\- Facebook Messenger (incl. Work)

All of these fine folks apparently use Electron or a similar technology...

<comment><USER41> Cisco Jabber also embeds a browser engine.

I think of Electron as Gresham's law in action.

</comment></comment></comment></comment><comment><USER42> I know pcwalton has been banging the "just use threads" drum for a while now. Rust used to use a green thread approach, but if I recall correctly, they ripped all that out for just native threads with the idea that in _most_ cases it's fast and efficient enough.

I remember when C10K was the big challenge, but even the naive approach of spawning a thread per connection now can handle that.

<comment><USER43> I believe the reason Rust dropped green threads was to get rid of the runtime system and become a more systems language.

Modern fast network code in Rust is based on async/cooperative multitasking distributed over num cpu worker threads which works very well.

</comment></comment><comment><USER44> Mmmmm. I set up a local kafka cluster serving millions of connections. I'm not Rachel by the bay though.

</comment><comment><USER45> And tomorrow there will be an article describing an exploit in a web service written in C/C++.

Does she even see the size of the batteries included in a framework like Django? Here is some brand new information. Everyone knows Python/Django is slow and inefficient (same for rails) but they still use it for those batteries.

It is easy to write some web service in a low level language that just crunches some numbers for your benchmark. It quickly gets complicated when you start thinking about accounts, security, passwords, databases, orders, carts etc.

</comment><comment><USER46> The premise is that people don't understand their system, or software, and need to be told how it works. So you have an admittedly ignorant person telling other ignorant persons that there are possibilities out there. That's... not really useful.

What is useful is to spend 2 days researching massively concurrent network applications, and find the ones that were already written, and see how they evolved over time. For the most part, it's just understanding your operating system and network protocols. Once you learn how they all work, the answers come quickly. This problem has been solved many times.

But for the most part, none of you need to know how this stuff works. You can write the crappiest network server in the world, and there are still so many other components you can wrap around it to make it scale that you never even need to get close to network optimization. So you may have to run 10 instances of your app; who cares? We have virtually unlimited everything these days. The answer to a poorly performing app is "just throw more cloud at it".

Also, I feel the need to remind everyone that single app instance handling 100K+ concurrent connections is a _terrible fucking idea._ What happens when the app crashes? What happens when the hardware dies? What happens when you need to, like, upgrade/restart the app? Several million SYN packets in 100ms, and default kernel TIME_WAIT settings, do not make great bedfellows.

<comment><USER25> Yeah, I know, I'm terrible. I don't know jack about this stuff. I should just stop now while I'm behind. Clearly.

</comment></comment><comment><USER47> Stuff like this worked great until someone tries to DDoS you, or found some buffer overflow which took over the box. I dont like the new world either but it was built this way for a reason.

</comment><comment><USER48> You must know a lot in order to write a slow performing program. If you just do a simple program like in the article, it will be fast, even if you do stupid things like spawning a new thread and file descriptor for each connection.

Now in order to make it slow you must know enough stuff to introduce complexity to the program. To make a slow program you probably have to learn about frameworks, php, micro services, cloud databases, etc.

</comment><comment><USER49> Would love to see the code.

<comment><USER50> Funny, when she revealed what the threads were doing, I wanted to search for Shakespeare in the output.

</comment></comment><comment><USER51> Old box, dumb code, few thousand connections, no big deal.

Old box, complicated code, few thousand connections, good fucking luck.

</comment><comment><USER52> should try this in erlang/elixir. i'm gonna bet you it could handle hundred of thousands of connections on a beefy machine (an million of connections w/ optimizations)

<comment><USER53> why tho? In real life if you're in need of handling millions of users per second, I bet you're already part of FAANG, at which point you simply open offices in each country and deploy local servers.

<comment><USER54> Doesn't WhatsApp story contradict to what you're saying? They handled millions of users per second, weren't part of FAANG and this helped them get acquired by FAANG and a high valuation for being a very nimble team and architecture.

<comment><USER53> how that contradicted me? I see this as exactly opposite, as in they became part of FAANG exactly because they managed to handle those users. And let's be real, they handled millions users/second after becoming Facebook. Like it or not FB is no.1 social network.

<comment><USER55> They got aquired in 2014. They hit 400M users in 2013. By then it was already THE messaging app for several European countries.

I know amount of users doesn't equal users/second. But we shouldn't pretend like they weren't able to handle high traffic before Facebook aquired them.

<comment><USER53> 400M in 2013 globally and millions per second does not equal. A simple math says 4x10^8 / 24 / 60 / 60 = ~4.7k users / second. Kind of 3 orders of magnitude lower. Even if you say most users are not spread evenly over those 24 meridians due to Earth being mostly water it will still not get to those millions per second.

<comment><USER54> In 2012 they were able to scale to 2 million tcp connections on a single box (and trust me, they actually needed that):

[https://archive.is/Jo6n](https://archive.is/Jo6n)

<comment><USER53> for how much time? you see lab result and real life results are different. Love it or hate it currently there is only one company that can deal with millions of users per seconds and that's Google. No one else does it, not Amazon, not Facebook including their whatsapp (a small math from my above calculation says the number goes from ~5k to ~50k if you say whatsapp has 4 billion daily users, which I doubt it has), not Microsoft.

Each time football Cup is active Twitter goes down. Same for plenty of big names when they launch a hyped service (Blizzard for example is another one). Scaling up from lab to real life dealing 24/7 with those millions/second is an entire different beast.

<comment><USER52> yeah no. a lot of people can do this. i don't know why you think google is special but it's not. maybe you can share so that i can understand your angle

<comment><USER53> already did. the bit about twitter and blizzard were 2 examples. you can google (pun intended) to search for more examples

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER56> Fastapi - a pleasant-to-use async python framework with strong type checking and lots of stuff built in - comes in at rank 68 on Techempower database update benchmarks. [https://www.techempower.com/benchmarks/#section=data-r18&hw=...](https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=update)

It's about 30% slower than the fastest golang equivalent.

For multiple-queries benchmark, its about 40% slower. [https://www.techempower.com/benchmarks/#section=data-r18&hw=...](https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=query)

Once I start bringing Numba for accelerating computational usecases... I guess the difference will be even smaller.

</comment><comment><USER57> Unfortunately this approach precludes GIL languages like Python. If you're able to use a language/runtime that is amenable to multithreading, then using a thread per connection works fine for most use cases (and it's probably easier than using whatever async/await interface your language has).

<comment><USER58> > probably easier than using whatever async/await interface your language has

Is it? I haven't used threads directly in a while, but I remember dealing with sinchronization issues. Problems that just don't exist in single threaded node with async await.

I find the async Promise or Task to be a more useful abstraction than the thread. Although, you need threads or a task dispatcher with a pool of threads if you need to run cpu intensive stuff.

<comment><USER57> If you're just handling requests there isn't much for shared state, so you don't have to worry about synchronization.

<comment><USER26> _> If you're just handling requests there isn't much for shared state_

No, but if your single listener/server thread is managing epoll for all your file descriptors, you do have to have a way of synchronizing the worker threads with it, so they know when and when not to read from/write to their fd's. I assume Rachel is using some kind of semaphore or other threading synchronization mechanism for this.

<comment><USER57> Right, there is some shared state, but it's very minimal.

</comment></comment></comment></comment></comment><comment><USER59> Does anyone have a link to the source code? I was thinking about writing some code in asyncio to compare but I couldn't find the code.

</comment><comment><USER60> That same old box could handle a million connections without the threads. In less than a hundred megabytes of ram too.

Try it today with NGINX.

</comment><comment><USER61> Thank you Rachel for saying loud what the erlang community say for years :)

This is basically the erlang design. And yes it works really well.

</comment><comment><USER62> uvicorn with quart in python 3.7 handle a 800-1000 req/s in gke :) porting flask to quart take a 2 months of work

</comment><comment><USER63> But this isn't engineering. This is the IT equivalent of building a bridge and driving successively larger trucks over it. In real engineering fields, you can do predictive analyses based on prior empiricism. There's none of that in our fields until you're talking about very small systems where, for example, the stack consumption can be determined in advance and the scheduler can give you guarantees about worst-case performance.

And this is to the detriment of all of us. That's why you'll never hear me call myself an engineer.

<comment><USER64> I'm glad someone else sees this the way I do. What the blog writer did was tinker with something. They didn't engineer it.

I was a mechanical engineer prior to switching to software. As a general rule, the things we do in software are very distant from engineering.

<comment><USER65> I think the analogy to mechanical engineering is something like this:

A seasoned engineer notices that everyone is only building suspension bridges all of a sudden.

They point out that you can span a stream with some bricks or rocks and a bit of mortar, and are ridiculed.

Next, they build a highway overpass out of concrete pylons, and stress test it to 10x the necessary engineering load, and point out it cost 10% as much as a typical contemporary suspension bridge. That's this article.

<comment><USER2> Or vice versa: they notice that everyone is only building concrete bridges all of a sudden, disregarding the traditional and much cheaper approach of hanging boards under some rope handrails; upon being ridiculed, they build a suspension bridge across a river for 10% as much as a concrete bridge.</comment></comment></comment></comment>


Ask HN: How did you attract contributors for your open source project(s)? - <AUTHOR> <comment><USER2> Generally there's the lossy pipeline of get people interested (1) -> give people a motive to contribute and obtain a first contribution (2) -> foster the contributions into long term collaboration (3).

Right now my primary target for attracting contributors is a synthesizer project. It's pretty interesting from the perspective of all the different working parts, though it does have sizable hurdles to attracting a large contributor base. Most users don't have a strong programming background and as such open source contributions appear to be too difficult (limiting 1->2). Individuals who are familiar with programming may still be unprepared for domain specific problems limiting the 1->2 transition as well.

Most of my attempts are trying to make the software more approachable in general, improving general internal quality, presenting the information to crowds who are familiar with it (multiple conference pubs), presenting it to groups mostly unfamiliar with it (HN/Reddit style threads/AMAs), encouraging community members who work on related sub-projects, using most popular hosting options, posting information on how to get started on the project's web presence, trying to convey that the project offers opportunities to learn skills rather than just apply existing ones, etc, etc, etc.

I'm not really sure which techniques really work or not since results are typically either delayed or are muddled with many different factors.

My newest attempt was based on a suggestion in a HN thread to list possible project roles (similar to if they were job listings). See [http://zynaddsubfx.sf.net/contribute.html](http://zynaddsubfx.sf.net/contribute.html) for that idea in action.

I'm not sure what the next effort will be, though I plan on discussing that on the project's mailing list this next week. Any suggestions or comments are welcome.

<comment><USER3> I'm the person who made the suggestion, just wanted to say that's a great page. It's really helping to figure out what are the desired contributions.

Only mildly related but I was actually looking for a synthesizer the past few weeks (I was just composing stuff I could actually play (and notate) myself before). Part of what was intimidating me about it was the user interfaces of that class of software are always super difficult to get started with... So I guess you gained a new user who does have a programming background!

<comment><USER2> Yeah, there's a lot of new lingo when you get started with synth interfaces (both from the design and usage perspectives). One of the goals I had for the new user interface was to start integrating more visualizations as a means of giving users feedback on what they're doing.

When you get started out, you just want to mess around an gain an intuitive feel for how to use something and what knobs do. Unfortunately many soft- synths miss out on that user feedback (since it's hard to do right).

</comment></comment></comment><comment><USER4> Typically supply a tool/app they need. Then hope they want to improve it. My project gets contributions from the community of users. What I would call "Drive by" contributors aren't a thing.

</comment><comment><USER5> need to have better documentation,

and should not have a complex code review/ commit procedure</comment>


The Nature of the Firm (1937) - <AUTHOR> http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0335.1937.tb00002.x/full <comment><USER4> This was written back when production cost dominated. The notes on scale don't really apply today, now that marketing cost and mind share dominate scaling.

Coase quotes Sir Arthur Salter: "The normal economic system works itself. For its current operation it is under no central control, it needs no central survey. Over the whole range of human activity and human need, supply is adjusted to demand, and production to consumption, by a process that is automatic, elastic and responsive." He's quoting Salter out of context. Salter was perhaps the first international central planner - he was in charge of Allied shipping during WWI.

Salter goes on to write, in his "Allied Shipping Control"[1], "By the exacting criteria of war conditions, this system proved to be, at least for those conditions, seriously inadequate and defective. By the new standards, it was blind and it was wasteful. It produced too little, it produced the wrong things, and it distributed them to the wrong people."

This is the fundamental critique of capitalism. The US produces about 4,000 Kcal of food per capita and still has hungry people. Even with huge productive capacity, the system doesn't deliver for the little guy.

Salter is describing economic socialism. Bertrand Russell once contrasted "Allied Shipping Control" with Marx's "Capital" as guides to the achievement of international socialism. Salter represents the bureaucrat's approach, as opposed to the politician's. Salter set things up so that the coal commission had to agree on coal policy across national boundaries, and the steel commission had to agree on steel policy, and then they'd go argue over shipping allocation. Arguments were by field, rather than by country. Marx is all about power - "Who tells whom what to do?", as Lenin put it.

Salter was one of the architects of the European Union, which started as the European Coal and Steel Commission, and a friend of Jean Monnet. He was influential in the design of the machinery of the Common Market. That's worked well; the EU's problems today are over immigration.

Coase and Salter were writing in an era when people considered these questions seriously. The Great Depression made it clear that unbridled capitalism could break down. All the productive capacity was still there, the workers still had their skills, yet people were starving. Something was fundamentally wrong with the system.

The Communists argued that their system was fundamentally better. (The failure mode of communism was that it centralized power so much that it encouraged tyranny. In retrospect, one can ask whether the problem was communism or just Russia. Since 1900, Russia has had a Czar with absolute power, a brief period of liberal communism that collapsed, dictatorial communism, bureaucratic communism, a brief period of liberal capitalism that collapsed, oligarchy, and now capitalist dictatorship. Russia without a strongman doesn't work. But that's another subject.)

The US coped with the Depression by trying to get capitalism under control, moving away from "greed is good, greed works". There was a lot of business opposition at the time. But not enough to stop it, for two reasons. One was that the Depression wasn't good for business, either. But the other was fear of communism as a competing economic system. It looked like communism was working - they weren't having a great depression. Capitalism had real ideological competition. This kept it honest.

This worked for a long time, from the 1930s to the 1970s. But by 1980, communism, as an economic system, wasn't working well. Nobody took it seriously any more. This freed up capitalism to go to its default "greed is good, greed works" mode, which is where we are today.

Now we have "secular stagnation" \- plenty of excess productive capacity, but we can't deliver a good standard of living to everybody. The new thing is that we need far fewer people to make production go. For most of history, the problem was making enough stuff. Coase wrote in an era when making enough stuff was still a big problem. We're past that now. And we have no clue how to organize an economic system that deals with it.

[1] [https://archive.org/details/alliedshippingco00saltuoft](https://archive.org/details/alliedshippingco00saltuoft)

</comment><comment><USER5> Coase is great. On a related note: not long ago, I discovered the wikipedia pages on "Internalization theory" and the "Eclectic paradigm", which descend from his ideas.

Those views deal a bit more with intellectual property and the fact that knowledge is a non-excludable good (as long as intellectual property rights are difficult to enforce). For me it's interesting to think of how this relates to giant tech companies, their secrecy, and internal-only software stacks.

(The eclectic paradigm relates to multinational corporations, which face some alternatives related to location/ownership/internalization.)

[1] [https://en.wikipedia.org/wiki/Internalization_theory](https://en.wikipedia.org/wiki/Internalization_theory)

[2] [https://en.wikipedia.org/wiki/Eclectic_paradigm](https://en.wikipedia.org/wiki/Eclectic_paradigm)

</comment><comment><USER6> Bar none my favourite question in economics, especially as it pertains to modern political dialogue. People claim central planning has been "proven wrong" while ignoring ie: Apple being very much centrally planned internally, which eliminates a ton of interesting discussion.

> _As D. H. Robertson points out, we find "islands of conscious power in this > ocean of unconscious co-operation like lumps of butter coagulating in a pail > of buttermilk." But in view of the fact that it is usually argued that co- > ordination will be done by the price mechanism, why is such organisation > necessary? Why are there these "islands of conscious power"? Outside the > firm, price movements direct production, which is co-ordinated through a > series of exchange transactions on the market. Within a firm, these market > transactions are eliminated and in place of the complicated market structure > with exchange transactions is substituted the entrepreneur-co-ordinator, who > directs production. It is clear that these are alternative methods of co- > ordinating production. Yet, having regard to the fact that if production is > regulated by price movements, production could be carried on without any > organisation at all, well might we ask, why is there any organisation?_

> _In view of the fact that while economists treat the price mechanism as a > co-ordinating instrument, they also admit the co-ordinating function of the > "entrepreneur," it is surely important to enquire why co-ordination is the > work of the price mechanism in one case and of the entrepreneur in another._

<comment><USER7> Yes, it´s a lot more complicated than that.

Fundamentally, it´s an issue of scale. It can be proven that central planning is the least efficient form of organisation at scale, because the information capacity of a hierarchically organised network will always be less than that of a mesh or federal mesh system.

However for small systems, those network capacity limits aren´t reached - <USER2> it doesn´t matter, and there´s also a joker in the pack for mesh networks, in that if they can´t self-regulate their traffic, the resulting broadcast storm (which socially we are now in worldwide) will drive their good-put (user traffic) to very low levels.

Besides the information capacity issue though, there are also considerations of command and control, so if you have a central planner that really knows what they´re doing (and the issue of information capacity is essentially that most central planners can´t), then that topology will be advantageous. Especially in competition with other centrally planned organisations - <USER3> Microsoft say.

<comment><USER4> We have so much information capacity now that's no longer an issue.

Gosplan, the Soviet state planning apparatus, ran on an monthly information cycle and a yearly planning cycle. WalMart runs on a daily information cycle and a weekly planning cycle. Amazon is near real time.

China, interestingly, goes to the opposite extreme. China is now on the 13th 5-year plan since the Revolution.[1] It's a set of 32 general policy goals to be emphasized for the next five years. These seem general, but this overall guidance determines on what the government spends time and resources.

[1] [http://news.xinhuanet.com/english/photo/2015-11/04/c_1347835...](http://news.xinhuanet.com/english/photo/2015-11/04/c_134783513.htm)

<comment><USER8> That's not the point. That information needs to be processed by a human being who then outputs decisions. If you have too much "big data", that data becomes useless because it overwhelms thereby making itself useless.

<comment><USER4> Only some of the information needs to be processed by a human being. How much inventory to carry of each item has been automated in retail for decades. Only unusual situations and new items need human attention.

</comment></comment></comment></comment></comment><comment><USER9> Coase's internal vs external transaction costs is a really good way of framing lots of tradeoffs between internally production vs outsourcing trade-offs. Should the government run jails or pay contractors to? Should apple build its own components or ship them in from China? It depends on whether internal or external transaction costs are higher

<comment><USER10> Over time, various economists have argued for additional factors. For example, there's often a degree of path dependence to why a given firm might be structured in a particular way. But, yes, especially for a paper that's almost 80 years old, its core concepts hold up remarkably well and provide a useful framework to think about sourcing decisions.

And it's often very clear (at least in retrospect) that those outsourcing trends that fall out of fashion over time often do so because many transaction costs associated with the outsourcing turned out to be ignored or at least underestimated. Conversely, there are countless examples today of companies using external services for things because the transaction costs associated with doing so have been so dramatically reduced.

</comment></comment><comment><USER11> Coase also had some fundamental insights into regulatory economics (insights which have been often misunderstood by modern neo-liberals). Two other excellent papers to read:

The Problem of Social Cost (1960): [http://econ.ucsb.edu/~tedb/Courses/UCSBpf/readings/coase.pdf](http://econ.ucsb.edu/~tedb/Courses/UCSBpf/readings/coase.pdf)

The Federal Communications Commission (1959): [http://web.ntpu.edu.tw/~guan/courses/Coase59.pdf](http://web.ntpu.edu.tw/~guan/courses/Coase59.pdf).

</comment><comment><USER12> 54 years later he got a Nobel prize for ideas that were essentially present in this paper.

</comment><comment><USER13> A small previous discussion from 2012: [https://news.ycombinator.com/item?id=3622639](https://news.ycombinator.com/item?id=3622639).

</comment><comment><USER14> Classic paper of a brilliant economist.</comment>


Qt 5.10 released - <AUTHOR> http://blog.qt.io/blog/2017/12/07/qt-5-10-released/ <comment><USER2> Glad to see the update. Most people associate Qt with GUIs which is unfortunate.

I see that when people think of Qt, they think of WxWidgets, Cocoa or MFC as alternatives. No, I submit to you that Qt framework is a more elegant, easier to use alternative to Boost as well.

This is not to say that QtQuick or QtWidgets aren't solid. However, the success of these two modules ends up occluding the others which to me are the real gems from the QtFramework: QtCore and QtNetwork.

QtCore provides a solid event loop, the most easy to use implementation of the observer pattern via its signal-slot mechanism, robust threading utilities and a bunch of other utilities that make writing apps in C++ an absolute breeze.

QtNetwork for a series of networking utilities that are elegantly simple.

If I were to write a command line app or a database or server, I'd reach for Qt in a jiffy.

Qt is not just for GUIs!

<comment><USER3> I really wish QtNetwork was better. I recently went and had to replace QtNetwork with our own event loop because it is extremely unreliable on Windows. The only mention of this is a footnote in the API docs:

[http://doc.qt.io/qt-5/qabstractsocket.html#waitForBytesWritt...](http://doc.qt.io/qt-5/qabstractsocket.html#waitForBytesWritten)

<comment><USER4> Yeah. It also turns out that _just having_ a Qt application running on a computer can tank network latency for _all_ applications on the computer:

[https://bugreports.qt.io/browse/QTBUG-40332](https://bugreports.qt.io/browse/QTBUG-40332)

There is a workaround, but developers have to know about it first. And it's such an unexpected and weird behaviour that 99% of developers don't.

This was our fix (1 line of code), in case that helps anyone:

[https://github.com/sqlitebrowser/sqlitebrowser/commit/739464...](https://github.com/sqlitebrowser/sqlitebrowser/commit/73946400c32d1f7cfcd4672ab0ab3f563eb84f4e)

<comment><USER5> Yup, i work on software with a Qt GUI and it took us ages to figure out why our daemon (no Qt dependencies) would experience latency when the GUI was open.

For Windows users we now put this var in to global environment via our installer, and probably break other software in the process.

<comment><USER4> > ... and probably break other software in the process.

You might also be fixing it for some too, if they're running other Qt applications. :)

</comment></comment></comment></comment></comment><comment><USER6> I've coded desktop apps (mostly editors and tools for games) for 25 years across X (Motif toolkit), Amiga, TOS, DOS, MFC, Cocoa, Win32, WxWidgets, Fox Toolkit and Qt. Qt has by far been the best, most rewarding, most empowering experience. It's a great library.

Now Cocoa would win but I prefer C/C++ to Objective-C, it's close though.

Good work people.

<comment><USER7> I'm very interested to hear your opinion on WxWidgets, as I have heard more desktop app devs endorse it over the years. What features makes QT win out over it?

Also, I know this might sound crazy, but what about (f/m)asm?

<comment><USER8> Not sure on Qt's features as I last used it in the KDE 3.5 days for basic basic apps, but I have used wxWidgets extensively. The Mac port has improved significantly recently, although there are still bugs.

It uses the native UI so fits in better with OSes, whereas Qt ones stand out to me (you can see the buttons aren't native). I did a lot of stuff that was "owner drawn" so reimplemented OnPaint to draw things myself for custom-look controls.

wxWidgets dropped its ODBC support a few years back so talking to databases likely needs another library. Also the ports/HTTP section of it isn't massively useful so I use libCurl instead.

Despite the bugs, it is quick to build and I found working with it enjoyable. Some controls (like the wxDataViewCtrl I think) were really slow so I wrote my own; the OpenGL wrapper works alright but I had to put some work into forcing it to resize; basically be prepared to put some work in - but it is rewarding. I like the layout mechanism - makes other systems like MFC look poor!

You can build wxWidgets and force it to use the STL for its container classes etc; Qt appears to implement everything again instead of just using the existing STL I think.

The developers are helpful and the forum is useful, at least for basic problems. I enjoy(ed) using it anyway.

</comment></comment></comment><comment><USER9> I'm an indie dev and have been developing a cross-platform (Py)Qt app for the past 1.5 years (~2100 dev hrs) [0]. Given that Qt is cross-platform desktop development, it's very solid. But there are a lot of things one has to do that are not required for (say) web apps:

* Creating standalone executables / installers for the app itself is already not so easy (I use - and recommend - PyInstaller [1]).

* Code signing the executables so users don't get an ugly "this app is untrusted" warning is tedious for the three different platforms

* Auto-updating is a pain to implement as well. I'm using Google Omaha (same as Chrome) on Windows [2], Sparkle on Mac [3] and Debian packages / fpm on Linux [4]. In total, I probably spent two to three months just on auto-update functionality.

* You really can tell that Qt is "drawing pixels on screen". Sometimes you have to draw pixels / perform pixel calculations yourself. The built-in "CSS" engine QSS works to some extent, but often has unpredictable results and weird edge cases.

I considered Electron as well. But its startup performance is just prohibitive. I blogged about this (and which other technologies I considered) [5].

I've been wondering for a while whether I should not open source my solutions to all of the above problems, to save other people the months required getting everything to work. Would anybody be interested in that? It would be something like a PyQt alternative for Electron.

[edit] People are very interested so I'm starting a MailChimp list. If you want to know if/when I open source a solution then please subscribe at [http://eepurl.com/ddgpnf](http://eepurl.com/ddgpnf).

[0]: [https://fman.io](https://fman.io)

[1]: [http://www.pyinstaller.org](http://www.pyinstaller.org)

[2]: [https://fman.io/blog/google-omaha- tutorial/](https://fman.io/blog/google-omaha-tutorial/)

[3]: [https://sparkle-project.org/](https://sparkle-project.org/)

[4]: [https://github.com/jordansissel/fpm](https://github.com/jordansissel/fpm)

[5]: [https://fman.io/blog/picking-technologies-for-a-desktop- app-...](https://fman.io/blog/picking-technologies-for-a-desktop-app-in-2016/)

<comment><USER10> > Creating standalone executables / installers

Python's Achilles heel, this is what caused me to walk away from Python. I usually make short very specific programs and I work on multiple of sites. Python was a pain to make executables. I now try to use Racket for everything and it makes executableson Windows, Linux and Mac.

Has things really improved the last 2 years?

<comment><USER11> I've seen people use Docker for this purpose, even on Mac/Windows. Personally, that seems a bit like using a howitzer where a flyswatter will do, especially since a lot of performance gains are going to be negated by Docker's virtualization on Mac and Windows.

</comment></comment></comment><comment><USER12> As good as QT and KDE are, I cannot really explain the fact that the most popular Linux distributions seem to show preference for Gnome; is Gnome more stable? is the user experience provided by Gnome more "polished"? or, is it the K-isms that push them away? is Gnome less resource-hungry?

<comment><USER13> While the official story is that it was a political issue (or "licensing hissy-fit" depending on your point of view), I often can't help but wonder if that was really somewhat of a cover for a bunch of stuck up C developers who simply hated C++.

Of course the one time I attempted to write something with Gtk+, it very much felt like a "We hate C++, so we're going to implement everything C++ does as conventions and macros on top of C" project.

In the early days, Linux/gcc also had some dynamic linking performance issues with complex C++ libraries. As a result, KDE apps actually did take a lot longer to start up than Gnome apps. However, once launched, the KDE stuff always felt like it actually ran faster. Thankfully those issues are long since fixed.

<comment><USER14> KDE is mostly developed by Europeans (Qt as well), Gnome (and GTK) mostly by Americans, though this has somewhat diminished over the years. The Gnome Foundation is registered in California, while KDE e.V. is a German non-profit.

KDE, in a sense, was the original free desktop.

> K Desktop Environment (KDE) was founded in 1996 by Matthias Ettrich, who was > then a student at the Eberhard Karls University of Tübingen. At the time, he > was troubled by certain aspects of the Unix desktop. Among his concerns was > that none of the applications looked, felt, or worked alike. He proposed the > creation of not merely a set of applications but a desktop environment in > which users could expect things to look, feel, and work consistently. He > also wanted to make this desktop easy to use; one of his complaints about > desktop applications of the time was that it is too complicated for end > user.

Gnome could be considered an American "but-its-not-all-under-my-gpl" response.

Personally I always found glib and friends always quite ugly from a technical standpoint, and the poor documentation and support for platforms other than Linux made me move to Qt quite early on when I started.

<comment><USER15> Dunno about KDE, as i can't say i have followed it much since the whole KDE4 mess. But Gnome/GTK is looking more and more Anglo-germanic, and not in a good way...

<comment><USER16> > But Gnome/GTK is looking more and more Anglo-germanic, and not in a good way

That's weird, AFAIK KDE development is predominately German, Scandinavian and Anglo and I was under the impression that many key gnome developers were South American.

</comment></comment></comment></comment></comment><comment><USER17> Is Qt a good, modern framework for desktop application development for a beginner? If not, can someone recommend something else?

<comment><USER18> Electron seems to be the preferred approach these days (much disliked by people who appreciate native apps). That said most Qt apps don't look native all the time either (it comes close enough)

<comment><USER19> Guess Qt gets more performance?

But more often than not Electron is good enough and there are even approaches to get JS on the desktop with less memory consumption than Electron.

[https://github.com/pojala/electrino](https://github.com/pojala/electrino)

[https://www.nidium.com/](https://www.nidium.com/)

<comment><USER20> Why would you even want to use JS on desktop if you have other options available? JS is the choice on the Web, because it's the only choice. It's the choice on Electron, because you can share the code base with Web version or make stuff like editors targeted to JS developers, so they can hack them while using them. But otherwise, it seems like the only worse option you could use for desktop apps is PHP. Why are people doing it to themselves?

Since we're in a submission about Qt: if you really don't want to touch C++ (which is kinda understandable, although C++ with Qt actually isn't that bad), Qt is very well supported in Python. Using them together is a breeze, and if you really do miss the craziness of JavaScript, you can still use it in QML :P

And if QML is what you only care about, you can even wrap it with Rust.

<comment><USER21> C++ is packed with undefined behaviour, complexity and cruft. I find JavaScript and extensions like TypeScript much more pleasant to work with.

<comment><USER22> > C++ is packed with undefined behaviour

This alone clearly states that you known nothing abou C++ beyond tired old, meaningless clichés about irrelevant stuff you've heard somewhere and are complaining about something you know nothing about.

<comment><USER21> How? Has C++ eliminated undefined behaviour recently? Yes, you can avoid it but it requires years of experience, a significant cognitive load as you work, linting/testing tools and you have to hope all the libraries you're using plus your team members have the same high standard that you're trying to follow yourself. I used C++ for years and knowing there are other languages out there I have no interest in making life hard for myself.

<comment><USER22> > How?

Because you're repeatedly showing profound ignorance on very basic aspects of using a programming language.

> Has C++ eliminated undefined behaviour recently?

You are aware that you're parroting on and on about an entirely irrelevant issue, don't you? I mean, any behaviour which has been left undefined in the international standard only means that there is a chance that two implementations may not implement the same obscure aspect exactly alike. And that's perfectly fine because these obscure corner cases shouldn't be used to begin with. That's what undefined behaviour does mean in practice.

But even if you for some obscure and irrational reason care enough to rely on behaviour left undefined then you also go another standard approach: adopt a specific implementation and check how the behavior was implemented.

Additionally, rust and go and python and other programming languages are not defined at all and somehow programmers don't get bothered by the fact that these programming languages are entirely undefined. But somehow you feel differently about a language you clearly know nothing about.

<comment><USER21> > But even if you for some obscure and irrational reason care enough to rely > on behaviour left undefined then you also go another standard approach: > adopt a specific implementation and check how the behavior was implemented.

By undefined behaviour, I mean things like null dereferences, buffer overflows, memory leaks etc. that are done by accident and result in serious issues at runtime. I'd rather use a safer language where I can be more productive where possible.

Sounds like you're talking about implementation defined behaviour or unspecified behaviour which are different from undefined behaviour (these three terms are defined in the C++ standard document).

> But somehow you feel differently about a language you clearly know nothing > about.

You're not even trying to understand the point you're leaping to attack. You clearly have a chip on your shoulder about something.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> This is awesome. I can't wait for the KDE binding generator to mature so we can finally start using it more properly from Rust.

</comment><comment><USER24> I enjoy using QML together with Python 3 for the logic, with the PyOtherSide plugin: [https://thp.io/2011/pyotherside/](https://thp.io/2011/pyotherside/)

</comment><comment><USER25> Is Qt 5.10 still stuck in the time prior to C++11? Does it have clear pointer ownership and move semantics?

<comment><USER26> Qt has moved into C++11+ world for quite some time. IIRC 5.7 was an important step in that direction, and they keep moving that way.

Move semantics are much better. Not sure what you mean by clear pointer ownership, though.

<comment><USER25> `unique_ptr` and `shared_ptr` semantics

<comment><USER26> Qt does have QSharedPointer and QWeakPointer. Also, you can use QScopedPointer for things like implementing RAII with custom deleters, etc.

</comment></comment></comment></comment><comment><USER27> Qt used to be an absolutely awesome C++ framework. Then Nokia happened, and it turned into a weird Rapid Development Environment™ for Symbian®©.

Does anyone have up-to-date experience with Qt? It's quite clear Symbian is no longer the main target, but did they ever get back to treating C++ as a first- class citizen, or is it still all about QML?

<comment><USER28> Not sure what you're talking about. Qt 5 didn't remove anything that was there in Qt 4 [1], and added quite a few nice things like native Wayland support and more OpenGL integration.

Sure, the main development focus is on QML etc., but that's mostly because desktop is a more mature platform than mobile.

[1] Except for the QtWebkit to QtWebengine transition perhaps. I haven't followed that very closely.

<comment><USER27> The main development focus is the very thing I was curious about. I remember Qt being slow to support new things eg. high dpi, unless you were using QML.

Getting the "feel" of a company's priorities can be hard, if you're not actively using their platform. I was hoping someone on HN had a somewhat realistic picture of where Qt is going these days.

<comment><USER29> It looks to me like they're very underfunded and have completely lost focus. The high DPI "debacle" is a good proof of it.

<comment><USER30> Sure /s. We have a Qt app with GUI completely written in Qt. Scales pixel perfect on HiDPI out of the box as long as you make sure to not hardcode pixel values.

</comment></comment></comment></comment></comment><comment><USER31> November 2017: [https://news.ycombinator.com/item?id=15617359](https://news.ycombinator.com/item?id=15617359)

>pknopf: _This reminds me of the project I am currently working on.

.NET/QML [https://github.com/pauldotknopf/net-core- qml](https://github.com/pauldotknopf/net-core-qml)

Not quite production yet, but it will be soon. I'd love to he[ar] some input. You can check the unit tests for how things are working currently._

</comment><comment><USER32> Haven't used QT in a long time.Have they done away the with 'moc' and code generation? I found it extremely annoying that while we were using QT we weren't actually using c++, but a strange, language that looked like c++, but was actually further processed by qt to generate the c++ code. Are they done with that shebang?

<comment><USER33> Without moc their is no signal slots and the handy stuff you get from plopping Q_OBJECT in your derived classes. So it's not going anywhere.

<comment><USER32> Can't they figure out a way to do that in pure c++ yet? In other words why does C++ lack reflection, and only seems to have a half baked RTTI? Perhaps the QT guys and other c++ application programmers didn't lobby C++ standards committee strongly enough for that?

<comment><USER34> Sure, but there's 1) a performance hit because things that can be done at compile-time currently, have to be done at runtime, and 2) syntactically verbose/cumbersome. They've decided that these things are not worth it to most people.

Besides, it still is C++: you can totally compile everything without the moc (although its probably not very useful to do so), since the extra keywords are just #defines.

I've personally never had any issues with the moc (but I did only ever use QtCreator/qmake, so...)

</comment></comment></comment></comment><comment><USER35> This is also an interesting project, if you want to use Qt with Rust:

[https://phabricator.kde.org/source/rust-qt-binding- generator...](https://phabricator.kde.org/source/rust-qt-binding-generator/)

</comment><comment><USER36> BTW, if you like both QML and Common Lisp, and want to reach for android, there's news: EQL5-Android [https://gitlab.com/eql/EQL5-Android](https://gitlab.com/eql/EQL5-Android)

</comment><comment><USER37> What JavaScript engine is Qt 5.10 using, and have there been any recent changes to that engine, or the way it's integrated with Qt?

</comment><comment><USER32> Annoying thing with QT, the official pronunciation sounds like 'cute', and some interviewers insist on calling it that.

<comment><USER38> Lots of people in the company call it 'Q' 'T' and lots call it 'Cute' and there is no "official" way it is called.

<comment><USER32> See Wikipedia: [https://en.m.wikipedia.org/wiki/Qt_(software)](https://en.m.wikipedia.org/wiki/Qt_\(software\)), it says pronunciation is 'cute'.

</comment></comment></comment><comment><USER39> R.I.P. QtWebkit

</comment><comment><USER40> Andy Brice has two successful desktop app products, Perfect Table Plan and HyperPlan [1], and both are written using C++ and Qt, IIRC; I read that on his blog [2], which I have been following for some time now. Lots of good info about product development and marketing there.

[1] At least, Perfect Table Plan is quite successful, he has been selling it for a long time now. HyperPlan is newer, but IIRC he had some sales for it too.

[2] [https://successfulsoftware.net/](https://successfulsoftware.net/)

<comment><USER40> And seeing a few comments about toolkits being cross-platform, reminded me: his apps work on at least both Windows and Mac (not sure about Linux, but I do remember the Qt version I tried earlier works on Linux too).

</comment></comment><comment><USER41> Awesome cant wait to check out the new features!</comment>


Containers and Storage: Why We Aren't There Yet - <AUTHOR> https://thenewstack.io/containers-storage-arent-yet/ <comment><USER2> There's a lot going on in this article, from a lot of different ends.

One of the big tensions (among many) that I'll focus on is rootless containers. Frankly it's a bit silly to think containers are useful when you need root on every system to use them- do people on academic systems simply not get the ability to use containers? As we move towards multi-tenant, "hyper-converged" systems (things running lots of workloads), figuring out who/what to bless as OK is a critical bottleneck, one that needs a lot of trust.

So the idea has been gesticulating around of "rootless" containers[1]. There's a lot of recent excitement and work- @lordcyphar has started a dedicated website for the idea, [https://rootless.containe.rs](https://rootless.containe.rs) [2], which gives a broad status overview of the pieces. The "runc" alternative to docker merged rootless recently, after a year in progress[3], and recent work focuses- as the article hints at- around the filesystem side of things. He's building orca to build oci standard images, and although strongly hinted[4] I'm not sure how exactly that factors in to supporting rootless.

[1] [https://www.cyphar.com/blog/post/rootless-containers-with- ru...](https://www.cyphar.com/blog/post/rootless-containers-with-runc) [2] [https://rootlesscontaine.rs/](https://rootlesscontaine.rs/) [3] [https://github.com/opencontainers/runc/pull/774](https://github.com/opencontainers/runc/pull/774) [4] [https://twitter.com/lordcyphar/status/849951277744181248](https://twitter.com/lordcyphar/status/849951277744181248)

</comment><comment><USER3> I don't appreciate using poor people picture to represent the awkward state Linux containers.

<comment><USER2> Not a great take. Your bias that rickshaw == poor isn't kind either. But I agree that the inference shown isn't a good one.

<comment><USER3> >Your bias that rickshaw == poor isn't kind either

why else would they be risking their lives by riding 12 to a rickshaw?

<comment><USER4> I think you are in a better position to answer your own question than anyone, given it's a leading question. I would rephrase that it's not "kind" or "unkind" but actually _blaming_ to a) be offended for a group of which you are not a member (riding rickshaw) and b) assuming the group that is riding it, is poor. For all we know, they may be doing better than the people walking on the street behind them.

Now, if the photo showed someone removing choice for another, or removing choice for a group, I could see the point of calling the use of the photo into question. In this case, however, I think pointing out the straightforward, yet slightly unsophisticated mode of transportation might do well to describe the container space. That's speaking from my experience with riding in such a vehicle, in such a city, and having had to deal with the container space, and the people in it, a group.</comment></comment></comment></comment>


MemSQL Does Oracle's Own Demo Ten Times as Fast, Sixty Times Cheaper - <AUTHOR> http://blog.memsql.com/memsql-does-oracles-own-demo-ten-times-as-fast-sixty-times-cheaper/

<comment><USER2> It is generally not that difficult to greatly out-perform Oracle on specific benchmarks. Postgres can do it too. Companies do not buy Oracle for the performance per se but as time has gone on Oracle has definitely been losing ground in that arena.

MemSQL makes a good product but nothing about their architecture is particularly novel, and other systems have a similar design. I've designed several database engines; I would _expect_ numbers like they demonstrated on a decent columnar implementation with the tradeoffs MemSQL has made. And Oracle has never been optimized for these types of queries because those tradeoffs adversely impact performance in other areas Oracle cares about. Features like compiling queries are mundane; many commercial databases did that a decade ago and it is a standard design element today. Same with the two-tier in-memory and columnar disk storage.

I'm sure MemSQL is fast, but "world's fastest" should be assumed to be hyperbole. Most of their benchmarks are table stakes for a modern, real-time analytical database kernel. It is apparently a well-designed product but nothing many other companies are not doing.

Here is what I tell people a properly designed database kernel on modern mid- range hardware should be able to do _per server_ : continuously insert millions of records per second all the way through storage, leaving virtually all of the cores idle for some large number of concurrent queries. If you can't saturate 10 GbE on ingest concurrent with saturating 10 GbE on query output, your design is most likely obsolete or broken.

Things have moved a very long way since I first started designing massive scale database systems (on Oracle, way back in the day).

<comment><USER3> What in your opinion counts as modern mid-range hardware?

<comment><USER2> The physical servers used in clusters I buy today run about $4k per server without storage. Outside of the dual 10 GbE interfaces, they are like most other budget servers. Storage costs per node vary depending on the size, type, and mix of disks. Driving the disk I/O at 10 GbE can be trivial with a good I/O scheduler design, so ironically the details of the storage device are almost irrelevant for many workloads. Cheap SSDs are the default target but cheap spinning disk works surprisingly well for many workloads.

Various AWS nodes are also used in clusters quite a bit (though we use ephemeral disks rather than EBS). Nothing exotic.

</comment></comment></comment><comment><USER4> There is a lot of use case optimization going on here. The sorting inside the columnar store is in and of itself a huge performance boost. And saying that it scans 138Billion rows per second is completely wrong because it uses a lot of short cuts to get the result and it doesnt even use rows to store the data.

There is nothing stopping Oracle using the same optimization for their use case, although they would be better suited if someone from the crowd queried something that MemSQL didnt sort/optimize for.

<comment><USER5> It's hard to define what a "fair" columnar store scan is. We use a technique that is called "segment elimination", which allows to skip over compressed segments of data as you scan. However every solid columnar database uses such techniques. It just happens that Oracle 12c doesn't have the ability to sort columnar indexes, and neither does it have the ability to store columnar indexes on flash or disk, which bloats the overall cost.

<comment><USER6> The much bigger problem with Oracle In-Memory is that a lot of operations like joins and sorts often happen in the SGA, in row form. So you end up needing a truckload of SGA for temporary calculations. And when the SGA runs out, which it invariably does, you end up using TEMPDB. And everything comes to a crawl.

</comment></comment></comment><comment><USER7> MemSQL is doing great work...However from a strategic point of view, I wonder how they plan to compete against 2 of the world's largest DB vendors (SAP and Oracle) at their core product. Oracle's flagship DB has more R&D, sales, marketing than a startup could ever muster even with the help of big name VC's. But I'm sure that's part of startup glory. "no one thought we could do it!" is how it always starts. I suppose they could be an acquisition target for HP or EMC or a company thats lagging in the data management world

<comment><USER8> They will be able to pull in customers who cannot afford to spend 3 million on hardware for a database but still need very high performance. There should be enough to build a business in that niche.

Those who can afford the high price tag will likely continue to buy from Oracle and the SAP in the short term. Over time, MemSQL can probably win some of these customers as they establish more credibility and a proven track record.

</comment></comment><comment><USER6> Doing some basic math... Wikipedia is around 80m rows a day, so 4 months of Wikipedia is around 9.5bn rows. But they show 17bn on the graph.

Typical columnar compression gives about 11GB per 1bn rows, so 17bn rows should be 187GB. The AWS machines they are using should be c3.4xlarge which are 30GB, and 6 of them is 180GB. But you can't run an in-memory column store at 100% RAM, you need to run it at 50-70% so you have capacity for calculations.

Is it just me or do the results not make any sense? Seems likely they actually had 9.5bn or 4 months data, which conveniently is what the graph shows?

</comment><comment><USER9> is memsql a free product, I can seem to find anything about pricing or otherwise on their site ?

<comment><USER10> The 60x cheaper benchmark is based on hardware costs, comparing 6 EC2 nodes (they say ~$20k worth of servers) vs a > $million Oracle server.

</comment></comment><comment><USER11> I'm amused that Nikita Shamgunov's bio suggests that he was a 'distinguished senior database engineer' at Microsoft. The title "Distinguished Engineer" at Microsoft refers to a pretty select group of people, not just any senior dev

<comment><USER5> That's right. "Distinguished Engineer" is a title at Microsoft. At the time of my tenure at Microsoft my title was "Senior Engineer". The bio was written without referring to the title, but rather the fact that I was "distinguished" with Gold Star awards and HiPo program. I asked to remove "distinguished" from the bio to remove any confusion. It's already live.

<comment><USER11> This is a classy response. Corporate bios, like any pr, can walk a fine line in what they communicate.

</comment></comment></comment><comment><USER12> I was hoping the blogpost would give at least a pointer or some insights about the actual experiment. Am I missing some context here?

</comment><comment><USER13> I wonder if it's a good idea to list comcast as one of you clients...

<comment><USER14> Sure, why not? It's a big company with a lot of brand recognition. The people most likely to be interested in this kind of product will first think 'oh, they have a big client!' before they think 'oh, they are used by a company I dislike because of anecdotal data or their stance on so called net neutrality.'

Heck, I'd be even more interested if they told me they serve the NSA or Palantir, as much as I dislike everything they stand for.

<comment><USER15> This would be of interest to you then: [http://siliconangle.com/blog/2014/09/29/heres-why-the- intell...](http://siliconangle.com/blog/2014/09/29/heres-why-the-intelligence- community-bought-a-stake-in-memsql/)

<comment><USER14> Ohhh... interesting. Thanks for the link!

</comment></comment></comment></comment><comment><USER16> A font with strokes 1px thick when you _scale it to 200 percent?_ What a great design! I'm sure it looks so cool on the designer's retina display.

You don't want me to read your page? Fine, then I won't.

Edit: Hehe, now the contrast of my comment is about as bad as the contrast of the blog. I guess criticizing Silicon Valley's favorite design fad of the 2010s is too much for the HN crowd to stomach.

<comment><USER17> That's just the 300-weight version of the Lato font, one of the most widely used fonts on the internet.

If you have trouble reading it, you may need a new monitor or get your eyesight checked.

<comment><USER16> _> you may need a new monitor_

No, I'm actually perfectly fine with my Eizo. It's not its fault that it isn't able to show half-pixels. That's by design.

 _> the Lato font, one of the most widely used fonts on the internet_

I seriously doubt that this is the case. Last time I stumbled across it was heartbleed.com, and I had to use the F12 console to de-Lato the page, that's why I remember. That was half a year ago, thankfully.

If you spend your day reading about cool parallax scrolling plug-ins for your site, then you might see a lot more Lato than I do, yeah.

<comment><USER17> > I seriously doubt that this is the case

[http://www.google.com/fonts#Analytics:total](http://www.google.com/fonts#Analytics:total) . 5th most popular font at the moment ( 2,059,357,755 views for Lato in the last 7 days and 64+ billion in the past year).

And that doesn't even include the Typekit deployments (or locally hosted ones).

<comment><USER16> _> 5th most popular font at the moment_

So, Google Web Fonts now is the benchmark for professional screen typography?

None of the sites that I visit on a regular basis use Google Web Fonts. The only one using any webfont for body text is The Guardian, which uses an excellent custom font designed by Christian Schwartz and Paul Barnes.

<comment><USER17> > So, Google Web Fonts now is the benchmark for professional screen > typography?

It's certainly a better benchmark than "none of the sites that _I_ visit on a regular basis"...

And if hard numbers like 64 billion+ views per year don't convince you then I won't bother.

</comment></comment></comment></comment></comment></comment><comment><USER18> The name of this makes me think of that MongoDB is web scale video[1]'s comment, 'Is /dev/null web scale?'.

[1] [https://www.youtube.com/watch?v=b2F-DItXtZs](https://www.youtube.com/watch?v=b2F-DItXtZs)

</comment><comment><USER19> Utterly stupid post from a company who should know better. The biggest reasons that companies buy expensive appliances is (a) data sovereignty and (b) ongoing support. The cloud e.g. AWS does have decent answers for this yet.

And the companies that are buying these types of appliances like we do couldn't care less about a few hundred thousand dollars.

<comment><USER20> > _The biggest reasons that companies buy expensive appliances is (a) data > sovereignty and (b) ongoing support_

Don't underestimate reason (c): because some C-levels went golfing together.

<comment><USER21> That's actually more likely to be reason (a).</comment></comment></comment>


My father, the good Nazi - <AUTHOR> http://www.ft.com/cms/s/2/7d6214f2-b2be-11e2-8540-00144feabdc0.html#axzz2STjeV3Ia

<comment><USER2> Your father was behind a paywall? It sounds awful, but not enough to make him into a Nazi.</comment>