Scaling to 100k Users - <AUTHOR> https://alexpareto.com/scalability/systems/2020/02/03/scaling-100k.html <comment><USER3> This is relevant only for multimedia apps.

I have fintech systems in production with 100k+ users with complex Gov app for entire country that runs on commodity hardware (majority of work done by 1 backend server , 1 database server and all reporting by 1 reporting backend server using the same db). Based on our grafana metrics it can survive x10 number of users without upgrade of any kind. It runs on linux and dot net core and Sql Server.

Most of the software is not multimedia in nature and those numbers are off the charts for such systems.

<comment><USER4> Thank you for this post. I read "10 Users: Split out the Database Layer" and about had an aneurysm.

I also work with fintech systems built upon .NET Core and have similar experiences regarding scaling of these solutions. You can get an incredible amount of throughput from a single box if you are careful with the technologies you use.

A single .NET Core web API process using Kestrel and whatever RDBMS (even SQLite) can absolutely devour requests in the <1 megabyte range. I would feel confident putting the largest customer we can imagine x10 on a single 16~32 core server for the solution we provide today. Obviously, if you are pushing any form of multimedia this starts to break down rapidly.

<comment><USER5> It seems like they recommend splitting out the database at the start because using a managed service is much easier than properly managing your own production database.

<comment><USER6> I can't speak for high performance/near or realtime system people but I would not trust a managed database service for those needs. My experience is that the managed offerings lag behind upstream versions substantially and usually are economical because they are multitenant. So you have a bit less predictability in io wait / cpu queue, lose host kernel level tunings (page sizes or hugepages, share memory allocation, etc), and - not naming names - some managed db services are so behind they lack critical query planner profiling features. That's not even going into application workload specific tuning for various nosql stores. This is a nice article but its audience is people that haven't scaled up a system but are trying to cope with success. It's not great generalized scaling advice.

<comment><USER4> All I will say is that our latency getting a business entity in or out of a SQLite database (running on top of NVMe flash) is on the order of tens to hundreds of microseconds. There will never be a hosted/cloud offering that can even remotely approach this without installing some "bring the cloud to you" appliance in your datacenter.

<comment><USER6> It almost kills me, just 10 years ago it was nearly six figures to get a RamSan that had 2TB storage and did 100k/25k r/w iops ([https://www.networkworld.com/article/2268291/less- expensive-...](https://www.networkworld.com/article/2268291/less-expensive-- faster-flash-array-served-up-by-texas-memory-systems.html))

Now a WD Blue NVMe does 95k/84k r/w iops at $215 and that's just off their website ([https://shop.westerndigital.com/products/internal- drives/wd-...](https://shop.westerndigital.com/products/internal-drives/wd- blue-3d-nand-sata-ssd), may be more depending on shipping method...)

That said, it's not a fair comparison and I wouldn't want to run a big service on a single sqlite/nvme setup for more reasons than are worth mentioning, but not prematurely optimizing can take you really far - scale _and_ money - <USER2> good design.

</comment></comment></comment></comment></comment></comment><comment><USER7> An hour of work put in by any of you reading this is worth several months of hosting for a starter project in an expensive provider like Heroku.

Do not invest time making sure your service runs for $6 a month if it can run for $50 with 0 hours invested. Invest that time talking to customers and measuring what they do with your service.

Most times a few customers pay for the servers.

This is just a friendly reminder. I see a lot of comments talking about running backends for cheap.

<comment><USER8> A friend of mine recently launched a side-project that does heavy processing of audio. He decided to invest ~2-5 hours properly setting up auto-scaling, a job queue, etc, before releasing v1.

Fast-forward two days later, his service and a competitor were both featured on Product Hunt. He's now making a profit on the service, as he managed to scale it up very fast, while the competitor buckled and completely lost momentum.

If you're talking about spending _a long time_ preparing a perfect infra, then your argument makes sense. Spending a few hours? It's both a great learning exercise and can literally save your project, so why not?

<comment><USER9> > completely lost momentum

I am sorry but you are really talking about 2 days of time interval and projecting predictions based on that? Unbelievable...

So what's to stop the competitors to do the same thing that your friend did to invest 2-5 hours and catch up on the third day???

I wish I was good at ascii art, then I would draw a nice facepalm here.

<comment><USER10> My take: depending on the nature of the business, and how the publicity was done, they may only have had one shot at gaining the customers. In 2-3 days time you might have fixed things, but by then the prospects moved on to the site that worked.

I'm not convinced that you need to superscale your infrastructure first. I think it's normally a waste of time and money. But for the example listed this is a likely benefit.

</comment></comment></comment></comment><comment><USER11> As the original creator of the presentation referenced by the blog author (later re-delivered by Joel in the linked post) I am super excited to see this still have an impact on people, but I'd say today in 2019 you'd probably do things very differently(as others call out).

Tech has progressed really far and there are tools like Netlify for hosting that would replace 90% of the non-DB parts of this. Cloud providers have also grown drastically and so again a lot of this would/could look a lot lot different.

Fwiw original deck from Spring of 2013, delivered at a VC event and then went on to be the most viewed/shared deck on Slidehare for a bit: [https://www.slideshare.net/AmazonWebServices/scaling-on- aws-...](https://www.slideshare.net/AmazonWebServices/scaling-on-aws-for-the- first-10-million-users/)

thanks, \- munns@AWS

<comment><USER12> Does it look that much different if you exclude solutions that increase vendor lock-in?

<comment><USER13> You always pay the vendor. Whether that's in sweat and tears or in dollars is up to you.

Fwiw, you are almost certainly shooting yourself in the foot by avoiding vendor lockin at stages before 8 revenue figures per year. Your engineering takes longer, is more brittle, and because you're only using 1 vendor actively, your solution is still vendor locked-in.

Love, ~ Guy who learned his lesson many times

<comment><USER12> I think it depends on the type of vendor lock-in -- sure, the trade off of having a managed Postgres instance is obvious, but it becomes less obvious to me when you're using things like a proprietary queueing or deployment service.

Writing service API integration code instead of code that interfaces directly with the technology that service is doing makes code quite brittle. If/when the vendor deprecates the service, introduces backwards incompatible changes, or abandons development of the product, you're left on the hook to engineer your way out of that problem. Often times that effort is equal to or greater than the effort of an in-house solution in the first place.

I had the same mentality as you until this happened to the SaaS product I work on for a few different services. Now at very least I try to make sure solutions are cloud agnostic.

<comment><USER13> My use case is that for the past 5 years I've built several API integrations in a vendor agnostic way. We never changed vendors.

Actually, we did once and we found that our abstraction was so tightly coupled to the underlying API that we had to remake it anyway. The core concepts between those APIs were just too different.

And I've had at least 2 cases where our attempt at being vendor agnostic made the integration completely fail and never work right. To the point the vendor told us "You're holding it wrong, please stop"

</comment></comment></comment></comment></comment><comment><USER14> Heh, this is one of the questions I liked to used for interviews.

"Let's work together and design a system that scales appropriate but isn't overbuilt. Let's start with 10 users".

Then we talk about what we need and go from there. The end result looks a lot like this blog post, for those who are qualified.

<comment><USER15> /offtopic

Heh, you're being modest. I'm sure you've dealt with far more complex distributed systems than the hypothetical one in the blog post.

<comment><USER14> Sure, but most of the people I was interviewing hadn't, so it was a good way to test their knowledge. :)

If you can scale to 100K users, you can probably learn the rest to scale to 100M users.

</comment></comment></comment><comment><USER16> It's probably a bad idea to switch to read only replicas for reads pre- emptively, vs vertically scaling up the database. Doing so adds a lot of incidental complexity since you have to avoid read after writes, or ensure the reads come from the master.

The reason punting on this is a good idea is because you can get pretty far with vertical scaling, database optimization, and caching. And when push comes to shove, you are going to need to shard the data anyway to scale writes, reduce index depths, etc. So a re-architecture of your data layer will need to happen eventually, so it may turn out that you can avoid the intermediate "read from replica" overhaul by just punting the ball until sharding becomes necessary.

<comment><USER14> The problem with going to the "top of the vertical" scaling so to speak is that one day, if you're lucky, you'll have enough traffic that you'll reach the limit. And it will be like hitting a wall.

And then you have to rearchitect your data layer under extreme duress as your databases are constantly on fire.

So you really need to find the balance point and start doing it _before_ your databases are on fire all the time.

<comment><USER17> Assuming you have a relatively stable growth curve, you should have some ability to predict how long your hardware upgrades will last.

With that, you can start planning your rearchitecture if you're running out of upgrades, and start implementing when your servers aren't yet on fire, but are likely to be.

Today's server hardware ecosystem isn't advancing as reliably as it was 8 years ago, but we're still seeing significant capacity upgrades every couple years. If you're CPU bound, the new Zen2 Epyc processors are pretty exciting, I think they also increased the amount of accessible ram, which is also a potential scaling bottleneck.

<comment><USER14> > Assuming you have a relatively stable growth curve, you should have some > ability to predict how long your hardware upgrades will last.

But that's not how the real world works. The databases don't just slowly get bad. They hit a wall, and when they do it is pretty unpredictable. Unless you have your scaling story set ahead of time, you're gonna have a bad day (or week).

<comment><USER17> If you're lucky, the wall is at 95-100% cpu. Oftentimes, we're not that lucky, and when you approach 60%, everything gets clogged up, I've even worked on systems where it was closer to 30%.

Usually, databases are pretty good at running up to 100%, though. And if you started with small hardware, and have upgraded a few times already, you should have a pretty good idea of where your wall is going to hit. Some systems won't work much better on a two socket system than a one socket system, because the work isn't open to concurrency, but again, we're talking about scaling databases, and database authors spend a lot of time working on scaling, and do a pretty good job. Going vertically up to a two socket system makes a lot of sense on a database; four and eight socket systems could work too, but get a lot more expensive pretty fast.

Sometimes, the wall on a databases is from bad queries or bad tuning; sharding can help with that, because maybe you isolate the bad queries and they don't affect everyone at once, but fixing those queries would help you stay on a single database design.

<comment><USER18> The minute your RDBMS' hot dataset doesn't fit into memory its going to shit itself. I've seen it happen anywhere from 90% CPU down to around 10%. Queries that were instant can start to take 50ms.

It can be an easy fix (buy more memory), but the first time it happens it can be pretty mysterious.

</comment></comment></comment></comment></comment></comment><comment><USER19> These posts are great and there is always great information in them. But to nitpick, it would be a lot easier to digest on face value if you lead with concurrency rather than raw total users as that's the true gauge of how your server infrastructure looks like.

<comment><USER20> Yeah I still don't understand the need to split servers at 10 users. Even if this is in parallel, it must still mean there is a well-beyond-average resource consumption per user.

<comment><USER21> Probably so when your 10 users grow to 1000, your efforts at 10x are good for 1000x, and you're working on 100000x.

</comment></comment></comment><comment><USER18> Conversely, rent or buy 1 bare metal server. That's how we went until we hit around 300k users. Back in 2008.

<comment><USER22> I think it's kind of crazy that we have 64 core processors available, but still need so many servers to handle only a hundred thousand users. That's what, a few thousand requests per second max?

Having many servers gives you redundancy and horizontal scalability, but also comes at a high complexity and maintenance cost. Also, with many machines communicating over the network, latency and reliability can become much harder to manage.

Most smaller companies can probably get away with having a single powerful server with one extra server for failover, and probably two more for the database with failover as well. I think this would also result in better performance and reliability as well. I'm curious to know whether the author tried vertical scaling first or went straight to horizontal scaling.

<comment><USER23> The bottleneck on a single big server setup is available network bandwidth to serve all those 100k users. If you run a simple site you can probably slap some cdn to serve all your static assets so it won't clog your network, but if your app uses more bandwidth per user than a typical website that can't be offloaded to cdn, then your single server might not have enough available bandwidth to serve all those 100k users and you'll be forced to scale horizontally even though your server still have plenty of cpu and i/o capacity. You might be able to increase your bandwidth but your mileage may vary as dedicated server vendors usually cap their offering to 1-3gbps per server.

<comment><USER24> That would have to be 100k concurrent users all streaming data at to 100kbps to saturate a 10gbps connection. Which are not hard to find these days. At least I came across several offerings when browsing bare metal server options recently. And they were not that expensive either.

And as a side note, anyone who is using up that kind of data is not going to be able to afford cloud egress prices unless they are making a mint on those users. Saturating a 10gbps connection would cost you around $450 an hour at AWS rates.

<comment><USER23> European providers seem to be more generous with bandwidth. On other locations if you want 10gbps per server you probably need to talk to someone first, and there are nonzero chance that they can't fulfill that if their datacenter is not that big.

</comment></comment></comment></comment></comment><comment><USER25> This blog post is almost entirely a re-hash of [http://highscalability.com/blog/2016/1/11/a-beginners- guide-...](http://highscalability.com/blog/2016/1/11/a-beginners-guide-to- scaling-to-11-million-users-on-amazons.html)

The primary difference is that this post tries to be more generic, whereas the original is specific to AWS.

The original, for what it is worth, is far more detailed than this one.

<comment><USER26> That's on purpose:

>> This post was inspired by one of my favorite posts on High Scalability. I wanted to flesh the article out a bit more for the early stages and make it a bit more cloud agnostic. Definitely check it out if you're interested in these kind of things.

</comment></comment><comment><USER27> A little bit of overkill recommendations here.

With 10 users you don't "need" to separate out the database layer. Heck you don't need to do that with 100 users. Website I ran back in 2007-2010 had tens of thousands of users on a single machine running app, database, and caching fine.

Users are actually a really poor way use for scalability planning. What's more relevant is queries/data transmission per interval, and also the distribution of the type of data transfers.

I'd say replace the "Users" in this posts to "queries per second" and then I think it's a better general guide.

<comment><USER28> Even in that case it's overkill. My site load tested at ~1k requests per second two years ago, when it was entirely on a $5/month DO droplet.

</comment></comment><comment><USER29> That seems pretty aggressive for just 100k users unless they mean concurrent users (in which case they should say so).

Let's say that maybe 10% of your users are on at any given time and they each may make 1 request a minute. That's under 200 QPS which a single server running a half-decent stack should be able to handle fine.

</comment><comment><USER30> We now use a DigitalOceans managed database with 0 standby nodes, coupled with another instance running Django. It is working good.

We are however actually thinking about switching to a new dedicated server at another provider (Hetzner) where we are looking at having the Web server and the DB on the same server, however the new server will have hugely improved performance (which is sometimes needed), still at a reduced cost compared to the DigitalOcean setup.

The thing we are doubting is if having a managed db is worth it. The sell in is that everything is of course managed. But what does this mean in reality? Updating packages is easy, backups as well (to some extent), and we still do not use any standby nodes and doubt we will need any replication. So far we have never had the need to recover anything (about 5 years). Before we got the managed db we had it in the same machine (as we are now looking at going back to) and never had any issues.

Any input?

<comment><USER31> I thought part of their managed service was that they optimized / tuned your postgres db based on how you were using it. If that is true, then moving off of the managed service means you are tuning postgres yourself now.

Also want to throw in there that it is important to not only compare specs, but to also compare hardware. If DO has newer chips and faster RAM, then you will take a performance hit moving to the new provider even if the machine is beefier.

<comment><USER32> Pretty certain that DO tunes to broad usage performance optimization (all the easy, obvious performance wins), not dynamically per client to each client's usage.

Here's their pitch: easy setup & maintenance, scaling, daily backups, optional standby nodes & automated failover, fast reliable performance including SSDs, can run on the private network at DO and encrypts data at rest & in transit.

<comment><USER31> huh, when I used them in the past we had someone specifically look at our RDS and data usage and tune it.

</comment></comment></comment></comment><comment><USER33> Eh, this article was disappointing.

It teaches good _business_ practice of tackling 1 thing at a time & not over- engineering.

But simply using read replicas, caches, CDNs, etc. does not mean things will scale.

Actually, often this will break your app unless you write concurrency-safe code. Learning and writing concurrent code is how you scale, the "adding more boxes" is just the after effect.

For instance, we have about 10M+ monthly users running on GUN ([https://github.com/amark/gun](https://github.com/amark/gun)), this is because it makes it easy/fun to work/play with concurrent data structures, so you get high scalability for free without having to re-architect.

But learning something new is never an excuse for shipping stuff today. Ship stuff today, you can always learn when you need it.

</comment><comment><USER28> I find this deeply unconvincing as I've scaled multiple apps to the 10k range while on low-tier hardware using the setup Alex suggests is only appropriate for _one_ user.

The old Stack Overflow podcast was also very instructive. They went a _very_ long way on a single server and had the Reddit founders on the show to talk about their scaling during their process of adding a second box. This was on servers of the mid-aughts, running ASP.NET.

</comment><comment><USER34> So, I just finished reading the article, and the last paragraph quickly mentions logging.

So, in the 'spirit' of this article, would that not be one of the first things to implement with the system?

Wait until you have added a caching layer and sharding the DB, to begin implementing logging?

I may not be reading this correctly.

I could see the case being made for distributed tracing, but having a logging strategy that can also scale and be flexible seems really important, to me at least.

</comment><comment><USER35> I am glad to see so many HN'ers here who run very successful projects on bare metal or simple VMs. We should do more to talk about those business verticals, use cases, and how we solve them practically from a tech point of view.

</comment><comment><USER36> I'm running ~1k DAU on a $6/mo Vultr VPS. Just a Phoenix Web App, no special things done to optimize. If I cache a little more aggressively on the frontend I should be able to handle even 10k. As always, advice in article depends on what you're doing.

</comment><comment><USER37> We can now achieve pretty high scalability from day 1 with a tiny bit of "engineering cost" up front. Serverless on AWS is pretty cheap and can scale quickly.

App load: |User| <-> |Cloudfront| <-> |S3 hosted React/Vue app|

App operations: |App| <-> |Api Gateway| <-> |Lambda| <-> |Dynamo DB|

Add in Route53 for DNS, ACM to manage certs, Secrets Manager to store secrets, SES for Email and Cognito for users.

All this will not cost a whole lot until you grow. At that point, you can make additional engineering decisions to manage costs.

<comment><USER38> Great, but this reads like a particularly blunt Amazon ad. Is there a way to achieve "high scalability" without selling my soul to Amazon?

<comment><USER37> Yes, it does read like that.

In the context of a start-up, cost is a big factor and then perhaps (hopefully) handling growth. You could start small and refactor apps/infrastructure as you grow but I am unsure how one could afford to do that efficiently while also managing a growing startup.

On the selling soul to cloud provider, I don't see it like that. I have a start-up to bootstrap and I want to see it grow before making altruistic decisions that would sustain the business model.

Once you are past the initial growth stage, there are many options for serverless, gateway, caches, proxies that can be orchestrated in K8 on commodity VMs in the datacenter. Though this is where you would need some decent financial backing.

(I am not associated with Amazon, Google or Azure. I do run my start-up on Azure.)

<comment><USER15> I'm down a similar route, but I must point out that beyond a certain number of users / scale, Serverless becomes cost-prohibitive. For instance, per back-of- the-napkin calculation, the Serverless load I run right now, though very cost- effective for the smaller userbase I've got, would quickly spiral out of control once I cross a threshold (which is at 40k users). At 5M users, I'd be paying an astonishing 100x the cost than if I hosted the services on a VPS. That said, Serverless does reduce DevOps to an extent but introduces different but fewer other complications.

As patio11 would like to remind us all, _we 've got a revenue problem, not a cost problem._ [0]

[0] [https://news.ycombinator.com/item?id=22202301](https://news.ycombinator.com/item?id=22202301)

</comment></comment></comment></comment><comment><USER39> Users has to be the worst unit you can possibly use, there's just to much variance between projects. If it's 100k users for a timesheet app people use once a week you've got very different scalability requirements than a 100 user app that people constantly interact with day in day out. Even then there's big differences depending on the domain, 100k users inserting data into a single table looks very different to the same entering information across 30 tables. You can't just pretend there's magic numbers that apply to everyone.

Many of those steps are reduce scalability if they're applied prematurely, splitting out the API and database layer at 1000 "users" is going to use more resources serializing things across the network than keeping it in process would. Same for seperating out the database, it's great if you need it but there's a cost if you don't. I worked on one system where we pulled out the API layer after realizing that this was where ~50% of our CPU time was being spent.

It also seems to focus on vertical layering more than horizontal splitting, being a photo sharing website I would have thought there was a lot of CPU intensive photo manipulation or something they can split off to services on the side that doesn't need to be done in real time.

<comment><USER40> Of course the article doesn't mean to give a solution to every possible scenario. It has to be taken as a general guideline of the necessary steps you should take in scaling an application but obviously every scenario is unique and needs to be analyzed before taking actions.

</comment></comment><comment><USER41> IMHO, I believe discussion about scale should begin with at least 1Million users these days. 100k has been old news for more than a decade.

<comment><USER42> As stated above, the number of users is not a measure of a system, it is the concurrency multiplied by the typical system loads per action.

Clearly a million users on Facebook is much heavier than a million registered with online banking and who only use it once a month.

</comment></comment><comment><USER40> I have a few questions:

1) I don't really get the "100 Users: Split Out the Clients".

It definetly helps in terms of understanding your customer profiles, whether they prefer the mobile app or the web interface for instance, and it might help from a usability point of view, but how does this help scalability per se if the API layer stays the same?

Splitting client happens, obviously, at the client level..

2) Also, I don't understand "This is why I like to think of the client as separate from the API."

Who considers the client and the API as the same thing?

You can consider the API as the client of the DB, sure, but why would you mix the user client and API together?

3) Caching: here I lack some knowledge. "We'll cache the result from the database in Redis under the key user:id with an expiration time of 30 seconds". I assume that every access to Redis _will not_ refresh the cache (aka reset the counter), otherwise you could potentially never get an updated data, right?

</comment><comment><USER43> Couple of questions for the author as well as the HN crowd:

1\. Should I interpret each X number of users as stated in the article to be "simultaneous" or "generally active over the past Y amount of time" or even "total user count ever?" "Unique per day?"

2\. What do you think in general about skipping the step "100 Users: Split Out the Clients" if one is reasonably certain to not want or need multiple clients? It would seem as though this could keep the deployments and testing simplified until later in the growth stage, as more code can be deployed/tested as a single bundle. But also, I want to be sure I'm not missing something by just trying to justify my own interests.

</comment><comment><USER44> Early stage social media/news feedy startup here, wondering if a strategy of just starting off with Firebase and praying will work.

I figure if we did get to a thousand or ten thousand users, we could recruit the DevOps talent to pull this off before the bills kill us!

<comment><USER45> Start with PostgreSQL as database and PHP/NodeJS/Java/<popular language> as web server. It doesn't sound sexy but leaves enough room to grow your app if need arises, and it's much easier to find people who know the stack. Firebase is a mistake in my eyes because when high bills come in, you are locked in. Either the bills or the migration to another stack will kill you.

<comment><USER44> I think you're right... We blew through the free tier 50k database reads just on testing, and that's just 3 users!

</comment></comment></comment><comment><USER46> Nice, but very simplistic (on purpose, it seems), write-up on the topic. For a much more comprehensive and excellent coverage of designing & implementing large-scale systems, see [https://github.com/donnemartin/system-design- primer](https://github.com/donnemartin/system-design-primer). Also, I want to mention that an important - and relevant to scaling - aspect of _multi- tenancy_ is very often (as is in Alex's post) not addressed. Most of the large-scale software-intensive systems are SaaS, hence the multi-tenancy importance and relevance.

</comment><comment><USER47> Good post, two thoughts:

* I'd imagine the website layer is frequently static (html/js) and could just be hosted on s3/cdn. One part of scaling avoided.

> This is when we are going to want to start looking into partitioning and > sharding the database.

You have to be at pretty huge scale before you really need to consider this. A giant RDS instance, some read replicas, and occasional fixing of bottlenecks will go a lonnng way. And scaling RDS is a few clicks. By the time you need to start sharding, you can probably afford a dedicated database engineer, or at least I'd hope.

</comment><comment><USER48> Jesus. If you're having to split your application across multiple machines at just 1,000 users you are poked by the time you have to properly scale.

</comment><comment><USER49> Questions for y'all. (Rather, I'm soliciting broad technical and business advice.)

I have built a very fast and efficient CPU-only neural TTS engine in Rust/Torch JIT that is the synthesis of three different models. I've got a bunch of celebrity and cartoon voices I've trained. The selling point is that this runs on cheap, commodity hardware and doesn't require GPUs. I can easily horizontally scale it as a service.

I've currently got it running in a Kubernetes autoscaling group on DigitalOcean, but I'm worried about the bandwidth costs of serving up potentially thousands of hours of generated audio. I haven't thrown any real traffic at it beyond load testing, but I think it can survive heavy traffic. The thing that worries me is the bandwidth bill.

Does anyone have experience with other hosts that are cheap for bandwidth intensive apps? Are there hosts that provide egress bandwidth on the cheap for dynamically generated (non-CDN) content?

Subsequent to this, I would really like to sell or monetize this app so I can fund the R&D / CapEx intensive startup I really want to undertake.

Who might be the market to buy a TTS system like this?

I was thinking Cartoon Network might want "Rick and Morty" TTS, but despite my engineering to scale this and make it sound really good, I doubt they'd pay me much for the product. I suppose $2M would give me runway to hire a few engineers and buy a lot of the equipment I need, but I have no idea who would pay for this.

Glass for optics is surprisingly expensive, and beyond that I have other extremely high R&D costs.

Alternatively, I also have a "real time" (~800ms delay) neural voice conversion system. I thought about running a Kickstarter campaign and selling it to gamers / the discord demographic. It's relatively high fidelity with no spectral distortion, and I have a bunch of hypothetical mechanisms to make it an even better fit.

I've also thought about slapping a cute animation system on top of my TTS service let people animate characters interacting. (Value add?) An earlier non-neural TTS system I built before the last presidential election cycle had something like this, but more primitive: [http://trumped.com](http://trumped.com) (The audio quality of this concatenative system is absolute garbage. The new thing I've built is unrelated.)

<comment><USER50> If you're worried about bandwidth, you could just let people download an offline app/sdk under a proprietary license.

</comment></comment><comment><USER51> odd, that was my google query of yesterday..

I'm curious what kind of hardware can sustain 100k concurrent connections these days.

<comment><USER42> We were running a speed test with node vs dotnet core and even on a small Linux box (4GB, 1 core), we could reach nearly 10K concurrent requests for a basic HTTP response but the exact nature of the system will affect that massively.

Add large request/response sizes or CPU/RAM bound operations and your servers can very quickly reach their limits with far fewer concurrent requests.

Architecture is a big picture task since you have to consider the whole system before implementing part of it, otherwise you end up having to start again.

<comment><USER51> thanks that's already a lower bound point of reference

</comment></comment></comment><comment><USER52> Use firebase or any other serverless architecture and forget about scaling and devops. Not only will you save development time but also money because you need less developers. Yes I understand at some point it will get expensive, but you can still optimize later and move expensive parts to your own infrastructure if needed

</comment><comment><USER53> I'm running 20$ Linode instance and I'll not upgrade that very soon :)</comment>


The opposite of a bloom filter - <AUTHOR> https://www.somethingsimilar.com/2012/05/21/the-opposite-of-a-bloom-filter/ <comment><USER2> I would argue that the opposite of a bloom filter doesn't really exist, at least not in a satisfying way. A bloom filter's size is dependent only on the desired false positive rate, whereas its opposite must be dependent on the size of the data. (And don't be fooled by data that can be represented by a primary key, that's not as general as a bloom filter.) I tried, with limited success, to explain my point of view in this answer on StackExchange: [https://cstheory.stackexchange.com/questions/6596/a-probabil...](https://cstheory.stackexchange.com/questions/6596/a-probabilistic- set-with-no-false-positives/14455#14455)

<comment><USER3> This probably runs afoul of your "at least not in a satisfying way" constraint, but:

It is pretty easy (an exercise) to implement the "opposite of a Bloom filter" if you start from a summary of the complete set of events and support deletion, rather than starting from the empty set and supporting addition.

What makes everything seem hard is the (often unstated) requirement that you start from an empty set and support addition, which is roughly as hard as implementing a Bloom filter that starts from the complete set and supports deletion. Neither of the links make this requirement explicit (though, it is implicit in their "motivation" sections).

</comment></comment><comment><USER4> TLDR: a cache with LRU eviction, but only storing the keys, not the values.

<comment><USER5> I was thinking the same thing

</comment></comment><comment><USER6> "A Bloom filter is a data structure that may report it contains an item that it does not (a false positive), but is guaranteed to report correctly if it contains the item ("no false negatives")."

I'm afraid that is not how it works. A Bloom filter can tell whether an item may be in the set (false positive) but can definitely tell an item is NOT in the set (no false negative).

<comment><USER7> It's correct; you've misread it — in fact, you're agreeing with it. What it's saying is, if the item is in the set, the Bloom filter is guaranteed to report that it is in the set. What you're saying is, if the filter says the item is not in the set, then it is guaranteed not to be in the set. Those two statements are equivalent (being contrapositives: "A implies B" is equivalent to "not B implies not A").

<comment><USER8> The article states a Bloom filter is "guaranteed to report correctly if it contains the item", however a Bloom filter cannot do this. The bits set by the various hash functions could very well be set due to some other key. What the Bloom filter _can_ say, is that if none of the bits are set, then clearly the key was never inserted.

<comment><USER9> I think it might just be a strangely formed sentence where the part "if it contains the item" is not the subject of the report, but a condition:

If it contains the item -> reports that it contains the item (correctly)

If it doesn't contain the item -> may report that it contains the item (incorrectly)

<comment><USER10> Nothing wrong with being pedantic when dealing with definitions.

<comment><USER11> Since we're on the topic of being pedantic, "well, you know, that's just, like... your opinion, man". I'd argue that there's nothing excessive about arguing if A may, in fact, be the opposite of A :)

<comment><USER10> Certainly not, I'm all for some arguing on the internet.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> "It's a cache." The use case is deduplication in an event-stream environment. This calls for exact matching without hash collisions.

</comment><comment><USER13> Low memory version:

`return false;`

<comment><USER13> Not sure why it's getting fownvoted so much. On large datasets it's just as effective...</comment></comment>


The Reporter Resists His Government - <AUTHOR> http://www.nybooks.com/articles/archives/2015/feb/19/james-risen-reporter-resists-his-government/

<comment><USER2> Jeffrey Sterling was an attorney working for the CIA as the handler for a Russian covert agent whose mission was to subvert the Iranian nuclear weapon program by feeding them faulty designs for the electrical firing mechanism of a nuclear explosive device. James Risen is one of the most celebrated investigative journalists of the last 20 years.

Sterling informed Risen of the program he oversaw for the CIA. Risen dedicated a chapter in his book to it.

The Bush administration opened an investigation into the incident. Obama's Holder-led DOJ oversaw a criminal prosecution of Jeffrey Sterling.

At no point that I can find was Risen ever criminally charged for anything. For a period of several years, Risen fought a subpoena compelling him to testify, naming his source (Sterling had already been indicted). That subpoena was appealed all the way to SCOTUS and ultimately held up. Holder, who already had enough evidence to convict Sterling, ultimately backed off. Risen was never compelled to name his source.

To me, this is sort of an acid-test case. It is hard to imagine a more consequential singular leak than one that reveals a carefully-planned human- intelligence _counter-proliferation_ mission against a hostile state. Sterling had a position that appears to have involved the utmost level of trust --- he wasn't a sysadmin disquieted by the documents he was uncovering, but an operations officer intimately familiar with and actively participating in the operation he revealed. There is a colorable argument that there should in a truly free government no state secrets. My sense is, if you think this leak was praiseworthy and the government's investigation "overreaching", as the Dean of the Columbia Journalism School writes here, you kind of have to sign on to that argument, don't you?

Two more thoughts:

First, Risen is a startlingly effective journalist, one of the best of his generation, up there on the leaderboard with Seymour Hersh. The lede of this piece sets up a narrative about a reporter persecuted by his own government. But (a) Risen wasn't prosecuted at all, and (b) his serious legal entanglements seem to involve a single case on his long resume, which resume is recounted at some length in this piece. The narrative effect is to suggest that the government fought as hard against his NSA and torture stories as they did against this one. But that's an argument that doesn't seem to be supported by evidence.

Second, more of a question: I've tried to read the indictments of all the leakers on the roster of the Obama administration leak prosecutions, often said to be the most zealous of any President. At one point in this piece, Coll states outright that the Holder DOJ had imprisoned "a half dozen" sources for journalists. That doesn't square numerically with the prosecutions that have been reported on, nor are most of those prosecutions about people who were sources for journalists (apart from Chelsea Manning).

<comment><USER3> > Second

Pretty easy to find. What's your question?

[http://www.propublica.org/special/sealing-loose-lips- chartin...](http://www.propublica.org/special/sealing-loose-lips-charting- obamas-crackdown-on-national-security-leaks)

<comment><USER2> This timeline shows 3 indictments for journalistic sources during the Obama administration (I'm not counting Kiriakou, who was his own source, and leaked CIA secrets during the promotion of a book he wrote, but even if I did, that's 4 indictments, not a half-dozen, and even fewer imprisonments).

So, back to my question:

Who are the half-dozen imprisoned journalistic sources Coll is referring to? I'm sure he's not just making it up!

FWIW: I _have_ done some research here; I asked a similar question the last time the sound bite about Obama/Holder's leak prosecution zealousness came up:

[https://news.ycombinator.com/item?id=5701813](https://news.ycombinator.com/item?id=5701813)

<comment><USER3> Ah, got it. I'll have to look into it a little more. It does seem there's some uncertainty about the total.

[http://www.politifact.com/punditfact/statements/2014/jan/10/...](http://www.politifact.com/punditfact/statements/2014/jan/10/jake- tapper/cnns-tapper-obama-has-used-espionage-act-more-all-/)

<comment><USER3> That said, the number of Espionage Act prosecutions by this admin. is concerning and pernicious in itself.

<comment><USER2> Can you support that argument with evidence?

<comment><USER3> Yes, the evidence would be every Espionage Act prosecution under the Obama administration. Which are more numerous than the total of every previous administration since 1917.

If you could justify how that's not worrisome that would be great.

[https://en.wikipedia.org/wiki/Espionage_Act_of_1917](https://en.wikipedia.org/wiki/Espionage_Act_of_1917)

<comment><USER2> If you read the link I posted upthread, I won't have to repeat my response to the claim that Obama is more zealous about the Espionage Act than his predecessors.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER4> It's chilling how far those in power will go to silence even the _possibility of dissent_ that honest journalism provides. In this case, the Obama justice department inherited a case from the Bush era and fought it all the way to the supreme court to get a journalist to incriminate his alleged source.

The information disclosed by Risen was already out there and the damage was done. That it embarrased a previous administration did not seem to matter. The DOJ seems to have wanted at least one head to roll in order to send a message to government insiders: "talk to the press at your own peril."

<comment><USER2> First, Eric Holder did not ultimately compel Risen to name his source.

Second, this wasn't a corruption or torture report, and the stakes weren't "embarrassment". The CIA turned a Russian nuclear engineer who fed to Iran flawed plans for the electrical firing mechanism for a nuclear explosive device. Risen learned about that from that Russian engineer's CIA handler, and after the NYT spiked a story he wrote about it, wrote a chapter a book about it.

<comment><USER5> It wasn't necessary to ultimately compel Risen to have a chilling effect on journalism, much like it wasn't necessary to ultimately imprison Thomas Drake for the rest of his life to have a chilling effect on IC employees who seek to expose waste, fraud, and abuse.

I agree with your perspective on the wisdom of Risen's reporting, but still think we should celebrate the heroic act of journalists protecting their sources even when they're on the hot-seat.

One doesn't necessarily have to be indicted or prosecuted for people to feel as though their rights are threatened. Needlessly forceful/aggressive FBI raids (Binney), wiretapping press organizations (AP), endless secret grand juries (WL), and border searches (Poitras) are all examples of the many ways USG has intimidated actual and potential whistleblowers and reporters.

</comment></comment></comment><comment><USER6> Wow. The _NYR_ reporting about the supine posture of the national press has been fantastic, and this is another example, one of the most revealing I've read.</comment>


Show HN: View random tweets from the Russian "trolls" - <AUTHOR> http://www.russiansinmy.house/ <comment><USER2> Some are suspended, for example: [https://twitter.com/PIGEONTODAY](https://twitter.com/PIGEONTODAY) Would love to know the real numbers. I remember a few articles saying how Twitter mass- suspended 1000s if not millions of these type of info-ops accounts just recently. Further reading: [https://en.wikipedia.org/wiki/Information_Operations_(United...](https://en.wikipedia.org/wiki/Information_Operations_\(United_States\)) focused on US psyops campaings, but the Russians are doing it too, and probably doing it better.</comment>


Knusperli: A deblocking JPEG decoder - <AUTHOR> https://github.com/google/knusperli <comment><USER3> Ah, this is a great idea, and I can see how this could be extended beyond simple deblocking.

In essence, it computes the uncertainty from the JPEG quantization step (the source of most of JPEG's compression) and uses that information to select a decoding that minimizes a particular error metric (here, discontinuities at block boundaries).

But it's not limited to just that, in principle. It could optimize towards any decoded block within the quantization range given a suitable prior - and I imagine that this could be paired with e.g. a DNN to faithfully reconstruct an image from a JPEG with much better accuracy.

<comment><USER4> Additionally, I'd love to see a decoder add noise proportional to the uncertainty and surrounding structure as over smoothed images look unnatural as well. See activity masking demo [https://people.xiph.org/~jm/daala/pvq_demo/](https://people.xiph.org/~jm/daala/pvq_demo/)

<comment> Ruud-v-A Pik can synthesize noise. [https://github.com/google/pik](https://github.com/google/pik)

</comment></comment></comment><comment><USER5> Nice. This is a decode-side thing; it can be used on existing images.

But the site should show more images. Some with hard edges, like text and line art, need to be included.

<comment><USER6> Why would that be interesting? Jpeg is the wrong format for these types of images anyway, is it not?

<comment><USER7> That's _precisely_ why it would be interesting. Getting cleaner decodes of jpeg-encoded line art and screenshots would be great.

Some images are only available as badly encoded jpegs.

<comment><USER8> Fonts suffer from ringing, their lines can actually look better if they are on the block boundaries.

</comment></comment></comment></comment><comment><USER9> See also [https://github.com/victorvde/jpeg2png](https://github.com/victorvde/jpeg2png) , which tries to maximize smoothness everywhere instead of just at block boundaries.

<comment><USER7> Maybe these two approaches, jpeg2png and Knusperli, could be combined to get potentially superior end result.

Knusperli would run first, to reduce quantization discontinuities a bit.

</comment></comment><comment><USER10> Test image still has perceptible blocking. Description suggests it should be possible to reconstruct source values/smoothness staying in stored intervals, and yet there are still visible boundaries.

Lower right corner you can clearly trace 4 horizontal blocks, looks like algorithm tries hard to maintain encoded hard boundaries despite there being gradual color shift. Maybe it needs to look further than direct neighbor block for clues.

</comment><comment><USER11> Unacceptable. Even though swiss-german doesn't have any spelling rules, I hereby declare a public vote to deport google from switzerland, because it's clearly spelled "Chnuschperli" in proper swiss-german.

Edit: Sorry, I forgot HN is allergic to jokes, obviously this is turning into reddit. I apologize.

<comment><USER3> So what does Knusperli even mean? It seems to mean "crunchy" or something from a Google search but Translate turns up nothing. Is the meaning at all related to the project, or is it just a cute name?

<comment><USER11> Yes, it means "crunchy thingy" basically. The name doesn't have any relation to the project. It's small pieces of fish fried in a dough mantle.

I was jokingly complaining about the spelling because the previous names "Brötli" and "Zöpfli" of similar projects were 'properly' spelled. They don't have anything to do with the name either. "Brötli" and "Zöpfli" are basically types of bread.

While Knusperli is a very german way of spelling a swiss german word. Though, it is what you would find in a supermarket.

Yes, too obscure of a joke, sorry :(

<comment><USER12> No need to 'joke' about deportation. That's what disgusted me.

<comment><USER11> > No need to 'joke' about deportation. That's what disgusted me.

Yeah, I should have considered global cultural influences more before posting. Sarcastically joking about deportation is probably a bit too swiss for an international audience to recognize it as such. Let me assure your though, this was an attempt by a swiss person desperately trying to be funny.

Obviously it backfired.

</comment></comment></comment></comment></comment><comment><USER13> I think visually speaking, while the deblocked version is smoother overall, the blockiness added extra perception of noise/texture. The extra noise almost always increase the perception of sharpness and level of details, and you can see from the braids coming out of the hat that it does look sharper with the blocking version, although the wall is smoother. Would be great to include more sample images in the page.

<comment><USER14> x264 video codec developer Dark Shikari blogged about [1][2] this "fake sharpness" phenomenon. Early H.264 encoders produced excessively smooth, deblocked video, which made them look worse compared to MPEG-4 ASP (like Xvid), despite former being a newer, objectively better format. Wavelet- transform compressors like JPEG2000 had a similar problem vs. blocky JPEGs.

People's tastes are subjective, though, so it's good to have options.

[1] Archive.org: [http://web.archive.org/web/20141124213159/http://x264dev.mul...](http://web.archive.org/web/20141124213159/http://x264dev.multimedia.cx/archives/317#more-317) [2] Archive.is: [http://archive.is/K7jKA](http://archive.is/K7jKA)

</comment></comment><comment><USER15> I would assume that JPEG like most lossy compression algorithms is specifically defined in terms of the correct uncompressed output for a given compressed input. This means a "deblocking JPEG decoder" is effectively actually just a bad decoder that gives the wrong output (albeit for some outputs maybe you happen to prefer this).

The definitions are this way around because it's the only way to have an opportunity to improve encoding quality over time, if you start by defining how encoding must be done then you lock in bad ideas. Lame for example is far, far better at encoding MPEG audio layer III than the demo encoders shipped when the standard was written, and this is only possible because the standard says how to turn MP3 back into PCM data, not how to encode PCM data as MP3.

<comment> Ruud-v-A That is what I expected as well, but it turns out not to be the case. See also [https://www.ece.cmu.edu/~ece548/hw/lab3/faq- do~1.htm](https://www.ece.cmu.edu/~ece548/hw/lab3/faq-do~1.htm) and [https://encode.ru/threads/2922-Knusperli-%E2%80%94-a-better-...](https://encode.ru/threads/2922-Knusperli-%E2%80%94-a-better- JPEG-decoder?p=56184&viewfull=1#post56184).

<comment><USER16> So could one define an "inverse" JPG standard that works with the current version but has the advantages that tialaramex spoke about?

</comment></comment></comment><comment><USER17> I thought that by this point in time, we would have more images than Lena to showcase image libraries.

<comment><USER18> Agreed, I believe the image of Lenna should be discouraged, as some people might find the image offending and unwelcoming to women (it originally appeared in a 1972 Playboy magazine issue).

[https://en.wikipedia.org/wiki/Lenna](https://en.wikipedia.org/wiki/Lenna)

<comment><USER19> Considering Lena doesn't object and by all accounts is quite happy her image is used, there really is no reason not to use her image. If you real want you can use Fabio instead.

<comment><USER20> I don't think that's a good standard for a general rule.

Why did you use the pornographic picture for this work related task?

Well the people in the picture don't mind. In fact they're quite happy about the exposure.

Oh well, carry on then.

<comment><USER19> The Lena image is not pornographic. It is a picture of a Lena naked (well apart from the hat) and if it was a painting nobody would object.

</comment></comment></comment></comment></comment><comment><USER21> Dissenting opinion: I think the blocky version looks better when not zoomed in. The sample on the left is less smoothed, and doesn't look as airbrushed. It looks better when zoomed in, but not when are regular zoom.

</comment><comment><USER22> This is one of those nice "let's do this properly" projects google does. I'd put into the same category as their work on compression. Unsexy and "so last decade" at first glance, but decidedly useful and universally applicable, given the number of jpg files created since the dawn of time.

That being said, I'd really wish people would stop using lossy formats for everything. I can live with it for simple icons and such (although, why not svg?) but photos? come on! Of course, that would have to come with a nice connection for _everyone_.

Oh and this is definitely going to make for some head scratching for devs who assume image decoding is somehow deterministic. :D

<comment><USER10> >their work on compression

Like that time they stole Jarek Duda ANS and are still trying to patent it fraudulently claiming to have invented it 4 years after Jareks public domain publication?

[https://encode.ru/threads/2648-Published-rANS-patent-by- Stor...](https://encode.ru/threads/2648-Published-rANS-patent-by- Storeleap/page4)

</comment></comment><comment> TD-Linux There is a lot of previous art on this sort of thing, like the ffmpeg spp filter: [https://ffmpeg.org/ffmpeg- filters.html#spp-1](https://ffmpeg.org/ffmpeg-filters.html#spp-1)

It's unclear to me if this method will perform better than those methods.

</comment><comment><USER23> Seems a new library could use something besides Lenna...

</comment><comment><USER24> C# wrapper? :D

<comment><USER25> I'm not sure if you are joking, but it does seem that more interoperability is usually a good thing.. (especially for something like a jpeg decoder - <USER2> it's less useful to have a toaster interface smoothly with some coffee beans)

<comment><USER25> Although I guess coffee roasters are a thing..

<comment><USER25> Assuming interoperability to be a good thing, then, coding something like a c# translation of this is a stimulating and useful intellectual exercise relative to like reading a book or solving a prescribed math problem. Assuming coding problems aren't like fossil fuel for jobs or something, and society hasn't yet reached the state where we can just neglect technology and say 'good enough,' given that (at least according to people I see while locomoting around town), there are still diseases, disabilities (physical and psychological), aging, puns, death, overpopulation, homelessness, variety of experience (I don't know what the correct word for this is, but something like 'non-crunchy-multitude- of-choice'), etc. all which may have solutions with sufficiently advanced technology (with the caveat that it may be like a pandora's box). (You could probably superficially post-justify a solution to these kind of problems in society by strapping an RC radio to everyones head and saying 'karma', but I would guess you can't do that without the being enacting this breaking large parts of the social contract, and essentially ridding life of substance, e.g. 'this thing happened to you because you ate a fish when you were a child, and that fish had a family, and so, relatively, your brain is like a fish-size (lets say chicken because birds are actually pretty smart in the animal hierarchy) to this AI we've constructed, and thus it's eating you, and so therefore, its karmically balanced (that's not really a solution is it, I mean it is, but it's like a military tit for tat solution (even if done subtly) rather than a diplomatic / medical type solution, which would provide more leeway for things like basic fight-or-flight response of humans / miscommunication / speed of actual human thought / etc., which doesn't really fit in the tit-for-tat model)' ). Wow I wrote freaking thesis here (and I didn't really put that long of thought into it, so judge as such, like git commit 1 of an more revised thought).

</comment></comment></comment></comment><comment><USER26> Curious how this affects visual quality indexes like SSIM.

(Also, come on, it's 2018, we need to retire Lena)</comment>


Facebook Offers $500 Bounty for Reporting Bugs: Why So Cheap? - <AUTHOR> http://www.pcmag.com/article2/0,2817,2389460,00.asp

<comment><USER2> Not for reporting bugs. For reporting security bugs.

Google also doesn't offer such bounties when merely reporting bugs. I report all kinds of Canvas issues to Chromium. Half of them aren't even acknowledged, even if they get fixed months later.

Google only gives bounties on security bugs, and as far as I know only significant ones (though we'll see, because I just reported an incredibly minor security bug to Chromium - getImageData in Canary is capable of getting image data from outside the Canvas, which is against spec and a security flaw)

<comment><USER3> Actually they give bounties on minor security bugs as well - the award depends on severity. A bug like you describe will most certainly get something (if Canary is an eligible target).

</comment></comment><comment><USER4> The total amount of these bounties are more or less meaningless. No one can make a living, 1st world wage with them - at least not consistently. All of the programs will pay you significantly less than what you'd get paid for an equivalent number of hours at a straight job.

What's more, most bugs will sell for 10x the bounty price on the black market.

Bug bounties aren't intended to be competitive compensation. They are intended to be tokens of appreciation given to people who would be reporting the bugs anyway. In this context, $500 vs. $1000 (a more typical reward from the chrome program) isn't too meaningful.

Whether this works for the researcher or not is up to them.

<comment><USER5> +1

At best, a bounty program gives me the incentive to look more closely at a particular website / application. But I've submitted plenty of vulnerability reports to companies that don't have bounty programs.

What this bounty really means: if you like finding and responsibly reporting security vulnerabilities, you now have one more reason to spend time doing so on Facebook.

</comment></comment><comment><USER5> Why so cheap? Because $500 is the base, not the maximum. $500 is the base reward for Google and Mozilla's web bounty programs as well. Another similar bounty program, from CCBill, lists a _maximum_ payout of $500 (although I can confirm that they can and will issue higher rewards under certain circumstances: <https://twitter.com/#!/williamlbell/status/97379423816384512>)

[Note: I've participated in the bounty programs for Google, Mozilla, and CCBill. I've submitted security vulnerabilities to Facebook as well prior to this bounty program]

</comment><comment><USER3> Article is incorrect about Microsoft. They still offer $0. The 250k is a bounty on turning in criminal botnet owners.

</comment><comment><USER6> "But a security researcher cited in ComputerWorld says reporting Facebook bugs can help budding security researchers make a name for themselves in the tight- knit security community."

"I can't pay you to make my website, but you can use it in your portfolio!" springs to mind

</comment><comment><USER7> I found a bug on pcmag.com -- how much is their bounty?

</comment><comment><USER8> Hey, at least they're way ahead of Donald Knuth.

<comment><USER9> Personally I'd rather have a check signed by Donald Knuth than a $500 bounty.

<comment><USER10> Which suggests a scheme where Knuth could be paid a one time fee of (say) $50k, then be asked to issue 0.02c cheques for each bug found.</comment></comment></comment>