The Death of "Free" Software or How Google Killed GPL - <AUTHOR> https://www.linkedin.com/pulse/death-free-software-how-google-killed-gpl-annette-hurst <comment><USER2> Google did not take Java code. It wrote it's own software with exactly same API.

Any API that is "in the open" is free to be emulated. Especially Open Source ones!

What this mean for open source software is that if I write a GPL library, and someone can't use GPL software and writes it's own implementation to be a drop-in replacement, it's free to do so. Which I believe is OK with everyone, except Oracle corporate rent-seekers, that can't get over the fact, that they can't monetize SUN Microsystems corpse even more.

<comment><USER3> > Which I believe is OK with everyone

What's the FSF's position on this, I wonder. The whole premise of the GPL as opposed to the LGPL is that even dynamic linking creates a derived work.

If someone creates a completely drop-in replacement for readline (I don't know if libedit and kin are drop-in or not) and releases it under a BSD license, there's no way to distinguish my app linking against one vs the other, I'm simply making use of the API in the abstract.

This isn't exactly the same as the Java case, but the copyright status of APIs per-se sure seems to have implication for the GPL's teeth.

<comment><USER4> Lets imagine that we create a third way to link software. There is static linking, dynamic linking, and now we got this imagined third thing called adaptive linking. In this made up linker, it magically combines the different parts and uses neural networks (ie, more magic) to figure out how to make the two parts communicate without a API.

Okey, in this fantasy world, do we still have the concept of derivative work? I would say Yes. Would thus GPL still apply? Yes. Is there an API? No.

GPL is not based on the technology of linking. As a software license, it used the concept of derivative work, distribution and other concept in copyright law to define the scope. LGPL in contrast do specify linking technology, but that is only an additional permission which would need to be modify if it were to function with new linking technologies.

If you created a drop-in replacement for readline, the argument that the complete work is readline + some code get weaker. However, if the police finds some emails on your drive that states your intention to combine readline + some code, and then go to copy readline by reimplemented the software in a identical (copy'ish) way, then you might very well be infringing on the right of the readline authors. It depend what a judge/jury think your intentions was.

<comment><USER3> It's an interesting thought experiment.

I agree that GPL doesn't get it's "teeth" unless linking creates a derivative work. The FSF (and Linus) for that matter have Opinions about what does,though, so I'm curious how and whether this ruling changes those.

I don't know whether to be thankful or not that there's still a circuit split in the US over what constitutes a derivative work in the software realm.

</comment></comment></comment></comment><comment><USER5> It's honestly pretty difficult to take an Oracle lawyer's word on what would be good or bad for open source or free software.

</comment><comment><USER6> I really wish that Google had bought Sun instead of Oracle. I worked at Sun up until 2007 and the feeling at Sun was very supportive of what Google was doing.

Oracle bought Sun's assets and they have every right to what they are doing. But a failed business effort to make additional fees is not, in my opinion, a risk to "free" software.

</comment><comment><USER7> This is a decidedly non-factual account. In more detail:

> Google won a verdict that an unauthorized, commercial, competitive, harmful > use of software in billions of products is fair use.

No, the only part of this description that is clearly correct is that Google won a verdict that its particular unauthorized commercial use (that the use was unauthorized and commercial are facts not in dispute) of Java APIs is "fair use". While the verdict required the jury to weigh whether and to what extent that use was harmful, a separate finding on that point was not made.

> No copyright expert would have ever predicted such a use would be considered > fair.

IIRC, several copyright experts publicly _did_ predict that, at least, a finding that the use in question was "fair use" was a reasonably likely outcome after the ruling that APIs were copyrightable in the first place.

> the free software movement itself now faces substantial jeopardy.

Certainly, any ruling which restricts the scope of copyright protection or finds broad fair use rights poses problems for enforcing the restriction of copyleft licenses, so, while I think the concern here is exaggerated greatly, there is _some_ impact to the GPL.

OTOH, there's very little impact to the utility of permissive free software licenses in achieving the goals of said licenses.

> Google's narrative boiled down to this: because Java APIs have been open, > any use of them was justified and all licensing restrictions should be > disregarded. In other words, if you offer your software on an open and free > basis, any use is fair use.

No, it didn't. Google's arguments included a lot of points, including arguing that Google's use of Java _increased_ the commercial relevance of the Java, and boosted Oracle's business. They did not argue, at any point that I can find documented that free licensing made any use fair. This seems to be a pure invention.

> No business trying to commercialize software with any element of open > software can afford to ignore this verdict. Dual licensing models are very > common and have long depended upon a delicate balance between free use and > commercial use.

While Hurst radically misrepresents both the substance of the verdict and the nature of the argument Google made in pursuit of it, there is some truth to this: certainly, firms relying on an extremely broad interpretation of the scope of copyright protection and an extremely narrow interpretation of the fair use exception in developing a business plan around dual licensing like the "copyleft or pay us" model, _should_ carefully consider what impact actual rulings on the scope of copyright and fair use like this one have on their business models.

OTOH, lots of commercial software that has an element of open software don't rely on the restrictive nature of copyleft license as a means to upsell to commercial licenses, instead packaging additional features and/or professional services with commercial licenses (e.g., EDB's Postgres-based products.) These kinds of commercial open-core systems aren't particularly impacted.

> It is hard to see how GPL can survive such a result.

Well, sure, its hard to see how the GPL could survive the rule that Hurst fantasizes was applied here becoming generally accepted, where any use of any software under an open-source license was "fair use". But that has little bearing on reality.

Viewing the use of the Java APIs in this case as fair use _might_ imply that some of the FSF's interpretation of where adherence to the GPL is required might be wrong, but this doesn't invalidate the GPL. (It might, if well- established in law, actually make the GPL more attractive for use, though it would no doubt be somewhat troubling to those who see the GPL as a wedge to force other and only tangentially-related software to be released under the same terms.)

> Nonetheless, Google exerts control over its APIs. Google prohibits copying > of its APIs for competitive uses.

I can't see any evidence of Google either asserting that its APIs are protected or acting to prevent competitors from copying them. Is there any example of this?

> In fact, Google has in the past settled with the FTC over the manner in > which it has restricted its APIs.

This is pure equivocation (using the same term, "API", for two different meanings.) What Google has settled with the FTC isn't over APIs-as-elements- of-intellectual-property, it was over APIs-as-specific-services. This has nothing to do with the APIs-as-IP issue in this case.

<comment><USER8> When a court finds API implementation to be fair use, there is an impact on free software, and that impact is basically positive. The idea of API's being copyrightable threatens the status of free software projects which implement ("clone") previously proprietary API's.

<comment><USER7> > When a court finds API implementation to be fair use, there is an impact on > free software, and that impact is basically positive.

It certainly is positive for the ability to create new free software. It is negative for the ability to control other people's actions through copyleft licenses, since the same ability to create new free software also allows creating new _permissive_ licensed or _commercial_ software rather than adhering to the copyleft terms.

So, for some interpretations of the "Free Software Movement", there may be a net negative effect (though, again I should emphasize, nowhere near of the size or nature of that that Hurst is trying to paint.)

<comment><USER9>
      It certainly is positive for the ability to create new free software. It is negative for the ability to control other people's actions through copyleft licenses, since the same ability to create new free software also allows creating new permissive licensed or commercial software rather than adhering to the copyleft terms.

Agreed. As a point of practicality I can't think of anyone whose created 'commercial' open source who depends on copyright protection 'of the API' for their commercial elements though, can you?

It feels like the OSS ecosystem has always concluded that creating drop-in replacements using API's are 'fair use'.

<comment><USER10> Hurst's argument would be somewhat convincing if there were people trying to apply the GPL to prohibit, say, the Windows Subsystem for Linux (or other efforts to make Linux binaries run natively under Windows). I don't think it would be easy to find a consensus that that is appropriate or supported by the law, and indeed, there's been tons of concern that proprietary software copyright holders will try to sue people for writing new interoperable implementations that use the same protocol or API.

Maybe there are some narrower cases involving proprietary Linux kernel modules (which many people have suggested could be improper derived works that constitute copyright infringement of the kernel); there this result might make the infringement argument more difficult.

But as people here have pointed out, almost nobody ever attempts to limit implementation or use of APIs as a "work based on the program" for GPL purposes, almost nobody assumes that would be successful, and almost all uses of the GPL are based on limiting the literal copying of code from one program into another.</comment></comment></comment></comment></comment>


The 12 bugs of Christmas - <AUTHOR> http://www.typemock.com/blog/2011/12/25/the-12-bugs-of-christmas/ <comment><USER2> The bug is that the song is O(n^2) ;)

<comment><USER3> You could make this O(n) with some memory trade offs...

</comment></comment><comment><USER4> even too geeky for me</comment>


System design hack: Postgres is a great pub/sub and job server - <AUTHOR> https://layerci.com/blog/postgres-is-the-answer/ <comment><USER2> Another neat hack is to use Postgres as a quick & dirty replacement for Hadoop/MapReduce if you have a job that has big (100T+) input data but small (~1G) output data. A lot of common tasks fall into this category: generating aggregate statistics from large log files, searching Common Crawl for relevant webpages, identifying abusive users or transactions, etc.

The architecture is to stick a list of your input shards in a Postgres table, have a state flag that goes PENDING->WORKING->FINISHED->(ERROR?), and then spin up a bunch of worker processes as EC2 spot instances that check for the next PENDING task, mark it as WORKING, pull it, process it, mark it as FINISHED, and repeat. They write their output back to the DB in a transaction; there's an assumption that aggregation can happen in-process and then get merged in a relatively cheap transaction. If the worker fails or gets pre- empted, it retries (or marks as ERROR) any shards it was previously working on.

Postgres basically functions as the MapReduce Master & Reducer, the worker functions as the Mapper and Combiner, and there's no need for a shuffle phase because output <<< input. Almost all the actual complexity in MapReduce/Hadoop is in the shuffle, so if you don't need that, the remaining stuff takes < 1 hour to implement and can be done without any frameworks.

<comment><USER3> If a worker fails or gets pre-empted, how does it retry anything? It's gone at that point, no?

Sounds like you'd end up with a bunch of dangling shards orphaned in WORKING state. And now you need timeouts and health checks and something to coordinate all that.

<comment><USER4> I used to do something a little like this with postgres advisory locks. Worker takes out a lock that signifies it's working on a task, if the worker fails or loses its connection in any way the lock gets dropped along with the transaction being rolled back.

Example: [https://gist.github.com/risicle/f4807bd706c9862f69aa](https://gist.github.com/risicle/f4807bd706c9862f69aa)

<comment><USER5> Yeah, in MySQL you can also do this with a named lock. Really like this workflow.

</comment></comment></comment></comment><comment><USER6> > It's rarely a mistake to start with Postgres and then switch out the most > performance critical parts of your system when the time comes.

This is pretty good advice in general.

<comment><USER7> Adding postgres there and there where it's clearly the wrong tool is a bad advice.

<comment><USER8> I probably would have gone with something like Kafka in this case, but it seems like Postgres serves them pretty well even if it's the "wrong" tool. If it works, fits your current needs, leads to faster development time, and isn't needlessly slow, then I say go for it.

<comment><USER9> Picking Postgres for alternative applications such as pub/sub services also simplifies both deployment and development. You'll already be using Postgres for persistence, so why waste time deploying another set of services and develop another set of clients just to get the same functionalities?

</comment></comment></comment></comment><comment><USER10> I've done this before with good results.

I was pleased to see they are using `SELECT FOR UPDATE SKIP LOCKED`. That is what this 2nd Quadrant article recommends, which I think is required reading if you want to implement this yourself:

[https://www.2ndquadrant.com/en/blog/what-is-select-skip- lock...](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for- in-postgresql-9-5/)

It goes into more detail about wrong ways to implement a queue and what the downsides are for its preferred approach.

</comment><comment><USER11> We use a lot this kind of tooling.. say, you need to check 20k URLs and you want to rate limit them.. add them to a Pg table (with state and result fields). A single thread worker that just takes a row (marks it as pending) and later updates it. With select for update and skip tricks you can horitzontal scale it to the number of workers you need.

I had seen it also for soft that sends massmail (our case around 100k/day).. it's state is a postgres queue.

We also use Pg for transactional mail. We insert it on a table. (There is a process that sends the row mails).. the so nice part is that the mail is joining the dB transaction for free.. (all or nothing)

</comment><comment><USER12> Excellent design hack. If anybody in the Node/TypeScript ecosystem is looking for this capability in a neat and supported library, it looks like the graphile folks have you covered:

[https://github.com/graphile/worker](https://github.com/graphile/worker)

<comment><USER13> There's also pg-boss that's really full featured

</comment></comment><comment><USER14> Postgres is an acceptable relational database / nosql database / pub/ sub/ job server / blockchain.

<comment><USER15> You forgot lisp.

<comment><USER16> and emacs too

</comment></comment></comment><comment><USER17> If you're working with Ruby I have had good experiences with Que[1], which implements a pattern similar to the OP using advisory locks.

[1]: [https://github.com/que-rb/que](https://github.com/que-rb/que)

<comment><USER18> Is there a similar library in Python?

I really like the look of this approach, but don't want to build it myself if I can avoid it.

<comment><USER17> [https://pypi.org/project/pq/](https://pypi.org/project/pq/) looks similar, although it looks like they use `SKIP LOCKED` instead of advisory locks. I'm not sure what the tradeoff is.

<comment><USER19> SKIP LOCKED is much simpler and doesn't really have any gotchas.

We had an internal library that used advisory locks which had all sorts of strange behavior we couldn't figure out until we just moved to SKIP LOCKED

</comment></comment></comment></comment><comment><USER20> Geat post!! At Rudder (open-source segment), we used Postgres as our streaming layer in a similar fashion. It has been super stable in production, very easy to setup and we easily get to > 10K events/sec insertion performance.

The code is open-sourced here in case anyone wants to re-use

[https://github.com/rudderlabs/rudder- server/blob/master/jobs...](https://github.com/rudderlabs/rudder- server/blob/master/jobsdb/jobsdb.go)

We had to built additional logic to clean-up old jobs (similar to Level merges in similar queing systems)

</comment><comment><USER21> Isn't hijacking a DB as a "distributed" message queue a pretty well trodden path? Enterprises have been doing this for decades.

<comment><AUTHOR> This pattern falls down if you need to poll the database, because if you have 3 queues and 100 workers you're making 300 queries per poll interval.

The feature of postgres that makes this viable in comparison to most other databases is the "channel"

<comment><USER22> What if your listeners crashed/were down at the time of the `PUB` message? Does this mean the message falls into oblivion (since it will never receive a reply/ACK/get worked on)?

<comment><AUTHOR> The pattern I follow is that workers immediately poll once when they start - this means that when your workers restart after crashing, they'll pick up any missing jobs.

<comment><USER23> I'm not seeing how this recovers from the failure scenario where a worker grabbed an event and then immediately crashed.

<comment><USER24> The comment you are replying to wasn't addressing that scenario.

Typically you would implement visibility timeouts and other such stuff. Depending on the use case you could make specific optimizations or keep it generic and have SQS like semantics or something.

<comment><USER25> at this point why not use something like rabbitmq and not reinvent it in postgres.

<comment><USER26> Because it's trivially easy. The dequeue operation updates the last_lease time; the query makes sure you don't dequeue something that was recently leased. It's far, far easier than setting up a separate job queue, especially if you are already using an RDBMS.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER27> The postgres module for node is quite unreliable when holding open a connection to listen on a channel. This helped: [https://github.com/andywer/pg- listen/blob/master/README.md](https://github.com/andywer/pg- listen/blob/master/README.md)

But we still see issues.

</comment><comment><USER28> At the scale I operate at, I wouldn't consider this a viable option. What's the backpressure like on NOTIFY/LISTEN? (Docs mention a maximum backlog of 8GB on the message queue, is that configurable? Monitorable?) Tons of constant churn on a table means we have to worry about vacuuming right? Now I have to monitor that too to make sure it's keeping up. Not to mention all the usual operational issues with running relational databases.

No thanks, I'll stick with GCP PubSub or AWS SQS, which are explicitly designed for this use case, and for which I have to setup _no_ infrastructure.

<comment><AUTHOR> \- postgres doesn't maintain a queue for notify/listen, it's purely pubsub. I'm not quite sure how the backpressure could make it to the database

\- A lot of use cases involve not dropping messages after they are processed (like CI jobs, in this example), so you don't have to vacuum the rows

\- If you're comfortable with SQS there's no really big reason to switch, but it makes it so that your project can _only_ run on amazon cloud servers, which is annoying.

<comment><USER28> [https://www.postgresql.org/docs/current/sql- notify.html](https://www.postgresql.org/docs/current/sql-notify.html)

> There is a queue that holds notifications that have been sent but not yet > processed by all listening sessions. If this queue becomes full, > transactions calling NOTIFY will fail at commit. The queue is quite large > (8GB in a standard installation) and should be sufficiently sized for almost > every use case.

My understanding of MVCC (correct me if I'm wrong), is every time you do an update, dead tuples are left behind that need to eventually be vacuumed. If vacuum isn't running or can't keep up, you'll run out of space or have a tx id wrap around.

Work queues are a simple paradigm and swapping out one queue for another is a trivial exercise that won't change the architecture of a system, and SQS or PubSub could be easily used, regardless of where the rest of the project is hosted. I'd rather pay that one-time cost than the ongoing cost of maintaining my own job queue service and infrastructure.

<comment><USER29> The LISTEN/NOTIFY queue has nothing to do with MVCC. Only reason why it exists is that postgresql synchronizes the notification events with transaction boundaries (ie. you get notifications only when you don't have active transaction). Given this the only case when you would care about the depth of the notification queue is when your application is exceptionally misbehaved and in that case you will run into more critical problems (stale locks, MVCC bloat...) well before that will begin to be an issue.

<comment><USER28> I mentioned MVCC with respect to vacuuming, not the queue.

</comment></comment></comment></comment></comment><comment><USER30> I'm not sure I quite follow this statement:

> In the list above, I skipped things similar to pub/sub servers called "job > queues" \- they only let one "subscriber" watch for new "events" at a time, > and keep a queue of unprocessed events:

If your job queue only allows one single worker (even per named queue), I'd argue it's a shit job queue.

<comment><USER31> The article mentions Celery and Gearman.

I'm not here to defend celery generally, but it has no such limitation.

<comment><USER30> Right, I spent some time earlier this year researching a number of job queue systems for a client and never came across one that had this limitation, which kind of makes me wonder what exactly the article is trying to claim.

</comment></comment></comment><comment><USER32> Can someone share their experience with scaling pg's NOTIFY and LISTEN?

The use case I have in mind has a lot of logically separate queues. Is it better for each queue to have its own channel (so subscribers can listen to only the queue they need) or have all queues notify a global channel (and have subscribers filter for messages relevant to them). I am mainly confused about whether I need a dedicated db connection per LISTEN query and also how many channels is too much.

<comment><USER33> Based on what I have read and some experimentation, I concluded that it will not scale very well beyond ~500 listener processes because there needs to be one persistent connection per listener. Furthermore, my understanding is using pgbouncer will not help since it needs to use session pooling in order to support LISTEN.

</comment></comment><comment><USER34> Hacks like this work at first, but long running transactions and postgres don't do well together.

After a few weeks of running on a multi-TB table, you'll find the dead tuples massively outnumber the live tuples, and database performance is dropping faster than the Vacuum can keep up. Vacuum is inherently single-threaded, while your processes making dead tuples as part of queries are multi-threaded, so it's obviously the vacuum that fails first if most queries are long running and touch most rows. Your statistics will get old because they're also generated by the vacuum process, making everything yet slower.

Even if you can live with gradually dropping performance, eventually the whole thing will fail when your txids wrap around and the whole database goes read- only.

<comment><USER35> Long running transactions? I didn't think the article implied doing something like that. The job state changes shouldn't take long to execute.

</comment></comment><comment><USER36> Postgres generally has a fairly low maximum connections. If you're running your own servers, you can adjust this, but in the cloud you may not be able to. For example, Google CloudSQL maxes at 1000, Heroku at 500.

At that point, people usually start looking at the connection pooling tools. Depending on how much work you need from the DB, connections pools can be a win. Anyone know how connection pooling works with listeners?

A

<comment><USER37> Can't say about listeners. But connection poolers introduced more problems than I cared to fix. So now I routinely use 3000 connections, and have 9000 set up for peaks. It eats a little more ram but it is more stable.

<comment><USER38> connections aren't free[1] after reducing number of connections from 200 to 60 my commit throughput doubled.

[1] [https://brandur.org/postgres-connections](https://brandur.org/postgres- connections)

</comment></comment></comment><comment><USER39> With today's hardware I argue you most likely do not need a pub/sub service. Considering the extra work needed in a distributed system, you could save a lot of time by keeping it simple.

</comment><comment><USER40> Postgres is now pretty much the ultimate nearly-all purpose backend, it seems. At this point, I won't need to use anything else.

And I'm more than perfectly fine with that.

</comment><comment><USER41> Keep in mind that a notify performed when nobody is listening is lost. The workers need then to catch up.

<comment><USER42> This is where the Debezium connector for Postgres [1] comes in: it will retrieve change events from the TX log and push it to brokers such as Apache Kafka or Pulsar, or directly to your application. When not listening, any consumer will continue to read from where it left off before (applying "at least once" semantics).

Disclaimer: I work on Debezium

[1] [https://debezium.io/documentation/reference/0.10/connectors/...](https://debezium.io/documentation/reference/0.10/connectors/postgresql.html)

<comment><USER43> I fail to see how Debezium would solve this as it's just as likely Debezium wouldn't be watching the WAL as if you had a minimal number of workers always listening and those went down for some reason.

Could you please elaborate?

<comment><USER42> Debezium stores the WAL position it has read ("offset"). So while it's down, of course it won't be able to read any events. But once it's restarted, it will continue to read the WAL from the last offset. So no events will be missed also when the connector isn't running for some time (WAL segments are retained by the DB until acknowledged by a logical decoding client such as Debezium).

</comment></comment></comment></comment><comment><USER44> How do you performance tune PostgreSQL on AWS and still keep it running under a reasonable cost?

<comment><USER45> In my experience, you don't. With tons of IOPS you need EBS or crazy expensive instances.

Instead you use a cloud like google cloud where you can add NVMe SSDs to whatever instance type you need and configure custom RAM and CPU instead of picking from the super expensive AWS instances with no configurable options and almost always the wrong resource allocations for your workload.

Source: testing my infrastructure that requires 60,000 iops on both google cloud and AWS and it being 1/4 the cost and higher performance on Google. Of note: this was a very high throughput streaming data application. YMMV for other applications.

</comment></comment><comment><USER46> I'll soon have to do a pub/sub for an application that's close to a multiplayer video game.

Most advices I have seen say that I'll probably want to code it myself, but I was wondering about the latency of that solution? I'll likely have a SQL store and that would be a good argument to use postgres...

<comment><USER47> Use Redis pubsub instead.

<comment><USER46> Looking into it, it seems to have something like 2ms of latency which is good. Do you think it would be a good idea to use for exchanging, say, realtime position of players in a virtual world?

<comment><USER47> Not with pubsub. You could use it as shared memory store for player states though, I guess.

</comment></comment></comment></comment><comment><USER48> I'm curious if the same holds true if you drop in Sqlite/MS Sql Server/Mysql.

I.e. is this good advice because Postgres in particular is a great implementation of sql, or because sql in general is good enough to solve this problem, or a mix of the two?

<comment><AUTHOR> Postgres in particular:

\- has strong performance vs, say, sqlite

\- has "channel" and "trigger" support so you can avoid polling (which either slows down your jobs or limits your number of workers)

\- is actually OSS (versus, say, mysql)

<comment><USER49> How is MySQL not OSS?

<comment><USER50> It is somewhat split. Not all MySQL features are open source (enterprise edition feature). Postgres is completely open source.

<comment><USER51> Well, yeah, but then there's stuff like enterprisedb which is postgresql + some proprietary stuff on top, and the company employs some of the postgresql core developers. So in some sense it's "PostgreSQL Enterprise Edition" in all but name.

I believe Citus DB(?) is something similar.

<comment><USER52> > I believe Citus DB(?) is something similar.

"Citus Unforks From PostgreSQL, Goes Open Source"

[https://www.citusdata.com/blog/2016/03/24/citus-unforks- goes...](https://www.citusdata.com/blog/2016/03/24/citus-unforks-goes-open- source/)

[https://news.ycombinator.com/item?id=11353322](https://news.ycombinator.com/item?id=11353322)

As for mysql - I don't know why anyone wouldn't prefer mariadb - but I suppose it's conceivable mysql has features that makes it worth dealing with oracle licensing. But I doubt it.

</comment></comment></comment></comment></comment></comment><comment><USER53> I've simply been using MySQL/Maria everywhere and am meaning to switch, but I'm not sure what makes Postgres this much better.

Can this hack not be achieved by a mariadb table too?

<comment><USER54> The key features used here for PG seems SKIP LOCKED, and the PG notification system, which informs all clients that are directly connected to the database that "this table has changed" so that they can restart their work.

The notification system comes with a cost. You can't scale up the number of connection to a PG database instance.

Therefore you need to go pooling.

And for SKIP LOCKED, I just checked and mysql/maria do have it.

</comment></comment><comment><USER55> I did something very similar with MySQL since that was the DB we already had setup for data (not my choice). Basically any database that has atomic operations can do this. It definitely is way more convenient and cheaper to do use your existing infra for pub/sub and then only scale out to other services once performance becomes subpar. Although if you already have a messaging service up and running, it's probably better to use that

</comment><comment><USER56> Thanks, seeing that atomic fetch in action is very useful.

</comment><comment><USER57> if you use Postgres on AWS, you'll easily exhaust IOPS with this and hell breaks

</comment><comment><USER58> I agree. I have prototyped this in the past, but our current pub/sub was not painful enough for us to go full steam ahead with PG.

However, my design was more bare bones. I was picking jobs by chaining CTE's that did the status update as they returned the first element of the queue.

</comment><comment><USER59> Should the "ci_job_status" in the line be "ci_jobs" instead?

         INSERT INTO ci_job_status(repositor ...

BTW, it looks Go becomes so popular that many tutorials are using Go for examples. ;D

</comment><comment><USER60> Interesting enough, but I'm not sure why this is called a hack. It's a fully supported feature of Pg

</comment><comment><USER61> Kinesis is based on DynamoDB...

Key-value stores could do a lot of things theoretically.

<comment><USER62> Kinesis is based on DynamoDB? Can you share where you saw that?

</comment></comment><comment><USER63> [http://mikehadlow.blogspot.com/2012/04/database-as-queue- ant...](http://mikehadlow.blogspot.com/2012/04/database-as-queue-anti- pattern.html)

<comment><USER64> The concerns in this post have already been answered in OP's post.

</comment></comment><comment><USER65> Another neat hack would be to use Postgres as a graph db

<comment><USER66> Check out AgensGraph, it's graph DB engine built on postgres. I haven't used it for much more than toy projects at this point though.

[https://github.com/bitnine-oss/agensgraph](https://github.com/bitnine- oss/agensgraph)

</comment></comment><comment><USER67> How do you manage the bloat?

<comment><USER68> Typically (at least when I've implemented variations of this pattern) a batch job run on a schedule performing something like "expire stale/completed jobs past a threshold."

If you needed the existence of completed jobs to compute future jobs, use a supplementary datastructure that stores spans or other heuristics, updated in a transaction with job completion

</comment></comment><comment><USER69> Used the "FOR UPDATE SKIP LOCKED LIMIT 1" trick to implement a job server in PG a few years ago for the first time.

It's a great solution.

<comment><USER70> Why the "LIMIT 1"?

<comment><USER69> It makes the job-fetching more atomic, as the article points out properly. If you take >1, you have to sort the situation where 1 or more jobs fail, which adds an unreasonable amount of complexity.

On my first go I was taking batches, on the reasoning that it optimized for query performance. It was a complete disaster and I quickly realized that I was engaging in premature optimization.

Batching is out, stream-processing is in. If you design your job table correctly, PG will perform very well as an advanced stream processor.</comment></comment></comment>


Show HN: Bash-My-AWS – CLI Commands for AWS - mike-bailey https://bash-my-aws.org/ <comment> mike-bailey It's probably my fault if you haven't heard of Bash-My-AWS.

Bash-My-AWS is a simple but extremely powerful set of CLI commands for managing resources on Amazon Web Services. They harness the power of Amazon's AWSCLI, while abstracting away the verbosity. The project implements some innovative patterns but (arguably) remains simple, beautiful, readable and easily extensible.

The project started in 2014 and while many hundreds of hours have gone into it, far less has gone into promotion.

I'm speaking about it at LinuxConf and have created a documentation site at [https://bash-my-aws.org](https://bash-my-aws.org)

[https://linux.conf.au/schedule/presentation/144/](https://linux.conf.au/schedule/presentation/144/)

<comment><AUTHOR> for anyone on this thread that is interested. I run [https://getcommandeer.com](https://getcommandeer.com) which is a tool to manage your AWS and IAC infrastructure from a desktop GUI. I love this bash- my-aws, as we are about to release Bash, Docker Compose, and Terraform Runners. We already have Serverless and Ansible runners. They enable you to run your command line system from a GUI, so that you can instantly switch between AWS accounts/regions and even LocalStack. Because it is a desktop, under the hood we are really running cli tools mixed in with some AWS JS SDK.

<comment><USER2> Why does your main website (linked in your comment) require JS to run? I've not seen anything that warrants that requirement.

I wish people would stop making websites that's require JS.

<comment><USER3> Spoiler alert: if you don't want to run JavaScript in your browser, you're not going to enjoy this application.

<comment><AUTHOR> yea, it works terrible without a monitor as well. The site does not work headless. what was I thinking? It's a desktop application. The site is where you download the app.

<comment><AUTHOR> To be a little more constructive with my answer. The website needs javascript for our chat service, our newsletter service, our ability for users to purchase licenses for the app, the night/dark mode switch. The codebase also shares components with our desktop app as they are both Vue, but with the desktop app utilizing electron. We will look to make the site be html only at some point, but it is not a focus of our dev team. If you download the app, you will see that it is a first rate experience for managing many AWS services. The app itself probably has about 1,000 man hours or more put into it. Would love to hear insight into how we can make it even more powerful for the community.

</comment></comment></comment></comment></comment></comment><comment><USER4> This looks like an awesome project!

Meta note: All things considered, Amazon has it pretty good. They put out a barely usable, bare-bones, but fully functional tool in awscli. Paying customers of AWS have to perform the engineering effort to make the API more usable, and some even open-source their projects like this. AWS is an incredible business model.

<comment><USER5> I have sometimes questioned whether I should be spending my personal time developing an open source tool so tied to a single companies services.

My reasons for continuing include:

\- I prefer to use command line over ClickOps

\- Using Bash-My-AWS makes me more effective at work

\- The emergent UX is equally applicable to other services (e.g. bash-my- github, bash-my-spotify)

\- The intrinsic satisfaction from creating

\- Helping improve the experience for others

<comment><USER6> I looked, no one created packages with those 2 names that I could fine, although there are some clis for controlling spotify it seems. Were you referring to specific packages somewhere for github and spotify?

<comment><USER5> bash-my-github and bash-my-spotify are two things I've made a start on but are not public yet. They've been on the backburner for a while due to competing priorities.

I was able to write a simple command that returned all songs a friend and I had in common in our public playlists.

I forget the exact syntax but it was something roughly along the lines of:

      $ sort <(
          user-playlists alice | playlist-tracks
          user-playlists bob | playlist-tracks
        ) | uniq --repeated

</comment></comment></comment></comment><comment><USER7> Coming from Google Cloud, I couldn't deal with the atrocity that is awscli, so I ended up eventually implementing the bare minimum of shell wrappers to at least start, stop, ssh into, rsync files to and from, etc, my aws instances _by name_, not by instance ID. Took me a couple of hours to cobble it together.

Google cloud CLI offers all of this out of the box. Why Amazon wants to make such basic commands difficult, I'll never understand.

</comment><comment><USER8> This looks pretty great, while the AWS CLI is very comprehensive, I always struggle to remember which flags are needed for each command, and it's not very consistent.

One thing I've not been able to work out with bash-my-aws yet was how to easily switch between regions and accounts. I noticed you can use `region` on it's own to set the current default region, but I'm often working with multiple regions, and it'd be a pain to have to run `region us-west-1` separately each time I want to use a different region. I couldn't see a way to just specify a region for a given command (eg how you'd do `aws get-instances --region us-west-1`). I guess you could do this with the environment variable `AWS_DEFAULT_REGION=us-west-1 instances` but that's a bit verbose.

Similarly with AWS accounts, I use multiple AWS accounts, which are accessed with different access keys, which are defined as profiles in my ~/.aws/config. Normally I'd use these with the AWS CLI like `aws ec2 get-instances --profile production`, I couldn't see any way in the docs to use or set this?

<comment><USER5> They're good questions. I can tell you how I manage regions and accounts but am interested in learning how people think Bash-my-AWS might better support users in this regard.

The AWCLI, as well as SDKs all support grabbing Regions and account credentials from environment variables.

For Regions, I work tend to use the following aliases:

      alias au='export AWS_DEFAULT_REGION=ap-southeast-2'
      alias us='export AWS_DEFAULT_REGION=us-east-1'
      alias dr='export AWS_DEFAULT_REGION=ap-southeast-1'

I normally work in a single Region and swap when required by typing the 2 character alias.

To run a script or command (doesn't have to be Bash-my-AWS) across all Regions I use region-each:

      $ region-each stacks | column -t
      example-ec2-ap-northeast-1  CREATE_COMPLETE  2011-05-23T15:47:44Z  NEVER_UPDATED  NOT_NESTED  #ap-northeast-1
      example-ec2-ap-northeast-2  CREATE_COMPLETE  2011-05-23T15:47:44Z  NEVER_UPDATED  NOT_NESTED  #ap-northeast-2
      ...
      example-ec2-us-west-2       CREATE_COMPLETE  2011-05-23T15:47:44Z  NEVER_UPDATED  NOT_NESTED  #us-west-2

For AWS accounts, I type the name of the account and I'm in. For accounts using IDP (ldap/AD backed corporate logins) I generate aliases so I have tab completion and simple naming.

In accounts that are only setup to use AWS keys, I use aliases that export credentials kept in GPG encrypted files. Last time I looked, AWS docs suggested keeping these long lives credentials in plaintext files readable by your account. That's asking for trouble IMO, especially if they're kept in a known location that a compromised node library could exfiltrate them from.

AWSCLI v2 beta includes support for SSO so it's probably a good time to look at how BMA could include support for auth.

</comment></comment><comment><USER9> Interesting this requires 'jq' when JMESpath is built into AWS CLI already.

[http://jmespath.org/](http://jmespath.org/)

<comment><USER5> jq is only used in three of the >120 functions (for sort-keys functionality). All the rest use JMESPath.

If anyone can help with a solution I'd be delighted to remove the dependency on jq.

[https://github.com/bash-my-aws/bash-my- aws/blob/b74d92a902bb...](https://github.com/bash-my-aws/bash-my- aws/blob/b74d92a902bb54ab7527eb40e79d397b3895bf0c/lib/stack-functions#L326)

<comment><USER10> I would definitely keep jq dependency there, especially if you plan to expand the code base to provide ecs and Fargate commands. You will quickly run into use cases where JMESpath is not capable of parsing json outputs for different tasksdefintion commands.

</comment></comment></comment><comment><USER11> Just the other day I was looking for an official docker image that includes the AWS CLI. On top of that, and mainly, I was looking to find more documentation or tooling to better automate the deployment of new AWS projects.

Does anyone here have any experience of (starting from scratch or with no AWS resources) setting up policies/users/resources/configurations via something similar to the Deployment Managers of GCP and Azure?.. preferably something declarative or via templates?

Bash-my-AWS looks like a great step towards the goal I have in mind but I may also just be unaware of other tooling or AWS capabilities.

<comment><USER12> Like it or not (I do...) terraform is the de-facto industry standard, and pretty much the only mature cloud resources management tool I'm aware of.

It is unwise IMHO to use CloudFormation currently unless you're provisioning resources so obscure they didn't yet make it to tf aws provider.

BTW your Dockerfile pretty much boils down to:

        FROM alpine:3.10

        RUN apk add --no-cache \
            python3

        RUN pip3 install awscli

        COPY config /root/.aws/
        COPY credentials /root/.aws/

<comment><USER13> Have a look at CDK. It's a framework on top of CF to use python/javascript/etc made by AWS. I've been trying it out recently to try to move away from TF and it's a promising alternative.

<comment><USER14> My problem with CF is the CF part, not the yaml. It takes just a few times getting stuck in a rollback loop to hate CF forever.

Especially when you contact AWS support and they tell you the only thing you can do is wait.

</comment></comment></comment></comment><comment><USER15> Interacting with individual DNS records in Route 53 is very hard using AWSCLI. I wrote a Python wrapper around the Route 53 API to make it easier to do command line records management (and also to do dynamic DNS with your own Route 53 hosted domain):

[https://github.com/ericfitz/r53](https://github.com/ericfitz/r53)

</comment><comment><USER16> Is this primarily required because AWS CLI is not good enough at listing resources in desired format (json, jsonpath, yaml, table..)?

<comment><USER5> AWSCLI is amazingly flexible and powerful.

Bash-My-AWS thinly wraps AWSCLI commands that would otherwise be too long to type. So you're still using AWSCLI and can improve your skill with it by inspecting the source of Bash-My-AWS functions.

[https://news.ycombinator.com/item?id=21931298](https://news.ycombinator.com/item?id=21931298)

</comment></comment><comment><USER17> If you want to easily manipulate your AWS environment from the command line use the AWS cmdlets for PowerShell. The fact that PowerShell cmdlets work on objects instead of text makes them miles better than this or the AWS CLI because you don't spend most of your time figuring out how to wrangle text into meaningful output.

<comment><USER18> Hasn't AWSCLI supported toggling the cmd output to either text, json or csv for quite some time now or have I misunderstood your comment here?

<comment><USER5> Bash-My-AWS wraps AWSCLI as thinly as possible and makes use of JMESPath and the text output.

The result is you have a simple set of commands that don't require you to type hundreds of characters.

      instances() {
        local instance_ids=$(__bma_read_inputs)
        local filters=$(__bma_read_filters $@)

        aws ec2 describe-instances                                            \
          $([[ -n ${instance_ids} ]] && echo --instance-ids ${instance_ids})  \
          --query "
            Reservations[].Instances[][
              InstanceId,
              InstanceType,
              State.Name,
              [Tags[?Key=='Name'].Value][0][0],
              LaunchTime,
              Placement.AvailabilityZone,
              VpcId
            ]"                                                               \
          --output text       |
        grep -E -- "$filters" |
        LC_ALL=C sort -b -k 6 |
        column -s$'\t' -t
      }

</comment></comment></comment><comment><USER19> I have developed something similar on top of the AWS CLI that incorporates a bunch of integrations with other tools like the cloudinit and various bits of Batch-related instrumentation: [https://github.com/kislyuk/aegea](https://github.com/kislyuk/aegea)

</comment><comment><USER20> The first thing I'd do after installing this is to alias prefixing "aws-" to every commands. I like namespacing things as I suck at remembering names...

<comment><USER5> I don't like to rely on my memory either! I forget the names of commands and use tab completion to list them.

You can just type bma[TAB][TAB] and it will list them all.

If you know the type of resource you are working with, you can use TAB completion to see it's commands:

      $ stack-
      stack-arn            stack-exports        stack-tag-apply
      stack-asg-instances  stack-failure        stack-tag-delete
      stack-asgs           stack-instances      stack-tags
      stack-cancel-update  stack-outputs        stack-tags-text
      stack-create         stack-parameters     stack-tail
      stack-delete         stack-recreate       stack-template
      stack-diff           stack-resources      stack-update
      stack-elbs           stack-status         stack-validate
      stack-events         stack-tag

</comment></comment><comment><USER21> What really sells me on this tool is the ability to examine the underlying awscli command and transformations. I'll be giving this a go in the new year!

<comment><USER5> Thanks!

The intent has always been to enhance rather than replace AWCLI (which is an amazing tool!).

If you're ever wondering how a Bash-My-AWS command works, use `bma type` (it even supports tab completion for all the commands).

      $ bma type instances
      instances is a function
      instances ()
      {
          local instance_ids=$(__bma_read_inputs);
          local filters=$(__bma_read_filters $@);
          aws ec2 describe-instances $([[ -n ${instance_ids} ]] && echo --instance-ids ${instance_ids}) --query "
              Reservations[].Instances[][
                InstanceId,
                InstanceType,
                State.Name,
                [Tags[?Key=='Name'].Value][0][0],
                LaunchTime,
                Placement.AvailabilityZone,
                VpcId
              ]" --output text | grep -E -- "$filters" | LC_ALL=C sort -b -k 6 | column -s' ' -t
      }</comment></comment>


Pakistan, A Popular Outsourcing Destination? - <AUTHOR> http://www.businessweek.com/technology/technology_at_work/archives/2009/06/pakistan_a_popu.html#comments

<comment><USER2> why not rest 19 above Pakistan?

<comment><AUTHOR> I don't see any reason for that except the "perceived risk"</comment></comment>


Ask HN: How do you decide your cloud provider - <AUTHOR> I've engaged with a number of startups over the years (as well as one's I've been part of) and the answer to the above question is either:
- it's what we know
or
- it provides what's needed
My specific question is more around choosing one of the larger "clouds" - heroku, digital ocean, azure, aws, google, etc.
Do cost, features, etc. play into it? <comment><USER2> Price efficiency, Stability, Optimized for developer time and not operational costs, aka capabilities.

I've used AWS, GCloud, DO in production and Heroku for side projects.</comment>


Google Launches Chrome-To-Phone Extension To Link Chrome With Android - <AUTHOR> http://techcrunch.com/2010/08/12/google-launches-chrome-to-phone-extension-to-link-chrome-with-android/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Techcrunch+%28TechCrunch%29&utm_content=FaceBook <comment><USER2> I've tried this. It's extremely neat. Much better than emailing a URL to yourself.

<comment><USER3> Agreed. And the maps functionality rocks, too.

</comment></comment><comment><USER4> For those stuck on pre-2.2 phones/firmwares, a friend wrote:

<http://linkpush.appspot.com/>

</comment><comment><USER5> would be great for android powered tvs

</comment><comment><USER6> I would be happy just accessing my chrome bookmarks on the phone, without having to send them from chrome first...

</comment><comment><USER7> No support for Google Apps accounts. Still.</comment>


Google ranks top 'Torrent sites'  - <AUTHOR> http://startupmeme.com/google-ranks-top-torrent-sites/

<comment><USER2> TvTorrents does a lot to organize the chaos of bittorrents of TV shows. There's only one torrent per episode for 'normal' reolution and another for high-def. Episodes are organized into TV series. There are categories and ratings as well.

If this sort of site was combined with an app that packaged this information to users, you'd have Tivo-like functionality.

<comment><USER3> Try Miro.

</comment></comment><comment><USER4> I'm always glad when I hear thepiratebay isn't #1. Hate the owners and their "we are invincible" attitudes.

Am I the only person? It always seems like they are seen as a modern day Jesuses by a lot of people...

<comment><USER5> Piratpartiet 2008!

</comment></comment><comment><AUTHOR> [http://trends.google.com/websites?q=mininova.org%2C+thepirat...](http://trends.google.com/websites?q=mininova.org%2C+thepiratebay.org%2C+isohunt.com%2C+torrentz.com&geo=all&date=all&sort=0)

check out the regions, pretty cool.

And according to <http://twitter.com/mininova> , it says "Mininova in July: 35.6MM uniques, 105.2MM visits, 512.1MM pageviews. 7.6% increase compared to June"

Thats some serious traffic.

</comment><comment><USER6> I think it's a little misleading, because whilst mininova is arguably the most popular torrent site itself, thepiratebay's tracker is the most used.</comment>


From Wordpress to GatsbyJS: A Solution - <AUTHOR> https://www.rodrigoplp.com/blog/migrating-blog-from-wordpress-to-gatsbyjs-a-solution <comment><AUTHOR> Migrating a blog from Wordpress to Gatsby is not so straight forward. There are some compromises that have to be made along the way. This is a detailed description of the choices I made.</comment>


TechStars: Boulder Demo Day from the Cheap Seats - <AUTHOR> http://boulder.me/techstars-demo-day-from-the-cheap-seats/

<comment><USER2> I wonder why Techstars and companies get so little coverage, Comments and Upvotes on HN.

<comment><AUTHOR> I think it depends. I've seen some decent discussion of TechStars on here before. I'm sure there is a Harvard and Yale aspect to it it as well. But overall I think that HN is a fairly impartial site. I would be much more inclined to think that people just didn't like the article then think they were boycotting TechStars.

<comment><USER2> I globally agree on HN fairness, was just curious on recent posts.

<comment><AUTHOR> In this specific case, there was a better article from TechCrunch covering the same info. I just didn't see it before submitting. Also, it was upvoted quite decently.</comment></comment></comment></comment>


Comcast claims it'll spend $50B because of net neutrality repeal and tax cut - <AUTHOR> https://arstechnica.com/tech-policy/2017/12/comcasts-network-investments-soared-with-net-neutrality-rules-in-place/ <comment><USER2> These claims are an important follow-on to claims made by the FCC that ISPs were holding back on investment because of net neutrality, therefore it had to be repealed.

So this is a way of publicly backing up the FCC, and heading off possible independent action by congress.

It's a kind of theater ... what if they never got it repealed? So they would never re-invest in their business?

<comment><USER2> I guess it makes sense if you read it as: we have this money that now we can spend on schemes that violate net neutrality and shake down businesses that cross our path!

</comment></comment><comment><USER3> So when are we getting the fiber at our doors, that is a decade late, and we've already paid out billions for?

<comment><USER4> After just the right amount of over paying and complaints.

</comment></comment><comment><USER5> Seems to fit:

[http://slatestarcodex.com/2017/01/03/trump-and-the-batman- ef...](http://slatestarcodex.com/2017/01/03/trump-and-the-batman-effect/)

"When US companies do something that sounds good in the next few years, whether it's hiring new people, or deciding to stay in the United States, or reporting high profits, some of them are going to credit President Trump. First, because it's going to get them good press. "Ford decides not to build plant in Mexico" is tenth-page news. "Ford decides not to build plant in Mexico because of President Trump" is front-page news.

But second, because it's going to make the President like them."

</comment><comment><USER6> And then they'll raise prices yet again and continue to deliver horrible service, packages nobody wants, etc.

</comment><comment><USER7> Sure, but they will also raise prices and charge for fast lanes to make up that spending and more.</comment>


The mystery of what causes the holes in Swiss cheese has been solved - <AUTHOR> https://www.theguardian.com/world/2015/may/28/swiss-scientists-hole-in-cheese-emmenthal-hay-particles-milk-bacteria <comment><USER2> Article is from 2015.

<comment><AUTHOR> Abstract: [https://www.sciencedirect.com/science/article/abs/pii/S09586...](https://www.sciencedirect.com/science/article/abs/pii/S0958694615000631)

Text of abstract: The production of Swiss-type cheeses with a typical number, size, and distribution of eyes is a difficult task, especially when bactofuged or microfiltrated milk is utilised. In this study, the potential of microparticles (plant origin) to influence eye formation in cheese, was assessed. Eight experimental Emmental cheeses were produced with one replicate from microfiltrated milk with addition of 0.0625–4.000 mg of powdered hay to the milk (90 L) and ripened for 130 days. Eye formation was quantified by means of X-ray computed tomography (between 30 and 130 days). The contents of fat, water, citric acid, lactic acid, and volatile carboxylic acids were determined at 130 days. The results demonstrate that microparticles of plant origin act as eye nuclei that control the number (P < 0.001) and size of the eyes in cheese in a dose-dependent manner. The findings also provide new insights into the formation of eye defects.</comment></comment>


May I help you? No thanks, just looking. - <AUTHOR> http://bijansabet.com/post/1108802698/may-i-help-you-no-thanks-just-looking

<comment><AUTHOR> Been thinking a lot about this space and am almost done with a prototype. Would love to connect with others who are similarly fascinated by it.

<comment><USER2> I'm not a skilled programmer, but I defintiely see potential for some far reaching machine learning opportunities for this.</comment></comment>


The Evolution of JavaScript Modularity - <AUTHOR> https://github.com/myshov/history_of_javascript/tree/master/4_evolution_of_js_modularity <comment><AUTHOR> Hi, everyone!

I want to share with all of you my article about modularity in JavaScript. I know that there are a lot of other articles about the same theme, so I took a different approach. instead of simple repetition what has been already said in the other articles about AMD, CommonJS, UMD, ES2015 Modules, you will learn a lot of other approaches for the modularity. Also you will find a lot information about the history of emerging these technologies and you will learn about the people who stood behind of them.

I wanted to write this article as comprehensive as possible. It may sound ridiculous but I was working on this project in course of more than a half of year in my spare time. So I hope you will probably learn something new, even if you are already seasoned JavaScript developer.

Enjoy!</comment>


Parents who digitize their babies' lives - <AUTHOR> https://www.theguardian.com/lifeandstyle/2019/mar/02/apps-that-track-babies-and-give-data-to-tech-firms-parents <comment><USER2> > I ask Workman why parents should trust him with all that data. "You know, > that's a great question," he says. "Honestly? Around data, I am not the > expert on how and where this is stored." Apparently, I would have to talk to > one of his data guys about that.

Is it really asking too much for the CEO to have a legal- and engineering- approved answer to this question?

I mean, I'm glad he's being honest, but it clearly shows that he's out of touch with his customers if he hasn't familiarized himself with the basic details of one of consumers' biggest concerns in this domain.

<comment><USER3> > one of consumers' biggest concerns in this domain.

I hear privacy concerns mentioned by people a lot, but in the non-tech crowd I see nearly zero online behaviour that values privacy. One of us is making incorrect assumptions about user behaviour and privacy wishes. I hope it's me.

<comment><USER4> > One of us is making incorrect assumptions about user behaviour and privacy > wishes. I hope it's me.

I heard a lot about people's privacy. Most non-tech people I talk with see Google as a faceless evil corporation that wants to control humanity.

But, they use Facebook, and browse the web while logged in in GMail, Facebook, etc. instead of opening a private tab.

That is the problem with lack of knowledge. People don't trust big corporations, sometimes to irrational extends. But, they do not understand how the technology really works.

If we want behaviour and concern to be one consequence of the other, we need better education and more government action.

<comment><USER5> >If we want behaviour and concern to be one consequence of the other, we need better education and more government action.

I think government action would more align with the realities of better education. The better education aspect, however, relies on people being willing to keep-up with the growth of technology after their compulsory education ends; which is, probably, going to be your hardest sell.

For those of us, for whom technology is a an every-day necessity and/or the source of our income, that premise in the interests of continued education in technology seems second nature. However, if you compare that to doctors, they're primary concern is keeping up with medical trends; so, this might be a harder "sell" to them.

Instead, much like other professions have governing bodies, technology should probably have it's own. I'm thinking along the lines of things like doctors and medical review boards or lawyers and the BAR.

However, there is another caveat that the governments benefit from: Ineptitude around technology is what governments actually benefit from. Telling the governments that they should care more about education around technology and creating laws to protect basic rights is going to be a hard-sell to some of them. Take the current issue with the US law versus the GDPR for a principal example.

I don't pretend to have an answer but, to me, it generally comes down to whether the government is far more concerned with the welfare of itself than the welfare of people (in general).

</comment></comment></comment></comment><comment><USER6> As with any third party hosting service, they exist until they don't and when they don't then you better be sure you have a copy of that data in your personal possession. If not then the dying hosting service can monetize your access to the data like a fire sale. Don't be put in that predicament as you simply can't trust that the vendor will be alive in five years times.

</comment><comment><USER7> This is the logical evolution of helicopter parents, considering the state of technology. I would call them "facebook parents".</comment>


Cos 333: Advanced Programming Techniques - <AUTHOR> https://www.cs.princeton.edu/courses/archive/spring19/cos333/ <comment><AUTHOR> "This is a course about the practice of programming, an attempt to expose students to the development of real programs. Programming is more than just writing code. Programmers must also assess tradeoffs, choose among design alternatives, debug and test, improve performance, and maintain software written by themselves and others. At the same time, they must be concerned with compatibility, robustness, and reliability, while meeting specifications. Students will have the opportunity to develop these skills by working on their own code and in group projects."</comment>


Dramatically Reducing Software Vulnerabilities [pdf] - <AUTHOR> http://csrc.nist.gov/publications/drafts/nistir-8151/nistir8151_draft.pdf <comment><USER2> Of the many recurring stupid ideas for startups I have had is a software company that does not write code. Instead it edits and removes code. This has been called "negative coding."

Few people write small programs anymore. UNIX, the OS that runs the internet and on which the "cloud" depends, is a time-capsule full of small programs. What this might mean is left as an exercise for the reader.

Today, large programs are celebrated without question.

Gratuitous complexity can be disguised as "productivity" and pitched as value creation. At the very least it can be a source of fascination for futurists. Any liabilities created in the process can be easily explained away.

Among the many things that can be done with this output, one of them to is to make small programs from the larger ones. As far as I can tell, this is not being done.

For myself, I have taken large, open source programs and gutted all the needless features in order to produce the small programs I want.

My goal is less complexity. Simplicity is a loaded term. For myself, this process of incrementally reducing complexity "creates value". It is an intellectual exercise, there is a sense of productivity, and yet almost no code is written.

Software companies are full of developers who cut, paste and compile. Today many of them are working with open sourced code. The software company I aim to start is comprised of "programmers" who also work with such open sourced code, they only cut and compile.

EDIT: Whenever I see the "Who's hiring" thread I always wish I could put in an entry for this. The vision I have is a group of "early employees" with day jobs. Absent connections and name dropping no VC would ever fund this. No one will be paid to do this "negative coding" work until we secure our first government or Fortune 100 contract and they see the results. And even then I could not guarantee the results would be appropriately valued. But were this fictional company to succeed, those early employees could claim they were an integral part of a monumental paradigm shift.

EDIT 2: The fictional company does negative coding on _single programs_ to reduce them in size or to break them up into smaller programs and is not limited to the embedded market or driven by resource constraints. Selecting parts of an open source OS distribution to create a new distribution is a different idea and of course many companies have been doing that for many years.

<comment><USER3> I even have a tag line for you: "Do better with less"

<comment><USER4> Ironically, that's the jQuery slogan [1]: Write less, do more

[1] [https://jquery.com/](https://jquery.com/)

<comment><USER5> ...while relying on more.

Using frameworks/libraries often can reduce the amount of code one has to write to perform a certain task, but it also can greatly increase the _total_ complexity of the system. The OP is talking about reducing total complexity, which may mean increasing local complexity slightly.

</comment></comment></comment></comment><comment><USER6> They should have pasted this diagram all over the document: [https://en.wikipedia.org/wiki/Automata_theory](https://en.wikipedia.org/wiki/Automata_theory)

I'm disappointed that the "Formal Methods" section makes only a passing reference to Finite State Machines. FSMs are arguably the easiest to understand and use formal methodology, and probably the most prevalent. Especially worth emphasizing given how difficult and opaque a lot of formal methods remain to even people with 4 year CS bachelor degrees.

This old thread [https://news.ycombinator.com/item?id=2949543](https://news.ycombinator.com/item?id=2949543) had lots of interesting discussions on FSMs, particularly their current widespread use in (sometimes mission-critical) embedded systems. An ongoing problem seems to be a lack of robust first-class support for FSMs in every day programming languages, which can limit debugging, static analysis, standard tools etc.

Disclaimer: I know nothing about security or formal methods.

<comment><USER7> People making arguments about automata theory in security are often affiliated with the LANGSEC program/project/movement.

[http://langsec.org/](http://langsec.org/)

There are lots of compelling ideas there (and LANGSEC researchers would tend to agree that if a functionality can be implemented correctly as an FSM, that would be a safer option).

A related phenomenon that people talk about is the unexpected Turing- completeness (where people have been able to prove that so many different parts of computing are Turing-complete -- things that were never _intended_ to be programming languages). LANGSEC people and others refer to the negative security implications of some of this as "weird machines", where you really didn't want Turing-completeness but you got it by accident or by default anyway, and it might be possible for an attacker who can corrupt control flow or other kinds of state to then perform arbitrary operations. For security people an example may be return-oriented programming, but there are evidently others that can be thought of in the same way.

<comment><USER8> This is fascinating. Do you have any examples or stories of such 'weird machines'?

<comment><USER7> The concept is presented in

[http://www.langsec.org/papers/Bratus.pdf](http://www.langsec.org/papers/Bratus.pdf)

I think people have found a number of other, well, weird examples. I found

[https://www.usenix.org/conference/woot15/workshop- program/pr...](https://www.usenix.org/conference/woot15/workshop- program/presentation/rushanan)

via Google Scholar and there are more papers on this theme if you search for "weird machines" there. It seems like a helpful way to think about this issue!

</comment></comment></comment></comment><comment><USER9> I wonder if this'll (or something like it) be the tipping point for governments to introduce more regulations for software? We already have regulations like PCI-DSS for dealing with payment card, maybe this'll end up with regulations for dealing with security?

The conclusion suggests better specification, better testing and more resilient architectures are the strategies that could make a difference. Perhaps software engineering will have to agree on a set of "best" practices that make security problems less likely to occur?

(typos)

<comment><USER10> Regulations in software tend to encourage compliance theater rather than delivering the spirit of their intent in my experience. It isn't that difficult to conform to regulations technically with an awful and careless implementation, and I've seen it happen many times. It is "best effort", in part because it is obscenely onerous or impossible to ensure some properties in a strict sense.

Best practices also drift over time and vary with context. A challenge is that there are very old, very large code bases where as a practical matter will never be modernized because it is tantamount to a rewrite, with all of the bug creation and cost implied.

Robust software is definitely possible today even in languages like C++, but the development process for software that rarely fails isn't that compatible with the popular "fast iteration" development model nor the current computer science skills of most software engineers.

<comment><USER11> All of this is true, which is why effective policy interventions would punish bad outcomes, not prescribe methodology.

Make bad software expensive and software quality will eventually increase.

<comment><USER12> Bad software is already expensive. This is just poorly understood.

It's simply not fair to non-specialists to expect them to have a working knowledge of how cost works in software.

<comment><USER11> Fines and vulnerability to class actions make the costs far more visible (and increase it besides). If non-specialists routinely make bad decisions because they don't understand how cost works, then the solution is to make the cost of bad software dead obvious.

</comment></comment></comment></comment></comment><comment><USER13> Although it does mention some risks of using 3rd party code/components in passing under 'Understanding and Controlling Dependencies' (835-), I would have expected them to make a stronger case for OWASP 2013-TOP10-A09: 'using components with known vulnerabilities'[1] type of analysis. This applies to almost any kind of modern application, not just web apps, but especially to those written in native languages (C/C++/...) as we increasingly build our applications by bringing in varying amounts of 3rd party code; open source or commercial. Like traditional static code analysis, more or less mature tooling exists in open source[2][3][4] and commercial capacity to perform static binary and source code analysis to discover and track 'offending' 3rd party code, which does make it practical to include this kind of analysis to sdl/sdlc/devops workflows.

[1] [https://www.owasp.org/index.php/Top_10_2013-A9-Using_Compone...](https://www.owasp.org/index.php/Top_10_2013-A9-Using_Components_with_Known_Vulnerabilities) [2] [http://www.binaryanalysis.org/en/home](http://www.binaryanalysis.org/en/home) [3] [https://www.owasp.org/index.php/OWASP_Dependency_Check](https://www.owasp.org/index.php/OWASP_Dependency_Check) [4] [https://github.com/OWASP/SafeNuGet](https://github.com/OWASP/SafeNuGet)

</comment><comment><USER14> This is an interesting read. Despite this, I still cannot understand what a Formal Method is.

It just says a method is 'proven'? How? Can someone please develop a piece of software, document/video themselves doing it and explain what/why they are doing parts of the project?

Basically walk a plebeian like me through making a program using formal methods.

<comment><USER15> It's a slide deck, not a paper, but you might find this presentation from my class helpful: [http://www.dwheeler.com/secure-class/presentations/Secure- So...](http://www.dwheeler.com/secure-class/presentations/Secure- Software-10-Formal-Methods.ppt)

A specific example of software developed using formal methods is Tokeneer: [http://www.adacore.com/sparkpro/tokeneer/](http://www.adacore.com/sparkpro/tokeneer/)

There's a severe lack of publicly-worked examples. Most formal methods applications are private, which inhibits improvement and use. I'm an advocate for the creation of more "open proofs" where the tools, software being proved, and the proofs are all available as open source software. If we had more open proofs then it'd be a lot easier to understand, apply, and improve them.

<comment><USER16> I believe the capability of ada to be formally proven was one of the reasons it has seen use in defence and real-time systems.

<comment><USER17> Well, there was the mandate plus the language itself was better than others at safety & programming in the large. The bigger reason, though, was that it was systematically designed to eliminate error from the syntax to memory handling to integrations. I mean, really thorough. Good description of all that in case anyone wants to cherry-pick safety features for use in a simpler language.

[http://www.adacore.com/knowledge/technical-papers/safe- and-s...](http://www.adacore.com/knowledge/technical-papers/safe-and-secure- software-an-invitation-to-ada-2012/)

</comment></comment></comment></comment><comment><USER18> I find this hilarious in light of what we know about the NSA. All solid recommendations, but they are bandaids when compared to the sheer weight of capital that is being misused by hoarding known vulns. Unfunded mandates are fine for grandstanding.

<comment><USER7> I strongly oppose vulnerability hoarding, but I'd point out that if we don't think it's easy to make political or cultural progress against it quickly, a technical alternative to make people safer is actually finding ways to make safe software. Plus, if "bugs are dense" advocates are right, we need major qualitative improvements to software development in order to make much of an impact (because they argue that different hoarders, or even different from- scratch researchers, would primarily know about _different_ bugs).

So it's not unreasonable for NIST (which actually wants people to enjoy computer security) to investigate what could be done in this area!

Edit: but it's true that it would be awesome to see government do something to make these improvements actually happen -- and a research report probably isn't that.

</comment></comment><comment><USER19> Incidentally, BSIMM 7 ([https://www.bsimm.com/download/](https://www.bsimm.com/download/)) also came out this month. Whereas BSIMM is descriptive and OpenSAMM ([http://www.opensamm.org/](http://www.opensamm.org/)) is prescriptive, I'm curious to see how NIST tackles the secure development lifecycle.

</comment><comment><USER20> _Goal:_

The goal of this report is to present a list of specific approaches that have the potential to make a dramatic difference reducing vulnerabilities – by stopping them before they occur, by finding them before they are exploited or by reducing their impact.

 _Impetus:_

In response to the February 2016 Federal Cybersecurity Research and Development Strategic Plan, NIST was asked to identify ways to dramatically reduce software vulnerabilities. NIST worked with the software assurance community to identify five promising approaches. This report presents some background for each of the approaches along a summary statement of the maturity of the approach and the rationale for why it might make a dramatic difference. Further reading was provided for each approach. Hopefully other approaches will be identified in the future.

 _Current Cybersecurity Approaches:_

1\. Formal Methods - how do you check your program while writing code? (static analysis, model checkers, pragmas, model-based programming, and more recently contracts)

2\. System Level Security - how do you partition components in a system? (containers/hypervisors/virtualization with smart API calls)

3\. Additive Software Analysis - how do you analyze completed software to check for problems? (expression and exchange standards, better IDE plugins, advanced static analysis tools)

4\. Mature Development Standards - how can you modify and reuse tried and tested code? (find, understand, learn, and combine the most relevant frameworks)

5\. Moving Target Defense & Artificial Diversity - how can you minimize your program's attack surface? (compile-time diversification, cryptography, OS interface scrambling)

6\. On Metrics - how do we even define and measure a bug? ("There are far too many proposed measures to evaluate or even list here. We can state that, as alluded to above, metrics and measures should be firmly based on well- established science and have a rational foundation in metrology to have the greatest utility.")

 _Government Calls to Action:_

1\. Engage the research community (funding, contests and prizes, infrastructure)

2\. Education & training for the dev community (school curriculum, training programs)

3\. Policy, Legal, etc. (procurement standards, liability for failures, insurance, standards, code repositories)

\---

In my opinion, this is a good piece to skim through as you'll probably come across 1 or 2 practices/fields you hadn't heard of before. The recommendations are (intentionally) vague.

Functional security is already taken very seriously in many life and safety- critical industries. One potentially-troubling trend we are seeing in the automotive/aerospace & defense industry is the consolidation of traditionally disparate systems into heterogeneous, multicore processors in order to save on hardware costs. These systems are relatively complex, and have the potential to mix safety-critical environments (brakes, steering, acceleration ECUs running an RTOS) with application environments (infotainment system running Linux). Partitioning is big here. There are well-defined and accepted standards for the majority of industrial automation, transportation, medical, automotive, and aerospace & defense targeted systems.

Data security and privacy is not taken seriously by the majority of consumer corporations (and even some government agencies), and won't be until the legal repercussions of failing to protect client information are a serious threat to the companies that build and maintain these databases and systems.

<comment><USER6> To back up your point, this was a great blog post on the fatal consequences of failure in mission-critical car software: [http://embeddedgurus.com/state- space/2014/02/are-we-shooting...](http://embeddedgurus.com/state- space/2014/02/are-we-shooting-ourselves-in-the-foot-with-stack-overflow/)

After reading one may think: Why the fuck is there _any_ dynamic memory allocation (stack) going on there? Why isn't the car software implemented as one big pre-allocated hierarchical state machine? You wouldn't expect such execution dynamism in hard circuits or even FPGAs. But in any case I think non-deterministic systems (especially complex CPUs+programs) should be outlawed for such mission critical signal paths.

<comment><USER12> You got me as to why this is considered remotely acceptable. This being said, you sort-of need a stack for a context switch ( or possibly an ISR ) in many RTOS offerings.

This also being said, I've had engineers react rather violently to the very concept of FSMs. I can't say why that is.

</comment></comment></comment><comment><USER21> My affection for F# likes that ML and F# are explicitly mentioned in section 3.4.2 "Better Code".</comment>


A NetHack adventure - <AUTHOR> http://garote.bdmonkeys.net/nethack/index.html

<comment><USER2> As ascension posts go, this is pretty mundane; a fairly normal first ascension. Don't get me wrong--it's an enjoyable read--but it misses out on what makes Nethack a truly unique gaming experience. Nethack is fundamentally a hacker game. It's about exploiting the rules, exploiting the programming in order to solve unreasonably difficult problems in creative ways.

Just as an example, he mentions black pudding farming simply as a tactic he's copied from others. But the true spirit of Nethack was when another player and I _invented_ black pudding farming: putting together the tactics and equipment to make it viable in order to solve a specific problem. At a high level of play, one is constantly doing that sort of thing on the spot. Ascension posts by good players commonly introduce completely new tactics.

<comment><USER3> Well laa dee daa, aren't you cool. Speaking as the author of that essay, I can tell you, I did my fair share of hacking. I'll go up against you in a hacker dick-measuring contest any day. Back in 1992 I was rolling my own builds of Nethack on a university server, OVER DIALUP, to play it my own way. And before Nethack I was hex-editing my copy of Beneath Apple Manor. So nyeah.

<comment><USER2> "I can tell you, I did my fair share of hacking."

This is a fair response. I didn't feel your essay brought out that aspect of the game very well, but I have no reason to doubt it happened. IMO, the use and abuse of the game's rules are the most interesting bit of Nethack, and I wanted to highlight that.

"I'll go up against you in a hacker dick-measuring contest any day."

This is ridiculous and in no way appropriate. I meant nothing personal by my comment.

</comment></comment></comment><comment><AUTHOR> Skip to Part 1 if you already know what NetHack is about. I found the narrative very entertaining to read.

<comment><USER4> Dwarf Fortress is one of my all time favourite twists on the Roguelike genre. Something Awful had a very amusing succession game that's worth a read too; <http://df.magmawiki.com/index.php/Boatmurdered>

<comment><USER5> If you have the right kind of mind to get into Dwarf Fortress, it is one of the most interesting games I've ever played. You start to really care about these ASCII characters, especially when one devastated by his wife being killed by raiding goblins goes into mourning, snaps, and murders your blacksmith to turn his skin into gloves... and they're the best gloves your town has ever produced, worthy of a name out of legend.

</comment></comment></comment><comment><USER6> I admit I could never _quite_ get the allure of NetHack. I was a Moria/Angband man in college. It seemed those games were more in line with the original Rogue, than the "everything AND the kitchen sink" special case-ism that filled NetHack. I'd rather play a game I have a reasonable chance of winning is a short period of time, than have to spend _decades_ trying for my first ascension unless I cheat by reading the spoilers (that is, the source).

<comment><USER7> Thanks for introducing me to alternatives; I am of the same predisposition but had not heard of Moria/Angband.

</comment></comment><comment><USER8> You can watch other players in realtime here: telnet://nethack.alt.org . More resources: <http://alt.org/nethack/>

<comment><USER3> That's where I played the game mentioned in the writeup, in fact. I wonder if the replay is still buried in the server somewhere...</comment></comment>


Show HN: StartShout - Find places to get coverage for your startup news - <AUTHOR>

It seems like there isn't a week that goes by where there isn't a post on Hacker News asking "where can I find coverage for my startup?" So, I decided to make something to finally help address part of this problem.
I'm happy to announce the launch of StartShout (http://startshout.com) today - a new way to find sources of coverage for your news about events, startup launches, new features, or other announcements. StartShout is both a directory of content publishers and a pitch writing service.
I made StartShout because I want to help hackers get their projects noticed, but I think it could be useful for marketers or anyone else involved with technology as well. I'm actually not a hacker myself, but I am an avid reader of technology news. I don't know how to program and this is one of the first websites I've ever fully built form scratch. I originally designed StartShout with the intention of making it a targeted newswire service for blogs. After only a lukewarm reception I decided to pivot and it evolved to the current form.
If English is your second language or if you'd just rather have someone else write a pitch or press release for you, I'm happy to offer paid copywriting. If you don't like what I prepare for you, tell me and I'll fix it or give you a 100% refund. I think I'm offering this at a competitive rate since most major online writing services charge over $300 for writing a press release.
If anyone has any suggestions for further additions to StartShout I'd love to hear them. I'm sure that there are other areas out there that I've missed so feel free to email me. I'm going to focus on improving and expanding the directory in the coming days. I hope that StartShout can serve as a useful resource and at least help curtail some of the "How can I find coverage?" questions on HN :) <comment><USER2> As I understand, it's a (large) directory of places where people can announce their startups and/or other tech related news.

That's cool - I'm sure it'll serve the objectives you've outlined above.

Congrats!

</comment><comment><USER3> This is interesting, but what are your qualifications? Have you gotten coverage for any startups in the past?

</comment><comment><USER4> I like it...thanks! I was definitely looking for a resource like that

</comment><comment><USER2> Deleted: OP posted clickable at the same time as I did. :-)

</comment><comment><AUTHOR> clickable: <http://startshout.com>

<comment><USER5> There is no problem HN readers can not solve. This is awesome. Have you heard of pressdoc. Euro company that seems to naturally extend from a service like yours.

Best of luck with this - you will probably be hearing from me!</comment></comment>