Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions - <AUTHOR> https://www.cockroachlabs.com/blog/parallel-commits <comment><USER2> From what I understand, with this new commit protocol they managed to improve response time of a writing transaction by shifting some work of determining its final status to the readers. Am I understanding correctly that readers' performance will degrade by the same amount?

While this is an achievement which some scenarios will definitely benefit from, like bulk loading or updating secondary indexes as mentioned in the article, what about other scenarios where read performance is more important, like the ones where there are much more readers than writers? Shouldn't there be a configuration option which commit protocol to use?

<comment><USER3> > Am I understanding correctly that readers' performance will degrade by the > same amount?

Not quite. The "slow path" talked about is only applicable when the coordinator node is unavailable (presumably a rare event). If it's unavailable, there's nobody left to clean up the STAGING txn record, so the reader is tasked to do it itself.

In normal conditions however, once the coordinator node receives acknowledgement for the successful persisting of all write intents and the "txn STAGING" record, it can simply record "txn COMMITTED" in memory (and return to the client, send off async intent resolution procedures, etc.) Any subsequent read requests that observes left over intents (yet to be resolved) are pointed to the coordinator node, which can simply consult the "txn COMMITTED" record in memory. This is all safe because the commit marker is not simply stored on the coordinator node, it's a distributed condition and can be reconstructed by any observer even if the coordinator failed.

<comment><USER2> >Any subsequent read requests that observes left over intents (yet to be resolved) are pointed to the coordinator node, which can simply consult the "txn COMMITTED" record in memory.

Another roundtrip performed by reader rather than writer? That's what I'm talking about.

Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node.

<comment><USER3> > Though I understand it differently, as reader just waits until all writes > and "txn COMMITTED" record arrive at it's node.

Actually, welp, I believe this is closer to the truth in implementation. But consider how read performance compares before and after introduction of Parallel Commits. Before, when readers happen over extant 2PC "prepare" phase markers, they would still have to wait for txn resolution (on the coordinator node of the other txn, or on the node the intent was seen). They simply continue doing the same in Parallel Commits, there's no extra latency added to the read path (except when there is failure, but even then as soon as the earlier txn is recovered, future readers no longer get stuck).

<comment><USER2> >Before, when readers happen over extant 2PC "prepare" phase markers, they would still have to wait for txn resolution

Again, I understand it differently, but maybe I'm wrong. A reader upon encountering unresolved intent looks up corresponding transaction record. Before Parallel Commits, it's either marked COMMITTED or PENDING. If it's PENDING, reader just ignores it and skips its data, since they use MVCC. There's no waiting here.

Now with parallel commits, transaction record can also be marked STAGING, in which case the reader cannot determine if it's commited without additional work and/or waiting (the author doesn't go much into details).

<comment><USER4> > If it's PENDING, reader just ignores it and skips its data, since they use > MVCC. There's no waiting here.

I think this is where the confusion is coming from. You're correct that a read can simply ignore writes, even pending ones, at higher timestamps due to MVCC. This improves transaction concurrency.

However, if a read finds a provisional write (an intent) at a lower timestamp, it can't just ignore it. It needs to know whether to observe the write or not. So it looks up the write's transaction record and may have to wait. If the write transaction is not finalized then it needs to either wait on the transaction to finish or force the transaction's timestamp up above its read timestamp. This is true regardless of parallel commits or not.

What parallel commits gets us is a faster path to transaction commit, as irfansharif pointed out below. So the write can not only be committed faster with parallel commits, but it can also be resolved faster to get out of other reads' ways. In that way, it improves both the synchronous latency profile and the contention footprint of transactions, assuming no coordinator failures.

<comment><USER2> Thank you for clarification.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> Why is it so important for transactions to be interactive? I feel like this is not a feature that I use frequently as an application developer and it makes distributed transactions so much harder.

It seems like something that accidentally came along from SQL, and is imposing a large cost.

<comment><USER6> Interactive may be a little misleading here. Consider a common case in any java application at my current place of employment.

1\. A controller starts a transaction.

2\. Some code looks up a record in a database.

3\. Based on the results of that lookup we run 1 of two possible writes to a table in that database.

Knowing what kind of write you will do in the transaction requires knowing the result of the lookup which itself must be run in the transaction. Now take this small case and imagine it in the real world scenario where there might many such lookup specific write logic all wrapped in a single transaction and the space of possible combinations of statements you will want to execute in the transaction is large. None of this is interactive in the sense of someone working in a shell. But it is interactive in a way that makes the transaction useful in the real word.

<comment><USER5> Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result.

This allows you to implement the pattern you describe, which I agree is common, but in a dramatically simpler way.

Having the database, which is not really a single thing, but a swarm of computers spread across the globe separated by unreliable links who are trying to stay in consensus, pause their work to hear back from the client just seems ... well it seems miraculous that it can work at all.

The Calvin way is so much easier, it seems like there must be some very good reason that it's not what CockroachDB does. But I've never heard what that reason is.

<comment><USER7> Calvin has been an elegant protocol to work with in practice, and has pretty radically simplified FaunaDB's implementation of transactions compared to classic 2PC. Writes are committed in one global communication exchange, read isolation is pretty straightforward, and not requiring transaction recovery cuts out a significant amount of complexity which tends to be overlooked.

In talking with others, my best guess as to why we've seen relatively few implementations of it in the wild is that it is just less well understood compared to 2PC, so misconceptions propagate. The original paper focuses on how it works, rather than how to apply it in detail to generic transaction processing, which perhaps is a shame in hindsight considering that is where most of the confusion lies, IMHO.

For example, there is no reason stemming from Calvin that FaunaDB cannot provide full SQL-style session transactions. We chose not to implement them because they aren't a good fit for the core use-cases the system currently targets. Specifically, interactive transactions are too chatty for client- server interactions over the internet where link latency dominates an application's perceived speed: Instead, FaunaDB's interface encourages packing as much logic into as few requests as possible. (But I suppose that's a topic for another comment thread.)

<comment><USER8> Would that include transactions for which the reads can query the whole database as opposed to a predetermined set of rows?

<comment><USER9> The SQL support? Yes, even that.

</comment></comment></comment></comment></comment></comment><comment><USER10> I like the elegance and simplicity of two-phase commits. I didn't understand the criticism in the article; maybe it's something specific to CockroachDB. In my experience with two-phase commits, if the system crashes before a transaction is fully processed and committed, it should be fully reprocessed (from scratch) once the server restarts.

In one of my open source projects with RethinkDB (which doesn't natively support atomic transactions), I implemented a distributed 'parallel' two-phase commit mechanism by assigning each pending record a shard key (an integer derived from a hash of the record id); then worker servers would decide which subset/range of DB records to process/commit based on a hash of their own server id (which would tell them which range of shard keys/records they were responsible for). Only when a record had been fully processed, its status would be updated as committed by writing a 'settled' flag on that record.

Whenever a server failed and restarted, it would pick up processing from the last successful commit. If a server did not restart, the worker count would be updated and remaining workers would redistribute the partitions among themselves based on the shard keys of the records.

<comment><USER11> > Whenever a server failed and restarted, it would pick up processing from the > last successful commit. If a server did not restart, the worker count would > be updated and remaining workers would redistribute the partitions among > themselves based on the shard keys of the records.

You don't describe how you come to a global view of which is the "last successful commit". Do you mean that the co-ordinator recovers (such as from its own log) and uses that information, or do you mean you use local information? In the latter case, what you describe doesn't sound like it preserves transaction atomicity across node crashes.

<comment><USER10> In RethinkDB, a table can be split into shards and each shard has at most one master and multiple replicas. With one master to perform all writes for a shard, there is no need for a co-ordinator. If the master of a shard goes down, a replica can be promoted as the master. For writes, RethinkDB prioritizes consistency over availability so a server failure may carry some downtime in terms of writes but reads can be configured to be highly available (though they could potentially be out of date).

But there is definitely a downside that if a RethinkDB master server fails immediately after a write (before it propagated to a replica) and does not recover, then there can be some data loss (but only the most recent writes on that shard).

<comment><USER3> So if I'm reading this right, the master process can lose writes (if there's permanent failure). So taking your 2PC scheme above with the 'settled' flag, you can respond to the client that the txn has been committed once you've written the flag, but this commit marker can also just be entirely lost? (again, permanent failure)

If this 'settled' flag exists on each 'shard' instead of just the coordinator, any random subset of those too can be lost? I don't understand what's going on here, or what guarantees this 2PC implementation provides.

<comment><USER10> Actually, I may be mistaken about my previous commment. I'm not completely sure if this loss of recent data would happen as I've described. It depends on client implementation. For example, a client could wait for a write to propagate to at least 1 replica before telling the caller that the data was inserted successfully. This is an implementation detail I'm not sure about.

Also the settled flag exists on each record, not each shard. A shard is typically made up of multiple unsettled records. Each worker is assigned to a shard using a hash function so it's deterministic and the worker only processes unsettled transactions from their own shard.

Also I said something else misleading in one of my previous comments. In my case, the shard key of each record (which determines which shard a record belongs to) was not based on its own record ID but on the account ID of the user who owns that record. So effectively the sharding was happening based on user accounts and it was designed so that the records created by an account could be processed independently of records created by a different account.

</comment></comment></comment></comment></comment><comment><USER12> Given how critical preventing future intent writes is to the protocol to ensure safety during recovery, it'd be nice to have more detail on how that works. Calling it an in memory data structure doesn't exactly inspire confidence.

<comment><USER3> Agreed, we should be talking about the "timestamp cache" in more detail generally. While I'm here, looking at [1] helped me confirm how everything is kosher despite being in-memory. The timestamp cache pessimistically maintains a "low water mark", this always ratchets up monotonically and represents the earliest access timestamp of any key in the range. Writes happening at timestamps lower than this water mark are not let through, and bumping this watermark past the point of the observed write intent's timestamp is how slow inflight write intents are aborted by recovering read requests. On server restart, the timestamp cache is initialized with a low water mark of the current system time + maximum clock offset, so "future" write intents (or more accurately: write intents sent in the past but previously stuck in transit) are simply rejected.

[1]: [https://github.com/cockroachdb/cockroach/blob/master/pkg/sto...](https://github.com/cockroachdb/cockroach/blob/master/pkg/storage/tscache/cache.go)

</comment></comment><comment><USER13> What prevents a transaction from being prematurely aborted while there are some intents in flight?

From what I understand, transaction A can still be replicating intents and transaction B, not knowing that coordinator for transaction A is still at work, start a recovery process. This makes transaction A abort which could be prevented by waiting for intents of A to be successfully replicated.

</comment><comment><USER14> Wow, this is impressive.

</comment><comment><USER15> What I'll say might come off naive but I accept this at an attempt for a reductionist's viewpoint.

A singular update that side affects *N (times N) where N is greater than 1 will always lead to either a race condition or latency.</comment>


Ask HN: Our CR (browsing/installs) is low. What's wrong? - <AUTHOR> Hey guys, What do you think about apps landing page? <comment><USER2> I like visualizing task importance like this. But there are several reasons why I will not install this app:

1\. The name is absolutely terrible. "Task Manager To Round" is nonsense to me, and I'm a native English speaker. Bad names imply poor design and lack of support.

2\. The visualization is too simplistic. When I think about laying my tasks out with visual feedback, I expect to be able to represent the task with a photo or some kind of attachment. Your app might be capable of that, but if it is, there's nothing on your pages that convey that.

3\. The screenshot of the widget on your Play Store listing implies that the task bubbles are all the same size on the widget view. That completely defeats the purpose I'd have for the widget, and I consider widgets crucial for task management, so that's a turn-off.

Again, these are just my impressions from your store listings. I wish you luck.

<comment><AUTHOR> Thank you so much! It's very helpful for me.

<comment><USER3> I have to agree about the name. Here are some other ideas: CircleDo Taskround Ringdo Circler BubbleDo

No idea if any of them are taken.

<comment><AUTHOR> Thank you for suggestion!

</comment></comment></comment></comment><comment><USER4> Listings in the Apple and Google app stores are slices of shelf space behind a counter not landing pages. Your company has no control over format, little control over content, and most importantly zero control over user engagement because here "user" means Apple/Google's user not yours. While your company pitches potential customers, Apple/Google are advertising competitors right there on their [not your] page.

Task management is inherently complex, that's why management is relevant. This means that users have to invest time in learning a technique. The app stores preclude dynamic presentation of your apps techniques. There's no way I can "get it" from a couple of static screen shots because the nexus between circles and task management nirvana is non-obvious.

I think many problems stem from the AppStore business model. It makes talking directly to customers next to impossible and the approval processes and upgrade policies create friction for quick iteration based on customer feedback.

Good luck.

<comment><AUTHOR> Yes, you are right! Thank you for detailed explanation. This is very valuable.

</comment></comment><comment><AUTHOR> Links:

[https://itunes.apple.com/app/id1044638681](https://itunes.apple.com/app/id1044638681)

[https://play.google.com/store/apps/details?id=com.toround.an...](https://play.google.com/store/apps/details?id=com.toround.android)</comment>


Ask HN: Service to Collect Money You're Owed - <AUTHOR> Anyone know of a service that provides drip pester-the-shit-outta-people messages until they pay you back?
Just enter the balance owed and contact information. Maybe keeps track of partial payments.
It can use constantly changing numbers and emails, so it can never be blocked.
You can select how aggressive or sweet the messages can be, like choosing the voice of a navigational system. Maybe have it play good cop / bad cop.
And in time you could use machine learning to perfectly time the messages and create the content.
I need this.
And I need to be paid back by many of my friends, who all promised to repay swiftly and most of whom have not paid a dime in a year.
Not to mention my hoodies, coats, boxers, psycho bunny, and socks hahaha.
PS: I refuse to lend money ever again to anyone for any reason. Because too many bad actors. I am highly jaded. Of course, I have never lent money that I couldn't live without. But it stings a lot and I need a solution to this problem. Might build it myself unless someone has one already.
PSS: I was very tempted to tag those that owe me on my Facebook post about this. But alas, I am not savage enough. I am looking for something that doesn't mess with the public reputations of the promise-breakers and maintains a high ground.
PSSS:  Maybe my buddies at Pardot / Salesloft could build a side tool for this lol. Or launch a business around it with me.
*Originally posted at:  http://notonlyluck.com/2019/02/23/startup-idea-service-to-collect-money-youre-owed " <comment><USER2> > Anyone know of a service that provides drip pester-the-shit-outta-people > messages until they pay you back?

There are federal and state laws about debt collection to protect debtors from aggressive debt collectors. If you harass someone for a debt that you can't prove they owe, you could get into deep trouble. If you continue to contact somebody after they tell you to stop, you can get into deep trouble.

> It can use constantly changing numbers and emails, so it can never be > blocked.

This could be a violation of anti-spam laws.

> You can select how aggressive or sweet the messages can be.

Aggressive? If you threaten someone with violence, you're going to get a visit from the cops. If you threaten someone with reprisals like ruining their reputation on social media if they don't pay up, that's extortion. You could go to jail.

<comment><AUTHOR> Thanks for all the info.

No no none of that aggression. I am a gentle father of 6 lol. I just mean like a more firm wording like "pay by XYZ date" versus a "hope ya pay soon."

And yeah, everyone should keep track of their financial transactions and purposes for it. But those get proven or unproven in court.

And drip emails happen all the time. I don't see that this is different.

</comment></comment><comment><USER3> > _I refuse to lend money ever again to anyone for any reason._

Sounds like you solved the problem yourself...

<comment><AUTHOR> It is a suboptimal outcome. I'd prefer to have a bot that knows it can get back X% for me over time. Then I can budget for my charity to friends and understand / maximize the results.

And it's a huge time sink to do personal AR, and a tedious ugly task.

Being scrooge is not optimal.

</comment></comment><comment><USER4> There's probably a market for this among freelancers.</comment>


Exciting times in Rails land (review of some 2.1 features) - <AUTHOR> http://www.caboo.se/articles/2008/3/30/exciting-times-in-rails-land <comment><USER2> This update will have more of an effect of my day-to-day life than the 1.2 -> 2.0 update. It is good to see that the Rails team is confronting serious issues (like the Migration numbering) quickly and not letting them fester.</comment>


Apple Said to Be in Talks to Buy Beats for $3.2 Billion - <AUTHOR> http://dealbook.nytimes.com/2014/05/08/apple-said-to-be-in-talks-to-buy-beats-for-3-2-billion/?_php=true&_type=blogs&partner=rss&emc=rss&smid=tw-nytimes&_r=0

<comment><USER2> Discussion: [https://news.ycombinator.com/item?id=7717961](https://news.ycombinator.com/item?id=7717961)

Other submissions, some have up-votes, some have comments:

[https://news.ycombinator.com/item?id=7718020](https://news.ycombinator.com/item?id=7718020)

[https://news.ycombinator.com/item?id=7717995](https://news.ycombinator.com/item?id=7717995)

[https://news.ycombinator.com/item?id=7717980](https://news.ycombinator.com/item?id=7717980)

[https://news.ycombinator.com/item?id=7718106](https://news.ycombinator.com/item?id=7718106)

[https://news.ycombinator.com/item?id=7718133](https://news.ycombinator.com/item?id=7718133)

[https://news.ycombinator.com/item?id=7718137](https://news.ycombinator.com/item?id=7718137)

[https://news.ycombinator.com/item?id=7718138](https://news.ycombinator.com/item?id=7718138)

[https://news.ycombinator.com/item?id=7718146](https://news.ycombinator.com/item?id=7718146)</comment>


Ask HN: What's Your DIY Home Security System - <AUTHOR> I did a HN search and of the 2 pages most are several years old [1]. What's your recommended DIY security system? I was thinking deadbolts but then those cheap battery operated door window alarms, one camera (wired or wireless) above front door, two cameras in back, and motion detector lights (battery or solar). I don't trust Smart Home setups that can be used to disable or open doors. I want to be able to monitor and have loud alarms. I was even thinking no keypad in the house to control everything. Maybe a wired siren in the attic thats really loud so if the alarm is tripped the neighborhood hears it (maybe after 10 second delay) so any possible intruder would know EVERYONE is noticing my house. What would/do you do?<p>[1] https://news.ycombinator.com/item?id=5174685 <comment><USER2> The answers to this post would be the Comstock Lode for a tech cat burglar?

What's your threat model? Ordinary random petty burglaries, by people who will run if they find people in the house? Less-predictable drug addicts? Home invaders who will be aggressive about getting the safe combination?

You can ask your local police department what's typical in your neighborhood. They might also have an advisories email list you can get on, to hear about current patterns or unusual events (e.g., a rash of breakins via unlatched ground-floor windows). The first priority measures might not be electronic at all.

Also, I'd be hesitant to install a neighborhood-shaking siren. Your system is probably going to have false alarms, and you don't want to ruin the sleep of your neighbor, the hospital emergency department worker, who you really don't want to be fatigued next shift.

</comment><comment><USER3> It really depends on where you live. I live in a rural area on a gravel road with only one way in and out.

I have two dogs that sleep on the front porch. Both of my nearest neighbors have a dog too. We all listen for them barking and it's easy to tell the difference between them barking at a stranger and howling out to neighbor dogs and coyotes.

If I lived in an urban area where stuff getting stolen was a problem I would get good deadbolt locks for the doors and keep the windows locked. I'd also get a camera for the front door and maybe a couple others if needed to surveil and record points of access, and that's about it.

I'd probably make them myself out of Raspberry Pi computers and their camera and have fun twiddling with them.

I'm not a big fan of loud alarms. There were a few times when I lived in LA that I wanted to smack the snot of owners of cars with those installed when they went off.

I haven't heard one of those since I left there.

</comment><comment><USER4> Get a dog. I saw a very ugly half-Pitbull, half-Husky the other day, I think if most burglars saw that they would run into the next time zone.</comment>


Thoughts on Metro - <AUTHOR> http://brooksreview.net/2011/09/metro/ <comment><USER2> I found it hard to get past this statement: "The last risk was Windows Vista and that was a risk done by necessity so that the OS could keep up with OS X's speed."

This opinion seems like an anachronism. Go to the Vista wiki page and every bit of referenced commentary is with respect to Windows XP, and how users still wanted to buy XP. I didn't see any linked articles mention Apple as a competitive threat. Wasn't Apple crawling back out of the grave when Vista planning started in 2001? Ipod may have been a big deal when Vista was released in 2006, but the iPhone was still a rumor, and I believe the next OS X was delayed because developers had been pulled to work on iOS.

Unless you transport today's competitive environment back 5 years, I just don't see how this statement jives with reality.

</comment><comment><USER3> I hope they don't force Metro to be used on non-tablets. The screens I've seen so far show a confusing jumble of colors and different sized and entirely unrelated boxes squished together. I've been using windows for a long time now, maybe too long even, because I just want to keep the traditional desktop layout.

It might be a great tablet interface, but I think they're severely overestimating how big the tablet market will be in 2013.

</comment><comment><USER4> _> There will be no "true" multi-tasking_

Really? What about the ability to snap two apps side by side? That doesn't count as multitasking?</comment>


'Karma': A hack used by the UAE to break into iPhones of foes - <AUTHOR> https://www.reuters.com/investigates/special-report/usa-spying-karma/ <comment><USER3> Whether or not this hack was developed with the help of Apple (a "backdoor") or by a third-party exploit, this is exactly what a "golden key" looks like after it gets in the wild.

An espionage tool developed by a major world power proliferates to totalitarian regimes, aided and operated by ex-NSA agents on the payroll, to compromise human rights activists and the political opposition.

If ever there was proof that our devices need to be striving — constantly striving — for absolute security, and can never allow any "trusted party" an authentication or encryption bypass, this article is it.

An exploit like this is incalculably valuable to intelligence agencies. That the exploit would proliferate is undeniable. And the ends to which it would be (has been) used is atrocious.

Probably the only thing different about how intelligence agencies exploited this, and how they would exploit a golden key, is that with the golden key they would be sweeping up every photo on every device, and not just some photos on some devices.

 _"It was like, 'We have this great new exploit that we just bought. Get us a huge list of targets that have iPhones now,'" she said. "It was like Christmas."_

<comment><USER4> I disagree with the conclusion of absolute security — it won't happen, and only encourages subversion by people who both need and have a right to access the content.

Instead of pontificating, the tech industry should innovate.

There's no reason that hashchains can't be used to timelock the key, and the enclave export it in response to a signed request. Then we can at least force the compromises through the legal system _and_ require effort to reverse the hashchain. That kind of court authorized targeted access removes the incentive (and justification) for other actors to more deeply compromise the system. In turn, this let's us provide more security, in practice.

What's not going to sell, and what the tech industry needs to get over is "lulz, it'll impossible to intercept military or terrorist information because I need absolute privacy for my saucy emails". I think it's been empirically demonstrated that won't happen.

Be part of the solution.

<comment><USER5> Wouldn't the police have a "right" to know if a person has any weapons? Detaining everyone and performing a full cavity search for any and all infractions is just the police exercising their "right" to such information. Will you be the first to bend over and spread for the cops "right" to peace of mind?

"That it is better 100 guilty Persons should escape than that one innocent Person should suffer, is a Maxim that has been long and generally approved."

<comment><USER6> > "That it is better 100 guilty Persons should escape than that one innocent > Person should suffer, is a Maxim that has been long and generally approved."

I wonder if that maxim is still generally approved. It seems like some authoritarians would prefer that 100 innocents would suffer than one guilty person should escape.

I suppose it depends how you define "innocent" and "suffer". Under modern law, everyone is guilty of something. And while we might not require suffering in prison, a little suffering of expensive legal fees, invasions of privacy of your digital data/at the border/in the airport, or searches and seizures of property by police in your car are commonplace.

<comment><USER7> I think, even in places that believed the maxim, 9/11 changed the calculation. Now the question is: How many innocent people should be jailed in order that 3000 innocent people not be killed?

Mind you, I'm not saying it's _right_. I'm just saying that this is how the authorities are thinking.

<comment><USER8> Or how many billions of hour-lives should we steal a minute or hour at a time from to save X lives?

</comment></comment></comment></comment></comment></comment><comment><USER9> If the US government gives itself the right to install backdoors / exploit vulnerable software (as opposed to notifying companies about vulnerabilities) then I feel pretty uncomfortable about ex-government hackers just becoming freelance mercenaries using knowledge they may have gleaned from those ops once they move onto their next gig.

I can't think of a great solution to this problem.

<comment><USER10> You can make it illegal for ex-NSA employees to use their knowledge of exploits learned while on the NSA payroll. It may well already be the case for all I know.

<comment><USER11> Perfect. Then, just hope people follow rules.

<comment><USER12> Sometimes you have to disincentivize behavior with prison time and things like that and then hope people don't do it. Trying to prevent some crimes ahead of time is a recipe for dystopia.

<comment><USER13> In this case, "trying to prevent some crimes [of government employees leaking the golden key]" possibly means "don't make a golden key that lets governments freely hack everyone", which is generally being regarded in this thread as the non-dystopian result.

</comment></comment></comment></comment></comment><comment><USER14> I realize it's a really sexy headline, but I'd like for there to be more than 0 proof that this is a real thing. Especially if they claim a vulnerability that's exploitable by only sending a text.

<comment><USER15> This article doesn't cite sources, but the other one cites Lori Stroud, a former developer of the application.

[https://www.reuters.com/investigates/special-report/usa- spyi...](https://www.reuters.com/investigates/special-report/usa-spying- raven/)

<comment><USER16> Lauri did not develop the program. She was an solely an intelligence analyst.

</comment></comment></comment><comment><USER17> I think that the idea of out-of-the-box privacy/security against even a semi- competent adversary on any computer (especially a mobile device) is completely fictitious, and these hack stories play an important role in helping people realize that.

Consider the thousands of people around the world that are involved in making phones in design, hardware, software, manufacturing, signal providers, platform providers, app writers to name a few. Any of them could be malicious actors or accidentally introduce exploitable bugs. The idea that such a complex stack can shield you from very smart and resourceful people that are actively trying to peek though is not reasonable. Everyone, especially people that are "annoying" to powerful entities (corporate or government), should _assume_ that everything they do with their mobile phone is accessible to the people they hope it isn't.

<comment><USER18> That a good, cautious stance. That said, thus may not be a good example of it.

We don't know the imessage bug, but a big one was patched in ios 9.3.3, released July 18, 2016. Meanwhile, the article says this exploit got a lot of people in 2016/2017.

So, presumably simply updating software would have protected a lot of the victims in this case.

The higher up in adversary skill level you go, the less this works. But up to a reasonably high level simply having up to date software thwarts most adversaries, no? And conversely, if you have very out of date software, even incompetent adversaries can break in.

</comment></comment><comment><USER19> >A team of former U.S. government intelligence operatives working for the United Arab Emirates

no non-competes? So, when Snowden tells to public about mere existence of NSA hacks - it is a crime, yet when an intelligence operative brings his NSA and the likes sourced detailed technical knowledge to a foreign government - <USER2> is kosher.

<comment> java-man welcome to reality

<comment><USER20> Now imagine what's going to be possible in Australia with their compulsory surveillance laws...

Though, I wouldn't be super surprised if they banned people they forced to implement exploits from leaving country =X

<comment> java-man I take a more pessimistic view. We should think that all this is already being done, just not acknowledged.

</comment></comment></comment></comment><comment><USER21> No one read the other Reuters article on this - it may have been criminal for NSA employees to participate in this, at the very least it was highly discouraged. If anything this is a good argument to pay IC employees on the GS payscale better so they're less likely to take jobs with other countries.

<comment><USER16> Large govt. contractors and sub companies run this and all the other programs like this. There is tacit approval for these ops. I'd even imagine there's a bit of intel being fed back to them from the new _employees_.

<comment><USER21> "The FBI is now investigating whether Raven's American staff leaked classified U.S. surveillance techniques and if they illegally targeted American computer networks" [1]

It's still illegal to use US classified information for a program like this and it's still illegal to target American citizens or networks.

[1] [https://www.reuters.com/investigates/special-report/usa- spyi...](https://www.reuters.com/investigates/special-report/usa-spying- raven/)

<comment><USER16> I didn't say it wasn't, I simply said all of it could be encompassed by the specific clause of "TTP's" that is common in anyones read on-off of information. The difficulty becomes in proving what was _classified_ and what wasn't when it comes to techniques and procedures. If you were to use a liberal sense of the term, many of your "Security Professionals" deploying their skills around the US right now would be in violation. It's why there isn't a simple non-compete / non-authorization of future work within that field, it's just not going to stand up in court.

It's all as clear as mud and in this instance the government was more than aware of their former employees working there, many returned and went back to their prior careers.. think about that one for a second.

</comment></comment></comment></comment><comment><USER22> To expect citizens to revolt against their _own_ country and force their "secret services" to not endanger public security any more by using and buying such 0days is one thing - that's unrealistic.

But seriously, I wonder why _other_ governments and their citizens are not demanding drastic actions, like trade suspensions, expulsion of diplomats or other sanctions, when other countries get caught in such ways of spying or otherwise just screwing all over human rights. This one would be a perfect example to take a stand on - UAE is far smaller in oil trading and political importance than e.g. Saudi-Arabia.

Or why there seems to be next to zero public funding for providing open source, auditable hardware and software that could _prevent_ such spying in the first place? The European Union could easily fund the development of a truly FOSS Android-based phone, down to the processors. Instead everyone seems to rely on Chinese or American products, which are both subject to non- European influence (in the US via NSLs, in China due to the massive influence of the Party on any major company).

</comment><comment><USER23> IA link[0] for people who get "Page Not Found", trying to visit from Europe.

[https://web.archive.org/web/20190130135641/https://www.reute...](https://web.archive.org/web/20190130135641/https://www.reuters.com/investigates/special- report/usa-spying-karma/)

</comment><comment><USER24> _> Three former operatives said they understood Karma to rely, at least in part, on a flaw in Apple's messaging system, iMessage. They said the flaw allowed for the implantation of malware on the phone through iMessage, even if the phone's owner didn't use the iMessage program, enabling the hackers to establish a connection with the device. To initiate the compromise, Karma needed only to send the target a text message — the hack then required no action on the part of the recipient._

Has anyone here heard about or is familiar with this malware?

<comment><USER25> I wonder if this is [https://www.theguardian.com/technology/2016/jul/22/stagefrig...](https://www.theguardian.com/technology/2016/jul/22/stagefright- flaw-ios-iphone-imessage-apple) again, the article mentions 2016.

<comment><USER26> Seems like a likely candidate.

</comment></comment></comment><comment><USER27> I can't help but giggle at the thought of Emirati hackers. For whatever reason my mind can't wrap around the fact that an extremely religious people can also be at the high end of tech (at the very least high enough to figure out 0days and such).

Does anyone have any info on since when this has actually been like this? I'd like to look up how their CS education works and that kind of stuff.

<comment><USER28> There are plenty of religious people in Hacker News. I have no idea why you would think being religious or spiritual would prevent anyone from developing technology.

My religious views do not stem from a lack of intelligence or education.

<comment><USER27> Not even talking about intelligence here. Sorry. That's an endless time waster I'm not touching.

As mentioned, for whatever reason, I'm having a hard time picturing how people who deem apostasy punishable by death can also manage, research, and exploit modern equipment, and am looking for some indication as to when exactly did they start getting good at it.

</comment></comment></comment><comment><USER29> Overlooked: the apathy required to become techno-mercenaries started with the agents convincing themselves that spying on nationals was different than foreign nationals while working in the NSA. To resist this apathy cyber intelligence agents working for any government/corporation should deploy a moral compass that assumes they are working for the UAE.

This also begs for international conventions. New international conventions would provide a psychological back-stop against the infosec industry's unchecked nationalism. When an agent asks themselves "is what I am doing okay" international convention and law would give them an alternative to compare with other than the militarist default of "yes".

</comment><comment><USER30> Sorry for slight meta, but does anyone know why this 323 pt 8 hour old story is near the top of the front page, while the 6 hour old 639pt story on apple blocking FB's certs is on the 2nd page?

</comment><comment><USER31> > and former American intelligence operatives working as contractors for the > UAE's intelligence services

At what point does this become considered treason?

<comment><USER32> _Treason against the United States, shall consist only in levying war against them, or in adhering to their enemies, giving them aid and comfort. No person shall be convicted of treason unless on the testimony of two witnesses to the same overt act, or on confession in open court._

</comment></comment><comment><USER33> You have an agency like the NSA with scores of genius mathematicians and hackers and a black budget. There's nothing beyond their means, spying wise.

I suspect that one day our internal thoughts and feelings will be under constant mass surveillance, Minority Report style, but it won't look like sci- fi when it happens.

</comment><comment><USER34> Am I the only one who feels like every time we get news of a government compromising an iPhone through some mystical exploit, the technology around it seems very fanciful?

<comment><USER14> Like an exploit where all you need to do is enter the target's phone number to compromise their phone?

<comment><USER3> TFA says they need to send the target a text message.

The exploit must be something like a buffer overflow in iMessage. Which we know bugs like this have been fixed. Remember the text of death which could crash any iPhone from a couple years ago?

<comment><USER35> Are you thinking of the "Stagefright" bug that did RCE via SMS on Android devices? Or maybe the Chinese censorship code that crashed iOS when people sent the Taiwanese flag emoji? 🇹🇼

<comment><USER36> Don't forget the Stagefright-like bug in iOS where a malformed TIFF file could lead to remote code execution!

</comment></comment></comment></comment></comment><comment><USER37> I read this as: "class of netizens who use iphone, targeted by scandal or bad actor" poignantly titled 'Karma.'

</comment><comment><USER38> While I find this absolutely disgusting and horrifying, at the same time I hope this becomes extremely widespread and rapidly. This needs to happen to Americans. Only then will we, the collective we, wake up and actually do something and maybe start to take privacy and security seriously.

I am rapidly becoming anti tech, as I think I can clearly see where this is all going. That's hard for me to say, as my whole life has been tech focused. I'm 47 and started coding when I was 10. My whole life centers around it, and always has.

Hitler, Stalin and Mao would have absolutely loved to be alive today and have these types of tools. Maybe we need another 100M deaths to see what this kind of information and power leads to. We are recording everything we do digitally, all to be easily analyzed by whomever comes to power at some future point of time, where the rules might be different. Most of what is recorded about us we don't even know. It will also be easier to find all of the relatives, so they can be killed off too. They like to make examples and ensure no one steps out of line. They don't just kill you, they kill 1-2 generations of your family.

This data won't go away. Ever. They will know who likes what, who supports what, etc. Just a keyword search away from getting a list of names and addresses. We think we are so clever. We are building our future jail. For the first time in history, we have the ability to track every single minute detail about a persons life from birth til death, in extreme, high resolution which grows by the day. I don't just know you went from point A to point B. I know the exact route you took, how long it took to complete each segment, how long you stopped at each place along the way, what those places were, etc. That's just gps data.

I saw the 60 Minutes piece on PlanetLabs recent launch of 300 satellites. They're taking pics of the globe in very hi resolution, constantly. Better than some of our spy sats. Oh, and anyone can access that data. It's free! They showed how they were able to go back in time to when the compound that Osama Bin Laden killed in was built. They were then able to create a very accurate model of the compound which led to the raid that killed him by going so far back in time to when they started building the thing. Obviously we think that's a good thing because it led to a mass murderers death, but think about that technology.... recording everything 24/7, globally, going back years in time to reconstruct something that happened in the past... [https://www.cbsnews.com/news/private-company-launches- larges...](https://www.cbsnews.com/news/private-company-launches-largest- fleet-of-satellites-in-human-history-to-photograph-earth-60-minutes/)

</comment><comment><USER39> What about Dragonfly? Is it also assisted by the government? It is censored but it sounds like a rat trap.

</comment><comment><USER40> TL;DR: the US attitude ...

      *It's fine to spy on human rights activists with all the powers of government as long as they're not American*

... really gets to the heart of how the US treats the rest of the world. The US is the biggest terror threat in the world today. Its pains are self inflicted and it's enemies created by their very own foreign policy.

<comment><USER41> >> ... really gets to the heart of how the US treats the rest of the world.

Really gets to the heart of how the ruling 0.0001% of the US treats the rest of the world. Fixed that for you. Some of us just live here.

<comment><USER40> yeah I actually thought it would be clear as crystal that when I speak of the crimes committed here we're not talking about those that pay their taxes, and struggle to get access to health-care ... Most are themselves immigrants so how on earth would anyone suggest it's the ordinary citizen here that's guilty[0]. I do acknowledge that my comment was "un-American" (not that I give a hoot).

[0] [https://en.wikipedia.org/wiki/The_Untold_History_of_the_Unit...](https://en.wikipedia.org/wiki/The_Untold_History_of_the_United_States#Critical_reception)

</comment></comment></comment><comment><USER29> Taking the sources of the article on their word...

They said the tools use faded in late 2017 due to apple patches and that compromise required only sending a text message. Examining CVE's up until late 2017 may give more of an idea of how this tool worked. Judging from a cursory review, there are many remote code exploits so it would be hard to narrow down. But this is what I chose to look at when considering CVE's between Jun 2017 and Dec 2017 that could effect iMessage. Many of these are classified as Denial of Service bugs but often those can be extended to code execution with extensive research.

IOKit [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-13847)

IOMobileFrameBuffer [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-13879)

CFString [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-13821)

CoreText [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-13825)

CoreText [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-13849)

Fonts? [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-13828)

ImageIO [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-13814)

Messages [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-7118)

SQLite [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-10989)

SQLite [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-7128)

SQLite [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-7129)

SQLite [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-7130)

SQLite [https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-20...](https://www.cvedetails.com/cve- details.php?t=1&cve_id=CVE-2017-7127)

Kernel: too many to count

These were compiled by reviewing the apple security mailing list [https://lists.apple.com/archives/security- announce/2017](https://lists.apple.com/archives/security-announce/2017)</comment>


Barilla pasta boss's anti-gay comments spark boycott call - <AUTHOR> http://www.independent.co.uk/news/world/europe/i-would-never-use-homosexual-couples-in-my-adverts-barilla-pasta-bosss-anti-gay-comments-spark-8841902.html <comment><USER2> That's his personal opinion. I have no idea why people are getting so upset about it.

<comment><USER3> It says that his opinion is not just a personal opinion but also extends to how the company does business. It quotes him as saying "I would never do an advert with a homosexual family…if the gays don't like it they can go an eat another brand."

Thus, those calling for a boycott are doing exactly what Barilla suggests they do.

</comment></comment><comment><USER4> (2013)</comment>


AquaSnap: Tiling Window Manager for Windows - <AUTHOR> http://www.nurgo-software.com/products/aquasnap <comment><USER2> If anyone has used this can you tell us what you thought of it ?</comment>


How Psychedelic Drugs Can Help Patients Face Death - <AUTHOR> http://www.nytimes.com/2012/04/22/magazine/how-psychedelic-drugs-can-help-patients-face-death.html?_r=2&hp=&adxnnl=1&pagewanted=all&adxnnlx=1335032670-ZqQUHmuDrFhKhrpwPFfupw

<comment><USER2> Psychedelic drugs can help nearly everybody, not just terminal patients.</comment>


Show HN: LosslessCut – Cross-platform GUI tool for fast, lossless video cutting - <AUTHOR> https://github.com/mifi/lossless-cut <comment><USER2> I guessed from the title it was going to be using ffmpeg, because I myself have used ffmpeg's `-ss`, `-t`/`-to`, and `-codec copy` numerous times for this exact purpose.

Then I saw that this bundles its own piece of Chromium and ffmpeg and is ~70 MB in size. Something is seriously wrong with today's app development ecosystem if it takes 70 MB, even when I already have both Chromium and ffmpeg on my system.

EDIT: On the other end is QtAV, a cross-platform multimedia player and SDK which took me only 1 MB of network download. It uses Qt and ffmpeg, both of which I already have on my system, so it doesn't have to redundantly bundle anything.

On OSes without Qt and ffmpeg in their package repos (OS X and Windows), the players are ~ 20 MBs in size. So even if one argues "cost of cross-platform compatibility", it still doesn't make sense to bundle the entirety of a web browser for something as simple as this.

<comment><USER3> I'm legitimately pissed off by this. Electron apps are always annoying to see but this one is genuinely offensive. I hate the culture and ecosystem Electron has allowed to flourish. This project is an insult to computer science.

<comment><USER4> 70MB is less then U$0.01

And for that, you get one hell of a text/layout engine, hardware accelerated 2D/3D graphics, integrated IDE, one the fastest interpreted runtimes around, which happens to be the most popular programing language…

All Open Source, cross platform.

Yeah, insulting indeed. Who needs those damn Electron apps.

<comment><USER3> >you get one hell of a text/layout engine, hardware accelerated 2D/3D graphics, integrated IDE, one the fastest interpreted runtimes around, which happens to be the most popular programing language…

All so you can provide a graphical interface for two ffmpeg flags.

<comment><USER5> > All so you can provide a graphical interface for two ffmpeg flags.

"everything old becomes new again"

reminds me also of the hot new trend of chat based interfaces. (hmmm, pretty sure that was our default computer interface back in 60's/70's/80's)

</comment></comment></comment></comment></comment><comment><USER6> Check out AviDemux, [http://fixounet.free.fr/avidemux/](http://fixounet.free.fr/avidemux/) it offers a simple UI and also lossless cutting.

<comment><USER7> Amusingly, AviDemux, which has a LOT more ability than just cutting... is about 71 meg.

</comment></comment><comment><USER8> So you're playing with version X of ffmpeg and cutting with version Y. It seems that X+Y equals disaster.

</comment><comment><USER9> I think including everything is super cool because I had the app installed and running in under 10 seconds. What wasn't cool was that I asked to cut a short 15 s video to the first second, and there was no feedback but the spinning gear for 5 minutes.

<comment><AUTHOR> That's a bug then: [https://github.com/mifi/lossless- cut/issues](https://github.com/mifi/lossless-cut/issues) Edit: could you check if ffmpeg was still running while it was spinning? Maybe we need to kill it if it takes too long.

</comment></comment><comment><USER10> How do you deal with the fact that some video tracks do not provide the needed cross-frame data or the times you're cutting are at unfortunate points which would require a re-encoding because a quick byte copy of the existing stream doesn't work or at the very least will complain later about missing things like color info (although it's played back correctly by mpv)?

<comment><AUTHOR> I don't know what you mean by cross-frame data. The program is using ffmpeg's -ss, -t, and vcodec copy functionality, and I'm not sure how it handles these issues. I know that if you cut between keyframes (which is very likely to happen), then the data before the next keyframe will be lost, so it is not an exact cutting mechanism.

<comment><USER10> That's another issue, but it can also lose other data which will cause warnings but play most likely with correct color reproduction.

I don't have such a source file handy to reproduce, but if you try enough ISO MP4 containers with H246, you will hit one where this is the case.

Is `-t` different from `-to` which I've been using?

<comment><USER11> libavcodec tries to recover from missing frames (you need this for TV broadcasts) but it's actually not very good at it for H.264. You'd almost always see major artifacts.

Any idea what the exact warning text is?</comment></comment></comment></comment>