Help me understand this ECB attack - <AUTHOR> It's pretty old i guess now (2012) but i'd still like to able to understand the adaptive chosen plaintext attack on AES-ECB. Specifically, how it's explained in this video from Thomas Ptacek's talk on crypto: https://vimeo.com/41116595
I can understand how we identify where our block boundaries are, but repeatedly injecting known plaintext (32 or 64 A's for example) and then appending another known string, but this time with  a different value (another letter in the alphabet). From this, build up a dictionary of ciphertext that we know the plaintext for.
My problem is i cant understand how the attack gets implemented practically, for example in the video it's suggested to include a pipe character | (thats @ 45 minutes in or so) but i cant see how this would lead to a decryption? <comment><USER2> In ECB mode, a block cipher (like AES) is used to encipher/decipher each block (of say 128 bits) with the cipher key independently of all other blocks. This means that two equal ciphertext blocks, say block 23 and block 61, have equal corresponding plaintext blocks so we can surmise that plaintext blocks 23 and 61 are identical. This weakness precludes using simple ECB mode in most cryptographic situations. It is too easy to exploit this property of ECB mode with replay or reordering attacks because the encrypted blocks can be rearranged and still remain valid.

The talk that you reference implys that is it possible to progressively decrypt ECB encoded data by progressively decrypting one byte at a time. I don't see how this makes any sense in the way suggested in that video, and I am guessing that this is why you are wondering how this type of attack would be applied in practice. First note that changing even one bit of the plaintext block will on average change half of the bits in the resulting ciphertext when using a strong block cipher like AES. Mr. Ptacek suggests using a chosen plaintext attack to build up a dictionary of ciphertext to plaintext such as

       Plaintext           Ciphertext (in hexidecimal)
       </comment><comment></comment><comment>----    </comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>--
       AAAAAAAAAAAAAAAA    219d992d9f5ae84d29ffd4dfbeaf1f7f
       AAAAAAAAAAAAAAAB    a20400dbb98b6da0ac0178d71204460d
       AAAAAAAAAAAAAAAC    cd20f8731ccf8e6643b287aadaf84f26
       ...                 ...

Now, whenever we see a block containing cd20f8731ccf8e6643b287aadaf84f26 we know that it's plaintext translation is AAAAAAAAAAAAAAAC. But this doesn't help us. Changing only the last character or even bit of the plaintext is no different than changing the first of 12th bit, the result is a new completely cryptographically random result in the ciphertext. We can't progressivly attack the plaintext one character at a time as suggested by the video.

We can continue to fill in the dictionary by just submitting chosen plaintext to the system and recording the results. The problem with this approach is that to be useful this dictionary will contain many many entries and will change completely for each different key. For arbitrary plaintext, the dictionary will contain 2^64 entries or worse 2^128 or 2^256 entries depending on the block size of the underlying cipher.

All of this doesn't mean that ECB mode is safe against attacks, just that the attack as described by the video using pipe characters, etc. doesn't make sense. However, there are times where we can brute force ECB mode with a chosen plaintext attack. For example, if we already know that a block of important plaintext starts with the eleven characters "Password = " this leaves only 5 bytes in the block to be guessed. Now we can run through all 5 character values and use this. It may only reveal the first five characters of a password in this case, but that is an important vulnerability.

I believe that the section of this video to which you refer is confusing because Mr. Ptacek is attempting to present a simple introduction to an important cryptographic attack against another mode of using block ciphers. This attack is known as the _padding oracle attack_. The CBC mode (cipher block chaining) eliminates the weaknesses I've described above for ECB mode, but it has an important vulnerability of its own which makes use of a padding error oracle. I won't attempt to describe it here, it requires quite a bit of explanation, but by using this attack, a system that returns detectable errors when there is a block padding error, _can_ be attacked and almost an entire message can be decrypted _one byte at a time_ as Mr. Ptacek suggests. It's just all more complex and applies only to the CBC mode not the ECB mode.

Unfortunately, on the web, there is a lot of hard to understand information on such cryptographic attacks. I find many blog posts are by those describing systems that they don't fully understand and papers written by researchers that do understand the attack but make heavy use of mathematical notation and describe the attacks in their full generality. There are few resources with easy to understand diagrams and clear descriptions. For this particular padding attack on ECB mode I suggest: [http://robertheaton.com/2013/07/29/padding-oracle- attack/](http://robertheaton.com/2013/07/29/padding-oracle-attack/)

<comment><AUTHOR> Thanks a lot for the response, you've given me a lot to go and research.</comment></comment>


The Internet is Broken - <AUTHOR> https://www.nytimes.com/2017/05/20/technology/evan-williams-medium-twitter-internet.html?rref=collection%2Fsectioncollection%2Ftechnology&action=click&contentCollection=technology&region=rank&module=package&version=highlights&contentPlacement=1&pgtype=sectionfront&_r=0 <comment><AUTHOR> "Ad-driven systems can only reward attention," Mr. Williams says. "They can't reward the right answer. Consumer-paid systems can. They can reward value. The inevitable solution: People will have to pay for quality content."

I for one am willing to do this, and I do already (NYT, The Economist).

The question is, how many other people will voluntarily pay to receive content they don't always agree with?

Maybe we'll end up with a system in which only wealthy people can receive factual, balanced news? (Silicon Valley types: by "wealthy" I mean top 50% not top 0.1%)

Sort of like we have a system now in which, by design or by consequence, it's more difficult, logistically, and in terms of price, for poor people to eat healthy compared to wealthy people.</comment>


Spanner vs. Calvin: distributed consistency at scale - <AUTHOR> https://fauna.com/blog/spanner-vs-calvin-distributed-consistency-at-scale <comment><USER2> The all-to-all dependency step between Calvin's sequencer layer and scheduler layer seems like it will be a problem as things scale, because it means that a single stalled sequencer [edit, orig: scheduler] blocks all writes in the system whether they conflict or not. This is the kind of dependence structure that magnifies outlier latencies and unavailability at scale.

In Spanner's design, on the other hand, a transaction can only be blocked by another one on which it actually conflicts. It will have worse average performance, but better tail latency as things scale.

Perhaps it is just a nit, but the blog post is somewhat inaccurate when it says that Spanner uses the commit timestamp to order transactions. It uses locks to order the transactions, then holds the locks for extra time to ensure that TrueTime order agrees with the lock-based order.

<comment><USER3> You should think of the sequencer layer as a shared log abstraction along the lines of the Corfu project from Microsoft. It is distributed and replicated, with the scheduler layer reading from their local copy. Stalled scheduler nodes do not block writes in the system.

<comment><USER2> I mistyped, I meant to say that a stalled sequencer stalls all schedulers. It's true that Corfu has impressive per-sequencer throughput (by moving most of the work onto other nodes), but you have to move to multiple sequencers to get to Spanner scale.

<comment><USER3> Yes, you move to multiple sequencers, and you have them reach consensus via Paxos (to avoid problems from individual stalled sequencers).

<comment><USER2> So to get back to Spanner's availability (if you need it), you need the Calvin sequencer Paxos groups to span data centers. Since you're not exploiting the commutativity structure of transactions you need either a leader-based consensus implementation, which will have latency stalls when a leader becomes unavailable (amplified by the all-to-all communication), or you can use old- school 2 RTT Paxos, and your end latency ends up the same as Spanner.

Lest it seem like I'm not actually a fan of the log-based approach, let me point out a way in which Calvin crushes Spanner: write contention. So long as the Calvin transactions can encode the application logic, they can support extremely high write rates on contended objects in the database. Spanner's 2PL, on the other hand, has single-object object update rates visible to the naked eye.

</comment></comment></comment></comment></comment><comment><USER4> Tangentially, while FaunaDB and Spanner both seem great, and I love consuming the technical details in articles like these, neither I or my companies are ever going to commit to using something which isn't open source.

I have a real problem understanding why Google (and now FaunaDB) are so clueless and/or inconsistent about software licensing. Google seemed to finally get it with Kubernetes, which has grown into a real and vibrant ecosystem; I will (and do) happily use Google's hosted Kubernetes product, while I know, at the same time, that I am not locked into a single vendor. With Spanner (and FaunaDB), all I get is lock-in. (The cynic in me suspects that Google's open-sourcing of Kubernetes is a cunning strategy to dominate the cloud by commoditizing container orchestration, and thus undermining competitors such as Amazon. After all, it's not like Google is open-sourcing Cloud Storage or BigQuery either.)

The fact that I can't run these products locally on my dev machine is another major downside unrelated to the philosophical aspects of free software.

I'm sure Google and Fauna will make truckloads of money on enterprises that are willing to bet on them, of course. Personally, I am holding out for CockroachDB.

<comment><USER5> > The fact that I can't run these products locally on my dev machine is > another major downside unrelated to the philosophical aspects of free > software.

You can't run a full spanner cluster unless you have the Google Hardware. Spanner needs to have an accurate clock with a known error bound. This makes it difficult to open source without open sourcing a bunch of hardware.

<comment><USER6> > This makes it difficult to open source without open sourcing a bunch of > hardware.

That's simply not true; there are many consumer-grade GPS clocks out there well within the bounds that spanner operates (iirc skew bounded at +- 5ms).

Not only that, but you can bet that if spanner were open-sourced, you'd see additional pressure to lower the cost of that hardware.

Not that the answer is any more nefarious: They don't open source it because it extensively uses google-only code, and open sourcing spanner would entail open sourcing many, many other parts of code, which is expensive. It'd be much easier to have something like the dynamo db local front to sqlite3 like amazon offers.

<comment><USER7> A critical design feature of Spanner is owning the packets end-to-end (as in, between data centers and across oceans). This lets Google minimize the "A" in "CAP", and would be quite difficult to replicate outside of a network like Google's SDN.

Spanner is the software, plus the network, plus the hardware.

(work on Google Cloud)

<comment><USER8> \+ The army of SREs monitoring it...

</comment></comment></comment></comment></comment><comment><USER9> The Spanner design seems more resilient in the face of server failures. The initial Calvin papers call for taking the entire replica offline if a single server in the replica fails. Are there more advanced versions of Calvin that get around this?

<comment><USER3> Yes --- the current version of Calvin (in the Yale research group) does not have this limitation. We're actually not sure which paper you're talking about, but either way, it's not fundamental to the Calvin approach. In general, if a single server in a replica fails, the other servers within the replica that need data from the failed server can access that data from one of the replicas of the failed server. (We can't speak for FaunaDB, but like the current version of Calvin, it is unlikely they have this limitation.)

<comment><USER9> My understanding was that the replica would go down in order to recover the failed server. This was a side effect of the way snapshots and command logging worked. You couldn't just restore the snapshot on the failed node because the multipartition commands would have to execute against the entire replica. Instead you would restore the snapshot on every node, and roll forward the entire replica.

</comment></comment></comment><comment><USER10> > Before we get started, allow me to suggest the following: Ignore the CAP > theorem in the context of this discussion. Just forget about it. It's not > relevant for the type of modern architectural deployments discussed in this > post where network partitions are rare.

Anyone worried about that statement. Ok so they are rare, but what happens when they do occur? File corruption is rare as well but in our large deployment over thousands of servers I see it happen pretty often.

<comment><USER11> Calvin is still a CP system, so nodes outside of the quorum cannot proceed. The point however is that partitions are rare enough that a CP system can still provide a high level of availability, despite the theoretical limitation. Eric Brewer, who came up with the CAP theorem explicitly makes this point here: [https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45855.pdf)

<comment><USER10> That's an excellent description. And post's author describes that systems behave like CP.

But in general, Google's network is very different than other networks. With their resources they can provide guarantees and capacities other run-of-the- mill data centers can't. So others probably shouldn't listen and assume it applies to them as well.

<comment><USER11> IME, while Google's network is really good (based on my experience w/ GCP at least), AWS cross-region traffic, for example, is still pretty reliable.

Reliable enough, at least, that trading off perfect resiliency in the face of partitions is worth it to gain strong consistency semantics.

IMO behavior in the face of partitions is a bit of a red herring. The latency tradeoffs required in a geo-replicated CP vs AP system is much more relevant.

</comment></comment></comment></comment><comment><USER12> I would like to point out that the original implementation of Calvin is open source: [https://github.com/yaledb/calvin](https://github.com/yaledb/calvin)

</comment><comment><USER13> "It's not relevant for the type of modern architectural deployments discussed in this post where network partitions are rare"

I think the post needs to be more precise about this. Modern networks are asynchronous and so they are essentially always partitioned. And this is important, because later, when snapshot and transactional read latencies are discussed, they are not exactly "very low", not on "AP" systems level. Because "CP" systems need consensus they can't just send requests to all of the replicas and wait for the first reply or even read one locally, "very low" there is more about talking to a remote node "low", with all of the problems associated with it, like poor tail latency, etc.

<comment><USER11> Snapshot reads require just as many network communications as Cassandra does at consistency ONE, and provide a better correctness guarantee because effect order has already been determined. Data replicas can the single valid state at the requested snapshot time, or delay or reject the read if they are not caught up yet.

</comment></comment><comment><USER14> Abadi has an interesting paper here about why so many NoSQL databases are missing transactions. [http://dbmsmusings.blogspot.com/2015/10/why-mongodb- cassandr...](http://dbmsmusings.blogspot.com/2015/10/why-mongodb-cassandra- hbase-dynamodb_28.html)

The fairness, isolation, and throughput (FIT) tradeoffs seem more interesting to me than CAP.

<comment><USER15> Efficiency? Not sure how one measures fairness.

</comment></comment><comment><USER16> No pricing information for the self-hosted version. Gonna pass on this. Why even bother reading shitloads of text for something I can't try?

<comment><AUTHOR> FaunaDB on-premises pricing will be by core and comparable to other commercial databases like Aerospike, Datastax DSE, etc.

Daniel's post is relevant to more than just FaunaDB though.

</comment></comment><comment><USER17> "The current public version of Spanner does not support client-side interactive transactions either."

Did not know that...why?

<comment><USER18> I can't comment on Google's decision here. I don't see any technical reason why they can't support client-side interactive transactions. But I'm happy to comment on technical points I made in the post ...

<comment><USER19> (Disclaimer: I work on Cloud Spanner).

Cloud Spanner fully supports interactive read-write transactions.

I'm not sure what the source of the confusion here is. Maybe Daniel is using a new definition of "client-side interactive transactions" that I'm unfamiliar with. :)

<comment><AUTHOR> Our source for that was this post: [https://quizlet.com/blog/quizlet-cloud- spanner](https://quizlet.com/blog/quizlet-cloud-spanner) (SQL Dialect)

Maybe that's no longer the case?

<comment><USER19> Yeah, I've read that post, and I have no idea where it gives the impression that we don't support interactive transactions. What paragraph are you looking at?

Could you be referring to the fact that we make you do writes via a mutation API rather than DML? Obviously that has no impact on interactivity...

<comment><AUTHOR> Yes, our misunderstanding and we misinformed Daniel. Fixed, and thank you.

It would be cool to know why Spanner is like that.

<comment><USER20> (disclosure: CockroachDB founder) The reason I've heard is that Spanner uses a separate mutation API instead of SQL DML because of a quirk of its transaction model. Writes within a transaction are not visible to subsequent reads within the same transaction (source: [https://cloud.google.com/spanner/docs/transactions#rw_transa...](https://cloud.google.com/spanner/docs/transactions#rw_transaction_properties)). This is different from other SQL databases, so the use of a different API forces you to think about your read/write transactions in a Spanner-specific way.

(FWIW, CockroachDB does not have this limitation - transactions can read their own uncommitted writes, just like in other databases)

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> Is there any good paper on Consistency Models across different Network Partitions types?

</comment><comment><USER21> Welcome Dan.

What other production implementations of Calvin are out there?

<comment><USER18> I said in my post: "influenced the design of several modern "NewSQL" systems" \--- I'm not aware of other production implementations of Calvin. But VoltDB's command logging feature came directly from Calvin. So basically I had in mind FaunaDB and VoltDB when I wrote that sentence. Neither is an exact version of Calvin, but FaunaDB is closer to Calvin than VoltDB. Obviously, the Calvin paper has been cited many hundreds of times, so many of its ideas have made it into other systems as well.

<comment><USER22> > But VoltDB's command logging feature came directly from Calvin.

VoltDev here. Huh? We added this feature in 2011 and read the Calvin paper sometime later IIRC.

<comment><USER18> "The case for determinism in database systems" paper (which described the technology that became Calvin) was written in VLDB 2010. At least one VoltDB developer told me that command logging came from a talk we gave about this paper to your team.

<comment><USER23> I think they happened independently, but it's long enough ago that I might not recall if I or Ning had inspiration from somewhere.

If you heard it from Ning or myself then it's probably what happened.

<comment><USER18> Yes Ariel, it was you who told me this! But I agree with jhugg that it was an obvious choice based on the VoltDB architecture.

<comment><USER22> lol

</comment></comment></comment></comment></comment></comment></comment><comment><USER24> I think it's often a better strategy to do sharding or micro-services and if possible keep the service so small that it's state can fit in a single machine. If you can have data boundaries, like for example one customer do not need to access the data of another customer, then you can separate customers's data and place them in different databases.

<comment><USER12> The state can never fit on just a single machine, unless you're perfectly confident that single machine will never fail :)

Distributed consistency as discussed in this blog post is not really about scaling write throughput (which is the problem solved by horizontal/vertical partitioning), but rather about keeping replicas in sync with each other, seeing the same state. The usual suspects, MySQL and Postgres, don't really have any sort of replication that's totally synchronous (although they're getting there[1, 2]).

As for scaling writes, I agree that chopping up your Postgres database is the safest choice for now, but it's generally true that this is hacky, and kind of a pain to architect your application around it. Solutions like Spanner and FaunaDB aim to free the application developer from this burden, _without_ having to give up the large feature set of relational databases by using a NoSQL database.

[1] [https://dev.mysql.com/doc/refman/5.7/en/group- replication.ht...](https://dev.mysql.com/doc/refman/5.7/en/group- replication.html)

[2] [http://paquier.xyz/postgresql-2/postgres-10-quorum- sync/](http://paquier.xyz/postgresql-2/postgres-10-quorum-sync/)</comment></comment>


Ask HN: YC W2020 Site Contains GoogleMap API? - <AUTHOR> I was checking my YC W2020 status, however I found something interesting. Why did it load "Google Map"? I did not remember that there was a google map widget inside the application form. Did I miss out something? Feel free to check my screenshot below.<p>https://twitter.com/richajak/status/1180696826216775680/photo/1 <comment><USER2> I think they are using Google Maps APIs without showing a map.

You can see the library included in the html file via view source and some custom code (search for google.maps) in

[https://apply.ycombinator.com/assets/application-4493d388285...](https://apply.ycombinator.com/assets/application-4493d388285847d71d3a46fcee4dbeffdc51a7db6bbb457fe041eea5dc7e183c.js)</comment>


A group of ex-NSA and Amazon engineers are building a 'GitHub for data' - <AUTHOR> https://techcrunch.com/2020/02/20/gretel-nsa-amazon-github-data/ <comment><USER2> I once took a call with an ex-NSA guy who was a CEO selling email security software- your MX would point to them, they'd scan your incoming email for exploits, then deliver it on to you. When I spoke to the guy, I expressed my concern, having working for a large multinational corporate whose fiber optic lines were tapped by UK intelligence to avoid NSA laws against spying on americans, that if I couldn't inspect his software, I couldn't feel confident of the security and integrity of the scanning system.

I said, in good faith, that I would consider his product if I could inspect the running system and the code.

He said several things: 1) the NSA never did anything illegal 2) the software was too large to audit 3) it was an insult to his employment in the NSA that I was even asking these questions.

Then he hung up.

<comment><USER3> It is just plain rude to apply his previous employer's actions, who employ over 30k people, solely on him.

<comment><USER4> If the NSA guy is selling himself based on his past experience, then he himself is the one dragging those issues into the conversation.

<comment><USER2> Yes, he sells himself on that experience (I had already done due diligence on a previous company he founded, sqrrl, which was organized around open source software, but touted the NSA creds):

Oren Falkowitz CEO and Co-founder Oren Falkowitz co-founded Area 1 Security to discover and eliminate targeted phishing attacks before they cause damage to organizations. Previously, he held senior positions at the National Security Agency (NSA) and United States Cyber Command (USCYBERCOM), where he focused on Computer Network Operations & Big Data. That's where he realized the immense need for preemptive cybersecurity.

</comment></comment></comment></comment><comment><USER5> When you hear a company talk about the promises of synthetic data, you should run far far away. The fundamental problem is that in order for synthetic data to be useful for model training the generative synthetic model must have already solved the problem at hand. Training on the synthetic data is just a charade: you would be better off simply extracting the model you need from the source generative model. For generative models with densities this is as simple as P(Y | X) = P(X, Y) \ P(X).

So, what do you get out of it compared to simply training models as a service for people? Almost nothing useful. All you get are:

\- Much worse privacy guarantees. (People selling synthetic data love to talk about improved privacy, but it's actually the reverse. The privacy guarantees for synthetic data are much, much worse than selling direct models to people.)

\- Much worse model performance. See the previous notes about how a synthetic data generating procedure must have already solved the problem at hand.

\- A much more complicated setup with much more expensive model training. Training generative models is hard and requires a lot of data and compute due to the difficulty in learning such a complex outcome space. This can easily cost 100-1000x as much as simply training a straightforward xgboost model.

<comment><USER6> I'd argue you're incorrect on most accounts. I work in this space, and have implemented what's being described related to synthetic data.

You can see some of my work here:

[https://medium.com/capital-one-tech/why-you-dont- necessarily...](https://medium.com/capital-one-tech/why-you-dont-necessarily- need-data-for-data-science-48d7bf503074)

The reality is, you can guarantee certain values are not passed. That's fairly easy, what you can't do is block general trends easily in synthetic data - else the model can't learn. So you have to be willing to accept that "leakage" when using synthetic data.

If you accept that leakage, then you can actually improve model performance in some domains. See ubers architecture search blog, there's a lot of material (also from Uber) showing this.

Regarding cost of training, yes model training costs could increase. Although, I'd suggest much much less than 10x more like 50% or something. Is that worth it for privacy?

<comment><USER7> > The reality is, you can guarantee certain values are not passed. That's > fairly easy

Nonsense. There's a fundamental requirement that for a model to have utility, it must have access to relevant data. If you truly find a leak-proof way to block certain values (which is borderline impossible in itself), then you make the model significantly weaker or outright useless if those values aren't uniformly distributed across the input domain.

And if you do have leakage (and you almost certainly do), then direct re- identification becomes a trivial problem and we're back to square one.

Synthetic data generated from statistical analysis of real data is either worthless or leaky. One of those two conditions is formally guaranteed to be true.

GP is correct that a viable synthetic data generator basically has to have already solved your problem for you, and in that sense it just becomes your model which is trained on real data. Training an additional model on top of that model doesn't add any privacy or mitigate reidentification.

</comment></comment></comment><comment><USER8> I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a similar amount of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data contests with The Economist - but at the end of the day, we couldn't find anyone with an urgent problem that they were willing to pay to solve.

I wish these folks luck! Perhaps things have changed; we were part of a flock of 5 or 10 similar projects and I'm pretty sure the only one still around today is Kaggle.

[https://www.youtube.com/watch?v=EWMjQhhxhQ4](https://www.youtube.com/watch?v=EWMjQhhxhQ4)

<comment><USER9> Sometimes an idea has a time. It may seem obvious now but it isn't like Instagram and Snapchat were the first image sharing applications developed. Slack was nowhere near the first chat app.

I happened to have a long discussion on the topic of data businesses last night with a friend. We brainstormed datasets that would be a combination of hard/expensive to obtain while also having resell ability to thousands of customers who would be willing to pay a high value for them. I don't want to get involved in datasets that are easy to obtain (too many competitors, no bar to entry) or datasets specific to a particular company (too much dependence on a small set of customers, cost of acquiring new customers also includes cost of acquiring data, no economy of scale).

It's easy to start with the tech problem: how to collect, clean and analyze data. But reasoning backwards from the business side is much more difficult. Expensive data I can sell once feels easy. Cheap data I can sell frequently feels like a race to the bottom. Expensive data I can re-sell 1000s of times to a niche audience feels like a perfect middle ground ... I just can't think of any examples.

<comment><USER10> What makes a dataset hard/expensive to obtain?

<comment><USER11> I had a friend tell me recently about a client using commercial real estate data for lead gen. He mentioned [https://compstak.com/](https://compstak.com/)

Basically, identifying companies that are doing well / expanding by how big the space is they leased. This sort of data is apparently very hard to get, but gives users a competitive advantage.

<comment><USER9> Real estate data, and companies like compstak are exactly the kind of niche markets I'm talking about. Agents are willing to spend large sums to get access to this data and it can be resold multiple times. Unfortunately it is also a market full of existing competition with some established players.

What other markets for data are similar? In general, data that leads to prospect generation is desirable because sales agents are willing to spend money to make money. Are there any other markets like that?

<comment><USER12> So it sounds like the salient aspect here isn't necessarily the type of data, but the manner in which that data is collected. Looks like compstak's success is a result of creating a platform that facilitates crowdsourced data points that are difficult to acquire using traditional data collection approaches...that scarcity is what makes the data valuable, especially since that data can be used for leverage in a negotiation. Also, they appear to prop up the overall scarcity by only granting access of existing data to users who provide new data.[1]

I'm curious how they figure out how much to charge companies for this data? And also how they stop real estate insiders from gaining access without sharing new data?

[1] [https://techcrunch.com/2012/10/18/compstak/](https://techcrunch.com/2012/10/18/compstak/)

</comment></comment></comment></comment></comment></comment><comment><USER13> I'm a bit unclear on the goals for this startup. When you say "Github for Data" I'm thinking of a repository of datasets used for ML training or for more traditional research. But this:

> This so-called "synthetic data" is essentially artificial data that looks > and works just like regular sensitive user data.

So its like Lorem Ipsum generator for data? Whats the use case here besides building apps with sample data? Notwithstanding potential privacy concerns, How am I confident that this is realistic if you literally say its generated?

General data repository for research with some mechanism to ensure cleanliness or integrity sounds much more useful to me.

<comment><USER14> As you have noticed, "Github for Data" is not what they're building. At all.

Our company is actually building Git for Data and Github for Data. We have an open source database called Dolt which combines the commit graph of git with the relational tables and SQL of MySQL:

[https://github.com/liquidata-inc/dolt/](https://github.com/liquidata- inc/dolt/)

Then we have a DoltHub, which is Github for data:

[https://www.dolthub.com/](https://www.dolthub.com/)

Dolt lets you version, branch, and merge your dataset so that you can collaborate on it with others. Dolthub lets you share your dataset with the world, submit PRs, fork other people's repos, and lots of other analogous features to Github.

<comment><USER15> You might want to re-think that name: [https://www.urbandictionary.com/define.php?term=dolt](https://www.urbandictionary.com/define.php?term=dolt)

> A mental retard who is clueless not only about current events, but also has > the IQ level of a rock. "Dolt" may be the most sophisticated insult in the > English language. Dolts commonly populate such stereotypes as jocks, nerds, > fruits, bookworms, and dorks.

<comment><USER16> It's probably intentional, because "git" is also an insult to imply somebody's dumb.

<comment><USER15> True, but the name `git` came from the open source SCM tool, written by Linus for the Linux source code - and now everyone is sort of stuck with that name. This is a commercial product, deciding on a name for themselves.

</comment></comment></comment></comment></comment><comment><USER17> I don't want a git _hub_ for data, I want a _git_ for data. The one reason that POS Perforce is used in game development is that it can handle binary files, especially large ones. LFS is..OK.

And this company isn't really even aping GitHub given that they are generating the content too.

<comment><USER18> What would you use a "git for data" for?

<comment><USER19> Data?

Although having said then when you say data to me, I do imagine some level of immutability. Logging scientific results using git would be pretty good if required, in the sense that there is a habit to just use folders and text files which is fine at the time but is really really hard to take over sometimes (Like code written by scientists with their single-letter variable names and hatred of functions, in my experience of Fortran - yuck)

<comment><USER20> We've released our branched versioning system for data at [http://dvid.io](http://dvid.io). It's entirely open source and uses a science HTTP API with pluggable data types and a versioned key-value backend. I'm currently developing a new backend for it: DAGStore, a lower-level ordered key-value store that will have an explicit immutable store and is tailored for distributed versioning of big data.

</comment></comment></comment></comment><comment><USER21> ex-NSA engineers. That has a negative brand to it. Especially and specifically when it comes to respect for data privacy.

<comment><USER22> I've met several ex-NSAers that used their time there to build their brand successfully.

At the end of the day, the dragnet surveillance decisions came from the highest levels of the Bush and Obama administrations, not the boots on the ground.

<comment><USER23> "I was just following orders," is not an excuse with a precedent of being accepted.

<comment><USER3> No, the point is that the average NSA employee was not involved in dragnet at any capacity.

<comment><USER24> Most people who were members of socially bad organizations throughout history weren't bad people or doing bad things, but willingly joining and serving a support role for such an organization is itself morally dubious. It's actively helping enable their activities.

For example, if someone got a job on a shark finning boat maintaining the knives, they're still complicit in shark finning even if they're never touched or seen a shark in their life.

And even if their actual role was spying on and helping detain Americans for arbitrary reasons, nobody's going to slap that on their resume. They'd say they were a data aggregating administrative officer.

</comment></comment></comment></comment></comment><comment><USER25> I think a Gitlab for data is a much better proposition . See [https://quiltdata.com/](https://quiltdata.com/)

It is also open source and written in python. [https://github.com/quiltdata/quilt](https://github.com/quiltdata/quilt)

</comment><comment><USER6> I work in this space, at the intersection of synthetic data and security:

[https://medium.com/capital-one-tech/why-you-dont- necessarily...](https://medium.com/capital-one-tech/why-you-dont-necessarily- need-data-for-data-science-48d7bf503074)

The market opportunity is very large, but also insanely difficult to tap.

For one, you have an uphill battle on trust. Customers have to trust your data is secure, and btw it'll never be 100% secure by many standards.

On the other hand, people have to trust the synthetic data is good enough to use Practically.

So you have to both convince management and convince engineers. Arguably management is easier to convince, but.. best of luck on the endeavor!

That's not even discussing the technical challenges - I've implemented this all technically and have had it deployed to production systems. Building a robust system that is secure and produces valid synthetic data is a challenge.

</comment><comment><USER26> I see 'Github for data' but I'm reading services to hide PII. I think a distinction needs to be made here. Is the primary goal to enable Change Cata Capture on data(Github does CDC on code) or is the primary goal to manage PII?

<comment><USER27> Hey thanks for the great question. So the "Github for data" is referring to the ability to collaborate on data. By streaming in data, you can view discoveries we made on the data (entity recognition, etc) then essentially make a new version of that data with automatic transforms, anonymizations, etc. So you're absolutely right, managing PII is part of it, but really its about enabling entity A to share data with entity B with a high level of confidence the sensitive data is stripped out.

We'll be releasing some of the packages to do the analysis and transformations as time goes on, so stay tuned for those so you can take them for a test drive yourself.

Thanks!

</comment></comment><comment><USER28> I pitched exactly this years ago when I was the founder of NoSQL.com [http://bit.ly/NoSQLPitchDeck](http://bit.ly/NoSQLPitchDeck)

The problems with a 'Github for data' are the 7 'V's Volume: too much data to have usable diffs and merges Veracity: How do you know which branch of data to commit? Velocity: The data coming in is a stream - batch processing does not cut it

Most companies end up creating a federation, not a true Data Mart.

</comment><comment><USER29> A lot of the discussion here seems to be about the methodology and value of the "synthetic data" concept, and squaring that approach with the analogy to github.

It feels like we can tease those two things apart:

1\. Is a github-style website/service of forkable data sets useful?

2\. Are anonymized, synthetic versions of those data sets, created via ML, useful?

Feels like the answer to both is "yes"?

(Also makes me wonder if there's a "rebase" equivalent for data in this sort of world...)

<comment><USER30> > 1\. Is a github-style website/service of forkable data sets useful?

Depends on the specifics, but it can be.

I've been working on and off on getting a GitHub style data thing going ([https://dbhub.io](https://dbhub.io)).

It's still a work-in-progress, and I really need to get the data visualisation piece working, which is a pretty key feature and lousy to not have. ;)

</comment></comment><comment><USER31> GitHub for sensitive data? Why would I trust an external company obfuscating my sensitive data and not my developers doing this for me?

<comment><USER32> the same reason you trust external companies to do all sorts of other stuff for you: specialization. in the average case, they can probably do it better than your developers because it's their primary business.

<comment><USER31> Sure, but we are talking about a twofold trust problem here. Trust in an entity (external company vs internal developer) and trust in the work itself.

While I do see your point about trusting an external company which is specialized in the problem I'm trying to solve more than my own developers, I still have to transfer my highly sensitive data to them for which I have to trust them even more.

</comment></comment></comment><comment><USER33> Engineers from the NSA, Google _and_ Amazon worked on this? That doesn't exactly inspire any kind of confidence...

</comment><comment><USER34> I haven't seen it mentioned yet so don't forget about the dat project: [https://github.com/datproject/dat](https://github.com/datproject/dat)

</comment><comment><USER35> Sounds very similar to the OSS DVC project: [https://github.com/iterative/dvc](https://github.com/iterative/dvc)

</comment><comment><USER36> "How do the CIA, NSA, DHS, and FBI all share critical data without breaking the law with regard to specific details about certain types of collection targets (like citizens?)"...

FEDRAMP it. Now run that as a service inside a secure cloud environment. You need enough runway for the FEDRAMP / engineering / sales process plus a contract win and then I'd imagine the income gets pretty steady.

Commercially? Unsure of the use case. I'd imagine as those sharing data are typically in competition. Not so in the government / intel community / finance space and I'd imagine you have to write a metric ton of policy and sign a bunch of MOUs to do this kind of stuff properly. People in government do care a great deal about these policies, believe it or not.

This is also a huge problem to solve in the "Know Your Customer" / Anti-Money Laundering space for financial institutions, where sharing data between companies or government and companies is often prohibited and/or really difficult. See the recent FCA TechSprint for more on this: [https://www.fca.org.uk/events/techsprints/aml-financial- crim...](https://www.fca.org.uk/events/techsprints/aml-financial-crime- international-techsprint)

Lots of talk of "Homomorphic Encrpytion" and "Encrypted Cloud Runtimes" as options but if you don't really need to share all of the data to get to an outcome (but rather synthetic data - though synthetic identity seems to be the hard bit here...) that could be interesting!

</comment><comment><USER37> FYI: You might enjoy my talk notes titled "Using Git (and GitHub) for (Publishing) Data" [1] includes many real-world examples too.

[1]: [https://github.com/geraldb/talks/blob/master/git_for_data.md](https://github.com/geraldb/talks/blob/master/git_for_data.md)

</comment><comment><USER38> Hey all- we just released code on GitHub and a research post on Medium demonstrating how to generate synthetic datasets from models trained using differential privacy guarantees, based on rideshare datasets. Please take a look and let us know what you think!

[https://medium.com/gretel-ai/using-generative- differentially...](https://medium.com/gretel-ai/using-generative- differentially-private-models-to-build-privacy-enhancing-synthetic- datasets-c0633856184)

</comment><comment><USER39> A little off-topic, since we're not talking about true synthetic data here (vs. wiping PII), but the future of synthetic data is no-data due to differentiable programming. Instead of a program outputting vast amounts of synthetic data, it is written with a library or language that is differentiable and whose gradients can be integrated straight into the training of a model. A few PyTorch libraries dealing with 3D modeling have been released lately that accomplish that, and a good deal of work in Julia is making promising advances. I'm curious to see how overfitting will be addressed, but there may come a time when large datasets become a thing of the past as low-level generation of data becomes just another component of a model's architecture.

</comment><comment><USER40> Synthetic data is a tool in a system of privacy-preserving analytics. I don't think a lot of companies would be comfortable with shipping their sensitive data into a model sitting not on their premises. The model, needs to generate good enough results that would make sense in a training/analysis scenario (doubtful) and preserve the privacy of that data (difficult).

People in the Privacy-preserving industry try to find the silver bullet in one technology, but the real solution is a combination of different technologies. Trusted delegation of computation + privacy-preserving techniques together solve this issue, but separately provide marginal value.

</comment><comment><USER41> Honest question: does the NSA have good engineers? Like, FAANG good? It seems like a funny "hype line" for a startup, because I think of government engineers as ok, but not like, rockstars.

But maybe I'm wrong?

<comment><USER42> Most of the top engineers would probably be contractors. The government pay scale pretty much forces you to become a manager if you want to move past GS-12. Including locality pay, GS-12 salary tops at a little over 100k a year[1].

I just looked at their website, and they don't mention anything about NSA, Amazon, or Google on the landing page, so I think it's just techcrunch adding it in as clickbait, even though the founders aren't going out of their way to advertise it.

[https://www.federalpay.org/gs/2020](https://www.federalpay.org/gs/2020)

</comment></comment><comment><USER43> The article does not have a link to the company's website; this will save you a few actions: [https://gretel.ai/](https://gretel.ai/)

</comment><comment><USER19> Not entirely sure what this is yet but I was looking for a resource of scientific data in nice formats and I was very surprised I couldn't really find any. If this could do that in a way that doesn't upset any licensing agreements (Don't really know what I'm talking about in that regard) that would be pretty cool.

I needed a Resitivity-Temperature dataset for a Tungsten alloy and I ended up having to manually type up a series out of the book, not fun!

</comment><comment><USER44> Is this the same Lazslo Bock, founder of Humu, formerly head of HR at Google? If so, very diverse set of founding experiences.

If not, is this a common name in Hungary? Small world.

<comment><USER45> Same one, confirmed on his twitter [https://twitter.com/LaszloBock/status/1230520252384993285](https://twitter.com/LaszloBock/status/1230520252384993285)

</comment></comment><comment><USER46> I think this is a great idea. Perhaps the "Github for data" copy could use some work, but the concept of obfuscating real data for use in building systems without the overhead or concern of operating on real customer data is valuable. Of course, the degree to which the obfuscated data represents real data is important to ensuring the systems built like this are robust, but this seems possible.

</comment><comment><USER47> Since they're not launched yet, I think we can safely assume that this press release is not targeted at potential users.

</comment><comment><USER48> I've been using Snowflake for a while now and the Data Sharing feature they have is weirdly close to what I'm reading here. Anonymization, masking a column etc. Anyone with similar experience as mine? Or anyone who's knowledgeable enough to compare this product with Snowflake for me?

</comment><comment><USER49> I see "Laszlo Bock" is one of the founding members. He was industry veteran an exec at GM and Google (as a head of HR, as far as I know). He then started an HR company Humu ([https://humu.com](https://humu.com)).

</comment><comment><USER50> Wouldn't a data repository considered more of a wiki than vcs? I don't understand the desire to associate the concept with Github instead of Wikipedia, which I would propose is more appropriate.

<comment><USER51> What if you wanted to work on a data set across groups and you wanted version control?

</comment></comment><comment><USER52> There is already "GitHub for data": [http://academictorrents.com/](http://academictorrents.com/)

<comment><USER53> How does that even come close to solving the problems that a "GitHub for data" product would solve?

<comment><USER52> Arguably, Academic Torrents is closer to what GitHub does than the product in question. I don't know why they compare it to GitHub TBH.

</comment></comment></comment><comment><USER54> When I read the headlines, I thought it was these guys: [https://www.dolthub.com](https://www.dolthub.com)

<comment><USER55> There's a 404 at that link

<comment><USER53> [https://www.dolthub.com/docs/getting- started/introduction/](https://www.dolthub.com/docs/getting- started/introduction/)

This is the correct link

</comment></comment></comment><comment><USER56> Dolthub and Data.world sound more like that. This looks like one of numerous synthetic data startups.

</comment><comment><USER57> I like this idea but the article doesn't do a great job of spitting out why it's valuable.

Consider a SAAS with 25,000 active customers and tons of structured (database) data.

You have a bunch of people that need to work on the dev system and the closer dev looks to prod the better you are.

\- Contractors in another country

\- A team working on basic compliance with GDPR and CCPA

\- Sysadmin team trying to manage backups/restores

When the contractors pull a version of the DB it needs to not have any customer data (emails, addresses, etc.) so there's a process that wipes all those out and fills them in with fake data.

When the GDPR team gets a data deletion request ("Please delete all my data my email is firstname.lastname@example.com") on Monday and the Sysadmin team restores from a backup from Sunday what happens?

Right now both of these actions are one-off things done with a mish-mash of scripts, organizational knowledge, and half-remembered processes.

So wouldn't this be better with a service that could talk to your DB and you could fork out versions that "knew" the current DB structure, that you could mark as purged of sensitive data, that you could apply (and re-apply) transforms to for structure to data addition/removal.

</comment><comment><USER58> I would like this as a marketplace where you could license your data to other people.

</comment><comment><USER59> It just happens that I have been working on something like that, posted on Show HN.

</comment><comment><USER60> Great idea. I was just thinking about how this would be a great idea the other day.

</comment><comment><USER61> It's not Github for data, but if it were, then one of the main issues is handling GDPR requests. Git is by construction averse to deletions in its history, so incompatible with hosting sensitive data. On the other hand it could be great to have deduplication in storage and versioning for datasets.

</comment><comment><USER62> How about a version controlled law lookup, can I get an Amen?!

<comment><USER63> I'm sure I've seen this somewhere already. Don't recall what country though

</comment></comment><comment><USER18> Assertion: GitHub for data can't exist without a Git for data.

Naive take?

</comment><comment><USER64> That sounds like github in pretty much no way.

</comment><comment><USER65> We already have:

      git lfs

</comment><comment><USER66> Why not use GitHub?

<comment><USER67> Well, Git itself is not a good tool for handling large dataset files. In most cases, you're not interested in deltas between commits. The size of your repo can also grow or of control pretty quickly. As a dirty workaround, you have Git-LFS to do that for you. People tend to use it in repos with a lot of multimedia assets. This works well in many cases, but it has its own pitfalls as well.

</comment></comment><comment><USER68> How about a GitHub for science?

<comment><USER69> GitHub itself works fine for science. I see no particularly compelling reason to use an science-oriented service given that people are more likely to be familiar with GitHub.

Here's an example (or shameless plug): I use GitHub to share research data: [https://github.com/btrettel/pipe-jet-breakup- data](https://github.com/btrettel/pipe-jet-breakup-data)

There's code in that repository too. The code merges a variety of different data sources and performs some analyses. Nothing particularly fancy, and the code is probably not much better than average as far as academic code goes (which is not good), but I'm slowly adding tests and improving the code otherwise.

</comment></comment><comment><USER70> _"We're building right now software that enables developers to automatically check out an anonymized version of the data set," said Watson. This so-called "synthetic data" is essentially artificial data that looks and works just like regular sensitive user data. Gretel uses machine learning to categorize the data — like names, addresses and other customer identifiers — and classify as many labels to the data as possible. Once that data is labeled, it can be applied access policies. Then, the platform applies differential privacy — a technique used to anonymize vast amounts of data — so that it's no longer tied to customer information._

Anyone taking bets on how long it's going to be before these idiots end up leaking the SSN of every US citizen because their categorizer failed?

<comment><USER71> How do you know they're idiots? Maybe you're the idiot

<comment><USER72> You don't have to be a machine learning expert to understand that no classifier is going to be correct 100% of the time. The laws against divulging PII don't contain exceptions for classifiers goofing.

<comment><USER73> That's not how it works. Synthetic data is entirely artificial rather than transformed, so you're not worried about "missing some PII".

See for example the videos on [https://hazy.com/product](https://hazy.com/product)

Disclosure: Hazy cofounder.

<comment><USER72> Why would you need "to anonymize vast amounts of data — so that it's no longer tied to customer information" or "appl[y] access policies" if the data contain no PII? Presumably the ML is anonymizing the data and the access policies are necessary because the data contain PII.

<comment><USER73> Yup, people tend to confuse concepts and refer to synthetic data as anonymised data. They are very different things.

Anonymised data or redacted data are transformations of a data set that _hopes_ not to leak too much PII / sensitive data. People don't use ML to anonymise but they do use ML to classify as a first step before splatting or generalising.

In that case, its absolutely right that the ML classifier not being 100% results in PII leaking.

This is a key reason why anonymisation and redaction are widely seen as problematic and are being replaced by synthetic data and, maybe in future, homomorphic encryption.

<comment><USER40> Homomorphic encryption and any encryption in-use technology is no guarantee of privacy on its own. Synthetic data has the same dillema of utility vs anonymity as any other anonymization tech.

<comment><USER73> Yes, but starting from the position of "entirely artificial data".</comment></comment></comment></comment></comment></comment></comment></comment>


Ask HN: How to test Email Clients? - <AUTHOR>

Hello all,
I want to test email clients in my project. I could come up with test cases like:
1. Send mail in outlook, delete mail in mobile app, check in outlook it shouldn't be there etc.
2. The supported attachments
3. Check sender has message limits.
and go on.
These are simple test cases that I could come up with, but what are other test cases that I "must" test while testing email clients? That can be in :
1. Security
2. Usability
domain.
Any ideas will be really helpful. <comment><USER2> One sane way to test anything is based on requirements, which you probably don't have, unless you want to extract them from RFC's.

Here's a number of tables of standards, UI's and features of email clients: [https://en.wikipedia.org/wiki/Comparison_of_email_clients](https://en.wikipedia.org/wiki/Comparison_of_email_clients)

You could study those tables, think about it for awhile, and come up with a subset of derived requirements, and test against that.</comment>


Largest known soft-shell reptile egg discovered in Antarctica - <AUTHOR> https://boingboing.net/2020/06/18/largest-known-soft-shell-repti.html <comment><USER2> No inner bones and no shell... In my opinion this speaks volumes, not so much about biology of this type of animals, as what you would expect in a scenery of Ocean Acidification. Would be my first candidate.

That could also explain the super-size (water intake by osmosis after losing the calcium layer).</comment>


New technology enabling scientists to identify origins of illegal timber - <AUTHOR> https://www.nature.com/articles/d41586-019-01035-7 <comment><USER2> Fascinating article, but it's not some miracle tech somewhere. Just powerful, tried-and-true techniques combined with a hell of a lot of grunt work collecting and characterizing thousands of samples. Most of this stuff is still expensive, time-consuming lab work, not like a tricorder you point at a log that spits out a location. Soon, hopefully!</comment>


Drupal exploit spreads Monero malware to site visitors - <AUTHOR> https://drupal.sh/drupal-me0ws-malware-drupalgeddon <comment><USER2> I had a server get hit with this, lucky I have hourly snapshots. Was an old centos6 server, rebuilt it with 7, and put selinx on it.

Big problem I have is, I have no server-based IDS. Snort rules detected this drupal flaw and would have blocked it. With the web server using SSL, snort and suricata wont work as a host based IDS.

If anyone knows of a freeware server IPTables IDS for centos that uses snort rules, please share. The closest thing I could find was ETPLC.

[https://github.com/rmkml/etplc](https://github.com/rmkml/etplc)

Using these rules

[https://rules.emergingthreats.net/open/snort-2.9.0/emerging-...](https://rules.emergingthreats.net/open/snort-2.9.0/emerging- all.rules)</comment>


Yahoo ex-employee sues, alleging performance review manipulation, gender bias - <AUTHOR> http://www.latimes.com/business/technology/la-fi-tn-yahoo-lawsuit-20160202-story.html <comment><USER4> I'm very interested in seeing how this case pans out, seeing that he cited "favoring women" as a criteria.

In many courts, similar cases like this have been laughed off. In today's world, in many established companies the gender bias (female given priority) is a real form of discrimination. At my University (a top 10), the average female graduates where picked off by Google, MS and the likes - the extremely hard working, talented and passionate grads (male) couldn't even get interviews at big G half the time.

It's an interesting world we live in, where discrimination is now being fought with discrimination.

<comment><USER5> Google's defense is probably algorithmic.

<comment><USER4> I don't know if it could be blamed on an algorithm. Some of the companies (MS was one) even hosted parties for the female members of IEEE and ACM. Took them out to their campus, free meals, mock interviews, free swag, networking events. But they where female only events.

</comment></comment></comment><comment><USER6> So apparently he had been given a promotion and took a leave of absence to participate in a prestigious fellowship and was fired for having a low performance review? What metrics were they going by for the performance review if he wasn't even working at the time?

I have a feeling this lawsuit will actually go somewhere.

<comment><USER7> mm yes sounds fishy a bit like women on maternity leave getting a bad appraisal.

and allowing only women to appeal sounds like an automatic win on sex discrimination grounds.

Sounds like HR had a quota to fill and took the easy route and screwed up.

<comment><USER8> Ah yes, to fight with inequality and racism companies now use quotas - <USER2> system where they hire based on gender and race.

<comment><USER7> No HR had a quota to get rid of x% and took what they thought was the easy target.

Its incompetence in Yahoos HR function

</comment></comment></comment></comment><comment><USER9> Not judging the merit of the complaints, but trying to "cure" gender imbalances by adding another imbalance will create strong non-linearities, that will attract people that like to exploit these non-linearities instead of attracting true talent.

For every individual, every probability is always 50/50 - either something happens, or not. If an individual, a man, is affected negatively by a pro- women policy, he won't forget it for a lifetime. Because he is not a statistic, he is a person that has only one life to live.

Personally, if I were affected by this, or a colleague were, I'd find another job immediately, and watch the ship sinking from a safe distance.

<comment><USER4> Affirmative action is discrimination, and has strong negative effects on local economies.

Consider a firm which operates at equilibrium (where peak profit is achieved), with a ratio of 80% male, 20% female. Artificially capping the number of males allowed in the corporation, or artificially increasing the number of females pushes the corporation out of equilibrium thus reducing profits in the long term.

On the flip side of the coin, if the firm discriminates (equilibrium 50% male, 50% female but it chooses 80/20) - the economy punishes the firm by reducing its potential profits.

Companies should be hiring on skill, and culture fit. Not to fill a quota or someone's agenda.

It's true that there is a gap within genders in tech - but the answer isn't reducing another equally skilled person's chances of getting the job - this is bad for everyone (except the person who got the job). Instead, we need to look at the root cause and try to learn why the gap is so obvious. Fix the issue at the source, rather than using affirmative action as a bandaid. Maybe it even turns out not to be an issue. Who knows.

<comment><USER10> > Companies should be hiring on skill, and culture fit. Not to fill a quota or > someone's agenda.

As opposed as I am to discrimination (including "reverse" discrimination), i have to point out that you're missing part of the picture.

Companies are optimising for _profit_ , and there are many things that influence it. Including, for example, the media - and unfortunately, in the current (pro-female, anti-male) media climate, it pays off to be seen as pro- women - such companies simply get a better media exposure and extra praise! In other words, if the whole country supports discrimination, it might be in the best interest of a company to go along with it.

In general, I don't think we, as a society, should expect _companies_ to influence the _culture_ (in a positive way) - it would have to be the other way around.

<comment><USER11> Yep. You're touching here on one of the common bridges from libertarianism to neoreaction. Libertarianism is based on a handful of simple principles that work -- in theory. Neoreaction takes a step back and looks at the practical issues surrounding implementation of libertarianism (e.g. profit being derived from culture as you mention here), and of the results of libertarianism, and thus all the philosophical deductions and political necessities that result from those considerations.

(new account because I would like to remain employed)

<comment><USER10> > You're touching here on one of the common bridges from libertarianism to > neoreaction

Actually, my reasoning is coming from an entirely different point of view. My basic assumption is that _people follow incentives_. Markets (i.e. libertarianism) are _very good_ at providing certain kinds of incentives (basically, they encourage competition and drive the world toward greater and greater efficiency), but there are also well-known examples of market failures: monopoly, duopoly, cartels; externalities (pollution, resource exhaustion); barriers to entry and wasteful competition (main example being infrastructure - do we really need 2 companies digging through the whole city to establish 2 sets of separate, but otherwise identical phone lines?); agency problem; tragedy of the commons and free-rider problem (i.e. who pays for the fire department). In this case, I believe that government regulation is required and welcome; it has to be carefully enacted and continuously evaluated to make sure it results in increase of competition in the markets (otherwise we get e.g. copyright, which significantly hurts competition). I don't think neoreaction is a good example here; I believe that parts of EU, allegedly Denmark in particular (with their "flexicurity" style labor markets) are particularly good examples.

<comment><USER11> Libertarianism is flawed in many ways, yes, and its proponents do not consider its peripheral realities. To what you wrote I would add the observations that markets are created from the will of a central authority, and that the mob can become tyranny when not restrained by a more level-headed authority in whatever form.

These critiques (and more) are prominent in the neoreactionary literature on liberalism/libertarianism (but nowadays they have more important things to talk about). An interesting hypothesis of neoreaction is that liberal ideals, or even a libertarian world, will invariably lead to democratic socialism -- due to the power of the mob.

Personally, the foundation of my disillusionment with liberalism/libertarianism is the fact that people are so much more than economic units, and that collectivism trumps individualism. What's important is preservation of heritage, culture, tradition, creation of art in all its beautiful forms, and the maximization of humanity's potential.

I can't see liberalism/libertarianism getting us much further than working on advertising systems to push junk food, while NASA spends half its time making reports to get a few more handouts, and cultures worldwide are systemically destroyed in the name of economic growth.

</comment></comment></comment></comment></comment></comment><comment><USER12> "It also alleges that women were treated better by managers in the media group. While men were immediately terminated after receiving low employee scores, women were allowed to appeal their ratings, the suit says."

<comment><USER13> Its hard to rectify gender bias without somebody crying foul.

<comment><USER14> Err, why not let everyone appeal their rating?

(I understand, percentage wise, due to bias or whatever, it's likely a higher percent of things would be overturned for women, but it's not like the error rate for non-women was 0 anyway, so why not just make a process that nobody is going to complain about?)

<comment><USER15> What would be the result of that? You appeal and find, oh management really did make a mistake and is incompetent of it's primary purpose of evaluating employee performance correctly. AND, management did in fact discriminate and now has proven it for any and all future and current lawsuits. There is literally no reason for management to allow ratings appeals.

<comment><USER14> "What would be the result of that? "

(Let's ignore the obvious point - not everyone is malicious, some companies really do want to try to make sure people get correct ratings and get rid of discrimination, and aren't worried about the cost of lawsuits. These companies exist, i've worked at them)

Again, i think you missed the point (I guess i'll be more explicit when it's a touchy subject like this). If you read the allegation, women were allowed to appeal, men were not.

Someone tried to say that this was okay, because gender bias affects women more.

But it's not okay. It makes no sense. Either you believe some people are getting terminated wrongly, in which case both women and men should be able to appeal, or you don't, in which case nobody should get to appeal :)

</comment></comment></comment></comment></comment><comment><USER16> This is a problem in business management. If you want to discriminate against an employee in a certain group, you just give them low scores in a performance review. That way you can claim they are a bad employee to justify laying them off or terminating them.

Even if they did nothing to get low scores.

It sort of avoids discrimination lawsuits if they can document that the employee got low scores.

But this is fishy because he was given low scores while on leave.

Something like that happened to me while I was on short-term disability and in a hospital and at home. They had employees sign papers that I was doing all sorts of things to give me low scores. The dates when they claimed I was at work doing those things, I was in a hospital or at home and not at work. When I returned from short-term disability I was put on probation and then fired. They tried to use the stuff against me to block unemployment benefits but the state saw the complaints were on the same dates that I missed work and sided with me considering the complaints were made up. I could have sued for discrimination but it was the best law firm in town, and they could have proven an undue hardship defense because they hired a software consultant for $100/hr to do my job because none of my coworkers could do it.

So I would like to see the manipulation of the low performance ratings to be investigated. You see the years prior I had gotten high scores and pay raises, then as soon as I got sick, low scores and complaints.

<comment><USER17> I'll be honest, I find this hard to believe. Any employment attorney would absolutely jump at the chance to take a contingency case where there was verifiable proof that a company and coworkers fabricated evidence in order to terminate someone to avoid paying benefits.

<comment><USER16> I focused more on getting another job than suing because I had a family and a mortgage to pay to keep the house.

In hindsight I should have sued but I let 180 days go by.

</comment></comment></comment><comment><USER18> I would like to see more laws to help create a better work environment for everyone. We all spend most of our time at work -- yet, somehow, we seem to give up every right as a human being when we become employees.

I do think that "hire/fire at will" is acceptable - but we should have decent lives and a great social safety net at the same time. After all, we live in a time where incredible wealth is being created (it just seems to be hoarded by the 1%)

<comment><USER19> "We all spend most of our time at work -- yet, somehow, we seem to give up every right as a human being when we become employees."

Workers in the IT industry probably have the cushiest jobs in the world. You make way above minimum wage, you aren't risking your life, and your job can be done from the comfort of a heated or cooled office environment.

I'm not sure what rights you think you are giving away as an employee. Employee/Employer relationships are usually legally bound by a contract. You are free to change the contract or question it before you work there.

If you choose not to do this or don't want to take the time and effort into finding another job, it's no fault but your own.

I think what's missing is education. Most people are so complacent, they don't want to read anything they sign and then act surprised when it contains something undesirable.

I've crossed out and questioned plenty of employee contracts. Most businesses will work with you on this.

I also don't know what more laws you want. As a business, you have to be so careful as to not step on someone's toes or offend them or you will find yourself in a lawsuit.

Employees have nearly all the power in today's society (especially with hashtag warriors starting digital riots against companies that do any little thing they don't like), and somehow you think you need more? It just doesn't make any sense to me beyond pure greed and entitlement.

"After all, we live in a time where incredible wealth is being created (it just seems to be hoarded by the 1%)"

Worldwide, you are in the top 1% if you are making more than ~$30,000/year.

I know the current narrative is to punish the 1% for 'hoarding' money and somehow screwing you out of your earnings, but it's just not happening.

<comment><USER18> "IT industry probably have the cushiest jobs"

This isn't selfishly just about me, this is about what makes sense for our society.

"Employee/Employer relationships are usually legally bound by a contract"

There is a long history of contract terms being set by the more powerful party - and it is almost always workers who need the job and have less power.

"it's no fault but your own"

This completely ignores that opportunities are defined by our society - <USER3> the US has very low economic mobility.

"what's missing is education"

For over a decade education has not provided an improved salary in spite of much improved GDP. The perception of better salary has been shown to be lowered salary for the rest of society (excluding the 1% of course.)

"what more laws you want"

We could start by emulating other countries. Denmark workers join Unions which are funded through government. Flexcurity offsets "at will employment" with a great social safety net and free education. Corporatism creates "interest groups" which can be involved with decision-making for employers and industries. Some countries require employees on the board. Some countries only purchase from companies with transparent finances (no offshores). And these are just some techniques proven to work, there are many others which may be a good idea.

</comment></comment></comment><comment><USER20> _Anderson also says he may have been terminated because he reported that an employee tried to bribe him to reduce a co-worker 's performance score. The lawsuit says that employee had a "personal relationship" with Anderson's manager, who later gave Anderson a low score._

This is the most believable line to me.

</comment><comment><USER21> It's obvious that many review systems aren't designed to help an employee, but rather provide management with justification to make staffing adjustments while providing CYA cover to all parties involved. Sounds like Yahoo's is really no different in that respect.

I have a non-scientific suspicion that performance reviews are going to become the next industry "thing" that prospective candidates evaluate before joining a company. On par with evaluation of benefits, comp structure, etc.

We're aiming to improve perf evaluation in my company, and I'll actively promote that with prospective candidates when we begin to see benefit. We want to identify and reward top performers, and provide support to the bottom to enable them to improve. (We're driving toward a culture that views under- achieving performance first as a failure of management.) We want this to be a strength, and something that our teams use throughout their time with us.

<comment><USER22> > I have a non-scientific suspicion that performance reviews are going to > become the next industry "thing" that prospective candidates evaluate before > joining a company.

It's not already? The performance review system is something I evaluate very critically before deciding whether to join a company. Use Microsoft-style stack ranking? I'll pass.

</comment></comment><comment><USER23> Wow if that's true, this is not the way to reduce the gender gap.

<comment><USER24> I'm very happy that a lawsuit on male discrimination is happening. Being a male programmer, I have seen many females around me being more promoted than men, in my 3 companies. I cannot determine whether there is actually a discrimination happening, but I can tell that "statistics don't match my experience".

So a lawsuit is one way of determining the "truth, according to the law". And on the other hand, there starts to be gender equality if both genders sue equally often for discrimination. It's ironic, but it's positive.

<comment><USER13> It could simply be, that your company recognizes talent wherever it finds it. And women flock there because they can succeed there. That would just make it, competition.

As society changes, there will be many apparent imbalances. To tell if something is wrong, we cannot just look at statistics. There must be a better test for 'inequality'.

<comment><USER24> Because women are more talentful? If something irritates me, it's newspapers full of articles saying that women are better at work than men, women lecturing men on being being accordingly stupid, and no quality attributed to men about work or private life, apart from being stronger (yay, stronger, a great brain-dead prejudice).

I included your lecturing in my original comment, it's an humiliation that you could think I oversaw such aspects.

<comment><USER25> Just as a note, as I know nothing about your or your situation, but reacting angrily will NOT help any impressions you are giving.

Frankly, when someone tells me they work in a tech place where women are successful, I'm jealous. So if you want to point out that there is some hostile bias at work and not just the lack of hostile environments that drive women away from so many places, you need to prove that. Coming across as a woman-hating man does not accomplish this (Again, I don't know you, I'm only saying the impression you are giving).

<comment><USER26> > Frankly, when someone tells me they work in a tech place where women are > successful, I'm jealous.

I think the key there is "where women are successful *at mens expense" (feel free to swap the genders as well).

<comment><USER17> Well promotions are by definition zero sum. Promotion of any individual is at the expense of every other qualified individual who wanted the job.

<comment><USER26> You're assuming there is a fixed pie. What if the company is growing?

<comment><USER17> It's still zero sum for the current employees. Only the following scenarios are possible whether a company is growing or not:

      1. Nobody gets the promotion
      2. Somebody outside the company gets the promotion
      3. Somebody inside the company who isn't me gets the promotion
      4. I get the promotion

With the exception of #1, I only get the promotion at the expense of my coworkers (who want and are qualified for the job).

Edit: Put a better way, it is zero sum because you and your coworkers cannot equally share in the benefits of a promotion.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER27> It would appear that Yahoo had forsaken the three D-s of firing: Document, Document, and Document some more. Employees fired for legitimate cause usually leave behind a trail of evidence wider than the New Jersey Turnpike. This documentation can take many forms, and is astonishingly easy to come by in today's increasingly data-driven workplace.

Yahoo's apparent inability to gather evidence is just another sign of a poorly run company. Who will miss them once they've gone for good?

<comment><USER28> You're right about what HR should have done, but you're overplaying the significance towards Yahoo's current position.

</comment></comment><comment><USER29> Sounds like an awful place to work at.

</comment><comment><USER30> The allegations sound very bad if true, but that's table stakes for a wrongful termination lawsuit. Given the timing, most likely this gets settled for a tidy sum.

</comment><comment><USER31> Noteworthy dirth of comments in the normally chatty HN audience. Untouchable subject sadly.

<comment><USER32> Wrongful firing lawsuits are incredibly common across all industries. The merit of the case is impossible to figure out from a news article. People can speculate all they want, but really no one knows the facts other than those privy to the actual lawsuit.

<comment><USER20> Where is the vitriol people directed at Ellen Pao?

<comment><USER33> IIRC, vitriol started to be directed at Ellen Pao only once she was cross- examined in court, giving specific details, and her claims of discrimination suddenly seemed much weaker than what the media had been reporting before the trial.

I would expect exactly the same thing to happen here if the case ever goes to trial and Anderson's testimony in the courtroom doesn't support his initial claims as presented in the media.

<comment><USER34> > _IIRC, vitriol started to be directed at Ellen Pao only once she was cross- > examined in court_

It certainly increased after the court case began, but certainly some, uh, strong opinions from the start:

[https://news.ycombinator.com/item?id=4008796](https://news.ycombinator.com/item?id=4008796)

<comment><USER20> I read the first thread. People are under informed about discrimination cases. Ending an affair in the workplace is a common trigger for retaliation. Yet, someone argued that a woman was more likely to fabricate complaints after ending a relationship. Who is more likely to be vindictive? The manager or the employee? Most people acknowledge power tends to corrupt.

Though Pao did have a few things going against her. Husband wasn't a pillar of the community. The actual case, according to my employment lawyer friend, was weak.</comment></comment></comment></comment></comment></comment>


Don't pay the for-loop tax - <AUTHOR> https://medium.com/@DanHomola/dont-pay-the-for-loop-tax-802d381d1cd4 <comment><USER2> Summary: use functional-ish idioms, library code and syntax sugars instead of explicit for loops, and give partially correct reasoning supporting the contention that it's better.</comment>


Reconsidering relevance (google talk) - <AUTHOR> http://www.youtube.com/watch?v=4LZNqV4qZR0

<comment><USER2> For those of you into IR he discusses the use of set oriented faceted search to refine the meaning of a users query, particularly with the goal of involving the user in the refinement. He finds a set oriented approach to be better than clustering, which tends to be opaque. The discossion of faceted search begins around minute 29.</comment>


Ask HN: HN censoring Facebook licencing controversy? - <AUTHOR> So, I was actively paying attention to a particular topic that was on the front page recently, Facebook saying no to a licence change request from the Apache Software Foundation.[1]
This thread had over 750 points, and 250 comments. However, all of a sudden, after 12 hours of being present, it is being marked as a duplicate of [2], which has under 150 points, not nearly as many comments (approximately 130), and no presence on the front page (or the next 10 pages for that matter).
Surely policy should prefer the thread with the most activity and more points? Particularly when it's on something I really wish to pay attention to (or rather, the commentary of HN participants which I value, of which [1] was receiving a lot of), given that it seriously affects the direction of present and future projects (which, at this stage, will no longer be utilising React).
Cheers.
[1] - https://news.ycombinator.com/item?id=15050841 [2] - https://news.ycombinator.com/item?id=15050705 <comment><USER2> A moderator marked the later discussion as a duplicate overnight, which is the default move, but didn't notice the disparity in discussion until a little later and restored it. We've just merged the threads as well.

If you email us at email@example.com, we will read it and respond. Users seem to imagine that moderation happens in dark rooms with seedy agendas but the truth is we're a part of the community and accessible. Once we've had an initial cup of coffee and replied to the latest censorship-accusation thread, that is.

</comment><comment><USER3> Ranking history for the two stories: one story decayed over many hours, one story was at the top of the front page for hours. Guess which one was marked as a [dupe]?

[http://hnrankings.info/15050841,15050705/](http://hnrankings.info/15050841,15050705/)

Is there a way for users to "vouch" that a story is not a dupe?

Edit: Ranking of this Ask HN:

[http://hnrankings.info/15053297/](http://hnrankings.info/15053297/)

Edit2 (1440 UTC): the [dupe] tag has been removed and the story is back on the front page, rank #11.

<comment><USER4> You can send an email to the mods email@example.com with a small one paragraph explanation of the details that are different and interesting in the new article. Sometimes they agree and remove the [dupe] label, sometimes they disagree.

</comment></comment><comment><USER5> HN actively attempts to avoid flamewars and other types of unproductive discussions on the front page, by means of de-ranking articles that have a high ratio of comments to points.

I don't know if this is the case here, but it's a possibility.

<comment><USER6> "Flamewar" has become a weasel word for interesting discussion.

It's a shame HN actively combats debate, the only discussion allowed here is about benign stuff where mostly everyone here agrees.

<comment><USER7> Yup. The new HN admin's feefees are easily bruised...

</comment></comment></comment><comment><USER8> What's with the title? You're not alleging censorship in the body of your post - instead you pose it as a question about what the correct policy for marking dupes should be.

<comment><AUTHOR> Well, I guess I just can't see any reason why a mod of HN would prefer the older thread over the more active one, and thereby removing its position from the front page and stop any further HN discussion altogether on the topic. Even the older thread is not present at all on the front page, or any subsequent pages.

Anecdotal, but it certainly _feels_ like an attempt to censor the topic at hand. Now that I think about it, I agree that my choice of title wasn't the best to be quite confrontational, but it is what spurred my creation of this thread.

<comment><AUTHOR> Wow, I did not intend for this thread to get on the frontpage, I was merely trying to get an answer on why, and treated HN like Reddit. I need to look into how the ranking algorithm works more. I certainly wasn't trying to start anything. Sorry. :-/

<comment><USER3> You helped return the story+comments to the front page, thanks.

</comment></comment></comment></comment><comment><USER9> They listed it as a [dupe] and then it's gone - and now is this post removed from the main page.

The post was clearly gaining rapid traction on a Saturday morning - it could of easily hit 2000+ upvotes.

This is disturbing. HN mods better chime in.

</comment><comment><USER10> Your link [2] came 1 hour before your link [1]. Don't think there's anything malicious going on there, they just marked the one that came later as a dupe.

<comment><USER11> If marking something as a dupe would transfer points to the duped thread I would agree. I would also agree if it was marked as a dupe within a reasonable amount of time. But marking a popular, 12 hour old thread as a dupe of a marginally older, unpopular thread and giving it the ranking of the unpopular thread feels wrong and doesn't respect the intent the community has expressed by voting.

<comment><USER10> This seems a reasonable argument, and I think the mods should adopt your idea of transferring the points to the duped thread (and possibly the comments?)

<comment><USER3> And the post ranking!

<comment><USER9> I assume points being transferred would cover the post ranking.

<comment><USER3> The ranking is based on arrival rate of points, ratio of comments, total points, and other factors including manual moderator boost or demotion.

</comment></comment></comment></comment></comment></comment><comment><USER12> It is possible it was flagged by too many users. Censoring can happen from the users themselves and I think the "flag" button should be replaced or deactivated for users with weak Karma.

</comment><comment><USER13> Unfortunately a lot of manipulation and de facto censorship is going on HN... what a shame...</comment>