Ex-Googlers Train Machine Army to Sift Out Crooks - <AUTHOR> http://www.wired.com/wiredenterprise/2013/03/siftscience/ <comment><USER2> $0.10/query is a hefty price to pay. For comparison's sake - <http://www.maxmind.com/en/ccfd_features>

If siftscience turns out to be significantly more accurate, it may end up being worth it.. but at somewhere between 6 and 20x their competition's price, it will take quite the improvement.

<comment><USER3> That was my main takeaway from skimming. I'm not a sales person, but I could see ten cents being a significant portion of the profit margin on certain items.

<comment><USER4> (I work at Sift Science.)

For what it's worth, you can score up to 5000 users per month completely free with Sift Science. So if you run a small site, there's no fee. If you run a large site, you can see for yourself whether we're saving you enough money to justify the price.

In addition, there are discounts from high volume, for pre-payment, and there's the possibility of scoring only high-risk users. If you run a web site, feel free to shoot me an e-mail at email@example.com, and we can figure out how to make it work for you.

<comment><AUTHOR> (I work at Sift Science too.)

Just to clarify, we charge 10 cents per _unique user_ that you query in a month, not per query. So, you can query the same user as many times as you like in a month and the cost will not exceed 10 cents for that user.

</comment></comment></comment></comment><comment><USER5> How would this sit with payment gateways from a PCI standpoint?

An ideal customer would be an e-commerce marketplace, I imagine that Sift Science would want to receive as much information about the customer as possible, including credit card / address details. Are you guys completely PCI compliant? You're taking 10 out of 16 credit card digits...

From a quick glance of your website you make no reference to PCI.

<comment><USER4> Great question. We should add it to a FAQ. The PCI-DSS rules apply to systems that store the entire credit card number ("PAN" in PCI-DSS parlance). We don't accept the full credit card number -- just the first six digits (which identify the type of credit card and bank) and the last four (typically printed on receipts), which the PCI-DSS rules allow for. So if you're PCI compliant already, you'll still be PCI compliant if you use Sift Science.

<comment><USER5> Thanks for the information, I think I'll be in touch about an account in the next few weeks for a marketplace I'm developing.

Perfect timing too, I just started looking at our options for developing something similar internally.

Would love to see the systems that Etsy / Ebay for handling this type of fraud.

</comment></comment></comment><comment><USER6> So what happens when a user scores high? Suppose I'm a firefox & windows xp user from elbonia shopping at 3am, does the site not let me make a purchase? Am I forced to provide extra information or are items delayed before shipping?

<comment><USER4> (I work at Sift Science.)

We provide a score, and then let our customers decide what to do. The majority of our customers have a human review the user, and sometimes as part of that review, they'll do extra verification such as calling the user up. In other cases, customers will delay charging the credit card until they can verify it's legitimate.

</comment></comment><comment><USER7> Interesting that a prepaid gift card strongly suggests fraud. What if a person simply wants to protect his privacy?

<comment><USER4> (I work at Sift Science.)

One thing to note -- our system analyzes a whole bunch of patterns for each user. So just shopping at 3am by itself won't cause problems, nor will using a prepaid gift card by itself. But if a user matches multiple fraud patterns, then they're likely to get a high fraud score.

<comment><USER7> Cool, thanks for clarifying.

</comment></comment></comment><comment><USER8> Since the Sift guys are responding: from where do you get the raw data about fraudulent transactions? I'm assuming you have streams of fraudulent and valid transactions, otherwise you can't figure out what correlates with fraud.

<comment><USER4> Our customers send examples of users that they've banned from their site or who have caused a credit card chargeback -- these are the $label events in our API and quickstart.

Those $label events let us do two really important things.

First, we can learn patterns that are unique to a particular site. Every site is a little different, so that has a big impact. Patterns that catch fraud accurately for an auction site may not work at all on a travel site.

Two, if a user gets banned from one site in our network, we can identify when they attack another site. That means as more sites join the network, the system gets more accurate for everybody.

</comment></comment><comment><USER9> This strikes me as a very dramatic title.

</comment><comment><USER10> There is a second point of feedback that should be exploited. He knows which services are used for fraud, it is not unreasonable to conclude that Yahoo (2x) has something wrong in their service (terrible security problem) that AOL doesn't (0.5x).

</comment><comment><USER11> I've used sift science, it's pretty neat! I recommend checking out their product.

</comment><comment><USER12> From what I've read about this project so far, it requires adding "a single line of JavaScript to the site." So what's to stop scammers and spammers from just blocking the file from loading?

<comment><USER13> (I work at Sift Science)

If there has been no JavaScript activity from a user who makes a transaction on your site, that is a fraud signal in its own right. (You can send us events from your server in addition to adding the JS to your site, so that we know characteristics of your users' transactions that can't be gleaned from the JS. In both cases you set the user ID in the call to Sift.)

<comment><USER12> Ah, okay that makes sense, considering that both client side and server side events will be used. Thanks for the reply!</comment></comment></comment>


Disappearance of Frederick Valentich - <AUTHOR> https://en.wikipedia.org/wiki/Disappearance_of_Frederick_Valentich <comment><AUTHOR> Also, this video details some intriguing aspects of the case : [https://www.youtube.com/watch?v=Je_bJ95MVmU](https://www.youtube.com/watch?v=Je_bJ95MVmU)</comment>


CKSideBarController, a UITabBarController for iPad - <AUTHOR> http://www.cozykozy.com/first-look-cksidebarcontroller-a-uitabbarcont

<comment><USER2> I like this UI style, and clean code. Good work!

We released as somewhat popular iPhone-based "springboard" like clone a while back on GitHub (<https://github.com/fieldforceapp/openspringboard>) and have since gone on to iOS6 and the iPad. The code isn't anything to be proud of.

For our iPad app, instead of your side bar style, we use a top bar with the ability to slide it down when needed, an hide it when not; the content is wrapped in a UIScrollView of buttons for the various options, using a simple highlight to indicate the selected option. You can find some screenshots on our website or in the app store description of our iPad product.

Two questions for you: how about extending your code for both side-bar and top-bar orientations? And related, can the CKSideBarController be hidden?

<comment><AUTHOR> This wouldn't be a difficult enhancement. I'd probably add some sort of enum to indicate what edge the bar should be along. Some of the view layout logic would obviously need to change, namely the glow effect would need to rotate and reposition based on what position the bar was in (not a difficult thing, of course). This is not necessarily a high priority for me right now (I'd like to add some customization API for look and feel in the near term).

An optional swipe-to-hide/reveal feature is something I was planning on implementing. Look for that soon!

</comment></comment><comment><USER3> Since you use ARC, is there any reason you use old retain/assign syntax?

<comment><AUTHOR> No, those should be weak/strong. Thanks for good eye.

</comment></comment><comment><USER4> It was a pleasure reading through your code.

<comment><AUTHOR> Thanks! I put this together over the span of a day or two, so there's a few dirty spots, but pretty happy with implementation so far.</comment></comment>


yc input on rehomer - free, open, disinterested marriage of technology and real estate - <AUTHOR> http://www.rehomer.com The rising popularity of <a href="http://en.wikipedia.org/wiki/Free_software">free open source software</a> and <a href="http://en.wikipedia.org/wiki/Main_Page">free knowledge</a> excite my fiancee and me to no end. I imagine many industries can benefit from open and free philosophies. This manifesto explains our attempt to bring some of this culture into the real estate industry. The project is in its early stages--heavy prototyping and feature research.
Thoughts? Concerns? Good idea poorly executed? Bad idea decently executed? We present it to this fine community first; I figure if a bunch of open-minded entrepreneurs hate it, then I am probably off target for any audience.
Regarding the length, I understand it's wrong for the general public. At the moment we are targeting the copy to those people with a twinkle in their eyes--the ones who like to sweat the details and meditate on text. Hopefully those who merely skim it will catch at least one paragraph that turns on a light bulb.
For those who are lazy, like me, here is an abbreviated edition:
Consensus decision-making + free disinterested service + community moderation of listings and individuals + mobile features = the future of real estate established at rehomer.com
If you like the idea, point your real estate friends to the site; we will need legions of bright, courageous leaders to establish new standards of quality, possibly while under scrutiny from powerful competitors. If you dislike the idea, I respect your right to rip into me.

<comment><AUTHOR> I apologize if this text is duplicated elsewhere in my post. I've been lurking here since launch but have never posted.

The rising popularity of free open source software and free knowledge, like Wikipedia, excite my fiancee and me to no end. I imagine many industries can benefit from open and free philosophies. This manifesto explains our attempt to bring some of this culture into the real estate industry. The project is in its early stages--heavy prototyping and feature research.

Thoughts? Concerns? Good idea poorly executed? Bad idea decently executed? We present it to this fine community first; I figure if a bunch of open-minded entrepreneurs hate it, then I am probably off target for any audience.

Regarding the length, I understand it's wrong for the general public. At the moment we are targeting the copy to those people with a twinkle in their eyes --the ones who like to sweat the details and meditate on text. Hopefully those who merely skim it will catch at least one paragraph that turns on a light bulb.

For those who are lazy, like me, here is an abbreviated edition:

Consensus decision-making + free disinterested service + community moderation of listings and individuals + mobile features = the future of real estate established at rehomer.com

If you like the idea, point your real estate friends to the site; we will need legions of bright, courageous leaders to establish new standards of quality, possibly while under scrutiny from powerful competitors. If you dislike the idea, I respect your right to rip into me.

</comment><comment><AUTHOR> Thank you both for taking the time to comment. I wholeheartedly believe you are both right, as was my fiancee who told me the same thing. Three consenting opinions is enough to convince me.

But what can I say--I have a bad influence (<http://paulgraham.com/articles.html>)? ;)

</comment><comment><USER2> You need to cut out 99.9% of that text on the main page.

I'm sure you have very important information there that probably applies to me, but theres no way I'm going to be able to find it.

If you have more to say, place it in pages off of the front page. Users will go there if they want to find out more.

</comment><comment><USER3> It's good that you thought this entire thing through and can validate its existence with a lot of words. But no one is going to read it. Worse, people will leave the site. Try to sum it up, even if it only covers a fraction of the "big picture."</comment>


World's first lagoon power plants unveiled in UK - <AUTHOR> http://www.bbc.co.uk/news/science-environment-31682529

<comment><AUTHOR> Link to the project itself: [http://www.tidallagoonswanseabay.com/proposal- overview-and-v...](http://www.tidallagoonswanseabay.com/proposal-overview-and- vision.aspx)

</comment><comment><USER2> Fantastic, recently I've been really wondering why this kind of thing wasn't looked at more - it seemed to me it'd be a lot more consistent than using wind turbines or solar energy.

The idea looks excellent - obviously not 100% on the environmental impact (and I wonder if we really can get a good idea until some years after it's finished) but surely it's much better than coal & nuclear options?</comment>


React License – Additional Grant of Patent Rights - <AUTHOR> https://github.com/facebook/react/blob/master/PATENTS <comment><USER2> IANAL, but by my reading, the patent grant is intended as additional protection for both Facebook and the consumers of React. Facebook is free to register patents that may overlap in some ways with code that they've open- sourced, and you are free to use their code without fear that you'll infringe upon a future patent.

The 3-clause BSD license is retained in its original form [1] and makes no reference to the patent grant. Therefore, I believe you could fork the project and remove the patent grant in your fork. You would then lose the patent protection from Facebook, but they also could not revoke your rights to your fork of React.

That said, if your _use of React_ somehow violates one of Facebook's patents, you're in trouble. The original version of React would be protected by the additional patent grant, but your fork wouldn't be.

I don't think Facebook is acting maliciously; I just think they're trying to provide a means to balance their (perceived) need to protect their patent portfolio vs. their desire to release OSS. If they start suing people that are using their open-source projects, it's going to undermine the (rather good) reputation the company has for releasing quality OSS.

[1] [https://github.com/facebook/react/blob/master/LICENSE](https://github.com/facebook/react/blob/master/LICENSE)

<comment><USER3> It's also pretty similar to the Defensive Patent License. With that, to receive a free patent license, you have to join the 'community' by licensing _all_ of your patents under the DPL to other community members. If you sue a member anyway, your license is terminated. The language in the DPL is more precise in limiting itself to actual lawsuits (rather than just "claims") and excluding counterclaims, but it's the same general idea.

edit: corrected; DPL is not Twitter's license. Twitter's is the "Innovator's Patent Agreement", which is even more broad in its definition of "defensive" lawsuits:

 _(a) against an Entity that has filed, maintained, threatened, or voluntarily participated in a patent infringement lawsuit against Assignee or any of Assignee's users, affiliates, customers, suppliers, or distributors;_

 _(b) against an Entity that has filed, maintained, or voluntarily participated in a patent infringement lawsuit against another in the past ten years, so long as the Entity has not instituted the patent infringement lawsuit defensively in response to a patent litigation threat against the Entity; or_

 _(c) otherwise to deter a patent litigation threat against Assignee or Assignee's users, affiliates, customers, suppliers, or distributors._

<comment><USER4> This is actually a much _less_ broad restriction because it stipulates "voluntarily participated". The react license terminates even if you're on the receiving end of a lawsuit - it's not defensive, it's potentially offensive.

</comment></comment></comment><comment><USER5> IANAL, but this seems to imply that, if you use React, Facebook (and affiliates) can infringe on any of your patents with impunity -- you can't even verbally accuse them of infringement.

The same applies to Facebook's other open source projects (e.g. immutable-js), so one must also avoid any libraries that depend on those projects (even if it's a dependency of a dependency of a...)

This seems to conflict with the spirit of OSS, since the grant doesn't apply to modified versions of React (i.e., trying to fork React could result in a lawsuit). Again, IANAL, so I might be misreading this.

<comment><USER5> Also, as pointed out in previous discussions of this clause, this is significantly more restrictive than the Apache license.

This would make me quite hesitant about using React for any project of significant size (especially for a large enterprise). I saw a post earlier about Netflix using React -- I wonder if they know about this.

<comment><USER6> I think companies like Netflix et al are in a mutually assured destruction scenario with Facebook and others. They have the lawyers in house that can defend, counter sue, etc & by the current state of the patent system I'm sure there is something Netflix could try to sue FB for & I would guess neither one wants to fire the first shot in the war. The companies where this will be an issue for is smaller companies that may be in the right but can't afford a defense. INAL, I'm also a big user of React, while this license does irk me a bit, in reality if I were to ever go up against FB's legal team, I know I would most likely lose anyways. Don't mean to be defeatist about it, but its the state of our current legal system.

</comment></comment></comment><comment><USER7> The license granted hereunder will terminate, automatically and without notice, for anyone that makes any claim (including by filing any lawsuit, assertion or other action) alleging (b) that any right in any patent claim of Facebook is invalid or unenforceable.

that means if facebook sues you for any reason, they can also take away react if you try to fight it!!!!!!! Which raises the question: Can software truly be open source if there is a company to gain by controlling it?

<comment><USER8> That's a retaliation clause. That means if YOU file a lawsuit against Facebook then you lose the patent grant, but only if the lawsuit you file claims that you own some right to the software in question.

Basically this avoids Facebook being unable to defend themselves if you claim you have a patent that covers React.

<comment><USER9> How does it exclude assertions made when you're defending yourself against Facebook? I don't see it.

<comment><USER8> "The license granted hereunder will terminate, automatically and without notice, for anyone that makes any claim"

For anyone who makes any claim. The legal definition of claim is that you must assert you own all or part of the software. If Facebook makes the claim against you, the only defense you need is their patent license grant and the software license.

</comment></comment></comment></comment><comment><USER10> There is actually prior art; React is not the first public use of an immutable virtual DOM; a basically identical approach was presented at Clojure Conj November 2012 (React launched publicly in May 2013 iirc, though React was used on instagram.com in September 2012 [0]):

[https://github.com/drcode/webfui](https://github.com/drcode/webfui) [https://www.youtube.com/watch?v=HeI5-D7SQe8](https://www.youtube.com/watch?v=HeI5-D7SQe8)

If I found myself in a position where Facebook took away my access to React.js and to the patented ideas, does this help me at all? E.g. could I use this as a base and implement the immutable virtual dom myself, or looking into alternatives like mithril?

[0] [https://twitter.com/floydophone/status/427160179616927744](https://twitter.com/floydophone/status/427160179616927744)

(Some edits)

<comment><USER11> > E.g. could I use this as a base and implement the immutable virtual dom > myself, or looking into alternatives like mithril?

Prior art is probably only going to help in the case that you're already embroiled in a legal battle over infringement. That or if a USPTO patent review were to occur (e.g. [1]). But it'd be best to not be a potential infringer at the same time you contest a Facebook patent, unless you want FB's legal team gunning for you in retaliation.

That said, I think there are going to be a _lot_ of developers who simply don't know about and/or ignore the existence of this patent and go merrily about their lives anyway.

[1] [http://appleinsider.com/articles/12/12/19/apple-pinch-to- zoo...](http://appleinsider.com/articles/12/12/19/apple-pinch-to-zoom-patent- invalidated-by-uspto)

</comment></comment><comment><USER12> I will never use React for anything but (possibly) toy projects.

You don't need a lawyer to tell you being unable to sue Facebook for patent infringement even if unrelated to the licensed software, while Facebook is free to sue you for infringing other patents without you being able to challenge such patents, is an unfair and dangerous license.

And counting on Facebook's good will is as foolish as betting your livelihood on a Microsoft "promise" not to sue you.

Edit: fixed typo.

</comment><comment><USER13> Previous discussion of this:

[https://news.ycombinator.com/item?id=8901357](https://news.ycombinator.com/item?id=8901357)

</comment><comment><USER14> I understand that these things are written as defensively as possible, but could someone explain how likely this is to _actually_ be used? It seems like something that's intended to protect FB, but I can't imagine a situation in which they would actually use it.

Perhaps someone with more legal experience could point out the precedent?

</comment><comment><USER15> Maybe it's time to make a truly F/OSS React clone?

For those who may want other options to React after reading this, mithril is small & has JS DOM:

[https://lhorie.github.io/mithril/getting- started.html](https://lhorie.github.io/mithril/getting-started.html)

How it's rendered:

[http://lhorie.github.io/mithril/mithril.render.html](http://lhorie.github.io/mithril/mithril.render.html)

Differences with React:

[http://lhorie.github.io/mithril/comparison.html#react](http://lhorie.github.io/mithril/comparison.html#react)

<comment> TD-Linux Cloning a project does not absolve you from patent claims, though it does from copyright.

What is the issue with the React patent grant, exactly? (edit: reread it again. seems it applies to all Facebook patents. I had thought it was like the Opus license, I was wrong)

<comment><USER15> In the flurry of IANALs in the comments, the readings of others seemed to match what bothered me:

nkohari put it best: _That said, if your use of React somehow violates one of Facebook 's patents, you're in trouble. The original version of React would be protected by the additional patent grant, but your fork wouldn't be._

As nkohari goes on to state (which I agree with), they're probably just trying to protect their patent portfolio and participate in OSS.

I, however, don't want to both even being anywhere close to an idea of compromising F/OSS, as there are lots and lots of options out there that are well within and have no additional legalese that could be misconstrued (whether positively or negatively).

</comment></comment></comment><comment><USER16> Does the revocation part apply to the entire license (code + patents) or just the patent clause? If you don't care about FB's patents, can you just treat React as being BSD licensed?

<comment><USER9> I think the implication is that Facebook might hold patents on part of React and you will lose the right to use work based on those patents (i.e. React and/or your code based on it) if you try to knock down any of Facebook's patents.

</comment></comment><comment><USER17> What happens if you stick with 0.11.2 - the last version that was released under the Apache 2.0 license? This specific license should allow you to make modifications like bug fixes or some other incremental improvements.

The existence of prior art is also interesting especially in the case with Immutable.js which is basically a JavaScript implementation of common data structures and algorithms.

</comment><comment><USER4> That's not a patent grant by any reasonable interpretation. You're still liable for any infringement, after all, the patent grant automatically is revoked if they accuse you of patent infringement and you disagree - regardless of whether the patent in question originally was granted.

Exactly which scenarios is this supposed to cover?

<comment><USER18> It doesn't seem quite that bad. If Facebook sues you over something related to React, you're allowed to argue that the suit is invalid because of the patent license grant. What you're not allowed to do is argue that the patent itself is wholly invalid.

Even so, the idea that you could sign away your rights to make a particular kind of argument in court sure seems like a crazy and nefarious way of hacking the legal system. But what do I know.

<comment><USER4> If you argue that "any right in any patent claim of Facebook is invalid or unenforceable" you lose the patent grant. So you can indeed argue the (patent) suit is invalid or unenforceable, and you'll probably win, but by doing so you'll lose that right in the future: you'd need to immediately stop using react.

In effect, the license _is_ revocable by facebook, without cause. In short; it's not a patent grant; it's merely an agreement not to sue over _past_ infringements.

<comment><USER8> That's incorrect. It's only revocable if you make a claim. As the person said, if the only claim you make is that you are using the patent grant itself, your rights are not revoked.

It's overly broad, but your interpretation isn't correct.

</comment></comment></comment></comment><comment><USER19> This seems like a paradox. Facebook grants us unlimited rights to the software, unless we assert that we have unlimited rights to the software?

</comment><comment><USER20> What does the whole thing say? In English?

</comment><comment><AUTHOR> The title was rewritten, specifically the concerning part is the 2nd paragraph. Apparently Facebook reserves the right to revoke anyone's license using React if FB feels there is a patent infringement.

Edit: I'm wrong. [https://twitter.com/SlexAxton/status/562318547674619904](https://twitter.com/SlexAxton/status/562318547674619904)

<comment><USER21> That's not what it says.

      The license granted hereunder will terminate, automatically and without notice,
      for anyone that makes any claim (including by filing any lawsuit, assertion or
      other action) alleging (a) direct, indirect, or contributory infringement or
      inducement to infringe any patent: (i) by Facebook or any of its subsidiaries or
      affiliates, whether or not such claim is related to the Software,

So if you're a patent troll, you don't get to use React.

<comment><USER22> This is actual fairly common nowadays. And rightfully so, if you sue us for patent infringement....no you can not use our OPEN SOURCE stuff anymore. Seems pretty fair to me.

<comment><USER23> Seems to me the more problematic part is clause b:

 _The license granted hereunder will terminate, automatically and without notice, for anyone that makes any claim (including by filing any lawsuit, assertion or other action) alleging ... (b) that any right in any patent claim of Facebook is invalid or unenforceable._

Doesn't that mean that if Facebook sues you for patent infringement in something else you did (possibly unrelated to React) and you disagree, they can take away your licence to use React? In other words, Facebook could arbitrarily shut down your use of React by filing a spurious patent lawsuit against you. And if your use of React was critical to your business and Facebook didn't like the look of your business, that could be dangerous.

<comment><USER5> In fact, it seems that they don't even need to sue you -- they could just "assert" that you violated their patents.</comment></comment></comment></comment></comment>


Detailed PHP vs. ASP.net - Windows vs Linux? - <AUTHOR> http://misfitgeek.com/blog/aspnet/php-versus-asp-net-ndash-windows-versus-linux-ndash-who-rsquo-s-the-fastest/

<comment><USER2> Maybe the title should read: Microsoft Worker finds ASP.net is faster than PHP. Likely true but means nothing to me as it is far from an independent survey.</comment>


Google gets behind 'hella' campaign - <AUTHOR> http://www.sfgate.com/cgi-bin/blogs/techchron/detail?entry_id=64240

<comment><AUTHOR> For consistency with the current SI units, the prefix should probably be derived from "nona" which is Latin for nine. 10^27 = (10^3)^9.

10^24 = (10^3)^8 is yotta which is derived from "y" + "octo". They added the "y" to prevent the "o" from being mistaken as 0.

<http://lamar.colostate.edu/~hillger/si-prefixes.html>

<http://members.optus.net/alexey/prefSI.xhtml>

<http://www.wordinfo.info/words/index/info/view_unit/1409/>

</comment><comment><USER2> I can't, in good conscience, support this. When someone utters the words "hella cool", I have to be physically restrained from punching the offender in the neck.

<comment><USER3> What if they said "wicked cool" instead?

<comment><USER4> were talking about NorCal, not the UK.

<comment><USER5> "Wicked" is a Boston expression as far as I was aware.

<comment><USER4> Perhaps it is, I only remember it from growing up in London (alongside with massive overuse of "brilliant")

</comment></comment></comment></comment></comment><comment><USER6> I guess I didn't really garner any intellectual pleasure from this article. Feels more like something i'd see over at digg. Marginally entertaining at best.

Not really what I'm used to seeing on HN.

</comment><comment><USER7> As a Berkeley grad from SoCal, who attended Berkeley during the rise of hella, I am very much against this. I did everything in my power to stop the spread of hella, but clearly I failed.

<comment><USER8> As a Berkeley grad from NorCal, your attempts were futile. ;)

Plus, hella was not nearly as annoying as SoCal kids who complained how bad Mexican food is NorCal. All they would do is go on and on about putting french fries in their burritos.

<comment><USER7> But we right -- the mexican food IS worse. It's made up for by the Thai and Indian food, so it's all good.</comment></comment></comment>


Ask HN: How do you identify programmers who finish things? - <AUTHOR> In the context of hiring. Assume you have the chance to work with them on a few short term projects, but not long enough that the newness wears off.<p>Alternatively: How do you identify programmers addicted to novelty/who have a hard time internalizing business goals? <comment><USER2> My experience is that "finishers" frequently have a hard time in their careers because they do the "80% of the work that gets 20% of the results." and aren't perceived to be productive as the "programmers addicted to novelty/who have a hard time internalizing business goals."

<comment><USER3> You can do the 20% that gets the 80% of the results, though, and still ship. I'd consider that "finishing". Whereas a lot of devs get distracted even before that point and _don't_ ship.

<comment><USER2> Shipping means getting to 100% of what is necessary. So if are getting 80% of the results and shipping, that means that somebody else is doing 80% of the work.

Let's put it this way. If Christopher Columbus was looking for the new world and he got 80% of the results the reward would be a watery grave.

<comment><USER3> Except he did get 80% of the results. His stated goal was to make it to India. He did the important part, which was to prove that you could cross the ocean to reach another land mass.

The other 80% of the work was sailing around the southern end of South America, and building the Panama Canal so you could actually reach Asia by sailing west from Europe.

</comment></comment></comment></comment><comment><USER4> Even if those are short term projects you should look for a few things:

0- They should care about getting projects done, in reasonable time and without the need for subsequent reworking. Regardless of what the project actually is.

1- They should have internalized that nothing is ever finished. After the "building" phase comes the "maintenance" phase and they should be aware of this and behave like they know that maintenance for every line they write will be one of their long term task (see #5, #6).

2- Willingness to do the least pleasurable and more mundane parts of the project (possibly trying to reduce the time needed through automation if it makes sense), that are necessary to finish it.

3- They should be able to always keep a clear idea of the current status of the project, what they still need to do and what is still missing. They shouldn't have a tendency to throw away time to look into unnecessary and irrelevant details (i.e. technical rabbit holes).

4- They shouldn't lose interest or lose direction when something doesn't go according to plan. In these cases the plan should be re-evaluated and a new plan defined.

5- They shouldn't be too hasty with what they have to do just to get it over with quickly and jump to the next thing.

6- Understand that there is a difference between a project that needs to be completed and used in production and something you can experiment on and has no hard requirements related to when it will be ready and how reliable it will be.

</comment><comment><USER5> I don't think question A: How do you identify programmers who finish things? and question B: How do you identify programmers addicted to novelty/who have a hard time internalizing business goals? are equivalent or even similar.

Many developers are "addicted to novelty" because that is how the industry works. You don't get very far in the game by only knowing VB.NET and C++. To get a high salary you need to change jobs many times. To change jobs many times you need to have knowledge about the latest and greatest (AJAX, jQuery, MongoDB, TypeScript, Android, iOS, Rust, Go, Scrum, TDD, whatever...).

<comment><AUTHOR> They may not be equivalent, but for the purposes of this situation they are both worthy of identifying.

You're justifying the game plan of a certain type of programmer. I'm asking how to find someone who's professional toolbox includes the ability to knuckle down and manage the boring shit until it's done, over a project-level time period.

In this instance I don't want to hire an alphabet soup programmer who will move on to the next company as fast as the next resume update allows. If it's a side effect of kicking ass on a variety of projects, broad experience is a nice-to-have, but I need someone who will hang around long enough to internalize and take responsibility for business goals.

</comment></comment><comment><USER6> By 'finish things' I am going to assume that you mean they complete their work in a manner such that a coworker can use it without revision. This takes practice and training.

If you have limited time with the person (say, 3 weekly meetings over a period of 3 weeks) then I would watch for the following...

a) did they do what they said they would do (in the previous meeting) -OR- are they asking for clarification on what was discussed in the previous meeting?

b) can you or another dev build and run their code from whatever they last checked in/committed -OR- do you get build errors, compile errors, or runtime errors?

c) do they ask questions and help you clarify what needs to be done?

Regarding the addicted to novelty, not internalizing business goals question, I see that as more of a management issue than an individual trait. They may be too busy learning a new technology to care about the business goals, in that case the engineering manager needs to make sure they are on track (and that the new technology is relevant to the work being done). Once they become proficient then they can easily internalize the business goals, but they will need repeated exposure.

</comment><comment><USER7> Also adding in this response. > In this instance I don't want to hire an alphabet soup programmer who will move on to the next company as fast as the next resume update allows. If it's a side effect of kicking ass on a variety of projects, broad experience is a nice-to-have, but I need someone who will hang around long enough to internalize and take responsibility for business goals.

I'll start with dev incentives and why it explains their behavior. Now one thing that is fairly common is internal raises are relatively much smaller when compared to market for jr. devs than sr. ones. This incentivizes most young/jr devs to switch jobs fairly often to get market pay raises(which are very substantial for young devs and usually much larger than internal raises, I quadrupled my salary in my first 3 years of working). To do this they need to build up technical expertise as quickly as possible because this is what the next job is looking for. And there really isn't any incentive to build up domain knowledge that will become useless the first time they switch jobs.

Senior/older devs for instance are usually on the opposite end where the pay raise for switching jobs is relative lower compared to internal raises. They are usually at a point where job security is more important because they fear being a hard to hire greybeard. They're already close to maxing out as a single contributor(and a lot of time if they wanted to move to management/tech leadership they would have already). And for them to pursue their goals of job security and maxing out pay by being valuable to one company as opposed to many companies they'll pursue and acquire lots of domain knowledge.

So I think you have two options. Hire an expensive older/sr. dev and convince them that the company would be a great place to spend the next 5-10 years. Or hire a younger dev and explain how it would be very beneficial for them to make a career there, talk about how much your company values domain knowledge and internalizing business goals. And then explain that you are willing to pay very handsomely for that and match what are the absolutely nuts pay bumps that go to jr. devs when they switch jobs.

*Note I'm generalizing like crazy and I don't think this is true in every case. I've met plenty of older devs who were fascinated by new tech and had little interest in domain knowledge and vice versa. But I think as a general rule it holds, and it holds for myself. When I first started I always got really excited about new tech and building out things the "right" way. Now 10 years in meeting business deadlines/goals and learning deep domain knowledge so the application really shines excites me more than learning Yet Another Framework.

<comment><AUTHOR> Not sure I agree with the hire a younger dev and start persuading option. In my managerial experience, it's almost impossible to change someone's core personality or their value system, either through words or through money. And even a temporary change takes monumental effort on my part. Better to find someone who's natural mode of operation fits this type of role.

I feel like the personal value system and personality are what drive long term performance. You can explain and cajole all you want, but inevitably a few months later the core personality starts coming back through. Establishing a culture around certain things will go a long way toward making those behaviors expected, but finding a person who has the appropriate temperament and skill level for a the types of activity needed for a role is so incredibly easier than doing everything the hard way.

Maybe I've been spoiled... I have a few engineers in other roles who do naturally get things done and don't need constant hand-holding to get through anything boring, but so far the programmers mostly just want to work on whatever they think is cool and turn into procrastination machines if I push too hard or give them an uncool task. I do have one older guy who I can leave to his own devices and the result will mostly make sense and happen without lots of my time pushing to finish. The other engineering roles with 'finishers' are mostly young guys, though, so I'm still hopeful any age could be this type.

<comment><USER7> I think you're right that changing a person is hard. But I disagree that changing a person's behavior by changing their incentives is nearly as difficult.</comment></comment></comment>


Venezuela: Without Liberals, There Is No Liberalism - <AUTHOR> https://econjwatch.org/File+download/887/FariaFilardoSept2015.pdf?mimetype=pdf <comment><USER2> It is unclear to me what exact definition of "liberalism" they are using, but it sure isn't the current American one. One of the clearest examples in the text:

"Teaching at the university level is highly influenced by Marxist thinking and, in the case of economics, Keynesianism. Liberal alternatives are marginal, at best relegated to elective courses. Private and public universities seemingly differ in their approaches, but both ignore liberalism. Private universities are Keynesian, while government universities are mainly socialist."

Marxism, Keynesianism, and socialism would be liberal/progressive positions in the US.

Given the odds of the submitter of a story to HN being an American-style liberal... are you sure this says what you think it says, jeffreyrogers? This sounds like classical liberalism, which is now definitely conservative by US terminology, although it may have other spins on the ideas too.</comment>


Show HN: Duit, simple money tracker built with Go and Mithril.js - <AUTHOR> https://github.com/RadhiFadlillah/duit/ <comment><USER2> +1 for an awesome readme.

It would be great if more projects would add screenshots and succinct overview as you have done.</comment>


The Pentagon tries to win hearts and minds in Silicon Valley - <AUTHOR> https://publicintegrity.org/national-security/the-pentagon-tries-to-win-hearts-and-minds-in-silicon-valley/ <comment><USER2> even wired was brought (and bought) into fixing the "strategic narrative" ?</comment>


The Coming Post-Platform Era - <AUTHOR> http://pwpwp.blogspot.com/2010/02/coming-post-platform-era.html <comment><USER2> The problem with this (and I think the difficulty in understanding what the author is trying to get across) is 'who owns the data?', or 'who owns the initiating platform. Sure all platforms can communicate with the others, which is more akin to e-mail, but you still have a platform and a suppliers.

You wouldn't want a platformless environment where anything you post is spread anywhere and everywhere without your knowledge. You kinda have that with twitter now, but you can still delete your posts.

<comment><USER3> Isn't this kind of what xmpp or pubsubhubbub are about, decoupling the data created by a user from a service?

I think it would be interesting to see this combined with the personel web server concept seen in Opera Unite, to serve an xmpp feed from an individual user (and device) that is ingested by multiple services. This would give users control over their data (they have a local copy) while allowing it to exist in the cloud as well. The user would also be able to publish "deletes" to tell services indexing her stream to remove a previously published message.

Having control of ones data is extremely important however. A push for existing web services to have an open intermediary data is essential, users should be able to backup or migrate their data with ease.

</comment></comment><comment><USER4> what?

<comment><USER5> It's what keeps social media consultants employed. Mostly nonsensical jargon.

</comment></comment><comment><USER6> "Instead, the web will be a meta-platform of content flows, unencumbered by actual applications."... that's what she said.</comment>


Mechanical Turk Officially Closed Outside of the US - <AUTHOR> http://www.webpronews.com/blogtalk/2007/09/13/mechanical-turk-officially-closed-outside-of-the-us <comment><USER2> The title is misleading. MTurk requestors must have a US address. MTurk workers can be outside the US.

</comment><comment><USER3> What a bunch of fools. But they have a pretty tight patent so it will be hard to operate in the same space as them.

<comment><USER4> eh...but their base claim is "comprising", but it's a business method and i fell asleep about half way through. i'm no patent attorney though. what makes you say it's tight?

patent: <http://tinyurl.com/2lomvg>

1\. A method for a computer system to use human assistance in performing tasks, the method comprising: automatically and under control of a first computer system, causing a task to be performed by, identifying a first and a second subtask of the task, wherein the first subtask involves speech recognition; retrieving information about past quality of results of one or more humans when previously performing subtasks other than the first subtask; using the retrieved past quality information to facilitate performance of the first subtask by one or more humans, the facilitating of the performance of the first subtask including, identifying one or more required capabilities for performance of the first subtask; dispatching the first subtask to a remote second computer system of a first human for performance by the first human, the first human identified as being one of one or more humans who have capabilities that satisfy the required capabilities for the first subtask, the retrieved past quality information including past quality information for the first human when previously performing multiple subtasks other than the first subtask, the past quality information for the first human being at least part of the capabilities of the first human that satisfy the required capabilities for the first subtask; and receiving a first result from the first human via the second computer system, the first result generated by performance of the first subtask by the first human; providing payment to the first human for the performance of the first subtask; and generating a result for the task based at least in part on the first result.

</comment></comment><comment><USER5> Am I the only one who thought of the chess automaton and was disappointed?</comment>


RSTTA: A Rosetta Stone for WebDev Tools - <AUTHOR> http://rstta.com

<comment><USER2> Neat start, but only 3 languages makes it a little wanting and the title of rosetta stone a bit of an overstatement.

<comment><AUTHOR> Agreed, just a start but with 3 popular language ecosystems.

Thought I could get away with 3 because the actual Rosetta Stone only had 3: Egyptian hieroglyphs, Egyptian Demotic, and Ancient Greek. :-)

Possible next additions: Clojure, Haskell, etc.</comment></comment>


Fixing my accents comprehension problem - <AUTHOR> https://medium.com/@vojto/fixing-my-accents-comprehension-problem-adda9a52a8ae#.ahimelp5m <comment><USER2> BEIJING: China will double the number of AIDS patients it treats with traditional Chinese medicine (TCM), officials have said, part of a broader push to increase the use of the ancient practice in the country's medical system.

The promotion of TCM is part of a five-year plan from the State Council, China's cabinet, to tackle HIV/AIDS.

"The number of people living with AIDS who are treated with traditional Chinese medicine should be twice what it was in 2015," the State Council said on its website Sunday (Feb 5).

The plan outlined collaboration between traditional Chinese medicine departments and national health and family planning commissions "to find a therapeutic regimen which combines traditional Chinese medicine (TCM) and Western medicines".

The TCM push aligns with a recent effort by the government to make the practice a priority for both development and publicity.

TCM, dating back thousands of years, treats ailments using herbal mixtures and physical therapies such as acupuncture and cupping.

The science behind such remedies has long been questioned. Last month medical researchers disputed a study claiming that acupuncture could cure babies of colic.

In late December the Chinese legislature passed its first TCM law, which will allow practitioners to be licensed and make it easier for them to open clinics.

There are about 450,000 TCM practitioners across the country, according to the State Council Information Office.

The government sees the practice as a cost-saving alternative to modern healthcare.

The new initiative to tackle HIV/AIDS will aim to reduce "AIDS-related homosexual behavior" by at least 10 per cent and mother-to-children transmission rates to less than four per cent.

In a 2015 report China told the UN that it had 501,000 cases of HIV/AIDS as of the end of 2014.

\- AFP

</comment><comment><USER2> WASHINGTON — President Trump's executive order on immigration has prompted a constitutional showdown that could leave a mark on the law for generations and seems likely to end in a landmark Supreme Court decision.

A ruling by the court on Mr. Trump's travel ban on seven predominantly Muslim countries could help answer some crucial legal questions: How much independent constitutional authority does the president have over immigration, and how much power has Congress given him?

The likely answer to both questions: a lot. But other parts of the Constitution may temper or defeat that power. Among them are the due process and equal protection clauses and the First Amendment's ban on government establishment of religion.

Here is a look at the leading arguments in the case. What have the judges said?

Many trial judges around the country have blocked aspects of Mr. Trump's executive order. But none have issued an order as broad as the one by Judge James Robart, a federal judge in Seattle, who blocked the key parts of the executive order, which had suspended travel from the seven countries and limited the nation's refugee program. Continue reading the main story Related Coverage

        Trump Clashes Early With Courts, Portending Years of Legal Battles FEB. 5, 2017
        Lifting of Travel Ban Sets Off Rush to Reach U.S. FEB. 5, 2017
        Where Trump's Travel Ban Stands FEB. 5, 2017
        Trump Protesters Borrow From Tea Party to Put Pressure on Lawmakers FEB. 5, 2017

Advertisement Continue reading the main story

The case is now before the United States Court of Appeals for the Ninth Circuit, in San Francisco. That court on Saturday declined to issue an immediate stay of Judge Robart's order, but it indicated that it would weigh in soon after additional briefs were filed, with the last one due Monday afternoon. After it rules, an appeal to the Supreme Court seems likely.

Judge Robart's brisk ruling contained almost no reasoning. By contrast, Judge Nathaniel M. Gorton, of the Federal District Court in Boston, issued a 21-page decision on Friday refusing to block the program and discussing the legal arguments in detail.

Judge Gorton also sketched out the broader picture.

"The rich immigrant history of the United States has long been a source of strength and pride in this country," Judge Gorton wrote. "Conversely, the public interest in safety and security in this ever-more dangerous world is strong as well." The balance, he wrote, tipped in favor of Mr. Trump. How broad is the president's constitutional power?

Article II of the Constitution confers authority on the president, the Supreme Court has said, to conduct foreign affairs and address immigration.

In their brief to Judge Robart, lawyers for Washington State, one of the two plaintiffs, along with the State of Minnesota, said there were constitutional checks on these powers. "While courts generally give more latitude to the political branches in the immigration context, this does not mean that the political branches can act with impunity," the brief said.

"Federal courts," the brief said, "have no more sacred role than protecting marginalized groups against irrational, discriminatory conduct."

In the Ninth Circuit, the Trump administration said judges were ill-equipped to decide cases involving national security. "Unlike the president," the administration's brief said, "courts do not have access to classified information about the threat posed by terrorist organizations operating in particular nations, the efforts of those organizations to infiltrate the United States, or gaps in the vetting process."

Noah G. Purcell, the solicitor general of Washington State, appeared to concede in court that there were areas in which Mr. Trump was entitled to act. But he asked the court to protect people whose lives had been changed by Mr. Trump's order in a flash.

"The focus of our claim," he said, "is on people who have been here and have, overnight, lost the right to travel, lost the right to visit their families, lost the right to go perform research, lost the right to go speak at conferences around the world. And also people who had lived here for a long time and happened to be overseas at the time of this order, which came with no warning whatsoever, and suddenly lost the right to return to the United States."

</comment><comment><USER3> [https://www.reddit.com/r/sb51live2017tv/](https://www.reddit.com/r/sb51live2017tv/)

<comment><USER2> HOUSTON -- As soon as New England Patriots kicker Stephen Gostkowski was satisfied with his Lombardi trophy selfie, the plan was to pass the 22-inch, 7-pound sterling silver bauble into the waiting arms of special teams captain Matthew Slater. By then, though, the trophy had already made two full circuits of the crowded, joyous locker room inside NRG Stadium. It had been smooched, hugged, pumped, cradled and caressed by nearly every team member. ("Chest hair or no chest hair?" Gostkowski had laughed as he fiddled with his shirt collar before posing with Lombardi.) The once-shimmering world championship prize was looking a little, well, "nasty" is what lineman Marcus Cannon shouted when he saw the splotchy, smudgy condition of the trophy headed for the beloved and respected Slater, a six-time Pro Bowl pick. "Naw, naw, naw, this won't do for Slate," yelled Cannon. "Give me that thing."

        "Chest hair or no chest hair?"
        Patriots kicker Stephen Gostkowski, while fiddling with his shirt collar before posing for a Lombardi selfie

Cannon, a West Texas native and TCU grad who made the key block on James White's 2-yard TD run in overtime to cap one the greatest comebacks in sports history, wasn't finished cleaning things up for the Patriots on this historic night. He intercepted the trophy before it reached Slater and grabbed a big blue bath towel with one hand and a fistful of Purell from a wall dispenser with the other. After a flurry of buffing and wiping, he delivered a proper Lombardi trophy, shiny and spotless, to his esteemed teammate.

It made sense. After five Super Bowl titles since the 2001 season, the Patriots have more recent experience than anyone in the care and handling of the most prized possession in sports.

The trophy, named in honor of the five-time NFL champion coach Vince Lombardi and built from a design sketched on a cocktail napkin during a 1966 lunch between Pete Rozelle and Tiffany & Co.'s Oscar Reidner, made its first appearance inside NRG Stadium with 1:57 left and while the whole world thought it was still going to Atlanta. A white-gloved rep from Tiffany & Co., which handcrafts the Italian silver in a four-month process each year, took the trophy out of its familiar robin egg blue velvet bag at the 10-yard line near the Falcons bench. There, he held it through the end of regulation and into overtime as the Patriots completed their stunning comeback.

Editor's Picks

        Behind the scenes of Super Bowl 51

        The Patriots' historic comeback against the Falcons was seen across the globe by hundreds of millions. We take you behind the scenes of Super Bowl LI.
        Super Bowl LI in pictures

        Behind Super Bowl MVP Tom Brady and New England's offense, the Patriots made a historic comeback to secure their fifth Vince Lombardi Trophy for Brady and coach Bill Belichick.

At game's end, tradition dictates that the Tiffany rep hand the trophy off to a former NFL great for a 30-yard walk through a gantlet of players and to the foot of the postgame awards podium. This year, the task fell to former Patriot and three-time Super Bowl champ Willie McGinest. In what has to be the most NFL moment ever, a worldwide audience of millions saw and heard McGinest take the shiny, clean artwork from a guy wearing white gloves and stomp through a mob of sweaty, bloodied, exhausted players who slobbered all over it while McGinest screamed at them to "kiss this [expletive] trophy, man, [expletive] kiss it!"

</comment></comment><comment><USER3> Researchers believe they have the answer to why some people find certain noises, such as another person's chewing, so annoying.

A study from Newcastle University, published in the Current Biology journal (view study here), suggests this phenomena is linked to a key part of the brain that processes emotion.

Experts say for those diagnosed with misophonia certain sounds can lead to severe anxiety and even panic attacks.

From the study:

Overall, our data show that for misophonics, trigger sounds cause hyperactivity of AIC and an abnormal functional connectivity of this region with medial frontal, medial parietal, and temporal regions; that there is abnormal myelination in medial frontal cortex that shows abnormal functional connectivity to AIC; and that the aberrant neural response mediates the emotional coloring and physiological arousal that accompany misophonic experiences. Together, our data suggest that abnormal salience attributed to otherwise innocuous sounds, coupled with atypical perception of internal body states, underlies misophonia. With the available data, it is not possible to decide whether misophonia is a cause or consequence of atypical interoception, and further work is needed to delineate the relation between the two.

Misophonia does not feature in any neurological or psychiatric classification of disorders; sufferers do not report it for fear of the stigma that this might cause, and clinicians are commonly unaware of the disorder. This study defines a clear phenotype based on changes in behavior, autonomic responses, and brain activity and structure that will guide ongoing efforts to classify and treat this pernicious disorder.

Read more here.

Neuroscientists reviewed patients' brain scans and found the emotions area of the brain goes into overdrive when the patient is listening to chewing, eating, drinking and even sounds such as heavy breathing.

Sign up for ClickOnDetroit breaking news alerts and email newsletters Search for a specific story:

Search ...

Copyright 2017 by WDIV ClickOnDetroit - All rights reserved.

Sponsored Content This Game Will Eat Days of Your Life This Game Will Eat Days of Your Life Vikings A Modern House Made of Redwood, Glass and Stone A Modern House Made of Redwood, Glass and Stone Mansion Global RANKED: NBA All-Time Rosters RANKED: NBA All-Time Rosters PointAfter The Worst Romantic Comedies of All Time The Worst Romantic Comedies of All Time PrettyFamous

Sponsored Content Ranking the Worst Rookie QB Busts Since 1980 PointAfter | By Graphiq NBA Player J.J. Redick Lands a Brooklyn Penthouse for $4.25 Million Mansion Global The Best No. 1 Picks in MLB Draft History PointAfter | By Graphiq 30 Baby Names That Were Briefly Trendy MooseRoots | By Graphiq Derek Jeter's Charming West Village Rental Can be Yours for $7.5 million Mansion Global More From ClickonDetroit 70-Year-Old Heads Back to College And Pledges Fraternity Teen Livestreams Suicide Weeks After 12-Year-Old Did the Same Neglected Girl, 9, Hadn't Showered in a Week, Was Only Fed SpaghettiOs: Cops Live With Kellyanne: It's the 1998 Stand-Up Comedy Video Trump's Top Aide Doesn't Want You to See Boyfriend Dresses in Panther Mascot Suit at 'School Assembly' to Propose to School Principal Recommended by

</comment><comment><USER3> How much power has Congress given the president?

On Friday, defending Mr. Trump's executive order in a Seattle courtroom, Michelle Bennett, a Justice Department lawyer, cited Youngstown Sheet & Tube Co. v. Sawyer, a 1952 decision in which the Supreme Court rejected President Harry S. Truman's assertion that he had the authority to seize steel mills during the Korean War.

The most famous part of the decision is a concurrence from Justice Robert H. Jackson, which set out a framework for considering clashes between presidential power and congressional authority. The president has the most power when he acts with congressional authorization, Justice Jackson said, and an intermediate amount when Congress is silent. The president's power is at its "lowest ebb," Justice Jackson wrote, when Congress has forbidden a particular action.

Truman's actions fell into the third category, Justice Jackson wrote. Ms. Bennett, by contrast, said Mr. Trump's order was in the first category.

"Here we have the president acting pursuant to power that Congress gave him, which means, under the Youngstown steel seizure case, he's acting at the apex of his power," she said.

A key part of immigration law does give the president broad power. It says, "Whenever the president finds that the entry of any aliens or of any class of aliens into the United States would be detrimental to the interests of the United States, he may by proclamation, and for such period as he shall deem necessary, suspend the entry of all aliens or any class of aliens as immigrants or nonimmigrants, or impose on the entry of aliens any restrictions he may deem to be appropriate."

But another part of the law forbids discrimination "because of the person's race, sex, nationality, place of birth or place of residence," but only "in the issuance of an immigrant visa." The Trump administration argues that the power to bar entry, the subject of the first law, is broader than the limits on issuing visas. What about arguments based on religious discrimination?

Lawyers for Washington State have said that the executive order violates the First Amendment's prohibition against government establishment of religion because its provisions on the refugee program favor minority religions. "President Trump and his advisers have made clear that the very purpose of this order is to tilt the scales in favor of Christian refugees at the expense of Muslims," they wrote in their brief to Judge Robart.

The Trump administration urged the Ninth Circuit to reject arguments based on religious discrimination, even though Mr. Trump has said he meant to favor Christian refugees. Judicial consideration of the president's motives, the brief said, would violate the separation of powers.

"The more searching inquiry envisioned by the states would create substantial separation-of-powers problems, by permitting probing of the president's subjective motive in issuing the order," the brief said. Could the case founder on the issue of injury?

The states challenging the order face the initial hurdle of demonstrating that they have suffered the sort of direct and concrete harm that gives them standing to sue. Judge Robart ruled that they did, relying on a decision from the federal appeals court in New Orleans, which said Texas could sue to challenge President Barack Obama's plan to defer the deportation of millions of unauthorized immigrants and allow them to work.

"The executive order adversely affects the states' residents in areas of employment, education, business, family relations and freedom to travel," Judge Robart wrote. He said the states had been hurt because the order affected their public universities and their tax bases.

In its Ninth Circuit brief, the Trump administration called the states' asserted injuries "attenuated and speculative" and did not address the Texas decision. In court in Seattle on Friday, Ms. Bennett said the government disagreed with the Texas ruling.

Follow Adam Liptak on Twitter @adamliptak.

A version of this article appears in print on February 6, 2017, on Page A11 of the New York edition with the headline: The President Has Much Power Over Immigration, but How Much?. Order Reprints| Today's Paper|Subscribe Continue reading the main story Related Coverage

        Trump Clashes Early With Courts, Portending Years of Legal Battles FEB. 5, 2017
        Lifting of Travel Ban Sets Off Rush to Reach U.S. FEB. 5, 2017
        Where Trump's Travel Ban Stands FEB. 5, 2017
        Trump Protesters Borrow From Tea Party to Put Pressure on Lawmakers FEB. 5, 2017</comment>


How Tunnel Boring Works (2011) [video] - <AUTHOR> https://www.youtube.com/watch?v=qx_EjMlLgqY <comment><USER2> Absolutely fascinating! After watching all that CGI, I wanted to see an actual machine in action and came across this video:

[https://www.youtube.com/watch?v=qlrZ3G0tAzY](https://www.youtube.com/watch?v=qlrZ3G0tAzY)

it's by the company that makes these massive boring machines.

</comment><comment><USER3> Interesting stuff.

One thing I was hoping the video would touch on is how do they know exactly where they are and how they are oriented while boring large tunnels? If you're digging 30km of tunnel, being off by half a degree in any direction would be disasterous, no?

<comment><USER4> Survey control is transferred down the tunnel from surface. How it gets down there depends on whether you launch from a shaft or pit, but once survey control is established in the hole it is straight forward to locate the TBM using standard survey techniques (eg - survey prisms). The TBM has a reflector that can be shot with a survey instrument to accurately place the machine head and the TBM itself is equipped with gyroscopic or accelerometers for backup control if survey is lost during mining.

</comment></comment><comment><USER5> Why is the video unlisted? Very rare for a 1M+ view video to be unlisted.

</comment><comment><AUTHOR> The pipeline video is fascinating, too. They coat the exterior of the pipes with a lubricant, and install new segments at the back end, thrusting the entire line forward each time. [https://www.herrenknecht.com/en/products/productdetail/85/?w...](https://www.herrenknecht.com/en/products/productdetail/85/?wvideo=y66srmx0vz)

</comment><comment><USER6> I wonder how software is engineered for those machines. Languages used, etc.

</comment><comment><USER7> Not sure if I missed it but how do you bring this kind of equipment underground in the first place? Do they have to dig a huge hole using traditional methods first?

<comment><USER8> Yes, they have to build a shaft. One way is a diaphragm wall. A ring of smaller shafts are dug and filled with concrete to form a circular wall. Then the centre of the ring can be excavated with the retaining wall already in place.

</comment></comment><comment><USER9> How do they steer? By using perfectly machined custom segments that turn them in the correct direction?

<comment><USER10> They touch on that briefly at 8:10 in the video. The rings aren't perfect cylinders; the top and bottom are angled so that each ring is slightly longer on one side. (I.e. They resemble radial slices of a torus.) If each ring is rotated 180 degrees relative to its neighbors, the lengths average out and the tunnel goes straight. If the orientation is kept roughly constant, the tunnel curves in the direction of the short side of the cylinders.

<comment><USER4> This is partially correct. This is how the curve is maintained with the concrete liner. Steering is also accomplished via articulation jacks at the head and controlling differential stroke of the main jacks.

It should be noted that only certain types of precast segments work the way that is stated with all rings having an angled face - other types have specific concrete segments for each particular curve with the rest of the rings being planar. This approach has gone away in the industry with advances in software.

<comment><USER11> Does the machine itself deform while going through a curve?

<comment><USER4> Technically the answer is yes (any stress state results in deformation) but for all practical purposes the answer is no. The TBM does not bend to any degree worth talking about and is considered rigid.

This leads to curves in tunnels generally having a very large radius because you have to build the leading part of the curve inside a rigid cylinder. The tunnel is slightly smaller than the TBM and so some curvature can be tolerated.

<comment><USER11> So I suppose the curvature is much smaller than what a typical train can tolerate.

</comment></comment></comment></comment></comment></comment><comment><USER12> This is how it worked before Elon Musk reinvented it by buying a sewer boring machine and running it dozens of feet under a desert.

<comment><USER13> I'm not sure if your comment is meant to be snarky. But seriously... What is Elon doing differently?

<comment><USER14> As I understand it (and I am certainly not qualified in any regard with respect to this type of engineering) he is aiming to increase velocity by decreasing diameter (ie couple meter diameter vs 10m diameter) and then also looking at how they can 'juice' the machine using engineering magic

<comment><USER15> The London Tube was built with a 3.58m diameter. Jubilee Line Extension had a 4.4m diameter, Crossrail will have 6.2m diameter. So the diameter reduction is not that much smaller, and it certainly isn't the smallest attempted.

The original Tube tunnels actually have several problems - the tight diameter means that air conditioning is basically an impossibility (no room for the equipment in vehicles and nowhere to output the heat), and it makes maintenance and emergency evacuation annoying and hazardous if you can't have a walkway for people to walk alongside the trackway.

<comment><USER14> most of the original tunneling was done by hand, so i'm not sure if it's directly comparable. There was an interesting article regarding the heat transfer to the surrounding ground recently - [0] that i'm sure you'll find interesting if you haven't seen it before!

<comment><USER15> Jubilee Line Extension was the '90s and Crossrail is opening soon, so those definitely weren't by hand!

I have seen articles about it, as well as a recent proposal to use that heat for heating homes. But it just illustrates the (many) potential pitfalls of small diameter tunnels, because you really don't want your tunnel to become an oven after a few decades of heat generated from braking.

Other pitfalls include:

You need space for adequate ventilation. This is less of a problem with electric propulsion, but you do still need it for emergencies (like a fire). And then there's the issue of noise; if your vehicle is too small compared to the tunnel, then the air being forced out once you get to a larger space (like the outside or a station) creates loud noise. Japanese high speed rail trains have much stranger aerodynamic shapes than European ones because they built smaller diameter tunnels and this ended up causing noise issues at higher speeds.</comment></comment></comment></comment></comment></comment>


Launch HN: Datafold (YC S20) – Diff Tool for SQL Databases - <AUTHOR> Hi HN! My name is Gleb. I'm here with my co-founder Alex to tell you
about our company Datafold (https://datafold.com). Datafold lets you diff large datasets for fast and powerful regression testing. We support databases such as PostgreSQL, Snowflake, BigQuery, and Redshift.
One of the biggest pain points in developing ETL pipelines –
chains of jobs that move, clean, merge and aggregate analytical data –
has been regression testing: verifying how a change in source code (mostly, SQL) affects the produced data.
Early in my career, as an on-call data engineer at Lyft, I accidentally introduced a breaking code change while attempting to ship a hotfix at 4AM to a SQL job that computed tables for core business analytics. A seemingly small change in filtering logic ended up corrupting data for all downstream pipelines and breaking dashboards for the entire company. Apart from being a silly mistake, this highlighted the lack of proper tooling for testing changes. If there had been a way to quickly compare the data computed by production code vs. the hotfix branch, I would have immediately spotted the alarming divergence and avoided merging the breaking change.
Without a diffing tool, the typical options for regression testing are: (1) Data "unit tests" (e.g. check primary key uniqueness, ensure values are within interval, etc.) – these are helpful, but costly
investment. Frameworks such as dbt make it easier, but it's often still prohibitively hard to verify all assumptions in a large table. (2) Write custom SQL queries to compare data produced by the prod and dev versions of the source code (e.g. compare counts, match primary keys). This can easily take up 100+ lines of SQL and hours of unsatisfying work, which no one really wants to do. (3) "Fuck It, Ship It" is always an option but too risky nowadays as analytical data not only powers dashboards but also production ML models.
As this problem is common in data engineering, some large organizations have built and open-sourced their solutions – for example, BigDiffy by Spotify. However, most of these tools are CLI-based and produce results in a plain-text format which is hard to comprehend when you are dealing with complex data.
To fit existing workflows of our users, we've built a web interface with interactive charts showing both diff summary statistics (e.g. % of different values by column) and value-level side-by-side comparison (git diff style). But since the mission of the tool is to save engineers as much time as possible, we also opened an API for automation through Airflow or other orchestrators, and built a Github workflow that runs diff on every pull request with changes to ETL code. Since billion-row-scale datasets are not uncommon nowadays, there is an optional sampling feature that helps keep compute costs low and get results within a few minutes no matter how large the dataset is.
We've found Datafold to be a good fit for the following workflows: (1) Developing data transformations – before an ETL job is shipped to production, it undergoes multiple iterations. Often it's important to see how data changes between every iteration, and particularly useful if you have 1M+ rows and 100+ columns where "SELECT *" becomes useless. (2) Code review & testing: large organizations have hundreds of people committing to ETL codebases. Understanding the impact of even a modest SQL diff is daunting. Datafold can produce a data diff for every commit in minutes so changes are well understood. (3) Data transfer validation: moving large volumes of data between databases is error-prone, especially if done via change data capture (CDC): a single lost event can affect the resulting dataset in a way that is tricky to debug. We allow comparing datasets across different databases, e.g. PostgreSQL & Snowflake.
We've set up a sandbox at https://app.datafold.com/hackernews so you can see how diffing works. Shoot us an email (firstname.lastname@example.com) to set up a trial and use it with your own data.
We are passionate about improving tooling for data engineers and would love to hear about your experience with developing data pipelines and ensuring data quality. Also, if you think that dataset diffing can be helpful in other domains, we are very curious to learn from you! <comment><USER2> I was curious about pricing, but I see it's "call me pricing" with buttons to schedule a demo, so at least I can see this is squarely aimed at the enterprise. If I'm being honest, I don't like seeing "call me pricing" on HN; there are no rules against it, but it just doesn't feel right on HN.

Are you able to say anything about pricing here?

<comment><AUTHOR> Definitely a fair point!

The primary reason we haven't provided pricing is that we have just launched and wanted to collect more data points before setting making the pricing public.

Our current offering is:

1) Free for diffing datasets < 1M rows 2) $90 / mo / user for diffing datasets of unlimited size 3) Cross-database diff, on-prem (AWS/GCP/data center) deploy, Single-sign-on are in custom-priced enterprise bucket.

We would love to hear your thoughts on this.

<comment><USER3> I have a quick suggestion for you: two options you can mitigate this "issue".

Option 1: make it free, up to a certain dataset size. You can harvest interested leads like the gentleman above.

Option 2: (if you don't want to deal with a huge volume) offer it for $50 one- time fee, up to X size, for Y months (e.g. $50, up to 1 GB, valid for 3 months). Nice way to filter qualified leads.

There are variations from the two options above, but I think you can easily get the general idea.

Thoughts?

<comment><AUTHOR> Thank you for the suggestion!

We're leaning towards Option 1: free diffing for datasets < 1M rows. Option 2 seems a bit tricker since we are in a way creating a new tool category and it can be harder to convince someone to pay before they try and understand the value (unlike, say, a BI tool – everyone knows they need some kind).

</comment></comment></comment></comment><comment><USER4> I see a lot of these things and I don't understand them. I've done too much ETL so I'm not naive. Now either 1) people are making a mountain out of a molehill (not saying that's happening here, but in other cases I think so) 2) there's something my experience of ETL hasn't taught me or 3) these tools are specialised for niches. This one talks about 'large datasets' but I don't know how large that is.

Some questions then

> Often it's important to see how data changes between every iteration, and > particularly useful if you have 1M+ rows and 100+ columns where "SELECT *" > becomes useless.

select is fine for diffing. You just do an either-way except , something like

      (
      select f1, f2, f3 ... f100
      from t1
      except
      select f1, f2, f3 ... f100
      from t2
      )
      union
      (
      select f1, f2, f3 ... f100
      from t2
      except
      select f1, f2, f3 ... f100
      from t1
      )

used this and it's fine on many rows (millions is fine but I do recommend and index and a DB with a halfway decent optimiser).

> (2)

Interesting. OK.

> (3) Data transfer validation: moving large volumes of data between databases > is error-prone

Really? I never had a problem. What is 'large'? what problems have you seen? There are easy solutions with checksums, error correction (comes free with networks) or round-tripping, is that a problem?

Edit, just done that with mssql tables, 8 cols, 38 bytes per row, ~776,000 rows (identical but for one row), diff as above takes 2 seconds without an index (with PK it takes 5 seconds. Sigh. Well done MS). The single row discrepancy shows up fine. Totally trivial to extend it to 100 columns (did that too in previous job).

<comment><USER5> Data diffs are easy when the tables are already exactly the same. The union/except query you wrote returns no results.

It's much harder when many things are wrong at once, and you have to debug: \- Do schemas match? (same column names, column order, data types) \- Are pk's unique? \- Do pk's intersect? (pk's exclusive to one table?) \- For intersecting pk's, how is data different? (string truncation, timestamp truncation, rounding, upper vs lower case, empty string vs null, etc)

<comment><USER4> > Data diffs are easy when the tables are already exactly the same. The > union/except query you wrote returns no results.

It won't if the tables are identical. And if the tables differ, it a) tells you and b) gives you a very good idea where to look.

In all honesty, none of things are a problem.

> It's much harder when many things are wrong at once

Then you find and fix each in turn. BTDT.

> Do schemas match? (same column names, column order, data types)

Well, if they don't and you're responsible you've really messed up, but that does happen (cough). Just do a similar diff via information_schema to show you schema discrepancies.

> Do pk's intersect?

this is just a subset of full-table diffing. Use my except/union but just on the PK fields (consult infoschema for the columns which are the PKs. If it has to be done for many tables, build the union/except dynamically then execute the string to allow for different tables with different columns names - I've done this too. Annoying but when done once you can just reuse it).

> For intersecting pk's, how is data different? (string truncation, timestamp > truncation, rounding, upper vs lower case, empty string vs null, etc)

The aforementioned comparison using infoschema views will tell you where the schemas differs. Comparing the contents show you where the contents differ.

All this is annoying but of _very_ little work compared to a whole ETL process, and - emphatically - is reusable on the next job.

<comment><USER5> > Well, if they don't and you're responsible you've really messed up, but that > does happen (cough). Just do a similar diff via information_schema to show > you schema discrepancies.

Analysts with 1-2 years experience in SQL are often writing ETL. Responsible Data Engineers can't be there every time they deploy to production.

> All this is annoying but of very little work compared to a whole ETL > process, and - emphatically - is reusable on the next job.

Yes, you can write your own tools for SQL QA. Making a standard tool that the entire company can and _does_ use is another story.

<comment><USER6> > Analysts with 1-2 years experience in SQL are often writing ETL

This is the essence of Big Data.

</comment></comment></comment></comment></comment><comment><USER7> Hey! Looks great! Is there an example of the Github integration - how does it looks like?

I'm one of the developers and maintainer of the DVC project and we recently released CML.dev- which integrates with Github and can be used to run some checks on data as well. But in our case it's about analyzing files more or less. I'm curious how does that integration look like in your case.

<comment><AUTHOR> Thanks! We are going to add an example to our website and perhaps publish it as a Github App.

The flow is: For every whitelisted (say, SQL) file changed by a pull request, our bot compares the production data (produced by the master branch) vs. the data output of the new branch and pastes the diff as a comment to the PR. Some YAML configuration is needed to, for example, adjust it to your repo file structure.

<comment><USER7> Thanks! And how and where does setup happens which database to use to run the query for the specific SQL file? Also if it's part of some pipeline will it have to run the whole pipeline from the very beginning?

<comment><AUTHOR> So there are two main workflows:

1) Run diff as part of CI (on code change), that can be done via Github workflow (or other CI). The Github workflow takes in a YAML config file, and also is customizable so you can fit it according your needs. For example, one of our customers names SQL files after tables they populate, so when the bot sees a changed SQL file, it knows what table to compare it to.

2) If you are looking to run a diff within the data pipeline itself (e.g. in Airflow), you can wrap Datafold API in an Airflow operator to integrate it as a task in your DAG.

I hope that I covered your questions but please let me know if you are interested in specific use cases!

</comment></comment></comment></comment><comment><USER8> It is always good to see new approaches to testing but I don't see how this one is going to work. I've worked at multiple database companies. Diff'ing data is one of the weakest and most cumbersome ways to verify correctness.

Diffs are relatively slow, when they fail you get a blizzard of errors, and the oracles (i.e. the "good" output) have to be updated constantly as the product changes. Plus I don't see how this helps with schema migration or performance issues, which are major problems in data management. And don't get me started on handling things like dates, which change constantly, hence break diffs.

If you really care about correctness it's better to use approaches like having focused test cases that check specific predicates on data. They can run blindingly fast and give you actionable data about regressions. They're also a pain to code but are most productive in the long run.

<comment><AUTHOR> Thank you for sharing!

I assume we are talking about analytical, not transactional data:

> Diff'ing data is one of the weakest and most cumbersome ways to verify > correctness.

It depends on the use case: if the goal is to assess the impact of a change in source code on the resulting dataset produced (extremely common in ETL dev workflow in my experience), then isn't diff the natural solution? Of course, it depends on how the results are presented. A row-by-row output for a billion-row dataset is useless. That's why we provide diff stats across columns/rows and data distribution comparisons while allowing the user to see value-level diff if needed.

> Diffs are relatively slow

In general – yes, that's why we've implemented configurable sampling. In the majority of cases, developer is looking to assess the magnitude of difference and certain patterns, for which you don't need a large sample size. Our customers typically use ~1/10000 of the target table row count as a sample size.

> when they fail you get a blizzard of errors We try to fail gracefully :)

> I don't see how this helps with schema migration or performance issues.

For schema migration, you can verify whether anything has changed in your dataset _besides_ the intended schema changes (which certainly happened on my watch).

> or performance issues

We certainly don't claim to solve all DBA issues with diff, but here's an actual real example from our customer: they are optimizing their ETL jobs in BigQuery to lower GCP bill by reducing query runtime. After refactoring the code, they diff production vs. new code output to ensure that the data produced hasn't been affected.

> If you really care about correctness it's better to use approaches like > having focused test cases that check specific predicates on data.

Possibly, but

> They're also a pain to code

...which is often a prohibitive pain point if you have 300+ analytical datasets with 50+ columns each (a common layout for companies of 250+).

And another problem: the more test cases, the more failures on every run, and unlike app code unit testing, you can't expect the cases to stay relevant since the data is changing constantly, so those "unit testing" test suites require constant maintenance, and as soon as you stop actualizing them, their value drops to 0.

I think that diffing and "unit testing" are complimentary approaches and neither one is a panacea. So my recommendation has been to use both: 1) Specific test cases to validate the most important assumptions on data 2) Diff tool for regression testing.

<comment><USER8> I'm unconvinced your approach works beyond a narrow range of use cases. The weakness is the "is this a problem" issue. You have a diff. Is it really significant? If it's significant, how did it arise? You can spend an inordinate amount of time answering those two questions, and you may have to do it again with every run. Diffs are cheap to implement but costly to use over time. That inversion of costs means users may end up bogged down maintaining the existing mechanism and unable to invest in other approaches.

If I were going after the same problem I would try to do a couple of things.

1\. Reframe the QA problem to make it smaller. Reducing the number and size of pipelines is a good start. That has a bunch of knock-on benefits beyond correctness.

2\. Look at data cleaning technologies. QA on datasets is a variation on this problem. For example if you can develop predicates that check for common safety conditions on data like detecting bad addresses or SSANs you give users immediately usable quality information. There's a lot more you can do here.

Assuming you are working on this project, I wish you good luck. You can contact me at rhodges at altinity dot com if you want to discuss further. I've been dealing with QA problems on data for a long time.

<comment><USER8> p.s., to expand on #2 if you can "discover" useful safety conditions on data you change the economics of testing, much as #1 does.

</comment></comment></comment></comment><comment> stephane-klein At the moment I use [https://github.com/djrobstep/migra](https://github.com/djrobstep/migra) to make PostgreSQL diff.

<comment><USER9> I also use migra - and love it. Migra only checks your schemas though, not your data.

<comment><USER3> From what I recall you are correct. But I also vaguely remember that there was another little open source tool that integrated with Migra to check data as well? Or was it a custom-made script?

Sorry, can't find it, I wish my comment could be more useful.

</comment></comment></comment><comment><USER10> 1) Will it allow me to diff AWS RDS Aurora/MySQL serverA database schemaA against AWS RDS Aurora/MySQL serverB database schemaB ? 2) Are there APIs to initiate and view/parse these diffs that you generate or is it all through UI?

<comment><AUTHOR> Yes, you can diff to physically distinct databases. MySQL support is in progress.

There is a REST API that accepts the same parameters as in the UI and returns the diff results in JSON + optionally, materializes the tables in your DB with row-level differences.

</comment></comment><comment><USER11> To whome it may concern, we have written a paper on the subject (I'm not affiliated with datafold): [https://www.researchgate.net/publication/323563341_A_Time- co...](https://www.researchgate.net/publication/323563341_A_Time- constrained_Algorithm_for_Integration_Testing_in_a_Data_Warehouse_Environment)

The paper describes the original algorithm with examples.

</comment><comment><USER12> Much needed! Analysts struggle with this all the time - trying to explain why an analysis is off and having to manually debug every column in a new database

</comment><comment><USER13> In case you're unaware, your logo looks unfortunately a bit uncannily similar to that of [https://www.sigmacomputing.com/](https://www.sigmacomputing.com/) given you are both in a similar broader product category! I actually thought your logo throughout the site was actually a reference to integration with sigma at first.

<comment><AUTHOR> Thank you for pointing it out, I've never noticed but now that you brought it up will do a few more iterations on our design to avoid the collision.

</comment></comment><comment><USER14> I got the chance to play with Datafold and I would have __loved __to have had it when I was working for Facebook on data pipelines.

</comment><comment><USER15> I've recently discovered, and highly recommend Daff [0]. It's an open source tool that can diff CSVs and SQLlite database tables. The tabular diff format is fantastic.

[0]: [https://github.com/paulfitz/daff](https://github.com/paulfitz/daff)

</comment><comment><USER16> Hey Gleb, congrats on the launch. This is an interesting tool.

Myself being a founder of a product in the space of tools for data-driven companies, I wanted to ask -

Is you product aimed entirely at data engineers? The description seemed very technical and a problem that mostly very large companies would have. Did I understand correctly?

<comment><AUTHOR> Thank you and that's a great question.

It largely depends on how critical the data pipelines are to the business. We've seen big companies suffer the most from issues with their data quality. At the same time, we've had early-stage teams using our diff tool because their core product depends on the data they ingest from third-party vendors and performing regression testing of that data and the transformation code has been taking them a lot of time.

The product in its current form could be useful to anyone developing data transformations (which is what data engineers typically do full time), and we are working on expanding it to help data consumers (analysts, PMs, etc.) have higher confidence in the quality of datasets and metrics they rely on.

</comment></comment><comment><USER17> Can this be used locally for datasets with non-transferable PII? Thinking about this for non-profit work.

<comment><AUTHOR> Yes, we have on-prem AWS/GCP and K8S-based deployments available, would be very happy to support your work, just shoot me an email email@example.com.

</comment></comment><comment><USER18> Great tool! I am only interested in running such a tool locally (on-prem). This avoids security/privacy issues and data transfer time/cost issues.

A good model for me would be 30 day free license to get it integrated into our worklows.

<comment><AUTHOR> Totally fair! We offer free trials for on-prem. Please ping us at firstname.lastname@example.com to set it up.

</comment></comment><comment><USER19> For Cassandra: [https://github.com/apache/cassandra- diff](https://github.com/apache/cassandra-diff)

(Designed for correctness testing at petabyte scale)

</comment><comment><USER20> Can you import/work with .bcp files? (Microsoft SQL server bulk export files). For example, diffing 2 bcp files, even if you need to import and set them up as databases again.

<comment><AUTHOR> If you can materialize those .bcp files as tables, then yes, absolutely.

</comment></comment><comment><USER21> How does it work behind the scene? Is it simply sample a portion of the data then do the diff? What if I need 100% accuracy?

<comment><AUTHOR> If diffing datasets within the same physical database, generate SQL, execute in the database, analyze and render results.

If diffing datasets across physically different databases, e.g. PostgreSQL <> Snowflake or 2 distinct MySQL servers, pull data in our engine from both sources, diff, and show results.

Sampling is optional but helpful to keep compute costs low for large Mill/Bill/Trill-row datasets.

</comment></comment><comment><USER22> Congrats on the launch.

How does it compare to enterprise ETL tools like Informatica, Talend - is it not possible to do these within them ?

<comment><USER5> At best, Informatica could output the diff of two tables to another table. But the paradigm is very limiting. I'm not sure if you can dynamically change the mappings every time you feed it a new table (different primary keys, columns, data types), and you'd still have to browse the results in SQL.

<comment><USER22> Thank you, not to undermine the work done in Datafold. But visual schema-diff is a small improvement in giant of tools like informatica which can pretty much do anything in database space.

It can be well argued that YC is taking in companies without realising what exists out there in enterprise world. There are quite a few no-code/api- stiching tools which launched recently as well. And those at best resemble poor imitation of powerful BI tools that are existing out there since almost a decade.

<comment><AUTHOR> Those tools are definitely vastly powerful. Have you used either of them?

TBH, I haven't, but judging from our current post-Informatica users and by reading questions on Informatica/Talend official user forums, I concluded that the diffing problem (to be specific – not only schema, but data diffing) is not directly addressed by them (the answers are in the realm of "there is no diff feature but you can write SQL..."

In general, we see data stacks becoming increasingly modularized and tools more specialized. For example, there are at least 20x more teams using OSS like Airflow/Luigi/Dagster for managing their data pipelines (and 2-5 other tools for the rest of the workflow) than using end-to-end platforms that you mentioned. We see Datafold as a regression testing tool in a modular stack.

<comment><USER22> Thank you for your reply.

I have used Talend in great detail 3 years ago but I didn't have the usecase of schema diff at the time. But for data diff you can easily define workflow. And have to admit these workflows are crazy powerful and even can help the data fix with any transformation required (nocode or code)

However, Im seeing the usecase for a light weight tool with visual aspect. I like this. But will this problem be big enough for VC investment is the question ? I see schema diff can be just a plugin in one of the existing database tools. And if you are getting into data diff - you got to see what those tools do too.

<comment><AUTHOR> > But will this problem be big enough for VC investment is the question?

That's a great question. Thinking about where problems arise in data pipelines, there are fundamentally two moving pieces: 1) Your data – you're continuously getting new data without a real ability to enforce your assumptions on its schema or shape. 2) Your code for ingestion and transformation that needs to evolve with the business and to adapt to changes in other parts of the infra.

Datafold's Diff tool currently mostly addresses #2. It can add value to any company that runs ETL pipelines but most impactful at large data engineering teams (similar story to CI or automated testing tools).

Regarding #1, wouldn't it be useful if we tracked ALL your datasets across time and alerted you on anomalies in those datasets? And I am not talking about rigid "unit" tests e.g. X <= value < Y, but actual stats-based anomaly detection, akin to what Uber does: [https://eng.uber.com/monitoring-data- quality-at-scale/](https://eng.uber.com/monitoring-data-quality-at-scale/)

So, with diff, we already compute and store detailed statistical profiles on every column in the table. Next, we are going to track those profiles across time.

Diff is just the first tool we've built to get a wedge into the workflows of high-velocity data teams and start adding value, but it's just the beginning of a more comprehensive and, hopefully, valuable product we aspire to deliver.

<comment><USER22> Much appreciate your response

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> Will there be integration with ORACLE Dbs?

<comment><USER24> This

<comment><AUTHOR> Yes, we can add support for Oracle quite soon if you are interested.

</comment></comment></comment><comment><USER25> Very cool, nice work! SQL Server / Azure SQL support available or on the roadmap?

<comment><AUTHOR> Thank you!

We are [almost] DB-agnostic (as long as there is SQL interface and JOINs are supported) and have been adding support for particular DBs based on the interest from our users. SQL Server & Azure SQL areon the roadmap, please let us know if you have a use case in mind to pilot Datafold on them.

</comment></comment><comment><USER26> It is interesting to see Ant Design used other than my personal projects.

</comment><comment><USER27> Very cool! Does it work with MS SQL in our own DC?

<comment><AUTHOR> Support for MS SQL is in progress. We can deploy in your DC using with Docker or K8S.

</comment></comment><comment><USER28> love it, fucked up and made a startup with a solution so you'll never make that mistake again.</comment>