Ask HN: Data diff tool for tabular data? - <AUTHOR> Is there a tool you'd recommend to diff tabular data / TSV files?
Every day I have a snapshot of a database stored as TSV. Given two days of snapshots and that every item in the TSV has a unique ID, I'd like to get a list of which lines are new, changed, unchanged, and deleted between the two days. These lists will then be used to update an external service with an awkward database-like API.
I can kind of do this with normal diff but perhaps there's something better suited for the task? It reminds me of data joins in d3 but I'm not working with a UI or Javascript. <comment><USER2> For diff on whole rows, use you could use Linux' CLI tools, in particular join(1) which is very similar to SQL's [LEFT/RIGHT] JOIN clause. Another option is the combination of sort(1) and uniq(1); with proper choice of options, and, in some cases, repeating the file with old or new data, you can get:

      cat old.csv old.csv new.csv | sort | uniq -u # only the added/changed records
      cat old.csv new.csv new.csv | sort | uniq -u # only the ol d/deleted records
      cat old.csv new.csv | sort | uniq -d # only the unchanged records

<comment><USER3> Also I think sort has a -u flag.

<comment><USER4> If you use sort -u you don't have two repeat files (:

</comment></comment></comment><comment><USER5> Using the hammer already in my hand, I would write a small R script to do the joins.

Example script:

      library(tidyverse)
      old = read_tsv("path/to/old/db.tsv")
      new = read_tsv("path/to/new/db.tsv")
      deleted   = anti_join(old, new)
      new       = anti_join(new, old, by = "id")
      unchanged = semi_join(old, new)
      changed   = semi_join(new, old, by = "id") %>%
                    anti_join(unchanged)

</comment><comment><USER6> Beyond Compare has a tabular data mode, where you set the key columns and it presents the differences in a nice interface. Works well for me. [http://www.scootersoftware.com](http://www.scootersoftware.com)

</comment><comment><USER7> Not quite sure if it fits your needs exactly, but I've been pleasantly surprised with visidata's [0] capabilities when munging through csvs.

You can totally join two 'sheets' by primary key and save and replay that workflow, I think you may have to write a python function to iterate over the sets of columns to look for differences. Not sure if there's a cleaner way

[0] [https://github.com/saulpw/visidata](https://github.com/saulpw/visidata)

<comment><USER8> There is a --diff option that works if the rows and columns are both exactly aligned. But this post makes me think about adding functionality to match by row key and column name, possibly via the "diff" join (current diff join is more like an "xor"). If you have ideas in this regard I'd love to hear them!

</comment></comment><comment><USER9> Python and pandas: [https://stackoverflow.com/questions/17095101/outputting- diff...](https://stackoverflow.com/questions/17095101/outputting-difference- in-two-pandas-dataframes-side-by-side-highlighting-the-d)

</comment><comment><USER10> I quite like daff (have used it for diffing Excel sheets):

[https://github.com/paulfitz/daff](https://github.com/paulfitz/daff)

</comment><comment><USER11> My team and I had the exact use case.. here is what we came up with: [https://github.com/chop-dbhi/diff-table#diff-table](https://github.com/chop- dbhi/diff-table#diff-table)

Supports delimited files and Postgres tables. Outputs a large JSON-based summary of record diffs or it emits an event stream as changes are found (and thus does not hold the summary in memory).

Prebuilt binaries: [https://github.com/chop-dbhi/diff- table/releases](https://github.com/chop-dbhi/diff-table/releases)

 __Edit __: add link to releases and description

</comment><comment><USER12> I helped make a tool called DataCompareR, which does exactly this, in R. It gives you diffs for column names, variable types, rows and a few other things. I hope you find it useful!

[https://github.com/capitalone/datacomparer](https://github.com/capitalone/datacomparer)

</comment><comment><USER13> Take a look at `lnav` [1]. It's ideal for chaining w/ other cli tools, has an embedded sqlite engine, and makes it trivial to parse and query any kind of reasonably structured data.

[1][https://lnav.org](https://lnav.org)

<comment><AUTHOR> I've used lnav for log navigation before, but how is it relevant here? Checked the features page but nothing jumped out at me...

</comment></comment><comment><USER14> You could import your daily snapshot in a new SQLite DB everyday, then use the sqldiff utility [[https://www.sqlite.org/sqldiff.html](https://www.sqlite.org/sqldiff.html)].

Another option that might fit would be a sort(1) pass on your TSV files (unless you already ORDER BY on the primary key), then using Awk to detect differences, and immediately print out a list of API commands to update your external DB. Awk conveniently handles tabular data, just set the field- separator (FS) to \t in this case (default it splits on all whitespace).

</comment><comment><USER15> Personally, I would do it in the database. Most established database products have some sort of "import" or "load" tool. Some even have "update_insert" types of ingestion mechanisms depending on your requirement.

So, keep a table of the latest "data" you have. Then import into a separate table. After that, your requirements are solvable with a few SQL joins. The upside is that you also have a local copy of the data in a table, which is always a plus when diagnosing/supporting your deployed product.

The data sets you then need:

        Deleted: Orig_table left outer join Import_Table with a null filter on Import_table.
        New: Import_Table left outer join Orig_table with a null filter on Orig_table.
        Changed: Inner join with not-equal check on all columns, joined by OR.

And each update to the external API also updates your local version of the data. I.e. Orig_table.

I know it sounds a bit verbose and maybe not an "all in one" solution that you're looking for. But sometimes, you just need to solve the problem.

</comment><comment><USER16> git diff --word-diff-regex="[^\t]" one.tsv two.tsv

For example:

      $ echo "1\t2\t3" > one.tsv
      $ echo "1\tx\t3" > two.tsv
      $ git diff --word-diff-regex="[^\t]" one.tsv two.tsv
      diff --git a/one.tsv b/two.tsv
      index d43017e..e2d5856 100644
      --- a/one.tsv
      +++ b/two.tsv
      @@ -1 +1 @@
      1       [-2-]{+x+}      3

</comment><comment><USER17> Python. Symmetric difference.

        >>> set(open('a.tsv')) ^ set(open('b.tsv'))

</comment><comment><USER4> [https://github.com/MouslihAbdelhakim/Quick](https://github.com/MouslihAbdelhakim/Quick)

I made Quick to solve your exact problem! we use it to test our batch processes that either generate flat files or write data into database tables.

please contact me if you need help with the tool, i know that the docs are not as complete as i would like them to be.

here is the wiki: [https://github.com/MouslihAbdelhakim/Quick/wiki](https://github.com/MouslihAbdelhakim/Quick/wiki)

here is a repo of some examples: [https://github.com/MouslihAbdelhakim/Quick- examples](https://github.com/MouslihAbdelhakim/Quick-examples)

</comment><comment><USER18> Got Excel? [https://superuser.com/questions/420635/how-do-i-join-two- wor...](https://superuser.com/questions/420635/how-do-i-join-two-worksheets- in-excel-as-i-would-in-sql)

<comment><USER19> whats that

<comment><USER20> It's like Google Sheets, with some weird retro desktop theme as default /s

<comment><USER18> Arf!

</comment></comment></comment></comment><comment><USER21> Are you not allowed to do this in the database itself? That would be much simpler.

</comment><comment><USER22> [https://github.com/attic-labs/noms](https://github.com/attic-labs/noms)

Import two snapshots of data like this:

      csv-import --dest-type="map:0" dump0.csv /noms/mydb::data
      csv-import --dest-type="map:0" dump1.csv /noms/mydb::data

Show difference between the two snapshots like so:

      noms log --max-lines=-1 /noms/mydb::data

See also the `noms diff` subcommand.

</comment><comment><USER23> Hi! I'm working on [https://stuckd.com](https://stuckd.com) which compares folders, so it's not exactly what you need, but I could add this as a new feature, and I think should be quite easy to implement. I'm still in a early stage, but if you don't mind being a beta tester, it's something you may be interested to try?

</comment><comment> GuB-42 Beyond Compare has a mode for tabular data.

It is not free but I find it much better than the usual visual diff free tools like kdiff3 or meld.

</comment><comment><USER24> If you store your data in a slowly changing dimensional format you will be able to create a table where you can query the data as it was at any point of time.

The easiest way to do this is to use an ETL tool as hand coding this can be a bit hairy. You can use an open source etl tool like Talend for this as it has inbuilt Postgres or MySQL scd modules.

</comment><comment><USER25> I use xlcompare for comparing Excel. It creates very good diff reports of Excel features (including formatting, formulas and Macros). I often use it for simple CSV compares. Be warned, it does crash occasionally

[https://www.xlcompare.com](https://www.xlcompare.com)

</comment><comment><USER26> Augeas might be helpful

[http://augeas.net/docs/references/lenses/files/csv- aug.html](http://augeas.net/docs/references/lenses/files/csv-aug.html)

But will require a bit of coding

</comment><comment><USER27> [https://github.com/paulfitz/daff](https://github.com/paulfitz/daff) plus version control

</comment><comment><USER28> Crazy talk here, but do you have control of the database? Can you get history added to it?

</comment><comment><USER29> Maybe rsync could also work well for you?</comment>


How Diane Greene Transformed Google's Cloud - <AUTHOR> http://www.businessinsider.com/how-diane-greene-transformed-googles-cloud-2016-6 <comment><USER2> Interesting article. For anyone working with Google's Cloud Platform, there are definitely changes happening both in product and culture.

As a company that's used all the major clouds and vendors out there (AWS, Azure, Softlayer, GCP, Internap), Google's has been the best so far with simple fast tech that works. They are clearly behind in features and breadth of offering but I agree with Diane's statements that the underlying tech is just better. Quizley has a good article recently that goes into more detail which is what we saw as well. [1]

I do wish the GCP team was easier to reach though. They have lots of engineers and PM's who are active on social media and discussions forums and such but it feels like things are too "informal" in conversations right now. It's nice to see everyone passionate about helping on their own time but AWS has them beat on the vast amount of help and resources they pour into getting clients of all sizes onboarded quickly. I'm guessing that will change soon enough with all the hiring at Google.

1\. [https://quizlet.com/blog/whats-the-best-cloud-probably- gcp](https://quizlet.com/blog/whats-the-best-cloud-probably-gcp)

<comment><USER3> Whom would you like to reach? (sounds like PMs and/or engineers). I'd consider it okay that you ping me and I route your issues to the relevant folks (there's always going to be routing), but for existing customers most questions naturally should go through support (either directly or via requests from Sales when a customer contacts them). Are you an active customer who is having trouble reaching us? Or a prospective customer who doesn't find the current channels working? (I ask to know what to work on, not to accuse).

Disclosure: I work on Google Cloud.

<comment><USER4> Hi,

sorry if it's the wrong place to report an issue that I faced recently with GCP.

I had reached the limit of max projects one can have in their dashboard. I deleted some projects and when I tried to create a new project, I was getting the same message .

I had to wait for a week to create a new project ( the time window in which user can stop project deletion )

this really prevented me from trying some new services in spite of GCP giving generous $300 credit.

<comment><USER3> Interesting, I thought there was a "Seriously, I know what I'm doing, delete right now".

I've never run into this, and apparently that's because the issue is that we have a limit of 10 _free trial_ projects (to prevent abuse). I'll look into the "Seriously, let me delete immediately" thing though.

</comment></comment></comment></comment><comment><USER5> I moved from AWS EC2 to Google Cloud a few days ago. Google really seems to have beaten AWS, at least in pricing and flexibilty. On AWS (Singapore region) a 2-vCPU, 7.5G RAM instance costs $143/month (not including IOPS and bandwidth costs), while a similar one on GC works out to about $56/month. That's a massive difference. In addition, GC allows me to customize cores and RAM flexibly to a point, which is important for me.

Also, AWS's reserved instances are somewhat of a nightmare. There are only certain upgrade paths you can take, and you're locked in to them for a year. And if you're not in the US, you can't even sell the instance.

<comment><USER6> Yeah, I've seen multiple folks putting off reserved instance commitments and continuing to pay on-demand costs since they don't want to get locked in for a year. Its a serious commitment for smaller companies.

The No-Upfront reservation options that AWS last year helped narrow the difference quite a bit - but Google automatic discounts for sustained use are so much better and less complicated for users.

<comment><USER3> And per-minute billing, and the ability to move between zones month to month ("Oh, Haswell zone just launched? I'll be moving there..."). It's night and day, and if I could, I'd happily take the other side of any RI deal ;).

Disclosure: I work on Compute Engine.

</comment></comment></comment><comment><USER7> I skimmed the article, and this seems to explain a few things for me. The image I have in my mind is that Google is Google, masters of the datacenter. If they can build the most reliable website on earth (e.g. Google's down? Internet must be down) then their cloud service must be stellar. And yet, having switched from AWS to Google a few months ago at work, our experiences have left me sorely disappointed. It's been a nightmare, honestly. Why such a difference between what I expected and what we experienced?

The article seems to allude that Google's cloud offerings are, like a lot of services they offer, just side projects for them. That would explain why a lot of their platform feels half-baked. And the article (clearly a PR piece) waffles on about how Diane Greene has brought in lots of sales and marketing muscle. I see that clearly in the large amount of PR articles about Google Cloud floating around (including on Hacker News) and the sheer quantity of Google employees on social media and on here. I don't see the same from AWS. At first I saw that as a good thing, but it seems like the same effort that goes into their marketing and customer engagement has not gone into the technical side of things, despite what the article may say.

Perhaps that's a necessity of business. Maybe Alphabet doesn't want to pour more resources into Google Cloud without seeing more revenue. So Diane Greene is beefing up the numbers so she can get the technical resources she needs allocated. Maybe. My personal opinion remains doubtful, given Google's overarching track record, but time will tell.

Google Cloud is by no means unusable. It's a nightmare to work with, yes, but it does work, and if you've got the engineers on staff to handle the extra workload, then it's fine. We're going to continue to put up with it, because Google's particular mix of offerings allows us to save money and at the end of the day that's the common tongue. I just wish they wouldn't make my life more difficult.

<comment><USER3> Sorry to hear about your troubles! Assuming you got hit by our painful network outage, we can only repeat: sorry, and we have taken serious action internally to avoid this again.

To explain the difference between your experience (outages taking you out) and "Google.com", I'd guess the difference is that "Google.com" is _massively_ distributed. Perhaps you were running in just one zone or region, or maybe even quite sophisticatedly running across two regions (say us-central1 and us- east1). For Google.com, we have 15 "major" datacenter locations ([https://www.google.com/about/datacenters/inside/locations/in...](https://www.google.com/about/datacenters/inside/locations/index.html)) which are approximately regions in Compute Engine / Cloud parlance.

To your other question though, Cloud is _not_ a side project. Google happens to be enormous, so even though we have thousands of folks across Technical Infrastructure (TI) working on Cloud, thousands divided by tens of thousands is still a "small" percentage (but TI is bigger than say YouTube or Android).

[Edit: And please reach out to support! Don't be silently unhappy, have someone call us up, even if it's just to strangle a PM about how difficult it is to use.]

Disclosure: I work on Compute Engine.

<comment><USER7> > To explain the difference between your experience (outages taking you out) > and "Google.com"

My comment was somewhat terse (to remain as brief as possible) so perhaps it wasn't clear. My logic was "Google.com is supremely reliable, which is a supremely difficult task, so Google must have really good engineering chops, so their products must be really well engineered." In other words, I was saying that I had in my head an image of Google being filled with great engineers building generally well engineered products. I wasn't commenting on reliability of service, which we haven't had any issues with that I can presently recall (we're only in us-central1-c).

> [Edit: And please reach out to support! Don't be silently unhappy, have > someone call us up, even if it's just to strangle a PM about how difficult > it is to use.]

Google provides no way to contact support without paying them for a support contract. It makes no sense for us to pay Google for the privilege of debugging their service/software. And I've crawled my way to support through sales before. For all my effort, and an inability to use the platform for a week, I was left with "Sorry, here's an issue tracker". The same kind of issue tracker that's filled with stagnant issues that are months or a year old.

The same argument could naively be made about AWS, which also charges for the privilege of reporting issues to them. But in all the years that I've used AWS I've only once needed to contact them with a problem. It was a billing issue, which they fixed, and then comp'd us with free service for the trouble. With Google Cloud we would need to diagnose, debug, document, and report an issue roughly every day of development.

</comment></comment></comment><comment><USER8> This is probably some kind of GCP PR's placed article, but regardless, Greene's impact on GCP has been incredible and highly visible.

Previously, the standard arguments that I've seen in sales conversations were that GCP wasn't enterprise ready or serious enough. All of that disappeared in the last 6 months.

Yes, GCP is still small compared to Google (or AWS) at $400M/yr revenue. Yes, AWS still "gets" enterprise better than GCP.

And yes, GCP is back in the cloud infrastructure game and will give AWS and Microsoft a run for their money.

<comment><USER9> I don't think we've discussed revenue anywhere, where did you get this number?

We'll certainly do our best!

Disclosure: I work at Google on Kubernetes.

<comment><USER8> Hi there,

This was my source: [http://www.recode.net/2015/10/22/11619964/google-has-its- hea...](http://www.recode.net/2015/10/22/11619964/google-has-its-head-in-the- cloud)

The # is from Oct 2015, so it's entirely possible that my number is very out of date. Couldn't figure out how to amend my original comment, so adding this one as a substitute.

</comment></comment></comment><comment><USER10> > _When pointing out that Microsoft also offers a computer vision API, > translation services, and APIs for Office 365, and that IBM also offers > weather data and language services, and so on, Greene 's got a comeback > ready.

"We have Chromebooks." _

Is she joking? I can't see how they make Google's cloud more attractive than others. Am I missing something here?

Edit: I also wanted to add that there wasn't much concrete information in this article at all. Sounds like a fluff piece/advertisement to me (IMO).

<comment><USER11> Like she said - Chromebooks are very secure and easy to administer. They're also the ideal choice for running Google Apps in the enterprise.

<comment><USER12> > Google Apps in the enterprise

Google Apps for business are so bad though, I genuinely don't know why a company would pick it over 365. I could maybe see an SMB, but 365 is so robust in its offerings and just so much better all around.

My former employer was on 365, my new employer is on Google Apps. It's like I've been thrown back into the stone age. (Went from a large corporation to a start-up.)

I feel like an SMB is not likely to do volume purchases of Chromebooks, but a large corporation is...which Google just fails at in their offering in the office space, imo.

<comment><USER13> > My former employer was on 365, my new employer is on Google Apps. It's like > I've been thrown back into the stone age.

Oddly enough, we had exactly the opposite experience when a previous employer went from 365 to Google. The Google apps just worked, while the Office 365 apps ranged from hilariously bad (Yammer & Lync) to not completely terrible (the web version of Word) to mostly decent (the web version of Excel).

But man, Yammer & Lync were like a joke after using Hangouts. A really bad joke, told by a sadist. A sadist who is shorting one's stock.

<comment><USER12> Yammer is useless, but I found Lync to be very reliable. Why use Hangouts if your company is using Slack for team channels, and it has group video chat? Of course Lync is now "Skype for Business" and I don't think its as good as Lync was tbh. I think Microsoft should buy Slack and toss it into its Office 365 offering, if you want my honest opinion.

The web version of Word and Excel are vastly superior to Google's docs and sheets, I don't know how you can argue otherwise. Though, I don't know why you'd use them when you get the full versions for free in your 365 (which totally demolish Google's offerings.) Docs hasn't really done anything since it first launched, while sheets does have decent integrations...but it's no Excel.

Now 365 has baked in project management, Delve, business insights etc. There is no comparison.

</comment></comment></comment></comment></comment><comment><USER14> Wow I did not know she gave her $150m on the sale of Bebop to charity. That is very very impressive and must be celebrated more.

<comment><USER15> Yes, but... At her tax bracket and future earning potential, the money is very likely to be worth _more_ as a donation than as income. Total tax rate would be above 50%, but as a donation you get a $1-for-$1 deduction against future income. So the choice is something like 'keep 40 cents on the dollar today, or negate 60 cents on the dollar in taxes tomorrow.'

Also, the money isn't really all gone to a 3rd party, it's in a foundation- type entity similar to how Zuck does it.

<comment><USER16> That doesn't make any sense. The deduction is in the same amount of the income she just received (I assume her basis (since she was a founder) was very low or near zero). It will not be available for "future income".

The foundation-type entity is correct though, she will still be overseeing how it's spent.

<comment><USER15> No, that's just it! You donate the appreciated security directly without exercising/selling them first, so there is no recognized income on the personal tax return. Plus you _still_ get to take the deduction on the fair market value of the security.

See: [http://www.fidelitycharitable.org/giving-strategies/tax- esta...](http://www.fidelitycharitable.org/giving-strategies/tax-estate- planning/appreciated-securities.shtml)

<comment><USER16> Ummm...the _whole_ point is that she _sold_ those shares to Google, donating large "paper profits" is a total other thing.

EDIT: Perhaps you're saying she donated her shares to the Donor Advised Fund first, and then the Fund sold the shares to Google? Possible I guess, but that's not what the SEC notice said.

[https://www.sec.gov/Archives/edgar/data/1652044/000090342316...](https://www.sec.gov/Archives/edgar/data/1652044/000090342316000722/xslFormDX01/primary_doc.xml)

"Diane Greene exchanged 7,244,150 shares of bebop stock for 200,729 shares of Alphabet Class C Capital Stock at $740.39 each in the Merger, plus cash for fractional shares. Ms. Greene intends to donate the shares exchanged to a donor advised fund."

<comment><USER15> Getting past my 'expertise' in this area, but she didn't personally sell her shares to Google. Her company was acquired by Google, and her shares in bebop were exchanged for shares of Alphabet. A merger/stock exchange is not a taxable event, she would keep her near-$0 basis in the now Alphabet Class C shares, and then donate the highly appreciated Alphabet shares to the Fund.

You have to structure the deals this way, otherwise you lose more than half of your company to the government. When marginal rates exceed 50% and you have dependable future earning potential, yes, giving away unrealized gains can be worth more than realizing them.

<comment><USER16> Yeah, but at that point, ABC shares are basically cash money--they may as well be Treasury Bonds given how liquid they are. She could easily sell half to cover the tax bill and keep the rest. So no, I don't see how "the the money is very likely to be worth _more_ as a donation than as income."

<comment><USER15> Liquidity of the shares is not in question. If the tax rate is over 50%, you have two choices; sell the shares and keep < 50 cents on the dollar by realizing the gain, or you can donate the shares to your Charitable Fund, and carry the charitable deduction against future income. The next year when you earn $XX millions, half of that is now tax free because the charitable deduction carry-forward. Dollars which otherwise would have been taxed at > 50 cents on the dollar. Selling the shares is worth < 50 cents on the dollar, but the charitable deduction is worth > 50 cents on the dollar because of the future tax liability that it eliminates. That is why I say it is "worth more as a donation".

By donating $150m to the Fund, you not only get the self-directed Fund with $150m dollars in it (which is a fun time in and of itself) but you also get $150m in charitable deductions. Each dollar of that deduction eliminates > 50 cents on the dollar of tax liability. Donating to the Fund effectively lets you have your cake and eat it too, again, with the caveat that you expect to have enough future income to use it all up, while remaining in the highest tax bracket.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> Diane Greene talking at Startup School in 2013, VMWare ~ [https://www.youtube.com/watch?v=zSEeFxq2X_c](https://www.youtube.com/watch?v=zSEeFxq2X_c)

</comment><comment><USER11> GCP will be adding 2 new regions by the end of 2016 and bringing an additional 10 regions online by the end of 2017. How does this compare to the scale of AWS?

<comment><USER3> AWS currently has:

\- three in the US (1 in Virginia, one in Oregon, one in Northern California)

\- two in Europe (Dublin and Frankfurt)

\- four in Asia (Singapore, Tokyo, Sydney, Seoul)

\- one in South America (São Paolo)

and both Montreal and India announced. So by part way through 2017, the "numbers" would be sort of comparable but the real factors are your latency requirements, data sovereignty (hello Germany!) and the features you need (and not all features are available in all regions!).

Disclaimer: I work on Compute Engine.

<comment><USER18> Also one additional region being added to the US this year, Ohio was announced last year.

</comment></comment></comment><comment><USER19> google cloud seems like a repackaging of internal google services which is fine but most companies don't want to migrate existing code to google's homegrown tech. the story for running postgres, oracle, mssql, spark, hadoop, kafka, etc on google cloud is vastly inferior to the same on aws where they either offer it as a managed service or someone else has done and documented the work required to do it

<comment><USER20> Managed Spark & Hadoop = Cloud Dataproc [https://cloud.google.com/dataproc/](https://cloud.google.com/dataproc/)

One-click deploys for Kafka and Postgres: [https://console.cloud.google.com/launcher/details/bitnami- la...](https://console.cloud.google.com/launcher/details/bitnami- launchpad/kafka), [https://console.cloud.google.com/launcher/details/bitnami- la...](https://console.cloud.google.com/launcher/details/bitnami- launchpad/postgresql)

<comment><USER21> To be fair there is a big difference between a button to deploy an image and something like RDS which manages that instance going forward.

</comment></comment></comment><comment><USER22> tl;dr - Greene convinced Google to get serious about providing cloud services to enterprise.

<comment><USER23> I know internet comments are pointless, but this isn't the truth. Google knew it had to take things seriously, and as a result hired her.

New hires, even at the exec level, usually don't have the social capital to make huge changes. So they usually tie in to an existing sentiment.

<comment><USER24> I think you'll see huge changes as a result ... and you also need to remember Diane was a Google board member for three years prior to taking this role.

</comment></comment></comment><comment><USER25> I use both AWS and GCP. GCP has better engineering but fails to offer a good business experience. And this aggressive GCP PR feels a bit desperate.

</comment><comment><USER26> The one thing I've found baffling about both AWS and GCE is the bandwidth pricing. At about 0.1$/GB, it's 10x more expensive than competitors like linode. Why is that?

Bandwidth cost is the primary reason I'm afraid of hosting side projects on AWS/GCE.

<comment><USER27> Because Linode and similar overcommit their bandwidth given that they're charging a package deal - most people won't actually use the full allocated bandwidth.

AWS/GCE charge for actual bandwidth usage, which means that everyone uses exactly what they pay for, and the folks who use less bandwidth don't subsidize the folks who use more.

</comment></comment><comment><USER28> How exactly did she "transform" it if they were number 3 before and are number 3 now?

<comment><USER28> Really? Down-voted for asking a legitimate question related to the article? Why? HN is becoming a bit silly with this isn't it?

<comment><USER2> I think it's because you severely limited the perception to a basic ranking. It's a massive undertaking to actually move up against AWS or Azure considering the incredible scale of these platforms so continuing to be #3 in customers is not really an accurate measure of progress.

The article is pretty clear on what's she's done so far. There's no denying that they have a lot of work to do but if you're actively using the platform then you'll notice that there has been a lot of change in just a few months with them, which should all help to increase that customer number.

<comment><USER29> Being #3 in a growing market is not a bad place to be.

</comment></comment></comment></comment><comment><USER30> > Google's competitive strength, Greene believes, is the breadth of the tech > it can offer an enterprise.

> Enterprise-app developers can tap into things like Maps, Google's computer- > vision engine (the tech that powers Google Photos), weather data, and > language/translation/speech recognition. They can build apps on top of > Google's Calendar, documents, spreadsheet and presentation apps.

> And, under Greene's new integrated organization, they can even tap into the > tech that powers Google's ads or YouTube, search, or its many other > services.

Probably a good time to reshare the famous Yegge rant about platforms: [https://plus.google.com/+RipRowan/posts/eVeouesvaVX](https://plus.google.com/+RipRowan/posts/eVeouesvaVX)

</comment><comment><USER31> Not to get into a technical discussion of the merits between the two. But, I don't get the excitement around Diane Greene? Wasn't she asleep at the wheel at VMWare when AWS/MS were building out their cloud offerings? Seems like a major blunder and sitting on your laurels.

<comment><USER32> > Wasn't she asleep at the wheel at VMWare when AWS/MS were building out their > cloud offerings?

She left VMware in 2008, that's only two years after AWS launched and two years before Microsoft Azure. Maybe with the benefit of hindsight "cloud" seems like an obvious next step but I imagine it wasn't as clear in 2008ish - even Amazon didn't move its retail operation to AWS until two years later[1].

[1] [http://www.slideshare.net/AmazonWebServices/2011-aws-tour- au...](http://www.slideshare.net/AmazonWebServices/2011-aws-tour-australia- closing-keynote-how-amazoncom-migrated-to-aws-by-jon-jenkins)

</comment></comment><comment><USER33> I'd like to play with google cloud using my personal details but here in the UK it says you need to be a business/commercial entity on the registration page with the following clause

> This service can only be used for business or commercial reasons. You are > responsible for assessing and reporting VAT.

whereas Amazon don't seem to stipulate that (at least not explcitly, there might be something in their t+c's) and they deal with all the VAT stuff.

Oh well.

</comment><comment><USER34> Someone from BI or anyone,please fix the """ text in the article.Its distracting considering nearly 30% of the article is quoted by Diane Greene.I guess I am nitpicking or is a fault with my browser?

<comment><USER35> Sorry if this is not much help, but I'm not seeing any instances of """ in Firefox or Vivaldi (Chromium-based) in Windows 7 Pro.

<comment><USER34> I just reopened the article on Firefox Aurora too.Its become worse now. I can see <p> and <img> tags too.Wow</comment></comment></comment>


The Circle of Fifths, Part One - <AUTHOR> http://www.coderedux.com/circle-of-fifths-part-one

<comment><USER2> The code he wrote for showing how to generate is pretty ugly and left me confused. I think this shows much better the relation between the 12 notes and a scale:

      NOTES = ['c', 'c#', 'd', 'd#', 'e', 'f', 'f#', 'g', 'g#', 'a', 'a#', 'b']
      # 2 is a whole step, 1 is a half step
      MAJOR_STEPS = [2,2,1,2,2,2,1]

      # returns the scale in the given key with the given steps
      def scale(key, steps)
        note = NOTES.index(key)
        step = 0
        notes = []
        while step < steps.length
            notes << NOTES[note % NOTES.length]
            note += steps[step]
            step += 1
        end
        notes
      end

      scale('d', MAJOR_STEPS) #=> ["d", "e", "f#", "g", "a", "b", "c#"]

I think this shows how a scale is just a list of steps applied to the notes from a certain key :)

<comment><USER3> Chris Ford gave an elegant explanation of scales at a Clojure talk in London: <http://skillsmatter.com/podcast/home/functional-composition>.

He starts with a basic sine wave and builds up abstractions until he has a Bach canon.

<comment><USER4> Only at the first third and already delighted. He's moving between music, physics, programming in such a simple and pragmatic way, thanks for the bottom-up dynamic evaluation demonstration. Many many thanks.

links aside, he uses (and is a project contributor) <http://overtone.github.com/>

ps: they also provide a nice emacs setup <https://github.com/overtone/emacs- live>

</comment></comment></comment><comment><USER5> Most musicians aren't aware of this, but an interesting corollary of the circle of fifths is the Pythagorean Comma: [http://en.wikipedia.org/wiki/Pythagorean_comma#Circle_of_fif...](http://en.wikipedia.org/wiki/Pythagorean_comma#Circle_of_fifths_and_enharmonic_change)

We generally treat Gb and F# as the same note, but if you actually count perfect Pythagorean fifths (frequency ratios of 3:2) in both directions from C, Gb and F# don't actually perfectly meet in the middle!

<comment><USER6> That's the cause of that famous piece being called the Wohl Temperierte Klavier, those little rounding errors add up, once you have settled on a system that works for everybody by convention Gb and F# have to be the same notes (since there is only the one key for them...).

<http://www.phy.mtu.edu/~suits/notefreqs.html>

The ratios between subsequent notes differ, but not by much.

Edit: a quickie spreadsheet gives me 0.009502 for the largest difference between two tones.

See also:

<http://en.wikipedia.org/wiki/Well_temperament>

<comment><USER7> But that again is just a convention, and before western music converged on some kind of well tempered or chromatic tunings, there were quite some experiments with keyboards featuring both flat and sharp keys, see [http://de.wikipedia.org/w/index.php?title=Datei:Cembalo_univ...](http://de.wikipedia.org/w/index.php?title=Datei:Cembalo_universale_Tastatur.JPG&filetimestamp=20111014032521) or <http://en.wikipedia.org/wiki/Archicembalo>

</comment></comment></comment><comment><USER8> Er... something's terribly wrong in this explanation. because the essential forgotten fact is that, as much as F# is also Gb, C is also B#, F is E#, B is Cb and E is Fb. So the G# scale exists and is perfectly valid, it reads

G# A# B# C# D# E# F##

And it is a very important point, because each scale on the circle of fifth has one more sharp than the previous one when turning in one direction, and one more flat in the other one.

Another point is that there is actually a comma between F# and Gb, and between F## and G, because the chromatic semi-tone is one comma shorter than the diatonic semi-tone -- Though of course the distinction isn't done on a tempered instrument such as a piano or a guitar.

It is all actually quite simple, but not /that/ simple :)

</comment><comment><USER9> Circle of fifths and circle of fourths allows one to algorithmically put together Irish traditional tune sets that mostly work. (Not exactly as stated, because Irish trad is really modal, but close enough for trad.)

Basically, take the home chord of a tune and apply one of the following:

If "major" then switch to the relative minor, or vice versa.

Go Round the circle of fifths or the circLe of fourths either direction, but stay on whichever side of "major/minor."

Go up or down the scale, staying on whichever side of "major/minor."

So for example, a tune set with E minor, G major, A major should work. (Again, not exactly it, but don't want to explain modes.)

<comment><USER10> I bet that all made sense in your head; not so much for the rest of us.

<comment><USER9> If you know enough music theory to know what a relative minor is, it makes sense.

<comment><USER10> You might think that, you'd be wrong. I know what a relative minor is, I play guitar daily. I can play up and down the circle of fifths/fourths and improvise endlessly over the harmonized major/minor scale playing right through the changes. None of that knowledge helps decipher the purpose or meaning of the instructions you gave.

I'll try and walk through it.

> If "major" then switch to the relative minor, or vice versa.

OK, home chord is Em, so we'll use G major.

> Go Round the circle of fifths or the circLe of fourths either direction, but > stay on whichever side of "major/minor."

OK, that's C or D given G as the starting point.

> Go up or down the scale, staying on whichever side of "major/minor."

So playing the G, C, or D scale over an E minor progression. G makes sense, same scale as Em; C or D... don't even parse for me. I don't think of chords and scales as separate, changing the scale changes the underlying chords I'd play under them, thus changing the key to something other than Em.

> So for example, a tune set with E minor, G major, A major should work.

What? Nothing you just said would lead me to think A major would work over an Em progression. G major, sure, obvious, it's the relative major of Em and just another name for G major scale, but A... why A, you haven't explained that above, your example doesn't match what the instructions seem to say.

<comment><USER9> Then what you don't understand is how a trad tune set works. What I'm talking about is how to put together a set of three tunes that work well as one track on an album or one "number" in a performance. Each tune internally has its own progression, which may be different. I'm only talking about the home chord of each tune. (Which I state.)

<comment><USER10> Well that clears up basically nothing; I still don't understand what you're saying, or where the A major comes in.

<comment><USER9> A certain level of density is indistinguishable from trolling.

<comment><USER10> True, but I wasn't. You really weren't as clear as you seem to think you were. If all you're saying is pick as roots nearby notes on the circle of fifths, you could have been much clearer.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> I vaguely remember using this knowledge of the scales to "cheat" on the violin. If you move your hand into whatever position puts your first finger on the first note in the scale then you can always move the rest of the fingers in exactly the same manner - whole whole half, whole whole whole half. My music theory knowledge is crumbling but what I mean is if you move into 2nd position on the A string to play a B scale its much easier than starting in 1st position with the 2nd finger.

I did this at a competition once when given some insane scales (like 3 flats in the clef) without having ever practiced the scale itself and got it right.

</comment><comment><USER12> Lots of editorial issues, for example when describing C# and F# some of the information is incomplete/mixed. Too bad there is no comment section.

</comment><comment><USER13> Looks really good (bookmarked to read later). Although maybe some more motivation as to why you start music theory with circle of fifths -- as someone who knows nothing about music, I had never heard of it before.

<comment><AUTHOR> That's a really good question. The reason why I started at the Circle of Fifths is because it's the foundation and it is highly mathematical. It's a tad bit painful to start with, but I wanted to start from the ground up. This will make the other stuff far more understandable. I can go on all day saying "this is how to make a chord" but starting here will let you understand the later ideas better.

I also started here to demonstrate how logical it all is.

<comment><USER14> You don't seem to provide any context about what the circle of fifths is, or even what a fifth is for, thus I have no idea why I should care, and in general it makes it very hard to assimilate the article in a useful way.

<comment><USER15> Yeah I agree. I think there should be some mention of why a "fifth" is important (3:2 ratio of frequencies, which is recognizable, and when this is continued, it eventually "loops back" to the original note (at a 2^n:1 ratio, where n is the number of fifths up until it repeats)

</comment></comment></comment></comment><comment><AUTHOR> Thanks for all the feedback guys. The links found here are all amazing as well. I know what to improve in this article and the next article will be much better.

Thank you so much;

</comment><comment><USER16> "Up" this if you read "Cradle of Filth" instead of "Circle of Fifths" lol</comment>


Chinese firm EHang reveals self flying MEGADRONE taxi - <AUTHOR> http://www.dailymail.co.uk/sciencetech/article-3387542/The-MEGADRONE-big-carry-passenger-Chinese-firm-says-self-flying-craft-used-smart-taxi.html <comment><USER2> Really the idea that the blades are not enclosed is so bizarre. Who would ever think having it as is was a good idea?

</comment><comment><USER3> so now you need a couple platforms that sit above everyone that you climb some stairs and bored. no pilot. flies to the next platform. 10 flights a day at $10 a flight should cover capital costs.</comment>


Should Startups Hire Consultants? - <AUTHOR> https://medium.com/@cdstamper/don-t-hire-contractors-42276eedd23a

<comment><USER2> There seems to be 2 sides to this; someone always says don't hire contractors. Another person says yes hire contractors.

If you can hire an employee for less than you are paying for an contractor (include your HR time, payroll, taxes, other paperwork) there should be some semblance of a balance.

If you hire a Consultant who doesn't care about you succeeding; then he is selling his future short. Any good employee or Consultant will be absolutely willing to do his part to see you succeed; as that leads to him succeeding.

There are bad employee's as well as bad consultants; let's not try and make it black and white.

</comment><comment><USER3> Ha, for a minute I thought you might have come up with a good reason not to hire consultants.

One important point is that consultants can be a way to buy a slice of a skill set or experience level that you couldn't afford to own outright, in the same way that Berkshire Hathaway benefits from buying slices of companies on the public markets that it couldn't own 100%.

<comment><AUTHOR> That's more or less the point I was trying to make. Consultants can deliever experience and skill levels that you can't afford or attract otherwise.</comment></comment>


My oil droplet is smarter than your lab mouse - <AUTHOR> http://arstechnica.com/science/news/2010/02/my-oil-droplet-is-smarter-than-your-honor-lab-mouse.ars <comment><USER2> Curious definition of smart. Sort of like saying a compass is better at finding magnetic north than I am, ergo it is smarter. The oil droplet is just a tool, like the compass, which always reacts the same way given external stimuli. Now, the people who knew enough chemistry to create such a droplet, those are some smart people.

I'm curious, though, how they'd do at running a person sized maze. Certainly they'd be beaten by the droplet, but I bet they'd also be outperformed by the mouse.

</comment><comment><USER3> Unless my lab mouse is really conducting experiments on you? Trying to make you think that your oil is smarter (RIP Adams)</comment>


Sivlebot, the Natural Language Processing bot, Updated after 5 years - <AUTHOR> http://sivlebot.com/ <comment><USER2>
        Sivle :  Hello, my name is Sivle.
        User  :  Hello Sivle - what would you like to talk about?
        Sivle :  Hi there!. How did you hear about Sivle?
        User  :  I followed a link from an internet forum. Why do you ask?
        Sivle :  Why?.
        User  :  I'm sorry - why what?
        Sivle :  There's no need to apologize.
        User  :  OK, let me re-phrase, why did you say "Why?" ?
        Sivle :  The more people who chat with me, the smarter I become..
                 Go ahead, Seeker, try to RE PHRASE WHY DID YOU SAY WHY.
        User  :  You asked me a rather bizarre question, and I was trying
                 to find out why.
        Sivle :  Oh now I remember asking you A RATHER BIZARRE QUESTION
                 AND it WAS TRYING TO FIND OUT WHY. What did you say?

Oh look, very Eliza-like.

How does anyone generate a plausible-sounding conversation with one of these things?

<comment><USER3> Both Eliza and Sivle run off of case based reasoning. Which is the basis for other more popular bots like Siri and Cortana.</comment></comment>


Tests on skull fragment cast doubt on Adolf Hitler suicide story - <AUTHOR> http://www.theguardian.com/world/2009/sep/27/adolf-hitler-suicide-skull-fragment <comment><USER2> This is particular evidence is relatively old news, the discovery was made during a History Channel special. There is currently a show running on the History Channel called Hunting Hitler that goes into much more depth on this topic and expands on the evidence (there is also a book from several years ago with the same namesake).

I agree that this information is not enough to say he escaped, but there is a plethora of other information that makes the case he escaped very compelling.

Some points the show makes: \- testimony from supposed eye witnesses in the bunker has been unreliable and inconsistent \- underground escape routes existed with possible tunnels (that have since been sealed) that lead directly to the templehof airport \- safe houses have been identfied that housed nazis after the war in spain and the canary islands \- eye witness uboat activity around argentina after the war ended \- various safe houses discovered in Argentina that housed Nazis \- ruins of a compound in remote Argentina that have nazi paraphernalia including medicines Hitler would have needed \- locations in nearby brazil that have nazi sitings after the war

There's a ton of new evidence, I really think there's enough to take these claims seriously. The only thing that I'm not sure about is the dental records that were verified in 1945. I'm not sure how well that was handled being that Hitler's personal dentist did the verifications.

[http://www.amazon.com/Hunting-Hitler-Scientific-Evidence- Esc...](http://www.amazon.com/Hunting-Hitler-Scientific-Evidence- Escaped/dp/1626361711/ref=sr_1_2?ie=UTF8&qid=1449388005&sr=8-2&keywords=hunting+hitler)

[http://www.amazon.com/The-Hunt- Begins/dp/B017WZWN7S/ref=sr_1...](http://www.amazon.com/The-Hunt- Begins/dp/B017WZWN7S/ref=sr_1_1?ie=UTF8&qid=1449388005&sr=8-1&keywords=hunting+hitler)

<comment><USER3> This is all the same sort of thing that the folks who keep "finding" Amelia Earhart do, and it's all completely backwards. They are not finding bits of evidence, investigating them at length, and then reaching a conclusion about Hitler. They're finding "evidence" and then asking "how can this prove my theory about Hitler"? It's the reverse of a real investigation, because it starts with a conclusion and then looks for evidence to support it.

<comment><USER2> As far as the show is concerned, I don't think this is true. They approach it forensically, first determining if they should even cast doubt on the historical assumptions. I know this is mostly dramatized tv, but it coming up with some great results using technology where it's never been used before. Of course a show needs a narative, and that's a bit dangerous, but it doesn't mean the new evidence is invalid. Funding archeology and investigations like this can only come though tv sadly. I hope some historians review the new material and give their feedback because I really don't want to have to rely on the History Channels accounts of these events.

<comment><USER4> Forensic science is also highly prone to confirmation bias. It's a rather low bar to say something was investigated forensically.

Unfortunately, this means that even valid discoveries funded by someone with a story already in mind can't be given much weight, because there's no way of knowing what discoveries their preexisting bias prevented them from finding or acknowledging.

<comment><USER5> Review this document, and decide for yourself:

[http://www.nl-aid.org/wp- content/uploads/2012/09/Sognnaes.pd...](http://www.nl-aid.org/wp- content/uploads/2012/09/Sognnaes.pdf)

</comment></comment></comment></comment></comment><comment><USER6> > Unknown to the world, Hitler's corpse was interred at a Smersh centre in > Magdeburg, East Germany. There it remained long after Stalin's death in > 1953. Finally, in 1970, the KGB dug up the corpse, cremated it and secretly > scattered the ashes in a river.

This sounds really interesting. Anyone have additional information on this topic?

<comment> m-i-l Hugh Trevor-Roper's "The Last Days of Hitler" is the nearest we have to a primary source in English. The 1947 edition didn't have any details from the Soviets, but there were a number of subsequent editions where additional details were added, most recently 1995.

</comment></comment><comment><USER7> After reading the article, I can't see how this discovery cast doubt on Adolf Hitler suicide story. The skull fragment may tot belong to Hitler, but that has nothing to do with the possibility that Hitler committed suicide. It only shows that the soviets were wrong to think that the skull fragment come from Hitler.

<comment><USER8> People want to know exactly what happened; the cyanide pill would have been enough to kill him so where did the story of the shot in the head come from? And now that it is not sure if it happened then did it happen or did he just take cyanide and was the self inflicted head shot story only added to make him more of a 'man' after the fact (as in; women use poison, men use guns/swords to kill themselves (and others))?

</comment></comment><comment><USER9> I don't get this fascination with hunting down the truth behind a single person. He died, whether then or 20 years later, case closed.

I find more interesting the that ex-Nazis worked on some of the most important discoveries and inventions of the modern age, as well as them working behind the scenes to fight against communists in West Germany ([http://www.spiegel.de/international/germany/wehrmacht- vetera...](http://www.spiegel.de/international/germany/wehrmacht-veterans- created-a-secret-army-in-west-germany-a-969015.html)).

History is not about one person, it's about society and civilization as a whole...

<comment><USER10> > I don't get this fascination with hunting down the truth behind a single > person. He died, whether then or 20 years later, case closed.

It's significant because it gives us a rare insight into Mr Hitler's late-war psychology.

The traditional story, at least in the English-speaking West, was that he finally realised that he had failed himself and Germany and took his own life to escape the inevitable wrath of the Soviets.

An alternative narrative that I have encountered is that he saw the whole conflict as a competition between national character. He felt that the majority of the German people had failed _him_ and Germany, in which case it is difficult to understand why he would have committed suicide.

</comment></comment><comment><USER11> I thought it is well known under conspiracy theorists that Hitler died near Bariloche, Argentina at the Nahuel Huapi Lake. The secret postwar center of the highest german NAZI criminals. [http://www.google.com/?q=hitler%20bariloche%20argentina](http://www.google.com/?q=hitler%20bariloche%20argentina)

</comment><comment><USER12> Not sure if a DNA test was necessary to determine that skull fragment wasn't the result of a suicide... not that this says much about whether it is Hitler's skull or not. The picture provided in the article clearly shows a downward angled hole high on the parietal lobe of the skull-- a typical suicide shot, assuming it was taken at roughly the temple, should be in the temporal or sphenoid bones, and angled flat or slightly upward.

EDIT: I don't know if Hitler was right or left handed, but a right handed person would likely use their right hand to fire a suicide shot. This may be another fact that is inconsistent with the existing skull fragment, which has a bullet hole on the right half.

<comment><USER13> It's not very unusual to shoot with the "wrong" hand. Sure, someone who had never used a gun before would naturally pick one up with her dominant hand. Those who practice a bit, however, often shift to using their dominant eye (occasionally opposite the dominant hand). If only one chirality of firearm were available when one were learning to shoot, one might acclimate to that as well. Right-handed bolt-action rifles are a regular annoyance to me.

<comment><USER14> Are there non-symmetric pistols?

<comment><USER13> Many pistol safeties that are located near the trigger operate side-to-side. That means that it's easy, depending on which hand is holding, either to go from safe to fire _or_ from fire to safe, but going the other direction requires using the other hand. Most shooters would prefer the safe-to-fire transition to be easy. Not all safeties on modern firearms are set up like this, however.

De-cocking mechanisms may be located on only one side, although most are on both.

I've never seen a slide release that wasn't limited to only one side. That would seem unavoidable from a physics perspective.

However, your question is a bit off-target. Most shooters, particularly early in the previous century, would learn with a rifle first, and their rifle preferences would transfer to their pistol use.

<comment><USER14> > _However, your question is a bit off-target. Most shooters, particularly > early in the previous century, would learn with a rifle first, and their > rifle preferences would transfer to their pistol use._

Oh, I was curious in general, not as pertaining to this particular historical question. Thanks for the answer!

<comment><USER13> You're welcome! b^)

</comment></comment></comment></comment></comment></comment><comment><USER5> Hitler's dentist and the techs who worked with him were captured after the war. They independently reconstructed the details of the dental work that was placed in his mouth. The reconstructions meshed with each other and matched perfectly with the dental work on the jaw fragments attributed to Hitler, including the famous "telephone bridge" which was put in his mouth. They also match with the skull X-rays of Hitler which were taken after the July 20th bomb attempt. Hitler died in Berlin, forensic analysis confirms it.

</comment><comment><USER15> While interesting...it is from 2009.</comment>


Show HN: JavaScript, ES6, ES7, ES10 Where Are We? - <AUTHOR> https://medium.com/engineered-publicis-sapient/javascript-es6-es7-es10-where-are-we-8ac044dfd964 <comment><USER2> That's the best breakdown of the ES proposals I've ever seen. I wish this existed a year ago when I was trying to explain it to a coworker.

I'm glad someone took the time to put this together. Also, I did not realize that the native functions for most of those features were available, and considering they are faster than transpilers like Babel that polyfill everything, I need to check my webpage compiler outputs to make sure i'm not needlessly polyfilling for the 1.8% of IE users I don't care about

<comment><AUTHOR> Thanks you very much! I really appreciate your comment!</comment></comment>


Yada yada ... scientists name new virus after Seinfeld catch phrase - <AUTHOR> https://www.sciencemag.org/news/2020/01/yada-yada-scientists-name-new-virus-after-seinfeld-catchphrase <comment><USER2> except the term yada yada was around long before seinfeld

I remember it from my childhood.

[https://www.phrases.org.uk/meanings/yada- yada.html](https://www.phrases.org.uk/meanings/yada-yada.html)</comment>


Elevator Pitch Friday: Truevert, The Green Search Engine - <AUTHOR> http://www.techcrunch.com/2008/11/14/elevator-pitch-friday-truevert-the-green-search-engine/ <comment><USER2> _"Truevert gives you special results—everything comes back through green- colored glasses. So a search for 'SUV' brings back HybridSUV.com as the top result. A search for 'building materials' brings back results for green building materials."_

Until you search for something that isn't generic, then it returns poor results -- or at least it did when searching for relatively simple things in Seattle.

 _"It is a Yahoo BOSS search mashup. Truevert is actually just a demonstration of some powerful underlying semantic technology developed at OrcaTec, a company co-founded by Roitblat and Brian Golbère. Truevert gets its search results from Yahoo BOSS and applies its own text-analyzing software to generate the most relevant green results. OrcaTec's software could just as easily be used to create a fashion search engine, a startup search engine, or any of a thousand other vertical search engines."_

Once it works.</comment>


Over 100k GitHub repos have leaked API or cryptographic keys - <AUTHOR> https://www.zdnet.com/article/over-100000-github-repos-have-leaked-api-or-cryptographic-keys/ <comment><USER2> * Erase creds from your repo's history with the BFG[1]

* GitHub scans public repos for API keys[2]. Why is it alerting the service provider, and not the owner of the repo?

[1] [https://rtyley.github.io/bfg-repo-cleaner/](https://rtyley.github.io/bfg- repo-cleaner/)

[2] [https://help.github.com/en/articles/about-token- scanning](https://help.github.com/en/articles/about-token-scanning)</comment>


The Baltic States: The New Hotspot of Tools for Developers and Designers - <AUTHOR> http://arcticstartup.com/2014/12/22/the-baltic-states-the-new-hotspot-of-tools-for-developers-designers <comment><USER2> Waw, there are a few quite cool products I didn't now before. And I know the community quite well ;)

<comment><AUTHOR> thanks, I'm getting comments about the ones I left out - looks like this will need a followup :)</comment></comment>


Carnegie Mellon Is Saving Old Software from Oblivion - <AUTHOR> https://spectrum.ieee.org/computing/software/carnegie-mellon-is-saving-old-software-from-oblivion <comment><USER2> "Think of it as a Wayback Machine for executable content." Internet Archive already doing some of this. [https://archive.org/details/softwarelibrary](https://archive.org/details/softwarelibrary) Jason Scott pushed hard for JSMESS, the in-browser emulator that brings the archived software to life. [https://www.archiveteam.org/index.php/Javascript_Mess](https://www.archiveteam.org/index.php/Javascript_Mess)

This is one of a few similar projects. The government Archives offices of most countries have to not just preserve digital records but also the software necessary to read them. Mick Crouch is the legend in New Zealand who does a pile of this work: [https://www.radionz.co.nz/news/national/346630/outdated- tech...](https://www.radionz.co.nz/news/national/346630/outdated-tech-trove- keeps-digital-archives-alive)

<comment><USER3> I find it it pretty ignorant for a publication as venerable as IEEE to miss the fact that archive.org already has this.

<comment><USER4> Near the end of the article it does say:

 _We are not alone in our quest to keep old software alive. For example, the Internet Archive is preserving thousands of old programs using an emulation of MS-DOS that runs in the user's browser..._

And also then goes on to refer to other Living Software Preservation initiatives.

</comment></comment></comment><comment><USER5> I find this especially relevant. Recently I found an old textbook that was amazing. Lots of great detail on nearly impossible to find subjects.

I emailed the professor (now near retirement) for the answers to the solutions as I'm no longer a student and the book has been out of print since before I was born. He told me he didn't have it anymore as it was on an old computer on an operating system that doesn't exist anymore.

All the code, example answers lost to time.

It made me wonder just how common this phenomena is.

<comment><USER6> Compare [https://news.ycombinator.com/item?id=18036255](https://news.ycombinator.com/item?id=18036255), where right now people are slamming the OP for posting solutions to Coursera exercises.

If the goal is education, the solutions should be published too.

<comment><USER5> Agreed. If there are 100 problems in a book and I work all 100, how am I to know if I did it right? If I get one wrong (which I assuredly will), how am I to see what I did wrong?

Some teachers are awful too and the only way to learn is to get your hands on a solutions manual and reverse engineer your understanding. It can be a very frustrating experience.

</comment></comment></comment><comment><USER7> Reproducibility is a huge issue in academia and in the software engineering world.

As an anecdote, we were trying to reproduce results for a c++ algorithm for which we had the source code, on the same machine, same compiler, only 2 years after its original creation. Impossible. Our best guess is that some os/hw patch / firmware update had an impact on the random number generation, but we never managed to figure out what exactly caused the discrepancies that we were observing.

<comment><USER8> The way some video game studios do this is simply put a whole computer into the archive with all the software on it, so all the hardware and software are captured in time.

<comment><USER9> What's the expected lifetime of a computer in storage? I'd think that bitrot sets in fairly quickly on storage medium.

<comment><USER10> That's data rot or disc rot. Bit rot is also known as software rot.

</comment></comment></comment></comment><comment><USER11> > For now, though, Olive is available only to a limited group of users. > Because of software-licensing restrictions, Olive's collection of vintage > software is currently accessible only to people who have been collaborating > on the project. The relevant companies will need to give permissions to > present Olive's re-creations to broader audiences.

What do you do when the original company no longer exists?

<comment><USER12> Depends on what the EULA for the product said and what happened to the company.

Companies of any value tend not to just disappear; they get bought by other companies who thus inherit their IP rights.

Sometimes after a while it can be hard to determine who owns a piece of IP. This can happen with old video games, for example, since the studios and publishers have split and merged so much over the years. At times it's a matter of interpretation who owns a property, and it can end up in court.

TLDR there's almost always someone ready to take your money; not always easy determining who it should be.

<comment><USER13> > Depends on what the EULA for the product said and what happened to the > company.

The EULA to dBASE III stated that the grant of license lasted for fifty years. If, after fifty years, you were still using dBASE III, you would no longer be using it legally and had to return your copy to Ashton-Tate.

All you dBASE III users out there -- you've got maybe 15-20 years left!

</comment></comment></comment><comment><USER8> Reminds me, if you haven't been to Paul Alan's Living Computer museum I'd highly recommend it. You can play chess against a real working PDP-8 which is pretty impressive.

[https://www.youtube.com/watch?v=ZHU4nzIsaIM&t=5m38s](https://www.youtube.com/watch?v=ZHU4nzIsaIM&t=5m38s)

<comment><USER14> I was wondering how they get the funds for such an expensive operation and I realised you mean Paul _Allen_

The Paul Allen who is the co-found of Microsoft and one of the richest people in the world.

</comment></comment><comment><USER15> I want to know how that will be done with cloud applications when the companies never have published their cloud/online software.

</comment><comment><USER16> Years ago, an online acquaintance undertook something with similar intent, collecting old web browsers: [https://browsers.evolt.org/](https://browsers.evolt.org/)

<comment><USER17> One of the oldest versions of Opera (in the archive) from 1999 (765.31 KB): [https://browsers.evolt.org/browsers/archive/opera/win/212/](https://browsers.evolt.org/browsers/archive/opera/win/212/)

<comment><USER18> Opera 2.xx was ~1996

</comment></comment></comment><comment><USER19> similarly: [https://www.inria.fr/en/news/news-from-inria/launching-of- so...](https://www.inria.fr/en/news/news-from-inria/launching-of-software- heritage)

[https://www.softwareheritage.org/](https://www.softwareheritage.org/)

</comment><comment><USER20> I thought the federal government did that via 20th century COBOL software few understand.

</comment><comment><USER21> I need to pay a visit to the CMU campus to see if I can see their work.</comment>


Reverse Engineering a Mysterious UDP Stream in My Hotel - <AUTHOR> http://wiki.gkbrk.com/Hotel_Music.html <comment><USER2> At least they play music. I once stayed in a Howard Johnsons Motor Lodge in Pittsburgh near CMU, where, at corridor intersections, they had speakers to generate some ambient noise to mask voices from the rooms. Most places that do this use white noise, or water sounds. But this was Pittsburgh. They were playing faint machinery noises - whirr, chunk, etc. At first I thought someone had left a PA microphone open in the boiler room or something, but no, it was deliberate.

<comment><USER3> I'm surprised public washrooms don't play music or it seems very few do. I bet so many people in stalls wait for someone to start the air dryer or turn on a tap i.e. strategic pooping. It can be a embarrassing disaster with those modern auto shutoff air hand dryers.

<comment><USER4> Japan has portable, artificial white noise makers for bathroom privacy. They're called otohime, or "sound princess."

\- video: [https://www.youtube.com/watch?v=f0eqo6jU9gc](https://www.youtube.com/watch?v=f0eqo6jU9gc)

\- example gadget: [http://www.japantrendshop.com/keitai-otohime-toilet-sound- bl...](http://www.japantrendshop.com/keitai-otohime-toilet-sound- blocker-p-851.html)

\- background and wall mounted examples: [http://travel.cnn.com/tokyo/shop/sound-princess- loo-001106/](http://travel.cnn.com/tokyo/shop/sound-princess-loo-001106/)

<comment><USER5> Oh my god, that name is an amazing pun. Younger princess -> sound princess

<comment><USER6> Can you explain this pun? Not getting it!

<comment><USER7> "otohime" is how to pronounce one of the terms of address for the second daughter of the first Kamakura shogun. The Second daughter, hence, the "younger princess".

The Japanese word for sound is also pronounced "oto". So, a device which makes noise to help you be discreet...

<comment><USER8> Plus oto- is a Latin prefix meaning ear. Multilingual puns are the best!

<comment><USER9> Interesting. I wonder how Japanese and Latin both ended up using 'oto' for an auditory related word.

<comment><USER8> Not sure on that, without invoking some vague talk about universal language mumble Chomsky mumble mumble. It could be by chance, but things like the kiki- bouba experiment suggest that the dice loaded, at the very least.

But things that look like cognates with Japanese do exist. Notably そう ("sou") is pronounced the same as English "so" and means roughly the same thing, at least in phrases like "is that so?" and "Make it so, Number One."

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> Can you send your own UDP packets to the elevator then?

<comment><USER11> It would be possible to send packets to the elevator, but the elevator playing them would be another issue. If there is no authentication at all (as it it just plays all packets it receives on UDP 2046) I would imagine you would get an interesting mix of "valid" elevator music and your own "invalid" music.

On the other hand, those first 8 bytes of the packet may be some authentication/verification scheme which would have to be reverse engineered. Also, it may only play UDP packets coming from 126.96.36.199:2046, which would likely mean you would have to convince the DHCP server to assign you that address instead of its intended host.

<comment><USER12> > convince the DHCP server

Or you could just manually configure your computer to use that address.

<comment><USER13> Or you can craft packets all day long with any source address you like and dump them onto the network.

</comment></comment></comment></comment><comment><USER14> Now I know what to do in a hotel!

[https://www.youtube.com/watch?v=1Un_oHaf798](https://www.youtube.com/watch?v=1Un_oHaf798)

<comment><USER15> Dude that's awesome! Thanks for the link.

</comment></comment><comment><USER16> I was in an elevator a while ago when there was a ringing followed by a voice trying to sell PPI services (a regular source of spam in the UK). The emergency system was just an embedded phone.

<comment><USER17> In the US too - the reasoning is that it's simple and should work in a power outage as the telephone company usually can power it from the central office. Conversely, VoIP would depend on too many technologies working well to be reliable enough for emergency use.

<comment><USER18> I've definitely seen elevator phones attached to the PBX with VoIP trunking, but all the components were on battery backups.

<comment><USER19> At a previous employer (large .edu), we moved to VoIP several years ago but keep several PSTN lines -- including all of the emergency ones, such as for the elevator. We could have moved them to VoIP as well but we figured, in a true emergency, it was one less thing to have to worry about.

<comment><USER20> In Chicago building codes mandate elevators have a pstn line in place.

Even luxury homes with small elevators.

<comment><USER21> Well, I mean people have died from being stuck in their own elevators: [http://www.cbsnews.com/news/elderly-couple-dies-trapped- in-h...](http://www.cbsnews.com/news/elderly-couple-dies-trapped-in-home- elevator/)

</comment></comment></comment></comment></comment></comment><comment><USER22> Binwalk is a great tool for finding potential files within a given binary stream: [https://github.com/devttys0/binwalk](https://github.com/devttys0/binwalk) It has an incredible list of supported file types.

<comment><USER23> Foremost is also extremely useful for the same purpose, and I've found it more useful at least in my work during CTFs

[https://en.wikipedia.org/wiki/Foremost_%28software%29](https://en.wikipedia.org/wiki/Foremost_%28software%29)

A `sudo apt-get install foremost` works on Ubuntu to install it, IIRC

</comment></comment><comment><USER24> That was just the carrier, hiding the steganographic payload.

<comment><USER25> I bet the payload is in that NES cartridge. Check the data against the NES cart header list, or just try loading it in NESticle. That's what I would have started with, rather than focusing on the meaningless LAME MP3 data.

<comment><USER26> There is no NES cartridge data. It looks like OP is using an outdated magic file with a bug that was fixed in april [https://fossies.org/diffs/file/5.26_vs_5.27/magic/Magdir/con...](https://fossies.org/diffs/file/5.26_vs_5.27/magic/Magdir/console- diff.html)

</comment></comment></comment><comment><USER27> Spoiler:

Revelation/Disappointment -- it is elevator music.

Or is it? Maybe he gave up too quick. Maybe that is how they disguise the secret spy transmissions!

[https://en.wikipedia.org/wiki/Steganography](https://en.wikipedia.org/wiki/Steganography)

<comment><USER28> Came here to say this.

There two modes. One for normal people. Parents, cops, investigative journalists, government employees etc.

Then there's the mode for geeks. Then your base case should be that everybody is Norman Bates

</comment></comment><comment><USER29> This is really well written. Good length for one quick bus ride. I was on pins an needles until the end. Great blog post.

</comment><comment><USER30> You had to follow that shaggy dog a long time, but it finally led you to the girl from Ipanema.

</comment><comment><USER31> Reminds me how surprised I was when I found out that many IP phone installations use multi-cast strictly to distribute on-hold music to all phones, instead of the phones pulling files from a server or storing it locally.

<comment><USER32> Two reasons:

1\. Multicast is 1:N so one stream can be played on all phones. Pulling the files from a server would be N:N so your network bandwidth would be consumed unnecessarily due to all the phones streaming the same data. Also, the phones are going to have limited memory so storing music locally is not going to scale well (phone memory is a cost that gets multiplied by N phones).

2\. Synchronization, especially for the elevator scenario: if the music outside the elevator door isn't synchronized to the music inside the elevator, it will be rather disconcerting.

BTW, I suspect the streaming audio you saw was "background music" that could be played from the phones speakers. "Ambiance audio" would tie in with #2 synchronization; having adjacent phones playing unsynchronized "ambiance audio" would also be jarring.

Music on hold will be inserted by the PBX (head end), not the individual phones. Inserting "music on hold" by the phone would mean that music would be sent by the PBX to the phone back to the PBX and then to the "on hold" line where having the PBX insert it involves only the link from the PBX to the "on hold" line.

<comment><USER31> Yeah, it's efficient and easy to update. Was just surprised since applications for multicast crossing subnet borders are fairly scarce, so I guess multicast routing got configured just for that, and is one of these things you don't see mentioned very often. Thanks for the details!

</comment></comment></comment><comment><USER33> Excellent story, although the URL spoiled it for me, might worth changing it to something more vague.

<comment><USER34> It's about the journey, not the destination. But are you seriously suggesting he change it to a clickbait URL?

<comment><USER35> The story _is_ the journey, and there is no story without suspense. The URL is the dust jacket- some people are going to see the URL even before the title. No one said anything about clickbait. Any URL without the spoiler is better.

</comment></comment></comment><comment><USER36> Next time you see a multicast stream, try playing it with vlc.

<comment><USER37> That was my thought, too -- I would have just tried vlc and missed all the coding fun.

<comment><USER38> Me too. I had a friend who was convinced that he's being attacked when a steady stream of UDP packets started reaching his router for days. Asked him to open it in VLC and it turned out to be local news from his ISP's misconfigiured IPTV service.

</comment></comment></comment><comment><USER39> A story with a decidedly less innocuous outcome: [https://medium.com/@nicklum/my-hotel-wifi-injects-ads- does-y...](https://medium.com/@nicklum/my-hotel-wifi-injects-ads-does- yours-6356710fa180#.y3bctq5uy)

</comment><comment><USER40> I'd start with

tcpflow -p -C -i eth0 port 80 | grep -oE '(GET|POST|HEAD) .* HTTP/1.[01]|Host: .*'

<comment><USER19> cf. _ngrep_ [0]

[0]: [https://en.wikipedia.org/wiki/Ngrep](https://en.wikipedia.org/wiki/Ngrep)

(Yes, I chose to link to Wikipedia instead of its home page on SourceForge.)

<comment><USER41> Beautiful. Where has this been all my life.

</comment></comment></comment><comment><USER42> As someone who doesn't really grok either Python or IP programming, I really enjoyed how simple yet educative (and ultimately, funny) this blog post was. Definitely will be trying this exercise the next time I end up at a hotel with a laptop.

I wonder how hard it might be to hijack the stream to have the receivers play your own packets.

</comment><comment><USER43> Have you tried throwing the raw recording against SoX? Worked pretty well when I had an unknown recording to play for which I had guessed the format. (Which turned out to be IMA-ADPCM with reversed byte ordering)

</comment><comment><USER44> ah damn, not a network guy but couldn't they put lifts on their own subnets and avoid populuting the constrained airway. good read :)

</comment><comment><USER45> It's probably a microphone hidden in his room that encodes the data stenographically into elevator music.

More seriously, no investigation as to what happens when you try to inject your own data?

</comment><comment><USER46> Does it drain battery of mobile devices not listening to the port?

Have you tried multicasting your own audio to the same port? That might have been fun.

<comment> david-given I did a bunch of multicast programming in the early days of Android, and I discovered that:

\- most Android devices back then required a special OS call to tall the wireless chipset to listen to multicast packets at all --- otherwise they'd just ignore them and not wake up;

\- in about half the devices, this switch didn't actually work;

\- in an astonishing amount of consumer routers, multicast routing doesn't actually work;

\- multicast on mobile is so, so, so not worth the effort.

I expect that multicast is way better supported these days. I would be totally unsurprised if it were not.

<comment><USER47> > I expect that multicast is way better supported these days. I would be > totally unsurprised if it were not.

I wouldn't be, how often is multicast really used? Coming from the broadcast era, it seemed like a no-brainer. But the internet is built for n:n communication. Special cases for n:1 cost more in terms of engineering effort than would be saved in bandwidth and processing overhead.

<comment><USER48> Some IPTV services in the UK use it (at least BT and TalkTalk with their YouView packages and probably off-the-shelf YouView as well). I had a bit of a nightmare trying to get it working with BT TV when I upgraded from their godawful HomeHub to an ASUS router.

Pretty much every time I've upgraded my router firmware I've lost TV service and had to reconfigure it. Apparently multicast is hard, even if you're a networking company!

</comment></comment></comment></comment><comment><USER49> Er, if they are transmitting data to a multicast IP address, what's to stop you doing the same?

If they haven't done a check on the IP address that they are receiving the data from, then it would now be trivial to panic people in elevators by recording a fake emergency broadcast.

Not jus that, but what else is that hooked up to? If they are multicasting to your IP address and your IP address isn't the lift, then you can do some IGMP snooping to see what else there is out there. Or you could do a DoS on the lifts to see what happens.

Of course, it might be nothing. But when I get in a lift, I'd hope this sort of thing wasn't possible.

</comment><comment><USER50> Now just copy the headers and send some multi cast audio of your own to hijack the elevator and bathroom audio :) Could make for a good prank

</comment><comment><USER51> Great network engineering. I think we all need a set of tools that allow us to find if we are being bugged by all the networked smartTV's, printers VOIP phones etc. Not to mention the Amazon Echo. I dig it if someone had a Wireshark/Python app that allowed everyone to listen to with Amazon Echo was sending to Amazon.

<comment><USER52> The hacker in me would love to see an analysis of the traffic, but the consumer in me hopes that the Echo sends and receives encrypted data.

</comment></comment><comment><USER53> I love knowing I get spammed with elevator music over WiFi at a hotel the whole time.

<comment><USER54> A relaxed computer is a happy computer.

</comment></comment><comment><USER55> I'd love to see someone spoof those UDP multicast and play death metal or something rather then elevator music.

</comment><comment><USER56> Out of general interest, do you have a copy of the mp3 we could listen to? I couldn't find one in the article.

</comment><comment><USER57> Is this legal? What if the media was from Hotel Cameras?

<comment><USER58> At least around here, if it was from cameras, the hotel would be the one committing an illegality.

<comment><USER25> You don't have an expectation of privacy in hallways and elevators.

<comment><USER58> Yes, I know that's the US concept. Around here the view is a little less binary, as the law recognizes a difference between being seen (or even being filmed occasionally, by an handheld camera) and surveillance cameras, which can be used to track your movements.

The hotel could still have them, mind you, but transmitting them in the clear would be a clear violation of the law, which in fact specifically cites the "transmission of the recordings over a network" as something that must be secured.

</comment></comment></comment></comment><comment><USER59> I applaud the effort and your curiosity. Any good tunes on that stream? Perhaps you're working on a remix?

</comment><comment><USER60> Okay, I got a good laugh out of this one and appreciate it.

</comment><comment><USER61> Hehe, nice story! Congrats ;)

</comment><comment><USER62> haha! Best reverse engineering of elevator music ever :)

</comment><comment><USER63> Nice story

</comment><comment><USER64> It was an ex-NSA agent talking to his mom :)

</comment><comment><USER65> Do you know you are a nerd when you laugh out loud after reading this?

My girlfriend: Was that something I would also laugh at? Me: Most likely not ;)</comment>