SQL Tricks - <AUTHOR> https://blog.jooq.org/2016/04/25/10-sql-tricks-that-you-didnt-think-were-possible/ <comment><USER2> Standard SQL sucks and this is Why...

I think standard SQL is not the solution to many of these problems. What we need is something new or in this case old, that is built for such queries. What do I mean, well let's look through their examples reimplemented in qsql and I'll show you how much shorter and simpler this could be: [http://www.timestored.com/b/standard-sql-sucks-and-this- is-w...](http://www.timestored.com/b/standard-sql-sucks-and-this-is-why/)

On another note I actually love jooq the java library and have reimplemented something similar for qSQL. If anyone from jooq is reading, I've made a free java snippet runner called jpad.io that I would be interested seeing used with jooq. I think your customers may like it.

<comment><USER3> You're probably right, and at the time SQL was "invented" (and standardised), a lot of people were right too. They wanted QUEL, they got SQL. But hey, the presentation (linked from the article) got a slide for you:

[http://de.slideshare.net/LukasEder1/10-sql-tricks-that- you-d...](http://de.slideshare.net/LukasEder1/10-sql-tricks-that-you-didnt- think-were-possible/35)

Sorry :)

<comment><USER2> "SQL is the worst form of database querying, except for all the other forms."

LOL. As long as many people believe things like that, innovators can go out, learn new tools that will give them 10x more productivity and be able to make better things faster, delivering more business value.

In the words of Paul Graham [http://www.paulgraham.com/avg.html](http://www.paulgraham.com/avg.html)

"But I don't expect to convince anyone (over 25) to go out and learn Lisp. The purpose of this article is not to change anyone's mind, but to reassure people already interested in using Lisp-- people who know that Lisp is a powerful language, but worry because it isn't widely used. In a competitive situation, that's an advantage. Lisp's power is multiplied by the fact that your competitors don't get it."

<comment><USER3> I know that article, it's excellent. And I like Paul Graham's point of view, usually. But that article addresses (and will ever only address) the "elite". Like QUEL, which didn't make it. SQL did.

This doesn't have too much to do with innovation. It has everything to do with product-market-fit. SQL is the technology for the masses. QUEL (or something else more recent, very lean and sophisticated) is the technology for the elite. But why doesn't the "elite thing" go mainstream? Wrong product-market- fit.

<comment><USER4> QUEL's problem was not that it was for the elite but that Relational Technology (later named Ingres) did not have any sense on how to market and sell Ingres RDBMS and promote QUEL. Oracle's strength has always been marketing and sales and they started to take over the market quickly from Ingres in the first few years of commercial RDBMS.

Yes there were other players in the market at the time but in the very early days it really was Ingres versus Oracle.

For those of us that are older we know that Ingres was a result of Dr. Michael Stonebraker's work at Berkeley. There was at the time a university project known as Ingres and a commercial product known as Ingres. Yes postgres is what happened at Berkeley after Ingres.

I always loved QUEL and was downright pissed when it was ignored over the inferior SQL.

<comment><USER3> We kind of agree, but from a different angle. I see a product in its entirety: Technology + Community. The latter should extend beyond the technology-affine end user. As you mentioned, Oracle was always better at that.

I wasn't alive back then, I only know from listening to the stories. But I'd say that most Oracle customers are rather happy being Oracle customers, while only a select "elite" regrets what happened back then.

<comment><USER4> My memory is failing me at 62 but back then the Ingres community was much more positive about the product than the Oracle community. At one point I was president of the North American Ingres User Group (Association, what did we cann it?). I had moved from using Oracle version 1.x to Ingres 1.x in the beginning because Oracle was so bad. I think that was about 1982 or 1983.

But the company (Relational Technology) just had no idea what they had and how to sell it.

I'm an Oracle customer today because my shop has a site license. I would use mysql for what I need an RDBMS for today (very light, low volume usage). Oracle is overkill for my work. But it is always available and there are advanced features should I need them.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> CSS3 is also Turing-complete [1], so even while you _can_ write a whole bunch of data processing stuff with SQL not means you _should_. There are indeed better tools for doing this, keep in mind what is "Q" in SQL.

[1] [https://news.ycombinator.com/item?id=2300836](https://news.ycombinator.com/item?id=2300836)

</comment><comment><USER6> I've read a few good Jooq blog posts, and am becoming a convert to Jooq.

The tool seems to work well as a SQL DSL and helps me write testable code. Instead of an ORM that insists everything is an object, I can be honest and admit that I have a database which is a separate tool with its own features. And I have access to them all. Lovely.

<comment><USER7> Agreed. The code I write with JOOQ is so much leaner, and easier to reason about than any ORM I've used. To me, JOOQ is a great example of what modern Java should look like.

My only complaint about JOOQ is that there isn't a .NET version.

</comment></comment><comment><USER8> Interesting post - I love this idea that everything is a table and with INSERT, "you're not really inserting individual rows. You're really inserting entire tables".

</comment><comment><USER9> After a coworker left, I was - among other things - put in charge of maintaining a bunch of database views he added to our ERP system to do some reports (on Microsoft SQL Server, in case it matters).

It wasn't even that it was so exceptionally clever, but the SQL code made my eyes bleed at times.

So while I know basic SQL fairly well, I am by no means a wizard, and I tend more and more to keep the queries I write as simple as possible and to do the heavy lifting on the client side. Obviously, there are limits - to get the sum of a column, for example, it makes much more sense to do that on the server instead of transferring lots of data to the client and have it compute the sum.

As soon as I want to do more sophisticated stuff, my queries tend to get fairly ugly, and some day some other poor soul might have to look at and understand the code. I'd rather write a simple query and do the more complicated transformations using a PivotTable in Excel.

So while it's nice to know that there are these tricks, I try not to use them (not more than I have, anyway).

<comment><USER3> So. Rather than having everything in one place (single view) you now integrated some "easier to understand" VB scripts in Microsoft Excel and tied that to the SQL Server?

This might have been easier for you at that very moment, but it doesn't sound like an easier to maintain solution...

Why not instead take your SQL skills to the next level? It will be very rewarding, and you won't look back to MS Excel based solutions.

<comment><USER9> When I have a program issuing SQL queries, I have to write both anyway, and it seems unlikely that either is going to be reused without the other. So I had to draw a line in dividing the work between my program (a Perl script) and the SQL queries it uses. Using simpler SQL queries and letting Perl sort out the rest resulted (IMHO) in lower total complexity.

The same goes for Pivot tables where using simpler queries and then organizing and processing the data in Excel again results in a simpler and more flexible solution in the sense that our controlling / accounting department can play around with the data themselves, not requiring any programming skills.

If I have to write a View for some external application, I have little choice but resort to more advanced techniques and readily do so. I never said it isn't fun. ;-)

But with SQL, as with all programming languages, I think avoiding unneccessary complexity is desirable. Sometimes using these advanced techniques makes things clearer, sometimes not.

</comment></comment></comment><comment><USER10> Knew most/all of these tricks already, but it's a good summary of some useful tricks.

Just one thing...

"Common Table Expressions (also: CTE, also referred to as subquery factoring, e.g. in Oracle) are the only way to declare variables in SQL (apart from the obscure WINDOW clause that only PostgreSQL and Sybase SQL Anywhere know)."

Can't say anything about other vendors, but in T-SQL (a.k.a. MS SQL) creating variables is easily done via the DECLARE keyword, e.g. DECLARE @x INT = 4; . You can create a number of structures this way, including temporary tables (there are few different types of temporary tables, but the ones specified with DECLARE are specific to the scope in which they were defined in).

Does the DECLARE keyword (or something similar) not exist within any of the SQL standards?

<comment><USER11> CTEs aren't for creating a simple value. They are for creating a table variable.

<comment><USER10> I know what CTEs are for (in practical terms they're useful for subqueries you need to reference more than once), and like I said before, you can define temporary tables with the DECLARE keyword. To give you an example, this is valid T-SQL code...

DECLARE @temptable TABLE (id INT, fulldescription NVARCHAR(MAX));

</comment></comment></comment><comment><USER12> > Common Table Expressions (also: CTE, also referred to as subquery factoring, > e.g. in Oracle) are the only way to declare variables in SQL (apart from the > obscure WINDOW clause that only PostgreSQL and Sybase SQL Anywhere know).

Unless you're using all server which supports variables?

Not sure if I misunderstood what he's trying to say or not.

<comment><USER13> That variable support isn't part of SQL, it's part of the higher query execution language -- for want of a better term -- made available by the RDBMS. Besides embedding them into a CTE or getting them from a predefined table(s), there's no way of getting arbitrary data (i.e., variables) into a SQL statement.

<comment><USER14> Local variable support is _sort of_ part of SQL as it's part of the SQL/PSM Persistent Storage Module extension to SQL-92. Very few databases implement it according to the standard though.

<comment><USER15> The key thing that is important is that a recursive CTE allows for the retrieval of data from graph-like table schemas.

I'd sort of forgotten about this feature of SQL! But common table expressions and window functions actually keep the expressiveness of SQL and fill in gaps that just straight subselects, derived tables and joins alone can't always achieve.

Take for instance the rolling update he highlights. Before SQL Server got windowing functions (and boy, were Microsoft ever slow in implementing _those_ in SQL Server!) the way to do a reasonable fast query (i.e. Not use cursors) was one a guy called Jeff Moden came up with in 2012:

[http://www.sqlservercentral.com/articles/T-SQL/68467/](http://www.sqlservercentral.com/articles/T-SQL/68467/)

It relied on undocumented behaviour he observed in cluster indexes - so long as he could keep the order of rows returned by the database, he figured out a very fast method of doing a running total.

Obviously it was a hack and couldn't really be used, but it shows what folks can do when their favourite vendor doesn't supply critical features needed to process data in their favourite tool quickly :-)

</comment></comment></comment></comment><comment><USER16> Not really a trick in the spirit of the blog post, but I often catch people out with this, after telling them that it's legal to divide by zero in SQL:

        SELECT *
        FROM ThisTable
        WHERE EXISTS
        (
            SELECT 1/0
            FROM OtherTable
        )

<comment><USER17> What's this doing???

<comment><USER16> It's a query with an existence constraint, meaning that the query will only return data if at least 1 value is returned by the EXISTS predicate. In this case, you should get back all the data from ThisTable as long as there's a row in OtherTable.

But that's not the "tricky" bit. I'm using a feature of SQL that ignores the values in the SELECT clause when used as part of the EXISTS predicate. In the past, people used to type in the EXISTS predicate SELECT 1... because it was (wrongfully[1]) assumed to be faster than SELECT <column(s)>. This query proves that the query engine does not attempt to compute the SELECTed values. If it didn't, it would fail with a division by zero error.

[1] See Chapter 7.9:

        If the <select list> "*" is simply contained in a <subquery> that
        is immediately contained in an <exists predicate>, then the <select list> is
        equivalent to a <value expression> that is an arbitrary <literal>.

[http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt](http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt)

<comment><USER15> As an aside, quite frankly I could never get EXISTS and NOT EXISTS to work in Qlikview. I honest wish Qlikview adopted a more SQL-like exists clause.

(P.S. I used to do SELECT 1 because it was shorthand... Funny how people thought it was a magic speed up!)

</comment></comment></comment></comment><comment><USER18> In the spirit of the post...instead of:

SELECT COUNT(DISTINCT(Some_Column)) FROM Some_Table

try:

SELECT COUNT(X.Some_Column) FROM ( SELECT Some_Column , MAX(Some_Other_Column_Which_Can_Be_Maxed) [MaxValue] FROM Some_Table GROUP BY Some_Column ) AS X

<comment><USER19> Why? I can't think of any reason that should be faster. If it is, abandon your database for something reasonable.

</comment></comment><comment><USER20> What a useful blog post! Thanks for sharing.

</comment><comment><USER21> Some of the claims made by the author are just false (or glib theatricity). For example, you can definitely calculate capped running total without vendor- specific grammar.</comment>


Show HN:Inc.'s 5000 into beautiful Infographics by industry - <AUTHOR> http://www.sagarshah.co/visualization-inc-s-5000-companies-indusry/

<comment><USER2> so, is this a pie chart with hovering parts? on a quick look, I can't get this working. Also, I would be interested to hear more about your goals or the implementation choices that you did. Thanks!

<comment><AUTHOR> Basically you wont be able to click on it as I have not entered more data yet. What I did was just grab data from Inc.'s website and put it on Infogram (took 5-7 minutes). I am planning to add all data with revenue and market share but it will take some time.</comment></comment>


The CDC's Coronavirus Death Rate Estimates Are Too Low, Many Scientists Say - <AUTHOR> https://www.buzzfeednews.com/article/stephaniemlee/coronavirus-cdc-infection-fatality-rate <comment><USER2> The integrity of all new sources have been challenged to such a degree that the real loser here is the average person. There is no way for many of us to assess who to trust and we are being mired in confusion and smokescreens created by the powers that be. There is no denying that it is being done intentionally and its just sad.

<comment><USER3> It's partially the fault of the sources, but I also feel like there are unrealistic expectations. Most people I've seen discuss it online expect a lot more details with a lot more certainty than could possibly be available.

</comment></comment><comment> enahs-sf I wonder how many states are also cooking the stats amidst the flurry of reopening.

<comment><USER3> It doesn't seem like there'd be much incentive to. If they have an uncontrolled outbreak, people are gonna notice regardless of what the stats say.

<comment><USER4> You really don't see the incentive to underreport?

Just being open for business is one thing, getting workers and consumers to fulfill their roles in the system is another.

If people see the death rates are still quickly growing, they'll be less likely to resume working and spending.

<comment><USER5> I agree there's a lot of incentive to reduce the counts in states where for political reasons in party they opened up their states aggressively, one example is Georgia. Georgia has somewhat surprising overall positive counts trending down. Compare them to Alabama which has been consistently going up. Georgia has reported incorrect results and Florida fired someone working on their website. [https://www.cnn.com/2020/05/20/us/florida-georgia- covid-19-t...](https://www.cnn.com/2020/05/20/us/florida-georgia- covid-19-test-data/index.html)

</comment></comment></comment></comment><comment><USER6> The problem arises when people talk about a single "death rate" without the proper caveats. It is well understood that the infection fatality rate varies with age, from 0.001% for those under 20 to ~10% for those over 80.[1] When you apply this to the entire US population, you get ~0.7%.

[1] [https://science.sciencemag.org/content/sci/suppl/2020/05/12/...](https://science.sciencemag.org/content/sci/suppl/2020/05/12/science.abc3517.DC1/abc3517_Salje_SM.pdf)

</comment><comment><USER7> The local hospital around me has about 600 beds. And the highest number of patients they had was 54. Nurses hours were cut back severely since there just wasn't much to do in the hospital. I think the us numbers are so slow because of the highish bed to person ratio. And we had plenty of warning time to free up beds and such. I'm only aware of a few hospitals that actually had major issues.

</comment><comment><USER8> I think the problem is that tons of people get infected without bothering to get tested. It's not a simple process to get tested and if you are not very sick, you just won't bother.

<comment><USER9> In much of the country, you can't get tested unless you have symptoms, which means you've been infectious for 2–14 days.

Even now, we're still in single digits in terms of the percentage of the population that's been tested.

</comment></comment><comment><USER10> From the CDC report making those "estimates"

>The parameters in the scenarios .. Are not predictions of the expected effects of COVID-19.

They are demonstrating ways of modeling the effects based only on the the available evidence of the viruses biological and epidemiological characteristics. These can later be used in more detailed models.

The observers using them to support reopening the country are misusing them, but this article is also missing the point.

</comment><comment><USER7> The only rational option I see is Americans must be some of the healthiest people. Kidding aside this is a hard problem to solve. Some states are over reporting some are under reporting, there just isn't much consistency. And of course there is tons of political bias so who actually knows.

<comment><USER11> > The only rational option I see is Americans must be some of the healthiest > people. Kidding aside

The US was fortunate to have considerable ICU capacity. That has kept the mortality rate far lower than it otherwise would have been had the US been overloaded to the extent that some other nations were. Immense ICU capacity is one of the reasons Germany faired so well compared to most other nations in Europe.

By the time Italy and Spain were reporting a mere 20,000 cases, their healthcare systems were already crashing and they were forced to let older patients die because they didn't have the capacity to treat everyone. The same thing quickly happened in France and Britain as well, producing high mortality rates.

Also, most European nations are dominated in terms of population by one or two cities, as a percentage share of the overall population. You can see this in the list of primate cities [1]. So if your national population is 50 million and you have a primate city the size of NYC, and it gets hit hard by the virus, it's going to produce an extreme mortality rate nationally. With the US, you see a far more spread out context because NYC as one example doesn't represent nearly so large a share of the national population, with the US having few hyper dense cities vs its 330m population and few major cities in the climate zone that was ideal for SARS-CoV-2 to spread. If Los Angeles had NYC's climate, it would have produced a horrific outcome.

[1] [https://en.wikipedia.org/wiki/Primate_city](https://en.wikipedia.org/wiki/Primate_city)

<comment><USER12> I think we should set Italian stats aside.

It turns out that they used nursing homes to bed corona patients, so the residents were wiped out.

I'm curious why NY had such a bad time compared to SF. Was it because in a cold climate central heating circulated the corona virus droplets?

<comment><USER13> NY was afraid their hospitals would be overrun, so it sent their elderly covid patients back to the nursing homes. And then the nursing homes got wiped out. I'm not aware of this policy happening in SF.

<comment><USER12> Interesting, first I heard that NY copied Italy. I thought it was winter HVAC and overzealous ventilator use, but yeah, that would do it.

SF didn't really do anything, including testing, until very late, so we never needed more patient space, ironically. (We actually gave away 500 ventilators early on.)

I believe that's due to many Bay Area residents shuttling between SF and China, essentially starting a very early herd immunity in Dec. The Santa Clara dashboard makes corona look like a seasonal flu - there's numbers, but nothing of concern yet.

It was reported that at the end of NAMM on Jan. 24, planeloads of Chinese travellers disembarked at LAX, all wearing masks, so they knew what was up. (That was also the day Disney closed their Shanghai property due to corona.)

[https://www.cnbc.com/2020/01/24/shanghai-disney-shuts-to- pre...](https://www.cnbc.com/2020/01/24/shanghai-disney-shuts-to-prevent- spread-of-virus.html)

You would think closing a Disney theme park would catch the attention of the CDC or somebody in our government, yet here we are.

<comment><USER13> Don't believe the hype. The CDC and government were taking steps all through January. Hindsight is always 20/20, so one might suggest certain things could've been done earlier, but it's not as if coronavirus was being ignored.

Jan 3: CDC Director Redfield formally requests permission to send experts to China to investigate the new virus

Jan 5: CDC Director Redfield repeats formal request

Jan 6: CDC issued level 1 travel notice for Wuhan

Jan 7: CDC established coronavirus incident management system

Jan 11: CDC updated a Level 1 health notice for Wuhan

Jan 17: CDC began public health entry screening at 3 US airports that received the most travelers from Wuhan

Jan 20: Fauci announces NIH is working on vaccine

Jan 21: CDC activates emergency operations center to provide support for coronavirus response

Jan 23: CDC sought "special emergency authorization" from the FDA to allow states to use newly developed coronavirus test

Jan 27: Trump tweeted he made an offer to Xi Jinping to send experts to China

Jan 27: CDC issues level 3 travel health notice urging Americans to avoid all nonessential travel to China

Jan 27: White House Coronavirus Task Force had its first meeting.

Jan 31: Trump declared coronavirus a public health emergency, announced Chinese travel restrictions

... the actions just get more aggressive from here

<comment><USER12> Thanks for the info.

If those dates are correct for the first week of Jan., then the CDC was actually very aggressive for a govt. agency, since Wuhan was still trying to cover it up that week. (See link below.) We need to think about why it took 4 weeks for US action, and what a major trading economy like the US can do.

But ... note that December is missing from the timeline:

The first known Wuhan hospitalization was Dec. 1, and since I travel a lot in Asia, I learned of a highly contagious, deadly flu there on Dec. 15.

1) So it took over a month for the CDC to find out, despite US funding of the Wuhan corona laboratory (the epicenter.)

2) When talking with mainland Chinese people, they say Chinese social media had active topics about flu in Nov. and Dec. Something to search for in Chinese.

3) I looked at the CDC web site recently for how to report an outbreak, and there is no direct way for a citizen to do so. They require you to report it via an allowed channel. Again, something to think about.

"What did China know, and when?"

[https://thehill.com/opinion/international/490258-what-did- ch...](https://thehill.com/opinion/international/490258-what-did-chinas-xi- jinping-know-and-when-did-he-know-it)</comment></comment></comment></comment></comment></comment></comment>


Ask HN: Labeling new datasets as a bootstrapped startup - <AUTHOR> Hi all. Im trying to validate and PoC an idea for a tool/product that centers around the complexities of local city traffic signage. The central idea is to simplify parking in cities by highlighting things on a map like "Street Cleaning Schedules", "Tow Zones with variable parking times", "Parking unavailable due to long-term construction" etc. This product will require some form of Data analysis and ML. The initial dataset I was planning to use was Google street views for larger cities and their satellite towns.
My question is, as a lean bootstrapped idea, what services are available to me in order to help label data that won't break the bank? This is not proprietary or overly complicated data but it can require several forms of labeling. Im thinking just the basics to start such as "Has street sign (yes/no)", "Street Cleaning sign (yes/no)" (if has a street sign is Yes), etc. Eventually I'll need to feed that labeled data into image processing pipelines that can extract what the signs actually enforce.
I know there are various companies out there like AWS Turk and others that employ teams to do this but Im not sure I want to sign an AWS contract before I've even validated the idea. Has anyone used these services before? If not, what are the alternatives?
All help is so much appreciated. Thanks in advance! <comment><USER2> There are a bunch of services that will let you do this.

At your stage, probably put up some notices in your local university offering an hourly $ amount (or beer/coffee) in return for some manual labour.

Also look at Figure Eight, CrowdAI, Eureka, etc. There are a lot of competitors in this space.

If you're looking for tools to help with this, look at [https://prodi.gy/](https://prodi.gy/).

Amazon Turk doesn't require a contract I think. There are a lot of other freelancing platforms out there where you can find low-skilled labour.

Feel free to contact me (details in profile) to discuss more. I am exploring this area at the moment in any case.

<comment><AUTHOR> I checked out CrowdAI and it appears they mainly focus on road and building features but are open to work with customers to build out new features? Im kind of confused by that one actually.

Prodigy looks interesting if I end up going down the "do it myself" route though admittedly I'm no expert in ML yet so I was hoping to focus most of my attention there but maybe labeling my own data to start is the way to go?

Figure Eight reminds me of your typical "consulting while using our tools in the background" kind of service which might be a way to go.

Im just talking out loud so please correct me if I got any of that wrong :)

</comment></comment><comment><USER3> > The initial dataset I was planning to use was Google street views

Does the Google Streetview license allow this? It could become a derived dataset of a licensed database.

[https://www.mapillary.com/datasets](https://www.mapillary.com/datasets) is in a similar business. They have pre-labelled datasets but also work with developers and researchers to create more labels. There's [https://www.mapillary.com/app/marketplace](https://www.mapillary.com/app/marketplace) to submit tasks.

<comment><AUTHOR> [https://www.mapillary.com/dataset/trafficsign](https://www.mapillary.com/dataset/trafficsign)

Found that at mapillary and going to look into it, could be useful!

</comment></comment><comment><USER4> Almost every city and state DOT already has this data, because they create it when they place the signs. If your business revolves around having unique data, this is the wrong path. But if the business model is about what you do with the data, I'd skip all efforts to gather it yourself and just get it from your local jurisdictions.

<comment><AUTHOR> Yea it's 100% what Im doing with the data, not curating a unique dataset myself.</comment></comment>


Where does Google actually say that they won't read Gmail or Google Docs?  - <AUTHOR> http://blogs.law.harvard.edu/philg/2011/11/03/where-does-google-actually-say-that-they-wont-read-gmail-messages-or-google-docs/ <comment><USER3> the tl:dr of this comment thread is that most HN'ers have sadly taken the question too literally and are commenting on the fact Google reads emails with a definition of "read" being that of "parse" (ie computationally). I think most people know and expect they obviously do that, for advertising targeting etc. I can't believe people are shocked or surprised at that.

Which is a shame because I think it's pretty clear the OP is actually asking about Google _employees_ reading your email, which is a valid question but sadly lost in the fight over semantics. :/

<comment><USER4> > Which is a shame because I think it's pretty clear the OP is actually asking > about Google employees reading your email

Unless you run your own smtp and pop/imap mail servers, there is no way to be sure no one will read your emails. Even then, your servers could get hacked or maybe your contacts use a hosted solution (and most people do) that could get hacked, there is again no guaranty. The only work around is to fully encrypt your emails, even then someone could steal your private keys, or you could be using an unpatched encryption software etc.

It's a matter of trust and time saving. Do you trust google, yahoo employees etc to not spy on you more than you trust your ability and your contacts ability not to get your mail servers hacked? How much of your time and money are you ready to waste maintaining such a server?

<comment><USER5> Except that Lucent have no real reason to put firmware in a fiber switch to read your email as it goes down the tubes.

A company that makes all it's money from targetted ads - <USER2>

<comment><USER6> This is getting off-topic, but did you forget about AT&T's secret rooms that were recording internet traffic? It's not just Google that wants to read your mail. Governments want to spy on their citizens, and ISPs are (allegedly) cooperating in the effort.

<comment><USER3> sure but that's basically a "constant" in that it's going to happen regardless of where you receive your emails.

</comment></comment></comment></comment></comment><comment><USER7> The real question is not whether google, dropbox or anyone else has explicit "policies" about reading content from users. They're just going to say whatever makes them look good from a marketing and legal perspective. Companies lie all the time, systematically and comprehensively. They act in their own self-interest period. From their point of view it is merely a question of how much they can get away with.

The more interesting question to explore is how much we actually trust these companies with our personal information and how do we decide when we reach a limit to our trust.

Do I believe that google can read my gmail? Yes. Do I think this capability will cause harm to come to me as a result? Hell no. There's nothing "in it" for them to do so for me or millions of others. I "trust" them for this reason.

<comment><USER8> Until there is something in it for them.

It's a pretty classic security question, sometimes usability means that you accept zero security and completely share your private correspondence with the world.

I think trust is the wrong word, you are really just accepting the risk of sharing all your data for the benefits because the risk is low.

<comment><USER7> Yes, I agree "trust" is the wrong word. It is perhaps better to think of it as a trade-off of some risk for some convenience and that's OK.

In any case, back to the OP's question, the stated "policy" about privacy means virtually nothing unless there exists a third party who can effectively verify that policy.

<comment><USER8> Yes, in my experience on either side of those "policies" they really do mean nothing.

You just have to assume that all the data you share is no longer yours and will be used however the company sees fit.

I am amazed by the number of people who assume that "that would be bad PR if they got caught" is good enough security to protect data they consider sensitive. Especially when the PR damage historically has been very low, especially if you are a sexy and loved company like facebook or google.

</comment></comment></comment></comment><comment><USER9> He answers his own question in the article. "[https://mail.google.com/support/bin/answer.py?hl=en&answ...](https://mail.google.com/support/bin/answer.py?hl=en&answer=1304609) says that the answer to "Is Google reading my mail?" is "No" but doesn't elaborate ..."

So, he's looking for a more elaborate answer? This is PhilG, so I should cut him some slack, but it seems pretty clear to me.

<comment><USER10> Misleading.

What that document actually says is this:

 _No, but automatic scanning and filtering technology is at the heart of Gmail. Gmail scans and processes all messages using fully automated systems in order to do useful and innovative stuff like filter spam, detect viruses and malware, show relevant ads, and develop and deliver new features across your Google experience. Priority Inbox, spell checking, forwarding, auto- responding, automatic saving and sorting, and converting URLs to clickable links are just a few of the many features that use this kind of automatic processing._

In other words, Google IS most certainly "reading" your email, at the very least with computers, for various purposes.

<comment><USER11> I think there is a basic disconnect.

There is no way for a computer to anything with an email (save it, give it to you etc) without 'reading' it's contents. So the only meaningful thing for people to worry about is decoding the content which IMO means a spam filter does not qualify as decoding. Nor does displaying adds for tires if the word cars shows up in your email.

<comment><USER12> So when you visit my website and I see Google pushing ads to you for Justin Bieber posters and pillowcases <http://www.amazon.com/dp/B003FCXDRM/> we'll know Google isn't reading your Gmail.

... or are they? ;-)

<comment><USER13> Google _does_ read your email for ads, and that fact is spelled out in their answer quoted above. AdWords/AdSense doesn't only apply to the content of the site you happen to be on, but is an aggregate of all they've learned about you over time. This means that data they collect from relevant keywords in an email _is_ used to show relevant ads to you on a separate website.

I've never used AdSense so I don't know how it works. Is it possible for the site owner to read the ads that are displayed to you on a website? Does Google "leak" collected ad data to other websites which use AdSense? Or are these ads typically displayed in something like an iframe, where possibly the "host" website can't read them?

<comment> X-Istence Google's Ads are served in an iframe.

<comment><USER13> I suspected as much, but as I've never used them I wasn't sure. Thanks for clarifying.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> Google made a related statement last year when they fired an employee for unauthorized access to private data including emails.

"We carefully control the number of employees who have access to our systems, and we regularly upgrade our security controls–for example, we are significantly increasing the amount of time we spend auditing our logs to ensure those controls are effective. That said, a limited number of people will always need to access these systems if we are to operate them properly ..."

[http://techcrunch.com/2010/09/14/google-engineer-spying- fire...](http://techcrunch.com/2010/09/14/google-engineer-spying-fired/)

</comment><comment><USER15> The same can be said about any IT department. One of my friends sent made the news papers over sending a racially shared joke through the email system to the "boys joke list" in a rather large and respected company.

Turns out some people in the IT department where also helping themselves to the list as a source of entertainment, one of them took extreme offense to the joke and reported it.

After the thing went through its investigation process it was uncovered that the majority of disliked staff in the department where infact reading everyones email. Most of them had set up notification filters to flag emails with there names to see what people where saying about them.

Now the question is how common is this considering how easy it is to do... and how the whole system more or else runs off trust?

I'd say if you have sensitive stuff going through your email that you don't want external parties to see, set up your own server and find someone who isn't petty to look after it.

<comment><USER16> For an IT department the expectation is that people will read your email. I honestly don't think what you describe is rare at all, based on stories I've heard.

And if you work for the gov't your email can often be requested by citizens via FOIA.

<comment><USER15> I was working for a government agency in New Zealand when they introduced a law that meant every email became a government communication and you couldn't delete anything. I usually operate on an empty inbox principal, it was a total nightmare for me. I folded that contract as soon as my term was up.

<comment> X-Istence Moving it to another folder wouldn't have sufficed?

<comment><USER15> Yep, but when you get hundreds of emails a day, you end up folders inside folders. Not being able to trim excess is a total pain. Not to mention the pain of having limited mailbox sizes, so you have to ring the infrastructure vendor to archive your mail. And then put in requests to get older archives so you can search for stuff.

</comment></comment></comment></comment></comment><comment><USER17> Has there been anyone working to make a comprehensive locally-hostable webmail/weboffice suite that replicates Google's functionality? I can imagine that this would be quite poular. Sure, some of the features like Priority Inbox might require complex heuristics, but things like labels should be easy to add support for.

<comment><USER18> Google used to sell hardware appliances, before they realized they would actually prefer to read all your documents.

</comment></comment><comment><USER19> How many individuals at google have the ability to read a particular person's personal data? If I'm a moderately important person[1], how many individuals, in or outside of google, will have access to any of that data over the life of it? And, how does google internally police against misuse? And, who watches the watchman?

[1] say, 1 in a thousand, of which google has 260,000 such gmail users

<comment><USER20> Probably about 50 altogether. Each of us promises to never ever look at personal data, before being granted privileges. And I'm pretty sure all those promises are serious.

I don't know how enforcement works, though a single offense would be firing- worthy.

<comment> X-Istence <http://www.pcmag.com/article2/0,2817,2369188,00.asp>

</comment></comment></comment><comment><USER21> From the comments:

> here's a strange little story that happened to me a while ago – I set up a > gmail account to deal with nigerian letters and such (I wanted to collect > some data to report the spammers/thieves, without compromising my actual > e-mail address in the process). I set this up with a fake username > (something like george.thompson or so) and a password which included the > word "nigeria" in it. Lo and behold, after my first login (before > sending/receiving any mail) the targeted advertising in gmail included some > nigerian ads (nigerian holidays, nigerian business bureau, etc). > coincidence?….

If true, it seems they matched ads to the guy's password. Which means they needed to be able to read it plain text. The plain text should only ever live long enough to create or match with a hash.

<comment><USER22> Don't be too quick to judge - maybe he had been searching for Nigerian related things before creating the account.

<comment><USER21> You, and the other respondents, are probably right. Sometimes I'm too quick on the draw.

<comment><USER23> I seriously doubt google would persist any parts of passwords in plain text.

</comment></comment></comment></comment><comment><USER24> What about Facebook?

A few weeks ago I had a situation where Facebook contacted me about a job, and it appeared that Facebook may have been reading (or at the least mining) private messages related to my startup (<http://news.ycombinator.com/item?id=3035376>). A Facebook employee replied to the thread but wouldn't provide details.

<comment><USER25> (I am neither a Facebook employee nor can I speak for them.)

I really think that was just coincidence. Considering how many people talk about startups over Facebook messages and how many people have been contacted by Facebook recruiters, there's bound to be at least one person who was contacted by a Facebook recruiter a day or two after talking about startups. It'll seem mighty suspicious to that person, and they'll blog about it. Congratulations, you're the lucky one.

Companies that big just generally don't _do_ things like read private e-mail, because they know it'll get out somehow - disgruntled employee, whistleblower - and the damage to their reputation is totally not worth whatever they can gain from it.

<comment><USER26> The thing people don't realize is that when contacted by a big company for a job, usually it's a bot that just sent hundreds of emails.

Receiving an email from Google or Facebook is nothing special.

<comment><USER24> What data sources does Facebook use to trigger the email?

And are emails manually approved before they go out? If so, does that mean a Facebook employee looks at the data source that triggered email, which in some cases may be the user's private messages?

<comment><USER27> It seems to be a large leap to go from "sends lots of emails" to "automates finding people to recruit." There is a similar leap from "automates finding people to recruit" to "uses private messages as signal."

</comment></comment></comment></comment></comment><comment><AUTHOR> philg's comment:

"Folks: The problem with Google promising to hold "personal information" confidential is that a document may not meet the definition of "personal information" given in [http://www.google.com/intl/en/privacy/faq.html#toc- terms-per...](http://www.google.com/intl/en/privacy/faq.html#toc-terms- personal-info) . For example, if you write down some ideas for a new product and keep it as a Google Doc there may be nothing in the document that identifies you as the author and therefore it might be okay for Google to read or distribute the document.

Generally speaking the Google privacy docs address a separate issue from the question of keeping email or documents confidential. They are about the question of whether your identify is kept confidential when you're browsing around the Web and Google is figuring out what your interests and demographics are. Worthy stuff to be writing about, no doubt, but it doesn't shed much light on the subject of whether a Google employee can copy and paste paragraphs from your email messages or Google docs."

[http://blogs.law.harvard.edu/philg/2011/11/03/where-does- goo...](http://blogs.law.harvard.edu/philg/2011/11/03/where-does-google- actually-say-that-they-wont-read-gmail-messages-or-google- docs/#comment-190817)

</comment><comment> mike-cardwell Google should provide the option of automatically encrypting your incoming email with your public PGP key, if you provide one. They could similarly automatically encrypt your Sent folder (not the actual sent messages). That way, even if your account is compromised, or emails subpoenaed, they can't read the emails without your private key.

That's exactly what I do on my system anyway. I host my own email as well though: [https://grepular.com/Automatically_Encrypting_all_Incoming_E...](https://grepular.com/Automatically_Encrypting_all_Incoming_Email)

</comment><comment><USER28> How could they? They have ~24,000 employees and 260 million Gmail users.

<http://www.quora.com/How-many-employees-does-Google-have> and <http://en.wikipedia.org/wiki/Gmail> respectively.

(Although off the top of my head, they do say they will automatically scan all email for the purposes of advertising, spam filtering, etc. and the debate used to be "does this count as reading if no human is involved?").

<comment><USER29> That just proves that they can't read everybody's email.

</comment></comment><comment><USER30> It is probably pointless for Google to use humans to read mails, especially the mails of regular people. And it is certain that they use automated processing of the information in the mails.

A most relevant question would be: Does Google match the profiles of its mail users to high profile people, such as company managers or other rich people who can be easily found on, for example, Linkedin? If so, does Google automatically filter their email in order to predict, for example, the stock market?

</comment><comment><USER31> I don't think terms provide too much assurance - maybe they're only really useful if legal recourse ensues.

There will always be the possibility of a rogue employee who goes out of his or her way to read data that doesn't belong to them.

In the case of Dropbox - to prevent this possibility - we can encrypt our data if we choose to (I've successfully used 'encfs' in the past). In the case of Google's email and document services I don't think this is possible?

<comment><USER32> Or use SpiderOak, Tarsnap, or Wuala, all of which handle encryption natively.

</comment></comment><comment><USER33> <http://www.wired.com/threatlevel/2010/09/google-spy/>

[http://gawker.com/5637234/gcreep-google-engineer-stalked- tee...](http://gawker.com/5637234/gcreep-google-engineer-stalked-teens-spied- on-chats)

Even if they say they won't it looks like they are lacking some internal controls.

<comment><USER34> Rogue employees are a fact of life. That they were caught shows that Google's internal controls are (at least reasonably) effective.

</comment></comment><comment><USER35> Right in their privacy policy (<http://www.google.com/intl/en/privacy/privacy- policy.html>):

We may collect the following types of information:

      <snip>

User communications – When you send email or other communications to Google, we may retain those communications in order to process your inquiries, respond to your requests and improve our services. When you send and receive SMS messages to or from one of our services that provides SMS functionality, we may collect and maintain information associated with those messages, such as the phone number, the wireless carrier associated with the phone number, the content of the message, and the date and time of the transaction. We may use your email address to communicate with you about our services.

      <snip>

In addition to the above, we may use the information we collect to:

Provide, maintain, protect, and improve our services (including advertising services) and develop new services; and Protect the rights or property of Google or our users.

If we use this information in a manner different than the purpose for which it was collected, then we will ask for your consent prior to such use.

<comment><USER10> Uh, that privacy policy (or at least the snippets you posted) provide NO explicit assurances that Google is not reading your stuff. At all.

It tells you what they WILL do. And it says if they decide to "use" (vague) "this information" (vague) in other ways, they'll ask for consent. Does simply reading it count as "using" it? Google doesn't say.

I wouldn't trust these assurances at all.

</comment></comment><comment><USER36> I am more concerned about Google making an arbitrary decision to delete my Gmail account.

<comment><USER37> That's why I got my own domain. Coupled with offlineimap (or any other mail backuping software) running regularly, I can move from Gmail to any other provider in the time it takes to create a new account and update the MX records.

</comment></comment><comment><USER38> Does end to end email encryption get any traction? I remember PGP supposed to do it. Not sure how far that went.

It seems for company using third party email hosting, it shouldn't be too inconvenient to set up a company-wide encryption on its emails.

</comment><comment><USER39> It's not google you have to fear.

It's the warrantless backdoors installed for most government entities.

What you think is harmless to discuss is probably being auto-indexed by them to use against you if you ever become a "problem" (ie. protest war, etc.)

</comment><comment><USER40> Aren't emails subject to the legal definition of private correspondence and therefore protected from this kind of behaviour? (I believe this is the case in France, Italy, Spain or Germany)

</comment><comment><USER41> Store your plaintext email on someone else's server. Act surprised it's not private.

<comment><USER42> what makes you think it's plaintext?

<comment><USER41> If it weren't, then it wouldn't be an issue. Right?

And what makes me think that is that I suspect 99.999% of email is in plaintext. I have no foundation for this. Just a gut feeling.

</comment></comment></comment><comment><USER43> <http://gawker.com/5637234/>

</comment><comment><USER44> Aren't there some laws about secrecy of correspondence that already cover this?

</comment><comment><USER45> No Google doesn't read your email. In fact their spam detection just works by flipping the worlds luckiest coin at every incoming email.

</comment><comment><USER5> Where does it say on harvard.edu that they wont kill students and harvest their organs?

Edit: Oh sorry forgot that people in the land of the free don't have a human rights act preventing Google reading their email (not sure about harvesting organs)

</comment><comment><USER46> The Gmail man reads your email and docs.

<http://www.youtube.com/watch?v=yXqrTfOWx60>

</comment><comment><USER47> They're definitely reading my email.

I know this for a fact as the last time I mailed them they replied.

</comment><comment><USER48> This should be covered by the statement "don't be evil"

<comment><USER49> If the FBI comes knocking on Google's door and says they have a terrorist under surveillance, and they need to read his email in order to save a busload of schoolchildren from a bomb, should Google let them do it?

Is it evil to read a bad person's email, or is it evil not to read it?

That's why a clearly stated policy that says exactly when Google will or will not let a human look at your documents and email is important. "Don't be evil" is cute, but it's not a policy statement.

<comment><USER48> _If the FBI comes knocking on Google's door and says they have a terrorist under surveillance, and they need to read his email in order to save a busload of schoolchildren from a bomb, should Google let them do it?_

From the Policy:

 _[When] We have a good faith belief that access, use, preservation or disclosure of such information is reasonably necessary to (a) satisfy any applicable law, regulation, legal process or enforceable governmental request_

About how and when they access your information:

 _We restrict access to personal information to Google employees, contractors and agents who need to know that information in order to process it on our behalf. These individuals are bound by confidentiality obligations and may be subject to discipline, including termination and criminal prosecution, if they fail to meet these obligations._

That "need to know that information" seems to restrict casual access, with the threat of termination to back it up. All of these are more specific declarations in line with a general "don't be evil" policy.

<comment><USER50> A very real threat of termination: <http://www.pcmag.com/article2/0,2817,2369188,00.asp>

</comment></comment></comment></comment><comment><USER26> The question is wrong. Any email provider will read your email, at least for spam detection.

The question should be - does Google keep a profile on you based on the content of your emails?

I'm pretty sure they are able to serve ads without keeping a profile, making the decision solely on the content of the email being read. However, that may not be the case.

<comment><USER51> _The question is wrong. Any email provider will read your email, at least for spam detection._

He's clearly talking about _humans_ reading your mail. No email provider has humans reading your mail for purposes of spam detection, at least not in the usual case.

The question is perfectly valid: seeing that other companies have clear policies on the extent to which employees can access your data, how come Google doesn't have any such policies?</comment></comment>


Google Fiber: Working with content providers to minimize buffering - <AUTHOR> http://googlefiberblog.blogspot.com/2014/05/minimizing-buffering.html

<comment><AUTHOR>
      We give companies like Netflix and Akamai free access to space
      and power in our facilities and they provide their own content
      servers.

      We don't make money from peering or colocation; since people
      usually only stream one video at a time, video traffic doesn't
      bog down or change the way we manage our network in any meaningful
      way — so why not help enable it?

      But we also don't charge because it's really a win-win-win situation.
      It's good for content providers because they can deliver really
      high-quality streaming video to their customers... It's good
      for us because it saves us money (it's easier to transport video
      traffic from a local server than it is to transport it thousands
      of miles). But most importantly, we do this because it gives
      Fiber users the fastest, most direct route to their content.</comment>


Ruby 2.1 In Detail - <AUTHOR> http://globaldev.co.uk/2014/05/ruby-2-1-in-detail/

<comment><USER2> Funny! I actually came across this article yesterday when trying to look into the current state of Ruby's global method cache. I had a question about the write-up, so maybe I can ask it here!

The page says:

> _This is now no longer an issue, Ruby 2.1 uses a method cache based on the > class hierarchy, invalidating the cache for only the class in question and > any subclasses._

How does this work with an _instance_ of a class? Does it invalidate only that instance's method cache, or all instances in that class? If I do DCI-style:

       @user.extend(PasswordConfirmable)

Does that blow away the cache for all User instances, or just that one?

<comment><USER3> Just that one. The "class" that the method is attached to here is the metaclass of `@user`.

That class is below `User` in the hierarchy:

        class User; def foo; "foo"; end; end
        # => :foo
        module M; def bar; "bar"; end; end
        # => :bar

        u = User.new
        # => #<User:0x007f2e1abc9e00>
        u.extend M
        # => #<User:0x007f2e1abc9e00>

        u.singleton_class.ancestors
        # => [#<Class:#<User:0x007f2e1abc9e00>>,
         M,
         User,
         Object,
         PP::ObjectMixin,
         Kernel,
         BasicObject]

<comment><USER4> Your example isn't conclusive that what you're saying is true. Your example is just showing that the extended module only appears in the instance's singleton class, this has nothing to do with method cache. It's how the method cache is implemented that matters. The new method caching changes can't be seen from user code.

I'm curious as to whether the ruby-core team implemented method caches for every Object in the system (eg: every instance of User would then have its own method cache because they all would have their own singleton classes) or just objects that are instances of the Class class. I seem to doubt that the former happened because it seems like it would be expensive to maintain a cache for _every_ object, that means a normal Rails request that generates 65,000 strings would all have their own method caches. Only instances of the Class class having their own method caches sounds more reasonable from a performance standpoint to me.

For what it's worth, I've been trying to get an answer about this from a ruby- core member, although my attempts have been quite lazy through Twitter.

<comment><USER3> > Your example is just showing that the extended module only appears in the > instance's singleton class, this has nothing to do with method cache. It's > how the method cache is implemented that matters.

The method cache works by invalidating anything below the class(es) whose methods have changed on the ancestor chain.

You're right that it can't be seen from user code, but if you take the implementers at their word about how it works, then this should be a perfectly convincing demonstration.

Or, if you don't believe me, read the source yourself:

[https://github.com/ruby/ruby/blob/ba6b0acdb6c371f9f7150bed6f...](https://github.com/ruby/ruby/blob/ba6b0acdb6c371f9f7150bed6f555c7f34fe3781/vm_insnhelper.c#L823-L841)

[https://github.com/ruby/ruby/blob/1aa54bebaf274bc08e72f9ad38...](https://github.com/ruby/ruby/blob/1aa54bebaf274bc08e72f9ad3854c7ad592c344a/vm_method.c#L78-L94)

The part that clears the class hierarchy is here:

[https://github.com/ruby/ruby/blob/1aa54bebaf274bc08e72f9ad38...](https://github.com/ruby/ruby/blob/1aa54bebaf274bc08e72f9ad3854c7ad592c344a/vm_method.c#L56-L61)

Alternatively, see Koichi's presentation at RubyKaiga:

[http://www.atdot.net/~ko1/activities/RubyKaigi2013-ko1.pdf](http://www.atdot.net/~ko1/activities/RubyKaigi2013-ko1.pdf)

</comment></comment></comment></comment><comment><USER5> There's some great little improvements in there, to_h, required keywords and being able to decorate methods are all things that I regularly want. They're really making good on their philosophy of optimising for developer happiness :)

</comment><comment> grey-area Hurray, they fixed a little bug I reported: Dir glob returns composed characters. I hit this bug a year or so ago when processing files on OS X.

Thanks to the Ruby team for all the hard work on 2.1, it looks like there's a lot of little changes in there.

</comment><comment><USER6> Does Ruby truly follow semantic versioning now? I still see the patch version.

        $ ruby -v
        ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-darwin13.0]

<comment><USER7> No they don't exactly follow semantic versioning, which is as useful as not following semantic versioning at all.

Specifically, I believe they're planning on breaking API compatibility within major version numbers.

<comment><USER8> > Specifically, I believe they're planning on breaking API compatibility > within major version numbers.

Yes. Full details: [https://www.ruby-lang.org/en/news/2013/12/21/semantic- versio...](https://www.ruby-lang.org/en/news/2013/12/21/semantic-versioning- after-2-1-0/)

<comment><USER9> > Semantic Versioning starting with Ruby 2.1.0

> MINOR: increased every christmas, may be API incompatible

That is not semantic versioning.

<comment><USER8> I know :'(</comment></comment></comment></comment></comment>


While you pay into your 401k, CEOs collect $250k in monthly pension payments - <AUTHOR> http://www.latimes.com/business/lazarus/la-fi-lazarus-equifax-pensions-20171003-story.html <comment><USER2> > Why such a sweet deal? Obviously it's no easy task running a major > corporation, so a commensurately meaty paycheck is to be expected.

Statements like this really grate on me. Corporations large enough to afford seven-figure annual salaries already have dozens, if not hundreds of great, capable employees that could do at least as good a job as these guys for 100th of the cost.

The obscene salaries are purely the result of collusion.

<comment><USER3> Are you saying the company should prioritize reducing CEO salary by finding a "potential" cheap guy over someone who is already doing a good job ?

Din't Ben and Jerry tried something like this and nearly killed their company twice ?

<comment><USER2> No, I'm saying that CEOs aren't beings with magical powers. Any company of a sufficient size is bound to have a number of people within their ranks that could fill the CEO role.

CEOs justify their high salaries by claiming they are one-of-a-kind individuals. I'm saying this claim is probably BS and that any sizable company is filled with people who are capable of performing at the level of most of these executive candidates. The fact that the company would also save money is just a nice side-effect.

</comment></comment></comment><comment><USER4> CEO? Why bother with the MBA? You can pull that kind of scratch (and more) down being a librarian in New York City, or an educator upstate, or a firefighter.

[http://seethroughny.net/pensions/](http://seethroughny.net/pensions/)

Or if you fancy over half a million a year, just be a retired history professor.

EDIT: I'm an order of magnitude out. $250k/_month_ in TFA. I misread that.

</comment><comment><USER5> Such a shame reading news like this. I'd like to point out that this isn't really even a conservative vs liberal view anymore, regarding corporate greed, corporate malpractice, etc. Even Steve Bannon has said this.

While we definitely will always have culture battles to fight between both parties, the reality is the Republican Party was essentially "captured" by the Koch Brothers starting in the 1980's, and it made it drastically shift to the right on corporate issues. The Tea Party, which in the end is a force for good (really shocked me to write that), was able to internally destroy the Republican Party and thus undo most of it's absurd corporate agenda, but by that time Citizen's United has passed and perhaps half of the corporate shills or so fled to the Democratic Party, which is now also captured.

In conclusion, it will be the extreme wings of both parties, Tea Party on the right and Socialists on the left, that will save the economic system, but this solution wasn't ideal and there could have been a better way.

<comment><USER6> I found this comment very fascinating and would love to learn more about the recent history of the two major American political parties, do you have any suggestions for literature on this topic?

<comment><USER5> This book by a couple of very good liberal sociologists (whose other work I also recommend): The Tea Party and the Remaking of Republican Conservatism [https://www.amazon.com/dp/B005PS3CFM](https://www.amazon.com/dp/B005PS3CFM)

The most fascinating part was probably that Tea Party individuals actually adopted and modified a lot of liberal tactics, specifically those from the book "Rules for Radicals". Now we see a trend where liberals and people like the Democratic Socialists of America are adopting those modified tactics BACK INTO the Democratic Party.</comment></comment></comment>


Why HN is broken - <AUTHOR> HN is broken for the same reason Google is. When something is more popular it does not mean it is more interesting. In fact, the opposite. The most popular links are bland and uninteresting. Upvotes are a great way to filter out spam, but not a good way to select for interestingness. <comment><USER2> Flagging content is broken too - or at least people's behaviour in flagging. Take for example, there was a very interesting link to Teacups and Storms re: gender politics. I had wanted to continue reading what other HNers thought about the article, not 40 minutes after the link was posted. But I couldn't find it anymore.

Then I revisited my history to find that the story is now marked [dead]. Seriously?

<comment><USER3> I thought this was one of the more refreshing articles relating to gender issues to appear on HN, of late. Perhaps others felt it was just rehashing the same, tired issues. Who knows.

</comment></comment><comment><USER4> Complaining about something without giving (or better yet, creating!) a solution isn't very productive.

</comment><comment><USER5> Paul Graham has written essays about this, so I feel like you're muttering to the choir here.

That said, brilliant demonstration of a known failure mode for HN: using copywriting tactic head lines.

        Why I ...
        How I ...
        What You ...

</comment><comment><USER6> That's interesting. I suggest we get rid of downvotes altogether, and replace upvotes with a tally of click-throughs. If you really find a topic of interest, surely reading it is the one true upvote. To then come back and downvote it can register your disapproval, but not your lack of interest! No interest=no click-through=no upvote.

<comment><USER7> Isn't that just a vote on the effectiveness of the headline?

<comment><USER6> An interesting headline might do it I suppose. I choose more on the subject. Either way "interest" is noted, not just votes recording whether you agree or not with the views presented.

</comment></comment></comment><comment><USER8> If we truly value discussion above all else, then the optimum metric for link quality should be the ratio of comments to click-throughs. When people are going to the article and then coming back to discuss it, the article must be a quality cause for discussion.

</comment><comment><USER9> > The most popular links are bland and uninteresting

Not to the people who upvoted them. Come to terms with the fact that what HN likes changes as HN's users change. Look at all the accounts from years ago that were rich with karma but are gravesites now.

</comment><comment><USER10> I read the frontpage and go immediately to new. Simple solution. If you read "new" you can upvote an interesting link to the frontpage. It's the same on reddit.

</comment><comment><USER11> Why do you use a question on the headline if you don't answer it on the post? I read:

 _Why HN is broken? Because I think it is._</comment>


Ask HN: B2B Pricing - <AUTHOR>

Hello, after reading through HN and viewing a recently submitted post regarding pricing for a user-focused, I had a question.
My startup is close to finalizing a business-to-business product and has some close connections within the industry that will result in likely first sales. However, we've been having trouble pricing the service. Every competitor on the market requires a request for a service quote, so price benchmarking is difficult. Also, getting this business to say what they're currently paying is unlikely.
Without getting too specific, we're developing an atmospheric science-based technology that is extremely disruptive. For example, the "error" reading for our product is nearly 10x less than current methods. However, it seems illogical to charge a price premium in suit.
As far as cost to provide the service, it is minimal. Should we charge on the hypothetical lower end of the spectrum for hopes of easy adoption, of charge a slight premium for quality? I have a monthly rate in my head that makes sense to me, but I have no idea if it is too high or too low.
On another note, this was a Ph.D. dissertation technology, and adjusting it for direct consumer use is not possible. <comment><USER2> There's no getting around understanding what the value of your technology to your customers is.

The starting point is to find out the raw ROI. So if they use your technology, how much money will they make or save? How does this compare to your competitor? So I would say, you need to take the final step and translate that 10x reduction in error to dollars.

I don't care if my GPS is exact, but to an airline company this is critical and they would spend well to make sure it takes into account the curvature of earth, etc...

But the raw ROI is just the starting point. Companies also pay a lot of money for other intangible features. There are some companies that spend millions on Oracle when they could probably get by with free open-source. Clearly Oracle is not competiting on price here, but on a feeling of 'security', knowing that my site will not go down and if it does, I have someone to yell at and come fix it.

If your product is truly revolutionary, maybe there are early adopter advantages. Could early adopters of your product steal revenue from competitors by using your tech? Price that in.

If your product is truly revolutionary, this final price may end up being pretty high and give some customers pause. A good approach would be reduce the price of your product for a pilot phase of say months. In return for reducing your price, you can ask for some specific feedback or a testimonial. Make sure they understand you aren't just reducing the price in return for nothing.

After they go through the pilot and have a real tangible sense of the value of your product, they may be willing to pay your high price you initially calculated.

</comment><comment><USER3> First, I'll recommend the same free e-book here that I recommended over there: <http://www.neildavidson.com/dontjustrollthedice.html>

It will give you a good framework for figuring out your pricing.

If I were in your shoes, I'd be asking myself what the value to the customer is. What's their ROI going to be?

Once you do get in the door with somebody, I suspect you'll hear what they used to pay your competitors.</comment>


What Are Fintechs and How Can They Help Small Business? - <AUTHOR> https://www.nytimes.com/2020/06/10/business/economy/fintechs-loan-small-business.html <comment><USER2> Has too big to fail become too big to lend?</comment>


Show HN: ReboundRobots a game to keep you sharp - <AUTHOR> http://reboundrobots.com <comment><AUTHOR> Hey Fellow HNers,

I built reboundrobots.com utilizing angular6, Spring Boot 2, http2, letsencrypt and websockets for multiplayer.

It was a lot of fun building this game. Check it out and let me know what you think.

It's a great game to keep you sharp.</comment>


Kingshard-A High-Performance Proxy for MySQL Powered by Golang - <AUTHOR> Overview
kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read/write sqls. Now it supports basic SQL statements (select, insert, update, replace, delete). The most important feature is the sharding function. Kingshard aims to simplify the sharding solution of MySQL.
Feature
splits reads and writes
sharding table across multiple nodes
client's ip ACL control.
supports prepared statement: COM_STMT_PREPARE, COM_STMT_EXECUTE, etc.
MySQL HA
Install
1. Install Go
2. git clone https://github.com/flike/kingshard.git src/github.com/flike/kingshard
3. cd src/github.com/flike/kingshard
4. source ./dev.sh
5. make
6. set the config file (etc/multi.yaml)
7. run kingshard (./bin/kingshard -config=etc/multi.yaml)
https://github.com/flike/kingshard <comment><USER2> Posts without URLs get penalized, so you'd be better off posting this with a URL, then adding your text as a first comment to the new thread.</comment>


Pebble iOS App Stuck "in review" for 43 days, Pebble Time app for 12 days - <AUTHOR> https://www.kickstarter.com/projects/597507018/pebble-time-awesome-smartwatch-no-compromises/posts/1251424?ref=backer_project_update

<comment><USER2> Man, the [edit: Apple's] Facebook Page is a mess. Pebble fans posting off- topic comments on all the posts... I feel like this is not an effective way to promote goodwill with a competitor and gatekeeper.

<comment><USER3> I'm not a competitor or gatekeeper, I'm a user. Frankly, my next phone is unlikely to be an iPhone.

<comment><USER2> But they're not encouraging users to switch to a different phone, they're encouraging users to harry Apple.</comment></comment></comment>


Abusing WebRTC to reveal coarse location data in Signal - <AUTHOR> https://medium.com/tenable-techblog/turning-signal-app-into-a-coarse-tracking-device-643eb4298447 <comment><USER2> The only universal fix I can think of for this class of attacks is to have routers bound latency to a lower limit (eg. 200ms), with fixed latency buckets (eg. 500ms granularity) when it goes beyond that.

That is, no traffic would traverse the router in less than 200ms, and every other flow would be fixed at 700ms, 1200ms, 1700ms, etc amounts of latency. Tweaked correctly that would limit location to continent, unless I'm missing something.

It would effectively trade quick responses to/from close networks for some extra amount of privacy (in the case that GeoIP has already been taken care of)

The latency would have to be controlled on both ingress and egress to account for internal and external threats. I've got a niggling feeling that an attacker that could control latency of enough geographically diverse networks could find the boundary by manipulating responses to get finer detail, but can't quite work the problem into a solution...

Is there a less horrible or more reliable universal mitigation that I'm not thinking of?

<comment><USER3> To some extent, this already happens. My cable modem adds about 30ms latency no matter the destination. I think this is a combination of buffer bloat (wait for buffer to fill before talking on the network) and waiting for a transmit time slot (shared access to the physical layer). I haven't looked at it in detail, but it is very surprising to me that I get 60ms RTT to Blizzard's servers in Chicago (a speed of light distance of 4ms one way) and 25-40ms ping to Google (in what they call "lga15", which is somewhere in Manahttan, probably 60 Hudson).

I realize that ping is a very poor benchmark as most routers do not handle ping in their fast path, but it's not adding 40ms of latency. So I suspect my modem.

<comment><USER4> This is extremely unusual to me. Personally I notice it when my ping to local game servers go from 5 to 25.

I'd recommended you look into it, and potentially get another modem. If all my requests started taking another 30ms, I'd consider my network degraded.

<comment><USER3> Seems like everyone with any cable ISP has this problem. It could be a modem problem, but it's happened with a variety of modems. It could be an ISP problem, but both Comcast and Spectrum users see the same thing in my experience.

I'd be interested in hearing counterexamples, but I imagine that people with good experiences are on DSL or Fiber.

<comment><USER5> I can ping 126.96.36.199 right now at 12-15ms, or Google at around 20ms, via Comcast. Less than 10ms to either one on Optimum. But it's 25-30ms to ping Optimum from Comcast.

</comment></comment></comment></comment></comment><comment><USER6> I recall seeing a paper where they showed how close you can geolocate with various numbers of peers to the target, by using network latency alone

<comment><USER7> That's funny, I was just working on a POC like this today[0] - it's accurate most of the time for my location but I haven't tested from other locations. You'd need to tweak the 'known' servers on and off to find the optimal arrangement because you'd need to be somehow inside the polygon. I was planning to find a way to discover these itself and other tweaks (like trying multiple times then averaging out)

Edit: the paper I found related to this is here[1]

[0]: [https://github.com/stagas/http- geolocate](https://github.com/stagas/http-geolocate)

[1]: [https://homes.cs.washington.edu/~tom/support/geoloc.pdf](https://homes.cs.washington.edu/~tom/support/geoloc.pdf)

<comment><USER8> [https://vercel.com/edge-network](https://vercel.com/edge-network)

This page contains websocket addresses of a CDN that returns ping pong from a huge number of locations. It works surprisingly well for working out very fine grain location in just playing with it.

<comment><USER7> Oh wow, they're actually doing the same thing, triangulating on latency. Hive mind, I guess. I used universities because I figured a)they won't mind, and b)they're more likely to have their servers on premises, then used a public reverse geoip to get their locations. I'll try to see if I can integrate with Vercel's edge network, seems more ideal.

Edit: no, I misunderstood, the location dot that's being displayed isn't the product of triangulation, they're just doing reverse geoip lookup. So, I wonder now if the edge network would perform better.

Update: it doesn't perform better. Either there is some kind of proxy redirecting their traffic or these servers aren't where they say they are, the center is skewed out completely. The universities win so far being correct and accurate most of the time.

<comment><USER9> Unimpressed with yours. Why? Because my ISP got some IPv4 space recently, which was formerly allocated to .ua, .ru, .iq, & .ir. While being physically next to or in Hamburg, .de. That led to all sorts of inconveniences for some people, suddenly barred from logging on, or using the sites they frequent, because they used outdated geo-ip data. I didn't even notice, except for the outrage in their customer forum.

Now yours consistently puts me somewhere into, or onto the shores of the Black Sea, while Vercels doesn't. So that makes me suspicious of your claim by using latency alone.

Edit: There seems to be outdated geo-ip information factored in somewhere. Why else it would put me IN the Black Sea?

<comment><USER7> I mentioned I just started working on it. To try for yourself you'd need to clone and tweak the servers list to find an optimal arrangement for you. The problem as I see it is that jumping continents occurs really artificial delays which skews the result significantly, so it first needs to identify your relative whereabouts, then decide on an optimal set of servers. If you clone and tweak the servers to place yourself inside the polygon you'd see it does locate you. Vercel is doing a reverse geoip lookup, so your location is preconfigured in some database based on your ip.

<comment><USER9> At least they are current ;-)

<comment><USER10> I hope you can come back in a day or so, and re-read this conversation. You're not being very nice, or fair, and it doesn't portray you in a good light.

<comment><USER9> I probably will. It may have come across harsh, but wasn't intended as such. IMO i outlined the reasons why sufficiently in my post and edit.

<comment><USER7> Maybe you misunderstood what the tool is doing. It is not doing geoip, it is using the latency of pings between your client and some selected servers with known locations, then calculates the center based on those timings and these known locations. Very simple, you can look at the code and see what it's doing. It's not factoring in any kind of real latency, such as the speed of light, hop delays, etc. Unfortunately, as others pointed out, this method works poorly, especially in such long distances. In my tests you can only find a certain window that performs well for your location, still within a radius of hundreds or thousands of kms, which makes it pretty unusable as you point out. Knowing what _doesn't_ work is also valuable information.

<comment><USER9> I understood that. Let me replay it from my mind, how i percieved it. There is the dark map with white contours. And yellow points on it, which represent the triangualiton servers, i guess? Even clickable, at least the RED point which represents _my_ assumed location sometimes moved. Most notably when i clicked on London, then it hopped to Odessa. Otherwise, it was mostly in the northeast region of the Black Sea. In the light of the recent IPv4-space acquisition of my ISP, which i wrote about containing addresses formerly used in the Ukraine, Russia, Irak and Iran, causing "funny" issues...what would _you_ have thought?

<comment><USER9> edit: btw. tested in latest Firefox with uBlock Origin on/off. and Iridium Browser (Chromium derivative), retested in Iridium only after 24h dyn-ip renewal. Same results.

Anyways, time to sleep. Sun's getting too bright.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> I can see where a FQDN candidate is no biggie in a browser's offer/answer since DNS lookups occur all the time. But I imagine the simple fix for Signal's WebRTC use, since they control both sides of the exchange, is to just disregard non-IP candidates. Or even better, don't do anything with the candidates until the call is accepted. Worst case, could just have a geographically centralized signaling server (or shared IP). Granted, since Signal controls both sides, might as well only serve fixed "host" candidates and disallow any offer/answer with custom crafted ones.

One also wonders, to prevent other forms of leaks, if Signal can make a blanket policy to prevent DNS lookups or in general get tighter control on outbound network.

<comment><USER12> Disregarding non-IP candidates is exactly what we've chosen to do (and which the new versions of the app do).

The downside of disregarding all candidates until the call is accepted is that post-accept connectivity would be much slower.

Going through a server to hide your IP is an option in the settings in the app, but it can potentially lead to higher call latency, so there is a trade- off.

To prevent issues like this in the future we are taking more control of WebRTC's behavior with a fork of WebRTC (Signal uses WebRTC) and are providing patches to upstream WebRTC as well.

(I work at Signal on calling)

<comment><USER13> >> " PINs will also help facilitate new features like addressing that isn't based exclusively on phone numbers, since the system address book will no longer be a viable way to maintain your network of contacts." [1]

Any idea when more information might be available on this? Asked moxie years ago to add this and know 100s of other have too.

Worth noting the FAQ as it relates to the PIN length is not correct, "How long can my PIN be? There is no limit. Feel free to add as many characters as you want." [2] ...tested it and longest PIN I was able to create was 20 characters all numeric.

[1]: [https://signal.org/blog/signal-pins/](https://signal.org/blog/signal- pins/)

[2] [https://support.signal.org/hc/en- us/articles/360007059792-Si...](https://support.signal.org/hc/en- us/articles/360007059792-Signal-PINs)

</comment></comment></comment><comment><USER14> > if a Signal user wishes to hide their private/public IP addresses even from > contacts who call, then it has an option "Always Relay Calls" in its privacy > options

I thought Signal was all about privacy _by default_? :D

Signal fans love to dunk on Telegram for secret chats not being the only kind of chat.. well turns out on Signal, private is not the only kind of call, and your IP address is exposed by default.

<comment><USER15> Signal fans like to selectively forget that the server-side is proprietary software - therefore, the whole platform can't quite be proven to be reliable.

Essentially, they are not much better than Whatsapp stans.

<comment><USER16> That's hilariously wrong, the server source code is at [https://github.com/signalapp/Signal- Server](https://github.com/signalapp/Signal-Server). How can you make such a claim in good faith when it's so absolutely trivial to refute?

</comment></comment></comment><comment><USER17> >Even Edward Snowden, the well known American Whistleblower, claims "I use Signal every day."

Well, 5 years ago...

<comment><USER18> Presumably he's been using Signal over Tor, defeating all such attacks as a side-effect.

<comment><USER19> How does Edward Snowden acquire a laptop or phone in a way that he can trust it? I don't think it matters what protocols and applications he uses: he does not enjoy privacy.

<comment><USER20> I order to get a device that is not explicitly compromised with custom targeted malware one could: take a walk, enter a random shop, buy a device. Now you only have the standard malware that everyone gets preinstalled on their devices.

How to keep it free of custom targeted malware? That is another question!

<comment><USER21> With a target like Snowden who is under constant surveillance and lives at the whim of his host country, he could expect that any off the shelf hardware he bought would be immediately compromised. His hosts would just make up some bullshit reason to part him from the device for several minutes and do an evil maid attack. Or from afar his hosts or another country's actors could exploit undisclosed vulnerabilities in his device's wifi or bluetooth layer that they have in their toolkit.

<comment><USER20> I agree I cover that in my OP.

</comment></comment></comment></comment></comment></comment><comment><USER22> WebRTC and signaling can be an interesting attack vector. If rooms are not protected technically from uninvited people to enter you can get all kinds of information but even worse you can sometimes even hijack a call.

</comment><comment><USER23> [https://archive.is/SYq8H](https://archive.is/SYq8H)

I got a blank page on the original domain, perhaps due to DNS adblocking.

</comment><comment><USER24> You already have the peers IP address for p2p call right? How is this better than that?

<comment><USER25> According to the article, by default Signal will relay all calls from people who aren't contacts. This means if a non-contact calls you or vice versa, they can't see your IP address and you can't see theirs, even if the call is accepted and you're both talking. They also provide an option to enable relaying calls to/from contacts, so that contacts won't see your IP address, either.

Here, regardless of if you have that setting enabled or not, and regardless of if you accept the call, contacts and non-contacts can cause your device to make a DNS request, which will leak your DNS server. And if using a DNS server with EDNS Client Subnets, the first 3 octets of your IP address will also be leaked.

I think there's another issue like what you're describing which can kind of obviate this, though: the vast majority of Signal users probably use Signal on their regular mobile phone and its number, not a burner phone/SIM/number. (Few users probably even own a burner phone/SIM/number or understand what that is or why they might want one or how they'd obtain one.) So... everyone can just see your phone number, which probably has an area code corresponding to your city or close to it, and the other digits can possibly pinpoint it even more precisely than that.

Anyone who isn't tunneling all of their DNS traffic with a VPN or otherwise probably also isn't anonymizing their phone number and just has the app installed on their personal, standard cell phone.

If they aren't traveling and haven't moved recently, you can probably see what city they're in just from that. (This exposure does allow coarse location detection even when someone's traveling, though it's a lot more coarse than the area code, unless the Client Subnet value is being sent.)</comment></comment>


BlackDuck Alternatives - <AUTHOR> We use BlackDuck at work to scan our jar for open source components with weak licenses. Thing is, BlackDuck's scan tool takes too long (60+ min on a 400mb jar), and their support kinda sucks.<p>Anyone have recommendations on a better tool? I'm googling the competitors but would be interested to know what people's experiences were (as opposed to just reading the shiny marketing-speak) <comment><USER2> [https://github.com/nexB/scancode-toolkit/](https://github.com/nexB/scancode- toolkit/)

</comment><comment><USER3> I'm not sure about better, or what features you are looking for specifically. But, check out: WhiteSource CheckMarx OSA tools - though very new Veracode has a nice OSA / CVE tool And there are a whole slew of open source tools too.</comment>


Show HN: Encrypted private bookmarks for incognito mode on Firefox and Chrome - <AUTHOR>

I've had issues saving bookmarks on my computer since my girlfriend always uses it. I'm talking about engagement ring related bookmarks, birthday plans related bookmarks, etc.
My buddy and I built an extension called Hush that allows you to save them behind a password of your choosing. They're encrypted using SHA-256/AES, and you can export/import them to your liking (in case someone accidentally uninstalls the extension). It only operates in private browsing for security reasons.
Check it out and let us know what you think!
Chrome link: https://chrome.google.com/webstore/detail/hush-private-bookmarking/hjmoaenjknbdehbiaeeijcppnljflkff
Firefox link: https://addons.mozilla.org/en-US/firefox/addon/hush-private-bookmarking/
More information: http://teameuler.com/hush/ <comment><USER2> Yes "engagement ring related bookmarks".

Great idea though! Thanks.</comment>


Ask HN: Where to find resources on population simulation? - <AUTHOR>

I'd like to try my hand on making a world population game similar to Pandemic[1], but with more realism and a much nicer interface using more modern web technologies. My background is more focused on the interface side of things, but I'm not afraid of spending time and learning algorithms.<p>Where could I find some papers or examples of this kind of simulation? I'm aware that there are many HNers that have math backgrounds, so I would appreciate the help.<p>[1](warning: sound and uses Flash): http://www.kongregate.com/games/DarkRealmStudios/pandemic-2 <comment><USER2> You might try starting from this link, which was posted a few days ago: <http://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equation>

<comment><AUTHOR> Thanks, that's actually really helpful!</comment></comment>


Remote part-time jobs - <AUTHOR> http://remoteparttimejobs.com/ <comment><USER2> This doesn't seem ready for a Show HN. What does it offer later in exchange for email addresses now? Convince me this isn't just another spam harvesting service, please.

I could really use a functioning service to help me find clients right now, especially if there's some mechanism to prevent a race to the bottom.

<comment><AUTHOR> Well, this is something like "scratch my own itch" thing. I wanted to start small and build the audience. I'm afraid I'm unable to convince you other than: trust me. I'm a real person ([http://jestem.tw](http://jestem.tw)) and I would like to solve a real problem. I hope you will sign up and I will be able to share some openings with you soon! Cheers!</comment></comment>


On the causes of the '80s AI winter [pdf] - <AUTHOR> https://www.computer.org/csdl/mags/ex/2008/02/mex2008020002.pdf <comment><USER2> This article is dated 2008, it might be an idea to note this in the title</comment>


Jill Bolte Taylor's stroke of insight [video] - <AUTHOR> https://www.youtube.com/watch?v=UyyjU8fzEYU <comment><AUTHOR> I submitted this a year ago, and hope to continue to do so annually. When I read the top story re: LSD: Life-Saving Drug, I felt today would be a good day for it.</comment>


November 5th is Clang Day - <AUTHOR> http://docs.freebsd.org/cgi/getmsg.cgi?fetch=147088+0+current/freebsd-current

<comment><USER2> Man that's actually really exciting to see Clang and LLVM come into their prime on a popular OS. Great work, guys!

<comment><USER3> OS X isn't a popular OS but FreeBSD is? ;-)

</comment></comment><comment><USER4> It's really cool that the FreeBSD project is investing into llvm like this, seems like a good bet for the future. But why so soon and so earnestly?

<comment><USER5> The GPLv3 is why. The current C compiler in FreeBSD is GCC 4.2, which is what, seven years old? They can't use a more modern one because they're all licensed GPLv3 which is incompatible with FreeBSD's base system. Clang has a more liberal license. As I recall, this is also why Apple switched to Clang (though there were many side benefits for Xcode).

<comment><USER6> Why can't they use GPLv3? The FreeBSD license is compatible with GPLv3. <http://www.gnu.org/licenses/license-list.html#FreeBSD>

Apple doesn't want to use GPLv3 because of the patent clause.

<comment><USER7> It's not a legal incompatibility; FreeBSD has never shipped modified GPL software anyway to my knowledge (beyond things like patches to cleanly- separated ports that can be distributed independently under the GPL).

It's a philosophy thing. The new provisions in the GPLv3 (principly the counter-tivoization requirements which prevent shipping integrated devices without reprogrammability) are seen as unacceptable to the FreeBSD team. So they simply ruled against the license, which means that recent versions of gcc are unusable and they need to look elsewhere for a modern compiler.

That's the official answer. The cynical answer is that they've always hated the FSF and their dependence on GNU software, and are finally jumping ship now that there's a non-copyleft compiler available for them to use.

<comment><USER8> What kind of non-reprogrammable, integrated device would ship with a compiler?

<comment><USER7> They wouldn't. Most situations where GPL exclusions are enforced are still willing to include a modern gcc toolchain.

But still, it's FreeBSD's distro, and they get to write the rules. And clang is a modern, attractive compiler. The feud may be dumb but the choice isn't.

</comment></comment></comment></comment></comment></comment><comment><USER9> As a side note: Clang really shines when it comes to producing useful error messages / warnings.

</comment><comment><USER10> Remember remember, the fifth of November...</comment>


3D video conferencing is now possible with realfaces.org - <AUTHOR> https://www.youtube.com/watch?v=z5C1OVLiDBE&t=0m50s

<comment><AUTHOR> This is a link to a live demo. The website is at [http://realfaces.org](http://realfaces.org)</comment>


Mapping High-Level Constructs to LLVM IR - <AUTHOR> http://llvm.lyngvig.org/Articles/Mapping-High-Level-Constructs-to-LLVM-IR# <comment><USER2> "does not differentiate between signed and unsigned integer"

This isn't really accurate. IT's accurate at some level, in that there are not two different types. But that's because it's not necessary, they have nsw/nuw (no signed wrap, no unsigned wrap) flags on the operations.

<comment><USER3> It _is_ necessary - you cannot reconstruct your function prototypes from the IR declarations. That's why, for example, SPIR provides signedness information in the metadata. This ugly hack would not be necessary if IR had explicitly signed types.

<comment><USER2> "It is necessary - you cannot reconstruct your function prototypes from the IR declarations. "

Why would you want to do this? You are trying to reconstruct high level, language specific info from low level, language independent info. Of course you can't do this.

Your comment comes out to "lowering loses information", which is, of course, true.

So let's try again: What _semantic_ information do you believe is lost here. IE where does it result in an incorrect translation, or the inability to optimize something?

Because that's what "necessary" would come out to.

<comment><USER3> > Why would you want to do this?

Because SPIR requires that you can load and enqueue an OpenCL kernel from an IR, without having a source. And you need this type information in order to be able to format your arguments properly.

Please note that I'm not commenting here on the very choice of LLVM IR as a common medium made by SPIR committee.

> Your comment comes out to "lowering loses information", which is, of course, > true.

Exactly. But the current common uses of LLVM IR do require some of the information which is lost, and, in case of signedness, it was not even necessary to lower it.

> IE where does it result in an incorrect translation, or the inability to > optimize something?

You're limiting IR uses to translation and optimisations. Fine. I would have welcomed this way of thinking. But, unfortunately, this is not the case.

<comment><USER2> " But the current common uses of LLVM IR"

Portability of a high level language is an explicit non-goal of LLVM :)

The fact that someone decided to do it just makes them silly :)

<comment><USER3> I know it's silly. I can go on forever on what I think about SPIR and RenderScript. But, unfortunately, now it's a fact, an objective reality we have to deal with.

</comment></comment></comment></comment></comment></comment><comment><USER4> Nice but damn, I'm about to implement closures in llvm, but there's only a TODO at that point ;)

<comment><USER5> What language are you working on?

I guess for closures you simply copy all the locals that they capture into a heap allocated structure that also has the pointer to the closure code.

<comment><USER6> I have never implemented this, so I'm sure this will be incomplete and possibly slightly inaccurate, but that is not quite true. Some of the issues: copying may not be appropriate if your data is mutable (multiple closures can share a value or it might be modified in the 'regular' code after the closure is created), or if the code does identity checks later on.

Also, for performance, it may be beneficial to skip the 'put a local on the stack' part and create a to-be-captured object directly on the heap.

I see somebody posted [https://news.ycombinator.com/item?id=8580501](https://news.ycombinator.com/item?id=8580501), which points to the Wikipedia entry on 'spaghetti trees', the conceptual view on the needed data structures (which one may recognize from reading SICP, although I do not remember it using the term)

Many implementations, at runtime, will implement the 'main line' of the tree as a 'real stack', but that can be risky, as you will have to make sure that no closures survive the point where any locals they refer to get removed from the stack (what does C++ here? Declare it undefined behaviour or make it impossible?)

</comment></comment></comment><comment><USER7> Can't wait until Apple will disclose more information about Swift IR that compiles to LLVM IR. Too bad they canceled this year LLVM Devmeeting keynote.

<comment><USER8> I'm guessing that it's only conventions to work around the hardware specific parts of LLVM IR. Stuff like pointer and integer size, etc.

If you want to look at something similar, OpenCL has an LLVM IR -based standardized binary IR.

<comment><USER2> Actually, this is very wrong :)

SIL has a very large number of differences from LLVM IR. It's essentially built as an IR that the can do static analysis and high level optimization on.

This means it has a number of higher level constructs that LLVM doesn't, in order to be able to achieve the semantics they want for static analysis, and in order to be able to do things like optimize dispatch.</comment></comment></comment>


Mozilla is going to sell VPN subscriptions within Firefox - <AUTHOR> https://www.theverge.com/2018/10/22/18011072/firefox-protonvpn-subscription-ad-mozilla <comment><USER2> What a horrible idea... pocketsync pissed me off enough to rm -rf ...

ProtonVPN ... geeze</comment>