Wireshark: Facetime on Iphone 4: Vanilla unencrypted STUN and SIP - <AUTHOR> http://blog.roychowdhury.org/2010/06/25/facetime-on-iphone-4-vanilla-unencrypted-stun-and-sip/

<comment><USER2> Hopefully once Apple "opens" up facetime, they will also enable calls outside the Facetime network. Otherwise the who's and what's of connecting to their registrar will probably be no better than the App Store situation today (eg, I'm sure gateways will be forbidden).

edit: Although the identifying info from those screenshots is scrubbed, the comments say the R-URI is basically user@ip:port. This is interesting: there's no SIP proxy in between. No wonder it only works over wifi.

<comment><USER3> _Although the identifying info from those screenshots is scrubbed, the comments say the R-URI is basically user@ip:port. This is interesting: there's no SIP proxy in between._

Unfortunately I see "Bandwidth Limit Exceeded" when I go to that page but your comment makes me wonder whether they are doing SIP URI discovery via DNS-SD[1] given their penchant for DNS-SD.

[1] <http://tools.ietf.org/html/draft-lee-sip-dns-sd-uri-03> (ignore the stuff about mDNS and substitute Wide-area Bonjour)

</comment></comment><comment><USER4> Its actually a little more complicated than that:

[http://www.packetstan.com/2010/07/special-look-face-time- par...](http://www.packetstan.com/2010/07/special-look-face-time- part-3-call.html)

</comment><comment><USER5> Since Facetime is running SIP over WiFi, does AT&T still charge you minutes for a Facetime call or is that only for the non-facetime portion of the phonecall?

<comment><USER6> Apparently no:

"The voice call ends as soon as the FaceTime call connects... the FaceTime call is over Wi-Fi so does not use carrier minutes."

<http://news.cnet.com/8301-13579_3-20008289-37.html>

</comment></comment><comment><USER7> This is cool. This, Google Voice and Sipdroid on my Desire (which lets you route all calls, automatically, to the SIP provider of you choice, either on Wi-Fi only or 3G as well) makes me feel good about the future of telephony.

</comment><comment><USER8> Why is "unencrypted" considered a good thing? I find it kind of scary especially for a video conference app.

<comment><USER2> Your phone calls aren't point-to-point encrypted, either.

<comment><USER9> No, not end-to-end, but they are encrypted between your phone and the cell tower.

It's considerably more secure than an open wifi network.

<comment><USER10> GSM supports optional encryption, but there's no notification to the user whether a given call is encrypted.

<comment><USER9> Yes, but I'm pretty sure all US carriers use it all the time

(Though it's true that this opens you up to a potential MITM attack by a well equipped adversary, it's still a far cry from an open wifi network)

</comment></comment></comment></comment></comment><comment><USER11> Does anyone think they're trying to figure out IPv6 to make this work over 3G?

<comment><USER12> What does one have to do with the other?</comment></comment>


Senators have no idea how Facebook works, want to regulate it anyway - <AUTHOR> https://reason.com/blog/2018/04/10/mark-zuckerberg-facebook-senate <comment><USER2> > Sen. Lindsey Graham (R–S.C.) asked whether Facebook had any major > competitors. Zuckerberg tried to explain that the company competes across > different categories related to Facebook's several main functions—as a tech > giant, against Google, as a social media site, against Twitter, and so > on—which led Graham to fret about Facebook being a monopoly and thus > incapable of self-regulation.

IIRC, Graham had a good point. At his point, Facebook has no real competitors that offer an equivalent service.

To continue the analogy from the question: Zuck's dodge was a lot like a monopolist automaker trying to pass off Sony as a competitor because they both sell products with AM/FM radios.

</comment><comment><USER3> This is why selling access to people based on private personal data should be a crime with prison sentences for the responsible parties. Motivate the people in charge to find a solution. Don't tell them your solution.

</comment><comment><USER4> Aren't these the same people who were being asked to regulate ISPs during the net neutrality debate?

<comment><AUTHOR> Aren't they also the same people you (generic you, in case you're not American) voted in?

<comment><USER4> Yep, not me personally, but a majority of voters in my country chose these senators.

<comment><AUTHOR> Exactly. And they are the ones who write laws for us. Some even far more important than ISP regulations or what to do with Facebook.

<comment><USER4> Exactly. Maybe we should not ask them to regulate what they do not understand?

<comment><AUTHOR> That's sort of their job, to regulate things.

I was gong to say trying to elect people who at least know what they don't know might be a better approach.</comment></comment></comment></comment></comment></comment>


Guile-lips: Scheme as a generic macro language - <AUTHOR> https://rbryan.github.io/posts/guile-lips-scheme-as-a-generic-macro-language.html <comment><USER2> For those saying that the code is too verbose, here is another version in Common Lisp:

        (defun jim (arg &optional path)
          (etypecase arg
            (string
              (format t "import ~{~(~A~).~}~A;~%" (reverse path) arg))
            (list
              (push (pop arg) path)
              (dolist (a arg)
                (jim a path)))))

Test:

        (jim
         '(java
           (util "HashMap" "HashSet" "Map" "Random" "Set" "UUID"
            (concurrent "ExecutorService" "Executors" "Callable" "Future"))
           (awt "Color")))

Gives:

        import java.util.HashMap;
        import java.util.HashSet;
        import java.util.Map;
        import java.util.Random;
        import java.util.Set;
        import java.util.UUID;
        import java.util.concurrent.ExecutorService;
        import java.util.concurrent.Executors;
        import java.util.concurrent.Callable;
        import java.util.concurrent.Future;
        import java.awt.Color;

<comment><AUTHOR> If you'd prefer common lisp you should check out the project that inspired mine: [https://github.com/zc1036/lips](https://github.com/zc1036/lips)

<comment><USER2> You did a great job and I am sorry that the discussion is mostly about the example. I saw the Scheme version and found it quite verbose, like some other people on this thread. That's what made me want to implement a CL one. Maybe if you had a less controversial example where you produce e.g. Markdown text, it would not distract people from the tool.

<comment><AUTHOR> It was verbose. I actually left a comment in the example about how it could have been written better. Someone else further down in the comments suggested a nicer way of writing it. I think most people got the idea though! Thanks for the feedback. :)

</comment></comment></comment></comment><comment><USER3> I think many will agree that the import statement example is fairly weak. If you are writing import statements by hand you should just seriously consider the benefits of using an IDE.

That being said I use code generation instead of the IDE all the time but for harder things like generating fluent builders , copying objects, generating visitor pattern for pattern matching etc.

What I do is write Groovy scripts and will either use Groovy GString templates or Squares Poet to generate code.

What I do though instead of making some sort of DSL is use a subset of the language itself to help the code generator. For example copying a list of private fields I can quickly turn into either an immutable object with builder or an interface or just a plain object copy etc etc.

I put my Groovy scripts in my ~/bin and then just keep my terminal running. I then copy from Eclipse/Intellij some code and then run:

       pbpaste | SomeGroovyTransformScript.groovy | pbcopy

Then paste back the code into the IDE. Ideally I would use APT libraries (and in same cases I have converted Groovy code into APT libraries) but some times I just want something lightweight.

<comment><USER4> > I think many will agree that the import statement example is fairly weak. If > you are writing import statements by hand you should just seriously consider > the benefits of using an IDE.

Agreed that the example is weak. I do agree that, for import statements, having and IDE fill those in for you is fair game. It can figure out what you are trying to use and offer to import those, that's fine.

The problem here is the slippery slope. You get an IDE writing import statements. Than it will write getters and setters for you – which is already an aberration that prevents language evolution. For instance, C# has a syntax for default getters and setters (and for properties, in general).

Then you get IDE plugins for writing boilerplate (for, say, JEE). Then you end up with a XML preprocessing pass that writes Java code (XDoclet).

It's amazing the amount of machinery that gets created when people do not have access to a useful macro system.

<comment><USER3> Yes but there is also abuse and or complexity that doesn't make it easy. One of the classic examples for me at least is OCaml's Camlp. For Lisp like languages it is easier to have a macro language but for typed languages I think it becomes far more difficult both to implement and understand.

A lot of people don't realize that Java offers a macro/preprocess like system (APT) and it is even sort of hygienic (an example project is Google's Auto [1]).

APT is not that powerful but it gets you quite far and going back to OCaml the new ppx language extension is actually fairly similar (ie annotation processing instead of a complete macro/preprocessing language). .NET has something analogous but even more powerful (IIRC as well as the whole access to the AST aka LINQ style).

 _> Then you get IDE plugins for writing boilerplate (for, say, JEE). Then you end up with a XML preprocessing pass that writes Java code (XDoclet)._

2001 called and wants it example back :) . Yes APT didn't exist for a while. Tangental but XDoclet sort of reminds me of Go's very weak annotation system. And again It takes time to get a macro/preprocessing engine right particularly for typed languages. e.g. Rust has had some fair challenges getting this right and even their macro system is not completely hygenic IIRC.

[1]: [https://github.com/google/auto](https://github.com/google/auto)

<comment><USER4> > ... but for typed languages I think it becomes far more difficult both to > implement and understand.

I don't think that types are the issue but rather that such languages were not designed with macro systems in mind from the very beginning. Some added complexity is unavoidable, of course.

> A lot of people don't realize that Java offers a macro/preprocess like > system (APT) and it is even sort of hygienic (an example project is Google's > Auto [1]).

I didn't. Will look it up.

> 2001 called and wants it example back

I had to use that abomination as far as 2005. I understand that's a dated example, but it is the best example of that slippery slope taken to its extremes that I could think of. It is also a good example in the sense that it used XML instead of expressing itself in the original language. As such, it is also a nice illustration of Greenspun's Tenth Rule.

<comment><AUTHOR> I was also completely unaware that Java had a macro system. I'll have to check that out.

<comment><USER3> Calling it a macro system is probably a stretch of the word :) but if you check out APT you should also check out the far sexier Java Agents [1] which is code that changes other code during runtime (ie alters structure and provides meta programming).

[1]: [https://docs.oracle.com/javase/7/docs/api/java/lang/instrume...](https://docs.oracle.com/javase/7/docs/api/java/lang/instrument/package- summary.html)

</comment></comment></comment></comment></comment></comment><comment><USER5> _< 20 lines of parensoup>_

 _> How easy was that!!?_

;)

The problem comes mostly from lisp's dislike for text (my personal opinion). Lisps work best when everything is a list, always. As soon as you cross the boundary into... Well.. Anything else , really, you need a little bit of ugly glue.

That's why it's so ugly, in case anyone was about to make the "ermagerd if lisp is so great explain this..." comment. :)

[edit: which, btw, is not to lisp's credit, imho.]

<comment><AUTHOR> I completely agree. This is actually better than a lot of cases because Java import statements are just lisp symbols delimited by periods. This is mostly only useful if you're transforming lisp statements into code. Transforming java to java or c to c is just a mess. Still, If I need to do some ugly code transformation, I'd rather it be in scheme than c. :)

Also, the example could and probably should be rewritten more cleanly. Hence the comment. That was the first thing I used the tool for and my scheme has improved since.

</comment></comment><comment><USER6> I don't like the style though. I'd write an accumulative tree traversal composed with a string formatting final step.

      ;; :: tree -> [flat-qualified-name]
      ;; :: flat-qualified-name -> string

<comment><AUTHOR> That does sound better. Build up the structure you want then convert to a string.

</comment></comment><comment><USER7> I really like the idea of a general-purpose text-replacement pre-processor like this. I especially like:

\- A "real" programming language (Scheme)

\- Only 1 character to avoid/escape (~)

\- Expression-oriented (as opposed to, for example, impure echo/print)

\- Completely orthogonal/independent of the code being manipulated ( _unlike_ the myriad Java/JVM examples cited by other comments)

One thing that would be nice is to distinguish between pure expressions and impure ones. That way, pure expressions can _always_ be processed automatically (e.g. in wrapper scripts around tools like static analysers) since they're incapable of performing any effects.

Impure expressions would still be useful, e.g. to scan filenames in the current directory, but can be identified as such and skipped by tools/scripts.

<comment><AUTHOR> Hmm, I think it's a dangerous thing to be skipping portions of code in analysis just because they're impure. I think you'd be better off just taking the code produced after processing and running that through your static analyzer as is. That's what's going to be running in the end anyway. Maybe I'm not fully understanding the problem though.

<comment><USER7> Well, you could choose to abort rather than skip anything.

The problem is that impure code can do dangerous things (depending on which i/o primitives are provided). For example, a macro might create and delete a bunch of temporary files; this might work fine in one scenario (e.g. during compilation) but misbehave during another (e.g. static analysis).

In which case "static analysis" would no longer be static.

</comment></comment></comment><comment><USER8> We have this at work, actually. I wrote a macro language for pascal in guile. At first it was a first draft, but it worked well enough and got to stay.

It generates enormous amounts of code that would otherwise have been unmaintainable. As it is now, it is just a couple of hundreds of lines of scheme and pascal code that recursively generates pascal.

We had some problems making it scale (as I said, it generates a lot of code), but the upcoming 2.2 guile solved most of our issues, and I rewrote parts of it to make it fast enough. Got it down from 1 minute to 10 seconds on a 10+ minute build when rebuilding everything.

I might be able to opensource it. With minor tweaks, it can probably be made a bit more language-agnostic.

</comment><comment><USER9> Here's the original lips on which this is based: [https://github.com/zc1036/lips](https://github.com/zc1036/lips)

Maybe I'll use this to hack generics into Go …

<comment><AUTHOR> Are you the author? I tried to find a way to get in contact with him, but github doesn't have a messaging feature and I couldn't find an email.

<comment><USER9> Nope, I'm not. Good luck finding him — maybe you could open an issue? Bit of an abuse of the system, but it would work.

<comment><AUTHOR> I considered that. I may do it yet.

</comment></comment></comment></comment><comment><USER10> Very cool!

I had thrown together something very similar using s7 for a super small self- contained preprocessor.

I'm using `@` as the symbol to consume the next-sexp and process it, otherwise it's mostly the same.

I'll try to throw together some documentation, examples and a LICENSE in case anyone wants to use it. But I've just been using it internally as a generator for Ninja files for my builds.

[https://github.com/Arelius/s7pp](https://github.com/Arelius/s7pp)

</comment><comment><USER11> Ouch!

I never write those import statements by hand anyway, that's one of the many reasons I have an IDE.

<comment><USER12> Meta-programming via your editor doing code-gen for you or lisp doing code-gen for you end with about the same code.

<comment><USER11> Yeah, except IntelliJ Idea is a refined product that does that and 100 other things.

I don't know what IntelliJ has inside of it, but I know Eclipse is based on EJC, the Eclipse Java Compiler, which can turn Java code into a Java abstract source tree, and then turn that either back to Java code or to byte code.

It's not too different from the read function in LISP but it is a bit more complex...

</comment></comment></comment><comment><USER13> In the case of Java you can use Clojure or Armed Bear Common Lisp anyway.</comment>


Ask HN: Is there a Yelp for Special Dietary needs? Can we build it?  - <AUTHOR>

I hope you forgive me if this all sounds rather naive, as I'm new to both app development and diabetes. I'm curious about either finding or building a tool that would be like Yelp, except for special dietary needs.
I personally do not have diabetes, but my girlfriend just got diagnosed with pre-diabetes and gout both in the past 2 months. The hardest thing for her is getting used to what do eat and what not to eat given the new diet. I am hoping for something to merely help find (via crowdsourcing) dishes at restaurants that are specific to different dietary needs. Think "Yelp", but for particular dishes at restaurants that friendly for diabetes, gout, vegetarians, kosher, halal... Users that categorize their own dietary needs will post pictures/descriptions of the foods they eat, allowing similar users to discover them too.
Is there an app like that currently, and if not, what would be the best way to create one for Diabetes patients at least? Honestly, could such an app gain the critical mass it needs to take off, at least in a dense urban environment like San Francisco? Would people be interested in actually using such an app? Would it even be useful for diabetics? If so, do you think it would take a lot more resources than a single dev working nights and weekends? Marketing costs? I'm not too concerned if other apps like Yelp or Urbanspoon take this segment over, as that would have also achieved my purpose. <comment><USER2> You need to broaden this to gain real traction. Not just diabetics, and not just San Francisco. Search for restaurants that suit your dietary needs anywhere in the USA.

<comment><AUTHOR> I am just skeptical about outside of large cities since even Yelp has very little coverage outside of major cities, and they have a much larger demographic. However, I can see that targeting other dietary needs doesn't really have any disadvantages, and especially since some overlap (vegetarian/gout), it could be good. Thanks for the advice!

</comment></comment><comment><USER3> Isn't there a website that ranks not the restaurant, but a particular dish at a restaurant?</comment>


Ask HN: How to scrape Google's similar searches? (challenge) - <AUTHOR>

Alrighty guys! I've been at this for about 4 hours straight with no dice!
I'm hoping some genius here can help me out. Open a new tab and type in a band name like: "All Time Low"
You'll notice a box on the righthand side with more info about the band.
If you click the down arrow you'll see a "People also search for" section.
My goal is to get those names.
I've tried everything I could possibly think of to do it. The only thing I got working was phantomJS and the time it took to scrape just one page was over 5 seconds.
Thats way too long...
Anyone got a better solution than me? <comment><USER2> So, I know the folks who do DOS protection for Google, and...well, good luck. Scrapers get blocked, and the folks in charge of that are very, very good at what they do. Your best bet is probably to put up with the slow query rate and mimic ordinary user traffic. You really, really do not want to end up on Google's bad side.

<comment><USER3> Google isn't _that_ difficult to scrape. I've worked on numerous projects doing it and it just takes a little thinking outside the box and a few proxies.</comment></comment>


Us.linkedin.com subdomain certificate expired - <AUTHOR> https://imgur.com/a/nxWNk <comment><AUTHOR> 311,000 (report) indexed pages in Google. Ouch. Helpful screenshot to have the next time it happens to you.</comment>


Design systems, agile, and industrialization - <AUTHOR> https://bradfrost.com/blog/post/design-systems-agile-and-industrialization/ <comment><USER2> While the author implicates Agile, it seems like their real problem is with software development at scale, or really, any production process at scale. Once you've come up with a design (the fun part), you are faced with the prospect of implementation (the not-so-fun part). If that implementation includes so much work that you need a bunch of people to do it, you are left with the inevitable task of decomposing and distributing the work (into widgets or services or subroutines, or however you functionally break down the system). Whether you do Agile or not, that work is going to _have_ to involve requirements and acceptance criteria, and tracking and reporting and scheduling.

But all of that _doesn 't_ have to be dehumanizing. Nothing in Agile says you have to call people "resources." Or that you can't say things like "Oh, Samantha would be a great fit for this project." And if your Agile team's primary motivation is to move tickets from one column to another, you should fire your Agile management team, because that is literally antithetical to the Agile process, which focuses on delivered value.

<comment><USER3> " Agile, it seems like their real problem is with software development at scale, or really, any production process at scale."

I think it's more unique to software development. You don't see this kind of approach in large scale construction project with layers of contractors and subcontractors covering dozens of disciplines. You don't see this in car industry where they can do design, engineering, testing, factory tooling in 3 years.

I really believe the only reason why we do it in software is because we can.

And we can get away with it.

I also think the proliferation of outsourcing and offshoring was a strong catalyst for various reasons, though I'm sure that's a sensitive topic.

When industries that cannot get away with doing so tries, plane falls out of sky.

I think Agile can be great, but having been in the industry for 20 years and being put to practice, I see more reluctant Agile than true Agile. Meaning, people don't like it, but it has perception of working, so they go along with it, because there doesn't appear to be impetus to move away from it.

<comment><USER4> First, construction and automotve have been around for over a century, and they definitely suffered through growing pains in their discipline - there are boondoggles galore in the engineering and infrastructure space, and for the same reasons as bad software - a lack of clear vision and product fit.

And of course secondly they are subject to a much more rigid set of constraints to define success - primarily the hard laws of physics, chemistry, and human physiology.

But the key difference hands down is the raison d'etre - the demand for cars, buildings, bridges, roads, airplanes is self-evident.

Almost no software rises to such an elemental need in our world. The ones that do (Linux, avionics firmware, etc) don't run on Agile.

The rest: who really ( _really_ ) cares?

</comment></comment></comment><comment><USER5> Enterprise web development (or any enterprise development) has always been dehumanizing. It´s the vertical organizational model applied to software development. The enterprise implementation of Agile is just the vertical model in new clothes (status meetings disguised as daily meetings, story points used for stack-ranking, etc). When you enter the enterprise you trade leverage for job stability and 9-to-5 working hours.

Startups and mid or small sized niche companies are a little more human. But here you tend to get long working hours and more stress, plus sometimes they are also "vertical".

</comment><comment><USER6> So I played a lot Overwatch for the two years after it came out. (I swear, this is relevant.)

For the uninitiated, Overwatch at release gave players more freedom than any shooter on the market. Pretty much any effect you wanted (mass teleportation? Energy shields? Hacking? Automated turrets? Sonar? Lifedrain?) was available to you if you switched to the appropriate hero.

And the players started eating each other alive.

See, diversity of abilities means that some will synergize well together. But those parts are not necessarily equal; it's usually more fun to mow down opponents than to carry ammo around, for instance. So gamers, being dicks, will insist that _other_ people do the boring stuff like carry ammo or whatever, then retreat to the motte of "Don't you want to play as a team?" when called on this.

I got really idealistic about this game, and wanted to play it in a way that respected and supported the _humans_ on my team.

What I found was that the best way to do that was to pick heroes with which I could _reduce existential risk,_ (channeling Taleb) thereby not constricting my teammates into "Do X or we lose the game immediately" situations. This increased the options available to my teammates.

In an ideal world, they'd have reciprocated and made _my_ flexibility a priority as well, but, well, it's online gaming.

So the status quo was players yelling at each other to do X_UNFUN_THING that (supposedly) synergized with Y_FUN_THING.

Well, after three years the developers got sick of the players arguing with each other and enforced some structure.

This is how I feel about Agile, and software in the large. There are a zillion degrees of freedom available to individuals, who use that freedom in ways that have negative externalities for other stakeholders. Those stakeholders are doing the same, of course, and now you have a coordination problem.

What does it mean to write software in such a way that you create/preserve options for other parties? How do _they_ act to do the same for you?

I think that's the place to start.

</comment><comment><USER7> [The Pure Function Pipeline Data Flow v3.0 with Warehouse/Workshop Model]([https://github.com/linpengcheng/PurefunctionPipelineDataflow](https://github.com/linpengcheng/PurefunctionPipelineDataflow))

It systematically simulates integrated circuit systems and large industrial production lines.

In the computer field, for the first time, it was realized that the unification of hardware engineering and software engineering on the logical model. It has been extended from `Lisp language-level code and data unification` to `system engineering-level software and hardware unification`.

It brings large industrial production theory and methods to software engineering. It incorporates IT industry into modern standardized large industrial production systems, This is an epoch-making innovative theory and method.

1\. The replaceability, insertability, observability, and readability of the pipeline are very strong.

2\. The observability, standard, fairness of the dataflow are very strong. Data standards (data interfaces, data specifications) are better than code interfaces. and non-IT practitioners can understand.

3\. The warehouse/workshop model is widely adaptable, simple, and reliable, and the workshops are isolated from each other.

There are only five basic components:

1\. Pipeline (pure function)

2\. Branch

3\. Reflow (feedback, whirlpool, recursion)

4\. Shunt (concurrent, parallel)

5\. Confluence.

</comment><comment><USER8> "a developer resource is needed"

I hate this phrase. In the last two years the word "resource" has completely replaced words like "developer" or "person" at my company .

I guess since it has been widely claimed that a corporation's purpose is to make money for the owners and nothing else it makes sense to rid the workplace of any semblance of humanity. No privacy, no loyalty from the side of the company and now slowly people aren't being addressed as people anymore.

<comment><USER9> We use that term a lot at my job. It's specifically used a generic term to track utilization rates and project headcount needs so we can guess how many people we need to hire. We've been calling this practice Human Resources for decades. It doesn't mean we don't know who our people are or that they are all individuals with unique personalities and talents.

And it seems to me that employee loyalty is degrading far faster than employer loyalty, so it goes both ways.

<comment><USER10> I always thought HR was meant to mean resources _for_ humans, not humans as resources.</comment></comment></comment>


Show HN: Anatine – Pristine Twitter desktop app - <AUTHOR> https://github.com/sindresorhus/anatine <comment><USER2> Hey, well done on making it. Just wanted to comment that it seems to be more of a web app wrapped in a frame? Running it looks like it's just JavaScript and not native.

<comment><USER3> Welcome to the future of desktop apps.</comment></comment>


How Google's Web Crawler Bypasses Paywalls - <AUTHOR> http://elaineou.com/2016/02/19/how-to-use-chrome-extensions-to-bypass-paywalls/ <comment><USER2> "Remember: Any time you introduce an access point for a trusted third party, you inevitably end up allowing access to anybody."

See also: [http://www.apple.com/customer- letter/](http://www.apple.com/customer-letter/)

:)

<comment><USER3> Except of course what FBI is proposing wouldn't give access to "anybody". Just unfettered access for FBI, CIA and NSA by way of gag orders and national security letters, forcing Apple to break security of their hardware and not speak about it publicly. Of note is the fact that it wouldn't even give direct access to FBI, since they don't have the firmware keys (yet).

<comment><USER4> In this example, the FBI is the "trusted third party", but by giving them access, we inevitably open access for everyone, as the system is no longer strongly secure. The trusted third party in the quote isn't asking for access for everybody either, but in the end that's what happens.

<comment><USER3> Apple isn't giving access. Apple would be required (by court, unless they manage to fight this off) to install a signed custom build of the OS in order to give access to that particular device. FBI would not have this build, nor a key to create their own signed custom build.

<comment><USER5> Yup, that's the FBI's pitch. The issue is that it sets a legal precedent as well as potentially leaking a backdoored iOS to the world. Yes I know "signed for a specific device", best of luck with that.

</comment></comment></comment></comment></comment><comment><USER6> If they're now blocking clicks _from_ Google, doesn't that mean that they're cloaking and violating the Google's Webmaster Guidelines [1]?

[1]: [https://support.google.com/webmasters/answer/66355?hl=en](https://support.google.com/webmasters/answer/66355?hl=en)

<comment><AUTHOR> Google is not okay with cloaking, but they will whitelist publishers if the publisher specifically includes a parameter that declares if the site requires registration or subscription. This is done in the sitemap.

[https://support.google.com/news/publisher/answer/74288?hl=en](https://support.google.com/news/publisher/answer/74288?hl=en)

<comment><USER7> WSJ is big enough to negotiate their own terms with Google Search.

<comment><USER8> They're really not. They need Google a _lot_ more than Google needs them.

One of these things is going to happen:

(1) They end this "experiment."

(2) They stop serving Google the full content. (And see their rankings drop accordingly.)

(3) They get delisted for cloaking.

</comment></comment></comment></comment><comment><USER9> If this is true, what WSJ is doing is called "cloaking" and should cause it to get de-indexed: [https://support.google.com/webmasters/answer/66355?hl=en](https://support.google.com/webmasters/answer/66355?hl=en)

<comment><USER10> conversely, everyone should actively cloak and use random generated numbers to dynamically serve variants of their content similar to how mapmakers use trap streets. That way, like, another company wouldn't be profiting directly off of their work and threatening to sort of, destroy their entire business if you disagreed.

<comment><USER11> What? Any site can very easily not be in Google if they choose to. It's a very dumb decision for a news site, but you're free to do it.

<comment><USER10> Its a false choice without a conpelling alternative. Like saying, anyone upset with the status quo should vote. I was joking a bit, but I also wasn't.

Google has end to emd control over some users internet experience, and much of it in other cases. They own:

* 100s of thousands of servers

* domain registrar

* ~50% of web browsers in US.

* code CDN, FontService

* define web standards

* hundreds of millions of emails.

* CA implementation

* ISP infrastructure

* Develop software for a large part of thr mobile ecosystem.

* decide what you see when you go to search (most search copy google, buy results, or both)

* also many of the web beacons and advert targeting.

* oh, and the largest collection of video and images in the world.

So when you say, just do what they say or get deindexed, and you present it as if that is reasonable(not just you but the collective you) I just think I must be insane.

I mean, assuming google is good (i fo mostly) doesn't mean I would let them become the entire internet.

Real question, if google were to disappear vs. the "too big to fail banks" that would have gone under, where a case could be madr for a few certainly failing, what would have bigger impact today?

Tl;dr everyone cares about single point if failure except at the macro system level: finance, banking, healthcare, etc

<comment><USER12> There are certainly more you omitted (similar to your code CDN for javascipt, fonts, etc.)... consider something like recaptcha.

In my experience, all those requests to api.recaptcha.net get forwarded to "www.google.com"

My experience has been that if a user for whatever reason cannot access the IP du jour for www.google.com (www.google.[cctld] will not suffice) then that user is prevented from using the myriad websites that rely on recpatcha.net.

Now, I could be wrong and maybe there is something I am missing, but in my experience this is a sad state of centralization and reliance by websites on Google. Quite brittle.

<comment><USER10> Right, that was my point. I actually did include code CDN and fonts in the original list, but regardless I think google is an awesome company but as a community it is just downright irresponsible to fork this level of control to an entity.

While I consider Snowden a proper hero it is almost a certainty that this could happen to a "friendly" entity like google. In that, the NSA likely has some top programmers who could get a job there and compromise something, learn enough info to find a vuln, or pass data out. This is of course making the massive assumption that they aren't already cooperating at a system level either voluntarily or involuntarily.

As you can see, as search deteriorates google is motivated to (in my opinion benevelontly) use any means neccessary to continue to fund their larger goals of a connected and automated techno-utopia. However, they will be tempted to leverage what amounts to almost literally 50% of the worlds thoughts to build systems to make short term profit while pressing forward.

Just a few that immeadiately come to mind:

* using their network to control an entire alt coin ecosystem

* using data trends to trade on global markets

* start a competing business and deindex or penalize a competitor.

* build skynet (kind of joke)

So basically, those scenarios are fairly suboptimal and I could certainly imagine that several thousand genius with knowledge of googles systems AND the worlds data could likely be profitable quickly.

<comment><USER13> > as search deteriorates

Can you expand on why search is deteriorating? Honest question. I certainly don't see the relevance of search sinking, nor can I see any competitor in the market that could even come close to threatening Google's monopoly on search.

<comment><USER10> It is my position that search can never be decoupled from the browser and when I say "search" in the statement you are referring to I mean Google, as it is Peerless for english lang search.

Search is in fact massively expanding as tooling and machine learning capabilities increase do to research and hardware. Similarly, Google, Apache and Elastic have many open source libraries for search, indexing, storage, caching and serving which allow for scalable architecture. Also, outside of the things above like crawlers and Hadoop, Solr, etc. Microsoft and Google have open sourced JS parsing engines and Node as well as the Electron browser, Brave Browser and Node Web-Kit are built on technology that leverages this.

So, as someone who is not an information architect or data scientist, it seems like we have an ecosystem where a scaled down version of google can be built and trained on the per user basis and completely private.

The solution I have hashed out in more detailed elsewhere, but on to the actual question, is search deteriorating?

* My results seem to be worse and I have much less control than before. Anecdotally, it seems as if qoutes and boolean ops are respected less.

* Discovery is a huge issue that Google solved well, now we have the opposite conditions but the same problem. There were very few sites and it was hard to know what content was on them. Now there is too much content.

* Without fine grained control over my search I can't get make destinction between _Information vs. Links_. This is needing a date or well accepted piece of content/documentation vs. finding some new apps or non-facts. DuckDuckGo is quite good for some things and Google is good for others. Sometimes you may want to eliminate all wordpress sites (many content mills built on this) or remove Alexa links from your queries if you need to discover something.

* Time is bad. E.g. I have a problem with JavaScript function. Get back results from 3 years ago. This is amazing and difficult to do, so commendable but I need newer info as pace changes. E.g. News.

* Need to eliminate sites and content I don't want. NOT something like a content filter for porn or whatever, something like:

         never return results from %news-websites older than 30 days.

         never return content posted %before nov-2014

         remove links from [%Alexa-1000, %Wordpress, #TLD(.co,.co.uk)] for reputation ranking

         decrease links from [%Alexa-1000, %Wordpress, #TLD(.co,.co.uk)] by [80%] for reputation rankings

There are other things but so far my point has been:

* Google provides no versatile results.

* Many pieces of well tested software would make it easy(for the right group of software engineers) to silo crawl data and parse it with a users own parameters.

There is a way to set up this ecosystem that I have been thinking about, but to conclude:

Google is fucking awesome and really really good at what they do. Search experience is getting worse in terms of control but tooling is leagues better. Google sees this and is working on loftier goals internally (I imagine), thus it has split up into a meta-company that will work as an accelerator for growth while capitalizing on some verticals like the Real Estate thing they are doing or Delivery they just announced to keep short term profitable before they can achieve their end goal. Also, advertisement is an unsustainable paradigm for internet growth for many reasons.

Notes:

The DOM is super fucking horrible.

The Parsing engine is a great fix for a fucking horrid DOM.

DNS security is fucking horrible.

The Next google will be a browser & an optimization marketplace.

I don't think compiling to web assembly makes sense but I could be totally wrong. I think something like Docker would provide a sandbox that would let people get performance and versatility and sidestep the entire DOM, only run JS, need Apps vs. Content thing. No idea how this works on mobile though.

<comment><USER13> Wow, awesome response. Will need to let that sink in.

> So, as someone who is not an information architect or data scientist, it > seems like we have an ecosystem where a scaled down version of google can be > built and trained on the per user basis and completely private.

I'm skeptical. Search is a huge problem just because of the bizarre amount of resources you need to throw at it. I can't afford to build my own datacenter(s) to host my custom search system. There might be huge advances ahead in terms of storage capacity on commodity systems, I don't know, but in any case, I'm only one person crawling webpages versus millions of people (and bots!) creating them.

You implicitly address that a bit later by talking about "silo crawling", but again, I'm skeptical. The only silo structure that I can easily see is large sites with useful content like Wikipedia or StackOverflow/StackExchange, but I'm likely to come across these anyway in any given domain, and I can easily filter for these on Google today, e.g. "site:en.wikipedia.org". The more interesting and hard part is the long tail of small, sparsely interconnected websites which might contain unusual insights but are unlikely to come across with a silo crawler (or with Google's current UI, for that matter).

> Search experience is getting worse in terms of control

I guess that's the classical problem of scaling a product to a large audience of mostly technically illiterate users. Maybe Google is learning from Apple, whose UIs have for a long time favored ease of use over giving control to the user.

<comment><USER10> > I'm skeptical. Search is a huge problem just because of the bizarre amount > of resources you need to throw at it. I can't afford to build my own > datacenter(s) to host my custom search system. There might be huge advances > ahead in terms of storage capacity on commodity systems, I don't know, but > in any case, I'm only one person crawling webpages versus millions of people > (and bots!) creating them.

I have been thinking about this, and I have come up with some ideas, other people obviously would provide more ideas and a solution could be reached, some of my thinking:

A service that behaves like AWS/GIT/DNS/Google combined. * A user runs the service and indexes data it receives and there is a central repository of information than a user can contribute to or not contribute to. Initially, a new user would either buy a crawler or cache of data from the market and store it locally or on a private server bought as part of the service. The blockchain, or a verification mechanism would be used to provide access to the initial seed data and a hash would verify the contents. The user now has a running cache of data s/he can connect to with a private DNS-like verification system. User runs search.

* The parameters do not return the results s/he wanted from their private store. Similar to DNS, they move up food chain to the service provider (whoever creates this system, or one of the companies/orgs providing the service) to get more data. Here there is a centralized repo of information. This can be a market or platform. People can buy and sell data, filtering mechanisms and crawlers. Also people can include all of their searches, or some of their search results into the master crawl. This would be the "datacenter", but it can also be a platform that maps to many peoples individual caches:

> So if I indexed and codified everything about the Beatles I could sell this > to the market by running my own server.

> I could sell a crawler that is really really good at finding all musicians > and music to the market.

> I could sell a filtering/parsing engine plugin for music guys crawl results > (or all results it is fed) that only delivers high quality FLAC audio files > and converts high-enough quality MP3s to FLAC, all this but only for tracks > with a Saxophone.

However, fucking music guys crawl stack doesn't have the shit I want in it.

I can buy (or write) a master crawler that goes out onto the internet and finds what I am looking for then delivers it to my private cache, and if I am generous, codifies it in a generally accepted meta language and inserts it into master.

Obviously there is much more here but what I am talking about is distributed and optimized search.

Notes: Google has a nearly impossible job:

* It does not allow a user to provide any filtering outside of some boolean operators and human language.

* Therefore it never knows exactly what the user wants.

* Provides a general service so to some extent it is one size fits all.

* Difficult to do machine learning because it can watch you make selections but may not ever be able to tell what the deliverable was or if you were successful.

* You can not backout or modify algorithim it uses to find results. Neccessarily, even when it knows what you want it is biased because it shows you the results and defines the algorithim. Also, in the fact I am baselessly making up right now, only 2.1% of users ever go to the 3rd page, which means that if google is wrong, it can't know and the problem compounds as users see the same bad pages and keep clicking them.

> I guess that's the classical problem of scaling a product to a large > audience of mostly technically illiterate users.

yes. I am not saying google is doing a bad job. They have a nearly impossible task if they only use a searchbar with natural language and 0 filtering to deliver trillions of terabytes of data to millions of people. I am not sure how much easier it would be, but certainly n times easier, if filters worked.

Also, I think basically the idea of HTML is shit EXCEPT for the meta language. If not some simple JSON the actual results need fucking tags not the content, then we could filer down further and better.

Group annotation.

File sharing.

Bitcoin payment for content/filtering/cooperation

Running arbitrary code in a sandboxed environment like docker, not a "DOM"

Also, the silo concept is like DNS if I didn't cexplain it super well. You have a cache on your computer, a cache in the cloud, access to a master cache of information (both receiveables and lookups of other silos) and an optimization market for searching through data, or finding more of it if neccessary.

100% obvious search will end up this way. Brave software seems to sort of get this. I am hoping they realize a browser can't be decoupled from search though because you can't just fork electron and put some plugins in it. They are super talented. I am hopeful. One of the systems similar to what I am suggesting is called memex-explorer. However, I have never used it as the build is currently failing. It was opriginally funded by DARPA and NASAJPL then one day all work stopped on it and I have emailed and tweeted some of the people and orgs with no response. So while doing research, the description seems somewhat inline with my thinking.

The large problem of scaling is handled by the market. Search is essentially an API to call APIs that call an RSS feed if you think about what your browser and google are actually doing. Knowing what those APIs do is pretty fucking important.

I try to share this info with people but they all think I am fucking insane. Does this sound that farfeched? Honest question.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> Correct me if I'm wrong, but wasn't there a long standing Google's policy that the version of the page served to their crawler must also be publicly accessible. That would then be the reason why WSJ articles were accessible through the paste-into-google trick, rather than because WSJ was incompetent and failed to "fix" the bypass.

So does it mean that Google will no longer index full WSJ articles or does it mean a change in the Google's policy?

<comment><USER8> You are correct, Google requires that you let users see the first click for free if you want to index content behind a paywall. [1]

Since this is billed as an "experiment" I'm guessing that WSJ is just testing the waters. If they roll it out to everyone, they will have to serve only snippets to Google or risk getting delisted.

[1] [https://support.google.com/news/publisher/answer/40543?hl=en](https://support.google.com/news/publisher/answer/40543?hl=en)

</comment></comment><comment><USER15> And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i).

From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the computer that the accesser is not entitled so to obtain or alter."

To prove you have committed this terrible felony, the FBI will now demand that Apple assist in disabling the secure enclave of your device in order to access your browser history. But remember, they only need to do this because they aren't allow to MITM all TLS and "acquire" \-- not "collect" \-- every HTTP request your machine ever makes. </s>

<comment><USER16> User agent strings have a long history of being intentionally misleading. IE 11 claims to be "Mozilla/5.0". Chrome claims to be "Safari/537.36". The User- Agent string is all lies, and has been ever since the first site started doing UA sniffing.

<comment><USER15> It's _intent_ that matters. Setting user-agent in order to properly render a page is legal. Setting a user-agent string to gain access to otherwise unauthorized content is probably not.

<comment><USER17> User-agent string is not an authorization mechanism.

<comment><USER15> Interestingly, the CFAA does not define the term "without authorization" however it does define "exceeding authorization" exactly as I quoted above;

      - to access a computer with authorization and to use such access
        to obtain or alter information in the computer that the accesser
        is not entitled so to obtain or alter.

So arguing User-agent is not an authorization mechanism probably won't help you, because exceeding authorization means, first, that you _were_ authorized to access the computer (HTTP GET returns 200) but then that you used that access to obtain information in the computer that you were "not entitled so to obtain."

<comment><USER17> If the computer on the public Internet responds to a standard HTTP request, it has explicitly authorized my access to whatever information it sent me.

<comment><USER15> Unfortunately this will not help your defense. Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this (although the conviction was later overturned on a technicality).

Again, exceeding authorized access means using your _authorized_ access to obtain information you were not "entitled" to. So the question is not 'were you authorized' but rather it is 'were you _entitled_ ' to that information? WTF 'entitled' means is another question entirely, but likely it is in the eye of the beholder. A jury decided Weev was not 'entitled' to the email addresses he downloaded from AT&T, and it's safe to assume we are not 'entitled' to free access to WSJ's content. So I would not rest your hopes on the "200 OK".

<comment><USER17> > Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this > (although the conviction was later overturned on a technicality).

So, your example is...not an example?

>WTF 'entitled' means is another question entirely

No, in this case it is very clear: a request containing a particular user agent string is entitled. I have not tried this myself, but presumably you could verify that is the case by sending a request with the appropriate user agent.

<comment><USER15> It's the best example we got. The case was overturned (after he spent quite some time in federal prison) not because it was found that he didn't violate the CFAA but because the charges were brought in the wrong jurisdiction.

Again I think you're confusing the fact someone could trick the server into delivering the content for free with WSJ intending to deliver their content to you for free. Since WSJ clearly intends their content to be delivered to only Googlebot for free and to users only if they pay, it is likely a jury would consider this a violation of CFAA.

A web server returning 200 OK is not ipso facto a guarantee the person making the request is not committing a crime. To give a more obvious example, if the request header contains a stolen authorization token. The law does not require the access control be non-trivial to defeat.

I don't like it, and I think the CFAA is seriously problematic, but it is the law and the Feds have been known to enforce it.

<comment><USER17> As far as the law is concerned, he did not violate anything. You do not have to prove yourself innocent, the burden is on the prosecutor to prove a violation. In the example you cite, no violation has been shown.

It is not at all clear that WSJ intends Googlebot to get their content for free while others must pay. Thus is actually against Google's policies, which would actually call into question whether WSJ's behavior is felonius. WSJ may not be entitled to be incorporated into Google's index, yet they are manipulating the Googlebot to the contrary.

<comment><USER17> If you will down mod, at least show how I am wrong!

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> Am I alone in feeling like this is akin to a tutorial on how you can shoplift without getting caught? WSJ, for better or worse, does not want to give you content without your paying for it. If you take that content without paying, you are stealing. Just because you have figured out how to get past their security does not mean it's not stealing.

(See the second precept here: [https://en.wikipedia.org/wiki/Five_Precepts](https://en.wikipedia.org/wiki/Five_Precepts))

<comment><USER19> Legally it might, I don't know.

Morally, I'm not sure.

1\. If they allowed all bots but disallowed all non-bots, that would raise questions of what defines a "bot". Couldn't I write a personal bot that fetches the story for me? As a browser addon, even?

2\. It's even more complex since allowing bots means they allow tools that provide the information to third parties, as the bots are not intended for private use by the bot maker. So the door is already open.

3\. But in practice, it seems they favor certain bots. Is it ok that the WSJ lets Google do things Google's competitors cannot? Try to use the "web link" trick from HN on any other search engine, and it doesn't work in my experience. That seems anti-competitive and discriminatory in favor of the existing dominant entity in this space, Google.

<comment><USER20> > If they allowed all bots but disallowed all non-bots, that would raise > questions of what defines a "bot".

Maybe, but I think its a pretty easy distinction. They aren't even allowing all bots - they're allowing a white list of them. You're not just writing your own bot to get around it, you're pretending to be someone else's bot.

> But in practice, it seems they favor certain bots. Is it ok that the WSJ > lets Google do things Google's competitors cannot?

That's the really important question. I personally have no context for answering except to say that I can see both sides argued. If you view their website as a physical store / private establishment, then I assume that they have every right to establish who has access to what and under what conditions.

Of course, that hampers a lot of legitimate use cases along the way.

<comment><USER21> > If you view their website as a physical store / private establishment, then > I assume that they have every right to establish who has access to what and > under what conditions.

Not true. There are very specific laws about not being able to discriminate against protected classes of people.

Bars have to serve minorities, bakeries have to cater to same sex marriages, etc.

Where you draw the line of legislated equality within private property rights is pretty intriguing. I have don't have any answers, but lean heavily towards the libertarian bent.

<comment><USER20> That's fair, though private establishments are still allowed to distinguish based on other criteria such as membership, partnership, etc. I didn't meant to be super specific. I'm merely pointing out that they are allowed to establish barriers to entry.

</comment></comment></comment></comment></comment><comment><USER22> This is an odd debate. Let's say a restaurant declares "veterans eat free." This blog post is like a friend telling you "Hey if you tell this restaurant you're a vet they'll give you a free meal." No one said it's legal or ethical. It's lying to trick someone into giving you something at their expense.

I think the relevant point, underscored by the author's last sentence, is it doesn't matter who you open a back door for - it opens the possibility for anyone to barge through.

<comment><AUTHOR> that's a good analogy.

</comment></comment><comment><USER23> This is not meant to be purely controversial, but I thought long and hard about WSJ back a few months ago when HN mod (always forget his name) said to stop complaining about HN links being posted because paywalls were ok. I agree paywalls are ok. But some things are not ok.

Take a look, for instance, at the WSJ.com home page with an ad blocker turned on (note all the missing letters and scrambled up titles). They want me to pay, and they want me to see ads, and they want to track my behavior? Should I send them my DNA also?

Organizations like WSJ are exactly the disease that causes ad blockers to proliferate and ruin the web for all the decent publishers. They're at war with my privacy (by breaking their site intentionally when I visit with a blocker on). They want it all, ads, tracking, your private data, and subscription revenue, not to mention...

# Agenda-Driven Content

I mean, we're basically talking about NBC or Fox here, just on the web. Imagine every morning when you woke up you turned on the television and tune to some "news" show. After talking about the weather, they start talking about a lost pickle that is thought to be potentially alive and moving about with free will. Over the next two years, talk about the same pickle extends to every other TV show. Before you know it, everybody in the nation is talking about the same pickle. Years go by, and that pickle has become a part of our society, and that's not because people are born with an innate care the well- being of pickles, but because "news" shows taught them to be.

That's not a good position to be in. I have to believe I'm not the only one in here that doesn't watch any TV. So, why do we all treat the same media giants differently on the web? We crave their content so much that we build browser add-ons to get to their content, etc.

<comment><USER24> If they make sending your DNA a requirement of consuming their content, then yes, you send it to them if you want their content. That's their right, as owners of something, to dictate its use.

You aren't entitled to WSJ.com, NBC, or Fox.

<comment><USER23> I'm not a collectivist, nor do I believe I'm "entitled" to anything, including human rights. I'm simply stating that "I don't support organizations like WSJ"... as you can clearly see, if you read my comment. I don't propose anyone ban them. I simply "don't know why we [as a society] support them", also clearly in my comment. My point, which your political diatribe disallowed you to notice, was that we're fighting awful hard to consume their garbage. Meanwhile, there is plenty of content out there there is free, not because of socialism or entitlement, but for the same reason as blockers became popular... because information is so readily available now that nobody _is_ ready to send their DNA in.

</comment></comment></comment><comment><USER25> I'm pretty sure Google will soon stop indexing WSJ. Why index something if the vast majority of users cannot access the pages behind the links?

EDIT: The "paste a headline into Google" trick still works for me, though. If this continues to be the case, they will keep indexing, of course.

<comment><USER26> >Why index something if the vast majority of users cannot access the pages behind the links?

So people can find it? I'd be pissed if Google de-indexed something like IEEE because it has a paywall.

Assuming the internet has to be freely available is a mistake. Especially with the continued growth of an adblocked internet. We could be facing an internet with significant paywalls in the future.

I'd support a "free" search term to weed out paywalled results.

Furthermore, Google shouldn't be making normative judgements about what people should see. It's an abuse of their monopoly.

<comment><USER8> Google doesn't forbid or de-index paywalls. What they forbid is cloaking (showing Google different content than what users will see). This is, of course, quite critical to maintaining search quality.

WSJ is free to institute a full paywall and only serve snippets to Google. They might now like what it does to their rankings though.

What they cannot do is continue to sniff the UA before deciding to put up the paywall. (Though I'm still able to use the Google trick, so it seems the experiment might have ended.)

</comment></comment></comment><comment><USER27> Well, that trick won't last long either. It's trivial to verify that an IP indeed belongs to Google:

[https://support.google.com/webmasters/answer/80553?hl=en](https://support.google.com/webmasters/answer/80553?hl=en)

<comment><USER28> Seems to work if you deploy a proxy on Google's app engine and use it to access WSJ ;)

<comment><USER6> A App Engine wouldn't have a IP with a reverse DNS *.googlebot.com, would it?

<comment><USER28> Nope, but it does resolve to something .google and if you check netblocks.google.com it will appear there so they might not be limiting it to googlebot only at this point.

[https://support.google.com/a/answer/60764?hl=en](https://support.google.com/a/answer/60764?hl=en)

<comment><USER29> I don't think that any server that random people can start a proxy on is in Google's SPF record.

<comment><USER28> not on the spf list, but on google's domain/ip blocks list.

<comment><USER29> The link you yourself posted says "The most effective means of finding the current range of Google IP addresses is to query Google's SPF record." While this is intended to be mailservers only, it has always appeared to me that the SPF list is all of their servers that 3rd parties can't run proxies on.

</comment></comment></comment></comment></comment></comment></comment><comment><USER30> Basically, the article is stating to change the User-Agent to GoogleBot or Bing or whatever other crawler UA you'd prefer. While that's doable, that's something that is easily detectable and prevented, as all of the big crawlers can be validated against DNS.

Additionally, I would like to point out that I wrote a Varnish extension for the express purpose of validating User-Agent strings through DNS lookups, and is available here: [https://github.com/knq/libvmod- dns](https://github.com/knq/libvmod-dns)

It was built because we had specifically a problem with bad bots crawling a large site (multiply.com) and this was one of the easiest ways to filter out the bad bots from the good, and to enforce robots.txt policies on a per bot basis. It works very well, as you can do any kind of DNS caching internally and prevent this kind of behavior, if that's your goal.

</comment><comment><USER31> I like wsj but I only read maybe 1 article every other day. They need a more reasonable price point, especially since the market will almost bear no price at all.

That being said I do enjoy their content, save for maybe the op-eds.

<comment><USER32> I'm surprised that most online papers won't sell you one day's worth online for a buck or so. Like buying a real newspaper.

They all seem to want to sell subscriptions, which are perpetual and probably difficult to cancel..

<comment><USER31> Even a dollar a day. I can pay Netflix $10 a month and stream unlimited HD video, but wsj wants $30 to read the first few paragraphs of a few articles a day?

The pricing here is much too aggressive

<comment><USER33> Videos are more relevant to rewatch than articles are to reread. One needs to output more articles than videos, because articles must be "fresh" or you'll lose an audience. Nobody is printing 40 year old news - people are still watching 40 year old movies.

Playing devil's advocate here. Pricing for many online goods is almost completely arbitrary and varies with little accord to service/product quality or even what that service provides.

Another related example of arbitrary pricing: people will pay $2 for a soda from a vending machine but won't pay $1 for a useful app on their phone.

There's something going on there... The sooner the $1 app's figure out what makes people buy $2 sodas is the day they become rich. And the sooner content providers figure out why people will pay $20/month to stream media (Let's say $10 to Netflix, $10 to Spotify or something) and charge people $20/mo for their articles... things will turn around for them.

<comment><USER13> > Nobody is printing 40 year old news - people are still watching 40 year old > movies.

Because news operate at a different scale than movies. People are still reading 40-year-old books.

A 40-year-old news article is not relevant anymore because it only fits within the momentary context in which it was created, whereas a non-fiction book or even an essay can span a broader context and thus stay as informative for future readers.

</comment></comment></comment></comment></comment><comment><USER34> I thought Google specifically disallowed returning different pages based on User-Agent targetting googlebot, and this included paywalls.

Are they running afoul of Google policies and going to get pinged by Google?

I can't find the text from Google now (when can you ever find any docs at google?), but I am very certain I remember reading from them that you may not return different content to GoogleBot based on User-Agent.

</comment><comment><USER35> Doesn't this kind of also hurt SEO? I'm would guess Google has some automated system to detect and apply a negative signal to sites that provide different content to a Googlebot user agent than a non-Googlebot user agent. I guess these sites are counting that the other signals outweigh that negative hit.

Otherwise, why would expertsexchange be obligated to provide the answers at the very bottom? Did something change?

<comment><USER36> I'm 99% sure I've encountered a Googlebot crawling pages with the UA of a regular browser, presumably for exactly this purpose.

<comment><USER37> They do this via an iPhone like UA, though googlebot is still in there too.

<comment><USER36> I'm pretty sure that's just to see if the site is serving different content to mobile vs desktop. I _think_ they also sometimes hit pages with no mention of googlebot.

<comment><USER38> If you do an nslookup on the IP, it should come up with crawl-xx-xx-xx- xx.googlebot.com, where xx-xx-xx-xx is the IP.

</comment></comment></comment></comment></comment><comment><USER39> If you hit a paywall or a "sign up to access this content" message from a google search result, report it. Google will remove them from the search results, they will lose their largest traffic source, and they will address the issue. Or they won't because they have enough paying customers.

</comment><comment><USER40> i thought of doing that when the "search google" trick stopped working, but i decided it crossed the point where i would feel like i was unfairly circumventing their clear desire not to serve me the content. i've just added wsj to my mental ignore list and count it as a few more minutes gained to do something else.

<comment><USER41> Yeah, same here. Every time I get to a tab where I see a paywall I just close that tab, probably saving 5-10 mins of my life!

</comment></comment><comment><USER42> If Google (or any other crawler) wanted to play nice with paywalls, they could issue a public key for their bot, and put a signature in their User Agent string that the domain could then verify.

Those signatures could obviously leak, but on a per-domain basis. Perhaps the domains could have a secure way of bumping the valid key generation if they had a leak.

<comment><USER43> There are two problems with this.

First, they don't want to. In fact, if a search engine can figure out that a link is going to lead to a paywall, they'll probably want to reduce the ranking of the result, because the user is not going to want results they can't actually look at.

Second, it would be a massive antitrust violation because it would prevent access by competing crawlers. The only way around that is to allow access to anyone who claims they're a crawler, which was the original problem.

<comment><USER44> The current situation with the WSJ could already be considered an antitrust violation. It's whitelisting one crawler and leaving the other ones out.

</comment></comment></comment><comment><USER45> Bypassing the paywall is more unethical that blocking ads. It is one thing to have control over your own browser but another to steal something from another site.

Also, isn't it illegal to bypass computer security?

<comment><USER46> How is modifying your own request headers any different than choosing to not display content returned in the response body?

Their server can choose to do what it wants with your request and you can choose what to do with the response it sends.

Are User-Agent headers legally protected identities?

<comment><USER47> The difference is substantial; in principle, one can modify his request to intentionally bypass the authorization mechanism occurring on the server. One cannot mislead anyone/anything by displaying his data in a customized way in private on his computer.

<comment><USER46> Fair point. Intent matters. However must it be demonstrated that there is some specific malicious intent behind changing a request header versus simply changing your user agent for the heck of it?

In a related point, some news sites load a modal and prevent scrolling over an article asking you to sign up. However if the full article is included in the response and I read it by simply viewing the response body (HTML) is that circumventing security? Actual example. In this case modifying how the response is rendered in my browser, I can bypass their intentions.

Obviously the better way of doing this would be to not send the entire article content until they've determined I should be able to view it.

<comment><USER47> > _must it be demonstrated that there is some specific malicious intent..._

I think so. Just the act of changing User-Agent alone does not mean a fraud is happening. User-Agents get changed often for valid reasons - research, detection of cloaking, testing etc.

> _However if the full article is included in the response and I read it by > simply viewing the response body (HTML) is that circumventing security?_

That is a hard issue. I think the answer should be no, it is not circumvention, at least not in the above sense.(\\) If you can get full article on your computer, directly readable without fraudulent behaviour on your part, this means that the sender did not place a security measure to guard it, so there can be no circumventing it. Switching javascript off or displaying source text of the document is fully in the control of the requestor and the sender should know it.

(\\) Unfortunately, it seems legal systems allow companies to restrict what you do with things they produce, even if you do it on your computer in private, so legally, this may not be successful in court.

</comment></comment></comment></comment></comment><comment><USER48> Based on the comments here, am I to understand that constantly browsing the web with my user agent string set to a googlebot string, I am committing a felony? How would I even know which sites I'm gaining unauthorized access to?

That is completely idiotic if there is a string you can put in a Mozilla browser config that is literally illegal to browse the web with.

<comment><USER47> I do not think using any User-Agent alone constitutes a crime. There are valid non-criminal reasons why one would like to use Googlebot's or other User- Agent. I think it is the intent to bypass the paywall + success to do so that may be regarded as offense or even crime, but I'm not sure.

</comment></comment><comment><USER49> > Remember: Any time you introduce an access point for a trusted third party, > you inevitably end up allowing access to anybody.

 _cough_ NSA _cough_

</comment><comment><USER50> New workaround: paste the article title into archive.is. I don't know what they're doing but they have a workaround of some sort.

<comment><USER44> That's actually really interesting! Could anyone chime in and explain how they might work around this issue?

<comment><USER50> My guess is they have a login, but they could just be using some workaround like described in OP.

I've used them to save Facebook posts before, and the pages were logged in to some "Nathan" IIRC. They probably have a bunch of hacks for specific sites that needed fixing.

</comment></comment></comment><comment><USER51> I just tried clicking on "Harper Lee, Author of 'To Kill a Mockingbird,' Dies at Age 89" from wsj.com's homepage and got the paywall.

I then pasted the headline into google and clicked on it from Google results and did not get hit by the paywall.

<comment><USER40> they're probably doing some sort of a/b testing by selectively letting some clicks through

<comment><USER52> This is basically true ^^

<comment><USER53> Any idea what the deal is with SEO impacts of WSJ taking the idea of blocking everyone who isn't a google bot?

</comment></comment></comment></comment><comment><USER54> I was under the impression that the "hack" whereby you searched for the article on Google and clicked through to that article (effectively skipping over the paywall) was a demand of Google's and not an oversight by the paywalled website.

I thought that google deemed providing search results which were behind paywalls as a "bad experience" for their search users, and would penalize websites for doing so.

Is this no longer the case?

<comment><AUTHOR> Google doesn't demand anything. If your paywalled website is not accessible by Google's crawler, then Google will not index it. Publishers want Google to index their pages and drive potential paying visitors, which is why they open the loophole themselves.

For the second point, Google does require that publishers specify "registration required" in their sitemap.

<comment><USER54> If you're showing Googlebot one thing, and visitors who visit your website through google another, that's essentially "cloaking" (a blackhat SEO technique). At least it used to be.

</comment></comment></comment><comment><USER55> Doesn't Google usually try to punish websites that show users something different and even mentions that somewhere?

Not an SEO Expert here, but wonder how and whether Google will end up handling that. I mean making an exception could also be considered abuse of power in some countries of the world. Don't have any strong opinion yet on that, just saying that because of how the EU exercised certain laws in recent years.

</comment><comment><USER56> Aren't you supposed to verify if a visitor is a googlebot by reverse lookup of the IP address? I.E.: [https://support.google.com/webmasters/answer/80553?hl=en](https://support.google.com/webmasters/answer/80553?hl=en)

User-agents are notoriously unreliable.

</comment><comment><USER57> I wonder how many Google Cloud customers use the servers to run spoofed Googlebot crawlers from the Google IP range in order to bypass paywalls and scrape large sites (like LinkedIn) without hinderance.

</comment><comment><USER58> It's broken already. Tried to access an article about new china rules for online news and it pay-walled me. They're probably looking for clients coming from googlebot.com now.

</comment><comment><USER59> So does HN now choose to not post articles from the WSJ? I was comfortable with the "google it" trick, and frankly was a little annoyed with constant "paywall, wah!" comments when what should be by now a well-known workaround was available. But that workaround no longer works.

<comment> mark-r They've been testing the new wall for a while now. I know I made one of those "paywall wah" comments when the Google workaround didn't work for me. Then the next time I tried it worked fine, so it must have been random selection.

</comment></comment><comment><USER60> My Windows anti-virus deletes the linked sample code automatically upon download, marking it as "Trojan:Win32/Spursint.A". Did anyone have the same experience? (I was actually more interested in using it as a template for writing a simple Chrome extension.)

<comment><USER61> Yep. I then pasted it but it didn't work on wsj.com. Oh well.

<comment><AUTHOR> try deleting cookies, then hit refresh.

<comment><USER62> Not working here either. I tried wiping out cookies and cache, and Chrome is sending the right user agent (Googlebot).

<comment><USER63> Try disabling other extensions that might alter the user agent.

</comment></comment></comment></comment></comment><comment><USER64> Solution:

Content providers register a (yet-to-be-written) Google News API account, get an API key, with which Google indexes the site and the site recognizes as legit.

</comment><comment><USER65> I've noticed that this has stopped working on WSJ if you've already hit the paywall and try to google the article to bypass.

</comment><comment><USER66> I wonder if anybody tried to do as suggested? I copied the files to Chrome as per instructions, and the paywall was still in place.

</comment><comment><USER67> You can also access WSJ for free at the library.

<comment><USER68> It reminds me of this: "Trying to save a quarter..." [https://www.youtube.com/watch?v=j4nRHHPpnVc](https://www.youtube.com/watch?v=j4nRHHPpnVc)

</comment></comment><comment><USER69> It's not bypassing at all. Googles crawlers are deliberately let in because a paywall that nobody runs into is useless.

</comment><comment><USER37> So soon they have to block anyone with a fake Google UA and whitelist the well known 66.249 IP range. Trivial.

</comment><comment><USER12> Does WSJ check visits from a Googlebot UA against a list of known Google IP addresses?

</comment><comment><USER70> Fix: replace the user agent string by a cryptographic challenge/response scheme.

</comment><comment><USER71> They'll start allowing only some IP addresses search engines agreed with them.

</comment><comment><USER72> Possible in Firefox? Some people won't use Chrome.

</comment><comment><USER73> Is there a version of this available for Safari?

</comment><comment><USER74> So their next move is check if IP is from Google

<comment><USER57> [https://cloud.google.com/](https://cloud.google.com/)

</comment></comment><comment><USER75> >Archaic news source does something to hurt their market penetration to internet

Great idea here guys

</comment><comment><USER76> Or simply use incognito mode and click on Google search result.

<comment><USER77> Did you read the article. It talks about how that trick no longer works on a lot of sites because they are now checking User-Agent strings too.

<comment><USER78> Actually I noticed sites have simply changed policies -- if you're a regular visitor your cookies will identify you and block content. The Incognito mode trick works for WSJ and others that would still check the referrer header. Allowing Googlebot access and checking the referrer header are two different things.

Also, Google has published IP addresses it uses, so this extension might not last long...

<comment><USER6> > Also, Google has published IP addresses it uses, so this extension might not > last long...

They do not [1], but you can find out by doing a reverse DNS query.

> "Google doesn't post a public list of IP addresses for webmasters to > whitelist"

[1] [https://support.google.com/webmasters/answer/80553?hl=en](https://support.google.com/webmasters/answer/80553?hl=en)

<comment><USER78> Sorry, that was what I meant. :)

</comment></comment></comment></comment></comment><comment><USER79> 1\. Google's Web Crawlers are not "bypassing" paywall. It's the paywall that let's crawlers through. I.e. exactly the reverse of what the author implies with their headline.

2\. The idea that this is somehow new is wrong. The way for a server to identify crawlers have "always" been to look at the user-agent, and, when done right, IP, verified either by net block owner or by doing PTR lookup and then checking that the A or AAA record for the claimed host points back at the same IPv4 or IPv6 address. Meanwhile, I do agree that paywalling is a more recent phenomenon, at least with regards to the extend it is popular among sites today, _but_ the concept of presenting different data to crawlers and visitors arose much earlier and is something Google have been aware of and has made sure to delist such sites when found, whereas in fact Google has since then moved abit in the direction of allowing it in that they do so for Google News if declared as explained by others ITT.

So in my view, it seems that the author is jumping to incorrect conclusions based on an incomplete understanding of what's actually going on here. What then about the HN readership, how come this article became so highly voted and I don't see these issues raised by anyone else? Or maybe I'm just crazy?

<comment><USER80> > Google's Web Crawlers are not "bypassing" paywall. It's the paywall that > let's crawlers through. I.e. exactly the reverse of what the author implies > with their headline.

Don't nitpick. It's just a shortened version of _How To "Be" a Google's Web Crawler to Bypass Paywalls_. You get it. I get it. Everyone gets it.</comment></comment>


DeafCon for Deaf designers, engineers, and enthusiasts - <AUTHOR> http://deafhack.github.io/

<comment><AUTHOR> We need help raising funds to hire ASL interpreters for this event. [http://www.indiegogo.com/projects/help-support-deaf-con- live...](http://www.indiegogo.com/projects/help-support-deaf-con-live- interpreters)</comment>