Web Scraping in 2016 - <AUTHOR> https://franciskim.co/2016/08/24/dont-need-no-stinking-api-web-scraping-2016-beyond/ <comment><USER4> Keep in mind that companies have _sued_ for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: [http://www.informationweek.com/software/social/linkedin- sues...](http://www.informationweek.com/software/social/linkedin-sues-after- scraping-of-user-data/d/d-id/1113362)

OKCupid did a DMCA takedown for researchers releasing scraped data: [https://www.engadget.com/2016/05/17/publicly-released- okcupi...](https://www.engadget.com/2016/05/17/publicly-released-okcupid- profiles-taken-down-dmca-claim/)

Since both of these incidents, I now only scrape if it's a) through the API following rate limits or b) if there is no API, and the data has the explicit purpose of being shared publically (e.g blogs), I follow robots.txt. Of course, most companies have a do-not-scrape clause in their ToS anyways, to my personal frustration.

(Disclosure: I have developed a Facebook Page Post Scraper [[https://github.com/minimaxir/facebook-page-post- scraper](https://github.com/minimaxir/facebook-page-post-scraper)] which explicitly follows the permissions set by the Facebook API.)

<comment><USER5> This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will.

Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

<comment><USER6> >Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

Not really.

Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info.

If you are slamming the site with requests because of your scraping, yeah you need to knock it off. If you throttle your scraper in proportion to the size of their site, you aren't really harming them.

In regards to "stealing info", as long as you aren't taking info and selling it as your own (which it seems OP is indeed doing), that is just fine.

tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business.

<comment><USER7> > If you throttle your scraper in proportion to the size of their site, you > aren't really harming them.

And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.

<comment><USER8> Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause _less_ impact because they usually don't download images or execute JavaScript.

I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume _at least_ 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was conserved too.

Because we were a specialized browser used by people looking for a very specific piece of data, we could employ caching mechanisms that meant that each person could get their request fulfilled without having to hit the data source's servers. We also had a regular pacing algorithm that meant our users were contacting the site way less than they would've been if they were using a conventional desktop browser.

Our service saved the data source a _large_ amount of resource cost. When we were shut down, their site struggled for about two weeks to return to stability. I think they had anticipated the opposite effect.

Our service also saved our users a large amount of time. We were accessing publicly-available factual data that was not copyrightable (but only available from this one source's site). There's no reason that the user should be able to choose between Firefox and Chrome but not a task-specialized browser.

It is true that some people will (usually accidentally) cause a DDoS with scrapers because the target site is not properly configured, but the same thing could be done with desktop browsers. It doesn't mean that scrapers should be disadvantaged.

<comment><USER9> A small counterpoint to this -- in the airline industry, it's relatively commonplace for seat reservations to be made for a user _before_ payment has occurred. In this case, if you're mirroring normal browser activity, you can (temporarily) reduce availability on a flight, potentially even bumping up the price for other, legitimate users, and almost certainly causing the airline to incur costs beyond normal bandwidth and server costs. I'm sure there are many other domains for which this is also the case, however rare.

<comment><USER10> If they don't do the seat reservation behind a POST, or at least blacklist the reservation page in robots.txt, I have no sympathy.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> Corporations will abuse your personal integrity whenever they get a chance, while abiding the law. Corporations will cry like babies when their publicly available data (their livelyhood) gets scraped. They will take you to court.

They consider their data to be theirs, even though they published it on the internet. They consider your data (your personal integrity) to be theirs as well, because how can you assume personal integrity when you are surfing the internet?

I have high hopes that the judicial system some time not too far from now will realize that since the law should be a reflection of the current moral standings it will always be behind, trying to catch up with us and that those who break the law while not breaking the current moral standings are still "good citizens" unworthy of prison or fines.

I guess Google won this iteration of the internet because of the double- standars site owners stand by, to allow Google to scrape anything while hindering any competitors from doing the same. There will only be a true competitor to Google when we in the next iteration of the internet realize that searching vast amounts of data (the internet) is a solved problem, that anyone can do as good a job as Google, and move on to the next quirk, around wich there will be competition, and in the end that quirk will be solved, we'll have a winner, signaling that is it time to move on to the next iteration.

<comment><USER12> > Corporations will abuse your personal integrity whenever they get a chance, > while abiding the law.

Call my cynical if you will, but I'd leave "while abiding the law" out of that, or at least replace it with "while hoping they aren't breaking the law". Due diligence on these matters is often sadly lacking. They'll take the information first and only consider any such implications when/if they come up later.

Large organisations like Google probably will make the up-front effort to remain legal, because they are in the public eye enough for lack of doing so to attract a lot of unwanted press, but you don't have to get a lot smaller than that to start finding companies who are a lot less careful (or in some cases wilfully negligent).

<comment><USER13> I would use Microsoft as a precedent. Sure they will attempt to stay legal but by pushing it as far as they can.

For instance the browser choice script that came with Windows imposed by the EU never worked. It was a "bug". Somehow they must have omitted to test the feature...

Until last year Microsoft started playing nice, and I think Google and Facebook have become the new corporate villains. But recently the Windows team seems to be minded to challenge them in that position.

</comment></comment></comment><comment> fake-name I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls.

I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt.

At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for message passing middleware, SaltStack for automated VM provisioning, and python everywhere for everything else. Using some randomization, and a list of the top n user agents, I can randomly generate about ~800K unique but valid-looking UAs. Selenium+PhantomJS gets you through non-capcha cloudflare. Backing storage is Postgres.

Database triggers do row versioning, and I wind up with what is basically a mini internet-archive of my own, with periodic snapshots of a site over time. Additionally, I have a readability-like processing layer that re-writes the page content in hopes of making the resulting layout actually pleasant to read on, with pluggable rulesets that determine page element decomposition.

At this point, I have a system that is, as far as I can tell, definitionally a botnet. The only things is I actually pay for the hosts.

\---

Scaling something like this up to high volume is really an interesting challenge. My hosts are physically distributed, and just maintaining the RabbitMQ socket links is hard. I've actually had to do some hacking on the RabbitMQ library to let it handle the various ways I've seen a socket get wedged, and I still have some reliability issues in the SaltStack-DigitalOcean interface where VM creation gets stuck in a infinite loop, leading to me bleeding all my hosts. I also had to implement my own message fragmentation on top of RabbitMQ, because literally no AMQP library I found could _reliably_ handle large (>100K) messages without eventually wedging.

There are other fun problems too, like the fact that I have a postgres database that's ~700 GB in size, which means you have to spend time considering your DB design and doing query optimization too. I apparently have big data problems in my bedroom (My home servers are in my bedroom closet).

\---

It's all on github, FWIW:

Manager: [https://github.com/fake- name/ReadableWebProxy](https://github.com/fake-name/ReadableWebProxy)

Agent and salt scheduler: [https://github.com/fake- name/AutoTriever](https://github.com/fake-name/AutoTriever)

<comment><USER14> How do you circumvent cloud provider IP blocks? For example, one site blocks all requests from AWS EC2 servers.

<comment> fake-name None of the sites I'm scraping do that, mostly.

I'm not scraping high value sites like that (I mostly target amateur original content). It's not really of interest to other businesses. As such, I tend to just run into things like normal cloud-flare wrapped sites, and one place that tried to detect bots and return intentionally garbled data.

If I run into that sort of thing, I guess we'll see.

</comment></comment></comment><comment><USER15> A neat trick I sometimes use to "scrape" data from sites that use jquery ajax to load data is to plug in a middleware in jquery xhr:

          $.ajaxSetup({
            dataFilter: function (data, type) {
              if (this.url === 'some url that you want to watch!') {
                // Do anything with the data here
                awesomeMethod(this.data)
              }
              return data
            }
          })

I remember last using it with an infinite-scroll page with a periodic callback that scrolled the page down every 2 seconds, and the `awesomeMethod` just initiated the download. Pasted it all in dev-tools console, and the cheap "scraper" was ready!

<comment><USER16> You can also build a chrome extension if you need to navigate to multiple pages and use a long-running scraping process. I've done this several times and it's really easy to get one up and running if you use an extension boilerplate (30 minutes tops).

<comment><USER17> do you have something? i was going to write the very same extension (but distributed so i could add it to my pc and my friends) but never did that

<comment><USER16> This is the boilerplate I used last time: [http://extensionizr.com](http://extensionizr.com)

<comment><USER15> Didn't know about `extensionizr`, Looks super cool. Thanks!

</comment></comment></comment></comment></comment><comment><USER18> This good list of tactics underscores, for me, how the state of the Web has made it a lot more difficult to teach web scraping as a fun exercise for newbie programmers. It used to be you could get by with an assumption that what you see in the browser is what you get when you download the raw HTML...but that's increasingly less common the case. So now you have to teach how to debug via the console and network panel, on top of basic HTTP concepts (such as query parameters).

(Even more problematic is that college kids today seem to have a decaying understanding of what a URL is, given how much web navigation we do through the omnibar or apps, particularly on mobile, but that's another issue).

I've been archiving a few government sites to preserve them for web scraping exercises [0] (the Texas death penalty site is a classic, for both being relatively simple at first, and being incredibly convoluted depending on what level of detail you want to scrape [1])). But I imagine even government sites will move more toward AJAX/app-like sites, if the trend at the federal level means anything.

That said, I think the analytics.usa.gov site is a great place to demonstrate the difference between server-generated HTML and client-rendered HTML.

But as someone who just likes doing web-scraping, I feel the tools have mostly kept up with the changes to the web. It's been relatively easy, for example, to run Selenium through Python to mimic user action [2]. Same with PhantomJS through node, which has vastly improved how accurately it renders pages for screenshots compared to what I remember a few years back

[0] [https://github.com/wgetsnaps](https://github.com/wgetsnaps)

[1] [https://github.com/wgetsnaps/tdcj-state-tx-us-- death_row](https://github.com/wgetsnaps/tdcj-state-tx-us--death_row)

[2] [https://gist.github.com/dannguyen/8a6fa49253c1d6a0eb92](https://gist.github.com/dannguyen/8a6fa49253c1d6a0eb92)

<comment><USER19> It's unfortunate that nearly every webpage these days is a Javascript State Machine which you have to execute in a sandbox and inspect its internal state to get stuff out of.

On a blog post by Paul Kinlan ('Open Web Advocate' at Google and Chromium) [1], I lamented that we ended up here instead of the semantic web because the semantic web was hard to execute. Instead, every web page is a black-box, only navigable by an intelligent and/or sufficiently persuadable human.

But this is also why I don't buy ethical arguments against scraping. Sure, _legally_ any company can unilaterally set any TOS prohibition against behavior they don't want, and these terms may be tested in court. But navigating a page in an automated manner that's designed to resemble interactions of humans (ie. through Selenium) is in my opinion _ethical_ , because it merely time-shifts a user's activity.

[1] [https://news.ycombinator.com/item?id=12206846](https://news.ycombinator.com/item?id=12206846)

</comment></comment><comment><USER20> Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I did not proceed to the next interview and ignored her emails ", this just shows a lot of immaturity.

<comment><USER21> Part of the turnoff for me was the middle-schooler tone and vocabulary. Good walkthrough with good code examples though, obviously written by a very smart JS dev.

<comment><AUTHOR> Ok, I'll try to explain to this thread. I actually thought about removing the Facebook part, but I kept it in there because that is kind of how I felt and it is real. The middle-schooler tone and vocab is probably because I don't read a lot of books, and English is my 2nd language.

In reply to XCSme - no I am not new to Node and my point of the post is to illustrate some of the techniques that I haven't seen published anywhere to HN and the community. My focus is quite different from what you think it is, so maybe it is my bad for bad writing skills, I'm still new to writing and learning.

</comment></comment></comment><comment><USER22> Not wanting to thread hijack, but just going to post an article I wrote a few years back as it covers a few other things that are still relevant; and often still gets referenced. May it'll help some people out in combination with OP's post.

[http://jakeaustwick.me/python-web-scraping- resource/](http://jakeaustwick.me/python-web-scraping-resource/)

<comment><USER23> I was surprised to not see Scrapy listed, but then I saw there were some comments about it - but seriously, doing by hand what Scrapy has spent _years_ perfecting is highly suboptimal.

I guess the distinction is between whether one wants to just "toy around" or run the spider for-real.

</comment></comment><comment><USER24> I wrote a fairly complex spidering and scraping script in Node a few months ago. I found downcache[1] to be absolutely invaluable, particularly as I was debugging my parsing scripts, a I was able to rerun them relatively quickly over the cached responses.

However, when the network was no longer a bottleneck, I found that the speed and single-threaded nature of Node became one. It wasn't really that slow, relatively speaking, but I had a few hundred gigs of HTML to chew through every time I made a correction, so it was important to keep the turnaround as fast as possible.

I eventually managed to manually partition the task so I could launch separate Node scripts to handle different parts of it, but it wasn't a perfect split, and there was a fair bit of duplicated work, where a shared cache would have helped a great deal.

In retrospect, I should have thrown my JS away and started again in something with easy threading like Java or C#. But -- familiar story -- I'd underestimated the complexity of the task to begin with, and by the time I understood, I'd sunk a lot of time into writing my JS parsing code and didn't fancy converting it all to another language, particular when it always seemed like "just one more" correction to the parsing would make everything work right. In the end, what was supposed to take a weekend took about three months of work, off and on, to finish.

[1] [https://www.npmjs.com/package/downcache](https://www.npmjs.com/package/downcache)

<comment><USER25> Threading in node is very easy, just use clusters. Alternatively, take any of the CPU intensive activity, like parsing the HTML and formatting as JSON, and just put that on an AWS lambda.

You can invoke as many lambdas from your application as you want in parallel and you're not going to be bottlenecked by your CPU :)

<comment><USER24> Clustering in Node creates isolated child processes, not threads. I needed to have shared queues, in-memory caches, and hashes to coordinate workers and avoid them doing duplicate work.

I'm did consider using clustering and having some master process coordinate everything, and using some shared-memory caching library. But it would not be "easy" to set up, especially compared to something like Java where you get thread pools and synchronized thread-safe collections out of the box.

And Lambda would have been totally impractical. As I said, I had hundred of gigs of data to process. If I'd been uploading this over my puny ADSL upstream every time, I'd still be waiting for a single run to complete.

I'm not trashing Node. I like it. There's a reason I used in the first place, after all. But for this particular use-case, I didn't find it was very good fit.

<comment><USER26> Threading for a crawler is just a dirty way of not handling distribution. When you will need more than one server your threads won't save you. It has nothing to do with Node.js and thread support.

<comment><USER24> I wasn't creating a new search engine, I was doing a one-off scraping job in my spare time. Creating a fully distributed solution would have been total overkill. But threading could and would have helped.

Honestly, stupidly hostile and ignorant comments like this are the absolute worst thing about Hacker News.

</comment></comment></comment></comment></comment><comment><USER27> Scraping with Selenium in Docker is pretty great, especially because you can use the Docker API itself to spin up/shut down containers at will. So you can spin up a container to hit a specific URL in a second, scrape whatever you're looking for, then kill the container. This can be done via a job queue (sidekiq if you're using Ruby) to do all sorts of fun stuff.

That aside, hitting Insta like this is playing with fire, because you're really dealing with Facebook and their legal team.

<comment><USER28> Serious question: What do you gain from having an extra layer like docker?

<comment><AUTHOR> Selenium grid runs in docker, so it's easy to have multiple instances running. Better control.

<comment><USER29> What are the advantages of this versus a thread pool of web drivers? I'm not really familiar with Selenium Grid.

<comment><USER30> Grid can dynamically dispatch based on the browser and capabilities you want when you create the session.

</comment></comment></comment></comment></comment><comment><USER31> > AngelList even detects PhamtomJS (have not seen other sites do this).

I run a site that aggregates/crawls job boards for remote job postings, and AngelList has been VERY difficult to crawl for various reasons, but you easily get PhantomJS to work (I have). Having said that, I've never felt very good about the fact that I'm defeating their attempts to block me (even though I feel like I'm doing them a favor) and will likely retire that bot soon.

It kinda sucks that I'm just grabbing publicly-available content in a very low-bandwidth way, but I really can't convince myself that what I'm doing is very ethical.

My to-do list includes making my crawler into a more well-behaved bot and that will have to go.

<comment><USER32> I think you may want to decouple your ethical analysis from which private company is making the most money. Remember that the only functional difference between you and somewhere like kayak.com or padmapper is business relationships.

</comment></comment><comment><USER16> I don't know why more people don't use chrome extensions for scraping. Using a boilerplate[1], you can get a scraper up and running in minutes. Start a node server that serves up urls and stores parsed data, and run the scraper in the browser. Best of all, you can watch it running and debug if something goes wrong. I know it doesn't scale well if you're running a SaaS, but for personal projects and research/data normalization it's the lowest barrier to entry, in my opinion.

[1] [http://extensionizr.com](http://extensionizr.com)

</comment><comment><AUTHOR> Sorry guys, hit by traffic - just scaling my EC2 at the moment.

<comment><USER19> No worries, we had your page scraped just in case ;)

Google Cache link: [http://webcache.googleusercontent.com/search?q=cache:https:/...](http://webcache.googleusercontent.com/search?q=cache:https://franciskim.co/2016/08/24/dont- need-no-stinking-api-web-scraping-2016-beyond/)

Archive.is link: [http://archive.is/DQccs](http://archive.is/DQccs)

<comment><AUTHOR> haha :)

</comment></comment></comment><comment><USER33> Good stuff.

I do a good bit of scraping, and made RubyRetriever[1] to make my life easier but it seems like I'm getting roadblocked on occasion, probably due to some of the things you mention in your article.

Is there any way for a site to verify that only their JS and CSS files are linked? Like preventing injection?

[1]: [https://github.com/joenorton/rubyretriever](https://github.com/joenorton/rubyretriever)

<comment><USER34> You could inspect the src attributes of script tags, and the href attributes of link tags with rel="stylesheet", for acceptable domains. I doubt it would cover all cases, but it might be a start.

</comment></comment><comment><USER35> At Feedity ([https://feedity.com](https://feedity.com)), we "index" webpages to generate custom feeds. Over the years, we've designed our system to use a mix of technologies like .NET (C#) and node.js, and implemented a bunch of tweaks and optimizations for seamless & scalable access to public content.

<comment><USER26> Any tips and tricks you are able to share about the technologies you guys developed? It would be especially interesting to see what you use for text extraction from HTML.

</comment></comment><comment><USER36> > But if you are automating your exact actions that happen via a browser, can > this be blocked?

Yes, by checking times between actions and number of actions in a time period, and blocking atypical activity. I was IP banned from a site once for a few months, after trying to scrape it too much and hitting links on the site that were hidden from humans.

The random wait settings specified in the post are better than nothing, but still too flimsy. You would need to put hours between requests, only request during a certain 15 hour periods, take days off, and eventually you aren't scraping regularly enough to do much good.

Scraping is not an API, and I should know- I used to do it for a living. Its unreliable. It requires constant maintenance. APIs can break too, but they are meant for the sort of consumption you are trying for.

If you scrape for a living, only do it as a side job.

<comment><USER37> It really depends on the data you are scraping. My main business relies on scraping and my data mining application has been running for over 5 years. If you have enough IP addresses available to you, it becomes almost impossible to distinguish it from normal users hitting the site...and bandwidth has gotten so cheap, the overhead is very affordable.

I've noticed that most sites actually don't change that often. I deal with changes once or twice every 3 months.

"If you scrape for a living, only do it as a side job."

This is true if you are scraping the low hanging fruit. I scrape 40+ sources (I do have access to a few APIs as well) and then have to extract the patterns/data I need to then integrate it into my business model. This is all automatic now and I only work on upgrading for speed and efficiency.

If you have to scan millions of urls daily from 1 site, it's probably not going to work out. You need to figure out clever ways of getting the data and using it without breaking any laws or pissing off the site owner.

</comment></comment><comment><USER38> I actually love Selenium for this purpose, for much the same reasons the author mentions here.

It's almost impossible for a website to reliably detect that a client web browser is being automated, and I find I can make Selenium scripts much more adaptable to breaking changes in websites when they occur than I can when hooking up my code directly.

I actually disagree with the contention that Selenium is slower than directly scraping though. The Firefox driver has always been lightning fast for me and the bottleneck is almost always server requests that would have been necessary either way.

</comment><comment><USER39> Whilst they mean well, I find this a fundamentally deceptive — the arduous parts of "real world" scraping simply aren't in the parsing and extraction of data from the target page, the typical focus of these "scrape the web with X" articles.

The difficulties are invariably in "post-processing"; working around incomplete data on the page, handling errors gracefully and retrying in some (but not all) situations, keeping on top of layout/URL/data changes to the target site, not hitting your target site too often, logging into the target site if necessary and rotating credentials and IP addresses, respecting robots.txt, target site being utterly braindead, keeping users meaningfully informed of scraping progress if they are waiting of it, target site adding and removing data resulting in a null-leaning database schema, sane parallelisation in the presence of prioritisation of important requests, difficulties in monitoring a scraping system due to its implicitly non- deterministic nature, and general problems associated with long-running background processes in web stacks.

Et cetera.

In other words, extracting the right text on the page is the easiest and trivial part by far, with little practical difference between an admittedly cute jQuery-esque parsing library or even just using a blunt regular expression.

It would be quixotic to simply retort that sites should provide "proper" APIs but I would love to see more attempts at solutions that go beyond the superficial.

<comment><USER40> > the arduous parts of "real world" scraping simply aren't in the parsing and > extraction of data from the target page, the typical focus of these "scrape > the web with X" articles.

I can agree with this after having written a scraper as part of core business functionality (we paid a company for access, but access was just to bare HTML blobs and CVS and not an actual API).

However, to what degree you want to do all this is negotiable whereas the 'core' of screen-scraping is not---all scrapers have to first figure out how to get text, parse it, then stick it back in their system.

An example of what I mean when I say 'negotiable' is....

> working around incomplete data on the page

Deciding how to do this depends on your problem domain. Sometimes, we'd get bad computed data from our source but not care because it just meant more work putting more work in calculating it from a more raw source.

> not hitting your target site too often

If they publish how often you are allowed to scrape, this isn't too difficult. If not, then trial and error is the only solution. On occasion, a site simply just doesn't know/care. For example, in my case, the site was static content behind a CDN, so that if we were anywhere under 200 req/second then no flags would ever be raised.

For most smaller sites, that you are unofficially scraping, you may be limited to 1 request every 2 seconds.

</comment></comment><comment><USER41> What bother me the most is that recently I wanted to extract and archive of all the threads I participated in from an Internet forum. The webmaster told me that the BBS he use don't provide such a function and that I just had to download each thread manually... (300+ thread in my case).

He then say that it don't bother him if I scrape theses thread. And I'm currently figuring out how to manage his site's cookie protected search feature, so that my painstaking effort (I'm not a dev, more a DB guy) could be reproducible more easily by other users of this service.

But this shouldn't appen in the first place because all post of this service are stored in a cleanly organized MySQl DB. Yet as no method is provided the only way to get back structured data is by scrapping (as the webmaster told me that no, he won't run custom SQL because "he don't want to mess his DB").

So even if all the data is publicly available through the internet forum only a geek can download a personal archive... or google because google scrape and store everything.

<comment><USER42> It's overkill for most things, but I have found that on occasion the best way to scrape stuff behind annoying frontends is with Selenium. pysaunter is a useful library that's one layer of abstraction higher, if you're familiar with Python.

<comment><USER42> Well I see now that I'm really late to the party with that comment.

</comment></comment></comment><comment><USER43> As someone who does a lot of scraping, I was happy to learn about Antigate :)

<comment><USER43> Just joking as I don't scrape unless scraping is allowed. :)

</comment></comment><comment><USER44> It's trivial to scrape public Instagram URLs...

[https://github.com/kingkool68/zadieheimlich/blob/master/func...](https://github.com/kingkool68/zadieheimlich/blob/master/functions/instagram.php#L421-L428)

</comment><comment><USER45> Does anybody know what the author means by "lead" (noun)?

I don't think it's any of the regular meanings: [http://www.ldoceonline.com/search/?q=Lead](http://www.ldoceonline.com/search/?q=Lead)

But it doesn't seem to be any of these slang terms either: [http://www.urbandictionary.com/define.php?term=lead](http://www.urbandictionary.com/define.php?term=lead)

<comment><USER46> [https://en.wikipedia.org/wiki/Lead_generation](https://en.wikipedia.org/wiki/Lead_generation)

</comment></comment><comment><USER47> Have you run into any issues from running all of your scrapers off of AWS, or just from sites detecting that you're accessing large numbers of pages in some sort of obvious pattern? I guess I was hoping there would be sites with more interesting ways to screw with web scrapers (rearranging certain page elements or something) than just throwing up a CAPTCHA.

<comment><USER6> Most really don't. A lot of big sites don't seem to care, at least in my experience.

The few that I've seen just 'ban' your IP for a few minutes. If you hit Wikipedia too much too quickly, they will essentially refuse to serve you for a while. It was a number of years ago I was doing it, but basically you would be scraping then you would just stop getting info (Maybe I wasn't reading response codes and could've realized quicker what was happening)

<comment><USER48> Wikipedia provides you with an API and guidelines on how to use it, so you really shouldn't be scraping it directly or so much you hit enforced limits.

<comment><USER49> Wikipedia provides archives of all its content.

No need to scrap it when you can readily download a nicely formatted .xml.zip file containing all knowledge written by mankind.

</comment></comment></comment></comment><comment><USER50> > But if you are automating your exact actions that happen via a browser, can > this be blocked?

Of course it can! You won't be able to defeat even the simplest attempt on anti-scraping based on statistical data. Like even keeping a list of individual rate-limits for /16 subnets of actual visiting users and you are in trouble.

</comment><comment><USER51> Does cheerio account for single page apps? In any case thanks for the tutorial!

Anyways I added your stuff here along with other data mining resource:

[https://github.com/kevindeasis/awesome-fullstack#web- scrapin...](https://github.com/kevindeasis/awesome-fullstack#web-scraping)

</comment><comment><USER52> To fight scrapers, we show some values as images that look like text (but not all the time)

And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors.

You can't stop them, but you can make their lives painful.

<comment><USER53> > To fight scrapers, we show some values as images that look like text

You are fighting screen readers more than anything; as well as legitimate plugins, form autofills, etc. If this is for captcha, you are fighting all the users as well.

> And we insert random (non-visible) html and css classes in our site to screw > with em, and use randomized css classnames.

Legitimate browser plugins, etc. I'd just use electron or selenium with `nth- child`, `:visible`, `[class*="…"]`, etc.

What you effectively doing is wasting time on useless stuff. This is even more useless than trying to prevent copying of DVDs or pirating games.

<comment><USER54> > What you effectively doing is wasting time on useless stuff. This is even > more useless than trying to prevent copying of DVDs or pirating games.

Can you be so sure? The Union blockade of the Confederacy had plenty of holes, and smugglers / privateers / blockade-runners made good money getting through (when they survived) ... but that doesn't mean the blockade wasn't effective all the same at weakening the Confederate military and economy.

<comment><USER53> Do you really not see the difference between military blockade and randomizing CSS classes?

<comment><USER54> Honestly, no. This one time I was pissed off at Egypt for undercutting me in cotton prices, so I tried to set up a blockade to prevent merchant ships going in and out of Cairo.

... and it would have worked, too, except for my naval vessels were all CSS classes. I even tried to name them cleverly, a la "USS hero unit" or "USS datatable table-consensed span9", but my plan was foiled.

</comment></comment></comment></comment></comment><comment><USER55> Hooray Melbourne! Would be interested seeing this at a meetup group if you were thinking of presenting.

<comment><USER56> Another 3000'nder. Would be great to see this turned into a talk somewhere.

<comment><USER55> For sure. Trying to think which ones. Probably the MelbJS one and maybe dddmelb? You could modify it to talk at the OWASP one perhaps. Which ones have you been to?

</comment></comment></comment><comment><USER57> While everyone is busy debating whether scraping is bad or legal, I just can't stop thinking a out Antigate.

Of the sweatshops that must have been setup to deliver this service. That, is to me the true horror of this story.

</comment><comment><USER58> I wonder how effective the CloudFlare anti-scrapper protection is against this approach of breaking captchas.

Also, I find it interesting that big websites don't just block all traffic from AWS IPs as they do with Tor.

<comment><USER49> There can be legitimate traffic coming from AWS, if not the site itself.

It's especially true when the site provides an API and is meant to be integrated by people/companies. In which case, the AWS traffic is likely to include major and/or important and/or paying customers. You really don't want to block that.

On the other hand, Tor is likely to be 90% evil. When in doubt, just block it. (That makes me think, I should run some proper stats and maybe publish a blog post about that. )

<comment><USER58> > There can be legitimate traffic coming from AWS, if not the site itself.

The traffic from the site itself, if it's hosted there, would come from the intranet IP address, right? Not the public facing one.

> It's especially true when the site provides an API and is meant to be > integrated by people/companies. In which case, the AWS traffic is likely to > include major and/or important and/or paying customers. You really don't > want to block that.

Agreed, but it's fairly easy to block the AWS IP traffic on web endpoints and not on the API endpoints.

</comment></comment></comment><comment><USER59> And from the trenches:

\- rails application

\- scraping with nokogiri gem on Ruby

\- simple models doing the scraping in rails app

\- some scraping is parsed with CSS selectors - <USER3>

\- some scraping is parsed with regex - <USER3>

\- persisting to DB, Text, even Google docs

\- presentation on web, text, pdf, xls

Boom

</comment><comment><USER60> How do you push a button like hit next on a paginated page?

<comment><USER61> Right click on the 'next' button in chrome and use 'inspect element' to find its id/class/css selector and then:

browser.findElement(webdriverio.By.id('#Next')).click();

</comment></comment><comment><USER62> There is so much that's missing from this. What about gathering tokens from customers vs. paying for social data feeds? How about canned services like 80legs?

<comment><AUTHOR> Hmm yeah there are a lot of other things that I could write about. 80legs seem like another Scrapy type of SaaS? Not sure what you mean about gathering tokens from customers.

<comment><USER62> I've heard of companies that scrape on behalf of customers who will walk marketing people through the process of creating an API token to help mitigate rate limiting.

</comment></comment></comment><comment><USER63> Currently getting 502 Gateway. Guessing this post is also trending on reddit and we hugged it to death :(.

<comment><AUTHOR> I'm on Reddit?

<comment><USER64> Google Analytics > Acquisition > Source/Medium > type "reddit" in search bar. Add secondary dimension "referral path"

</comment></comment></comment><comment><USER65> When I was building liisted.com I scraped using Selenium and it worked great.</comment>


Ask HN: What's wrong with my JS? - <AUTHOR> I recently applied for a frontend JavaScript developer position. The first step of the interview process the company asked me to build a car selector in JavaScript. After reviewing the instructions, I decided to use plain JS and EventEmitter.
I was quickly rejected after I submitted my work. The company said they are moving forward with candidates who exhibit stronger coding indicators. I was surprised to be rejected for work that I thought was good, given the boundaries of the task. I created modular code, wrote tests to influence the design, and even used some functional programming.
Should I have put more love into the UI? Should I have used a JS framework like angular or react? Maybe my code is crap and I don't see it.
Obviously, the company can't offer any insight, but can you? I'm stumped. I very much want to know what things I did wrong or areas where I can improve. Thanks!
Task description is in README.md http://bl.ocks.org/luk3thomas/512896ffd7a5dd325de1 <comment><USER2> To start as a couple of others have already pointed out you haven't implemented all the criteria.

If I was evaluating your solution based on what they are asking for I would have passed on you as well. Your solution is unnecessarily complex and over built. They are asking if can you use JS to manipulate a select box. Not including tests you have brought in 400+ lines of code (184 lines of code + the EventEmitter library) to implement 75% of the functionality you were asked for. That is a lot of code to take an item from a select box and add it to a list. Knowing that developers leave I would not want to inherit a solution like this.

To put it in car terms you built a 911 when the customer was asking for a vw bug.

</comment><comment><USER3> As someone who has interviewed 50+ developers in the past six months: that code sample would absolutely get you an in-person interview. Your code is fine and the company was probably just giving you a cop-out answer for whatever reason. Don't be discouraged; you should keep interviewing!

</comment><comment><USER4> "After a car has been added to the wish list, the select box should default to the first alphabetically available car from the same make".

Doesn't happen. Leaves the make and returns to top make after selection.

<comment><AUTHOR> omg, you're right.

<comment><USER5> Kind of sneaky to put important details in parentheses like they did. I would have let that omission slide, especially considering the typical candidate we get.

</comment></comment></comment><comment><USER6> You missed #3 as far as I can tell.

</comment><comment><USER7> this requirement:

"it also only need to work in Chrome – no IE testing necessary"

sounds like they had a preference for AngularJS. I guess your eventEmitter solution was too advanced for that company.

<comment><USER2> That sounds like a huge leap that because they are only going to test the solutions in chrome that he should have used angular.

<comment><USER7> without putting value to Angular, many companies that say "works in Chrome" mean "developer friendly". And from "developer friendly", it seems a small step to Angular. Maybe because many developers don't want to invest time learning real JS.

</comment></comment></comment><comment><USER8> Just curious, was it a junior dev position or intermediate?

<comment><AUTHOR> The company didn't make a distinction -- just JavaScript Developer.

</comment></comment><comment><USER9> Were you limited by time for this test?

<comment><AUTHOR> No, there was no time limit</comment></comment>


How to Survive an Atomic Bomb - <AUTHOR> https://jasonlefkowitz.net/2012/03/how-to-survive-an-atomic-bomb/ <comment><USER2> This is some good solid writing, I enjoyed reading it.</comment>


Hacking POS Terminal for Fun and Non-profit - <AUTHOR> http://h30499.www3.hp.com/t5/HP-Security-Research-Blog/Hacking-POS-Terminal-for-Fun-and-Non-profit/ba-p/6540620#.U8lhTvldXXp

<comment><USER2> Thought I could contribute my 5c here. I have been working in retail for around 4 years now, and dealt with PoS systems extensively: searched for the __right __software ( _protip: there is none_ ), developed a master DB for synchronizing the product catalog across several locations, gathered reports and maintained the infrastructure. After all, I find the lack of good PoS software frustrating ( _at least here in Norway_ ). The more or less affordable solutions are decades old, running VB macros on top of MS Access databases, they are often incompatible with versions of Windows above XP. Everything seems stitched together over the course of many years by various WYSIWYG Studio dilettantes. Extending such software with new functionality is often hard or even impossible, which their developers admitted on several occasions. For instance, I've once got an estimate of $5000 for integrating an old DIGI scale, with a condition that I'd manage to find a datasheet for that protocol. I ended up writing a small COM port emulator and used it as a proxy between PoS and the real port.

Fun thing about PoS is the huge assortment of various hardware, and all of it could be abstracted in business logic. Almost every country has it's own ways of collecting taxes, own currency, own central company registers and different ways of reporting -- all of it should be considered when designing, a dream of every architecture astronaut. In addition come these necessary things like stock control, handheld scanners, BI, printing, Unicode ( _never seen it supported by any PoS; imagine running an international foods store with ISO Latin-1_ ) and last, but not the least, user-friendliness.

I guess the PoS software market is much larger in the States, but what about the average software quality? PoS systems are not going anywhere soon, and I feel that a startup dealing with these issues could definitely see success. If anyone else feels like doing it, I'd be more than glad to contribute.

<comment><USER3> I'm the CEO of startup tackling this space. Hit me up hello at touchpoint dot io. You're right that this market is rife with garbage technology which represents a large opportunity, but the MVP to run someone's business is complex. Still, we're just starting to ship and have the attention of major chains. And not just because of our Unicode support, distributed dbs, or PCI- exempt security.

</comment></comment><comment><USER4> The fact that the writer could even get a used system that had not been wiped by the previous owner is a huge attack vector. This coupled with the fact that employee SSN's were stored in plain text makes a huge incentive for anyone interested in buying these things up on Ebay. Even if 1/10 wasn't wiped I bet it would still be worth it.

<comment><USER5> I legitimately don't understand why a machine like this would ever need to know an employee's SSN. I can understand why the employer might need to know it for tax reasons or similar, but why does the POS terminal?

<comment><USER6> I'd guess that they'd want to give each employee a unique id, and they used the SSN because it's something that every employee _has_ to have, so they didn't have to generate a new id number and sync it up with their accounting software. This has obvious security issues, as we see here, of course.

They'd need it integrated per-employee with accounting, since it'd be tracking tips via credit cards which have to make it into payrolls and taxes etc.

</comment></comment></comment><comment> slipstream- I saw this on /r/netsec a few days ago. One of the comments there mentioned that some POS devices are still vulnerable to the VNC authentication bypass exploit from 2006. The "we lose money if we patch"/"don't have time to patch, got this stuff to do" mentality is strong with this one.

<comment><USER7> That doesn't surprise me. I've seen these things (Aloha terminals) still running Windows 98. BOH was on Windows XP at least. From my experience, it's less about not having time, and more about being completely unaware of the requirement to patch the things.

<comment><USER8> The management of OS updates also falls between resposibilities. Probably, the shop/chain owners will be responsible for OS and virus scanner updates, and because of lack of knowledge on their part, have been cause of critical disruptions in the past.

I have some hands-on experience with POS-systems from the vendor side of things, and it has happened multiple times that a customer led security patch carnival has led to major issues (windows needing to reboot 3 times before it became stable enough to make a print again, seriously?).

</comment></comment></comment><comment><USER9> I worked technical support for a POS company that was in a very large chain of fast food pizza joints, which was owned by a company that did POSes for a lot of customers.

Their security was abysmal. My SSH key had to be manually added to 75% of systems because the automated system didn't work. All of the systems were running on an EOL version of a desktop version of Linux.

On the topic of this video, on the machines were all employees SSNs in a Postgres database, along with loads of other personally identifiable information.

</comment><comment><USER10> I've worked with aloha in the past. These are usually installed in their own physical switch by the server. They think that makes them secure... It's a joke. Squirrel POS is just as bad. I try to push these kinds of folks onto cloud based systems these days.

<comment><USER11> Cloud based POS sounds like a nightmare if you had downtime...

<comment><USER12> Most Cloud based POS have an offline mode, which still allows you to process transactions. Vend (vendhq.com) stores a local copy of the information needed to process sales (products, taxes, payment types) and allows you to continue processing sales, although it restricts you to selling and you can't manage inventory etc

</comment></comment></comment><comment><USER13> Just because Automatic Updates is turned off doesnt mean they haven't been applied. If they are running a custom build of XP-SP2 they could easily deploy and install updates on their own, no? Id be interested to look at what's installed.

</comment><comment><USER14> I did some work with PoS systems in the early 90s and I shit you not the VNC password on the customer deployed machines was 'godmode'. These machines were still used, with the same password most likely, in 2010. Scary shit.</comment>


MyDiago - Never Lose Your Stuff Again - <AUTHOR> http://www.kickstarter.com/projects/mydaigo/mydaigo-is-helping-you-get-more-of-that-found-it-f There are many applications for this device, but the one i'm focusing on is people and animals - so any guidance, access you can provide I would greatly appreciate it! <comment><USER2> This product is awesome, and will help solve a problem that all of us face - losing our valuables like our smartphones and keys. Definitely worth backing the project on Kickstarter!

<comment><AUTHOR> Thanks Steve!

</comment></comment><comment><USER2> Here's the link to back the project! [http://kck.st/16G2Afh](http://kck.st/16G2Afh)</comment>


The Internet Made Me Sad Today - <AUTHOR> http://raganwald.posterous.com/the-internet-made-me-sad-today <comment><USER2> Those ads on the rest of the page, and the sad part about all of this, remind me of what Steve Jobs said about network TV:

"When you're young, you look at television and think, There's a conspiracy. The networks have conspired to dumb us down. But when you get a little older, you realize that's not true. The networks are in business to give people exactly what they want. That's a far more depressing thought. Conspiracy is optimistic! You can shoot the bastards! We can have a revolution! But the networks are really in business to give people what they want. It's the truth."

<http://www.informationliberation.com/?id=36783>

</comment><comment><USER3> This reminds me of an episode of South Park:

[<http://www.southparkstudios.com/clips/154597/find-the-heart>]

The kids try to find the "heart of Wallmart" so they can destroy it (being the evil mom-and-pop store demolisher that it is). When they finally locate the mythical Heart of Wallmart, it turns out it just a mirror. In other words, the problem is not Wallmart (or Google, or whatever), it is us.

On another note, games (in general) are not so bad (although the ones they were advertising probably were). Even the games many people might view as mindless (such as your average FPS) are actually deeply embedded with strategy. I actually _learned_ a lot from games - not just strategy or spacial intelligence or hand eye coordination, but many of the same things I would have learned if I picked up a science fiction or fantasy book. Children today are glued to screens, and I think the only cure may be to embrace this and find a better way to use games as educational tools.

<comment><USER4> The problem is actually a mutually reinforcing feedback cycle between culture and corporation that gives rise to addictive superstimuli. But I suppose "the problem is us" is an equally useful observation.

<comment><USER5> 'the problem is us' is a useful observation in that clearly humans are vulnerable to addictive stimuli that were not abundant as we evolved.

That vulnerability isn't likely to change soon.

We can however recognize that activities designed to exploit it are harmful.

</comment></comment></comment><comment><USER6> Get ready to be even more sad, as everyone on Hacker News completely misses the point of your rant.

</comment><comment><USER7> > _Scanning down underneath the diagram, I saw "Play Games on Google+" along > with a lot of small text. I kept scanning, thinking that the column of text > had something to do with playing games._

Funny, I had the _exact opposite_ reaction. I am so used to skipping over ads with my eyes that I literally didn't even notice there was an ad there; I read the article about dinosaurs, and when I read the above sentence, I had to go back up to the screenshot and verify that there was indeed a Google+ ad there, that I'd totally failed to see.

<comment><USER8> I'm like you in that I skipped over it. However, physorg needs to take some of the heat here because they pick the background, fonts, and colors for the ad insert (not the bolding). So they consciously choose to make their ads look like copy and that is unethical.

</comment></comment><comment><USER9> I applaud this fantastically succinct point: "Some foods taste good but kill you. Some drugs make you feel good but rot your brain. And some ads lead the curious away from knowledge."

I do think, however, that it's incredibly difficult to construct an entirely machine-learning- and auction-based ad network without these sorts of problems popping up. Especially with presumably low dinosaur-related ad inventory to choose from. (Perhaps dinosaurs -> kids -> games isn't a far leap in terms of correlation?)

Based on physorg's deceptive ad placement, text styling, and terrible anti- aging ad selection for the bottom, I would argue that our collective sadness is better directed at them than at Google.

<comment><USER5> Perhaps the 'machine-learning-and-auction-based-ad network' just isn't a good thing. Is this really the end-state for human civilization?

<comment><USER9> I think it's a good thing in that it's led to the most commercially successful search and ads company in history (which - in large part - does deliver very relevant content, and relatively lower sadness than many competitors). But it's clear there is much room to improve.

That point you're addressing was meant to illustrate one way in which it is difficult to find an ad match that doesn't induce sadness. (As opposed to defend the current approach as the "end-state")

<comment><USER5> How about not showing ads in those cases?

<comment><USER9> I'm no expert, but two things come to mind:

1a) That is sub-optimal for ad publishers. I think in building such a system you have to assume they want to make the most money. And while ANY ad may not make much money, it's certainly not capped at the $0 they can expect from NO ad.

1b) Also: imagine if you implement AdCompanyX's html, and you see nothing. Is it broken? Out of ad inventory? Not yet parsed your page? It's not an impossible problem, but it adds UX complexity.

2) As I said in the original post, It's possible it's actually too far of a leap to go from dinosaurs (to kids?) to games. If this is the case, imagine some sort of fitness function scoring how close the best inventory match is to the content of the page. There's a cutoff under which you show no ad. Dinosaur could likely still be close enough to pass this test. Especially given the relatively little text in this example.

<comment><USER10> FYI your cutoff idea is what Google actually does. Ad quality is a problem Google works on. The main complaint people have is that Google doesn't do a good enough job of overruling users "for their own good" when those users are consistently actively telling Google what they want to see.

<comment><USER5> If that were true, why would there be a need for Adblock? Google would just stop showing ads to people who don't like them, or would be showing us all such great stuff that we'd be grateful for them.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> This viewpoint as raganwald expressed it is too oversimplifying for my tastes.

It's pretty .. well, wrong.. to say that all Google engineers are working on is putting ads on web pages. Similarly it's not true that a whole _generation_ of people are working on ads. Google is working on a tremendous number of things -- as is the rest of this industry. It boggles the mind.

I do; however, agree with the generalization of his point (and it's not a unique or new one) that what the technology industry has produced in the first 10 years of the 21st century is mostly a letdown compared to what got done in the last 10 years of the 20th century.

The first dot-com bubble left us with the worldwide infrastructure that powers the Internet today. What will this social-networking bubble leave us with when it pops?

Hopefully strong AI and the best possible tools for dealing with petabytes of data -- but this is likely just wishful thinking on my part.

<comment><USER12> Yeah I think in many cases its all a matter of perspective. You can look at any Google product, analyze it long enough and then understand how it will lead to a stronger advertising business model for them. On the other hand you can look at it and say that is potentially exciting technology for reasons that have nothing to do with advertising.

Probably most of the engineers at Google don't think of themselves as working in advertising and probably shouldn't but Id say almost indisputedly their paycheck is mostly coming from advertising and people clicking on mindless ads.

I think the main problem is how do you solve and build business around interesting problems that don't have the ability to be monetized through click advertising which most of the internet seems to be funded by.

</comment></comment><comment><USER13> You're looking at remnant inventory.

Inventory (advertiser speak for "a page view plus a slot available on that page which we could sell an ad against") are a lot like cuts of meat. Some are worth a lot of money. Some are not, and end up getting sold to be turned into fertilizer or paste to infect cows with all matter of diseases.

Things that tend to make inventory worth more money:

1) Better matching algorithms. This is, fundamentally, why Google is rich beyond the dreams of mortal men.

2) For contextual ads, being matched against content which suggests immediate purchasing intent for something which has a high customer LTV, high transaction value, and/or high margins.

3) Failing that, for brand ads, characteristics of the inventory suggest a "desirable" person at the keyboard. Ideally, you're a rich white American, male or female depending on which brand is trying to reach you, and this is one of the very few ads you will see today.

4) Within this particular session, this is the first ad you've seen.

5) You're American, Canadian, British, northern European, _gap here_ German, Japanese, _big gap_ other first world nation, _titanic gulf_ anyone else.

6) The page the ad appearing on is itself of high social esteem such that the ad will have the halo effect.

Inventory which matches the above descriptions can be sold to advertisers using dedicated sales representatives. Everything else is remnant inventory -- literally, that which remains after our ad sales guys had as many steak-and- booze meetings with the marketing execs at Audi as they can possibly have.

Options for remnant inventory are: a) backfill ad channels like AdWords, which makes some remnant inventory almost as valuable as brandable inventory (e.g. if I write a blog post about credit cards and you happen to read it you're very nearly as useful as someone reading a post on the NYT about credit cards), b) house ads, and c) backfill from ad networks, such as Google house ads. (House ads are, e.g., "We couldn't sell a commercial for our TV channel so instead we'll show a commercial for another show on that channel or another business affiliated with ourselves.")

So, why is your page view remnant inventory?

1) Because generic Internet news sites generate a metric truckload of page views.

2) Because your user behavior was, sorry to say, probably not seeing that ad as the first thing you saw today.

3) Because algorithmically you don't look like a rich white American.

4) Because there are very poor transactional options for monetizing someone if all you know about them is "Possibly interested in dinosaurs," so direct response marketers are unlikely to bid on that site.

5) Because physorg.com does not have a sophisticated ad sales operation which routinely has steak dinners with ad buyers at Proctor & Gamble or any other large brand advertisers.

OK, so you're remnant inventory. Why are you seeing Google+ game ads?

1) Because Google has virtually infinite funny-money to buy ads on Google AdSense at low, low prices because they're crappy backfill remnant inventory that costs them CPMs below $1.

2) Because Google thinks promoting Google+ is their overarching strategic priority right now (they're wrong) and ...

3) ... because Google thinks that engagement on social networks is in large degree driven by games (they're right)

Is remnant inventory necessarily a bad thing?

No, because remnant inventory will inevitably be generated by attempting to monetize (with ads) any website which can't afford a direct steak-and-drinks ad sales force who can charge stupid amounts of money to brand advertisers. If you perceive any value at all out of physorg.com, you should be happy that Google is underwriting it out of their massive pile of money that they get out of controlling navigation and advertising on the Internet.

There exist non-advertising methods by which you could subsidize the creation of content which you like. Many of them involve paying non-trivial amounts of money, or _praying_ that your interests happen to align with the interests of someone who is willing to subsidize your own consumption.

Remnant inventory also enables the creation of actual value, though you're not seeing it when you see diet pill scams. For example, almost all of Bingo Card Creator's substantial advertising budget is spent on remnant inventory at 3 to 8 cents a click, and while the websites that $15k+ a year goes to are not ones you'd really love your mother to be spending time on, selling BCC to the teachers who happen to be on them does at least help children learn to read and underwrite my quirky online pursuits like writing excessively long comments.

Remnant inventory is also virtually synonymous with Google, which a multinational advertising company which occasionally produces industrial biproducts of advertising which generate non-trivial amounts of value for almost every human alive.

<comment><USER14> This is very informative and you make many cogent observations. However, you don't even remotely address the point of the article, which, as far as I understand it, is that something is fundamentally wrong if a child wants to learn about dinosaurs and he/she is lured away with addictive games. Even more so if this is considered the pinnacle of our combined capabilities and at the same time a hint of the future into which we are heading.

<comment><USER13> It's a comment, not a debate. I'm not trying to crush raganwald's constructive and collect many finger snaps from the judges. I'm just trying to explain how Internet advertising works for a community which a) viscerally despises advertising, b) often dreams of working for an advertising company, and c) even if they don't work directly for the ad company will have their careers determined in large part by their ability to form an accurate mental model of how advertising actually works. To the limited extent that I disagree with raganwald about anything, I think that any mental model where "An ad for a BBC documentary about dinosaurs" sounds like a more attractive placement for this pageview than "generic remant inventory" economically is likely flawed. That mental model has not come to grips with the fact that the conversion math would suggest a click on that ad would be worth fractions of a hundredth of a penny to the BBC and cost several orders of magnitude more.

<comment><USER14> Patrick, your argument is flawless, but you've become a bit of a robot. There must be some reason why they don't put Marlboro ads in a lung cancer clinic or a Ballantine's ad in an alcohol rehab center. Surely they would be perfect choices for the well-defined audience that frequents those places and they would make perfect sense economically, more than whatever else they put on those walls. Someone might even come along and say, hey, as long as it pays for the heating, it's good for everyone, right?

Also, I think you misrepresent Raganwald's point. I don't think he claims that _economically_ a BBC ad would be better than an online game ad, he must be thinking along some other axis, along which it would rank higher.

<comment><USER15> OC does not misrepresent the point, he does not engage it, (those are distinct). In not engaging it, he provided tremendous value for everyone ITT.

[ _Thanks, Patrick_ ]

However, if you want someone to bite - I will. Why bother engaging in moralizing if you don't care enough about your values to pay for them? Better yet, how much is your morality worth to you?

 _"I know that this engine is driven by the money, and the money is in luring people into Google's social thingamajiggy instead of trying to sell someone a book or a course or even a BBC/Discover/National Geographic edutainment special on dinosaurs or natural history._

 _But you know, the whole point of having values is that sometimes you don't do the most expedient thing or the most profitable thing or the easy thing. That's what makes them values, you value them more then pecuniarum."_

Well, if Raganwald wishes to educate children, why not create a service that serves educational ads instead of selling add slots to the highest bidder? Because it wouldn't be able to compete with Google outside of some tiny niche. Clearly, society as a whole is not willing to pay for this moral value, namely educating children.

It might be tempting to trivialize this by the way of "heartless businessmen" argument, but _that_ would be misrepresenting the point. The point is - in a society where the only over-arching value is money, morality becomes a liability. One could argue that this is an inevitable state of human affairs due to $characteristic_of_human_nature. However, to me this seems to be more of an inevitable consequence of implementation of a universal means of exchange. Not everything boils down to money - true. But most things do. And most things will always be enough for some people. And the rest will ultimately have to follow. That, or find a clever way to monetize their "moral liabilities".

<comment><USER16> > However, to me this seems to be more of an inevitable consequence of > implementation of a universal means of exchange.

Disagree.

Anytime humans get ahold of a metric, they fuck it up and make it mean something it never did intend to mean. But, hey, animals did it, so we don't need to rise above that.

</comment></comment></comment></comment></comment></comment><comment><USER17> So _that's_ what a page looks like without Adblock. Interesting.

<comment><USER18> 3 months ago I'd say you're mad for sawing the branch you sit on, taking bread away from content providers whose content you consume. I have content sites myself that rely on ads, I believed in this but my internet experience got increasingly frustrating within last year for some reason - Flash popover or whatever you call them which you can't close, disturbing images of old women tearing their skin as in this article, google ads sneaked into content or navigation in hopes of me mistakingly clicking them.

I've tried readability next, clicking "read now" in my chrome extension on every interesting article..but I'm now with Adblock, at least 3 years later than all my techie friends.

I'll silently consume my purified internet now, thanks.

</comment></comment><comment><USER19> For many of us, our entire career eventually boils down to getting people to click on ads. Sit down and think about your product:

\- Are you chasing the hockey stick, making a free product? You're getting paid to create an ad base that will focus on ads.

\- Are you a content provider? A blog? You're getting paid for page hits to generate ad revenue.

\- Perhaps you're a recommendation engine that makes money on Amazon referrals. Ditto.

</comment><comment><USER20> I'm actually looking forward to the day when micro-transactions become easy and friction-less and we move onto paywalls. I'd love to support sites with content by paying $0.02 per article.

I hate seeing ads, and I hate my websites relying on ads for revenue. My parents have many times clicked on ads accidentally thinking it was the content they want. The last thing I need to is to clear out their laptop from endless toolbars and viruses so I've installed ad blocker on all of our computers.

It's gotten out of control. Even legit websites have started taking on questionable ad inventory. The kind that has a download button and tries to trick you. The kind that looks like thumbnails for the software that you're trying to read about. Or those annoying video ads that play automatically.

I'm actually looking forward to the day when I can say "Goodbye and good riddance, advertising killed itself."

<comment><USER21> Seriously: like throwing some coins in the guitar box of the musician you hear in the subway whose melody really sets your mood straight for the day!

</comment></comment><comment><USER22> The webmaster of physorg.com made you sad today.

I don't get mad at Toyota when an inattentive Prius driver nearly sideswipes me in traffic.

</comment><comment><USER23> I think that even worse are the fake download buttons served with free software that lead users to adware:

<http://jakepoz.com/why_do_google_ads_point_to_adware.html>

</comment><comment><USER24> I get the authors point, and yes... it is indeed sad that "play games" are the most profitable types of ads. And stupid weight loss ones, and look younger, and hot women in my area.

It might be because those specific industries see the most value in online ads, since they can tie a click to a sale end-to-end and can justify spending $1 to make $1.01.

Large multi-nationals who dominate TV advertising (Beer companies, car companies, restaurants, groceries, consumer products, things you have to go to a store to buy) have a harder time spending that $1 since it's a lot harder to track.

<comment><USER25> It's likely not a profitable ad at all -- that's a google "house" ad for a google product, which means they have nothing paid to show.

</comment></comment><comment><USER26> Advertising makes me sad. I don't care about how good or bad algorithms are and how well the ads match the context or my personal interests.

If any company tries to sell me something via advertisements, I walk away. It's anti-communication that assumes I'm an idiot.

<comment><USER16> The web is only usable with ad-blocking software.

Many ads are garbage for the soul. You know, the sort that imply that you _should_ have this lifestyle, or _need_ to look a certain way. I respect users too much to subject them to it. I'll probably never become rich, but, at least I have my integrity.

</comment></comment><comment><USER27> Do you remember the days when they had "punch the monkey" ads? This is progress in plain advertising. Also, it's the responsibility of the website owner to be smart. He/she could have spent the effort in finding dinosaur book publishers, but apparently they took the lazy route. As well, the web designer didn't think the ad aspect through -- it's an "oops" on their part. It's not greed, it's poor execution.

</comment><comment><USER28> I'm surprised no one has mentioned that AdWords can target users and offer them ads based on what they think they might want. They're not always contextual.

Do you like to play games Reg?

<comment><USER29> This is a very interesting point, thank you. So here's a thought experiment: I look at this article about dinosaurs, and Google shows me an ad about mountain bike holidays.

Great timing, Spring is looming and I am making plans for Summer (Google knows this from watching my search history or annual credit card spend on travel.) It's late afternoon, when I historically am distractible (witness reading about dinosaurs during my work day. And I ride quite a bit (easy to figure out when you're Google).

I might be quite likely to click an ad for Sacred Rides (Free plug for Mike's business!). But I would still feel sad that at the moment when I was reading something educational, I was lured away to think about something else.

I don't know if Google can come up with a way to make quadrillions of dollars and to keep me from being sad. Maybe my feelings are simply a cost of doing business, or maybe what I want is impossible. But I think my sadness is independent of the attraction the ad holds for me personally. My issue is with the relationship between the ad and the content.

<comment><USER6> _I don't know if Google can come up with a way to make quadrillions of dollars and to keep me from being sad. Maybe my feelings are simply a cost of doing business, or maybe what I want is impossible. But I think my sadness is independent of the attraction the ad holds for me personally. My issue is with the relationship between the ad and the content._

I don't think there can be another way. Advertising necessarily extracts value out of your attention, while simultaneously devaluing your browsing/watching/listening/driving/etc experience. Attention is a (mostly) zero-zum game.

If Google could find a way to put subliminal advertising on the web (and those glasses would sure be a pretty badass way to go about it), maybe it wouldn't make you as sad?

<comment><USER30> > _Advertising necessarily extracts value out of your attention, while > simultaneously devaluing your browsing/watching/listening/driving/etc > experience._

While it it usually, maybe even almost always, does, I don't think advertising is required to be a distraction.

If I'm searching for information on a product, say, a pair of headphones for my wife or a diet for myself, and I see ads for headphones or diet books, then that advertising can be very beneficial.

The problem comes when, like in the article at hand, there's a product or service, totally unrelated to what I'm searching for, that's being injected in my attention stream.

Arguably, Google's incredible financial success is founded on making that information _infinitesimally_ more likely to be useful to the reader compared to, e.g., prime-time television.

</comment></comment></comment></comment><comment><USER31> Maybe google knows that you are personally interested in social games? Or perhaps they're just trying to reach people who are on the internet _right now_.

Anyway, point is, the page isn't the only signal used for ad targeting; _you_ (and your search history) are another, as is the current time. As well as a variety of other things, I'm sure (I have no direct knowledge of google's ad targeting).

I visited the page myself, and got an ad for the galaxy nexus.

I do agree that the placement of the ad, which makes it look like part of the content, is pretty crummy. As are the ads about e-cigarettes and penny stocks and car insurance.

<comment><USER32> I get a lot of nexus ads, but Google knows I bought one months back so it is wasted inventory...

</comment></comment><comment><USER25> Why blame google? The problems here are: 1) the publisher has chosen to embed ad tags within editorial content; 2) the editorial content is so worthless that google can only show a house ad.

<comment><USER33> > _the editorial content is so worthless_

I think it's important to qualify this statement. Worthless to who? Google's ad bots? Perhaps. There's no obvious ad category that relates to dinosaurs.

But just from the snippet the OP included in his screenshot, it is clear the content is not mindless drivel and is actually worth reading (to some people).

<comment><USER34> The publishers chose that he wants to show ads in combination with that content. Apparently it is worthless to ads. What solution would you propose?

I can't think of many commercial products that would benefit from dinosaur associations, except movies, toys and games...

</comment></comment></comment><comment><USER35> "We take a generation of incredibly smart people who have been rigorously trained to deliver amazing code, running on a massive computing engine, and when confronted with a human being trying to learn something, they try to distract him with games."

The same can be said for Wall Street in the past 2 decades. Generations of top graduates have gone not to medicine, not to law, not to real business, but to Wall St. where they create leveraged financial products and led the economy to the 2008-9 Financial Collapse.

</comment><comment><USER36> When I see misplaced advertising like this, I just remember that it's not Google's fault; they didn't consciously make the decision that dinosaurs are childish/recreational fantasies that should be associated with gaming. It's the fault of dinosaur's representation today across the entire internet. Google's bots picked up that correlation mechanically, so while it's unfortunate that they propagate stereotypes like this further, it's not the root cause.

<comment><USER5> If Google perpetuates a lowest common denominator stereotype, it's their fault whether they use automation to achieve it or not.

</comment></comment><comment><USER37> I'm very interested in the point made regarding interactive children's education. High schools across the country seem to think that giving students iPads to use as text books is a good idea. I think that if I was in high school and I had an iPad, I'd be surfing the web the entire time I was in class. I'm certain that's what the next generation will be doing if high schools do indeed end up going that route.

<comment><USER38> And then some entrepreneur will realize this problem, and sell a solution that prevents students from surfing the web while in class.

I never got all the fuss about digitizing education, because it seems like the biggest downside is 'distractions.' Even if there is no way to prevent distractions, are digital textbooks not worth the improvement in costs, accessibility, and breadth (and type) of information?

<comment><USER37> It's hard to estimate the power of distraction for students. Having interactive textbooks would be really awesome, but what does it matter if kids are just playing Angry Birds instead? Also the cost benefits aren't really all that great. Schools typically reuse textbooks and iPads are very expensive and are more prone to damage.

</comment></comment></comment><comment><USER39> From my own experience, it seems that Google tests different contextual content from time to time. They have an algorithm that is supposed to automatically adjust and optimize the ads that get the most click throughs. Perhaps gaming related content works well on that site for some weird reason.

The "sad" part, if anything, is that the publisher decided to place the ad there. In my opinion, that is too misleading.

</comment><comment><USER40> Someone finally noticed that the Internet replicates the internal chatter of the mind. The web is designed to prey on human weakness, to encourage wilfing through the Internet, and to discourage learning and meditation. If our machines are to help us realize our potential, we must teach our machines to meditate. We must quiet their computations.

<comment><USER5> The web ism't designed this way - advertising companies make it this way.

</comment></comment><comment><USER41> Directing this as a criticism of Google (or other advertisers) is a little bit misplaced; the site designer has a lot of control over where ads appear and how they look. Nobody but physorg.com decided that a good place for an ad was right in the middle of their content like that, and they chose the colors, size and style to match as well.

</comment><comment><USER34> I'm saddened that you think games are a bad thing. While there are certainly crappy games, I think I owe part of the survival of my childhood to computer games, and I rank them as the highest possible form of art and expression. At least they have the potential for it.

Also, making the ad look like the text is the doing of the site owner. There are people who will recommend you do that to increase click rates, but I don't think Google actively promotes that strategy.

The placement of the ad is also by the site owner. For my own site I chose to keep ads to the side bar. Admittedly Google keeps telling me that I could put more ads on my site, but I choose not to.

There is also perhaps an economics problem: perhaps most schools, being funded by public money, have no money for ads. What entities in the education market would spend big bucks on dinosaur ads? How can you make big money with dinosaurs?

</comment><comment><USER42> "Someone other than Google is involved in deciding that stock market scams and anti-aging scams and auction site scams are the best way to extract value from my curiosity about dinosaurs."

Those clicks are essentially auctioned off so that someone is quite simply the market. The market, when it comes to online advertising, is largely benevolent. Tons of businesses are built on the fact that Google allows those to buy traffic who can benefit from it the most. The overall wealth creation of such a system is massive, far beyond the wealth created for Google.

Lots of people have jobs because a system exists that allows crappy game ads to interrupt your dinosaur article. it sucks (and the physorg site is kinda shitty for formatting it to look like a headline) but overall the system is a great thing.

</comment><comment><USER43> For me, this site shows an ad about wine - which makes sense since I work for a company that deals in epicurean goods, with our primary product on our site being wine. Based on my browsing habits, it is an appropriate ad to show to me, from a user analysis point of view. Does it suck that it's not related to the article - perhaps - but that's not what will make the most money. Just because you are reading a single article pertaining to dinosaurs, doesn't mean you have a history of being interested in dinosaurs.

The play games ad could be remnant inventory as others pointed out, it could be based on your usage analysis (do you play a lot of games? or share a computer with someone who does?), or lack there of if you have circumvented ways that Google can snoop on/analyze you.

</comment><comment><USER44> The text ad is the only one served on that page by Google, if you go and look at the actual page the rest are served by AdBlade or Tribal Fusion. I feel like it's pretty misleading to complain about those ads as if they all came from the same company.

<comment><USER29> The article is quite clear on the fact that the other ads need not be coming from Google.

<comment><USER44> It wasn't the impression I got from reading it, the text leading into the slideshow is:

>Between them, physorg.com and Google value money, and that's it. > > There's more on the same page, of course:

Which I felt kind of set the tone that they were Google related.

FWIW, I think that the aging secret, penny stock and money ads are scammy as hell and I think the internet would be better off without the bottom feeding ads that take advantage of people's ignorance or insecurities.

The Google+ ad, while poorly targeted to the content of the page, might actually be targeted towards you (I'm not sure that's possible though). In my case I get text ads for elbow tendonitis, probably because I've done searches for "wrist pain" and I've ended up on medical related websites recently. That and the hints they pass to adsense are:

google_hints = "animals elbow muscles";

The combination of those came up with that ad.

</comment></comment></comment><comment><USER45> Google Adsense provides publishers the option to show ads relevant on a user's browsing history (via tracking cookies) as well as contextual ads. Since Google's goal is to make both the publisher and the advertiser more money, Google will choose to show whatever it has found to be more profitable.

On top of that, if Google does not have ad fill, such as in the case of articles about dinosaurs, it will sometimes advertise its own products (which the publisher will still get paid for) instead of showing a blank spot (as some advertising companies will do.

tl;dr Google shows ads based on a visitors browsing history if it thinks it will make you more money that way.

</comment><comment><USER46> Just to note, I subconsciously skipped over the examples in the article as if they were ads -- then wondered what he was talking about for a few seconds afterward. Must be my years of kung foo internet training.

</comment><comment><USER47> Am I really the only one who started reading this thinking the author was going to discuss the paper's claim that dinosaurs, in popular perception, have a lizardlike/reptilian, splayed stance? I've been a dinosaur nerd for years, and the only times I've heard anything about such a possibility is in discussions of _old_ perceptions of dinosaur posture - in every modern reference (intellectual discussion or pop culture appearance) I've seen, dinosaurs are always portrayed with nothing other than a mammalian stance.

</comment><comment><USER48> What's the alternative ?

Having content locked behind a pay-wall so even less people can access it ?

Running ads which are more relevant but don't generate enough revenue to make the site viable in the long term ?

Not having the content at all ?

<comment><USER29> How do you feel about the fact that certain farmers will starve if they can't grow coca or poppies as a cash crop?

<comment><USER49> Hmm.. good point. You know this is a problem too in general. So many useless jobs and useless industries, serving little value to society, yet they are needed because they provide a living for so many people.

This is a result of too many humans on Earth. So many people are just not needed on this Earth. So we have to come up with surrogate occupations

<comment><USER50> I think the problem may be less with too many humans and may be with the preoccupation of "Everybody must have a job! 9-5, Monday to Friday. No exceptions or you're a failure and the country is failing and it's all our leaders fault."

I mean really, we're heading towards more and more automation. There just aren't going to be as many jobs as there once where. Yes, we might get more programming jobs and management jobs. But not enough to replace the hundreds of jobs replaced by a few robots in a factory.

</comment></comment></comment></comment><comment><USER51> This post reminded me of Steve Yegge's cat pictures (<http://www.youtube.com/watch?v=vKmQW_Nkfk8>). In a good way.

</comment><comment><USER52> So who is the author blaming? He's not blaming the end user (the one that makes those ads worth the most to both google and content sites). \- Is he blaming Google for maximizing ad revenue for the content site? \- Is he blaming the content sites for wanting to maximize their own revenue on page views? (or, perhaps, for being too lazy to sell ads themselves?) \- Is he blaming the people who spend the money to get their scammy ads to show up on these sites?

<comment><USER53> Based on my reading of the article and raganwald's comment below, my understanding is he's blaming talented employees or entrepreneurs who choose to use their skills to create low-value (in societal terms, not monetary) products and services. And it seems he's not so much blaming as bemoaning.

The post struck a chord with me because I am evaluating several employment offers. The one that gives me the opportunity to work with the coolest technologies and receive the highest compensation would leave me feeling the ickiest at some level (it involves mobile ad analytics).

<comment><USER10> Without mobile ad analytics, prices are inefficient and therefore ads are biased to the most profitable buyers, not the highest quality.

Do you have any reason to suspect that quality is more profitable?

</comment></comment></comment><comment><USER54> I recently appointed a legal firm. Today I saw their ad embedded as copy looking for cheap clicks via google content network. Screenshot here <http://portablepixels.com/images/google.png>

Seems to me like Google are stooping to new lows. The ad makes me feel like the legal firm I have appointed are selling diet pills and Im minded to terminate my relationship with them. Cheap clicks can be costly.

</comment><comment><USER55> I've actually had ad companies complain about banners too close too buttons in apps before. They can tell, statistically, when users are clicking something by accident rather than interest. Raganwald's screenshot seems similar to me. The web site designer chose to blend the ad into the text content and Google may not even be happy about it due to the lower quality incoming traffic on the ad.

</comment><comment><USER56> This is covered by the Tao of Leo: <http://justinmchase.com/2012/02/20/tao-of- leo-3/>

"For each 2x improvement in performance we get from Moore's law, half will be spent on showing better advertisements, and the other half will be spent on emulating legacy platforms."

</comment><comment><USER57> Using Readable (<http://readable.tastefulwords.com/>) it looks great: <http://www.blueatlas.com/images/readable.png>

Good job Readable

Full Disclosure: I have no affiliation with Readable.

</comment><comment><USER58> The worse thing is that Google lets ads like the "look younger" one buy time. They offer a sample to try for free, but if you don't return it they sign you up for a year's subscription for a $100 a month and you also get charged for the "free" sample. Simple ripoff.

</comment><comment><USER59> The look and feel of the ad is publisher's responsibility. Can't blame Google for that. Rest, Google gets the blame. Before I get down to that, however, I would be suspicious of the veracity of a publisher who tries to blend the ad and the content so closely.

</comment><comment><USER60> I get ads on nanotechnology and Siemens. Maybe the Google Ad machine works different for each person. Oddly too, a search for Rick Santorum leads to Republican primary results fist and then ricksantorum.com second, but, then again, this is my work computer ;)

</comment><comment><USER61> Another day of "5-minute Google Hate Sessions"? Google leaves webmasters the freedom to customize and place ads however they want. Physorg.com opted for the scumbag strategy of mixing ads and content, so it's 100% their fault.

Plus, the entire waxing lyrical about kids being fed ads for games is simply ignorant: he sees those ads because his advertising profile matches those interests, regardless of the page is looking at in this particular moment. A kid interested in dinosaurs might see ads for plastic dinosaurs or paid trips to natural history museums.

Honestly, I wouldn't be surprised to learn Facebook has resumed his paid mud- slinging campaign, these days people are clutching at the smallest straws just to badmouth Google.

</comment><comment><USER62> What ads? I thought all the smart people were running AdBlock...

<comment><USER63> Actually you can learn a lot by seeing ads with a critical eye.

</comment></comment><comment><USER64> Install AdBlock Plus. Solved. Not that big of a deal, huh?

<comment><USER29> I see that I somehow have made the _entirely wrong_ point. I didn't say that it's sad that I have to wade through ads. As it happens, I have this "reader" thingie in Safari, which solves the ad problem and the readability problem in one click or keypress.

The thing I was trying to say that makes me sad is that there is a generation of smart people toiling away with the power of what may be the world's biggest computing infrastructure, all to put _those_ ads on the page.

I am lamenting the waste of people and resources, not the waste of my time.

<comment><USER64> So.. you're sad because of the Ad industry, when there's A LOT more money in the military industry being thrown away? I mean, if you want to be sad about something, choose your argument subject more carefully..

<comment><USER65> These aren't mutually exclusive things, and his argument could be extended to the military industry as well. It's just a reflection on his part based on an experience he had.

</comment></comment></comment></comment><comment><USER66> AHAHAHAHA HE SAID VALUES _falls out of chair_

Americans don't have those anymore, silly. We have money, and to hell with everything else.

Edit, for those who don't get it: Consider what you do at work and whose benefit you do it for. If you're in the states, your job is to make money for shareholders and/or investors, period. You are not building a product, you are driving inequality. Always keep this in mind in any industry to understand "business decisions" and politics, because it's all the same game; "I want to be better than _those people_ "

This attitude is literally destroying our planet.

</comment><comment><USER67> I'm all for that mechanism that would distribute a fixed amound say 5 USD amongst the websites I browsed in that month so they could run without ads.

<comment><USER68> <http://flattr.com/> ?

<comment><USER67> Yes, this. I hoped this would catch on.

</comment></comment></comment><comment><USER69> Heh. That's funny. when i go to those sites, i don't even see the ads. That's why I block facebook connect, google plus, and a raft of other sites.

</comment><comment><USER70> Why doesn't Google offer some across-the-board AdSense subscription-- $10 a month to never see a single "Ads by Google"?

<comment><USER71> "Nice browser yous gots there... would be a shame if something were to happen to it, eh? Like if some ads was to get put on the pages... 10 bucks a month, me and my boys 'take care of' that for yous". The Google Protection Racket

<comment><USER70> What makes it qualitatively different from typical subscription-or-ad- supported services?

<comment><USER71> I would be a little put off if Google were to sell ads, then turn around and sell the opportunity to never see those ads. Seems wrong, like they're saying to the advertisers "Oh sure, we'll get your great ads out right away, all of our customers will see them!" and then saying to the rest of us, "If you're sick of those annoying ads (which we put up), just pay us and we'll make them go away!"

</comment></comment></comment></comment><comment><USER72> Furthermore, Google often encourages publishers to theme their ads so that they blend into the content.

</comment><comment><USER73> This is a common theme I see, often with respect to Zynga.

People want things. Some people want things that other people think they shouldn't want (games, drugs, sex, whatever).

Some people get mad (or in this case, sad) when someone provides a thing they think people shouldn't want to people who want it.

<comment><USER29> Or to put it in a word, they have "morals." And when they talk about it, they are "moralizing."

<comment><USER73> Do people who disagree with you lack morals? Or are their morals wrong?

<comment><USER16> Moral relativism is an intellectual shield for laziness.

Take a stand for something, especially an idea bigger than yourself. You may be pleasantly surprised at who you become.

</comment></comment></comment></comment><comment><USER74> You echoed me & stole my voice by wavelengths

</comment><comment><USER75> Nothing wrong with games.</comment>