Tone: An experimental Chrome extension for instant sharing over audio - <AUTHOR> http://googleresearch.blogspot.com/2015/05/tone-experimental-chrome-extension-for.html

<comment><USER2> I've been poking this the last few hours. It uses both audible and inaudible (ultrasonic) frequencies. The URL is passed to a Google service called Copresence to transform it into a short token - it's this token that's played. The Copresence client, and the playing/listening code are part of Chromium/Chrome 43.x. I think earlier versions won't work with the extension.

Google Tone is similar to [http://chirp.io/](http://chirp.io/), which has iOS and Android apps. There are third-party transmitters (but not listeners) which work across browsers. [http://piupiu.ga/](http://piupiu.ga/) is the most complete. I wrote a bookmarklet [https://moreati.github.io/chirpweb/](https://moreati.github.io/chirpweb/).

<comment><USER3> It would be nice if this had an encryption feature, so that it could be used in public and used to send more sensitive information.

Do listening browsers require the extension or just that the Dev version of Chrome?

Is there any information published about the token?

<comment><USER2> This is based on Copresence, which will probably ultimately include something like AirDrop. The audio portion would be used for neighbour discovery, before hand off to e.g. wifi direct. Copresence is already used for the guest play feature of ChromeCast.

Listening browsers require the extension. Neither require a Dev release of Chrome - 43.x is the current stable release.

Nothing you'd call documentation. Copresence is part of Chromium, so you can dive into the source code if you wish. I'm attempting to reverse engineer the audible portion (just for fun). I'm tweeting as I go [https://twitter.com/moreati](https://twitter.com/moreati)

</comment></comment></comment><comment><USER4> Note that this technique has a pretty neat malicious use case: bridging air gaps, in a way that removing/disabling wireless chips won't stop. You'd have to have already installed malware to the airgapped machine, of course, but once that's done, chances are it'll frequently be in range of (presumably easier to compromise) Internet-connected machines, and ultrasound allows slow but bidirectional communication.

This was alleged in 2013 to have been done in a piece of malware dubbed "badBIOS", but said malware is likely imaginary.

[http://arstechnica.com/security/2013/10/meet-badbios-the- mys...](http://arstechnica.com/security/2013/10/meet-badbios-the-mysterious- mac-and-pc-malware-that-jumps-airgaps/)

<comment><USER5> This is the security downside of using laptops for everything, or motherboards with every feature: we can't rip out the hardware easily.

Such as the camera or microphone built into every machine potentially listening/watching at all times.

<comment><USER6> With usb3 and the ara project, there's a possible window for an 'ATX' standard for portable devices to exist. Tiny connectors, form factors over usb or pcie. It's not a big market. The MacBook Air way is aiming at the opposite but I'm not sure it's going to go much further; these things have almost everything soldered, the next step is closer to a large SoC rather than a mainboard.

</comment></comment></comment><comment><USER7> Oh hey, I actually built something extremely similar in a hacknight recently [1]. Except instead of just converting URLs to fixed size tokens (also how chirp.io works), I allowed arbitrary bit streams and forward error correction.

[1] [https://github.com/rraval/pied-piper](https://github.com/rraval/pied- piper)

</comment><comment><USER8> I think it's worth mentioning that this is yet another fairly cool project that relies on an always-on* microphone. I don't remember who said it, but there was a very nice quote on HN recently about the currency of freedom i.e. it's not something you should never spend, but you should be certain you're getting your money's worth. :I

*I realize you can turn it on and off, but it's not designed to be done easily.

</comment><comment><USER9> I guess this is like how the Square device and modems work, except the tones are played out loud?

</comment><comment><USER10> It would be interesting to see something like this with but tweaked to produce sounds like, say, the background bleep bloop perchweeep noises in Star Trek.

</comment><comment><USER11> Cool concept, with perhaps the most utterly pointless "demo video" I've ever seen!

<comment><USER12> It's an even cooler concept if you imagine that it's transferring data using the groovy bass riff in the video instead of the bleeping and blooping.

Add a bongo and some beatnik poetry as extra data channels, and use it for the network connection at coffee shops.

</comment></comment><comment><USER13> would be nice if humans and computers could both produce/consume the audio. That way you would have a secret language that you could perfectly communicate to computers with.</comment>


OpenSSH for Windows - From Microsoft - <AUTHOR> http://www.hanselman.com/blog/ItsHappeningOpenSSHForWindowsfromMicrosoft.aspx <comment><USER2> It's about time! I would love more details on this. Putty has been the go to for decades. Having SSH capabilities standard on all windows machines might be a bit of a concern given MSFT's track record with security on their machines.</comment>


AMA with Mozilla CEO Mitchell Baker - <AUTHOR> https://answers.thenextweb.com/s/mitchell-baker-aGY62z <comment><AUTHOR> Hi HN, I work for TNW, and tomorrow (17th) we're hosting an AMA on our site with Mozilla's CEO Mitchell Baker. Please come ask a question if you like!</comment>


Samsung Goes Beyond the e-Book Concept with Their New SNE-50K - <AUTHOR> http://www.akihabaranews.com/en/news-18562-Samsung+Goes+Beyond+the+e-Book+Concept+with+Their+New+SNE-50K.html

<comment><USER3> The photo reminds me of a (likely HN, but I can't find it right now) article on why you shouldn't have pretty models look at the camera in product shots/websites. The viewer will look at the model instead of the website/product.

<comment><USER4> Maybe I'm just a nerd, but my eyes went straight for the hardware.

</comment></comment><comment><USER5> _Kindle is just a book, pity since Kindle could be much more_

This seems to insinuate the last thing I should want from my eReader is for it to be highly focused on the one task I actually use it for. For my money, the opposite is true.

<comment><USER6> Seconded. I have a Sony PRS-505 and I'd never go for an eReader without over- the-air or at least WLAN again. I couldn't care less about taking notes on the thing. Feel free to bolt that on _after_ you have wireless added, but not before. Wireless support is a system seller. Note-taking is just "Why would I want this" and "Will I scratch my screen with the pen?" fluff.

And even if the note-taking expirience is completely awesome (which I doubt, given the response time of E-Ink screens). Do I really want to plug the device in all the time and use some proprietary windows-only crapware to transfer my notes? Certainly not.

</comment></comment><comment><USER7> Thanks for the heads-up. A 5-inch e-ink touch screen is a novelty - I believe only larger Sony and iRex models have it.

A search on "Samsung reader SNE-50K" yielded a few good short notices - <USER2> it's only S.Korea for now, priced at USD 274, in line with other 5 or 6 inch e-ink readers.

P.S. The note at jkOnTheRun ([http://jkontherun.com/2009/07/27/samsung- sne-50k-e-book-read...](http://jkontherun.com/2009/07/27/samsung-sne-50k-e- book-reader-and-notepad/), with stylus photo) ends with a fully deserved barb: "The Samsung reader features no wireless capability, unlike Amazon's Kindle. Maybe that might be a good thing to protect reader's content."</comment>


Podcast Interview with Peldi of Balsamiq (500K of revenue in only 10 months) - <AUTHOR> http://techzinglive.com/?p=46

<comment><USER2> For those who don't have an hour or so to listen, here are my notes. I think a lot of this stuff is also covered in Peldis blog.

The motivation for starting Balsamiq was to learn all about the business side of things. Being a sole founder was hard, but it forced him to learn everything. He doesn't recommend this for people who need to launch quickly.

Mockups solved his own problem. He suggests to look for a problem that you're happy to work on for the next 10 years, and the products will come from that. This also means that you will identify with the community who also has these problems, ie customers. If you don't have a specific problem, try partnering with someone who does.

He doesn't like the typical freemium model as he thinks the free version can give a bad impression of the premium version. He prefers a useful but inconvenient demo - it gives people a taste.

He didn't seem to give a lot of thought to setting the price of Mockups, but did say that it is important because it indicates quality and a commitment to being in for the long haul. He suggests thinking of how much time the product will save customers, what other products sell for, and the accepted pricing tiers.

His launch was completely informal, and he keeps that attitude to marketing. He says be interesting and you'll be written about, so focus on the product and making your customers happy. Have a good website, and send the message that you understand the problem and are committed to solving it. Transparency is also important - you can't hide on the web these days so why try?

Gets most feedback from Get Satisfaction, and likes it because it puts customers and company on the same level. He gets a lot of feature requests but tries hard to keep with his vision of Mockups. He won't compromise on anything that reduces his competitive advantage. He does whatever he can to reduce the number of support requests, but those he does get he handles in a way that reflects his committment to customers.

Other miscellaneous thoughts: \- the two things that matter most are user experience and customer service \- paid to get rid of ads on Get Satisfaction \- stayed with GS because he liked and trusted the people behind it \- would rather pay for software because it's more likely the company will continue to exist

He also wanted to stress that his recent success was 10 years in the making.

The podcast was pretty good listening, although maybe a bit too long. I'll definitely be checking out the other interviews.

<comment><USER3> Nice summary.

On finding a problem you'd be happy working on for a few years, he said "fall in love with the problem". I like that.

One of the interviewers asked what source control he used, and admitted that he himself just makes directory tree backups periodically. Peldi spent some time explaining why you should use source control. He himself started with Perforce since that's what he knew, then switched to svn because he's a boring old fart. The cool kids use git or hg.

<comment><USER4> I am using hg (distributed is better than centralized) now but it is so funny because svn just barely took hold in 2004-5 and 4 years later you are a boring old fart for using it. Git and hg should be boring old fart tools in abut 2 years then.</comment></comment></comment>


Is Orion's Armpit About to Explode? - <AUTHOR> https://www.nytimes.com/2020/01/09/science/astronomy-supernova-betelgeuse.html <comment><USER2> I'd love to see a suitably scaled 3d view to reinforce these elements of the constellation don't lie in a plane.</comment>


Top Words in Italian - <AUTHOR> http://wastonchen.com/6230.html <comment><USER2> How are they collected? And listed (by frequency)? I can understand categorizing the most written words (still the actual book/articles/sites/whatever samples may skew the result sensibly) but most _spoken_?

>This is a list of the 1,000 most commonly spoken Italian words.

I can vouch that #974 "quarto di gallone" is actually one among the less common sentences ever spoken in Italian, more probably it is #25896746 in the real world, but it's still better than #541 "am" which is not Italian.</comment>


Intellij-Rust Rust Plugin for IntelliJ IDEA - <AUTHOR> https://intellij-rust.github.io <comment><USER2> Great to see that one of the key contributors looks to be an employee at JetBrains ([https://github.com/alexeykudinkin](https://github.com/alexeykudinkin)), increasing the chance that this may already be or may grow to be an official project.

<comment><USER3> The Erlang plugin for IntelliJ is also developed by a JetBrains employee. I'm always very impressed how much passion the guys have for their product.

</comment></comment><comment><USER4> Does the completion engine handle macros? So that:

      let x = vec![1]
      x.//auto complete here

Will work? If it beats Emacs for Rust overall (I assume IntelliJ has full Vim support) I'd definitely buy it.

<comment><USER5> I'm not a rust programmer but I installed the tools and the plugin to Intelij (latest version) and when I typed the x. auto completion was not shown.

<comment><USER6> did you install Racer? [https://github.com/phildawes/racer](https://github.com/phildawes/racer)

<comment><USER7> Racer is not required for this plugin.

</comment></comment></comment></comment><comment><USER8> Apparently Eclipse has a great Rust plugin: [https://www.reddit.com/r/programming/comments/4qmr3t/state_o...](https://www.reddit.com/r/programming/comments/4qmr3t/state_of_rust_survey_2016/d4ugbr4)

</comment><comment><USER9> How do I make autocomplete work? I installed the plugin, let it download and install rust sources from zip, but no autocomplete suggestions work, even after ctrl+space. Should it work out of box?

</comment><comment><USER10> I'm pumped. IntelliJ is a great IDE with a sufficient VIM plugin -- this will be much better than dealing with a mishmash of plugins for VIM that usually don't work on windows.

</comment><comment><USER11> I can't wait to try this out. I'm already writing a ton of Rust in my free time. When the tooling improves, I'm going to start advocating for it everywhere.

</comment><comment><USER12> Just this Sunday, I was looking for an IDE that I could use to start experimenting with Rust. Happened to find [https://plugins.jetbrains.com/plugin/8182?pr=idea](https://plugins.jetbrains.com/plugin/8182?pr=idea) which is what got released more officially today. But I also then found this on the Rust site [https://www.rust-lang.org/ides.html](https://www.rust- lang.org/ides.html) which talks about lack of on-the-fly, quick compilation support within the Rust compiler itself, a feature that's a major requirement for IDEs. I haven't yet started using this plugin (plan to do over the weekend), but I'm curious if this plugin supports on-the-fly compilation and error detection.</comment>


Preventing Site Scraping - <AUTHOR> http://www.techjunkie.com/preventing-site-scraping/ <comment><USER2> How to prevent site scraping:

1) Don't have any data worth scraping.

2) Charge for access.

3) Provide APIs so people don't need to scrape your site.

Trying to essentially DRM your web site so that it's human-readable and not machine-readable is not only inherently impossible to do effectively (like any DRM), but is also solving the wrong problem.

<comment> jack-r-abbit #3 is only an option if you are trying to prevent scraping just to reduce your bandwidth caused by rapid, repeated page requests. If you are trying to prevent someone one from just coming in and scooping up all your data, then providing an API is worse than just allowing the scraper to scrape.

There is no bullet proof way to _stop_ it. So you make it as painful for the scraper as possible. I like the randomized classes/ids and the extraneous random invisible table cells and divs.

<comment><USER3> In other words, "you can't puts ads on an API."

<comment> jack-r-abbit That is true. I'm sure there are sites that don't provide an API for that reason. Ad revenue on the site might be what allows them to provide the data for free. But that wasn't what I was talking about.

</comment></comment></comment></comment><comment><USER4> Not one of the methods listed here would detract a decent scraper. Moreover you would either screw with your users or with SEO if you wanted to make this or that technique more aggressive.

If your database has really great content it's not because some kid has copy of your website online that you'll lose users. Stackoverflow has been scraped to death and nobody goes to the other sites to check out answers.

<comment><USER5> I don't think there is even any need to scrape stackoverflow. They make all their data available for free.

<comment><USER6> Right, but other sites scrape that free info and republish it with the goal of drawing traffic. Apparently, Stack Overflow has an API. I don't know the functionality of their API, but it doesn't matter: via API or web scraping, the content of the SO database is all over the web, yet SO is thriving.

<comment><USER5> You said "other sites scrape that free". I agree that info is all over the web but it isn't scraped, and I don't think you should keep calling it scraping.

</comment></comment></comment></comment><comment><USER7> Try running a search engine. :-) Needless to say we get folks all the time who are trying to create or enhance data bases out of our index all the time. We even have an error page that suggests they contact business development in the unlikely event they don't "get" the fact that our index is part of our economic value.

One of the humorous things we found is that scrapers can eat error pages very very quickly. Some of our first scrapers were scripts that looked for a page, then the next page, then the next page. We set up nginx so that it could return an error really cheaply and quickly, and once an IP crossed the threshold, blam! we start sending them the error page. What happened next was something over 20,000 hits per second from the IP as the page processing loop became effectively a no-op in their code.

We thought about sending them SERPs to things like the FBI or Interpol or something so they would go charge off in that direction, but its not our way. We settled on telling our router to dump them in the bit bucket.

</comment><comment><USER8> Ajaxification can be defeated if you scrape using a headless browser like PhantomJS [+]. Actually all the markup/visual techniques you propose can also be defeated using Phantom. Dump the page as PNG and OCR it.

Honey pots suppose that the scrapper is an idiot... And even in that case, if he's dedicated, he'll come back later and will be more careful.

The only potentially effective solutions are those that preclude usability for everyone: truncating the content for logged out users. And even then, with PhantomJS and some subtlety/patience in order not to trigger flood detection, an attacker could probably get away with it.

[+] <http://phantomjs.org/>

</comment><comment><USER9> > By loading the paginated data through javascript without a page reload, this > significantly complicates the job for a lot of scrapers out there. Google > only recently itself started parsing javascript on page. There is little > disadvantage to reloading the data like this.

Well, unless you're visually impaired and using a screen reader... and it doesn't really complicate things for any halfway dedicated scraper, as your AJAX pagination requests probably follow the same predictable pattern as the non-AJAX ones would've.

<comment><USER10> AJAXification could make it much easier. GET -> json import.

<comment><USER11> This is exactly what I used for my project to scrape Path of Exile item data.

<comment><USER10> Ha! I did the same for KoL once upon a time.

</comment></comment></comment></comment><comment><USER12> I love the first comment on that post: _It will be more of a pain than usual, but I will get the data. I always get the data._

\- As mentioned: AJAXification the data makes it easier to grab.

\- Convert text to images? I'll OCR it. <http://code.google.com/p/tesseract- ocr/wiki/ReadMe>

\- Honeypot a random link? I don't scrape every link on the page, only links that have _my_ data.

\- Randomize the output? And drive your real users crazy?

I have found that the best deterrent to drive-by scraping is to not put CSS id's on everything. Apart from that, you'll need to put the data behind a pay wall.

</comment><comment><USER13> A lot of the people commenting on these techniques being fallible are missing the point: the idea isn't to make scraping impossible (despite the misleading title), it's to make it hard(er).

A determined scraper will defeat these techniques but most scrapers aren't determined, sufficiently skilled or so inclined to spend the time.

I've been curious about a variation of the honeypot scheme using something like Varnish. If you catch a scraper with a honeypot, how easy would it be to give them a version of your site that is cached and doesn't update very often?

<comment><USER7> C'mon cletus give us a bit of credit. Are you telling us that a company with the World's Smartest Engineers(tm) doesn't already do exactly this with their custom front end machines? :-) It's one of the more entertaining new hire classes.

You are correct that perfection is not achievable and you don't even want to get so close that you get very many false positives. But honey pots are bandwidth, which for folks who pay for bandwidth as part of their infrastructure charge, its a burden they are loath to bear. Rather to simply toss the packets into the ether whence they came rather than bother waking up their EC-2 instance.

<comment><USER14> They sure love to use GWT with an indecipherable exchange format though, have tried to scrape a few things in adwords before. I'm sure it is possible but there was enough of a deterrent for me to not bother.

</comment></comment></comment><comment><USER15> 4\. Provide a compressed archive of the data the scrapers want and make it available.

No one should have to scrape in the first place.

It's not 1993 anymore. Sites want Google and others to have their data. Turns out that allowing scraping produced something everyone agrees is valuable: a decent search engine. Sites are being designed to be scraped by a search engine bot. This is silly when you think about it. Just give them the data already.

There is too much unnecessary scraping going on. We could save a whole lot of energy by moving more toward a data dump standard.

Plenty of examples to follow. Wikimedia, StackExchange, Public Resource, Amazon's AWS suggestions for free data sources, etc.

<comment><USER16> One might argue that indexing from a data-dump will lead to search results that are only as up to date as the last dump.

In StackExchange's case, most of these are now a week or more old.

Maybe it's a good idea, but I'm not sure how many would want to dump their data on a daily basis to keep Google updated, when Google can quite easily crawl their sites as and when it needs to.

<comment><USER15> Have you considered rysnc? Dropbox uses it. So lots of people who don't even know what rsync is are now using it. We could all be using it for much more than just Dropbox. And if you have ever used gzip on html you know how well it compresses. The savings are quite substantial. Do you think most browsers are normally requesting compressed html?

</comment></comment></comment><comment><USER17> Here is my approach on how to find scrappers.

They are already supplying fake data to see if they are being scrapped.

Using this fake data they can find all the sites that are using their scrapped data. Congrats we now know who is scrapping you with a simple google search.

Now comes the fun part. Instead of supplying the same fake data to all, we need to supply unique fake data to every ip address that comes to the site. Keep track of what ip, and what data you gave them.

Build your own scrapper's specifically for the sites that are stealing your content and scrape them looking for your unique fake data.

Once you find the unique fake data, tie it back to the ip address we stored earlier and you have your scrapper.

This can be all automated at this point to auto ban the crawler that keeps stealing your data. But that wouldn't be fun and would be very obvious. Instead what we will do is randomize the data in some way so its completely useless etc.

Sit back and enjoy

</comment><comment><USER18> In general I think that getting into an arms race with scrapers is not something that you will win, but if you have a dedicated account for each user you can at least take some action.

If this data is actually valuable, they should put it behind some sort of registration. Then they can swap out the planted data for each user to something that links back to the unique account, without wrecking things for users with accessibility needs or unusual setups.

</comment><comment><USER19> I have yet to see any anti-scraping method that can protect against a full instance of Chrome automated with Sikuli. It's obviously very expensive to run, since you either need dedicated boxes or VMs, but it always works. In my experience the most consistent parts of any web application are the text and widgets that ultimately render on the screen, so you easily make up for the runtime costs with reduced maintenance. You could in theory make a site that randomly changes button labels or positions, but to the extent you annoy scrapers you're also going to annoy your actual users.

</comment><comment><USER20> As pointed out by others, many of the suggestions here break core fundamentals of the web, and are generally horrible ideas. It's unsurprising to see suggestions in the comments such as, "add a CAPTCHA", which is nearly as bad of an idea. If you're willing to write bad code and damage user experience to prevent people from retrieving publicly accessible data, perhaps you should rethink your operation a bit.

</comment><comment><USER21> Generally speaking, if you're in the business of collecting data, but you have a competitive incentive not to share and disseminate that data as broadly as possible, you're in the wrong business. This article seems to address a problem of business model more than anything else. And if you're using technology to solve a problem in your business model...

</comment><comment><USER22> Let me start by saying that I am a sadochistic scraper (yeah I just made up that word) but I will get your database if I want it. This goes the same for other scrapers who I am sure are more persistent than even I am.

You don't have to read any futher, but you should realise that...

* _People will get your data if they want it_ *

The only way you can try and prevent it, is to have a [1] whitelist of scrapers and blacklist useragents who are hitting you faster than you deem possible. You should also paywall if the information is that valuable to you. Or work on your business model so that you can work on providing it free.... so that reuse doesn't effect you.

\</comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>---

I thought I would provide an account of the three reasons why I scrape data.

There are lots of different types of data that I scrape for and it falls into a few different categories. I'll keep it all vague so I can explain in as much detail as possible.

[1] User information (to generate leads for my own services)...

This can be useful for a few reasons. But often it's to find people who might find my service useful.... So many sites reveal their users information. Don't do this unless you have good reason to do so. If I'm just looking for contact information of users, I'll run something like httrack and then parse the mirrored site for patterns. (I'm that paranoid that check out how I write my email address in my user profile on this site).

[2] Economically valuable data that I can resuppose....

A lot of the data that I scrape I won't use directly on sites. I'm not going to cross legal boundaries.. and I certainly don't want to be slapped with a copyright notice (I might scrape content, but I'm not going to so willfully break the law). But, for example, there is a certain _very popular_ website that collects business information and displays it on their network of websites. They also display this information in Google Maps as Markers. One of my most successful scrapes of all time, was to pretend to be a user and constantly request different locations to their "private API". It took over a month to stay under the radar, but I got the data. I got banned regularly, but would just spawn up a new server with a new IP. I'm not going to use this data anywhere on my sites. It's their database that they have built up. But, I can use this data to make my service better to my users.

[3] Content...

Back in day... I used to just scrap content. I don't do this any more since I'm actually working on what will hopefully be a very succesul startup... however, I used to scrape articles/content written by people. I created my own content management system that would publish entire websites for specific terms. This used to work fantastically when the search engines weren't that smart. I would guess it would fail awfully now. But I would quite easily be able to generate a few hundred uniques per website. (This would be _considerable_ when multiplied out to lots of websites!!!).

Anyway, content would be useful to me, because I would spin in into new content, using a very basic markov chain. I'd have thousands of websites up and running all on different .info domains, (bought for 88cents each) and running advertisements on them. The domains would eventually get banned from Google and you'd throw the domain away. You'd make more than 88 cents through affiliate systems and commission junction and the likes that this didn't matter, and you were doing it on such a large scale that it would be quite prosperous.

\</comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>

I honestly couldn't really offer you any advice on how to prevent scraping. The best you can do is slow us down.

And the best way to do that is the figure out who is hitting your pages in such a methodical manner and rate limiting them. If you are smart enough, you might also try to "hellban" us, by serving up totally false data. I really would have laughed, if the time I scraped 5million longitude and latitudes over a period of a few months, if at the end of the process, I noticed that all of the lats were wrong.

Resistance is futile. You will be assimilated. </geek>

<comment><USER23> _I honestly couldn't really offer you any advice on how to prevent scraping. The best you can do is slow us down.

And the best way to do that is the figure out who is hitting your pages in such a methodical manner and rate limiting them. If you are smart enough, you might also try to "hellban" us, by serving up totally false data._

Well, no, there are other ways too.

For example, any site behind a paywall probably has your identity, and unless you live in a faraway place with impotent copyright laws -- and there aren't that many of them any more -- there are often staggeringly disproportionate damages for infringement available through the courts these days, certainly enough to justify retaining legal representation to bring a suit in any major jurisdiction. Given a server log showing a pattern of systematic downloading that could only be done by an automated scraper in violation of a site's ToS, and given a credit card in your name linked to the account and an IP address linked to your residence where the downloads went, I imagine it's going to be a fairly short and extremely expensive lawsuit if you upset the wrong site owner.

<comment><USER24> Not all valuable scrapeable data is copyrightable. I also know of a number of sites I've scraped that don't even bother attempting to restrict your access to their data through T&Cs even though its the basis for their site (not that they'd have much of a legal basis for enforcing that, anyway). Ultimately if you're in the business of selling raw data with no value added, the problem is your business model, not scrapers.

<comment><USER23> _Not all valuable scrapeable data is copyrightable._

Sure, but a lot of it is, and even the bits that aren't may be protected by other laws such as database rights depending on your jurisdiction. I think anyone who maintains that you can't stop scrapers as a general principle is possibly a little unwise.

</comment></comment></comment></comment><comment><USER25> There's nothing worse than spending lots of hard work scraping sites to build your search engine and then having bad guys perpetrate the scraping of your search engine.

Maybe it's some sort of karma. If you scrape, then you will get scraped.

</comment><comment><USER26> I don't know what kind of site this is, so it's hard to say if it applies, but do note that several of these can significantly harm usability for legitimate users as well. For example, someone might be copy/pasting a segment of data into Excel to do some analysis for a paper, fully intending to credit you as the source; if you insert fake cells, or render the data to an image, you make their life a lot more difficult.

The first suggestion (AJAX-ifying pagination) can be done without a major usability hit if you give the user permalinks with hash fragments, though, so example.com/foo/2 becomes example.com/foo#2.

</comment><comment><USER27> I am currently working on a project that involves some scraping as well. The most annoying things I came across so far are:

\- Totally broken markup (I fixed this by either using Tidy first or just using a Regex instead of a 'smart' HTML/XML parser)

\- Sites that need Javascript even on 'deep links' (I fixed this by using PhantomJS and saving the HTML instead of just using curl)

\- Inconsistency, by far the most annoying: different classes, different formatting, different elements for things that should more or less be identical (basically fixing this whenever I come across a problem but sometimes it's just too much hassle and well, ask yourself if you really need to get every single 'item' from your target)

One more thing: RSS is your friend. And often you can find a suitable RSS link (that's not linked anywhere on the site) by just trying some URLs.

PS: No, I am not doing anything evil. If this project ever goes live/public, I'll hit all the targeted sites up and ask for permission. Not causing any significant traffic either.

</comment><comment><USER28> The best techniques to stop scraping, are ones not discussed in public.

</comment><comment><USER29> Anything that can be displayed on a screen can be scraped.

An approach I used to prevent scraping in the past is to start rate limiting anything that hits over N pageviews in an hour, where N is a value around what a high-use user could manually consume. Start with a small delay and increment it with each pageview (excess hits*100ms), then send HTTP 509 (Exceeded Bandwidth) for anything that is clearly hammering the server (or start returning junk data if you're feeling vengeful).

Added bonus is that the crawler will appear to function correctly during testing until they try to do a full production run and run into the (previously undetectable) rate limiting.

This project did not require search indexing so we didn't care about legit searchbots, but you could exclude known Google/Bing crawlers and log IPs of anything that hits the limit for manual whitelisting (or blacklisting of repeat offenders).

<comment><USER30> Not a 509, but a Deny rule in the firewall config. Works miracles.

</comment></comment><comment><USER31> More trouble than it's worth. Plus, none of these solutions actually prevent site scraping... if the person is dedicated enough, they'll find a way. The time spent on implementing any of these approaches would be much better spent on site optimization, features, etc.

</comment><comment><USER32> Sorry to say, that these techniques will work for _Enterprisey folks_ who anyway won't have to scrape.

For example you say 'Randomize template output'. Scrapers used a mixture of various techniques. Say, HTML Path does not work (despite supporting HTML wild cards in the form of body/table[1]/tr[*]/. Then you fall back to just some patterns could be the title of your data or anything.

Have scraped content coming in Flash also. Basically how can you stop anybody from understanding either a) data exchanged between the browser and the server Or (in case its encrypted/encoded) b) The HTML once its rendered.

Only way of doing it is by having your own custom browser, and preventing its source code getting leaked.

PS: I scrape, and our clients (whom we scrape) know that.

</comment><comment><USER33> I've written a lot of scrapers myself and let me start off by saying there is no such thing as a site that can't be scraped. You can add in honey-pots all day long and at the end of the day once I've discovered my bot has been detected, I'll find a way around it. If the content is worth scraping and the site owner doesn't have an API (free or pay to use), then people will find a way to get the data regardless of what you do.

A well-intentioned article that puts forth a few great ideas for amateurs, but at the end of the day it's wasted time and effort that could have been better spent, oh I don't know, developing an API for your users instead.

If I want data I'll get the data, I always win.

</comment><comment><USER34> A less aggressive approach I've encountered is to insert links to other pages of your website with full URLs (<http://www.example.com/page.html> over just /page.html). Usually, a scraper will copy the links too. This should then make it obvious that the content's been scraped.

This could become a nightmare to maintain if you don't automate it. It'd be trivial to automate on a CMS. I know WordPress has loads of plugins for exactly this. I don't think I've come across something that can do this for static websites though, which make up for the brunt of the websites I maintain.

<comment><USER12> wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

</comment></comment><comment><USER35> IHMO, preventing site scraping is really really hard. There are couple of startups that offer products that claim to stop site scraping.[1]

1\. Ajaxified paginated data ->

With Firebug looking at XHR requests, one can easily reverse engineer the ajax call and extract data directly from the json - in fact you are making the scraper's life easier as you will probably be storing the information in a structured manner in the json string. Or one could use a more sophisticated tool like MITMProxy to study how requests are made.

If you somehow managed to implement a highly obfuscated method of ajax requests (re: microsoft aspx.net), there is always selenium to get through them.

2\. Randomize template output ->

You are going to annoy users if you are going to display a different template altogether. If its just randomizing div and class ids, one can write clever xpath expressions or css selectors to circumvent this. Or worse case there are always the ever reliable regex expressions.

3\. HoneyPot ->

Scrapers only crawl pages that they are specifically looking for. A good scraper only runs through pages he wants to scrape. Nothing more. This is probably the least effective strategy.

4\. Write data to images on the fly ->

Use an OCR api to decode them!

5\. Alternatives ->

Putting in a login screen is also not effective as not only will that annoy users, it can easily be circumvented by using selenium or passing the cookie/session information to the scraping script.

Blacklisting ips is not going to a very effective strategy. With tons of free proxies, Tor and cloud based services (especially PiCloud[2] - which offers a scraping optimized instance!!), ip blocking can easily be circumvented.

Best strategy would be display corrupted content or start throwing CAPTCHAs if you sense a large number of requests coming from a particular IP.

But once again you may want to do some sort of machine learning on server logs based on the various ips and the specific urls visited and build a model that could predict if a particular user is a bot or a human before you start throwing fake data or captchas. Just to be on the safe side so that you don't annoy anyone.

[1] <http://spider.io> and <http://blockscraping> [2] <http://www.picloud.com/pricing/> s1 instance plan

</comment><comment><USER36> Some ideas:

\- Use Unicode RTL override, then put in numbers/text backwards (this is also fun for messing with web forums, but that's another story...)

\- Inject zero-width characters (such as ZWSP U+200B, WJ U+2060, ZWNJ U+200C, ZWJ U+200D)

\- Use intentionally broken HTML in places, to catch out some parsers (browsers won't care, some scraping parsers will)

\- Occasionally don't maintain element order in HTML, use CSS tricks to make it appear in the correct places

\- Add useless dummy data hidden in places using complex CSS rules

\- Require JS-generated auth token using an obscure, obfuscated algorithm to download the data, which expires immediately

</comment><comment><USER37> Google Scholar's solution is to show a captcha when a given IP has made too many requests in a given time period, although a scraper can easily throttle to avoid this.

You could require a captcha every n page views, or you could render the text of the page as a distorted image, which would defeat the OCR approach others have suggested here. These would make scraping difficult, but they would also mean throwing UX out the window, and they could still be defeated with Mechanical Turk.

</comment><comment><USER38> Great! I learned a few scraping tricks from this article. In the past, all I did was using "time.sleep(3)" to pace my scrapping so it stayed off the scrapee's radar screen.

</comment><comment><USER39> A friend of mine has been working on anti-scraping for a while. The question is: is there a market for an anti-scraping service? Would you pay so such a service and how much?

Some examples out there: [http://blog.cloudflare.com/introducing-scrapeshield- discover...](http://blog.cloudflare.com/introducing-scrapeshield-discover- defend-dete#!/)

<http://www.blockscript.com/>

</comment><comment><USER40> I've run into a few sites that "prevent" scraping. I just jump on <http://developer.yahoo.com/yql/> and scrape away. The surface level defenses (ip blocking) are usually sufficient, but if there is a real developer behind the scraper, they will get to your content.

Too much prevention could be counter-productive, as you may inadvertently deny the friendly spiders.

</comment><comment><USER41> In addition to mentioned suggestion I will add some here. Make difference layers difficult to understand and use (which will affect real user too!):

1\. Script delivery. Use dynamically loaded modules.

2\. Content delivery. Use websockets to deliver data. Does any developer tool show content going through websockets?

3\. Content rendering. Use canvas to render content on the screen. Use fonts that make OCR difficult. Handwriting? :)

4\. Use of browser plug-ing like Flash and Java.

5\. Content delivered over video.

6\. Quota based content delivery for requests originating from the same source. Use of multiple signals like cookies and IP addresses to pinpoint the source. Progressive

However, I would not recommend doing most of these. As mikeash aptly compared it to DRM, usability of the website will be affected negatively.

Like software, we can make it hard to reverse engineer them but cannot prevent a determined person from reversing it. There is no way to prevent a determined scraper. They will scrape your data using appropriate tools like real browsers, OCR or real people (mtrurk?).

Edit: formatting.

</comment><comment><USER42> Great ideas. Love the honeypot.

One more thought: once you discover a badguy, if you signal this immediately you are just speeding up his learning cycle. Don't just cut him off, tarpit him (gradually slow down responses) or return bad data.

</comment><comment><USER43> I agree with the argument * People will get your data if they want it* Reading the discussion here and on original post, I can make a scraper that would be difficult to detect and block ;-)

</comment><comment><USER44> You could just print the data in a book instead of putting them on a website.

Oh woops never mind (<http://www.diybookscanner.org/>).

</comment><comment><USER45> Honeypots are pretty worthless. The article counters itself.

</comment><comment><USER46> I can write a scrapper for any text-based website within an hour. As far as I know, site scrapping can't be prevented. You could make it harder, though.

</comment><comment><USER47> the only way to prevent scraping is to shut down access to your website, with the new modern libraries like phantomjs and selenium any one can write a scrapper that executes javascript and reads website pretty much like any human user.</comment>


Ask HN: What would you ask YC founders? - <AUTHOR>

At tomorrow's London Hacker News meetup we are doing a Q&A session with several YC founders from 07, 09 and 11 sessions. We have received many great questions already, but have room for some more. Is there anything you would love to find out about the YC experience, fund raising or building a successful startup? Please submit your questions and we'll do our best to get them all answered by our panel.
We will be recording the session and you will all get the opportunity to see it in its full glory.
Our panel will consist of:
Pete Smith founded Songkick in 2007 and as COO has scaled Songkick's operations -- premises, finance, legal, visas, HR, hiring, revenue -- from three friends with their own laptops to a team of 25 based in Hoxton and Palo Alto.
Phil Cowans was employee number 2 at Songkick and the company's original CTO, having joined in 2007 and spent some time in Boston during the Y Combinator summer programme. Now in the role of Chief Architect, he's been closely involved in with all aspects of the engineering process since then, from building the original prototypes to scaling the site and hiring and managing a large development team.
Josh Buckley is a 19 year old entrepreneur from Kent, in England. After selling his first company at age 15, he started angel investing. In January 2011, aged 18 he founded the youngest ever Ycombinator funded company, MinoMonsters. He has since raised a seed round from the most prestigious Silicon Valley investors.
Colin Beattie who founded Tuxebo at the end of 2009.  As CEO Colin undertakes a 'jack-of-all' role leading product research, design and development, SEM, strategy, financing, HR and strategic partnerships. Tuxebo is a marketplace, comparison site and booking agent for renting things; essentially a cross between ebay and Kayak for renting stuff.  It currently focuses on the construction segment of this enormous market but aims to be a single access and invoicing point for all your rental needs regardless of where you need to rent, what you need to rent and from whoever you want to rent it. <comment><USER2> What was your lowest low and how did you deal with it?

<comment><AUTHOR> good one.

</comment></comment><comment><USER3> Enterprise vs. everyone startups, e.g., fewer but more lucrative customers vs. popularity contest. What's awesome and not about each?</comment>


Python from scratch - <AUTHOR> http://go.hodspot.com/2012/01/python-from-scratch.html <comment><USER2> Congratulations hodspot. Hopefully this will inspire others.

<comment><AUTHOR> The name is Hod and thank you.

<comment><USER2> Sorry for butchering your name :( Btw, what are you planning on doing with Python? Develop web applications or use it for testing?

<comment><AUTHOR> I guess both: To test since i am a tester anyway. To develop because i want to create something. Somewhere in the distant my goal is to open my own Start-Up</comment></comment></comment></comment>


Audio from Scratch with Go: Extracting Breakpoints - <AUTHOR> https://dylanmeeus.github.io/posts/audio-from-scratch-pt7/ <comment><USER2> This is exactly the type of thing I'm on HN for. Someone in a different part of the world walking us through a very specific technology with examples.

<comment><USER3> My thoughts exactly!

Great blog, love to see more

</comment></comment><comment><USER4> It's nice to see this sort of thing here. FWIW in computer music parlance this is usually called an envelope follower.

Here's an example of a realtime implementation: [https://ccrma.stanford.edu/CCRMA/Courses/220b:1998/220b-98-0...](https://ccrma.stanford.edu/CCRMA/Courses/220b:1998/220b-98-07.html)

<comment><USER5> it's also not a very clever one. I think what the author wants to do is actually segmentation which is much more complex program. the best algorithm Ive used for doing any kind of amplitude following is a detrender with two envelopes running at different rates. next to that is Foote's audio novelty audio algorithm.

</comment></comment><comment><USER6> I enjoyed this and am going to take a look at the rest of the series, always wanted to learn a bit more about audio processing.

<comment><AUTHOR> Thanks! Glad you like it :)

</comment></comment><comment><USER7> Great series. I always wanted to learn about audio processing. Any learning material for this? Besides this series obviously

<comment><USER8> It depends on what programming language you'd like to work in and what exactly processing means to you. Are you interested in audio analysis or writing programs that generate audio like a synthesizer for a DAW? For audio analysis I'd recommend essentia a C++ library with Python bindings. For writing plugins, I'd recommend you take a look at JUCE, which has a decent set of tutorials on their website. You'll notice that C++ is pretty standard in industry because speed and hardware access are important. If JUCE is the sort of thing you're looking for, I'd also recommend the Audio Developer Conference talks on Youtube.

<comment><USER9> I've found rust to be very capable as well, if lacking in runtime libraries with native bindings. It's not too difficult to bind into coreaudio and jack, though.

<comment><USER8> I've actually been doing a bit of research into using Rust for an upcoming project. The JUCE library is just so fully featured that it would be an undue burden to use any other language to write audio software in. Still, I think Rust is an attractive language for this space and there's definitely some talk about expanding libraries for a JUCE alternative.</comment></comment></comment></comment>


Who's Behind the Sad, Anonymous Truth of 'My Startup Has 30 Days to Live'? - <AUTHOR> http://www.theatlanticwire.com/technology/2013/06/my-startup-has-30-days-live/66588/

<comment><USER2> Looks like our virtual reality haz virtual constant news update

<comment><USER3> Writing for HN is a long-standing journalistic tradition that dates back to them noticing all those free and easy pageviews.</comment></comment>


Ask HN: how to track replies to your own comments on different blogs - <AUTHOR>

Hello, I guess a lot of readers are in the same situation: you leave comment on a lot of blogs or websites (including HN), and would like to answer any question tot these comments. How do you keep tracks of replies to your comments? Only a few blogs have an option to receive an e-mail for each reply.
I found sometimes week later that somebody ask a question to me regarding a comment, I would like to be able to answer quickly without visiting 100+ sites manually daily.
Any free webapp that would take care of it? <comment><USER2> We (BackType) track hundreds of thousands of blogs, as well as sites like HN, digg, reddit, etc. In addition to claiming your comments, you can use BackType Subscriptions to follow activity on the posts you contribute to. There's also an option to auto-subscribe:

<http://www.backtype.com/>

<comment><AUTHOR> It could be the the solution, if it was working :-( The "Claim your comments " section is very confusing:

* it did not like my Disqus login. When I click on submit, there was no error message, just a thin pink border around disqus. It took me several attempt to understand what it means

* I entered 2 other URLs (from my websites). Go the error "One or more URLs are already claimed". I doubt it. It would be nice to know which one. When I go back to "Claim your comments", the URLs disappeared, as well as all the login I entered previously.

It doesn't look like I can use this website, unfortunately.

</comment></comment><comment><USER3> Aha!

This is something I've wanted for a long time [1]. No, I don't think anyone built it yet, sadly.

Basically I was thinking about a browser extension/plugin that would understand that I am posting something on a webpage, and then it would monitor/crawl this webpage, and let me know when a reaction (say, a reply or upvote) happens.

It's something that would require a lot of work, but doable, specially if you start targetting the largest websites (social networks, digg/reddit) and platforms (wordpress installations).

[1][http://www.reddit.com/r/programming/comments/1d46g/hey_reddi...](http://www.reddit.com/r/programming/comments/1d46g/hey_reddit_got_any_ideas_that_i_could_steal/c1d86f)

</comment><comment><USER4> I guess I've always posted only on HN and not individual blogs, mainly because I hate to spend even a few seconds signing in to these various sites.

The disadvantage is that the bloggers might not actually see what's said here; but I've found that a surprising number of them do. (And "threads" at the top of the page shows any replies to your comments here.)

Some blogs accumulate inflammatory or vacuous comments, which is another reason it can be nice to focus only on the discussions at HN. :)

</comment><comment><USER5> I believe Disqus is what you are looking for.

<comment><USER5> Also forgot Backtype

<comment><USER6> I think intensedebate and mybloglog also do similar things if the blogs are part of their network.</comment></comment></comment>


Night Science - <AUTHOR> https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1800-6 <comment><USER2> Important read, IMHO. Reminds me of several related topics, including topic mapping, and literature-based discovery.</comment>


Build a Blog Engine in Go - <AUTHOR> http://0value.com/build-a-blog-engine-in-Go <comment><USER2> I like the trick in Golang marketing, where we get a code full of references to some libraries. Same could be done about other languages.

Oh, look, web server in 5 lines of code, in Java!

        public static void main(String[] args) throws Exception {
            Server server = new Server(80);
            server.setHandler(new HelloWorld());

            server.start();
            server.join();
        }

The trick is cheap, but it works!

<comment><USER3> Of course, the difference is that the Golang code isn't using "some libraries", but the standard library. Well, except for this one, which is also using some custom handlers for gzip and such.

<comment><USER2> It is also the only one library available for Golang atm.

Actually I think that this example written in Java8 with lambda expressions would be more impressive.

<comment><USER3> _It is also the only one library available for Golang atm._

Except it isn't: [http://go-lang.cat-v.org/pure-go-libs](http://go- lang.cat-v.org/pure-go-libs)

<comment><USER2> Nice one, a fun list of hobby side projects made by Golang enthusiasts.

I'm gonna drop Jetty and Java altogether in favor of AeroNotix/wedge!

<comment><USER3> Well, your sarcasm really showed all those people who said you should!

</comment></comment></comment></comment></comment></comment><comment><USER4> A few years ago, a greybeard of my acquaintance quipped that blogs are this generation's unix shell.

My main nitpick would be that I see a standard blog as requiring a comments facility. Inbuilt, I don't count disqus as having reached the minimum feature level.

I'm also wary of flat file fetishism. Where data is strictly hierarchical and there is only one pattern of access, YAML would suit fine.

In blogs you pretty get into not-hierarchical-really stuff pretty quickly. As soon as the first per-tag or per-author archive page is added, for instance.

<comment><USER5> Author here. As I mention in the opening lines, I agree that this is really not the most original idea, barely above web frameworks :)

This was an attempt to build the minimal features required to have a useful yet ridiculously simple blog engine. Of course it is very subjective, and to me there are enough options to comment outside the blog itself (this thread here being a good example), I didn't feel this was required. I think a more "controversial" choice may be the absence of a distinct template for the index page. I know this is not acceptable for some, but then there are a few other blog generators to choose from!

<comment><USER4> To each their own, friend.

</comment></comment></comment><comment><USER6> I've written a static blog generator, too. But I haven't written a lot of blog posts since then. I realized that I'm missing a browser based backend to edit the posts.

 _sigh_

<comment><USER5> Well, I edit with vim and git push when I'm ready. But if you really want a browser-based editor, [Draft][0] is very nice.

[0]: [https://draftin.com/](https://draftin.com/)

</comment></comment><comment><USER7> I had a similar thing going 8 months ago.

[http://gokcehan.github.io/glog/](http://gokcehan.github.io/glog/)

The upside is that you are able to push your generated site to appengine with a single command afterwards. Too bad even I have switched to jekyll nowadays.

</comment><comment><USER8> I've wanted to get into Go for a while now. This looks like it will be a good reference. Thanks!

</comment><comment><USER9> Neat, but I keep wondering with these static generators... why wouldn't you just use pandoc [http://johnmacfarlane.net/pandoc/](http://johnmacfarlane.net/pandoc/) for the HTML generation?

<comment><USER10> Building a [static|dynamic] blog in $language is the most classic weekend project for us web-nerds. It's the one field where universal reinvention of wheel is culturally accepted. And it's just fun, I guess.

<comment><USER9> That's cool, and there's lots of things people should play with: the Go templates for the design and layout, the file system watcher for auto-updates, the file writing, etc.

But when it comes to formats, pandoc is so damn feature-rich, and would allow authors to write in whatever was best for a particular post. Markdown most of the time, restructered text occasionally, TeX rarely.

If anything it would extend the capabilities whilst introducing to the weekend project the execution of an OS command and using stdout and stdin to communicate.

A good addition to the weekend project, and a comprehensive addition to the features of the static blog generator.</comment></comment></comment>


SSL Server Test - <AUTHOR> https://dev.ssllabs.com/ssltest/index.html <comment><USER2> This is the development version, and has been around a long time.

Among many others, previously mentioned at [https://news.ycombinator.com/item?id=20218517](https://news.ycombinator.com/item?id=20218517)</comment>


Next Generation Pixel Qi screen improves over iPad3 Display - <AUTHOR> http://pixelqi.com/blog1/2012/04/21/next-generation-pixel-qi-screen-improves-over-ipad3-display/ <comment><USER2> Like many, I think, I'd really like to see Pixel Qi succeed, but I've about given up believing I'm going to have a (mainstream) product that includes a PQ screen, or a PQ screen replacement kit that I can readily order and install into such a product.

I don't pretend to understand PQ's market, at this point, but if you want to (re)engage me (and, I think, many others), you need to meet the above.</comment>


Google flights - <AUTHOR> https://www.google.com/flights/ <comment><USER2> This has been around for some time. I believe they're using technology from ITA, which was acquired by Google in 2010 [1]: [http://matrix.itasoftware.com/](http://matrix.itasoftware.com/)

(It's also what most flight hackers use to search for tickets.)

[1] [http://www.google.com/press/ita/](http://www.google.com/press/ita/)

<comment><USER3> It's now coming up because they're integrating it into the new version of Maps. If you search for a driving distance that's sufficiently far away, they will also suggest flights as an alternative.

</comment></comment><comment><USER4> One thing you have to be careful of is with connecting flights of differing airlines. Sometimes it will propose an itinerary that just isn't realistically possible to make because you won't be able to get to the other airline in time. I've only encountered this once or twice. Otherwise, I use it all the time, successfully booking many flights through it.

</comment><comment><USER5> As mentioned above, this launched in mid 2011: [http://insidesearch.blogspot.com/2011/09/early-look-at- our-f...](http://insidesearch.blogspot.com/2011/09/early-look-at-our-flight- search-feature.html)

Also accompanied by Google Hotel Finder: [http://googlesystem.blogspot.com/2011/07/google-hotel- finder...](http://googlesystem.blogspot.com/2011/07/google-hotel-finder.html) [https://www.google.com/hotelfinder](https://www.google.com/hotelfinder)

</comment><comment><USER6> This is disappointing. With all the data they now have access to through the ITA purchase, this is still much worse than what I can achieve with ITA Matrix ([http://matrix.itasoftware.com/](http://matrix.itasoftware.com/)).

<comment><USER7> We support a lot of that on Hipmunk (e.g. [http://www.hipmunk.com/flights#;kind=flight&locations=sfo,ve...](http://www.hipmunk.com/flights#;kind=flight&locations=sfo,vegas&dates=Tomorrow,Next%20Week&pax=1)) including the routing language

<comment><USER8> Can you offer a bunch of destination based on a budget on hipmunk? This is the one particular feature I found different from other sites.

<comment><USER7> [http://www.hipmunk.com/cheap-flights/San- Francisco](http://www.hipmunk.com/cheap-flights/San-Francisco)

</comment></comment></comment></comment><comment><USER9> A handy reminder of why flight search is Hard with a capital H:

[http://www.demarcken.org/carl/papers/ITA-software-travel- com...](http://www.demarcken.org/carl/papers/ITA-software-travel- complexity/img0.html)

(ITA being the company purchased by Google)

</comment><comment><USER10> Wow that's ridiculously fast. I'm used to seeing long-running spinners and "loading" pages.

Anyway, for US travel, no Southwest = no deal for me (not that Google or the other searches can do anything about that).

<comment><USER11> > no Southwest = no deal for me

Agreed, although I always thought the sticking point for SWA was paying commissions and not data access. For a sample search from BWI-SFO their price was only about $20 more expensive than the lowest fare ($520).

<comment><USER12> On the two occasions I have used it, Google flights was able to find me cheaper tickets than what Southwest offers. It is now my first stop, and then I double check Southwest to see if they can beat what Google has shown.

</comment></comment></comment><comment><USER13> The first impression I got from this was an interesting combination of creepy and incompetent.

The creepy part: The site gave 5 suggestions for places I might want to fly to. 3/5 were cities I've traveled to multiple times in the last few years. And while I might accept e.g. London as a default suggestion for anyone in Europe, something like Helsinki is a lot harder to justify.

The incompetent part: Google knows damn well I live in Zurich. They have Google+ profile information, they have my Android information, they have ip geolocation (which definitely gets my ISP right). So why is the suggested starting airport 150 km away, in another country? Ridiculous.

<comment><USER8> Maybe your isp is buying ip addresses from a provider there?

(It positions me in Calgary, where I have done the search from).

<comment><USER13> I used to work a few rooms away from the guys doing Google's IP geolocation database. I am pretty sure that this city + ISP has good coverage and is well tested :-) Also, all other location-dependent Google services get it right.

<comment><USER14> As far as I can tell, Flights doesn't "officially" support flights from Switzerland yet. If you try to select ZRH as your only departure airport, you'll get an error message stating as much.

If you were recommended flying from MLH (aka BSL), I'd wager it's just the closest (active) airport to your geolocation. If you were suggested STR, your geolocation may be off by a bit.

To get some results for flights out of ZRH, select MLH as your departure, then, on the map, click the marker at MLH. A list of close airports (BSL, BRN) will pop up. Click "more airpots" and add ZRH to your selection. It then somewhat works. Take note of the list of unsupported airlines at the bottom of the map.

</comment></comment></comment></comment><comment><USER15> Any info on how does it stack up against skyscanner / esky ?

edit: after a few minutes i can say that while the UI is 100x better than the aforementioned competiton, the flights my usual route i take few times a year (LDN - KRK) only display the price for the most expensive companies, which sort of defeats the point of using a crawler/search engine for this. hopefully they will add easyjet/ryanair/etc soon.

</comment><comment><USER16> Begs me to ask the question, how does most of HN search for flights?

I'm pretty addicted to the kayak +/\- 3 day matrix.

<comment><USER17> Honestly, I just buy from United most of the time unless the price is higher than I think it should be, then I'll search.

Frequent travelers are generally best served by sticking with a single airline.

<comment><USER2> Something about searching on United: [http://travel.stackexchange.com/questions/16581/does- browsin...](http://travel.stackexchange.com/questions/16581/does-browsing- ticket-sites-in-incognito-mode-and-or-from-different-ips-result-i)

I've experienced it myself searching on United.com. But yes, sticking with a single airline's usually a good idea — there are benefits towards acquiring elite status on a single airline's frequent flyer program (even when we're talking about airline alliances).

</comment></comment></comment><comment><USER18> For those that live in Europe and want a powerful search engine that includes low-cost flights (a deal-breaker for me), I recommend Azuon.

[http://azuon.com/en/feature-tour](http://azuon.com/en/feature-tour)

It's incredibly addictive.

Combined with airbnb you can plan some much cheaper and adventurous routes.

The payment plans are as follows:

\- €7.89 for 3 months (subscription)

\- €12.03 for 6 months

\- €16.54 for 12 months

\- €19.87 for 1-year

I've subscribed for 3 months now. It's worth every penny.

<comment><USER19> Thanks! First time I've ever heard of it but it sounds awesome!

Can you provide an example of discount you're able to get with this app and not with Kayak or Google Flights?

</comment></comment><comment><USER20> "our goal will be to refer people quickly to a site where they can actually purchase flights, and that we have no plans to sell flights ourselves."

By the look of it, this indeed follows the guideline they set. Google has always had a strong dislike of aggregators, and this certainly could easily be a big hit to sites like that.

That said, while it is a slick interface, I know quite a few people who are going to be quite unhappy about this being released.

</comment><comment><USER21> Wow, type in a place as a destination that doesn't have an airport (like Flagstaff, United States) and it will give you a map of the area, showing the lowest price for all the surrounding places.

I had intended to fly into Pheonix and rent a car to drive to Flagstaff, now I can clearly see where it's cheaper to fly into.

Nice.

</comment><comment><USER22> "Flights from Austria are not supported". Ok, I'll just keep using [http://matrix.itasoftware.com/](http://matrix.itasoftware.com/).

</comment><comment><USER23> I crave an API. Really want to play around with some visualisations.

</comment><comment><USER8> I'm the OP on reddit (posted this morning), note that there are a couple of other sites that do the same, and some which seem to have a better choice or airlines and prices.

I put a comment together with all the suggested sites:

[http://www.reddit.com/r/technology/comments/1hhqs7/google_fl...](http://www.reddit.com/r/technology/comments/1hhqs7/google_flights_choose_dates_and_a_budget_and/cauo9mg)

</comment><comment><USER24> "Flights from Turkey not supported" are you kidding me? I'm guessing they didn't integrate the ITA stuff yet.

</comment><comment><USER25> This gives me an urge to be spontaneous for a weekend. I don't always want to know where I'm going until a few days before it; and, knowing where there are airports and which ones have non-stop flights at a moment's inspection is actually quite a cool and convenient thing!

<comment><USER26> Our friends have a concept called an Orblitz. You got to orbitz.com on Wednesday and search for the cheapest tickets out of the state to anywhere. Take a backpack for the weekend and no hotels book - figure it out there.

Has been a blast every time. I actually used the concept at a Noisebridge party once:

[http://makezine.com/2010/10/13/noisebridges-3rd-birthday- par...](http://makezine.com/2010/10/13/noisebridges-3rd-birthday-party-end/)

A lot of fun for the winner (who actually beat me in the rock paper scissors tournament on the way to 'winning')

</comment></comment><comment><USER27> Is there a reason why they don't include something similar to Bing Flights' Price Predictor? I haven't actually booked a flight since they released that, but it seems like a nifty/useful piece of tech that Google would want an answer to.

</comment><comment> mark-ruwt For anyone feeling disappointed, take a look at Flight Explorer. It quickly became my first step when searching for flights or inspiration.

[https://www.google.com/flights/explore/](https://www.google.com/flights/explore/)

</comment><comment><USER28> Hmm, Google is missing the flexible dates and calendar view, while [http://matrix.itasoftware.com/](http://matrix.itasoftware.com/) is missing EasyJet and presumably other low-cost airlines.

<comment><USER29> [http://rense.com/general96/shockingplanerad.html](http://rense.com/general96/shockingplanerad.html) Who wants to fly when all that radiation is up there from Fukushima? Or is this rense guy way off the mark about this radiation you get when you fly?

</comment></comment><comment><USER30> It's a conspiracy: [https://www.google.ca/flights/#search;f=SVO;d=2013-07-18;r=2...](https://www.google.ca/flights/#search;f=SVO;d=2013-07-18;r=2013-07-22;mc=m)

;)

</comment><comment><USER31> "Sorry, flights from South Korea are not currently supported."

Which is fine, because local websites always have 30%+ cheaper rates than ITAsoftware, Hipmunk et al.

</comment><comment><USER32> Does Anyone know any API that I can use to search for flights? or how expensive or difficult to get an agreement to use ITA software?

</comment><comment><USER33> "Sorry, flights from Germany are not currently supported."

Seriously? Disappointing.

<comment><USER34> Same for Belgium.

<comment><USER35> And Australia. Seems like this is a long way off from becoming more useful than the many competitors.

</comment></comment></comment><comment><USER36> $700 million and 2 years later, and flight search is _still_ slower than the first random site I picked (dohop.com).

In the meantime I'm left wondering just how much global travel information now finds itself in the hands of the US feds as a result of the purchase.

<comment><USER37> What cost $700 million?

<comment><USER38> Google bought ITA for 700 million.

[http://www.nytimes.com/2011/04/09/technology/09google.html?_...](http://www.nytimes.com/2011/04/09/technology/09google.html?_r=0)

</comment></comment></comment><comment><USER39> Made with alien technology. (Common Lisp) ;-)

</comment><comment><USER40> I wonder if it displays any special flights out of Moscow.</comment>


White 64GB and black 4G TouchPad due in August, Opal and Pre3 by fall - <AUTHOR> http://www.precentral.net/white-processor-bumped-64gb-and-black-4g-touchpad-due-august-opal-and-pre3-fall

<comment><USER2> Did HP really have to name their product "TouchPad"? I was wondering why a laptop mouse needs 4G and 64GB disk before I clicked through the link.</comment>


I Switched from LastPass to 1Password (and You Should, Too) - <AUTHOR> https://www.reviewgeek.com/47843/i-switched-from-lastpass-to-1password-and-you-should-too/ <comment><USER2> I actually did the same, but then (as I commented on in a hn post yesterday[0]), I switched to BitWarden as I couldn't find anything that 1Password offered to make me want to pay for it over BitWarden. Not to say that 1Password doesn't have anything worth paying for, but for my personal use, BitWarden has been the ideal so far.

[0] [https://news.ycombinator.com/item?id=23924543](https://news.ycombinator.com/item?id=23924543)

<comment><USER3> Another mostly happy Bitwarden user here.

In spite of a few annoying usability issues the developers won't address, it's a pretty flawless replacement for LastPass and 1Password [both of which I've used previously] --at least on desktop. And it's free and open source, which is a bonus.

Unfortunately, I've yet to find any password manager, bitwarden included, that works reliably on Android.

Nine times out of ten they don't recognise that I'm on a login page at all and, on the rare occasion Bitwarden does actually launch when I enter a login form, it tells me it can't find any logins for the URL of my browser [yandex.com] rather than the URL of the site I'm actually on.

</comment></comment><comment> mongo-db-hater I switched from 1Password to Bitwarden and could not be happier.</comment>


100 million Facebook users (download torrent) - <AUTHOR> http://thepiratebay.org/torrent/5722635/Facebook_directory_-_personal_details_for_100_million_users

<comment><USER2> Before people get too bent out of shape/excited, the data within the files is just people's names and counts as well as the URL to their profile. All of these profiles were already public.

<comment><USER3> But if you went and got it from Facebook, I could alter my settings and make it go away. The torrent is forever.

However, to believe what I just said is completely naive. If you screw up and make something public, the likelihood that it is already forever somewhere is close enough to 100% that you should just expect it.

<comment><USER4> The URL to your profile and your name are always public information on Facebook, regardless of what settings you tweak. All this torrent supplies is your name and your URL.

</comment></comment></comment><comment><USER5> Any 1337 hackers interested in pulling off the same heist on LinkedIn are invited to peruse the list at this notorious Warez site:

[http://www.google.com/search?sourceid=chrome&ie=UTF-8...](http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=site:linkedin.com/in/)

</comment><comment><USER6> Facebook won't allow this link to be posted in the feed. Its considered abusive.

<comment><USER7> I believe Facebook doesn't allow any links from The Pirate Bay.

</comment></comment><comment><USER8> You can also get Google profile URLs for 16,256,271 users from the top post in this thread: <http://news.ycombinator.com/item?id=1537968>

<comment><USER9> I don't know if you've ever tried, but Google generally won't actually let you look through millions of results. It also eventually presents you with a CAPTCHA to solve if it detects that you're scraping its SERPs.

</comment></comment><comment><USER10> This is basically the business model of RapLeaf.

</comment><comment><USER11> Facebook's fuckup is not to have some sort of anti-scraping system running. If they'd limited the number of public-facing profile views to 100 a day per IP address then this would have taken 41,666 days for one IP address (obviously you'd use more, but it illustrates the point).

<comment><USER12> And many college students would be blocked from Facebook... even if you increased the number to 1000 :)

<comment><USER13> Do many colleges use NAT? Every college whose network I've been on gives external IPs to each network device.

<comment><USER14> The University of Alabama in Huntsville uses NAT for the folks who use its wireless network, or the network in the dorms; everyone on those networks gets a 10/8 address. Additionally, UAH is connected to I2, but doesn't provide any IPV6 connectivity for those who use its I1 connection. :/

</comment></comment></comment></comment><comment><USER15> What is the background on this file? What data is in it, where and how was it leaked, why does it exist, why do FaceBook users need to be concerned, or not?

<comment><USER16> This has more info : [http://techcrunch.com/2010/07/29/facebook-hacker-all-i- did-w...](http://techcrunch.com/2010/07/29/facebook-hacker-all-i-did-was- compile-publicly-available-information/)

</comment></comment><comment><USER17> I haven't looked at what's in the leaked data but is it some guy just incrementing id numbers and getting the JSON result via <http://graph.facebook.com/someidnumber> ?

<http://developers.facebook.com/docs/api>

</comment><comment><USER18> What's the difference of this torrent with this one ...

[http://www.google.co.uk/#hl=en&q=site%3Afacebook.com%2Fd...](http://www.google.co.uk/#hl=en&q=site%3Afacebook.com%2Fdirectory%2Fpeople%2F&aq=f&aqi=&aql=&oq=&gs_rfai=&fp=4da16c650c5b86c8)

<comment><USER18> ... and some profiles in HN are searchable...

[http://www.google.co.uk/#hl=en&q=site%3Anews.ycombinator...](http://www.google.co.uk/#hl=en&q=site%3Anews.ycombinator.com%2Fuser%3Fid%3D&aq=&aqi=&aql=&oq=&gs_rfai=&fp=4da16c650c5b86c8)

</comment></comment><comment><USER19> I would like to offer a bounty to the first capable script author who can take this 'proof of concept' and correlate names with email addresses and/or telephone numbers.

</comment><comment><USER20> If this included friend information it would be much more interesting.

As it stands, it's just a proof of concept.

<comment><AUTHOR> You could gather friend information easily. The info is there, this guy just didn't collect it.

</comment></comment><comment><USER21> Now for someone that scrapes the friend graph and turns it into a torrent. That might be more useful.

</comment><comment><USER22> I guet a malware warning when visiting the site.

</comment><comment><USER23> What is this useful for?

<comment><USER24> I guess it's useful for tracking people who might have deleted their facebook account.

</comment></comment><comment><USER25> While this is cool, I'd be truly impressed if it was expanded to all 500 million Facebook users, and included all the friend graphs. Basically a fully scraped copy of Facebook's data, kept up to date, and available for free via Bit Torrent. I suppose the best shot at doing that would be using a bot-net.

<comment><USER26> This is supposedly everyone with publicly searchable profiles so I'd be truly impressed if 329 million people adjusted their settings, and not at all surprised if the 500 million figure did something cute like classify visitors as users, eg: digg and their "40 million users" of which 0.0047% liked the current most popular story.

<comment><USER27> How do you "visit" facebook? You get a login page.

<comment><USER26> Lots of pages and even apps don't require you to login.</comment></comment></comment></comment>


Dive Into Python 3 now finished and downloadable as HTML and PDF - <AUTHOR> http://diveintopython3.org/?dupe=withhonor

<comment><USER2> The footnote is funny,

"""

This site is optimized for Lynx just because fuck you.

I'm told it also looks good in graphical browsers.

"""

<comment><USER3> Yeh, it confused me too.

One day, I asked diveintomark on twitter:

      Why there's an unpleasant 'fuck you'?

and get the reply from Mark: <http://twitter.com/diveintomark/statuses/1914075749>

Now I'm just wondering, has Mark been greatly annoyed by the Internet Censorship?

<comment><USER4> That's a really funny reply. I suggest Mark try "^__^ fuck you ^__^"

<comment><USER3> Oh, you ruined all fun

</comment></comment></comment></comment><comment><USER5> "Also available in dead trees" for the print edition. Can't stop laughing.

<comment><USER6> "dead trees", that's a very common expression; why would you find it funny?

<comment><USER5> Because I hadn't heard/read it before.

<comment><USER6> you, my friend, are a bleeding-edge internet newb :-P

<http://catb.org/jargon/html/D/dead-tree-version.html>

<http://catb.org/jargon/html/index.html> <\-- start here and spend a weekend.

</comment></comment></comment></comment><comment><USER7> Thanks Mark!

<comment><USER3> I followed a lot of diveinto series by Mark, so, Thank you!

</comment></comment><comment><USER8> _Über-geeks can also clone the Mercurial repository:

you@localhost:~$ hg clone <http://hg.diveintopython3.org/> diveintopython3 _

I know I'm a DBA, but I think this is the first time I've qualified as an Uber-geek.

</comment><comment><USER9> Hacker News continues to keep me informed. I was just looking two or three weeks ago for a refresher course on Python and found everything outdated. Stellar.

</comment><comment><USER10> Why is the section "What's New in Python 3" the last section?

</comment><comment><USER11> Yes! Now, only the small matter that more than 9/10 modules will break under python 3, but hey, at least now we have a finished Dive Into Python 3!

<comment><USER12> Nothing is stopping you from using Python 2.x. Writing a new version of a language is never intended to be backwards compatible. Indeed, breaking backwards compatibility is essentially a perquisite because a new version is the opportunity to add all the useful features that couldn't be added before.

<comment><USER13> That doesn't seem completely true. I could imagine default language syntax changing from one version to another, but still being modifiable via "from __future__ import ..." or "from __past__ import ...", respectively. Unless the bytecodes of the VM are incompatible (which is possible), there isn't any reason py26 code can't talk to py30 code...</comment></comment></comment>


US non-profit group sues cnn.com for uncaptioned online video - <AUTHOR> http://mediaaccess.org.au/latest_news/general/us-non-profit-group-sues-cnncom-for-uncaptioned-online-video

<comment><USER2> CNN provides those videos at essentially no cost to the viewer. While every website should strive to be 100% accessible, this litigation strikes me as frivolous, unfair, and more than a little scary. There are places accessibility should be legislated and mandated. An essentially free news and video site on the Internet is not one of them.</comment>


X-15 Flamethrower - <AUTHOR> http://www.ebay.com/itm/291629652846?ul_noapp=true <comment><USER2> Ships from Liberty, Texas. So you know it's good!

</comment><comment><USER3> eBay listing removed. How about an Amazon link?</comment>


The Future of Docker Desktop for Windows - <AUTHOR> https://engineering.docker.com/2019/06/docker-hearts-wsl-2/ <comment><USER2> So, I think the main sticking point here is the lock-in of Hyper-V. By making a new popular feature completely dependent on a technology that explicitly disables the use of competitive hypervisors, they're giving with one hand and taking with the other.

If I was on VM-Ware's executive team, I'd be seriously thinking about filing an anti-trust complaint and the open source community should be thinking about whether submarining virtualbox is worth what Microsoft is doing here.

<comment><USER3> Don't all hypervisors disable other hypervisors?

<comment><USER4> Hypervisor Framework and VirtualBox seem to cohabit just fine on MacOS.

<comment><USER5> Yeah. VirtualBox installs its own kernel extension at this point in time.

<comment><USER3> But can you run VBox and (say) Docker at the same time?

<comment><USER6> Yep. I ran docker and Fusion at the same time and it works great.

</comment></comment></comment></comment></comment></comment><comment><USER7> I'm forced to use Windows at work and am so looking forward to wsl2.

Starting a new terminal with wsl1 takes so long and I'm less than happy with all terminal emulators.

ConEmu has broken copy paste, alacrity doesn't have proper tiling... Heck, I've started using hyper of all things.

I realize that I probably never be _happy_ on Windows when I can actually use i3wm at home, but if it could just _suck less_ I'd be so ecstatic.

<comment><USER8> I am also pretty happy with WSL, but I'm even happier that I was able to get a mac at work lol.

Although I admit, sometimes the BSD toolchain poses some rather hilarious speedbumps. At that point WSL can be smoother? Weird.

<comment><USER9> I switched from Mac to WSL last year and yes, in some ways it is smoother. Having for example apt-get on top of the NT kernel is better than Homebrew on Darwin. It's worse in other ways that are slowly getting fixed, such as the sad state of terminal emulators on Windows. At least one thing will probably always be better on Mac: Settings/Defaults system with Plist files vs. Windows settings and the Registry. But to me that doesn't justify the lack of good laptop hardware choices anymore - PC laptops are so much better nowadays.

<comment><USER10> > Having for example apt-get on top of the NT kernel is better than Homebrew > on Darwin

Can you go into more detail about why that's the case — package selection, versioning policies, etc.? The main thing I've typically found is testing version-matched deployments and Docker has made me care about that a lot less.

<comment><USER9> Sure, we're heavily docker based too, but for development toolchain it's really valuable to have such a well supported package manager as apt. It's not just about package availability but community support - you have the same tooling locally that is widely used server side, so lots more documentation exists for it. Same goes for Linux userland vs. BSD.

</comment></comment></comment></comment></comment><comment><USER11> We work with a lot of independent contractors, many of whom get by fine with Window Home, so this is great news:

> ...because WSL 2 works on Windows 10 Home edition, so will Docker Desktop

Glad to see this too:

> bind mounts from WSL will support inotify events and have nearly identical > I/O performance as on a native Linux machine

<comment><USER12> I didn't think Hyper-V was available on Windows 10 Home Edition; has this changed or was it always available?

<comment><USER13> It's not. It's possible Microsoft is making an exception for WSL2, but I highly doubt the user will be able to run arbitrary VMs on HyperV without a Pro edition of Windows.

<comment><USER14> According to others in this thread, if HyperV is enabled then the windows host actually partially runs on HyperV (which is part of the reason you can't use other hypervisors with HyperV turned on).

The running theory is that they want HyperV on for all installs (like turned on by default).

</comment></comment></comment></comment><comment><USER15> Boy I'm happy Microsoft is making Windows 10 into a great development platform for the web!

One of the rare things to feel upbeat about is that we have great choice when it comes to OSes.

<comment><USER16> Competition at the OS level is good. Windows getting better will only make others try harder.

As a former Windows user, then macOS convert, I look at what's happening on Windows with envy. macOS is getting prettier and perhaps more consistent, but buggier. It's very frustrating. I've had so many problems with security updates, with both the latest and other supported versions. Apple support is terrible these days. I filed a bug report with them and weeks later they give me absolutely lame replies.

The only thing that gives me pause, still, about Windows 10: it updates when it wants to and that is difficult to turn off (you can set your network to "metered" and I hear that will do it). Also the telemetry. I feel like it spies on its users more than Apple's product does.

<comment><USER15> The updates btw got a lot better even for Windows 10 Home users. It can be argued if letting users turn off the updates is a overall good idea or not but at least you have more flexibility in postponing them with 1903.

</comment></comment></comment><comment><USER17> I've been following this longstanding Docker issue: [https://github.com/docker/docker.github.io/issues/6910](https://github.com/docker/docker.github.io/issues/6910)

The comments make it pretty clear how the Github crew feels about it, but I'm curious the response over here.

<comment><USER10> It seems unrelated to this article?

</comment></comment><comment><USER18> This is great news. No more CIFS bind mounts required for volume sharing with Windows drives. This is a particular issue with on corporate / Enterprise windows boxes.

<comment><USER19> It was also pretty unstable. Low memory conditions inside the docker VM => SMB times out => I/O terminatedn => File corruption. At least it did support mmap, which was an issue with the original virtualbox-backed docker for Mac...

Now, fingers crossed for transparent inotify...

</comment></comment><comment><USER20> From what I understand, this will only affect Linux containers running on Windows. Is that correct? What about Windows containers? I know they're not nearly as popular, but we did evaluate them at one time for running a legacy Windows application. I doubt we're alone in that use case.

</comment><comment><USER21> So... will WSL2 need to be installed as per WSL? because WSL will not install on my work machine as the machine is locked down by the firm.. So will I lose local Docker?

</comment><comment><USER22> I'm very skeptical of this new technology. I used VirtualBox for years and I will continue to do so. I just don't see any reason to switch to that new technology. Mounts? VirtualBox can do that. Fast boot? Well, VirtualBox boots in seconds, that's more than enough for me. Dynamic memory allocation? VirtualBox has memory balooning, if I ever would run into memory limits, I can always do that. And zero problems with all those Hyper-V conflicts, like Android emulators, snapshot support, VM import/export, actually any OS rather than just Linux, etc.

<comment><USER23> I used VMware since 2007, HyperV and VirtualBox for more than 5 years and I don't have such strong opinions, I use the one that is available and does the job. Because I had less problems with VMware and HyperV, I almost gave up to VirtualBox, YMMV.

</comment></comment><comment><USER24> It's sad that all this needs Windows 10 Pro, which makes all learner developers unable to use Docker to learn properly unless they upgrade.

</comment><comment><USER25> Amazes me how far people go to avoid simply running Linux, why isnt that the preferred option here ?

<comment><USER26> Because they want a laptop with decent power management that doesn't freak out when connecting or disconnecting additional displays, but they don't necessarily want a Dell XPS or a ThinkPad. They might also appreciate the ability to quickly nuke their Linux environment and create a new one if they really mess up their configs somehow.

<comment><USER25> None of the issues you speak of are relevant to running linux on a laptop instead of windows. Linux has decent PM and I havent had glitchy issues with displays in years.

The idea of nuking the linux environment gets easier when you run one natively as you start to understand the system better.

I understand the idea where everyone wants to have the same ecosystem to run things on (docker) but the best part of linux was the fact it wasnt the same as windows

This convergence of the operating systems is concerning

<comment><USER27> I dual boot Linux and Windows 10 on my personal Lenovo X1 and my Linux setup definitely has its moments with multiple displays... It mostly works until it does not then it's a headache.

Power management seems OK but I dunno how that relates to the fan controller because in Linux my laptop sounds like a jet engine 90% of the time while in Windows it rarely spins up.

<comment><USER25> Probably could use some configuration on fan control or CPU throttling :)

I have more issues with windows than I do linux but I have been using linux on a desktop for ~15 years or more now ( would say 19 but there was a BSD period in there )

</comment></comment></comment></comment></comment><comment><USER28> So WSL2 would use a lightweight VM running a Linux kernel to avoid the slow windows I/O.

</comment><comment><USER29> Will I have full access to the GPU?

<comment><USER4> Based on this FAQ, no.

[https://docs.microsoft.com/en- us/windows/wsl/wsl2-faq#can-i-...](https://docs.microsoft.com/en- us/windows/wsl/wsl2-faq#can-i-access-the-gpu-in-wsl-2-are-there-plans-to- increase-hardware-support)

<comment><USER30> Note that during the talk they said they heard the cries for GPU access loud and clear (they mentioned AI). It will hopefully come later.

</comment></comment></comment><comment><USER31> This is a big step toward the full merger of Windows and Linux on the desktop, which is the most logical way forward for both platforms. Kudos to Docker for taking a primary role and following Microsoft in this direction.

<comment><USER32> Can you explain how this is the most logical way forward for Linux?

For me, it feels it could be the opposite with Windows potentially impacting how Linux moves forward by having more devs using WSL instead of actual Linux and not actually optimizing for native Linux support.

For an example, Valve's Proton vs. native linux games, if I was a game dev, I wouldn't bother porting any Windows games to Linux, just rely on Proton instead.

<comment><USER33> >For an example, Valve's Proton vs. native linux games, if I was a game dev, I wouldn't bother porting any Windows games to Linux, just rely on Proton instead

I see what you mean, but on the other hand, most game development studios already didn't bother to port anyway.</comment></comment></comment>