Show HN: Kimono – Never write a web scraper again - <AUTHOR> http://kimonify.kimonolabs.com/kimload?url=http%3A%2F%2Fwww.kimonolabs.com%2Fwelcome.html <comment><USER4> The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward.

I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an error that something went wrong. Well, crap. That was a waste of time. Oh well, maybe it's just me.

Alright, I'll give it another shot using the website they used in the demo. Opened up a Hacker News discussion page and started to give it a try. Immediately it was far less intelligent than the demo. Clicking on a title proceeded to select basically every link on the page. Somehow I clicked on some empty spots as well. Nothing was being intelligently selected like it was in the demo. Fine, that wasn't working tremendously well, but I wanted to at least see the final result.

Same thing: just got an error that something went wrong and it couldn't save my work.

Disappointing. I still might try it again when it works 'cause it's a great idea if they really pulled it off. So far: doesn't seem to be the case.

<comment><USER5> HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like [http://ihackernews.com/](http://ihackernews.com/) or [http://hckrnews.com](http://hckrnews.com) (no comments though).

<comment><USER6> I really wish that HN wasn't even in the running for a "worst case". For a community that seems to be all about UX and innovation, shouldn't it run on at least a marginally user-friendly piece of software with this-century markup?

I get that it has a sort of kitschy or retro appeal, but it's just basically a pain to use and looks terrible.

I can't tell you how often I click next page to find that I've taken too long and my session or whatever has expired.

<comment><USER7> No. This site isn't about UX or innovation, it's about tech start ups. It's a constant reminder that something can be successful even if it was written in a LISP dialect and has a bunch of UX misses as long as the core idea is valid and the product usable enough.

I've used a bunch of HN skins that were supposedly better designed, but none of them stuck. Apparently it's just plain unnecessary for HN to be better.

<comment><USER8> So because it's written in LISP it gets a free pass in other areas? That seems like a dubious claim.

<comment><USER9> No, despite it being written in an obscure language and sparse UI it has massive traction anyway. There's a lesson in there somewhere.

<comment><USER10> > There's a lesson in there somewhere

The website that lets technologists hang out with people who might give them large sums of money to see their ideas succeed doesn't have to be good, or pretty.

If the website were better, or prettier, that would not add any additional value to the previously mentioned large sums of money.

(Oh, and advice, and being able to chat with industry leaders and experts on diverse arrays of topics, etc.)

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> The Simile group at MIT did something similar back around 2006. Automatic identification of collections in web pages (repeated structures), detection of fields by doing tree comparisons between the repeated structures, and fetching of subsequent pages.

The software is abandoned, but their algorithms are described in a paper:

        http://people.csail.mit.edu/dfhuynh/research/papers/uist2006-augmenting-web-sites.pdf

<comment><USER12> Oh, hey, memories. I worked one summer with David Huynh (who you're linking to there) and David Karger (his thesis advisor) on one of the Simile projects.

I vaguely remember playing around with this tool you mentioned. I thiiiiink it was this one[0], although it seems to be superseded by this one[1] now.

[0] [http://simile.mit.edu/wiki/Piggy_Bank](http://simile.mit.edu/wiki/Piggy_Bank) [1] [http://simile.mit.edu/wiki/Sifter](http://simile.mit.edu/wiki/Sifter)

<comment><USER13> Just had to chime in and say that David Huynh and his fellow programmers will be forever heroes to me and a small group of data journalists who depended on Gridworks/Google Refine/OpenRefine

</comment></comment></comment><comment><USER14> Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.

<comment><AUTHOR> Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.

<comment><USER15> how are you going to do it without having to know the actual authentication key(s)? if i don't trust anyone enough to give my auth away, and so unless the site being scraped has some sort of oauth support, how are you going to get any data?

of course, if this was an offline product, or self-hosted product, then it would solve that problem of auth instantly.

<comment><USER16> Would there any way to fake the beginning of an OAuth session with Facebook, Google or any other OAuth authenticated site? Kind of like replaying cookies to hijack sessions?

<comment><USER17> The route of proxying the web page presents much difficulty in doing actual authentication on Facebook or Google's website via the proxied webpage without first rewriting most of the javascript and hijacking their Ajax calls on the fly.

The approach I took was to hijack the Cookies from the browser once the user has signed in after on e.g. Facebook via the browser extension.

The route of proxying the website does in fact do away with the need to install any external 3rd libraries.

This browser extension I built coupled with the web service its integrated to does allow for scraping of pages from Facebook, Google and LinkedIn logged in pages as well.

[https://chrome.google.com/webstore/detail/krakeio/ofncgcgajh...](https://chrome.google.com/webstore/detail/krakeio/ofncgcgajhgnbkbmkdhbgkoopfbemhfj)

<comment><USER18> Hah, I've been working on this recently with Facebook, on a TV set-top-box. It was painful and I ended up giving up. xd_arbiter.php is the key, I think.

</comment></comment></comment></comment></comment></comment><comment><USER19> I've written more web scraping code than I care to admit. A lot of the apps that ran on chumby devices used scraping to get their data (usually(!) with the consent of the website being scraped) since the device wasn't capable of rendering html (it eventually did get a port of Qt/WebKit, but that was right before it died and it wasn't well integrated with the rest of the chumby app ecosystem).

This service looks great, good work! But since you seem to host the APIs created how do you plan to get around the centralized access issues? Like on the chumby we had to do a lot of web scraping on the device itself (even though doing string processing operations needed for scraping required a lot of hoop jumping optimization to run well in ActionScript 2 on a slow ARMv5 chip with 64mb total RAM) to avoid all the requests coming from the same set of chumby-server IP addresses, because companies tend to notice lots of requests coming from the same server block really quick and will often rate limit the hell out of you, which could result in a situation where one heavy- usage scraper destroys access for every other client trying to scrape from that same source.

<comment><AUTHOR> Access, legality and rate limiting issues come up a lot. We're working on a couple things to address them. The first is an intelligent job distribution system that consolidates scrapes across users and hits sites (and pages) at human-like intervals. the second is to create a portal for webmasters that allows them special privileged access to analytics on data being extracted from their sites, and the ability to "turn on or off" kimono APIs if they see fit. this way, via kimono, a webmaster at chumby could "provision" certain kimono users. we're still yet to see whether the later works out. thanks for the input

<comment><USER20> Use a user-agent containing a URL to find out who and what you are, and honor my robots.txt.

Having a panel for webmasters _along_ with that would be fine.

<comment><AUTHOR> Great suggestion... thanks for this one. We're putting this on our list

<comment><USER21> Please tell me that the robots.txt suggestion is something that you're already doing and the user agent part is whats going on the list.

<comment><USER17> You could try doing IP address rotation by rotation EC2 instances or some other cloud services.

I wrote a library for that.

[https://github.com/KrakeIO/resque-my-aws](https://github.com/KrakeIO/resque- my-aws)

<comment><USER20> That has nothing to do with what's being discussed on this thread.

</comment></comment></comment></comment></comment></comment></comment><comment><USER22> I'm curious how you plan to avoid/circumvent the inevitable hard IP ban that the largest (and most sought after targets) will place on you and your services once you begin to take off?

I could have really used a service like this just yesterday actually, I ended up fiddling around with iMacros and got about 80% of what I was trying to achieve.

<comment><AUTHOR> It's a great question. What we're really trying to do is make data accessible programmatically and at scale. We want to connect data providers and data consumers with APIs in a way that's mutually beneficial vs. being a tool for data theft. Our hope is to (once we scale) actually work with data providers directly on the on the distribution of their data so the IP ban becomes a non- issue.

<comment><USER23> But isn't the point kinda to let the users come up with data providers themselves? If you say "Only these 500 data providers are available for scraping", you don't have a business. If you _don 't'_ have such a limitation, you'll not be able to work directly with all data providers. You'll have IP problems.

</comment></comment></comment><comment><USER24> This is excellent. Even it if doesn't work for scraping all sites, it simplifies the average use case so much that it's not even funny.

Feature proposal: deal with pagination.

<comment><USER25> Another feature, simple one: Allow to add some filters to the data stream. For example: only posts that contain word "bitcoin" in the name or only those with 50 upvotes or more.

<comment><AUTHOR> Thanks for the suggestion... adding to the list :)

<comment><USER26> Make sure to include regex matching =)

<comment><AUTHOR> Thanks. We support regex matching now. Try dragging to select text, if there's a relevant regex pattern and kimono will find it (there's an example inthe blog post). You can preview (and soon, you'll be able to edit) the CSS and Regex also in advanced mode.

<comment><USER25> Too bad I couldn't edit selectors and regex-es at this step. I could implement the filters I needed myself manually like this.

</comment></comment></comment></comment></comment></comment><comment><USER27> Constructive Tone: I figured that it might be nifty to scrape cedar pollen count information from a calendar and then shoot myself an email when it was higher than 100 gr/m3.

This would be a pretty difficult thing to grab when scraping normally, but the app errors before loading the content:

[https://www.keepandshare.com/calendar/show_month.php?i=19409...](https://www.keepandshare.com/calendar/show_month.php?i=1940971)

JS error: An error occurred while accessing the server, please try againError Reference: 6864046a

<comment><AUTHOR> Thanks for letting us know. Just tried and am getting the same error. The page is loading content dynamically from another source... We'll look into this and see if we can get it working on this page

<comment><USER27> Do you support POSTs for fetching dynamic data? I found where it's pulling from, here's the curl command:

curl "[https://www.keepandshare.com/calendar/fns_asynch_api.php?r=0...](https://www.keepandshare.com/calendar/fns_asynch_api.php?r=0.04834879608824849&fromapp=calendar") \--data "action=getrange&i=1940971&from=2013-12-26&to=2014-02-06"

<comment><AUTHOR> No, we don't have POST support quite yet. We're working on a solution.

</comment></comment></comment></comment><comment><USER28> Great work so far. The tool was very intuitive and easy to use.

My suggestion: once I've defined an API, let me apply it to multiple targets that I supply to you programatically.

The use case driving my suggestion: I'm an affiliate for a given eCommerce site. As an affiliate, I get a data feed of items available for sale on the site, but the feed only contains a limited amount of information. I'd like to make the data on my affiliate page richer with extra data that I scrape from a given product page that I get from the feed.

In this case, the page layout for all the various products for sale is exactly the same, but there are thousands of products.

So I'd like to be able to define my Kimono API once - lets call it CompanyX.com Product Page API - then use the feed from my affiliate partner to generate a list of target URLs that I feed to Kimono.

Bonus points: the list of products changes all the time. New products are added, some go away, etc. I'd need to be able to add/remove target URLs from my Kimono API individually as well as adding them in bulk.

Thanks for listening. Great work, again. I can't wait to see where you go with this.

Cheers!

<comment><AUTHOR> Thanks a ton for the feedback. Getting data from multiple similarly structured URLs programmatically is something we're working on now. We love hearing about the use cases you want to use this for so we can make sure we build out the right features to make kimono useful for you.

</comment></comment><comment><USER29> Just use [http://selectorgadget.com](http://selectorgadget.com)

<comment><USER30> You should write a blog post on lessons learned when we spent a year making ~this in 2008.

<comment><USER31> Thanks so much for creating SelectorGadget! I used it a lot when scraping some fanfiction and Wikipedia data.

</comment></comment></comment><comment><USER32> Undo button is awesome.

More web apps need an undo button.

</comment><comment><USER33> Are you familiar with ScraperWiki? I'm wondering how your work fits in with it.

Edit: looks like they've moved away from that space, but have an old version available at: [https://classic.scraperwiki.com/](https://classic.scraperwiki.com/)

<comment><USER34> The people who scrape data to avoid paying for APIs are the same people who will not pay for a service to make scraping easier ;)

<comment><USER35> The people who scrape data at Scraperwiki -- which was made by the same people who opened up parliamentary transcripts in the UK for the first time, and the UN's proceedings, and data about how MPs in London vote -- generally don't have an option to buy anything because the data's hidden by governments from the people who paid for it, on purpose.

But by all means take this opportunity to dismiss all of us as freeloaders.

<comment><USER34> I have 17 scrapers on ScraperWIki classic for government data [https://classic.scraperwiki.com/profiles/maxious/](https://classic.scraperwiki.com/profiles/maxious/)

It would have cost me $348/year to move those to new scraperwiki.

<comment><USER36> That reads as "Less than $1/day" to me...

</comment></comment></comment></comment></comment><comment><USER37> This is a great tool! In a past life we needed a web scraper to pull single game ticket prices from NBA, MLB, and NHL team pages (e.g. [http://www.nba.com/warriors/tickets/single](http://www.nba.com/warriors/tickets/single)). We needed the data. But, when you factor in dynamic pricing and frequent page changes you are left with a real headache. I wish Kimono was around when we were working on that project.

I love how you can actually use their "web scraper for anyone" on the blog post. Very cool!

</comment><comment><USER38> That UI made me go wow, this could be an awesome tool. Idea that pops into my mind is being able to grab data from those basic local sites run by councils, local news papers etc and putting it into a useful app.

How dedicated are you guys to making this work because I'd imagine there are quite a few technical hurdles in keeping a service like this working long term while not getting blocked by various sites?

<comment><AUTHOR> Love your suggestion. We're committed to making kimono better and we're working on it all the time. We want to make sure it's a responsible scraper, so want to work together with webmasters in cases where there might be blocking but the data is legal to share...

</comment></comment><comment><USER39> >Sorry, can't kimonify

>According that web site's data protection policy, we were unable to kimonify that particular page.

Sigh... Oh well... Back to scraping.

<comment><AUTHOR> What page were you trying to hit? We'll check it out

<comment><USER39> Pages buried in here: [https://fannin4.wcjc.edu/](https://fannin4.wcjc.edu/)

The course catalog is public, so no login is needed. I want to scrape various data related to courses, to populate forms automatically and such.

<comment><USER40> Yeah, of course it won't work with HTTPS sites. They'd have to proxy those HTTPS sites and perform a MITM just to do it.

<comment><USER39> I kind of just assumed that's what they were doing.

<comment><USER17> HTTPS is definitely a problem for proxy servers unless you the proxy server rewrites all the URLs in the html pages loaded as well as all the URLs of the Ajax calls to point back to the proxy server.

<comment><USER39> It may be to their advantage to come up with a solution for this, given the popularity of https these days.

</comment></comment></comment></comment></comment></comment></comment><comment><USER41> > _Web scraping. It 's something we all love to hate. You wish the data you > needed to power your app, model or visualization was available via API. But, > most of the time it's not. So, you decide to build a web scraper. You write > a ton of code, employ a laundry list of libraries and techniques, all for > something that's by definition unstable, has to be hosted somewhere, and > needs to be maintained over time._

I disagree. Web scraping is mostly fun. You don't need "a ton of code" and "a laundry list of libraries", just something like Beautiful Soup and maybe XSLT.

The end of the statement is truer: it's not really a problem that your web scraper will have to be hosted somewhere, since the thing you're using it for also has to be hosted somewhere, but yes, it needs to be maintained and it will break if the source changes.

But I don't see how this solution could ever be able to automatically evolve with the source, without the original developer doing anything?

<comment><USER42> Perhaps this could be automated by finding the same content in two versions of the dom and then doing a diff on the structure, updating the rules?

<comment><AUTHOR> It would be great to automate this eventually. For now, we're trying to make it really easy to set up and rebuild the scraper. If it goes down, you'll see it in the status on your user dashboard. We're also implementing alerts, so you can opt to get an email notification if a scrape fails

</comment></comment></comment><comment><USER43> Wow, this is looking good, I wish I had it available to me 6 months ago! Nice job :D

I don't know if it's just me or not, but it's not working for me in Firefox (OSX Mavericks 10.9.1 and Firefox v26). The X's and checkmarks aren't showing up next to the highlighted selections. Works fine in Safari.

<comment><AUTHOR> Thanks for letting us know. We've tested on some versions of Firefox, but not v26 on Mavericks. We'll look into this

</comment></comment><comment><USER44> Great tool!

I'm coming at things from a non-coder perspective and found it easy to use, and easy to export the data I collected into a usable format.

For my own enjoyment, I like to track and analyze Kickstarter project statistics. Options up until now have been either labor intensive (manually entering data into spreadsheets) or tech heavy (JSON queries, KickScraper, etc. pull too much data and my lack of coding expertise prevents me from paring it down/making it useful quickly and automagically) as Kickstarter lacks a public API. Sure, it is possible to access their internal API or I could use KickScraper, but did I mention the thing about how I dont, as many of you say, "code"?

What I do understand is auto-updating.CSV files, and that's what I can get from Kimono. Looking forward to continued testing/messing about with Kimono!

</comment><comment><USER45> looks promising!

to be fully usable for me, there are some features missing:

\- it lacks manual editing/correcting possibilities: i've tried to create an api for [http://akas.imdb.com/calendar/?region=us](http://akas.imdb.com/calendar/?region=us) with "date", "movie", "year". unfortunately, it failed to group the date (title) with the movies (list entries) but rather created two separate, unrelated collections (one for the dates, one for the movies).

\- it lacks the ability to edit an api, the recommended way is to delete and recreate.

small bugreport: there was a problem saving the api, or at least i was told saving failed - it nevertheless seems to be stored stored in my account

<comment><AUTHOR> Thanks for the feedback. We're working on a feature that will allow you to edit APIs you've created and also edit the selectors and regex (right now, in advanced mode, you can see them, but cannot edit). We're looking into your bug now...

</comment></comment><comment><USER46> I would seriously consider rethinking that Favicon.

<comment><USER47> Seconded. I can't show that to anyone at work.

<comment><USER48> To me, the favicon merely looks like a sumo wrestler's head with a short ponytail and scowling/serious eyebrows. I can't tell what NSFW thing you see it as.

<comment><USER49> Until I read this thread I also saw a sumo or an angry onion, but I believe the picture is actually a person facing away from us undoing their kimono.

<comment><USER50> You see all that in a 32x32 pixel image?

<comment><USER20> It's much larger than that [http://kimonify.kimonolabs.com/favicon.ico](http://kimonify.kimonolabs.com/favicon.ico)

</comment></comment></comment></comment></comment></comment><comment><USER51> I'm experiencing login errors (PEBKAC caveat: password manager, 2x checked, reset), but the support confirmation page is a nice surprise.

[http://i.imgur.com/w01CoUy.jpg](http://i.imgur.com/w01CoUy.jpg)

</comment><comment><USER52> Nice work, this is much better than I expected! Does it require Chrome? It doesn't seem to work in Safari for me. Also, does Kimono work for scraping multiple pages or anything that requires authentication?

<comment><AUTHOR> Great, it should work well on webkit browsers, what version of Safari are you using?

<comment><USER52> 7.0.1, the latest. I also don't have Flash installed, but it doesn't look like you're using Flash. The entire top bar doesn't show for me. Feel free to email me and I can send you screenshots.

<comment><AUTHOR> Thanks - we're not using flash, so it must be something else. Will follow up over email

</comment></comment></comment></comment><comment><USER17> I found the one click action for selecting an entire column of values as well as the UI/UX on the top column of the page to be very impressive. We were thinking of a nice clean way to represent that particular UI/UX flow in this browser extension we built as well. Will incorporate that in our next release.

[https://chrome.google.com/webstore/detail/krakeio/ofncgcgajh...](https://chrome.google.com/webstore/detail/krakeio/ofncgcgajhgnbkbmkdhbgkoopfbemhfj)

Would love to meetup and exchange some ideas if you are based in Bay area.

</comment><comment><USER53> I like how you've thought through the end to end use case: not just generating an API, but actually making it usable. I've done my fair share of web scraping and it's not an easy task to make accessible and reliable -- good luck!

It makes me wonder if there isn't a whole "API to web/mobile app with custom metadata" product in there somewhere. I can imagine a lot of folks starting to get into data analysis and pipelines having an easier time of it if they could just create a visual frontend in a few clicks.

<comment><AUTHOR> Yes, we're excited about the possibilities of an end-to-end use case as well... in fact we were surprised when we found a lot of interest in front-end output layers on top of the APIs than just the APIs themselves. Would be curious to know what output features would be most valuable for you.

<comment><USER53> Well. Think about spreadsheets. Think about live spreadsheets powered directly by APIs. Bingo!

I'm doing a couple of data mining projects right now and simply being able to query and look into the API outputs, as well as my local database, without building a custom frontend would've saved me a bunch of time. But I'm thinking more of the knowledge worker, or even the power user who wants to view their Fitbit, Up and Lark data all in the same dashboard. Can't help but thinking this already exists somewhere though.

<comment><AUTHOR> Love your idea. Would love to follow up on this with you

</comment></comment></comment></comment><comment><USER54> We all know there are a lot of existing tools that does the same things. But I've not met one with such a polished UX. Kudos to the Kimono team, I'll definitly recommend your product.

</comment><comment><USER55> Very nice job. What about scraping data from password-protected pages?

<comment><AUTHOR> Great request... it's on our feature shortlist. Definitely a feature we want to implement as soon as we can (after we tackle some basics like pagination and getting images)

</comment></comment><comment><USER56> Like the parameter passthrough feature. Take a look at places where the parameters are part of the URL structure. For example a Target product page [http://www.target.com/p/men-s-c9-by-champion-impact- athletic...](http://www.target.com/p/men-s-c9-by-champion-impact-athletic- shoe-black/-/A-14656388#prodSlot=medium_1_1)

In order to get data for a different product, I will have to modify the URL itself. I think same holds true for blog posts.

<comment><AUTHOR> Yes, it's a great point. We're working on updating the query param passthrough to handle params within the URL structure.

<comment><USER56> Apart from that here are other items. You may have these on your list, but can count my vote to prioritize.

1\. Pagination 2\. Image URLs 3\. Focus on page types such as product pages, posts etc. That way its easy to go from content to content. Will help crawling too 4\. Link back to original page included in JSON

Finally common sites/pages used by multiple users of your systems should not count against the API count requirement under pricing. You may want to charge against total calls, like Parse.

<comment><AUTHOR> Thanks for the suggestions!

</comment></comment></comment></comment><comment><USER57> I really like how you guided me in to demoing. Nice job.

</comment><comment><USER58> This is awesome. Really nice implementation and so useful for many different applications. Just signed up and looking forward to trying this out.

</comment><comment><USER59> Cool concept. One concern I'd have about this type of tool is that when it encounters something it can't handle, I'm stuck. Writing your own scraper means that you can modify it when you need to. I think the ultimate solution would be something like Kimono with the ability to write snippets of custom javascript to pull out anything that it can't handle by default.

<comment><AUTHOR> We're in the middle of implementing a more power developer version of the tool to handle the use cases you're talking about. The beginning of this is surfaced under the "advanced" tab in the data model view where we show the selectors and regular expressions that are produced. we want to ultimately let you edit those to customize the extractor. From there it'd be super cool to implement the javascript snippet feature you suggested.

</comment></comment><comment><USER60> I'm normally a bit worried when a thread quickly fills up with praise, but this looks very nice.

It's something I have thought about, as I'm sure many people who have done any amount of scraping have, but never went forward and tried to implement. The landing page with video up top and in-line demo is a pretty slick presentation of the solution you came up with. Good job.

<comment><AUTHOR> Thanks we were pretty surprised as well, but we're really grateful for the encouragement

</comment></comment><comment><USER61> Please get this off the ground. I would also possibly suggest a separate business, website regression testing.

Selenium is WAAAY to painful.

<comment><AUTHOR> Thanks for the suggestion... we're working hard to get auth up and running!

</comment></comment><comment><USER62> Thank you for building a tool I been wanting so I don't have to!

Can't wait to play around with this tonight.

Suggestion. Allow one to select images.

<comment><AUTHOR> Great add. It's on our shortlist - a popular request!

<comment><USER62> Also, the ability to style it myself would be nice :)

</comment></comment></comment><comment><USER63> This looks really slick. What happens if a website you're scraping changes its design? Do you respect robots.txt?

<comment><AUTHOR> If it changes the format significantly, the scraper will break, so for now you'll have to use the tool to rebuild. You will see on your API status page that it's down. As for robots.txt, we do respect it... for now we're leaving that to the user, but we're trying to implement a proactive way of checking for disallows and stopping those scrapers from being built.

<comment><USER21> Please clarify: are you saying that right now you leave respecting robots.txt to the user?

<comment><AUTHOR> At the moment, we rely on users to be responsible. We spell it out in the terms and FAQ. We've been in private beta, keeping usage very limited until today. We fully understand the seriousness of the issue as we scale. We're committed to becoming a responsible bot that respects robots.txt

<comment><USER64> I would say how you're scraping differs from say how Google, a search engine, scrapes. I'm not sure there is a way in robots.txt to define for each use? Knowing the data in a structured way, but then allowing it to be displayed in full off-site is quite different than using the scraped data for linking into a website.

<comment><USER21> But robots.txt provides minimums: don't scrape this page, don't refresh more than once every x, these crawlers are allowed this access, etc.

</comment></comment></comment></comment></comment></comment><comment><USER65> There's a huge business here if you keep at it. I'll throw money at the screen if you can make this work.

</comment><comment><USER66> Definitely awesome presentation and product.

The example doesn't seem to work right on Firefox. On Chrome, if I click "Character" in the table then it highlights the whole column and asks if I want to add the data in the column. On Firefox, clicking "Character" just highlights "Chatacter" and that is it.

Ubuntu 12.04

Firefox 25.0.1

<comment><AUTHOR> Thanks for flagging. We'll get on this to figure out what's going on when running on ubuntu

</comment></comment><comment><USER67> I built something very similar last year, but sadly never got around to polishing and launching it: [http://exfiltrate.org/](http://exfiltrate.org/)

(There's a prototype of an API generator hidden in a menu somewhere but it's nowhere near production ready)

<comment><AUTHOR> Yeah, we've been working on this for a while too... took a while to polish it a bit before we could put it out there. Will check out exfiltrate.org - <USER2> cool!

</comment></comment><comment><USER68> Nice tool, slick UI. It worked for some pages and not for others. Currently I'm using Feedity: [http://feedity.com](http://feedity.com) for all business- centric data extraction and it has been working great (although not as flexible as kimono).

</comment><comment><USER69> Great job guys.

One problem I've had though is that I think you guys are hosted on AWS - a lot of websites block incoming connections from AWS.

Are there plans to add an option in future to route through clean IPs? Premium or default, this would be cool and make it a lot more useful.

</comment><comment><USER70> Nice job! I really liked, it's a fantastic idea! And your UX is great! Just one thing I've found when testing: I've had some problems with non-ascii characters, when I was visiting brazilian websites, such as this : www.folha.com.br.

</comment><comment><USER71> Well done on the product & solving a clear need! This is extremely useful for hackathons/prototyping. I also loved the live demo in the blog post and you did a wonderful job with the design/layout/colorscheme of the site.

</comment><comment><USER72> Very cool, and I like that the link is your announcement page running inside of the demo. Really drives home the idea.

That said, it looks like it can't do media right now. I would love it if it could at least give me a url for images/other media.

<comment><AUTHOR> It's a great suggestion, thanks! ... image extraction would be cool, and it's on our shortlist of features to build next

</comment></comment><comment><USER73> Does it do logging in to websites then fetching? Do you plan to add scripting to it?

<comment><AUTHOR> We don't support logging in yet, but it's a feature we're working on adding. Scripting will also be cool, but it's right now further down our feature queue

</comment></comment><comment><USER74> I don't think this can beat the speed of a hand-tune crawler. When I write crawlers, I skip rendering page and javascript execution if it isn't needed, which massively speed up the crawling process.

<comment><AUTHOR> There's definitely things that a custom-built scraper can do more efficiently than kimono, but our focus right now making scraping accessible across a broad enough range of web sites.

</comment></comment><comment><AUTHOR> Thanks guys, glad you like it. Welcome any feedback so we can make it better!

</comment><comment><USER75> Really cool idea and tool. Still need to test this out properly. Is it possible to scrape note just one page but a stack of them? For example - <USER3> product catalog of 1000 SKUs extending upto 50pages.

<comment><AUTHOR> We don't support that quite yet. It's our #1 feature request though, and we're working to get it ready soon

</comment></comment><comment><USER76> Does such Webscraping is allowed legally. Since it is not done directly from our servers and if any legal action will be taken by the scraped website , will it be on kimonolabs..or the user..

</comment><comment><USER77> really excited to see this. i've had the idea (and nearly this execution) in mind for years but no use or ambition to get it done.

given the pricing though i'm almost motivated to make my own. as a hosted service the fees make sense with the offerings. but not only would i rather host my own- it would be cheaper all around. would you consider adding a free or cheap self hosted option?

aside, i think there is a mislabel on the pricing page. i'm guessing the free plan should not have 3 times the "apis" than the lite plan.

<comment><AUTHOR> Yes, we're in beta right now, while we're still working out the bugs. For beta, it's free for 30 APIs.

<comment><USER77> thanks for the response! any chance at a self hosted option? i'd even still pay (once) for a self hosted version.

</comment></comment></comment><comment><USER78> Really sleek interface, and looks like it could be extremely useful (I just spent a few hours cranking out Nokogiri this morning).

Oh, typo: "Notice that toolbar at the toop of the screen?"

<comment><AUTHOR> Awesome, thanks for the kind words. And for catching that typo, will fix that now

<comment><USER79> I thought it was intentional. Swedish chef style. I like it, but I'll need to go back and re-read to understand how I can use this on other pages than the homepage/demo page. I've nothing immediate to try it with right now. :)

Edit: I'll watch the video after work, probably will clear everything up for me.

</comment></comment></comment><comment><USER80> This looks really useful, and I'm trying to figure out if I could use it on a project I'm working on, but hitting an issue. I sent a support message. Nice job!

<comment><AUTHOR> Thanks, the support tickets really help us debug. We'll look into it and get back to you

</comment></comment><comment><USER81> Looks awesome, however I keep getting errors and 404s. Could this be an issue on my end (seems to be working for others) or just HN making the servers beg for mercy?

<comment><AUTHOR> Where are you getting the 404s? We will check into it now

</comment></comment><comment><USER82> Reminds me of Dapper

[http://open.dapper.net/](http://open.dapper.net/)

This allowed you to do similar, before being consumed by Yahoo. Might be worth a look.

</comment><comment><USER83> Awesome! Hats off.. How about extracting hashtag/GID of any record if applicable, which are typically not rendered on page, but hidden under the hood.

</comment><comment><USER84> the reason i ever have to write a scraper is because of pagination. while this looks awesome, i'll have to stick to scraping until that is solved. :(

<comment><AUTHOR> It's probably our #1 feature request at the moment. We're working on it and hope to have it ready for you to try soon

</comment></comment><comment><USER85> I thought to myself oh boy yet another web scraper as a service but got surprised. I haven't been this impressed with a product video since Dropbox.

</comment><comment><USER86> Wow looks amazing. I tried doing some queries on public directories, and it even supports parameter passing. Will be using this for some side projects.

</comment><comment><USER87> How (if at all) does this run on javascript heavy sites?

<comment><AUTHOR> JS-heavy sites can be tricky. We position it so it should execute after most of the on-page JS, so it handles a lot of cases. There are still sites that break it though.... we're trying to tackle these guys one by one right now, as we try to generalize a broader solution

</comment></comment><comment><USER88> Any chance you guys plan to add link hrefs to CSVs? I'd love to use this now, but I need the href for backlinks and future inference.

<comment><AUTHOR> Thanks for the suggestion, we're adding to our list

</comment></comment><comment><USER89> The UX is great and a journalists everywhere will thank you.

But outside of government websites I don't see how a lot of this is even legal, per se?

<comment><AUTHOR> Thanks... yes, public data from governments is a great use case. Often a lot of apps built using scrapers will wind up driving up traffic/ sales a the source site so it's okay. We want to do responsible web scraping, so will respect webmasters robots.txt files to make sure it's legal.

<comment><USER90> I love the execution, but I also see inherent problems.

Robots.txt is just a convention to advise crawlers. I'm confident most sites explicitly state this is against their terms of service.

You will encounter terms along the lines of:

 _" Unauthorized uses of the Site also include, without limitation, those listed below. You agree not to do any of the following, unless otherwise previously authorized by us in writing: Use any robot, spider, scraper, other automatic device, or manual process to monitor, copy, or keep a database copy of the content or any portion of the Site."_

<comment><AUTHOR> You've got a valid point. We want to eventually create a space that allows responsible scraping - so webmasters can have access to analytics on what's being scraped and can explicitly turn off kimono APIs for their domains if they see fit. We also think there are use cases for people who own their own data. Often, APIs will provide a way for companies to streamline their internal app development and figure out what to expose to the developer community before investing in an expensive API deployment.

</comment></comment></comment></comment><comment><USER91> One of my favorites: [http://htmlagilitypack.codeplex.com](http://htmlagilitypack.codeplex.com)

</comment><comment><USER92> It would be nice to have a view also on the raw html code, e.g., to create a field containing the url of an image in the page.

<comment><AUTHOR> Thanks, for the suggestion. We're rolling out advanced mode soon, which will allow you to edit the CSS selectors and RegEx operating on the page's HTML to define the selected data elements

</comment></comment><comment><USER93> It looks cool, but very expansive compared to Visual Web Ripper, which you pay way less for (but has to host yourself).

</comment><comment><USER94> as someone building a home grown proprietary scraping engine. Consider alternative locations of elements. Most sites are using templating engines so its fairly reliable to find things in the same place, but more often than you might expect, things move a round ever so slightly. Navigation is a fun one also. ;)

</comment><comment><USER21> This is my third time trying to get an answer to this question: does your crawler automatically respect robots.txt?

<comment><AUTHOR> So sorry for missing this earlier. See our response in comments below: "At the moment, we rely on users to be responsible. We spell it out in the terms and FAQ. We've been in private beta, keeping usage very limited until today. We fully understand the seriousness of the issue as we scale. We're committed to becoming a responsible bot that respects robots.txt"

</comment></comment><comment><USER95> You can use the utility without registration or login by blocking the login prompt with, for example, AdBlock.

</comment><comment><USER96> This is fantastic. Congrats on launching it! Once it has pagination & auth I'll be all over this :)

</comment><comment><USER97> What about some navigation tools there?

Looks pretty good, but it does not really replace my scrappers. Maybe some of them...

<comment><USER98> re: nav tools -- you mean the ability to crawl multiple pages?

</comment></comment><comment><USER99> Seems it can't see the stuff inside angular views.. well at least mines..

But for the rest, awesome product. Thanks.

<comment><AUTHOR> Yes, you're right... we can't handle angular quite yet. We're working on this.

</comment></comment><comment><USER100> Looks very nice. There seems to be an issue with international characters though (æ/ø/å).

<comment><AUTHOR> Yes, thanks for spotting. We'd discovered that Chinese, Japanese and Korean failed but didn't know about these characters. Thanks!

</comment></comment><comment><USER101> It appears that it doesn't work with websites containing international characters.

<comment><AUTHOR> Yes, thanks for noting. This is a bug and we're working to get these characters supported as soon as we can

</comment></comment><comment><USER102> great idea. i'll have to keep this in mind for future projects.

</comment><comment><USER103> It's easy not to write web scrapers even without this tool ;)

</comment><comment><USER104> I like the concept. Would love to see page authentication

</comment><comment><USER105> Is there an ability to scrape more than one page of data?

<comment><AUTHOR> Not yet, but it's our #1 feature request, so we're working on it now. For now, you can make multiple APIs (one for each URL). If the URL takes query parameters though, you can re-use the same API and programmatically cycle through query parameters.

</comment></comment><comment><USER106> This is really slick! Btw. Who made your intro video?

<comment><AUTHOR> It's pretty homegrown right now, we did the intro video ourselves on our laptops

</comment></comment><comment><USER107> I like the concept and it looks similar at Import.io

<comment><AUTHOR> Yes, in concept quite similar. We wanted to make something that you can use from within your browser as part of your natural workflow, without installing any other software.We also really wanted to figure out the right data association intelligently based on user selections vs. asking users to think through a data model up front

</comment></comment><comment><USER108> I kind-of enjoy writing web scrapers.

</comment><comment><USER109> How does this compare with Mozenda?

</comment><comment><USER110> Man that demo is impressive!

<comment><USER111> Agree demo is awesome. But I don't think scraping for any web page is that simple. Lots of exceptional cases.

</comment></comment><comment><USER112> That looks quite swift

</comment><comment><USER113> use any chrome xpath plugin and give that to YQL

</comment><comment><USER114> so what do you do that import.io doesn't?

</comment><comment><USER115> i love this! and amazing video!

</comment><comment><USER116> OMFG.

</comment><comment><USER117> neat.

</comment><comment><USER118> "Never write a web scraper again"... yea right.. sick and tired of such gimmicks and self promotion on the net today.

</comment><comment><USER119> actually I love scraping :(</comment>


Ask HN: Where's the best place to purchase a domain name? - <AUTHOR>

Whilst GoDaddy has had a lot of brand name recognition through show like Diggnation, the extensive "pimping" of extras and the lack luster user experience leave me underwhelmed.<p>Does anyone know of any good, reputable and well designed Domain Name Provider? (worthy of HN!) <comment> tsetse-fly <http://news.ycombinator.com/item?id=186369>

<http://news.ycombinator.com/item?id=377301>

<http://news.ycombinator.com/item?id=150561>

<http://news.ycombinator.com/item?id=339823>

<http://news.ycombinator.com/item?id=436364>

</comment><comment><USER2> I always start at <http://instantdomainsearch.com/> ... very handy for quickly checking availability.</comment>


"Bootstrap or Die - Lessons Learned From a Web Startup's Murder/Suicide - <AUTHOR> http://bootstrappy.blogspot.com/2010/02/bootstrap-or-die-lessons-learned-from.html

<comment><USER2> Is anybody else frequently bothered by this type of voice in blogs? If you are going to go out of your way to research all these points and write ten thousand words about them, please don't finish it off by telling me I might be ass-fucked. It's not a "firm slap in the face," it's a simple vulgarity that makes you appear incapable of using expressive language. It seems inappropriate for the type of blog the author seems to be trying to write. Most confusing moment: using asterisks in ass-fucking. You either think it's appropriate to use or you don't, don't use it and simultaneously censor yourself.

<comment><AUTHOR> Thanks for the comment and sorry if my writing style's not to your liking. My post was not written to hurt anyone, quite the opposite. I have my own writing style, yes. It's not for everyone, yes. But you don't need to come here and try to organize a mob against, to try to force me to conform to your joy- sucking rules.

It's your choice and your right not to accept people who are different from yourself. God knows half of America is full of rabid, Jesus-loving attack dogs who try to force their beliefs on the rest of us. If you don't mind, however, could you please try to suffer in silence and keep your crying, bitching, and trolling to yourself. Thanks :)

BTW, if you don't like words like "ass" and "fuck" and all the rest of those naughty words that cause you cry out and try to rally a mob for a witch hunt, you might want to avoid blogs such as "500 Hats" by Dave McClure. Here's Dave's bio: <http://500hats.typepad.com/500blogs/about-dave-mcclure.html> here's one of his posts: [http://500hats.typepad.com/500blogs/2006/11/why_you_want_yo....](http://500hats.typepad.com/500blogs/2006/11/why_you_want_yo.html) and here's every post he's written with the words "ass" or "fuck" in it: [http://www.google.com/search?hl=en&source=hp&q=ass+o...](http://www.google.com/search?hl=en&source=hp&q=ass+or+fuck+site%3A500hats.typepad.com&aq=f&aqi=&oq=) Can you still see, or did all those naughty words blind you?

Better stay away from TechCrunch, too. That Arrington has a mouth on him! Of course, it's nothing like that Zed Shaw. Hehe, I love those guys. You know, what really burns me up, though, is little people who use words like "shit" and "fuck." Who do they think they are! We gotta teach them a lesson, don't we. We gotta stomp them back down where they belong. After all, only "important" people are allowed to use "bad" words.

Oh shit, better not read any William Shakespeare, either: [http://socyberty.com/education/evolution-of-profanity-how- sw...](http://socyberty.com/education/evolution-of-profanity-how-swear-words- came-to-be/)

Damn, or watch award-winning movies, such as "Pulp Fiction," "Mystic River," "The Departed," etc. Or any of those award-winning HBO shows, such as "Sopranos," "Deadwood," "Entourage," "The Wire," etc.

Fuck, fuck, fuck, what's left for you?! Well, at least you've still got your Fox news, church, and tea party friends, right? Speaking of which, where you at the rally? I heard Palin's speech was great! She's so smart and wholesome and funny and right. I love her! She's got my vote in 2012, I've had about all the "hopey changey" stuff I can handle outta that Obama guy!

Do you now see how ridiculous your attack was, Brett? And do your employers and university (Amazon.com, Trilogy, LiquidPlanner, and Yale, if this is you: <http://www.linkedin.com/in/brettbender>) know you're harassing people on Hacker News when you should be working and making them look good? In both cases, probably not, but I'm not going to argue. Life's too short to waste time with miserable people. Good luck. Seriously.

Cheers!

PS: I took out the asterisks and did some polishing and now the post is making the rounds all over again. Thks.

<comment><AUTHOR> Another interesting and, at least somewhat, related article. This time it's from the Washington Post (14 Feb. 2010): "The case against banning the word 'retard'" [http://www.washingtonpost.com/wp- dyn/content/article/2010/02...](http://www.washingtonpost.com/wp- dyn/content/article/2010/02/11/AR2010021103896.html)

</comment></comment></comment><comment><AUTHOR> Here's the premise: Taking that VC money is a root cause of most/all of the "Deadly Mistakes" Rob and Josh made with EventVue (read their post-mortem here: <http://blog.eventvue.com/post/372936164/post-mortem> and the TechCrunch article here: <http://www.techcrunch.com/2010/02/05/eventvue-deadpool/>). There are some updates and clarifications there, too, based on comments from Josh of EventVue. Thanks, Josh.

Whatever you think about that premise, hopefully you'll agree that it's not a bad idea to stop, think, and discuss _before_ you make an irreversible, and potentially fatal, decision.

<comment><USER3> And the weakness of the premise can be seen in "Update: Josh of EventVue clarified that they raised angel money, not VC money."

<comment><AUTHOR> Thanks for the comment, but it's a red herring (<http://www.fallacyfiles.org/redherrf.html>). Whether the money comes from an angel or a VC or a bank or China or your dad, you still need to stop, think, and discuss before you do choose to take outside financing, under what terms, from whom, when, and why (three times). That's the point :)</comment></comment></comment>


Mental health survey for people in startups - <AUTHOR> https://jamesroutledge.typeform.com/to/K4lCc9 <comment><AUTHOR> Hi Guys - I'm looking into mental health issues in startups.

It only takes 2 minutes

If nothing else, a great place to write down your thoughts

Thanks</comment>


Snowden touches down in Moscow ... tweeted by Wikileaks - <AUTHOR> https://twitter.com/wikileaks/statuses/348792474833133570

<comment><USER2> Everyone seems to be going very public with his travel information. Either it's a way of making it harder for something "unfortunate" to happen to his (ie, for his safety) or this could be misinformation.

How likely is it that this is misinformation?

<comment><USER3> Given that he's been so incredibly honest and open so far, I doubt it's misinformation or misdirection. If he did try to mislead regarding his travel plans, no matter the reason or strategy, it could undermine his credibility in terms of other things he says. He seems intentionally very candid.

We'll see though.

</comment></comment><comment><USER4> This is going to be a movie someday, just hope it doesn't end with tragedy.

<comment><USER5> It already has ended with a tragedy. Our privacy rights and 4th amendment protections have been thrown out the window. The making of Snowden into a Kardashian spectacle is to distract us from this fact.

<comment><USER6> If anyone turned it into a Kardashian spectacle it was Snowden himself. His video interview and Q&A have only served to publicize himself. He could have easily and safely leaked incriminating documents to a journalist and achieved the same effect.

<comment><USER7> Also, by making his identity public, he made sure that "Booz Employee dies in car crash" does not happen, its hard to make a public figure "disappear" without a fair trial.

<comment><USER8> Snopes agrees with you, the 34 people listed by them on the following page who were associated with either Bill or Hilary Clinton all died of explainable natural causes, suicides or natural deaths during the 90s, mostly 1992 to 1994:

[http://www.snopes.com/politics/clintons/bodycount.asp](http://www.snopes.com/politics/clintons/bodycount.asp)

<comment><USER9> Please don't drag that tired old conspiracy theory into this. If you start with just 500 people and kill them off at ~0.8%/year (roughly the annual US death rate), at the end of ten years close to 40 of them should be dead, and there are a lot more than 500 people "associated" with the Clintons.

</comment></comment></comment></comment></comment></comment><comment><USER10> Any ideas on why he have chosen Venezuela over Russia? Is russian government more likely to trade him for something with the USA? Or maybe he just was unable to obtain asylum in Russia on reasonable terms(e.g. russian government demanded him to disclose other classified information)

<comment><USER11> It isn't clear yet if Snowden has chosen Venezuela. He might be choosing Ecuador.

</comment></comment><comment><USER12> A few reports have been speculating that his next destination will be Cuba, for obvious reasons.

edit: Per reports, Cuba will be a layover before Venezuela, his final destination.

<comment><USER11> According to some sources, his final destination could be Ecuador.

</comment></comment><comment><USER13> Maybe a silly question, but is there any reason he couldn't have flown to Moscow in the first place?

<comment><AUTHOR> Every decision Snowden has made has been very well thought out....

1\. He pushed US media (WaPo) to publish by going to UK media (TheGuardian).

2\. He released with reservation and regard to security, not all docs were published.

3\. He voluntarily disclosed his identity for protection...if the public knows him, it'll be awkward, if he just disappeared.

4\. [my opinon] He went to Hong Kong (specifically HK, not China), because it is still somewhat it's own country, yet a formidable adversary to the US through China's protection. Also, it's unbeknownst to us, if he released more information while in HK.

5\. Russia recently had captured a CIA agent (although I thought it more comical then serious), and no extradition to the US. Again, another formidable adversary to the US, and it may have been a red flag for an intel agent to go to Russia.

By formidable, I imply that the US will not send a seal team in to these territories that have the military capability to strike back (I won't say the name - but Pakistan is what I'm implying here).

<comment><USER14> Uh, someone should learn the difference between "clandestine" and "covert".

The USA is most definitely capable of undertaking operations in "denied areas."

edit: typo.

<comment><USER15> No way to know this today, right? Any pointers to cladestine operations in russia since 1990 available? Those could help to debunk the claims that certain kind of those "cladestine every think is possible" ops are just marketing.

<comment><USER16> [http://www.dailymail.co.uk/news/article-2324303/Russia- seize...](http://www.dailymail.co.uk/news/article-2324303/Russia-seizes-CIA- agent-Moscow-US-diplomat-arrested-suspicion-spying.html)

</comment></comment></comment></comment></comment><comment><USER17> He sold me on Hong Kong and now he's leaving. This doesn't look good for our protagonist!

<comment><USER18> Why? Hong Kong still appears to have been a brilliant move. It's one of the few places he could actually travel to pre-whistle-blowing and it has apparently allowed him to travel to a countries that are even less likely to extradite him.

</comment></comment><comment><USER9> BBC is reporting Russian media as claiming a "diplomatic car" met Snowden's plane on the tarmac and he debarked directly into it. They're speculating he might be spending the night in the Venezuelan or Ecuadorian embassy.

[http://www.bbc.co.uk/news/world- europe-23022926](http://www.bbc.co.uk/news/world-europe-23022926) (in video, not mentioned in text at this time)

</comment><comment><USER19> Okay now I've lost respect for him. I appreciate that he told us what the gov was secretly doing, but to now go to the enemy? Come on, that's treason.

<comment><USER20> Who's enemy?

I have no enemies. My government says that I do though.

<comment><USER21> If you work in infosec or even just maintain a popular website, 100% of your security problems come from either Russia or China.

<comment><AUTHOR> Very true! Seeing as you are a hacker like I am, it's not 100% accurate to say we are where our ip addresses are.

<comment><USER21> Hmm? I am not a hacker. I design t-shirts.</comment></comment></comment></comment></comment>


The Secrets of Cloth Simulation in Alan Wake - <AUTHOR> http://www.gamasutra.com/view/feature/132771/the_secrets_of_cloth_simulation_in_.php <comment><USER2> Is this why so many games have some muscle-bound guy/gal in near skin-tight clothing be the lead? I never knew cloth simulation was such a difficult problem. I remember Lucas talking about this stuff when he was making the prequels, but I assumed everything has been long solved by now.

That said, I was pleasantly surprised in GTAV that when your character gets wet, his clothing and hair look wet and stay wet for a bit. I don't think I've seen that before.

<comment><USER3> Wet clothes in GTA V is actually a shader effect rather than a geometry effect: It's just adding a specular shader to the normally diffusion-shaded surface so it looks like your suit, hoodie, or wife beater is soaked with water. I'm pretty sure the geometry stays the same.

<comment><USER4> I don't know about games, but in movies, at least, a common trick to make simulations look wet is to simply increase the mass of the wet object. This causes hair and clothing to drape.

</comment></comment></comment><comment><USER5> Conspicuously missing (or I'm reading too fast), footage of the actual results :P

[https://www.youtube.com/watch?v=s40yXGCcorE](https://www.youtube.com/watch?v=s40yXGCcorE)

<comment><USER6> Easier-to-see daytime view in the video

[https://youtu.be/s40yXGCcorE?t=3m29s](https://youtu.be/s40yXGCcorE?t=3m29s)

<comment><USER7> The edges still make the whole thing look like polygons, which is a pity, since the edges should not take that much computational effort to get right.

</comment></comment></comment><comment><USER4> It is interesting that they refer to the simulation system as 'Verlet Physics' \-- that sort of constraint satisfaction system is called 'Position Based Dynamics' in most of the graphics lit. This type of system is also used in Maya's Nucleus and NVIDIA's PhysX. [1] is a nice overview.

[1] [http://matthias-mueller- fischer.ch/publications/STAR2013.pdf](http://matthias-mueller- fischer.ch/publications/STAR2013.pdf)

<comment><USER8> They're calling it Verlet physics because they're using a Verlet scheme for integration - which is a relatively elegant approach if you don't ever need velocities.

<comment><USER4> But it isn't really verlet, I don't think it will converge like a 'standard' verlet implementation with constraints (like molecular dynamics cats use) - the effective stiffness of the objects here is dependent on the time step.</comment></comment></comment>


Are you developer? Then play XSS game (by Google), find bugs, and get 7500$  - <AUTHOR> http://techmafia.net/google/2014/05/31/are-you-developer-google-gives-you-a-chance-to-play-xss-game-find-bug-and-get-paid-7500.html

<comment><AUTHOR> Please, give more points to my post, in order more developers will know about this interesting game idea created by Google</comment>


Panopticlick – How Unique, and Trackable, Is Your Browser? - <AUTHOR> https://panopticlick.eff.org

<comment><USER2> Previous discussions:

[http://news.ycombinator.com/item?id=6980085](http://news.ycombinator.com/item?id=6980085)

[http://news.ycombinator.com/item?id=1082464](http://news.ycombinator.com/item?id=1082464)

[http://news.ycombinator.com/item?id=1081309](http://news.ycombinator.com/item?id=1081309)

[http://news.ycombinator.com/item?id=6188543](http://news.ycombinator.com/item?id=6188543)

[http://news.ycombinator.com/item?id=1087975](http://news.ycombinator.com/item?id=1087975)

</comment><comment><USER3> Firefox 28 restricts the enumeration of navigator.plugins[] to reduce the impact of plugin fingerprinting. A website can directly query navigator.plugins["Some Plugin Name"] to check for a plugin, but no website should need to enumerate all installed plugins just to look for one particular plugin. Unfortunately, many major websites do, so enumerating Firefox's navigator.plugins[] will reveal QuickTime, Adobe Flash Player, Adobe Shockwave Player, and Java to avoid breaking those websites.

[https://bugzilla.mozilla.org/show_bug.cgi?id=757726](https://bugzilla.mozilla.org/show_bug.cgi?id=757726)

</comment><comment><USER4> Forgive me if I'm wrong, but it looks like if you could install a system font on a computer then you could create a unique fingerprint for that computer that is detectable by any website?

I am uniquely identifiable out of the 3.7 million samples because of my system fonts.

<comment><USER5> Also the funny thing is you cannot be tracked by this, because whenever you install a new font, update a plugin, install a plugin etc, you change your unique data.

<comment><USER6> I'm not sure how your conclusion follows from your premise. You are still trackable until you install a new font, update a plugin, install a plugin, etc. This may not happen for some time.

Even when you do make a change, you could still easily be tracked in many cases. If I see a new signature that I have never seen before that differs from an existing signature only by the version of a plugin, I can probably safely assume it's the same person, especially if I see that the plugin was updated between the last time I saw the existing signature and now.

I have a feeling that web developers are extra vulnerable to this type of tracking because we tend to install several useful developer extensions, and many of us have our own unique combination of extensions.

<comment><USER7> > Even when you do make a change . . .

On top of that, if you use a resource that had only previously been used by your previous fingerprint, your identity can probably be smeared that way too. This is only measuring client-side entropy, but there is also server-side entropy that can be used to make inferences about clients.

</comment></comment></comment></comment><comment><USER8> I'd like to have more control over the Javascript runtime in the browser. Defeating this identification trick is only one of the reasons.

Suppose you had a list of options and could selectively disable, for example, monitoring of mouse movements on one site, or ajax on another. And for this in particular, something that would feed the site random values from a particular range for fonts installed, plugins installed, screen size and other such information.

Using that data in development would still work because 99% would keep the default "true" values, and the few geeks who would change them would get what they should/would expect on sites that rely on those values. But everyone should have the power to control what info they're giving out, and what Javascript is allowed to do on their own device.

<comment><USER9> They didn't use any JavaScript to get this information. Most of it is sent by your browser in the request headers, and the font detection used Flash and Java. They could have used JavaScript to detect fonts as a fall-back when Java and Flash are disabled, but it's relatively complicated to do so (requiring you to know the rendered width of a string for each font you're trying to detect), and it was not included in this example.

Sending incorrect information for Java or Flash fonts is an interesting idea, and likely would not affect user experience, as non-standard fonts are often served with the animations. Sending the wrong screen size might get you a mobile site served when you were wanting non-mobile or vice-versa. IP address and ISP are valuable bits of identifying information as well, and those are more difficult to address without using a proxy. But I would bet that randomizing your screen size for each request would break most fingerprinting code, since that would be assumed to be static.

<comment><USER10> "They didn't use any JavaScript to get this information."

The list of installed plugins is retrieved via JS: window.navigator.plugins.

Not sure about fonts, though.

<comment><USER9> Ah, good catch. I guess the JS navigator.plugins list is quite a bit more thorough than the plugin information sent in your request headers (which I believe is limited to Java: yes/no).

</comment></comment></comment></comment><comment><USER11> The concept of a browser fingerprint being "unique" implies "worrying" in the sense that a user could then be tracked by their browser fingerprint.

But simply installing a font, changing screen resolutions, upgrading Java or Flash ("Browser Plugin details") or entering/leaving daylight savings time will result in the fingerprint changing.

So the browser fingerprint, as presented, isn't really a great way for websites to track users.

(And removing the aspects of the fingerprint subject to change, such as resolution and Browser Plugins etc., would then result in the fingerprint being less unique.)

<comment><USER12> How often do people on TFT displays change their screen resolution? I don't have either Java or Flash installed and I'm still unique.

<comment><USER13> The fact that java and flash are not installed makes you quite unique by itself.

<comment><USER12> Well, that's the default state in which Macs are shipped :) So I'm surprised that I'm unique to begin with.

</comment></comment></comment></comment><comment><USER14> If anyone is wondering how to disable the font list: [http://superuser.com/questions/292666/how-to-disable- permiss...](http://superuser.com/questions/292666/how-to-disable-permission- to-read-system-fonts-and-browser-plugin-details-in)

<comment><USER15> thanks!

</comment></comment><comment><USER16> What are the browser makers doing to reduce the number of identifiable bits of data leaked?

Most of the bits of identifiable information listed there has a reason to leak since 99.999% of front-end developers have no reason to need that information to create an acceptable cross-browser experience for all users.

</comment><comment><USER17> And whenever you wish to test if your browser can be "hard-tracked" (not just by probability), you can use this tool: [http://www.canyoutrackme.com/](http://www.canyoutrackme.com/)

</comment><comment><USER18> Mine is unique too, I checked Firefox, chrome and IE. I guess it is trivial for large companies to generate unique ID numbers for these unique fingerprints and crosscheck against cookie/login databases to extract e-identity. Is there an easy way from stopping browsers to broadcast this information? Using the Internet anonymously is really hard these days...

EDIT: Maybe it is even better for browsers to broadcast the most common settings if EFF discloses this information.

</comment><comment><USER10> I don't know how useful is this to track unique visitors, the next month I will probably have a different fingerprint.

All it takes is a new release of Firefox (different version in the User-Agent string) a new font or any plugin update.

So with that fingerprint you can possibly identify me now, but you cannot track me over time.

<comment><USER9> You can account for those factors by parsing only certain bits of the User Agent string, and allowing for the addition of fonts to the list (most typically don't uninstall fonts). With the plugins, you can ignore the version number and just go by the names. There are bits like browser name, OS name, screen resolution and the presence of all previously detected fonts and plugin names that you can be pretty sure won't change for most users. As long as you can uniquely match by certain factors, it'll be enough to link you to your previous session.

For a purpose like ad tracking, the period of time you need to track people is likely pretty short, as in from when they click on a banner or text link until they complete a purchase, so you can compare lots of data points to identify them. If you need to track for longer periods, like to retarget an ad to people who have completed purchases for x, then you would need to compare fewer, more stable points and hope you find a unique match.

</comment></comment><comment><USER19> I'm somewhat surprised to see that Chrome on my Nexus 10 shows up as unique, based primarily on user agent and screen size/color depth. Both seem to be surprisingly less common than I'd expect inasmuch as they would seem to be identical across all such devices.

</comment><comment><USER20> Does anyone know any good Firefox/Chrome plugins that help make your browser less unique?

<comment><USER21> Well I am using my own extension, with which you can block selectively cookies/javascript (among other stuff) and...

With cookies/js/plugins enabled I get: 1 in 3,719,197.

With cookies/js/plugins disabled I get: 1 in less than 160,100

Extension: [https://github.com/gorhill/httpswitchboard](https://github.com/gorhill/httpswitchboard)

EDIT: redid the tests with clearing cache before.

</comment></comment><comment><USER22> Browser plugin that randomizes the user agent and other details with each request?

<comment><USER23> Great for destroying user experience that uses these for valid reasons.

Use throw away virtual machines if you must, otherwise there is no easy workaround

</comment></comment><comment><USER24> Interestingly, I just tired this using Vidalia (the Tor browser).

> Your browser fingerprint appears to be unique among the 3,726,837 tested so > far.

I may be using an outdated version of Tor. Did they reset their data at some point? I can't believe I'm the first person to have tried the test using the Tor browser in my time zone.

<comment><USER25> Is your NoScript activated on the eff domain? If it isn't, that statistic is highly possible.

<comment><USER24> It's the Tor browser, so I haven't installed any extensions on it (the whole point of the Tor browser, as I understand it, is to present the same fingerprint to all browsers).

<comment><USER25> That is not the point of the Tor browser, though it is at times a goal.

That goal is absolutely impossible if you don't use NoScript, though. Tor browser includes NoScript by default, as well as many other extensions, but NoScript is initially set in "globally allow" mode which means it won't block any JS or Flash.

</comment></comment></comment></comment><comment><USER26> Why does it need to run a java applet to do this?

<comment><USER27> It uses flash and Java to try to access the system font list.

<comment><USER26> Ah ok, it is just getting all of the info that a website could potentially get from your browser.

</comment></comment></comment><comment><USER28> Why does it need java?

<comment><USER29> it does not. But if your system allows it to run, it will get more information.

</comment></comment><comment><USER7> iPad == anonymous.

</comment><comment><USER30> will they share the 4 million browser stats?</comment>


CGo-free SQLite Go database/SQL driver for Linux/amd64 v1.4.0-beta1 is released - <AUTHOR> https://gitlab.com/cznic/sqlite <comment><USER2> Is it go implementation of sqlite3 data format?

<comment><AUTHOR> Yes it is. The driver uses the original sqlite3.c code mechanically translated to Go.</comment></comment>