I Don't Need No Stinking API: Web Scraping For Fun and Profit - <AUTHOR> http://blog.hartleybrody.com/web-scraping/ <comment><USER3> I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques, you're driving a real browser and your requests look real (if you make 10000 requests to index.php and never pull down a single image, you might look a bit suspicious). There's a bit more overhead, but micro instances on EC2 can easily run 2 or 3 Selenium sessions at the same time, and at 0.3 cents per hour for spot instances, you can have 200-300 browsers going for 30-50 cents/hour.

<comment><USER4> In regards to the javascript problem. I'd suggest checking out the mobile versions of the sites first before you hop to a weighty solution like Selenium. Could be a _very_ simple solution to the problem :)

I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an update on my phone when it dawned on me that the mobile.twitter site is, for the most part, simple html stuff. Once I realized that, I was able to programatically log into my account with no problems, and the rest of twitter was unlocked for me. I could scrape and post to (almost) my heart content.

However, there were a few very big problems. Which makes me feel that scraping is not the way to go about things. I certainly wouldn't build a service based around scraping a particular site's data.

When I had my twitter bot operational, I would get blocked from twitter for hours at a time. It seems anytime I hit their servers too hard, or crossed some threshold, I would be locked out. I'm assuming it was some kind of IP level ban, because I wasn't even able to access the site from an actual browser.

I was able to deal with the setback by setting up a script to repeatedly check its access the site, and then relaunch the scraper upon discovering access, but the solution was just a band-aid. That would translate to significant downtime if I was running a service with counted on access to their data. The ban-hammer is too easily laid down.

Finally, just as a word of caution, I'd warn prospective scrapers to be careful of just who you scrape. I've inadvertently "DDoS'd" a site when a multiprocessed script got away from me. It spawned 1000+ instances of this particular request, all of which were doing their best to beat the bejesus out of this small websites servers. The site ended up going down for a couple of hours; I assume because of a bandwidth cap or something.

So, my point being, scraping is cool, but (1) I'm unsure if I agree with relying on it over a proper API, and (2) with great power comes great responsibility! Be nice to smaller guys, and don't punish their servers to bad.

<comment><USER5> For my twitter bot I extracted the xauth keys from twitter's official Mac client(s) (Tweetie 1 and 2 have different keys) and used those to access the API. To twitter the bot looked like the official client and they couldn't ban it without banning their official clients.

And XAuth made account creation and log in a breeze as there was no need for OAuth tokens - username/password was enough.

But you're not always as lucky as that and many websites are heavily JS driven. For Reddit I had to resort to selenium.

<comment><USER6> Reddit JSON api (just add .json to any URL) is not good enough for you ?

<comment><USER7> Another proof that spending 15 minutes on research can save you days in development and production.

<comment><USER5> And your comment is another proof that people tend to assume everyone else is an idiot.

<comment><USER8> I think his comment was quite appropriate, and did not feel there was an implication that he thought anyone was dumb for not having known about the alternative approach to getting Reddit data.

Often times programmers and the managers that drive them are way too quick to get going building or solving something with brute force. If they would just be patient and stop for a moment. Spending even a mere 30 minutes extra doing your homework on a problem can save hours or days in dev time.

<comment><USER9> Saying that that was a proof of spending 15 minutes to search about Reddit API would save his time implied that kybernetyk didn't do that research.

But kybernetyk already said he did the research before and that Reddit's API is not good enough for his requirement.

So this is not the case that 15 minutes of research will save the time. And his comment meant he assumed kybernetyk didn't do research, i.e. being dumb for not doing search.

<comment><USER7> Let me tell you what I thought when I wrote it...

I did not assume that kybernetyk was dumb or anything, I simply chuckled and thought to myself ouch haven't I done similar mistake before?! Please don't assume the worst when reading someone's comment.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> (shameless plug) I can scrape asynchronously, anonymously, with JS wizardry, and feed it into your defined models in your MVC (e.g. Django). But! I need to get to a hacker conference on the other side of the world (29c3). Any other time of year, I'd just drop a tutorial. See profile if you'd like to help me with a consulting gig.

EDIT: Knowledge isn't zero-sum. Here's an overview of a kick-ass way to spider/scrape:

I use Scrapy to spider asynchronously. When I define the crawler bot as an object, if the site contains complicated stuff (stateful forms or javascript) I usually create methods that involve importing either Mechanize or QtWebKit. Xpath selectors are also useful for the ability to not have to specify the entire XML tree from trunk to leaf. I then import pre-existing Django models from a site I want the data to go into and write to the DB. At this point you usually have to convert some types.

I find Scrapy cleaner and more like a pipeline so it seems to produce less 'side effect kludge' than other scraping methods (if anybody has seen a complex Beautiful Soup + Mechanize scraper you know what I mean by 'side effect kludge'). It can also act as a server to return json.

Being asynchronous, you can do crazy req/s.

I will leave out how to do all this through Tor because I don't want the Tor network being abused but am happy to talk about it one on one if your interest is beyond spamming the web.

Through this + a couple of unmentioned tricks, it's possible to get _insane_ data, so much so it crosses over into security research & could be used for pen-testing.

</comment><comment><USER11> And this is why we can't have nice things.

Web scraping, as fun as it is (and btw, this title _again_ abuses "Fun and Profit"), is not a practice we should encourage. Yes, it's the sort of dirty practice many people do, at one point or another, but it shouldn't be glorified.

<comment><USER12> So you're not so hot on the whole search engine thing?

The article does slide into the sketchy side (I've always wanted an excuse to do that client side javascript trick too) but I found it more interesting because of that, these aren't secrets. Maybe if I put my "won't somebody please think of the children" hat on I agree that glorifying using trojan code to potentially ddos someones server to get around rate limits desired by the owners of the server is bad. Adults and especially adult self described hackers should be able to read this without mock outrage, it's interesting and it's happening all the time.

You can't condemn web scraping though, that's the backbone of the services we all depend on for most internet related things. That's the whole point of structured markup and the world wide web itself.

<comment><USER13> > So you're not so hot on the whole search engine thing?

They scrape to generate links for _users_ to _go to the site_. That's quite different than scraping for...any other purpose? So it seems. Would you (anyone) argue otherwise? (genuine curiosity).

<comment><USER14> They are also using title, description, some snippets from the page and taking a cached version of the site and images you can view without having to visit the site itself. They are also using this data as a product to sell advertising against.

If there wasn't so much benefit for most of all sites to be in search engine indexes you would thinking at least some would object to this scraping.

There would be lots of other scraping that websites want to prevent that takes even less data than this. It just doesn't provide much in return for the website.

<comment><USER15> Google is even moving into the territory of scraping content to display. Relevant wikipedia snippets are now being displayed on the search page as a side bar. While Wiki probably doesn't care...there are plenty of other sites that would not like Google to scrape the content and display it on the search page.

<comment><USER14> Yeah, Wikipedia is creative commons so that should be okay? You are right though I wonder if they have the rights to sports results and weather that they are pulling.

They have even convinced us all to go mark up our page to help them pull stuff like ratings and reviews out.

<comment><USER16> Sports results are facts and are statutorily not subject to copyright in the US.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> There are some recent federal cases (Weev [http://www.wired.com/opinion/2012/11/att-ipad-hacker-when- em...](http://www.wired.com/opinion/2012/11/att-ipad-hacker-when-embarassment- becomes-a-crime/), Aaron Swartz[http://www.wired.com/threatlevel/2012/09/aaron-swartz- felony...](http://www.wired.com/threatlevel/2012/09/aaron-swartz-felony/), and a prosecution of scalpers <http://www.wired.com/threatlevel/2010/07/ticketmaster/>) that view scraping as a felony hacking offense. The feds think that an attempt to evade CAPTCHAS, IP and MAC blocks is a felony worthy of years in prison.

In fact, the feds might think that clearing your cookies or switching browsers to get another 10 free articles from the NYTimes is also felony hacking.

Which is to say, be careful what you admit to in this forum AND how you characterize what you are doing in your private conversations and e-mails.

Weev now faces a decade or more in prison because he drummed up publicity by sending emails to journalists that used the verb "stole".

<comment><AUTHOR> Very good point, I've added the following disclaimer:

      While scraping can sometimes be used as a legitimate way to
      access all kinds of data on the internet, it's also important
      to consider the legal implications. As was pointed out in the
      comments on HN[1], there are many cases where scraping data
      may be considered illegal, or open you to the possibility of
      being sued. Similar to using a firearm, some uses of web
      scraping techniques can be used for utility or sport, while
      others can land you in jail. I am not a lawyer, but you
      should be smart about how you use it.

[1]: Linking to this (parent) comment

<comment><USER17> Thanks for adding that and linking. The feds are nutter butter these days.

</comment></comment></comment><comment><USER18> From the article:

       Since the third party service conducted rate-limiting based on IP
       address (stated in their docs), my solution was to put the code that
       hit their service into some client-side Javascript, and then send
       the results back to my server from each of the clients.

       This way, the requests would appear to come from thousands of
       different places, since each client would presumably have their own
       unique IP address, and none of them would individually be going over
       the rate limit.

Pretty sure the browser Same Original Policy forbids this. Think about it- if this worked, you'd be able to scrape inside corporate firewalls simply by having users visit your website from behind the firewall.

<comment><USER19> > Since the third party service conducted rate-limiting based on IP

By the way, that's one of my projects. You can use a basic fibonacci-related algorithm to figure out (in the most minimal number of requests) what exactly the rate limit is. This way, you can scrape at just under the maximum limit. I am still working on this core library though. :|

<comment><AUTHOR> Sounds pretty interesting! Be sure to share it when it's ready.

</comment></comment></comment><comment><USER20> The issue with web scraping is that it relies on the scraper to keep up with changes made to the site.

If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.

<comment><USER21> This should be stressed - sites like Facebook do exactly this. Constant changes mean constantly updating your scraper. When it comes to A/B testing? Your scraper needs to intelligent find the data, which might not always be in the same place.

Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.

<comment><USER22> _I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping._

In my spare time, I've been playing around with "scrapers" (I like to call them web browsers, personally) that don't even look at markup.

My first attempt used a short list of heuristics that proved to be eerily successful for what I was after. To the point I could throw random websites with similar content (discussion sites, like HN), but vastly dissimilar structures, at it and it would return what I expected about, I'd say, 70% of the time in my tests.

After that, I started introducing some machine learning in an attempt to replicate how I determine what blocks are meaningful. My quick prototype showed mix results, but worked well enough that I feel with some tweaking it could be quite powerful. Sadly, I've become busy with other things and haven't had time to revisit it.

With that, swapping variables and similar techniques to thwart crawlers seems like it would be easily circumvented.

<comment><USER12> I would be really interested in knowing which heuristics or machine learning techniques produced decent results. That's if I can't convince you to open source the code. I'm working on the same problem at the moment.

<comment><USER23> What about something like <http://> tubes.io

<comment><USER12> We're fine with scrapers and scraping infrastructure, although tubes.io is a very interesting idea.

I'm more interested in what I can do to write fewer scrapers since the content is, at a high level, relatively similar. I've just started with experiments writing "generic" scrapers that try and extract the data without depending on markup. It's going to eventually work well enough but to get the error rate down to an acceptable level is going to take a lot of tweaking and trial and error.

There's a few papers on this, but not much out there. That's why I was interested in someone else working on the same problem in a different space.

</comment></comment></comment></comment></comment></comment><comment><USER24> Great read!

In the past, I have successfully used HtmlUnit to fulfill my admittedly limited scraping needs.

It runs headless, but it has a virtual head designed to pretend it's a user visting a web application to be be tested for QA purposes. You just program it to go through the motions of a human visting a site to be tested (or scraped). E.g., click here, get some response. For each whatever in the response, click and aggregate the results in your output (to whatever granularity).

Alas, it's in Java. But, if you use JRuby, you can avoid most of the nastiness that implies. (You do need to _know_ Java, but at least you don't have to _write_ Java.)

Hartley, what is your recommended toolkit?

I note you mentioned the problem of dynamically generated content. You develop your plan of attack using the browser plus Chrome Inspector or Firebug. So far, so good. But what if you want to be headless? Then you need something that will generate a DOM as if presenting a real user interface but instead simply returns a reference to the DOM tree that you are free to scan and react to.

<comment><USER3> Headless: Xvfb on Linux. (Virtual Framebuffer, let's you run apps that require a GUI) You can use one of the many options that include Webkit (like phantom.js, the capybara-webkit gem, or Selenium if you want a real browser like Firefox to do the work)

<comment><USER25> PhantomJS doesn't need Xvfb anymore it can run headless without this dependency.

<comment><USER26> well, it wasn't due to anything in phantomjs actually -- it was because qt introduced project lighthouse & their qt platform abstraction. project lighthouse was a fork that got integrated into qt 4.8 that phantomjs includes. (you can see the entire qt source tree if you git-clone phantomjs)

</comment></comment></comment></comment><comment><USER27> I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :(

I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

<comment><USER28> If you are using Python, you can also use pyv8 to evaluate Javascript code.

<comment><USER19> Yes, but if you want the DOM you would have to use something like webkit. So something like pyphantomjs might hit the right spot. It's a python re- implementation of phantomjs.

<https://github.com/kanzure/pyphantomjs>

</comment></comment></comment><comment><USER3> Another issue not covered: file downloads. Let's say you have a process that creates a dynamic image, or logs in and downloads dynamic PDFs. Even Selenium can't handle this (the download dialog is an OS-level feature). At one point I was able to get Chrome to auto-download in Selenium, but had zero control over filename and where it was saving. I ended up using iMacros (the pay version) to drive this (using Windows instances: their Linux version is very immature comparably).

<comment><USER29> I've done this successfully with Ruby Mechanize.

<comment><USER3> Awesome. I'd love some hints or links, as I'm always looking to refactor.

<comment><USER4> In general, if you're going the mechanize route, _.retrieve()_ is the function your looking for.

e.g.

      br = mechanize.Browser()
      br.retrieve("https://www.google.com/images/srpr/logo3w.png, google_logo.png)[0]

Mechanize doesn't really have a proper doc, but just about everything you'd need could be figured out from the very lengthy examples page on their site.

<comment><USER3> Playing with it now, and while it seems to hit my download need, I can't seem to get it to play nice with sites that are JavaScript dependent. Am I missing something, or is there a way to plugin an underlying WebKit engine?

<comment><USER30> PhantomJS is capable of downloading binary content from js dependent sites but it is a journey to get it working as it is not an out-of-the-box feature. Instead use CasperJS to drive Phantom and get a ton of snazzy features including simple binary downloads. Happy scraping!

</comment></comment></comment></comment></comment></comment><comment><USER31> I'm surprised that no one has attempted to write a Twitter client based solely on scraping to get around the token limits.

<comment><USER32> Or an alternative API that uses the scraped data from Twitter to make requests... but that might be getting a bit ambitious (and legally dodgy)

<comment><USER33> Create an script that scraps proxies and then use those to scrap twitter, use it in a Russian domain, claim 140 characters can't be copyrighted, claim that the tweets are being extracted from third party sites that use the twitter API but lack any kind of TOS and disclaimer; sell API access, profit!

</comment></comment></comment><comment><USER34> Scraping could be made a lot harder by website publishers, but they all depend on the biggest scraper accessing their content so it can bring traffic: Google ...

The biggest downside of scraping is that it often takes a long time for very little content (e.g. scraping online stores with extremely bloated HTML and 10-25 products/per page).

<comment><USER35> As a pioneer of scraping (NetProphet, the first interactive stock charting app with push-data) we initially scraped every quote we had in our database from other sites.

The fundamental problem is, web pages can change a lot. We constantly had scraper scripts fail either because the web pages changed for some innocuous reason, or they noticed the scraping and blocked us.

We resorted to a list of scrape targets and constantly-updating scrape-scripts to adapt continuously to the 'market'. We also pinged each target to find the least congested.

Eventually we got our own stock feed (guy that did that is a research scientist at Adobe now) and stopped scraping altogether. But it was a wild ride.

<comment><USER34> We still need to scrape many (several 100) clients' websites because they are unable to give us product feeds (adequate ones or any at all) for their stores. But hey, it gives us a small edge because we try harder than the competition.

</comment></comment></comment><comment><USER36> An important topic.

The main caveat is that this may violate a site's terms of use and thus website owners may feel called upon to sue you. Depending on circumstances, the legal situation here can be a long story.

<comment><USER37> Yes, it is complicated. That said, this is partly just because there aren't enough cases - and partly because the law hasn't stabilised (took a century to stabilise after invention of printing press). It isn't clear what rights society should grant yet, for maximising business.

My take on it, from ScraperWiki's point of view: <http://blog.scraperwiki.com/2012/04/02/is-scraping-legal/>

</comment></comment><comment><USER38> Related: If you fancy writing scrapers for fun _and_ profit, ScraperWiki (a Liverpool, UK-based data startup) is currently hiring full-time data scientists. Check us out!

<http://scraperwiki.com/jobs/#swjob5>

<comment><AUTHOR> very well played :)

</comment></comment><comment><USER39> The title makes it sound as if there is going to be some discussion of how the OP has made web scraping profitable, but this seems to have been left to the reader's imagination.

Otherwise, great article! I agree that BeautifulSoup is a great tool for this.

</comment><comment><USER40> It's pointless to think of it as "wrong" for third-parties to web-scrape. Entities will do as they must to survive. The onus of mitigating web scraping, if in the interests of the publisher, is on the publisher.

As a startup developer, third-party scraping is something I need to be aware of, that I need to defend against if doing so suits my interests. A little bit of research shows that this is not impractical. Dynamic IP restrictions (or slowbanning), rudimentary data watermarking, caching of anonymous request output all mitigate this. Spot-checking popular content by running it through Google Search requires all of five minutes per week. At that point, the specific situation can be addressed holistically (a simple attribution license might make everyone happy). With enough research, one might consider hellbanning the offender (serving bogus content to requests satisfying some certain heuristic) as a deterrent. A legal pursuit with its cost would likely be a last resort.

Accept the possibility of being scraped and prepare accordingly.

</comment><comment><USER41> People seem to wonder how to handle ajax.

The answer is HttpFox. It records all http-requests.

1\. Start recording

2\. Do some action that causes data to be fetched

3\. Stop recording.

You will find the url, the returned data, and a nice table of get and post- variables.

<https://addons.mozilla.org/en-us/firefox/addon/httpfox/>

<comment><USER42> Isn't this the same as what the Net tab from Firebug does?

<comment><AUTHOR> Yah, I don't understand why people make things so complicated once Javascript gets involved. Just inspect the XHR traffic to your browser ("Network" tab in Web Inspector, Firebug, etc) as you update the information on the page. You'll quickly discover what are essentially undocumented APIs returning the data used to generate the page. You don't need to use or even read through the Javascript that's calling them, you just need to figure out what parameters and cookies are being sent, and tweak those as you wish.

You might have to spoof the Referer header so that it thinks the request is still coming from their website.

</comment></comment></comment><comment><USER43> From a site owner's perspective: if you have a LOT of data then scraping can be very disruptive. I've had someone scraping my site for literally months, using hundreds of different open proxies, plus multiple faked user-agents, in order to defeat scraping detection. At one point they were accessing my site over 300,000 times per day (3.5/sec), which exceeded the level of the next busiest (and welcome) agent... Googlebot. In total I estimate this person has made more than 30 million fetch attempts over the past few months. I eventually figured out a unique signature for their bot and blocked 95%+ of their attempts, but they still kept trying. I managed to find a contact for their network administrator and the constant door-knocking finally stopped today.

</comment><comment><USER44> when i need to scrap a webpage, i use phpQuery (<http://code.google.com/p/phpquery/>), it's dead simple if you have experience with jQuery and i get all the benefits of a server-side programming language.

<comment><USER45> A similar module for node.js: <https://github.com/mape/node-scraper>

<comment><USER46> Better than that is <http://node.io/> Also, don't use jsdom (it is slow and strict), <https://github.com/MatthewMueller/cheerio> is much better.

</comment></comment></comment><comment><USER47> What I wish I could do is capture Flash audio (or any audio) streams with my Mac. All I want is to listen to the audio-only content with an audio player when I'm out driving or jogging, etc. Audio-only content that has to be played off a web page usually runs into the contradiction that if I'm in a position to click buttons on my web browser (not driving, for example), I'm in a position to do my REAL work and have no time to listen to the audio. I'll go to the web page, see whatever ads they may have, but then I'd like to be able to "scrape" the audio stream into a file so I don't have to sit there staring at a static web page the whole time I'm listening.

<comment><USER48> I used to work at a company where capturing flash video and audio streams was a regular part of our work. You're not going to like the answer.

You basically have to proxy everything through a proxy that can be given a command or otherwise instructed to capture the top 3 or 4 streams from the website. From there you can either dumbly accept the largest one or start checking byte headers.

</comment></comment><comment> SG- When scraping HTML where data gets populated with js/ajax, you can get a web inspector to look at where that data is coming from and manually GET it and it will likely be in some nice JSON.

Scraping used to be the way to get data back in the days, but websites also didn't change their layout/structure on a weekly basis too back then and were much more static when it came to the structure.

Having recently written a small app that was forced to scrape HTML and having to update it every month to make it keep working, I can't imagine doing this for a larger project and maintaining it.

</comment><comment><USER49> To all HN: All this being said, how do we prevent our sites from being scraped in this way? What can you not get around, and what are the potential uses for an 'unscrapeable' site to your mind.

<comment><USER50> if you don't want something to be scrapped, don't publish it on the internet. scrapping prevention reminds me of blocking right-click and other ridiculous solutions back in the day. hey, if I can view it, it means the data reached my end point.

</comment></comment><comment><USER51> I think the author just completly missed the point with API vs Screen scraping. The API allows for accessing structured data. Even if the website changes once, the datas would be accessible the same way through the API. Whereas, the author, would have to rewrite his code each time an update his made to the front-office code of the website.

A simple API providing simple json response with http basic auth is far more efficient than a screen scraping program where you have to parse the response using HTML / XML parsers.

<comment><USER37> This isn't always the case - APIs often change. Facebook, for example, is (at least was, a few years ago) notoriously bad at changing in an unpredictable and buggy way, and I stopped using it for that reason. Some HTML scrapers are more reliable than that.

As for efficiency, again not such an issue. HTML is very good these days, compared to 10 years ago, a simple CSS selector often does the job.

<comment><USER51> This is true, but APIs are often versionned.

Concerning efficiency this is true CSS / XPath processors, at least, both offer very nice performances.

But download 70KB of HTML each time you only need a single data, where the API request cost only a few (avg < 2KB), can be such a pain if you need to do this frequently. This can be handled by a scalable configuration but I find it a bit the overkill.

</comment></comment></comment><comment><USER52> This illustrates the significant difference between the use-cases of "web APIs" and conventional APIs, that the former are more like a database CRUD (including REST), rather than a request for computation. They (usually) are an alternative interface to a website (a GUI), and that's how most websites are used. e.g. an API for HN would allow story/comment retrieval, voting, submission, commenting.

They _could_ be used for computation, but (mostly) aren't.

</comment><comment><USER53> Not every site. There is data I would really love to access on Facebook without having to gain specific authorization from the user. It's odd that for most user profiles the most you can extract via the graph API (with no access token) is their name and sex. Whereas I can visit their profile page in the browser, see all sorts of info and latest updates (and not even be friends with them)

Tried scraping Facebook. They have IP blocks and the like.

<comment><USER54> Do it in JS, client-side with 3 second delays. I have used this to get available data (location, name, status, etc) from the latest 500 available likes of a page I manage

</comment></comment><comment><USER55> This is a shameless plug but I've created a service that aims to help with a lot of the issues that OP describes such as rate limiting, JS and scaling. It's a bit like Heroku for web scraping and automation. It's still in beta but if anyone is interested then check out <http://tubes.io>.

</comment><comment><USER56> I have done a bit of scrapping with ruby mechanize, when we hit limits have circumvented by proxy and tor

google as a search engine crawls most all sites, but offers very few usable stuff to other bots

<http://www.google.com/robots.txt>

Disallow 247 Allow 41

</comment><comment><USER16> Be careful. I got banned from Google for scraping. I did a few hundred thousand searches one day, and that night, they banned my office IP address for a week. This was in 2001, so I estimate I cost them a few hundred dollars, which is now impossible to repay. :(

</comment><comment> clark-kent The problem with scraping instead of using the API is that when the website makes even a slight change to their markup it breaks your code. I have had that experience and it's a living hell. I can say it's not worth it to scrap when there is an API available.

</comment><comment><USER57> There is just one major trouble with not needing stinking API. You can not POST as a possible client without requiring them to give their password to you, which actually would give you full access to their account instead of limited access with API.

<comment><USER58> You seem to be talking about a specific site? Which one?

<comment><USER57> Any social network where you can post messages for the user in the users message stream.

</comment></comment></comment><comment><USER59> I had to do some scraping of a rather Javascript-heavy site last year - <USER2> found the entire process was made almost trivial using Ruby and Nokugiri. Particularly relevant for a non-uber-programmer like me, it's simple to use, as well as powerful.

</comment><comment><USER60> So bloody true. A web page is a resource just like an xml doc, there's no reason public facing urls and web content can't be treated as such and I regularly take advantage of that fact aswell. great post

</comment><comment><USER61> If it's not automated and a fewer times, I will prefer IMacro to perform tasks on my behalf. The best part of it that you can integrate a Db to record your desired data.

</comment><comment><USER62> Automated web testing tools, such as Watir and Selenium, are also pretty good options. I'm especially surprised Watir hasn't been mentioned yet in the comments.

<comment><USER63> Indeed - or WatiN, the .NET port of WatiR. I've done some pretty heavy duty scraping and automation with WatiN, which included building a OO framework that trivialized writing scripts. Good stuff.

</comment></comment><comment><USER64> Checkout <http://selectorgadget.com> as a useful tool for coming up with CSS selectors.

</comment><comment><USER65> How about publicly available web scraping tools as a way to encourage sites to provide good APIs? Everybody wants efficiency, after all.

</comment><comment><USER66> _No Rate-Limiting_

Clearly someone's never spent time diagnosing the fun that is scaping HN (yes, unofficial API is available).

</comment><comment><USER67> Node.js is excellent for web scaping, especially if you're scraping large amounts very often.

<comment><USER68> I made this module for this exact reason: <https://github.com/icodeforlove/node-requester>. Supports horrible things like proxy rotation.

<comment><USER19> > Supports horrible things like proxy rotation.

Do you have any plans to track which proxies are actually working, or how quickly each one is blocked? I want a reverse proxy on my outgoing requests that knows how to shift my traffic around properly so that I don't get banned. I don't want to be rate limited and I don't want to sit here for weeks trying to figure out wtf the rate limit is in the first place.

<comment><USER68> interesting, im sure you can build this into it. you could hook onto the didRequestFail method and flag IPs (log this.proxy out to see what the proxy was). all i would need to do is add a method that makes it easier to add/remove proxies.

</comment></comment></comment></comment><comment><USER69> What is it with all the headlines this week abusing the classic "for fun and profit" title?

</comment><comment><USER70> relevant: [http://www.codinghorror.com/blog/2009/02/rate-limiting- and-v...](http://www.codinghorror.com/blog/2009/02/rate-limiting-and-velocity- checking.html)

</comment><comment><USER71> I've found diffbot to be quite useful for scraping.

</comment><comment><USER72> I so not agree with that article, it makes me sick. And this guy basically is some "marketer" so no wonder he gets quite some stuff wrong, imo. :p

<comment><USER73> What did he get wrong?

</comment></comment><comment><USER74> Craigslist, anyone?

<comment><USER75> I actually wrote a CL crawler in Ruby - <https://github.com/marcamillion/craigslist-ruby-crawler>

I used it to crawl for freelance web dev gigs, but it can be re-purposed to do anything.</comment></comment>


Show HN: I made a free Admin template pack using React. What do you think? - <AUTHOR> https://github.com/DesignRevision/shards-dashboard-react <comment><USER2> Before I click that link, what does it solve?

What makes it different, better, than other admin template packs?</comment>


Asking 'Why' in Design – A Cautionary Tale - <AUTHOR> https://spin.atomicobject.com/2016/02/15/asking-why-resources/#.VsHfMZ82rXQ.hackernews <comment><USER2> >MS Project went wrong in neglecting to ask "why is someone exporting to CSV?"

>What's the deeper need/problem to be solved here? The user is trying to import the data into another application!

This is why Free Software is important: if Project was open source the user could have just have submitted a pull request or else started a fork.

Instead we have predictably bad behavior from one of the most notoriously user-hostile companies of all time.

Microsoft products are heavy-handed and unwieldy? Colour me surprised.</comment>


VR interested gamer form the US wanted for Interviews (Study project) - <AUTHOR> Hey together!
I´m Jenny from Germany and I´m searching for some US woman, man at all ages who love to play games by steam or on smartphone. You should be interested in a VR headsets. Maybe you have tried one, it doesn't matter which one, it can be a smartphone or a stationary VR headset.
It is a university-project where we should define a sales and marketing-strategy for VR-headsets. I would be very happy if some would help me and answer my questions.
Each who would answer my questions will get a personal link to their Question-sheet. For this please mail me: email@example.com
Thank you for your time and interest!
Best wishes from germany :) <comment><AUTHOR> Is there no one from the US?

Thanks for the only one mail from Australia!</comment>


Contact-tracing apps are not a solution to the Covid-19 crisis - <AUTHOR> https://www.brookings.edu/techstream/inaccurate-and-insecure-why-contact-tracing-apps-could-be-a-disaster/ <comment><USER2> Since "Some countries such as Singapore, South Korea and Israel have enlisted technology" , tell us how is it not working over there? There are real case studies. No need to speculate.

<comment><USER3> > Since "Some countries such as Singapore, South Korea and Israel have > enlisted technology"

Testing and tracing works only if you're not already overwhelmed with infected people.

The US received 100,000 airline passengers per week directly from China (including Wuhan and Shanghai) from Dec. 10 to Jan. 31, then a stream indirectly after that via Europe and Africa.

With millions of cases in the US and inadequate testing and tracing, technology isn't going to be a magic bullet.

(Also, I believe the CDC wasted 6 weeks developing their own test that didn't actually work. This is documented in detail in the Frontline episode about US Patient Zero in Seattle.)

This time around, the US lost to Mother Nature, and our only hope is herd immunity at this point.

<comment><USER4> > Testing and tracing works only if you're not already overwhelmed with > infected people.

Which is basically the normal mode of operation for institutions like the CDC or similar organizations in other countries like the RKI in Germany. In the past this was a very successful approach to keep really nasty diseases in check. I have read that both RKI and CDC had a difficult time to switch from contact tracing singular clusters into pandemic mode.

> This time around, the US lost to Mother Nature, and our only hope is herd > immunity at this point.

I wouldn't phrase it so negatively, there is a major pandemic every 100 years. Everybody is still doing far better than at any previous similar event. Even if things seem to devolve into chaos, that's nothing in comparison. Already the fact of omnipresent real-time news probably makes a huge difference. Even without optimal government interventions, people come up with their own precautions.

<comment><USER3> > I wouldn't phrase it so negatively

I said herd immunity in a positive way.

Since we don't have a vaccine, that's all we get this year.

<comment><USER4> If that would happen in any country, it would actually be quite bad because it would mean hospitals would be over capacity for a long period of time. Developing herd immunity in a controlled way, i.e. not DoS'ing hospital capacity, would take far longer.

</comment></comment></comment></comment></comment><comment><USER5> It was a solution in search of a government check.</comment>


Ask HN: Whats the best book on your favorite programming language or topic? - <AUTHOR> I've been enjoying Fluent Python and started looking for a nice book on SQLite, which made me wonder what great books there are in other areas that could help me be a more well-rounded developer. <comment><USER2> The Little Schemer is a mind bender and short read. I would highly recommend it to any programer as something you should read ASAP. You'll regret not reading it sooner if your unfamiliar with the concepts.

Structure and Interpretations of Computer Programs is very good. It was MITs old CS textbook, it's still highly relevant. It takes a while to get through and you probably would get the most value out of it if you already have a lot of programming experience.

I also recommend Programming in Haskell by Graham Hutton 2nd Edition (Don't get the 1st edition). It will warp you mind to think in a functional way. I would put it up there with K&R in terms of how well written it is.

I'm a student at a "Java school," reading these has made me far better than my peers at programming.

Paul Graham has a lot of good essays on programming too, as well as life stuff.

</comment><comment><USER3> The C Programming Language by Kerningham and Ritchie.

If you're interested in something closer to the hardware than Python, why not read about the language that implements Python?

It's a programming classic, very concise, easy to read and informative. It'll take you from basic control statements to eventually writing a memory allocator.

</comment><comment><USER4> Designing Data-Intensive Applications by Martin Kleppman. It's the other half of software engineering as far as I'm concerned.

<comment><USER5> I spent the summer reading this book cover to cover. (I am not a particularly fast reader.) I absolutely adored it. Now I am eager to put what I've learned into practice.

</comment></comment><comment><USER6> For learning Rust: "The Rust Programming Language".

Available for free online [1] and hard copy [2]

[https://doc.rust-lang.org/stable/book/](https://doc.rust- lang.org/stable/book/)

[https://www.amazon.com/Rust-Programming-Language- Covers-2018...](https://www.amazon.com/Rust-Programming-Language- Covers-2018/dp/1718500440/)

</comment><comment><USER7> I actually think that books which cover software engineering practices, rather than language specific books, have helped me the most. This is due to the fact that the concepts and principles you learn can be applied to many languages.

Two I've enjoyed are: "The Pragmatic Programmer" by Andy Hunt and Dave Thomas "Clean Architecture" by Robert C Martin

That being said, no harm in deep diving individual languages. Secrets of a JavaScript Ninja by John Resig was an excellent introduction to JavaScript!

</comment><comment><USER8> "Project Oberon: The Design of an Operating System, a Compiler, and a Computer" by Niklaus Wirth and Jürg Gutknecht

I have the orginal hardbound 1992 edition [0], but it looks like there is a 2013 revision available as a collection of pdfs [1]

[0] [https://www.amazon.com/Project-Oberon-Design-Operating- Compi...](https://www.amazon.com/Project-Oberon-Design-Operating- Compiler/dp/0201544288) [1] [https://inf.ethz.ch/personal/wirth/ProjectOberon/index.html](https://inf.ethz.ch/personal/wirth/ProjectOberon/index.html)

You are unlikely to encounter Oberon in the wild, but this book is the clearest and most elegant exposition of a systems design I have seen. They walk through the code for a complete operating system from hardware drivers to file system, compiler, graphic user interface and everything else in less than 10,000 lines of code. The focus on modularity and minimalism is very refreshing.

</comment><comment><USER9> Thanks for Fluent Python, is this one [1]?

Java: Even for an early version I still revisit [2] Kathy Sierra SCJP 6 book beside of new ones, a great reminder for core concepts and basics, also 'effective java' by Joshua Bloch

C: As HeckFeck mentioned it was fun to follow C Programming Language by Brian W. Kernighan, Dennis Ritchie

[1] [https://www.amazon.com/Fluent-Python-Concise-Effective- Progr...](https://www.amazon.com/Fluent-Python-Concise-Effective- Programming/dp/1491946008)

[2] [https://www.amazon.co.uk/Certified-Programmer-Study-Guide- CX...](https://www.amazon.co.uk/Certified-Programmer-Study-Guide- CX-310-065/dp/0071591060/ref=sr_1_10?crid=2GCHKVVFHB7QH&keywords=kathy+sierra&qid=1573592180&s=books&sprefix=kathy+si%2Cstripbooks%2C152&sr=1-10)

<comment><AUTHOR> Yes that's the book, although I got it very cheap through the Humble Python bundle which I think recurs periodically.

</comment></comment><comment><USER10> I've really enjoyed Programming Rust by Jason Orendorff and Jim Blandy as a good overview of Rust for an intermediate level programmer with experience in other languages.

Herlihy and Shavit's The Art of Multiprocessor Programming was fantastic for learning about parallel/concurrent algorithm and data structure design

</comment><comment><USER11> When I'm stuck on a problem at work or when there's down time, I find myself reaching for The Algorithm Design Manual.

</comment><comment><USER12> I've been enjoying Practical Common Lisp. I don't think I'll ship any Lisp code for business purposes but it's been a lot of fun to tinker with and it's conceptually interesting. I think my Python programming ability has grown somewhat indirectly through learning more about Lisp programming.

</comment><comment><USER13> Learn You a Haskell For Great Good ([http://learnyouahaskell.com/](http://learnyouahaskell.com/)). Despite Haskell's reputation for its difficultly due to its strictness and fundamental differences with mainstream imperative languages, the book manages to smoothly introduce the purely functional paradigm with tons of effective and concise examples. The book ends just where the infamous Monad is introduced and doesn't touch any of deeper capabilities of Haskell's type system, but still serves as a wonderful introduction to the language and pure functional programming. A lot of people mention SICP as the book that made them rethink/rediscover computer science, for me this book had the same effect.

</comment><comment><USER14> Type-Driven Development with Idris. It's a great introduction to dependently typed programming languages and type-driven development. Some knowledge of Haskell may be beneficial but isn't required.

</comment><comment><USER15> If you are learning Scala, then the red book.

<comment><USER16> Second this. Don't be dissuaded if it seems a bit esoteric. It's really fascinating and has an eye towards pragmatism.

</comment></comment><comment><USER17> "Coding" by Jonathan Locke, on software development in general. Short and surprizing. [https://www.goodreads.com/book/show/12625166-coding---on- sof...](https://www.goodreads.com/book/show/12625166-coding---on-software- design-process)

</comment><comment><USER18> The Java Black Book was my favorite a long time ago - [https://www.amazon.co.uk/Java-Black-Book-Steven- Holzner/dp/1...](https://www.amazon.co.uk/Java-Black-Book-Steven- Holzner/dp/1588800970/)

I learned React using Stephen Grider online courses.

</comment><comment><USER19> Secrets of the JavaScript Ninja by Resig. My favorite to learn the intermediate level for JS.

</comment><comment><USER20> "The well grounded Rubyist" is a fantastic book to learn Ruby. Right balance of theory and practice. Easy to read and still offers a lot of depth about the internals of Ruby.

</comment><comment><USER21> Check out Programming pearls by Jon Bentley. Reads like a novel, teaches phenomenal foundations.</comment>


Show HN: Atmel AVR 8-bit Emulator in Rust and React - <AUTHOR> http://realscout.github.io/avr-emulator/

<comment><USER2> Cool. I've been sporadically spending time going through the microcorruption game since I was clued into its existence a few weeks back when Starfighter was on the front page, and find it really enjoyable (if frustrating much of the time ;)

I haven't really had a compelling reason to pop into a gdb-style debugger for over a decade (sice I was in college), but it comes back pretty fast. My biggest problem is actually not letting my half-remembered knowledge from back then keep you from researching what is really going on because you _think_ you know what's happening.

<comment><USER3> Yep, Starfighter was definitely the inspiration for this.

It was super fun dusting off the cobwebs from my EE degree (over a decade ago for me too) and diving back into Assembly.

<comment><USER4> Very cool! I've been working on the microcorruption CTF myself, and it inspired me to work on a MSP430 emulator. From the newsletter that the folks at Starfighter sent out a few weeks ago, emulating the AVR is tricker so kudos to you.

<comment><USER3> Cool!

To be fair, we've only implemented 7 opcodes out of ~150 so far. But thanks!

<comment><USER5> OK that makes me feel much better about mine. I have about 20 instructions left to do.

</comment></comment></comment></comment></comment><comment><USER5> Cool. I've been slowly implementing instructions for a 8 bit AVR using Haxe.

A somewhat less than impressive display is at [http://fingswotidun.com/avr/](http://fingswotidun.com/avr/)

I have wondered though, what is the IP status of implementing an instruction set? I would like to think that it is something that should be freely implementable for interoperability purposes, but then again the entire Java/Android lawsuit thing makes me wonder if compatible interfaces are a minefield in general.</comment>


Ask HN: Finding the intersection of music preferences on Spotify with a friend? - <AUTHOR> I feel like I'd heard of some software or a plugin that did this, but I cannot find it.<p>It would be nice to be able to see, based on a group of people's spotify data, the intersection of music preferences to create the optimal group playlist <comment><USER2> Spotify has an API. I'm not sure if you can get your friends' public playlists without them authorizing the application, though.

You could probably squeeze this into a < 50 line python program. Just make an API call for all the playlists you want, toss their songs into sets, and intersect them.

Agreed it would be cool to see! You should build it. :)</comment>


Repeat after me: Unicode is not UTF-\d{1,2} - <AUTHOR> http://enjoydoingitwrong.wordpress.com/2009/06/22/unicode-is-not-utf/ <comment><USER2> From the comments on the article: "UTF-\d+ IS Unicode like a Ford Focus is a car. Not every car is a Ford Focus. Thus UTF-8 IS Unicode, not vice-versa."

I think that's a better explanation than the whole rest of the article/rant. That author is a little too nuts to follow.

<comment><USER3> That's not a great metaphor though, because there are many types of cars, but only one type of Unicode. 'Unicode' is not a generalization for character encodings the way 'car' is a generalization for car models. I think this is exactly the point the article author is trying to make.

A better analogy would be color pixels. Consider a pure-red pixel; there is only one particular shade of red a pixel can have and be pure-red. However, there are multiple ways to represent that color: RGB, HSV, HSL, YUV, CMYK, etc. These are all encodings of the same color. None of them /are/ pure-red, but they all /represent/ pure-red.

Similarly, the 1-4 byte sequences within a UTF-# encoded string aren't Unicode characters themselves; they represent individual Unicode characters. There is only one A in Unicode, but there are multiple ways to encode that A in a stream of bytes.

<comment><USER4> Well there are at least a half-dozen valid unaccented capital a characters in Unicode, but there is only one LATIN CAPITAL LETTER A, which sits at U+0041.

You are correct that there are many ways to encode U+0041 in a stream of bytes.

</comment></comment></comment><comment><USER5> The analogy I prefer is that Unicode-encoding is like network protocol layers. In a (very simplified) network, the goal is to present an interface of a pipeline for sending a continuous stream of bytes. To achieve this, you have a protocol like TCP, which consists of routines for transforming a series of packets running through a wire into a stream of bytes. To achieve a series of packets running through a wire, you have a protocol like Ethernet, which consists of routines for transforming electrical pulses into a series of data packets. You need both Ethernet and TCP to turn electrical pulses into a stream of continuous bytes.

Similarly, the goal of Unicode is to present an interface of a stream of abstract characters (the letter "A", the number 3, the nonbreaking space, etc.). To achieve this, you use Unicode, which consists of routines to convert a stream of arbitrarily-long numbers (packets) into abstract characters. To achieve a stream of arbitrarily-long numbers, you use an encoding such as UTF-8, which consists of routines to convert a series of bytes into a stream of arbitrarily-long numbers. Now, using an encoding and Unicode, you can convert a stream of bytes (i.e., an arbitrary file) into a series of abstract characters.

</comment><comment><USER4> Any article that attempts to lay down the truth about Unicode that does not mention UCS-2 is seriously deficient.

There's way too many morons out there treating UTF-16 input as UCS-2, or writing UCS-2 and calling it UTF-16 (or "Unicode" as the article nicely addresses). Both Windows and Java have fucked this up pervasively in the past.

</comment><comment><USER6> "when transforming from (byte) strings to Unicode, you are decoding your data"

Oh, so in memory they are not bytes anymore, but "code sequences"? Fair enough to attempt to clarify a point, but please don't make it even more confusing than it actually is.

I guess (what I take away from the article, even though it is not written in it) the actual "transforming" stage only applies to single letters then - "unicode" would be the mapping of a number to a letter, and the encodings (utf-8 and so on) are different ways to represent the number?

Also, is it true that utf-16 can represent all of unicode? Because I was under the impression that it can't?

<comment><USER7> It's true. UTF-16, like UTF-8 is a variable length encoding. UTF-8 uses less memory, and with UTF-16 it's much easier to determine the string length.

<comment><USER8> How is it easier in UTF-16? Most "normal" Unicode characters fit in two bytes, sure, but you can't just count bytes and divide by two if you want the right answer. It is just as difficult as UTF-8 to implement.

</comment></comment></comment><comment><USER9> Wow, there's a UTF-EBCDIC? What next, UTF-Morse code?

</comment><comment><USER10> This is the kind of thing that doesn't matter till it does, which for most programmers (at least in the US) is never.

When something bad happens you go off and learn some more, but, for a lot of people, it flat out won't be an issue.

<comment><USER11> .. and then you rewrite all your string manipulations?

Unlike other new technologies, you're already using it from the beginning (and in your approach, incorrectly). Spend the two hours to figure out how to do the basic work right at the beginning.

<comment><USER12> This. At my workplace we had legacy code written by people who either didn't understand or didn't care about this stuff, and now our databases are full of crap that we literally have to _guess_ how to correctly convert and render.

</comment></comment></comment><comment><USER13> So, unicode isn't called multicode because...?

<comment><USER4> ...because Unicode is a Universal set of Code Points that uniquely refer to Characters.

Even if some asshole decreed that there'd be variable-width encodings, you'd still have endianness issues and combining characters to trip over. Shit ain't easy.</comment></comment>


Jokob Nielsen: Guesses vs. Data as Basis for Design Recommendations - <AUTHOR> http://www.useit.com/alertbox/guesses-data.html

<comment><USER2> "In our discussion group example,

"100% of the designers who provided external data were right, whereas

"25% of the designers who relied on their personal opinion were right.

"Most strikingly, 75% of guessers were wrong. You'd be better off tossing a coin than asking advice of these people."

A very important point. Confident guessing can send more convincing than data- based hypothesizing, even when it is flat wrong. This is why I appreciate any participant here on HN who can provide data to back up statements, and why I also appreciate comments that ask for data--yes, including from me--if a statement is first made without data.

<comment><USER3> > "Most strikingly, 75% of guessers were wrong. You'd be better off tossing a > coin than asking advice of these people."

That claim above is absolutely ridiculous. If it were true you'd do much better by always doing the exact opposite of what they propose. If you can do as well as by doing the opposite of what they guess as by doing real research, then the guys clearly know far more than a random coin.

Consistently getting 75% wrong is difficult.

The data sets are too small to do any meaningful extrapolation from. The important message should've been about falsifying claims.

When you're designing anything you have a whole bunch of assumptions. Those assumptions ("everybody knows how to do X") can be falsified by a single counterexample. But this absolutely doesn't imply that if two people happen to think X that that is a meaningful reason to assume X is correct.

<comment><USER2> _If it were true you'd do much better by always doing the exact opposite of what they propose._

How do we know how many possibilities there are? Maybe there isn't just one exact opposite choice.

But, agreeing with Jakob Nielsen, if guessers are getting it wrong more than half the time when dealing with yes-no questions, it isn't a very good idea to guess, and guesses are poor guidance for important business decisions about Web services.

<comment><USER4> Of course depends on what the metrics are and how correct they are for measuring what they intend to measure.

</comment></comment></comment></comment><comment><USER5> Part of being an expert is finding the right balance between accumulated knowledge and tactical flexibility.

"Intuition" and "common sense" are nothing more than predjudices. They may be right; after all, prejudices are usually founded in real-life experiences.

The problem comes when you over-generalize those experiences and crystalize that over-generalization as knowledge.

A silly example: you run a test on a website about button colors and red is the best color and that it made a large difference.

There are a lot of lessons you could possible take from this, e.g., 1\. Users prefer red buttons over any other color 2\. You should always test button color 3\. It's important to think about what draws users' eyes and test that, including buttons. 4\. It's important to test everything, no matter how small

Learning (1) is probably wrong. Red might work in this situation, but not in another. This is the problem with learning tactics in general. As the competitive landscape evolves or the situation "on the ground" changes specific tactics become less useful, and might even be worse than an alternative you've precluded because of your "expertise."

(2) and (3) are probably good lessons to take away. (3) is better because it opens your eyes to new, possibly fruitful things to test, although (2) isn't bad.

(4) is the opposite of (1) in that it's an overbroad lesson that is hard to apply and if done religiously would probably paralyze the decision-making process.

Anyhow, that's how I think about applying "data-driven" processes to business, marketing, and product decisions.

<comment><USER6> Interesting, how do you feel about all the talk on Google's data-driven UI development?

<comment><USER5> In principle I think it's fine, but from the story it sounds like the decision-making process was busted.

What I've found is that companies that live or die by data don't have better arguments, necessarily, they just have different arguments.

At the end of the day it's about someone making a decision and taking responsibility for it. Sometimes forward momentum and the feelings of your employees are more important than minute details like that.

The story was more about Marissa Meyer's ego and how she uses the data-driven process at Google to frustrate her internal enemies, IMO, and less about the specific fact that they tested 41 shades of blue.

Personally I'd just make a decision and get on with it. If, for whatever reason, we thought the shade of blue would really, really matter you can always test it latter.

The risk of being data-driven is that you wind up in a world of Cartesian uncertainty. Do I really know ANYTHING? What if this thing I don't think matters really DOES matter? Doesn't that mean I should test every detail no matter how small?

Anyhow, these are issues you have to grapple with when you commit to being data-driven.

There's no simple answer, although it's easy for Google to justify testing every single thing because they have a gargantuan amount of traffic and even an improvement of a fraction of a percent could mean millions of dollars in revenue won or lost.

</comment></comment></comment><comment><USER7> Interesting quote: "Fair enough: All people are experts on their own preferences". I have mostly found this to not be the case. Most people have preferences, but mostly don't know why. People are, however, entitled to their own preferences.

PS:It's "Jakob"

<comment><USER8> > PS:It's "Jakob"

You mean "it's that Jakob again"? You must be designer.

Oh, I get it: you are a hacker, you mean the typo in the headline.

<comment><USER7> //designer !(!=) hacker;

</comment></comment></comment><comment><USER5> Forgive me for tooting my own horn: <http://20bits.com/articles/decisions- without-data/>

I like the fact that Jakob took it one extra step and included data that shows making decisions with data is (empirically) better than making decisions without data.

Very meta. :)

</comment><comment><USER4> I know programmers don't like to hear about intuition and that is why I am probably being down voted for this too.

But that is what makes what you do stand out and what separates the excellent designer from the mediocre designer.

Just the same that makes the excellent developer stand out, that which cannot be put into a data set.

I challenge anyone to find a common design functionality that couldn't be solved with common sense and intuition.

Something that you could prove the actual results would be better if they where based on data.

The problem of course is that no such data exist.

<comment><USER9> I have a hunch that if you use data to separate the excellent designers from the mediocre designers, then the designers who use data turn out to be the excellent ones. If you use intuition to separate the excellent designers from the mediocre designers, then perhaps the intuitive designers fare better.

</comment></comment><comment><USER10> I enjoy the fact that he uses two data points to support his conclusion that using two data points gives you better conclusions than guessing. Nothing circular about that reasoning at all. :-)

</comment><comment><USER11> This seems more like an advertisement than an article. Make them worry real big that their experts or expertise just isn't good enough, then close with a paragraph about our training. Hmmmm.

I like reading expert articles from people _not_ trying to sell me their expert services.

<comment><USER2> I wonder what Peter Norvig (who doesn't sell Web consulting services, being employed by Google full time) would say about Jakob Nielsen's general approach? It happens that Nielsen has consulted with Google before, and Google is acclaimed for its usability, so I think many of Nielsen's ideas make sense. I'd hate to have experts stay silent to avoid accusations of having commercial interests, when they could do what Nielsen does and share a lot of good advice for free.

</comment></comment><comment><USER12> Typo: 'Jokob' must be 'J _a_ kob'.

</comment><comment><USER4> "Summary: Even the tiniest amount of empirical facts (say, observing 2 users) vastly improves the probability of making correct UI design decisions."

Yes if you are a complete newbie or academic without any experience about webdesign. Otherwise not in a million years.

<comment><USER13> Is that a guess, or do you have empirical data to back it up?

<comment><USER4> It's an informed opinion.

The map is not the territory.</comment></comment></comment>


A Network-Based Comparison of Popular Privacy Extensions - <AUTHOR> https://redmorph.com/articles/proof-in-the-packet.html <comment><USER2> Adblock, Ghostery, and Disconnect have been around for so long. It's surprising the results that come from this article. Would be curious to see if someone could replicate the results.</comment>


Popular kids' tablet patched after flaws left personal data vulnerable - <AUTHOR> https://www.zdnet.com/article/popular-kids-tablet-patched-after-flaws-left-personal-data-vulnerable/ <comment><USER2> These tablets might be the next big target for hackers/phishers. I would imagine children make relatively easy targets for click-bait phishing links. Stay safe, kids.</comment>


Alibaba Raises $21.8B in Initial Public Offering - <AUTHOR> http://www.nytimes.com/2014/09/19/business/dealbook/alibaba-raises-218-billion-in-initial-public-offering.html <comment><USER2> Alibaba going for a higher selling price just makes the continued undervaluing of Yahoo itself increasingly hilarious. At some point, I would think someone will want to buy them for all of this free money they have sitting around.

      YHOO market cap : 42.3bn
      + liabilities : 3.7bn
      = 46bn (to own outright)

Meanwhile:

      Cash on hand = 1.1bn
      + Other assets = 15.3bn
      + Cash from Alibaba sale = 8.3bn
      + Value of remaining Alibaba stock = 22bn
      + Value of Yahoo Japan stock = 9bn
      = 55.7bn (of value, if you can unlock it)

So simply by buying them, firing everyone and selling everything in a firesale, without even cashing those checks from advertisers, you can make $9.7bn. Seems like a sweet deal.

<comment><USER3> No need to fire everyone. Yahoo made $38M net profit on operations last quarter. All they would have to do is maintain their current assets and employees and the owners get $160m/year in free cash. Then, using their effectively unlimited supply of capital in the form of Alibaba stock, they could make some very smart acquisitions (think for example of buying an instagram for $1b cash and growing it to what it is today) and they could pretty quickly be a $100b company.

I am not not sure if Marissa is up to the task, but they do have a legion of smart employees and an insane amount of capital. If they fail to build meaningful growth now, it's her fault.

<comment><USER2> They do still have some smart people, but I don't think anyone or any amount of money can turn things around for them in any meaningful way.

Their pre-Marissa acquisitions were almost universally wastes of time and money. It's too soon to tell about the recent acquisitions, but I don't see them making that $1bn back from Tumblr in my lifetime.

Yahoo is an also ran, constantly playing catch up and trying to emulate the successful behaviors of other companies. The negative public perception of a lot of their services in the technical community is mirrored inside the company, and there is nothing anyone can do to force positive change. Things that Google and Facebook and others announce were projects 3-4 years ago inside Yahoo, that never launched. Once you see that happen a few times, you get sufficiently disillusioned and leave. I don't think for all of Marissa's impact that this has fundamentally changed.

For example: Yahoo employees have to use Yahoo Mail for all of their work e-mail. The web interface only, with ads. It's a great exercise in dogfooding, _IF_ you commit to rapidly iterating on the product to make it best of breed for your employees (not to mention real users). But that will never happen at Yahoo, there is no will to make an excellent product for a technically demanding audience.

Now your employee productivity suffers, which means every other product in the company suffers. Eventually you will have to let them go back to using e-mail the way they want to or they will find a way to screen scrape the messages into fetchmail in order to get back to work.

<comment><USER4> _> Yahoo employees have to use Yahoo Mail for all of their work e-mail. The web interface only, with ads._

That is... terrifying. No wonder smart people leave.

</comment></comment></comment></comment><comment><USER5> Gosh....that's barely more than WhatsApp. Woulda done better selling it to Facebook.

More seriously, the Economist put a valuation at $55-$120b. This puts it at $168b. Or 9 WhatsApps. The IPO was expected to raise $20b, so this is really good.

It took 3 years after founding in 1998 to reach profitability.

It defies most conventional notions of a startup, despite having started in somebody's apartment. It's unbelievably unfocused, it does pretty much every kind of business you can do on the Internet.

It might be one of the first Asian-style conglomerates to be born on the Internet.

There are plans for it to open brick and mortar stores.

[https://en.wikipedia.org/wiki/Alibaba_Group](https://en.wikipedia.org/wiki/Alibaba_Group)

<comment><USER6> >It defies most conventional notions of a startup... It's unbelievably unfocused, it does pretty much every kind of business you can do on the Internet.

It defies the Silicon Valley notion of startup. China's internet market is still new, and Alibaba is like Yahoo in the old days - it does everything online.

Hopefully it doesn't turn out like Yahoo.

<comment><USER7> It sells more product than both ebay and amazon combined.

<comment><USER8> While we're having fun with numbers that don't make much sense to compare given the differences in businesses...

Apple generates ~1,000% more net income than Alibaba (projected 2014).

Amazon has about ~900% more revenue than Alibaba.

Apple has about ~2,000% more revenue than Alibaba.

eBay does 100% more in revenue than Alibaba

<comment><USER9> Crazy part is how much illegal stuff they sell. (knockoffs and fakes)

<comment><USER10> Its not fair comparing Amazon's revenue vs Alibaba. Alibaba doesn't sell products they take ad fees. Which means the product price itself isn't part of the revenue. It is more fair comparison with Ebay's revenue numbers.

</comment></comment></comment></comment></comment></comment><comment><USER11> Well....you guys have to look at the bigger picture: Alibaba was rejected by HK stock market the first time around, then Jack decided to take it to NYSE. Of course 21B is nothing in comparison to the other big brothers like Facebook, but ask yourself this question: what really is Alibaba doing in China? For one, Alibaba is about trading on the international scene, and its seed Taobao is like eBay China version; then they are into car manufacturing as well, not to mention Alipay being the paypal of modern day China....clearly, that 21B he raised can't even cover a lot of the expenditures for development. What Jack really is doing is to get his company that global recognition. Any of you can name ONE brand/company that based in China on top of your head? Well, that's what Jack is trying to do. You think he's unfocused? How about this: for someone who owns the e-commerce platform of the world's 3rd biggest country, he is utterly focused on becoming "China's international brand." Don't forget, China is the jumping board, and Jack is the pioneer. Once he claimed first, who will remember the ones from the subsequent ranks?

</comment><comment><USER12> This seems like a pretty bad deal for foreign investors.

[http://www.forbes.com/sites/investor/2014/09/15/four- reasons...](http://www.forbes.com/sites/investor/2014/09/15/four-reasons-to- avoid-the-alibaba-ipo/)

<comment><USER13> I don't know why more people aren't talking about this. The Motley Fool guys went over this in their podcast yesterday. I'm in full agreement.

[http://marketfoolery.libsyn.com/marketfoolery-09182014](http://marketfoolery.libsyn.com/marketfoolery-09182014)

</comment></comment><comment><USER14> "[Yahoo] will have raised nearly $8.3 billion through the offering"

If I'm reading that right Yahoo stands to triple their cash position almost overnight which today stands at around $4b.(1)

Given Marissa Meyer's appetite for acquisitions this puts them into a potentially interesting new ballpark of scale.

(1) [http://finance.yahoo.com/q/bs?s=yhoo](http://finance.yahoo.com/q/bs?s=yhoo)

<comment><USER15> You would have to consider the after-tax amount, which is going to be a bit less...around $5-6 billion.

<comment><USER16> They could buy an Instagram or two.

<comment><USER17> They should buy Buzzfeed

<comment><USER18> This a pretty solid recommendation. Mayer has pivoted yahoo strongly towards content and BuzzFeed is pretty much the most solid content start up out there right now. It sounds like an obvious purchase.

<comment><USER19> You judge a merger by how the combination will unlock new value. Just because the business models are similar doesn't necessarily mean it's a good acquisition. I don't see how BuzzFeed can benefit with Yahoo other than perhaps accessing its ad sales network?

</comment></comment></comment></comment></comment></comment><comment><USER20> Here's a link to the Alibaba IPO Roadshow Presentation, [https://www.youtube.com/watch?v=sa9R2SqZsHM](https://www.youtube.com/watch?v=sa9R2SqZsHM)

I highly recommend watching it. To save time, watch it on Chrome at 2x speed.

<comment><USER21> Sorry, how do you watch it at 2x speed on in Chrome?

<comment><USER22> 1\. Enable HTML5 at [http://youtube.com/html5](http://youtube.com/html5)

2\. Click the gear icon in the bottom right of the player and change selection in the "Speed" dropdown

</comment></comment></comment><comment><USER23> Planet Money had a decent piece on Alibaba recently: [http://www.npr.org/blogs/money/2014/09/03/345310125/episode-...](http://www.npr.org/blogs/money/2014/09/03/345310125/episode-565-the- story-of-alibaba)

</comment><comment><USER24> > Though it did not claim the title of biggest initial public offering ever, > Alibaba will still lay claim to having held one of the biggest stock sales > on record, surpassing offerings from Facebook and General Motors.

If you're curious (I was), Wikipedia puts the Agricultural Bank of China as the largest IPO, with Facebook at number 6.

[http://en.wikipedia.org/wiki/Initial_public_offering#Largest...](http://en.wikipedia.org/wiki/Initial_public_offering#Largest_IPOs)

<comment><USER25> Yet it was _America 's_ biggest IPO to date, right before Visa Inc. with $19.7B. The other three went public in China (Hong Kong Stock Exchange).

</comment></comment><comment><USER26> A bit off topic: Is there a good site to keep track of all tech IPOs and including historical data and upcoming/withdrawn IPOs?

<comment><USER27> Professor Jay Ritter of the University of Florida is a subject matter expert in this area:

[http://bear.warrington.ufl.edu/ritter/ipodata.htm](http://bear.warrington.ufl.edu/ritter/ipodata.htm)

</comment></comment><comment><USER28> Yahoo can buy 30 tumblrs with the alibaba money. That's neat.</comment>


Why Did Yishan Wong Resign as Reddit CEO? - <AUTHOR> http://www.quora.com/Why-did-Yishan-Wong-resign-as-Reddit-CEO?srid=X&share=1 <comment><USER2> [https://news.ycombinator.com/item?id=8605847](https://news.ycombinator.com/item?id=8605847)</comment>


U.S. Navy Looks to Replace IBM Servers for Security After Lenovo Purchase - <AUTHOR> http://www.wsj.com/article_email/u-s-navy-looks-to-replace-ibm-servers-for-security-after-lenovo-purchase-1432047582-lMyQjAxMTE1MjE0OTQxNzkxWj

<comment><USER2> There's no mention of Superfish in this article, but is it safe to assume that this is where these "security concerns" are rooted? Curious as to what other considerations there may be.

<comment><USER3> It's because Lenovo is a Chinese company.</comment></comment>


How to find reputation for a college's engineering program? - <AUTHOR> HN's, my nephew is being courted by several colleges in the mid-Atlantic area. He has top-1% test scores and is really interested in aerospace engineering.<p>How would you learn about a school's program reputation? I will send this question to some industry recruiters but could use some other ideas. His family lives 2000 miles away from me & will be heavily dependent on scholarship money. Thoughts? <comment><USER2> Most engineering education in the US is fungible, particularly at the undergrad level. The curricula are standardized. There are elite universities like MIT and individual engineering programs at very ordinary universities with special circumstances, e.g. aerospace engineering at UA-Huntsville's proximity to Marshall Space Flight Center and Redstone Arsenal provides students easier access some world class opportunities that rankings might not reflect.

Elite universities aside (possibly), making a decision on the basis of economics is not unreasonable because engineering credentials are mostly treated as equivalent. By design, engineering credentials indicate basic competence not relative expertise.

<comment><AUTHOR> Thanks - that's good to hear. He is getting offers from schools that I didn't know had AE programs. The UA-Huntsville option is one that I bet he hadn't considered.</comment></comment>


The differences between your eyes and your camera - <AUTHOR> http://www.pixiq.com/article/eyes-vs-cameras

<comment><USER2> It's also worth noting that the rods in your eyes are much more sensitive to low light than the cones in your eyes (which detect color). That's why it's hard to see color in low-light conditions.

So I had a fantastic surprise when I was photographing comet Hale-Bopp because it looked white to my eyes but when I developed the photos the camera had picked up that it had one red tail and one blue tail: <http://imgur.com/XKJr1>

</comment><comment><USER3> Another difference is the human eye's retina is built the wrong way around with respect to nerves - light has to go through the nerve layer and capillaries to reach the photosensitive cells.

There's more on <http://en.wikipedia.org/wiki/Retina>

<comment><USER4> AFAIK, most image sensor chips are built the same way; there's a layer of wires on top of the sensor. (Back-illuminated sensors are an exception.)

</comment></comment><comment><USER5> Don't ccd cameras also see in a slightly larger wavelengths? I seem to remember unfiltered ccds from normal cameras that could see IR flashlights.

<comment><USER6> Turn your webcam on, point your remote control at the camera and press a few buttons.

I recently had to do this to figure whether a new remote was working or not. Worked a treat.

</comment></comment><comment><USER7> How much longer until I can implant a camcorder in my eye and think about taking a video, having it take a video, then watching it again whenever I want?

<comment><USER8> As a doctor who has seen implants, long-standing implants, get infected, let me urge you to never, ever want this.

<comment><USER3> "Ever" is too broad. What about a genetically engineered, personally tailored biological implant? It may be in the distant future, but I'll guess it'll one day be possible.

<comment><USER7> I want a GPS, compass, infrared vision, gyroscopic (rotational sensors), accelerometers, optical zoom, be able to sense the frequency my cellphone is broadcasting on. I want to feel electrical and magnetic fields, be able to distinguish between every smell, have 360 degree radar/lidar, and optional echo location, temperature sensors, I want to be able to think http requests and sense the responses.

I want to be able to close my eyes, disconnect all of my 5-10 senses and plug them into a computer generated world where the rules are as I define them. So I could be lying in bed yet genuinely feel that I am running a marathon.

<comment><USER9> The ability to feel electrical/magnetic fields has been achieved by a couple of people.

<http://wiki.bmezine.com/index.php/Magnetic_Implant>

<http://www.wired.com/gadgets/mods/news/2006/06/71087>

Of course it didn't go well in the long term, but it's a very interesting experiment. Here is the article about how it went wrong (contains some graphic imagery):

[http://news.bmezine.com/2006/04/01/another-two-months-of- mag...](http://news.bmezine.com/2006/04/01/another-two-months-of-magnetic- implants-the-publishers-ring/)</comment></comment></comment></comment></comment>


Mathematics of the Small-World Phenomenon (from Networks, Crowds, and Markets) - <AUTHOR> http://www.cs.cornell.edu/home/kleinber/networks-book/networks-book-ch20.pdf

<comment><AUTHOR> Includes a discussion of Stanley Milgram's famous experiment ["with a budget of only $680, he set out to test the speculative idea that people are really connected in the global friendship network by short chains of friends.. he asked a collection of 296 randomly chosen 'starters' to try forwarding a letter to a 'target' person, a stockbroker who lived in a suburb of Boston.." & found that the MEDIAN length of successful paths was just six] and clarifies a common misconception about it:

"It doesn't establish a statement quite as bold as 'six degrees of separation between us and everyone else on this planet' - the paths were just to a single, fairly affluent target; many letters never got there [all but 64 in fact]; and attempts to recreate the experiment have been problematic due to lack of participation."

<comment><USER2> Full book (to be published by Cambridge University Press in 2010) at

<http://www.cs.cornell.edu/home/kleinber/networks-book/></comment></comment>


Ask HN: Best issue tracker in 2015? - <AUTHOR>

What do the people of Hacker News use as their issue tracker out there in 2015? We run Bugzilla, but I find its UI lacking. The UX is clunky. It does not integrate well with GitLab. Now that it has  been 4.5 years since the Make Bugzilla Pretty contest happened and failed to affect change, even in Bugzilla 5.0, I feel it may be time to switch and I'd love to know what you use in your daily development life. If it has a good kanban experience, then all the better! <comment><USER2> Honestly, as something of a minimalist, I've been happiest with Github's issue tracker lately. GitLab's looks similar in screenshots, but I've not used GitLab. (Github also has an Enterprise version if you need on-premises install these days.)

It has just enough features to get the work done but also not feature/configuration overload so that it's also not eating a lot of "management time from you". Github in particular you can tell that Github themselves dogfood their issues and PRs and want them to be great products for themselves.

There are actually more than one "Kanban overlays" for Github issues I've seen. The first example to my mind is [https://waffle.io/](https://waffle.io/)

I would presume if GitLab has similar APIs for their issues there may be a Kanban overlay for it as well.

If you are already using GitLab then maybe you should just use GitLab's issue tracker? I think the convenience of using your source control host as your issue tracker is a big, useful deal. (#synergy)

<comment><USER3> It took me a bit of "Selling" to management to begin to use Github issues as a solution at a number of organizations but every time I have it has been a rousing success! Most of the concerns around using issues have been that there "aren't enough fields for all the things we want to track on an issue." After a bit of discussion it is usually decided that they should try to track without so many fields and see what happens.

Ultimately, people love the simplicity and the integration with the repository. I know for a fact that GitLab's has a very similar mechanism and works wonderfully. Good suggestion and a very successful one!

<comment><USER2> Issue tracking is one of those areas in computing where there's a compulsion to over-organize things and store a lot of peripheral information that may or may not be germane to the actual problem of issue tracking. It's fascinating how much applying the YAGNI [You Aren't Going to Need It] principle to Issue Tracking reveals. A lot of fields are either to duplicate stuff that should already be searchable (in the main body, for instance) in a good issue tracker [1]. A bunch of other fields are business process related and it's not surprising how many of those business processes themselves could use a good YAGNI refactor, and the few that remain after tree-shaking are often easy enough to store in a label, a "tag" in the title of an issue, or a status on a kanban board.

[1] I honestly think that a lot of blame goes back to the very slow search of early issues trackers (including and especially, but not alone, Bugzilla) and needing to lump things into specialized fields to get any sort of search improvements.

</comment></comment></comment><comment><USER4> Yet once again I recently have made decision to use mantis bug tracker. At a first glance it does not look too sexy but it is matured and have all functionalities I need.

<comment><USER5> Ha, I wanted to say Mantis!

I have used that to track stories/tickets/bugs for almost 15 years and it's my favorite. I love how the colors let me see at a glance what is most pressing. I find non-technical clients can use it without problems.

One thing a lot of modern systems suffer from in my opinion (Trello, Pivotal Tracker, Asana, Agile Zen), is that the space to leave a comment is about 2 inches by 1 inch. Who wants to communicate via that? I find that they work well at the top level of defining cards, but within a card I think their UI psychologically discourages communication, so people resort to email and other things. With Mantis I've always felt less friction getting people to keep the communication inside the issue tracker. When you consider that, I'm not surprised that people are saying they like Github Issues. It's one of the few systems where you don't feel like you're peering through a microscope.

<comment><USER4> In addition I can add that there is an interesting project called "mantis kanban" \- look for their project here: [https://github.com/cgaspard/mantiskanban](https://github.com/cgaspard/mantiskanban)

And check the demo here: [http://mantiskanban.com/mantisbt/mantiskanban/](http://mantiskanban.com/mantisbt/mantiskanban/)

This made mantis look very impressive - and even my friend who loves those visual tweaks said he won't use trello anymore having mantis with mantiskanban.

</comment></comment></comment><comment><USER6> 1) Can it be hosted by someone else / SaaS? or must you host it yourself? (regulatory / contract / funding constraints sometimes dictate one or the other)

2) What's your budget?

3) How many people involved, and what are their jobs? All programmers? Designers? QA? Marketing? Support? Do customers get direct access to the issue tracker?

4) Is it really supposed to function as an issue tracker? or as a CRM system?

For "technical people only, non crm, cheap, potentially hosted at home" I still recommend Trac; If it's large scale company wide, JIRA is a good option (though it costs).

Haven't used SaaS solutions.

GitLab itself has its own issue tracker - what's wrong with that?

</comment><comment><USER7> TBH the only trackers I've recently interacted with are github, bugs.debian.org and launchpad. I used to manage (and use) a few trac instances. Trac is more of a framework for making a project manager than a ready-made tracker - I'd have a look at:

[http://www.agilofortrac.com/](http://www.agilofortrac.com/)

Or maybe:

[http://bloodhound.apache.org/](http://bloodhound.apache.org/)

If you want a more out-of-the-box opinionated tracker than standard trac. Have you looked at phabricator?

</comment><comment><USER8> I liked using Jira at my last job.

</comment><comment><USER9> Phabricator's Maniphest, especially if you need custom fields and advanced sorting, and a relatively good looking gui.

</comment><comment><USER10> Jira.</comment>


Haskell on Android - <AUTHOR> http://keera.co.uk/blog/2014/08/13/most-inspiring-green-screen-you-will-ever-see/

<comment><USER2> "...that Java was calling Haskell as C via it's Foreign Function Interface (FFI), which in turn was executing correctly (including its Runtime System), and calling SDL's C API also via the FFI. It meant all communication between Android, Java, Haskell and C was working fine..."

Is this efficient? Will end users notice any performance hit from all of these layers?

<comment><USER3> It sounds like it's layered:

        </comment></comment><comment>===JAVA</comment><comment>====
        ==C (SDL)== ==Haskell==

Rather than:

        ===JAVA====
        ==Haskell==
        ==C (SDL)==

If that's the case, I wouldn't expect much of a performance hit _from the layering_ given that (as I understand things)

        ===Java==
        ==C/C++==

is common for people writing games.

You'll certainly have a memory hit of the Haskell runtime (though I'm not sure how that compares to the size of other relevant things), and the performance differential between Haskell and whatever you would have been writing that code in otherwise (presumably C or Java) - which could be in either direction depending on the details.

<comment><USER4> Haskell runtime is about a meg in size. Rather, most memory use will come from additional libraries and garbage generated, I expect.

</comment></comment><comment><USER5> > To the inexperienced, this may look just like any green screen. But it meant > much more.

Reminds me of "Black Triangles": [http://rampantgames.com/blog/?p=7745](http://rampantgames.com/blog/?p=7745)

</comment><comment><USER6> The potential for this has been around for a bit, though not a long time. It's great to see it getting some traction. iOS/ARM support was added in 7.8 which is now the standard with Haskell Platform. Now we need tools that target and deploy our apps to all platforms :)</comment>


Gogs: A self-hosted Git service written in Go - <AUTHOR> http://gogs.io

<comment><USER3> Previous discussion: [https://news.ycombinator.com/item?id=8217286](https://news.ycombinator.com/item?id=8217286)

</comment><comment><USER4> The project looks great, but it's ironic that it uses github.com to host itself [1].

The only message "not eating your own dog food" sends is this: "we're not doing it for ourselves, we're doing it for those who might find it interesting", which in turn sends a strong warning about the lifespan of the project.

[1] - [https://github.com/gogits/gogs](https://github.com/gogits/gogs)

<comment><USER5> It's possible the project doesn't (currently) have all the features they like when hosting on Github. Nothing says the project won't be moved in the future.

<comment><USER6> They could at list mirror it.

</comment></comment></comment><comment><USER7> I always like to see projects reducing centralization of code hosting.

However, my initial reaction to the UI is that it almost feels like a ripoff of GitHub, since many UI elements are very close in layout and design - <USER2> to the same large icon on the left of the author's name (legally copied from GitHub via octicons, but still).

<comment><USER8> Requiring some programmers to also do an interface design different and better than one that has had millions poured into it is a pretty tall order.

</comment></comment><comment><USER9> Very nice. I can not see a graph of the commits. Kallithea (a fork of gitlab) in docker is an alternative we use.

</comment><comment><USER10> I wonder how it compares to GitLab, which we use internally and it is really great, with very frequent updates.

<comment><AUTHOR> I'm actually running this in favor of GitLab which might be more feature complete, but it was much too resource intensive for my taste. I haven't tested it, but the developers claim it can run on an rPi. What I _can_ say is that running it has only added a negligible amount of load to my VPS.</comment></comment>