Show HN: Free, instant, secure, disposable chat rooms built in Go - <AUTHOR> https://niltalk.com/ <comment><USER3> I only feel safe using end-to-end encrypted chatrooms. Currently, niltalk can read every message. At the very least, AES encrypting messages by the chatroom's password will reduce reliance on SSL. But it really should use public key crypto for a key exchange between users. This is what's done by other disposable chatrooms:

[https://crypto.cat/](https://crypto.cat/)

[https://ephemeral.pw/chat/](https://ephemeral.pw/chat/) (Also written in Go)

<comment><USER4> The problem with end-to-end encryption is not the encryption but the key- exchange (and especially so for multi-user setups).

If you are trusting the server to create or associate identities with keys, the end-to-end encryption is easily subvertible.

<comment><USER3> New keypairs would be generated on the client every time you join a chatroom. Another member of the chatroom sends you the shared_key encrypted by your public key. Server knows nothing, stores no keys. Keys exchanged between users.

Javascript crypto is still a problem though: [http://matasano.com/articles/javascript- cryptography/](http://matasano.com/articles/javascript-cryptography/)

When you re-download the codebase on every use, there is no way to ensure integrity of the code. This is the reason cryptocat ships as a chrome extension, because it is downloaded once. Even with these issues, I'd take javascript crypto + open source over nothing (or just SSL).

<comment><USER4> > New keypairs would be generated on the client every time you join a > chatroom. Another member of the chatroom sends you the shared_key encrypted > by your public key. Server knows nothing, stores no keys. Keys exchanged > between users.

The question is - how does the first public key exchange happen? It has be done outside of the site for it to be secure and your private key must exist locally on your device - which is contradictory to the premise of these websites.

<comment><USER5> It's asymmetric encryption. Even if the server got a hold of the public key, it would not be able to decrypt the contents.

How to ensure the server doesn't get a hold of the private key is the issue (can you really trust the code you're running?).

<comment><USER6> The bigger problem is "how do you ensure that the public key the server sent is actually the other user's, and not a MITM?".

<comment><USER4> Exactly, you have to exchange public-keys via another method - which is also potentially vulnerable.

<comment><USER7> But all forms of exchange are potentially vulnerable, the point of using multiple channels for authentication is to increase the challenge-space for potential attackers. Indeed the chief benefit of public key encryption is that the key can be exchanged over a multitude of channels and a compromise of just some of them does not jeopardize the entire operation. Perhaps we need more authentication systems where this is made implicit, with trust based on the number of different mediums the key is transferred over (or the number of different third party signers).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Have two buttons: "vote to keep open" and "vote to close". Colour them orange and blue. Release it to Reddit. Potential viral hit.

<comment><USER9> Have one button. Release to Reddit. Potential viral hit.

<comment><USER10> For anyone who doesn't understand this reference, there is a massive social experiment going on at Reddit: [https://www.reddit.com/r/thebutton](https://www.reddit.com/r/thebutton)

</comment></comment></comment><comment><AUTHOR> You can checkout the public repo here: [https://github.com/goniltalk/niltalk](https://github.com/goniltalk/niltalk)

</comment><comment><USER11> Good work. Though I have to say I've seen so many of these web-based "secure, private, anonymous" chat services now, I've lost track.

What we need is end-to-end encryption and with an open source client that just has to be downloaded and built/installed once (and in such a way that it's verifiably secure, think reproducible builds).

<comment><USER12> why does the client need to be built locally? Are you inherently suspicious of anything delivered over HTTPS?

I'm genuinely interested in why people feel local clients are more secure than something running in a browser. It's something I came across when writing an ssh client in browser (www.minaterm.com).

I guess it's the potential for a HTML page to updated overtime so it no longer reflects an audited version. However it seems that it's really a failing in our browsers that this is the case. Perhaps an external service that verifies the hash of a page would help? But this would need browser support of course.

The only thing I could think of that could be implemented in current browsers was a small stub page which calculates and displays a hash of the HTML/Javascript to be launched. The stub would need to be small enough that a user could manually check that nothing malicious has been added here.

<comment><USER13> I was looking for something like minaterm the other day, trouble is i'd be scared to put my credentials into it. A when I think about it logically that isn't rational (putty can grab my credentials just as easily), but still.

<comment><USER7> It's not entirely irrational. If putty wants to grab your credentials they have to ship a broken binary that once downloaded exists forever and can be examined and reverse engineered in the wild. Someone running a web service (or someone who has compromised said service) can target a particular user for a single session and the evidence that an attack occurred will only exist until a few caches get cleared.

</comment></comment></comment></comment><comment><USER14> Have you been measuring the server load of this at all since this thread has been open? Very curious as to what that looks like.

</comment><comment><USER15> Is this open-source?

If it is could you post a link to a public repo. Thanks :)

<comment><USER16> Author here. Yes of course - [https://github.com/goniltalk/niltalk](https://github.com/goniltalk/niltalk)

PS: The "source" link is in the footer of the website as well.

<comment><USER17> Wait, you're the author but someone else submitted it as "Show HN"? That's... not how it's supposed to work, but hey, at least you're in the thread.

Here's another room (pass is dontclickdispose):

[https://niltalk.com/r/8L5MD](https://niltalk.com/r/8L5MD)

<comment><USER18> And it's gone.

Last messages were about how the Dispose button is conspicuous and easy to press by mistake.

</comment></comment></comment></comment><comment><USER19> I've been killed kicking around the idea of doing something similar, in go, with the domain I own ChatFor.Us

JavaScript encryption, as others have mentioned is the thing I see I was planning that's missing from yours.

I'm planning on going a different direction with the domain, this functionality for private messaging for a platform set up for chat rooms as well.

Right now though I'm investigating a node.js and rethinkdb infrastructure, but that's also because I will need to persist data somehow.

Thanks for building this, at least validates that someone else has similar ideas.

<comment><USER16> Nice. From my crude and unscientific benchmarks, I've found that Go is able to handle a lot more concurrent WebSocket connections than Node.

<comment><USER19> I assumed it would. But I'm thinking socket.io would provide better client support and have been looking at meteor as a way to get something built fast. Going Roth a prototype first as I have a lot less time to do this stuff these days. I'm about equally proficient in both go and node which is to say I can stumble my way to a solution with both.

</comment></comment></comment><comment><USER4> And this is secured how exactly?

<comment><USER20> Password-protected and no public listing, I assume. Nothing on secure data transfer, though.

<comment><USER21> The number of bcrypt rounds is extremely low, too[1]. While the Go bcrypt lib will actually accept a cost of 5, that seems an unreasonably low value to me.

Coupled with absolutely no encryption of the messages in memory, I think "anonymous" would be a better term than "secure" for this.

1:[https://github.com/goniltalk/niltalk/blob/master/api.go#L75](https://github.com/goniltalk/niltalk/blob/master/api.go#L75)

<comment><USER22> Your bcrypt complaint is pretty petty. They aren't storing the hash on disk at all, and the chat rooms are only temporary.

I do have privacy concerns about this and agree they can eavesdrop if they wish. Increasing the bcrypt rounds from 5 to 15 would in no way help with any of that.

<comment><USER21> It's in a redis database, so it's not all that hard to get at, either, should someone compromise the system. "Not on disk" stops being such a good defence when it's stored in a semi-persistent DB.

My main complaint, though, is that there's simply no reason to choose such a low number of rounds. Using the exact same code as in this app, 5 rounds takes 3551534 ns/op, 10 rounds takes 3583632 ns/op and 15 rounds takes 3623005 ns/op. In other words, it's only 2% slower to use 15 rounds than it is 5, and the default (10) is less than 1% slower.

<comment><USER22> If someone has an active compromise on a running machine, they can intercept network traffic and bypass bcrypt completely.

> In other words, it's only 2% slower to use 15 rounds than it is 5, and the > default (10) is less than 1% slower.

So are you arguing that your complaint is petty or isn't? Because this isn't helping your case.

Overall your attack scenario is where an attacker has just enough access to the machine to read memory in the redis database, but not enough access to read memory in the web-server, or at the point before bcrypt has been run in the process.

If redis was stored to disk you may have a valid point. As it stands your argument actually doesn't make sense. If they can access Redis they can access pre-bcrypt passwords and therefore making bcrypt's rounds completely unimportant.

<comment><USER21> > If they can access Redis they can access pre-bcrypt passwords and therefore > making bcrypt's rounds completely unimportant.

No. The unhashed passwords are not stored in redis. What I think you're missing is that there's a significant difficulty gap between connecting to, and reading data from, redis compared to gaining root access and reading arbitrary memory on the server.

> So are you arguing that your complaint is petty or isn't? Because this isn't > helping your case.

You make a good point - even if it's not the one you were trying to make - <USER2> that it's that my benchmark was not particularly helpful as it measured per operation, not per hash.

You missed the point I was really trying to make, though, which is that difference between 5 rounds and 15 (your choice, not mine - I probably wouldn't choose 15) isn't that significant when you're doing legitimate stuff, like hashing chatroom passwords. It is significant if you're brute-forcing.

<comment><USER22> > The unhashed passwords are not stored in redis.

Never claimed otherwise. They are stored in memory though. They're in the web- server process, and the process which actually conducts the bcrypt hashing.

> What I think you're missing is that there's a significant difficulty gap > between connecting to, and reading data from, redis compared to gaining root > access and reading arbitrary memory on the server.

You don't need to read arbitrary memory on the server, you only need to be in the same scope as the web app runs in.

> It is significant if you're brute-forcing.

If you're in a position to steal the bcrypt-ed passwords in this case, you're in a position to steal the plain text passwords (both in memory, both in the same scope, why waste time breaking bcrypt?).

If the author altered the code so it DID store on the file system medium to long term, sure, it might be worth while increasing bcrypt's rounds. In the mean time bcrypt is almost pointless in this case as plain text exists in the same execution scope and is accessible to processes with access to Redis.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> How one runs this? I installed go, and redis. The ran "go get github.com/goniltalk/niltalk", which installed. The previous command created three directories under my $GOPATH, one on which has a 'nilktalk' executable.

For someone who has never dabbled with go, how do I run nilktalk after all of the above was done?

<comment><USER24> The README has the full instructions. Edit the file config.json and then do "./run" on the terminal.

<comment><USER23> Thank you. More on the installation steps on an, now closed, issue[0].

[0] [https://github.com/goniltalk/niltalk/issues/3](https://github.com/goniltalk/niltalk/issues/3)

</comment></comment></comment><comment><USER25> Nice idea, sadly only as secure as https.

<comment><USER26> As all clients need a password to enter a room, the messages could be encrypted with that password. There are a lot of JS libraries that could do this, e.g. Triplesec

<comment><USER16> Author here. Right now, it's only as secure as https, but I'll look into JS encryption. It's just a fun project that came out of some Go experiments.

<comment><USER27> Still would only be as secure as https if the client is downloading your JS crypto lib every visit.

<comment><USER25> Would it be safe to keep the crypto lib on the client somehow? Browser addon? local storage? How would we do that?

<comment><USER28> [https://webpg.org/](https://webpg.org/)

</comment></comment></comment></comment></comment></comment><comment><USER29> This is an awesome service. Thanks for making it available to everyone. Can I ask what the use case is for this? I talk to my friends using FB messenger or Google chat and my customers using a chat widget on our site, so I'm curious when I would use this.

<comment><USER16> Thanks! Use cases could be, quick private convo. at a workplace, taking a discussion on a public forum private (like HN or Reddit), talking to strangers (eg: Craigslist) without adding them to your FB or Google Talk, exchanging secrets with your friends without leaving logs on your Google talk etc. :)

<comment><USER20> The problem with taking a discussion from a public forum private, is that in the current state, everyone can choose to dispose of the room.

As proven in this very thread, it doesn't really work. The idea that everyone can dispose of the room is interesting, but there probably should be an option so that only the creator (or the first to join) can dispose of the room, for such public place cases.

<comment><USER16> It's actually meant for small group of people to have private conversations and is not really ideal for take a huge public discussion private. The idea of marking a peer the creator or making the first peer an owner complicates the whole privacy and security aspect.

<comment><USER30> Perhaps an option where a majority of users (if more than two) need to opt for deleting the room ?

</comment></comment></comment></comment></comment><comment><USER31> To make it even more instant (in terms of UX), I would display the message immediately so you don't get the little delay. From where I'm at, it's about 250 milliseconds from the point I hit ENTER to when I see the text displayed.

<comment><USER16> Author here. This is how it was meant to be but I somehow overlooked it. Thanks, will implement.

</comment></comment><comment><USER32> I have opened a room. [https://niltalk.com/r/8CKyw](https://niltalk.com/r/8CKyw) I am not going to tell the password though. Let's see how long it lasts!

</comment><comment><USER33> first off... this is great! I wonder if you could make it so when you create a room, you can attach a message.

So for instance i could generate a password then sign it with my partners public key then paste that in the message box so theoretically only they could get access to the channel.

and and, create rooms that are meant for someone, so their public key is the index and their private key decrypts the message to get the password into the channel.

<comment><USER16> Thanks :)

Public key encryption is definitely a good idea and could be an optional feature for an upcoming version.

</comment></comment><comment><USER34> hn chat time!

[https://niltalk.com/r/bfT9W](https://niltalk.com/r/bfT9W), hn-923732

<comment><USER20> Disposing of the room does remove it and kick everyone out, indeed. And then the link is invalid. Neat. According for the privacy page, it's all living in RAM only, so in theory there is no logging ([https://niltalk.com/pages/privacy](https://niltalk.com/pages/privacy)). Guess we can check the code and see for ourselves, of course.

<comment><USER24> _Guess we can check the code and see for ourselves, of course._

Alas, you can't really know if the code on Github is actually the same running on their servers.

<comment><USER16> Author here. I concur, just like any other open source software running as a hosted service. It has to be trust based.

<comment><USER35> [http://www.daemonology.net/blog/2012-01-19-playing- chicken-w...](http://www.daemonology.net/blog/2012-01-19-playing-chicken-with- cat-jpg.html)

It doesn't need to be trust-based, and in fact shouldn't be trust-based, because even if I trust you, I also have to trust the people who could coerce or bypass you, or people who could maliciously access/modify your systems.

This is why end-to-end encryption is really the only way to make promises as a server about not reading / storing logs.

</comment></comment></comment></comment></comment><comment><USER36> Really interesting to read through the source code and get an idea of how you're using Go to write APIs, thanks for sharing!

<comment><USER16> Deciding on a pattern on writing http APIs in Go was a bit of chore. Ended up using the `pat` library for chaining middleware. Quite extensible and light weight. Also, using context to pass objects through the requests chain is a neat trick.

</comment></comment><comment><USER37> This site uses insecure 1024 bit Diffie-Hellman parameters for Diffie-Hellman key exchange! Please fix!

<comment><USER37> Why was this comment downvoted? The NSA has built custom hardware to crack 1024 bit DH in a few days[1], so the site owner really should regenerate the DH parameters and use 2048 bits.

It would also be nice to disable 3DES ciphers and only allow ciphers with forward secrecy.

[1] [http://blog.erratasec.com/2013/09/tor-is-still- dhe-1024-nsa-...](http://blog.erratasec.com/2013/09/tor-is-still-dhe-1024-nsa- crackable.html#.UipD1z9Bx8E)

</comment></comment><comment><USER38> [https://niltalk.com/r/h8XLk](https://niltalk.com/r/h8XLk) pw hnchat

don't delete the room! lol

edit: this doesn't work on a public forum. some asshole always deletes it.

<comment><USER39> Aaaaand it's gone

<comment><USER31> Let's keep this one open: [https://niltalk.com/r/PEiMn](https://niltalk.com/r/PEiMn) hnhnhn

The honor and glory of Hacker News users will keep this from being closed.

<comment><USER40> gone.

</comment></comment></comment></comment><comment><USER41> I'm curious to see if this takes off, how long before the powers-that-be start to say something.

<comment><USER24> If the powers-that-be are annoyed, they'll just require the Niltalk operators to log the messages. It's not like they're encrypted end-to-end.</comment></comment>


Anyscale, from the creators of the Ray distributed computing project, launches - <AUTHOR> https://techcrunch.com/2019/12/17/anyscale-ray-project-distributed-computing-a16z/ <comment><USER2> Project Ray is specifically more focused on scaling AI systems it seems. That is sort of important but also missing from the article.

[https://bair.berkeley.edu/blog/2018/01/09/ray/](https://bair.berkeley.edu/blog/2018/01/09/ray/)

<comment><USER3> Previous discussions on the Ray project:

[https://news.ycombinator.com/item?id=15481169](https://news.ycombinator.com/item?id=15481169)

[https://news.ycombinator.com/item?id=16510610](https://news.ycombinator.com/item?id=16510610)

[https://news.ycombinator.com/item?id=20064241](https://news.ycombinator.com/item?id=20064241)

</comment></comment><comment><USER4> Ray user here. Their adoption is exponential. Much faster than Spark when it was "the same age." It's not limited to AI workloads, that's just its flagship use case with RLlib.

<comment><USER5> Has Ray grown beyond reinforcement learning? Ray appears to lack a datastore or locality mechanism for at-rest distributed data, which is really important for ETL and SQL workloads. So far Ray looks like it has some nice features that are missing from the Spark RDD API, but for ETL / SQL it looks like you'd only choose Ray there for your own entertainment.

It would be compelling if Ray were to provide Horovod support versus the authors re-applying some of same research in their own thing. Ray programming API + code distribution + Horovod performance primitives is what the community probably wants.

<comment><USER6> Interesting. What do you mean when you say: "versus the authors re-applying some of same research in their own thing."

And does "code distribution" just mean to release as open source?

<comment><USER5> code distribution I mean like cloudpickle. Which Ray has, but I wonder about an equivalent to Spark files (torrented and cached). A Kubernetes runner is also important ...

I wish the Ray peeps would consider just trying to merge some with the Spark RDD API. Reynold (at Databricks) is kinda hard to deal with, but so far to me it looks like the aim of having Ray team build things from the ground up has simply re-validated a lot of the systems work (but not all) that's already in Spark.

</comment></comment></comment></comment><comment><USER7> another one!

berkeley rise lab is such a powerhouse - spark, mesos, etc. just in the past few years. it really puts some of these big tech companies to shame.

<comment><USER8> I think spark and mesos are about a decade old at this point. I'm not really sure anyone is being put to shame either, there are wildly successful alternatives to all of these.</comment></comment>


Barack Obama On Steve Jobs - <AUTHOR> http://allthingsd.com/20111005/barack-obama-on-steve-jobs/

<comment><USER2> why not just link to the actual whitehouse.gov page?

[http://www.whitehouse.gov/blog/2011/10/05/president-obama- pa...](http://www.whitehouse.gov/blog/2011/10/05/president-obama-passing- steve-jobs-he-changed-way-each-us-sees-world)

<comment><AUTHOR> When I figured the actual link, I tried reposting it, some one had already posted it.

I wanted to delete my post, HN was timing out.</comment></comment>


Show HN: Search platform, get own search for your bookmarks and any resources - <AUTHOR> https://bubblehunt.com/ <comment><USER2> I think this is genius. I don't know how am I going to use it. Probably I wont because I couldn't think of any practical use of it now, but it really is nice, I'll recommend it whenever I find the opportunity, I hope it succeeds.

<comment><AUTHOR> Thank you friend! :)

We working on usecases and I think soon we make our system more understandable. I see you wrote about errors, we fixed them. In our system uploading thousands of sites for indexation and we have a big load, we will soon increase capacity and will be able to index sites faster.

Thank you so much! Stay tuned :)

</comment></comment><comment><AUTHOR> Hi everyone! We developing Bubblehunt ([https://bubblehunt.com](https://bubblehunt.com)) - common-based search platform, where you can create own search system without code.

Add interesting sites and resources in your profile and get your search system. Now it's public beta, you can add unlimited resources, but now 1 link - 1 indexed page. We working on deep indexing system and soon you can see our crawler, but we need improve our platform, architecture and other. On our platform now >150.000 resources, that handpicked and added by users.

I think when you are designer and add 1000 best resources for design - it can be better search system for this topic.

On Bubblehunt you can provide your opinion for ANY topic. It can be more transparent, flexible, democratic search system, where people select best resources for search.

And where every opinion is important and can be different.

If you have questions, feedback or any ideas you can write here or drop me lines on firstname.lastname@example.com

<comment><USER3> Very nice work. A couple of questions.

If I create a search bubble with my take on "everything tennis", will the contents of that bubble be available in your global web search when someone searches for tennis related concepts?

Can bubbles be composed of content not on the web?

I think if you made your site look less like a social platform and more like a cloud tool, businesses might become more interested.

Edited: thought I saw a bug but I was wrong so removed the bug report

Edit2: maybe you have a bug after all? Compare these results: [0][1].

[0] [https://bubblehunt.com/@web?q=president%20donald%20trumq%20i...](https://bubblehunt.com/@web?q=president%20donald%20trumq%20islam%20arab)

[1] [https://bubblehunt.com/@web?q=president%20donald%20trumq%20i...](https://bubblehunt.com/@web?q=president%20donald%20trumq%20islan%20arab)

No fuzzyness on "islam"?

<comment><AUTHOR> Hi! Thank you :) Yes, this is a bug, we must correct words with errors and we improve it soon.

"Can bubbles be composed of content not on the web?" Yes, soon you can upload .pdf and other files to your search system. Maybe you have other suggestions?

<comment><USER3> I like your ideas.

</comment></comment></comment></comment><comment><USER4> What stack are you using behind the scenes to power the search? Are you storing each webpage and then doing full-text search or are you trying to pick out keywords from the resources and then match that way?</comment>


Grass-Fed Beef: The Most Vegan Item in the Supermarket - <AUTHOR> https://medium.com/@drewfrench/grass-fed-beef-the-most-vegan-item-in-the-supermarket-8c46b45a0d47 <comment><USER2> Finally! Someone with a rational viewpoint on the whole issue of humans eating animals.

Factory farming has to change but it's quite natural for people (and other animals) to eat animals. As the OP points out, it's probably better for the planet, too.

</comment><comment><USER3> I thought vegan just means no animal products.</comment>


A Message from Mariette DiChristina, Editor in Chief - <AUTHOR> http://blogs.scientificamerican.com/at-scientific-american/2013/10/13/a-message-from-mariette-dichristina-editor-in-chief/

<comment><AUTHOR> They are apologizing for pulling the piece re-posted here:

[http://mikethemadbiologist.com/2013/10/12/want-to-know- what-...](http://mikethemadbiologist.com/2013/10/12/want-to-know-what- privilege-looks-like-the-tell-someone-no-get-called-a-whore-edition/)</comment>


Ask HN: Why did the Amazon fire phone fail? - <AUTHOR> Link to phone on amazon - http://www.amazon.com/Fire_Phone_13MP-Camera_32GB/dp/B00EOE0WKQ <comment><USER2> They had initially charged $200 for it on contract when comparable phones existed for free on contract. (The Prime offer was not sufficient)

When the cost was removed, it was just another Android phone.

<comment><USER3> I think it's worth pointing out that too that while "just another Android phone" even lay consumers can see the differences between Amazon's Android shop (and the Fire platform) and Google Play, fairly easily. From a lay perspective where "sideload" is a meaningless made up word, Amazon's platform certainly looks more like an "Android knockoff" than "just another Android phone".

</comment></comment><comment><AUTHOR> I am also curious on people's thoughts relating to the design chops at Amazon.</comment>


Ask HN: Who's using Google App Engine? - <AUTHOR>

What big sites use Google App Engine? Have you tried it? Is it flexible enough to run big, serious projects on? <comment><USER2> Disclaimer: In a past life I was a PM on App Engine (and not surprisingly, we're using it for our new company), so the following comment has a fairly high bias.

Though it doesn't have the same HN buzz that Heroku does, App Engine has a fairly large developer community and a lot of applications using it. Like almost all PaaS product, there are restrictions that limit what you can do (library support, native code support, etc) and these limits frustrate developers that go in expecting the same experience as having a virtual machine. The limits mean that App Engine isn't the right solution for every application under the sun. But if your application doesn't have super unique requirements, the value that App Engine provides is hard to beat.

Ten second deployments, one click version management (and roll back), logs and monitoring, and autoscaling mean that we can be really agile pushing new versions of the application. App Engine's recent changes to the scheduler and Always On feature have really solved the slow instant start issue that still plagues a lot of Heroku developers. Last year the datastore had a ton of performance and reliability issues, but the new high replication configuration appears to have solved the problem. The biggest issue for us is the one that's mentioned here a couple of times and that's SSL.

For now, App Engine is nearly perfect for us. If we end up with unique requirements, we can definitely spin up a few EC2 instances and hook them up to App Engine's Task Queue. But I'm not excited about having to manage machines. Personally, if we hit a limit with App Engine, we'd be looking to another PaaS offering like Heroku or dotCloud, or Infrastructure as a Service like MongoHQ/RedisToGo before bothering to deal with machines again.

<comment><USER3> > For now, App Engine is nearly perfect for us.

Have you launched yet? My experience is that it becomes very difficult to manage once you have customer data.

<comment><USER2> Launched, although not particularly large (www.splitterbug.com). What becomes difficult to manage when you introduce customer data?

</comment></comment></comment><comment><USER4> We use it at bynd.com for a bunch of projects; we've deployed at least 30 applications on it now. It has taken me quite a while to get used to the AE way, and I don't love it, but it is very good for some things. I can think of lots or pros and cons, but off the top of my head:

Pros:

\- scaling is amazing; we've had sites go from 1 to over 1000 instances in a few minutes; from <1 to 1600 queries per second in the same time.

\- reliability is pretty good now

\- if you let go of the idea of doing everything synchronously life gets a lot easier, and deferred processing is really easy to do.

\- the development process is great: download SDK, checkout code, point SDK at code and you're off. Then you develop locally and have it run exactly the same (with a handful of caveats) as it does appspot. It's a really low barrier to entry.

Cons:

\- webapp (the default framework) sucks. Once you find something like tipfy, webapp2, or flask life gets a lot better

\- changing your datastore structure can really suck with a live site. There's still no simple way of getting data on and off, or of doing backups. Messing with your live database if you need to fix some bug can be _scary_

\- it takes some time to grok the right way to model data, and to use the datastore.

\- there's a lot of "here's how to get started / build a simple blog" info, very little on best practices, how to model complex data, how to do maintenance...

<comment><USER5> What framework did you wind up using? Tipfy seemed great at first but the docs aren't quite up to date, so I wound up sticking with webapp for the prototype for my project.

</comment></comment><comment><USER6> App engine has the following awesome characteristics:

\- the datastore is extremely scalable if you use it as intended.

\- your app can automatically scale up and down (as if an invisible genie were moving the little "workers" slider that Heroku uses)

\- always on instances make your app mega fast, even at low volume (but cost a tad bit more).

\- there are a lot of useful apis (memcache, queue, email) and a nice development environment.

\- it usually "just works" and "just scales".

a few bad things:

\- they had some very annoying downtime while they sorted out the datastore architecture problems (where theory and practice had differed with horrible results)

\- the OSX dev tools don't bundle the correct version of python and associated libs, which makes it a pain to get a working dev environment on OSX without breaking your system's python.

\- no naked domains, SSL support for normal apps that aren't priced per user.

All in all I'd rather write an app for AppEngine than any other cloud system, though Heroku is catching up -- both could really benefit from prioritizing 24 hour support via irc though.

</comment><comment><USER7> App Engine has a lot of really awesome features going for it. There are some pains for sure, but not the ones you find people typically griping about. The datastore and all of that aren't a big deal. What's annoying is the lack of library compatibility, and no real SSL suport. Having access to a relatively small subset of available libraries for Java/Python sucks, each time you find a library you'll have to find out for yourself it it works on AE or not.

As for SSL. I mean come on. It's 2011, it's absolutely unbelievable they don't support SSL for custom domains. Unbelievable.

<comment><USER8> There are ways to hack around the lack of SSL on third party domains with Ajax, but yeah it's pretty insane that they still haven't got that in place.

<comment><USER9> Completely agree that it's a much needed feature. We've worked around that by setting up an nginx server in AWS which has proxies to the <https://*.appspot.com> name. So it's still SSL all the way, but we just didn't want appspot.com as the URL we gave out.

</comment></comment></comment><comment><USER10> The UK's Met Office - who provided Weather Forecasts use it for their new Weather Observations Website (WOW): <http://wow.metoffice.gov.uk>

It's a site that allows the public to submit weather observations from anywhere in the world. Since launching 31 days ago, we've got 1.7 million weather reports and probably spend about 0-30 seconds a day (on average) on maintenance.

App Engine has some big pains (SSL has already been mentioned) but if you don't require it (we don't), I think it's a fantastic tool.

The multiple versions, quick deployment (new build inc all testing&configuration can be completed by us in <5 min), automatic scaling and the fairly good dashboard are fantastic. Plus, our infrastructure costs in development (before heavy load testing) was $10 for 3 months of work - absolute bargain! Even after heavy load testing, it was a fraction of what a real server would have cost.

I'd say the biggest difficulty (or barrier to entry) is the datastore design. It's very easy to think you're in SQL land, and you're not. And because of that, it's very easy to burn yourself quickly. But, the design gives great scalability. We wouldn't be able to manage the size of the data we're expecting in 2 years+ in a typical environment.

So, I think it's great :)

(Disclaimer: I was part of the team at PA Consulting that built WOW)

</comment><comment><USER11> I don't know what big sites use App Engine but I personally use it for everything. Every weekend project, every work prototype, everything! I just love the fact that it's free to start with, and the idea that it's on Google's server sort of brings a level of confidence for up-time and scalability.

Sure the fact that are some limitations (no IO and not many libraries) are a bit of a down-side, but the payoffs of using such a service instead of managing your own servers is really great. You're just able to crank a lot on the app code part without worrying at all about system administration work.

The simplicity of the datastore is also a great benefit, and it scales like nothing out there.

</comment><comment><USER12> I use App Engine a ton. I won't try to list all the positives. Of course "is it flexible enough to run big, serious projects" depends on the requirements of your big serious projects. Feel free to get more specific here, or (better yet) in the google app engine forums.

One thing not mentioned yet that is a legitimate concern applying generally to PaaS offerings including App Engine - that is vendor lock-in. There are some way to address that including choice of apis and frameworks and also projects like AppScale and TyphoonAE.

There is much to evaluate when considering App Engine for your projects. the flexibility of the platform from a technical perspective is one of things.

Also note, watch closely the changes that are coming in pricing and the way that Google is offering the service. You don't want to get caught evaluating the App Engine of 2010 for 2012.

</comment><comment><USER13> We use it here at Optimizely and have been very happy. It hasn't been smooth sailing all the time, and we've gradually grown our stack to include EC2, S3, and several CDNs as well, but overall we're very happy with it and have no plans to leave.

The biggest benefits for us are the scalability (huge), ease of deployment, and the monitoring. The biggest cons have been lack of support for SSL on custom domains (HUGE hassle) and lack of official support.

<comment><USER4> I'm interested to hear that you think monitoring is easy - it was something I forgot to add to my "cons" list; all you get for free is the web dashboard that lets you look at logs, and ereporter to send you daily exception summaries. One of my biggest "want"s in fact is API access to logging.

</comment></comment><comment><USER14> Google itself is using App Engine on both internal and external projects as far as I know.

The websites below has tens of thousands of registered members, there are also similar websites for some other European countries (Italy, Spain, Germany etc)

Getting British Businesses Online: <http://www.gbbo.co.uk/>

Getting Australian Businesses Online: <http://www.gettingbusinessonline.com.au/>

So yeah, App Engine seems to be suitable for big projects if you play by their rules. Only problem seems to be backing up data, as their App Engine's bulkloader tool has rate limitation, so it takes quite a while to download/upload all data.

</comment><comment><USER15> It would be great if Google would allow for managing of a custom reverse proxy with unique IP into an app.

This could solve the naked domain and SSL issues and enable other cool customizations.

Has anyone done this themselves? Just put Varnish or Nginx in front of GAE?

<comment><USER9> We've put Nginx in front of GAE and it works without any issue. I haven't tried any other methods such as Varnish though. We aren't really throwing too much traffic at the GAE app which is "behind" Nginx, but we fully expect it to handle a reasonable amount of traffic.

I'm sure there are many people who have done this. I'd be interested in hearing how other people have solved the lack of SSL...

</comment></comment><comment><USER16> Appengine is extremely good for auto scaling, if u have 100 users today and and u get 1 million users tomorrow, it will just work without u having to do anything, u dont need a scaling engineer.

I use appengine for all my web apps, it works for most of basic webapps, here are a few I have used appengine for: <http://gramfeed.com>, <http://shaloc.com>, <http://checkinmania.com>, <http://misotrendy.com>

</comment><comment><USER17> I have used AppEngine for a few fun projects where I have tried to use FB Login/Connect for authentication. I like the fact that you can rapidly prototype and improve your skills and have it all published to the web for free.

Question: I was looking for a library for authentication mechanism other than FB or Google . Any suggestions ? (All I need is basic user/password)

<comment><USER18> I use Django exclusively on it, and thus use Django's authentication framework, which works very well.

<comment><USER17> so are you using one of those helper libraries/project django-norel ?

</comment></comment></comment><comment><USER19> I'm using Google App Engine in a new project ( <http://www.homeyplanet.com> \- allowing travelers to share their favorite spots off the beaten track) and I have to say I am very happy about it. This project requires geo spatial queries, so at first it wouldn't be a good choice given how inadequate the datastore is for it, but the trick is always to think how things work in GAE and see if you can optimize it. It only scales well if you do it their way...so it won't work for everything, but most problems can be solved quite well, you will just have to scratch the way you normally would solve them.

Note: I noticed that some have accessed the site from this link. The site is beta and is newly launched, and for now we are only covering Europe. If you want to check out how fluid are the geo spatial queries, you can try checking out Portugal, where half our spots are now.

</comment><comment><USER20> AppEngine is great for rapid prototyping, but it still has several limitations that make me hesitant to base a real infrastructure off of it.

I've used it for <http://www.minehub.net>, <http://www.quickcite.it>, and a few others.

</comment><comment><USER21> I used appengine for my now defunct <http://www.unedditreddit.com> project, which sucked up all reddit comments for a few months. It was extremely easy to use and seemed to preform just fine, with the exception of those few weeks where the datastore had stability problems.

</comment><comment><USER22> Windsoc uses App Engine for our Unified Social API, and while it has many benefits, we've always had our minds on how to move off of it someday. SSL is a big issue for us, and another is the support model (which should be changing soon from what I've seen).

So what we have now is a product that runs both on App Engine and on DotCloud. Since DotCloud is awesome, too, and we know that we can talk to a specific person when something goes wrong, it's currently a front-runner for us. We're also looking at Joyent, too, as opposed to EC2.

But even if we switch large parts of our product to somewhere else, I wouldn't be surprised if we still used App Engine for some things.

</comment><comment><USER23> I was using this on a project for a while. But compared to running the application with PHP, it was always slower. Sure, I've developed applications with PHP since 2001, so I really know how to tweak it and optimize it. But, with App Engine it was really frustrating. Requests really took long to complete, and there was no real way to figure out what was going on behind the scenes. We created an 'always on' instance, but it was still quite sluggish compared to the exact app running on a standard linux, apache, PHP system.

I would still try it again down the road, but it's really not a good fit for what I'm currently doing.

<comment><USER12> Just out of curiosity, how would you describe what you are currently doing? Can you elaborate on why it might not be a good fit? I'm honestly just trying to learn from your experience, not disputing anything.

<comment><USER23> Sure - no problem - the application is Generic Domain Finder - just search it on Google. The current app in the search results is still running GAE.

The things I loved: \- tight integration with Google accounts \- easy integration with other Google apis (we used a few)

The frustrating thing was dealing with the black box. We used an api to get domain availability. Every time it returns more slowly than the equivalent call on our linux based cloud server.

With GAE, you're basically stuck with what you get. With a generic Linux based cloud server - you have more room to optimize your environment, or even take your code and run it anywhere... AWS, Rackspace, Voxel...

I still believe in what GAE is trying to do - and I'll definitely try it out again for future projects.

</comment></comment></comment><comment><USER24> It takes you a bit to get used to the catchas... I am using it in one of my side projects and it's working very well although we had some throuble to move from SQL to the BigTable and that slowed us down a bit in the architecture phase.

But I am betting on it because I think it will make my life easier in the future...

The memcache is ok to get.

The java subset is the part that bothers me as I had a lot of already made stuff I liked to use in my new site, and had to redo a loooooot of stuff to fit the dawn java subset.

</comment><comment><USER25> We use it at www.bugsense.com and we love it. It has lot of great apis included but it requires a mind switch and you should study the limitations. Datastore is really scalable but slow.

</comment><comment><USER26> Most, if not all the comments are from people/companies that use GAE. I'd love to hear from people who evaluated it and then decided not to go for it.

</comment><comment><USER27> I've used App Engine for web projects and backend server for Android apps. My web TV site <http://www.topchan.tv>, and the webpage snapshotting site <http://www.previouslook.com> are in App Engine.

</comment><comment><USER28> I built <http://www.sparkmuse.com> and <http://thesparkfoundry.com> on AppEngine. You'll have to write a little more code to handle not having a RDMS, but that's only if you need joins.

</comment><comment><USER29> I have been using GAE Python since it came out. Used it to build www.thesponty.com and www.gamedaytycoon.com (social game on FB). The latter has thousands of users and GAE scales super well. We deploy > 10 times a day. GAE is really awesome.

</comment><comment><USER30> Optimizely. I can't remember other big ones but there more. App engine has proven to be serving millions of requests per day without any issue. There are some case studies around, also some web caches hosted on app engine.

</comment><comment><USER31> I've only just started using it (this week) as an excuse to get my feet wet with Go. It seems easy enough to get started, things are well documented so far though I've admittedly only touched about half the packages.

</comment><comment><USER32> What about the channel API, any experiences? I want to do a multiplayer quiz app as a side project and I'm wondering if I should go with the GAE or if I should dive into node (using socketstream, maybe).

</comment><comment><USER33> I built <http://www.onesong.me> in appengine using the java playframework, and I quite like what I get for free.

</comment><comment><USER34> We use GAE for Pulp Sync (acrylicapps.com/pulp). It's scales very well if you handle requests quickly and take advantage of the memcache and taskqueue APIs.

</comment><comment><USER35> At 2XL we built the backend for Ricky Carmichael's Motocross Matchup on Google AppEngine. We might switch to EC2 as it grows.

</comment><comment><USER36> we use it at ligertail.com and dont see a reason to switch to aws so far :)</comment>


Ask HN: Observability tools for Early Stage companies - <AUTHOR> Which Observability tool is recommended for early to medium stage ventures to effective Log/monitor/trace the events? <comment><USER2> Disclosure: I contribute to the software mentionned next.

Definitely implement OpenTelemetry [0] instead of specific providers sdk even if promised to have better features. Lock-in is hard to escape when you need ship 2 features for yesterday. However OTel is quite young though (still not GA) so you'll feel some missing features (logs are still WIP for ex).

On the question of where to host the data, on the early days i would recommend to opt (if available) for your cloud platform products (Stackdriver for GCP for ex) since it will be easier to setup. Later on you will be able to easily switch to something better as your needs will evolve.

[0]: [https://opentelemetry.io/](https://opentelemetry.io/)

</comment><comment><USER3> Infrastructure/software or Business?</comment>


For the next three weeks, if you boot up Windows 95, the clock will be wrong - <AUTHOR> https://twitter.com/Foone/status/1105181992843337729 <comment><USER2> Went in for Windows 95 timezone-related tidbits. Came out knowing more than I ever wanted about weather station data collection.</comment>


Show HN: News Timeline Concept - <AUTHOR> http://jamie.st/blog/news-timeline/

<comment><USER3> I like the concept. though I think the layout needs a bit of work though. I would want to be able to see a bit more information in the presentation in a final product, maybe decouple the navigation from the feed so the size of the views aren't tied to the size of the timeline bubbles. It would be an interesting navigational element though.

Also it seems to crash entirely when I post an empty form. Which is no big deal since it's a mockup but still..

<comment><AUTHOR> Thanks for your suggestions, I wasn't aware of that crash, thanks for the heads up!

</comment></comment><comment><USER4> timeline: like! i wish more news sites presents history in a straightforward manner colors: not so much; not meaning & some combi are hard to read; favicon/logo/color-scheme-based-on-logo will be more useful http: if the form method is "GET", then I can easily share or bookmark for updates later text: some body text will be nice

<comment><AUTHOR> Thanks for your suggestions, hoping to get all of these sorted out over the next week or so :)

Trying to find open API's with quality content is a bit tricky though - <USER2> that could be the breaking of this project!</comment></comment>


On Internal Engineering Practices at Amazon - <AUTHOR> http://jatins.gitlab.io/me/amazon-internal-tools/ <comment><USER2> This… feels like it toes the line of NDA breaching. Fascinating story, though.</comment>


Ask HN: What info do you web scrape for? - <AUTHOR> I have been web scraping for several months and am starting to teach it at Meetups. I'm lucky enough to work for a company that has a few pre-crawled copies of the web that I can query against and a distributed processing platform to speed up any scraping I do.
I'm running out of ideas of what to build though. I build scrapers to produce content for the company based on the data and insights I find. They are usually marketing verticals though, such as finding all websites using feedback tools (I search based on their javascript widgets), and do analysis on that info.
So if you had these resources, what would you be looking for? I love building tools that help people so any feedback/ideas would be great!
I'm also open to hearing what you would scrape for on the live web.  I find that if I'm doing broad analysis then the pre-crawled copies are best, and for specific sites/information I use the live web. <comment><USER2> There is a closet industry for scraping any sort of data that can move markets. Fed, crop, weather, employment,etc.

Anything that is released at a certain time on a fixed calendar, you can bet that multiple parties are trying to scrape it as fast as possible.

If you can scrape this data( the easy part), put it in a structured format( somewhat hard) and deliver it in under a few seconds(this is where you get paid) then you can almost name your price.

It's an interesting niche that hasn't been computerized yet.

If you can't get the speed then the first 2 steps can still be useful to the large number of funds that are springing up using "deep learning" techniques to build a portfolio over timelines of weeks to months.

To answer the question of: > Wouldn't this require a huge network of various proxy IPs to constantly fetch new data from the site without being flagged and blacklisted?

This is why I gave the caveat of only looking at data that comes out at certain times. That way you only have to hit the server once, when the data comes out, or atleast a few hundred times in the seconds leading up to the data's release:)

<comment><USER3> That is a fairly surprising opportunity. I have experience monitoring/scraping thousands government websites for a different purpose. Considering some government sites have a round trip of well over 5 seconds, seems like it'd be a fun challenge to parse, format, and deliver it that fast.

What types of data formatting are you talking about here? Would it require a unique template for each individual site?

</comment></comment><comment><USER4> I work as a research analyst for a Canadian provincial opposition party. Most government data is in terrible HTML tables, often dynamically generated, and almost none of it is in an easily machine readable format. I spend a lot of time downloading PDF files of data and converting them to JSON formats.

I have two main recurring scrapes:

\- political donations. Every donation to a political party in my province above ~$300 is posted publicly on a gov't website (in a PDF). I use the data to run machine learning algorithms to predict who is most likely to want to donate to my party.

\- public service expenses. My province has a "sunshine list" which publishes the salaries and contracts for all senior government officials. We grab it weekly (as once someone quits the gov't, their data disappears).

One tool that you could consider building is an easily accessible expense website, where people can enter the name of a public official and see all their expenses, including a summary of the total amount spent. There have been a number of massive expenses here in Canada related to this [1, 2].

[1] [http://news.nationalpost.com/tag/alison- redford/](http://news.nationalpost.com/tag/alison-redford/) [2] [http://en.wikipedia.org/wiki/Canadian_Senate_expenses_scanda...](http://en.wikipedia.org/wiki/Canadian_Senate_expenses_scandal)

<comment><USER5> This is very interesting work, regarding your last point, I came across this site which is searchable by employee name or government organization [http://canada.landoffree.com/](http://canada.landoffree.com/).

I think the salary and expense disclosure is only for Ontario, based on the sunshine list.

<comment><USER4> I don't know much about the civil service, but I know that most provincial assemblies have expense disclosures, at least for the elected officials, e.g. [1], [2]. Federal cabinet ministers also have to disclose all of their office expenses (can't find a link, but I know it exists).

[1] BC: [http://www.leg.bc.ca/Mla/remuneration/index.htm](http://www.leg.bc.ca/Mla/remuneration/index.htm) [2] Alberta: [http://alberta.ca/travelandexpensedisclosure.cfm](http://alberta.ca/travelandexpensedisclosure.cfm)

</comment></comment></comment><comment><USER6> I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated.

1\. Reformatting and content archival (lag times of hours to days are no prob).

As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tree structure to shake out all the irrelevant comments leaving only the necessary context. Real time isn't an issue. Up until recently it ran on a weekly cron job. Now it's daily.

2\. Aggregating and structuring data from disparate sources (real time can make you money).

I work in real estate. Leasing websites are shitty and the information companies are expensive and also kinda shitty. Where possible we scrape the websites for building availability but a lot of time that data is buried in PDFs. For a lot of business domains, being able to scrape data in a structured way from PDFs would be killer if you could do it! I guarantee the industries chollida1 mentioned want the hell out of this too. We enter the PDFs manually. :(

Updates go in monthly cycles, timeliness isn't a huge issue. Lag times of ~3-5 business days are just fine especially for the things that need to be manually entered.

This is exactly the sort of scraping that Pricenomics is doing [1]. They charge $2k/site/month. Hopefully y'all are making that much.

3\. Bespoke, one shot versions of #2.

One shot data imports, typically to initially populate a database. I've done a ton of these and I hate them. An example is a farmer's market project I worked on. We got our hands on a shitty national database of farmers markets, I ended up writing a custom parser that worked in ~85% of cases and we manually cleaned up the rest. The thing that sucks about one shot scrape jobs from bad sources is that it almost always means manual cleanup. It's just not worth it to write code that works 100% when it will only be used once.

Make any part of structuring scraped data easier and you guys are awesome!

[1] [http://priceonomics.com/data-services/](http://priceonomics.com/data- services/)

<comment><USER7> There are services that cover at least part of what you mentioned. These effectively provide you a tool to visually build a scraper and then they automate the scraping in the background, creating an API or spreadsheet of the data.

Import.io is one example, and I think there's another more recent YC-backed one. However, I tried using import.io a little while back but without much joy.

<comment><USER6> We have a non-tech intern and import.io looks like like a great tool to get him chewing up data. I'm playing with it now. Why didn't it work out for you? Beyond the wrapped browser interface being a little funky lol. (Edit: eugh, selecting data for import is really clunky.)

Ask HN: Anybody got a visual scraping service they like?

<comment><USER7> It was the data extraction and selection process I couldn't get to work. I was trying to scrape a particular search on autotrader.co.uk (I wanted more up to date results than their daily emails provide, and I wanted to filter out cars that had been written off). I don't remember all the details, but I followed the tutorial video and got to the stage where you select a single item that matches your criteria and it's supposed to extrapolate from there. However I just seemed to be stuck in an infinite loop of it asking me to do this.

<comment><USER8> I found you often have to select two, then it figures it out. I assumed it was probably because of alternating odd/even row CSS classes.

</comment></comment></comment></comment></comment><comment><USER9> Partial plug, but very related to the topic: if you're doing large scale analysis on the web and you don't want to have to actually run a large scale crawl, use the CommonCrawl dataset[1]! Common Crawl is a non profit organization that wants to allow anyone to use big web data.

I'm one of the team behind the crawl itself. Last month (July) we downloaded 4 billion web pages. Thanks to Amazon Public Datasets, all of that data is freely distributed via Amazon S3, under a very permissive license (i.e. good for academics, start-ups, businesses, and hobbyists). If your hardware lives on EC2, you can process the entire thing quickly for free. If you have your own cluster and many many terabytes of storage, you can download it too!

People have used the dataset to generate hyperlink graphs[2], web table content[2], microdata[2], n-gram and language model data (ala Google N-grams)[3], NLP research on word vectors[4], and so on, so there's a lot that can be done!

[1]: [http://commoncrawl.org/](http://commoncrawl.org/) [2]: [http://webdatacommons.org/](http://webdatacommons.org/) [3]: [http://statmt.org/ngrams](http://statmt.org/ngrams) [4]: [http://nlp.stanford.edu/projects/glove/](http://nlp.stanford.edu/projects/glove/)

<comment><USER10> What would it take in terms of resources, etc. to get to the point where Common Crawl was doing web scale crawls on a regular basis?

<comment><USER9> We're aiming to do monthly crawls from this point on. The main hold was automating the intensive manual steps of our crawl process. Now we have scripts that make running our 100 node EC2 cluster and processing the terabytes of web data relatively trivial.

If anyone wants to discuss sourcing well distributed crawl lists for billions of pages per month, we'd love to chat. We want to make sure we cover a diverse variety of languages and domains. Given that we're trying to get a good sample of the web, that's a difficult proposition!

</comment></comment></comment><comment><USER11> A lot of services with online billing refuse to send bills by e-mail, instead requiring users to log into their websites.

No doubt the companies would justify this by saying e-mail isn't secure enough. The side-effect that it'll stop many users bothering to look at their bill isn't why they do it at all, no sir.

I've been considering making a web scraper that goes to the phone company, electricity company, gas company, broadband company, electronic payslips, bank, stockbroker, AWS and so on; logs in with my credentials; downloads the PDF (or html) statements; and sends them by e-mail.

Of course, such a web scraper would need my online banking credentials, so I'm not in the market for a software-as-a-service offering.

<comment><USER12> I started working on a tool like this over the weekend to pull down my bills, and pay the balance.

I think there is a market for something like Mint for bill paying - it's a it's a bit of a pain to remember when I have to pay all of my bills each month, and make sure to go through each one, and pay the balance on time.

</comment></comment><comment><USER13> I scrape about 60 blogs and news sites that deal with a niche topic and examine all the hyperlinks. If more than one of them links to the same page, I assume that it's a page that's generating some buzz, so I send it out in an email. It's proved to be a generally reliable assumption.

<comment><USER14> Would love it if you posted a blog article sometime about the technical details.

<comment><USER13> There's really not much to it. Scrape each site/feed every X minutes, find all the hyperlinked URLs on the page, add them to a database table ... and if they're already in the table, send out an email with links to the "buzzed about" URL, as well as all of the sites/feeds that mention it. I keep the links in the table for about a month.

</comment></comment></comment><comment><USER15> I scrape Gumtree and eBay hourly using a python script for certain things I want under a certain price. The script sends me an email with the link in it and I get on top of it sharpish.

Managed to bag a lot of stuff over the last couple of years for not much money.

If someone bags this up as a service I'd pay for it.

<comment><USER16> @allegory could you share the the script with us ?

<comment><USER15> Would love to but not at the minute because it has hard coded credentials for eBay API in it. It's on my list as a TODO to tidy it up. Will stuff on github and post a Show HN on it soon :)

I've got one that monitors amazon prices for sudden lows as well.

<comment><USER17> I'll second a want for this script, let us know when it's up!

</comment></comment></comment></comment><comment><USER18> I do a lot of scraping for my day job. We have a business intelligence team that will build us reports that we need from the data that we have. However I find that this process is so incredibly slow and sometimes we only need to compile the data for a one-off project. I used to use vb.net for this as that's what I started learning programming with. Now I use python/requests/bs4 for all my scraping scripts.

I've started working on a new website that will use data scraped from several vbulletin forums. I've found that even 2 vbulletin forums running the same version may have completely different html to work with. I'm assuming that it's the templates they are using that changes it so much.

I'm setting up the process so that the webscraping happens from different locations than the server were the site is hosted. The scraping scripts upload to the webserver via an api I've built for this. Mostly did this because for now I'm just using a free pythonanywhere account and their firewall would block all of this without a paid account. And then also none of these sites would see the scraping traffic coming from my website, etc...

<comment><USER19> At Diffbot, we have an automated discussion thread parser, currently in beta testing, that might be exactly what you need. Send me a note at firstname.lastname@example.com and I'd be glad to hook you up.

(disclosure: I work there)

</comment></comment><comment><USER20> When I worked at MyEdu, I didn't actually sign on with the dev team originally — I worked on "the scraper team". We scraped college and university websites to get class schedule information: which classes were being taught, broken down by department and course number; by which professors; at which times on which days. If you're ever looking for an interesting challenge, I would encourage you to try getting this data.

Well-formed HTML is the exception rather than the rule and page navigation is often "interesting". Sometimes the school's system will use software from companies like Sungard or PeopleSoft, but there's customization within that... and of course, there's no incentive for the schools to aggregate this information in a common format (hence MyEdu's initiative), so there are plenty of homegrown systems. In short, there's no one-size-fits-all solution.

* NOTE: If you do attempt this, I insist that you teach throttling techniques from the very start. Some schools will IP block you if you hit them too hard; other schools have crummy infrastructure and will be crushed by your traffic. Scrape responsibly!

<comment><USER21> I've attempted to do this before for a side project (RateMyProfessor, but for textbooks!) and its incredibly hard to do accurately. One of the (many) issues that I ran into was that some schools still have all of their course data PDF format, in addition to the problems Cyranix listed above.

Much respect to any person or team that has to wade through this stuff.

</comment></comment><comment><USER22> We do a lot of live web scraping of product information from retail sites for [http://agora.sh](http://agora.sh). We basically scrape all of the essential product info and offer it to the user in an optimized view (we call it the 'product portal') that can be accessed without having to load a new page. This reduces tab sprawl and provides a general improvement to a lot of shopping workflows, such as just wanting to see larger/more images of the product (likely clothing) and being able to do so with an overlay on the existing page.

</comment><comment><USER23> My first scraping project was well over 10 years ago in college. I was a member of the education club and we wanted to get funding so I convinced the college of education to allow us to charge $10 automatically to students in their school. But then the administration dragged their feet to give us a list to submit to the accounting office for billing. 1 (of the many professors) submitted a list to me via outlook that they copied off the site so I was able to look at the HTML structure of their list. The university used basic security (htaccess) and didn't verify that you had permission for a task once you were in. I had access because I worked for the dean of men. So I scraped all the faculties student lists and then used another system behind the htaccess point to get all the relevant information on each student. Compiled a list of 300 students and submitted it getting the club $3,000 in funding. The college of ed office staff were freaked because they had no clue how I came up with the student roster (no one in their office gave it to me) but nothing came of it.

Been scraping a lot lately but mostly:

\- government website for license holders

\- creating lists of businesses for different segments (market research/analysis)

\- using those lists to scrape individual sites and make analysis (how many use facebook/youtube/etc)

</comment><comment><USER24> I am currently scraping for brand product and nutrition data. Having to build custom scrapers per brand is hell.

I have a dream to use something closer to OCR against a rendered page, rather than parsing DOM. That way it would be less custom, and I could say, for instance, "find 'protein', the thing to the right of that is the protein grams".

I, personally, don't know how to do this, but I'd be willing to pay for a more generic way to scrape nutrition data (email in profile :) )

<comment><USER25> There's companies out there that already have this info. Checkout [http://kwikeesystems.com](http://kwikeesystems.com) .

</comment></comment><comment><USER26> I once wrote a scraper for a Yellow Pages site in Python. It pulled down the business category, name, telephone and email for every entry, and returned a nicely formatted spreadsheet. The hours I spent learning the ElementTree API and XPath expressions have paid for themselves several times over, now that I have a nicely segmented spreadsheet of business categories and email addresses, which I target via email marketing.

<comment><USER27> As someone responsible for search on a yellow pages company, I can confirm that most YP websites have little to no protection against this. Company information is usually public anyway. We just make it very easy for you to get it :)

</comment></comment><comment><USER28> I'm working on a startup that has web scraping at its core. The vision is a bit larger and includes fusing data from various sources in a probabilistic way (e.g. the same people, products, or companies found on different sides with ambiguous names and information. This is based on the research I've doen at uni). However, I found that there are no web crawling frameworks out there that allow for large-scale and continuous crawling of changing data. So the first step has become to actually write such a system myself, and perhaps even open source it.

In terms of use cases, here are some I've come across:

\- Product pricing data: Many companies collect pricing data from e-commerce sites. Latency and temporal trends are important here. Believe it or not, there are still profitable companies out there that hire people to manually scrape websites and input data into a database.

\- Various analyses based on job listing data: Similar to what you do by looking at which websites contain certain widgets, you can start understanding job listing (using NLP) to find out which technologies are used by which companies. Several startups doing this. Great data for bizdev and sales. You can also use job data to understand technology hiring trends, understand the long-term strategies of competitor's, or us them as a signal for the health of a company.

\- News data + NLP: Crawling news data and understanding facts mentioned in news (using Natural Language Processing) in real-time is used in many industries. Finance, M&A, etc.

\- People data: Crawl public LinkedIn and Twitter profiles to understand when people are switching jobs/careers, etc.

\- Real-estate data: Understand pricing trends and merge information from similar listings found on various real estate listing websites.

\- Merging signals and information from different sources: For example, crawl company websites, Crunchbase, news articles related to the company, LinkedIn profile's of employees and combine all the information found in various source to arrive at meaningful structured representation. Not limited to companies, you can probably think of other use cases.

In general, I think there is a lot of untapped potential and useful data in combining the capabilities of large-scale web scraping, Natural Language Processing, and information fusion / entity resolution.

Getting changing data with low latency (and exposing it as a stream) is still very difficult, and there are lots of interesting use cases as well.

Hope this helps. Also, feel free to send me an email (in my profile) if you want to have a chat or exchange more ideas. Seems like we're working on similar things.

<comment><USER4> How do you use a probabilistic approach to scraping data? Were you able to get a low number of false positives?

<comment><USER28> Sorry for the confusion. They are used for "merging" scraped data from various sources, not in the scraping process itself. For example, they help in figuring out if similar-sounding listings on related websites refer to the same "thing".

If interested, take a look at this (and related) papers: [http://www.cs.ubc.ca/~murphyk/Papers/kv- kdd14.pdf](http://www.cs.ubc.ca/~murphyk/Papers/kv-kdd14.pdf)

<comment><USER4> That makes more sense. Thanks! I'll check out the paper. I was hoping you had some revolutionary new scraping method.

</comment></comment></comment></comment><comment><USER29> YQL is surprisingly quite brilliant:

[https://developer.yahoo.com/yql/](https://developer.yahoo.com/yql/)

</comment><comment><USER30> I'm currently scraping data such as "tweets, comments, likes" a website gets each day so I can graph them over time.

One thing I am having a hard time scraping backlinks to websites. Currently using bing but they are paid after like 5000 queries. I really wonder how other companies like seomoz do this daily against millions of websites.

</comment><comment><USER31> 1\. Monitoring competitors. By monitoring product/service offerings close to my own operations, I can get bizdev people on the phone and speak to partners when I see indications in the public marketplace that someone has a better sourcing deal than I do. Haven't done this in five years or so.

2\. Gathering basic data that should be freely available anyway (like currency exchange rates, global weather, etc.). Always this is done carefully and with a light touch, with maximum respect for load inferred on targeted systems. Again, haven't bothered in about five years.

3\. Automating content acquisition. For search engines, media libraries, etc. This is more like ten years ago. These days there's so little call for it... maybe if I ran a boutique hotel chain in a copyright-isn't-respected jurisdiction and wanted to provide a fat library of in-room entertainment...

</comment><comment><USER32> I just started getting into scaping (mostly been using import.io) mostly because it is a complement to what I really care about - data visualization. I've gotten a ton of interest for my side project and despite that I haven't opened the beta I'm still worried that it won't be as lucrative as creating some niche reporting services for various verticals (real estate, auto, etc). Essentially data that is very tabular and not hierarchical or qualitative. You can think of my work as pivot charts on crack. If someone already pre-compiled this data, I'd much rather pay for it that do it myself. My value add is the analysis/viz done on top of the data. If you want to chat, feel free to email me, contact info is in profile.

<comment><USER33> Machete.io is awesome!

<comment><USER32> Thanks!

</comment></comment></comment><comment><USER34> In terms of ideas, how to scrape Javascript-heavy sites. This one has broken me and the Import.io helpdesk: [http://www.usrentacar.co.uk/](http://www.usrentacar.co.uk/). I'm now trying in CasperJS/PhantomJS but no joy there either.

I'm looking to buy a house, and not all local estate agents post to Rightmove (or some post with a 24-hour delay). Trying to submit the search form on the agent's own hideous website, parse the results and get a standard structure between them all is hideous - I gave up in the end.

Once I have the data the challenge is then analysing it (geolocation, how long are the commute times, distance to amenities etc) which is its own separate challenge

</comment><comment><USER35> I made a javascript based automated scraper in a win7/vista desktop gadget. It was originally for dislplaying the remaining credit on my mobile. You put in search terms and a website and it scrapes it and tries to return what it thinks you are looking for (weather, stock prices, remaining blance etc). It works ok. I think there is definetly demand for a well made scraper/alert app/service though.

App is here [http://robotification.com/creditile/](http://robotification.com/creditile/)

Also didn't yahoo make a thing for scraping this 'Yahoo Pipes' or something.

</comment><comment><USER36> I scraped around hundreds of the major universities of US for one of my client. And he used the data to build a mobile platform for students which integrated different services. I mainly grabbed course, class routines, bus route and email of both professors and students. I still have around one and half million of email address of academics.

I also did some e-commerce information scraping.

One of the most interesting one was for a data selling company. They asked me to collect data of geo information, disaster, finance, tweets etc. We used to apply ML and statistics to give forecast with historic data.

</comment><comment><USER37> What about [https://www.kimonolabs.com/](https://www.kimonolabs.com/) ? Makes it pretty easy to collect data and presents in a structured format (JSON).

</comment><comment><USER38> I have a side-project which scrapes play-by-play data from NBA games to gain more insights into these games.

Here is an example of the (un-finished) side-project: [http://recappd.com/games/2014/02/07](http://recappd.com/games/2014/02/07)

I'm far from the only person scraping this data. Look at sites liked [http://vorped.com](http://vorped.com) and [http://nbawowy.com](http://nbawowy.com) for even better examples.

</comment><comment><USER39> Scraping really is a quite complex process, and not everybody does it right. Do you employ a (distributed?) crawler pool? What if a scraped page goes offline (404/410)? And, how do you handle network errors, and 403's / getting caught (and possibly blocked) - if at all? Do you conceal the scraping by employing a fake user agent? Do you (sometimes?) request permission for scraping to relevant webmasters? These are the things that can make it or break it IMHO.

<comment><USER39> BTW I write tailor made PHP+MySQL scraper scripts, targeting English or Italian language sites; contact me for more info :)

</comment></comment><comment><USER40> A while ago, I had the idea of creating a travel site that catered to the group of people that enjoy traveling but aren't bound by time (i.e. I want to go to X, but I don't care when -- just show me the cheapest weekend for the next 3 months).

Anyway... it turns out that flight APIs are ridiculously non-existent. I ended up scraping two different airline sites, but since it was against their terms, I never took the site any further.

<comment><AUTHOR> The hospitality and travel industries are very slow to update their technologies. I used to work with Ritz Carlton and St. Regis and even those brands are practically in the stone age, so I can't imagine how scraping for flight info would go.

I've thought of even building a simple event aggregator for some friends in the industry and they are blown away that it's possible. Then I remember how many venues are in cities like Charlotte and San Francisco and realize why these industries lag in technology. There just isn't a large pool of developers who want to solve their problems.

Do you have any projects you are currently working on?

<comment><USER40> Completely agreed on the "lag in technology" \-- couple that with the fact that even if you do manage to book a flight as an affiliate, most providers only give you a flat fee (as opposed to hotel affiliates which tend to give a percentage).

Disclaimer: Those claims were relevant the last time I investigated (>1 year ago). It could have changed by now.

On the plus side, building out the flight tool really endeared me to hacking on things - which eventually lead me to leave my last job and co-found something new. :)

</comment></comment></comment><comment><USER41> Frankly I'm annoyed to see this topic here. Most people who have taken to scraping are low-life scum. They see content that others have spent months or years producing, simply set up a site that aggregates all of that information, and then sit back and collect revenue from ads or reseller links you paste everywhere.

People who put in a few hours of work to take advantage of other people's hard work piss me off. :/

<comment><USER42> Are you against hyperlinks too? Seriously, computers naturally make content reusable. Try to imagine the next level when we don't depend on hoarding. The semantic web imagined this, though it was too complicated. But the idea still has huge benefits; every web site a linked database, with content precisely described. But I think many orgs are too afraid they don't really have something to offer in the big picture (that's what so much of business is about).

I do a lot of work in scraping, but it's for non profit healthcare, academic, and general knowledge augmentation. It's painful, but the only way to get to the next level without waiting a thousand years for everyone to make their own consistent API and metadata descriptions.

</comment></comment><comment><USER43> I used to scrape web for a daily deals search engine i wrote for a client in 2010. But we scraped in realtime as the number of sites were really low (in 10s).

pre-crawled copies with distributed processing platform could be cool. you could come up with a better search engine with programmable rules that are edited collaboratively (like wikipedia)

</comment><comment><USER44> I've used Mozenda for Web scraping. They have a free trial and can scrape some complex formats, like drilling down several levels in a Web site or database. They can also parse PDF's.

See [https://www.mozenda.com](https://www.mozenda.com)

</comment><comment><USER45> I love scrapping and even made a subreddit for the purpose where I showcased few of my public work. Any scraping lover can join in.

[http://www.reddit.com/r/scrapingtheweb/](http://www.reddit.com/r/scrapingtheweb/)

</comment><comment><USER46> We do so to determine new pages on websites within our industry. Often the new information here is not formally announced or done so only weakly. We regularly uncover valuable new info about company developments and changes.

</comment><comment><USER47> I currently scrap my lending club account to automatically trade loan notes on the secondary market. This way I can buy/sell notes that satisfy my criteria. If anyone is interested in this, I can send you the scripts.

</comment><comment><USER48> I scrape for Google Playstore for app data, top ranked apps, etc. Unlike iTunes, they don't have a public API/feed. So, in order to get the data I require, I scrape playstore on a pretty regular basis.

</comment><comment><USER49> I scrape Google to save search results.

[http://www.labnol.org/internet/google-web- scraping/28450/](http://www.labnol.org/internet/google-web-scraping/28450/)

</comment><comment><USER50> PlainSite ([http://www.plainsite.org](http://www.plainsite.org)) uses about 20 different scrapers/parsers to download and standardize legal materials.

</comment><comment><USER51> I scrape craigslist for side by side comparisons of stuff I want to buy from there. Eg: Cars, motorcycles, etc. Maybe even real estate would be a good target.

</comment><comment><USER52> Sports scores, statistics, etc are always high in demand for scraping and great to get people interested when learning scraping techniques.

<comment><USER53> Yeah, very much agreed. I have to scrape schedules together to populate games for my ticket site ([http://www.boxrowseat.com](http://www.boxrowseat.com)).

The popular sites are wising up to scraping of information, so it's really an exercise in futility. But when it works, it's super rewarding.

</comment></comment><comment><USER54> I am looking for data having list of all universities and associated colleges . I didn't found anything related to it anywhere

<comment><AUTHOR> So like the dmoz.org or higher education?

<comment><USER54> Yes, sort of like that only. I found a good list on [http://www.4icu.org/](http://www.4icu.org/) , and wikipedia too have list of many colleges and universities.

But information about all colleges associated with university is generally available only on specific site of university, and many times those are in pdf.

Having said that, my above requirement(college associated with university) itself is secondary. I found the list of universities at many places , but not the list of colleges.

So for now just getting a list of all college with location is enough for me.

</comment></comment></comment><comment><USER55> I scrape the Hacker News website to get links and numbers of points. These allow me to produce my "top 10" lists.

</comment><comment><USER56> I regularly scrape financial data - historical prices, live quotes, company information, quarterly reports, etc...

</comment><comment><USER57> Reddit and twitter account for tendancy.

but twitter is so vast you may want to categorize account.

But reddit is a good source for a lot of info.

<comment><USER58> Why not use PRAW? It's very mature, useful library using the Reddit API.

<comment><USER57> i will dig this thks

</comment></comment></comment><comment><USER59> I used to scrape a bunch of webcomics to turn them into RSS feeds. I still have one or two running, actually.

<comment><USER60> Try [https://www.comic-rocket.com/](https://www.comic-rocket.com/) and see if it has all the comics you want to read. It supports either RSS, a web-based reader, or an Android app.

</comment></comment><comment><USER61> i'm building a database of games and scores. web scraping has been very helpful.

maybe instead of trying to change up the content, try to change up the method. ie. do a talk on running crawlers/scrapers to seed your database at an interval. (instead of just "scraping").

</comment><comment><USER62> if you could publish a price list for items sold at major grocery chains, I am sure that many people could use it (bonus if it includes aisle numbers)

<comment><USER63> Similar to [http://www.aislefinder.com](http://www.aislefinder.com) and [http://www.supermarketapi.com](http://www.supermarketapi.com)

</comment></comment><comment><USER64> Online personal banking</comment>


Pyweek 4 - Registration Open.  Great coding challenge. - <AUTHOR> http://pyweek.org/ <comment><AUTHOR> I highly recommend pyweek to anybody interested in coding. It's a one week long competition where you and a team of unlimited size build a game from scratch(mostly) in Python. To prevent cheating, or at least make it harder, a theme is voted on and announced at the beginning.

I'm posting this here because I competitions like this fit well with the ycombinator theme of getting something working out the door quickly. If you've never actually tried to produce something complex quickly by burning hard give this a shot. It is very inspirational to be able to look back at the end and see what you've done. If you're planning on doing a web startup I'd also recommend it as a team building activity. The stress is high, the time limiting, and the competition hectic. You'll find out quickly how well you can work with people in a crunch. And you will learn a lot both about coding and the people you work with.

This is the third one that Erik and I will be entering into. It's a lot of fun and I look forward to seeing some skilled competition coming from here.

</comment><comment><USER2> I'm very interested in this. A portion of my background is in game design at Atari and marketing at EA.

How does team selection work?

<comment><AUTHOR> Once you sign up you can create an entry which sets up your team. You just enter any team member names that happen to sign up there as well. Your entire team doesn't have to sign up officially.

<comment><USER2> I'm guessing you have a team set up? Working remotely?

Thanks for the info, btw -- all of this sounds exciting.</comment></comment></comment>


Google Bans Payday Loans Ads - <AUTHOR> http://googlepublicpolicy.blogspot.com/2016/05/an-update-to-our-adwords-policy-on.html <comment><USER2> If only they demonstrated more control over their third party ad networks.

Breaking mobile on purpose, dubious products, etc.

I miss when ads were text only and had to be clever and accurate.</comment>


My spare time project, a simple weather site. what do you think? - <AUTHOR> http://www.sunlikely.com

<comment><USER2> why are you asking for the user's location? you know it already.

<http://goingtorain.com/>

</comment><comment><USER3> I like the design, but the ads interrupt it, so maybe move those off to the margins. Other than that? Excellent!

</comment><comment><USER4> I chuckled at the name--good choice.

</comment><comment><USER5> Ra himself would be proud.</comment>


Las Vegas casino workers prep for strike over automation: 'Robots can't beat us' - <AUTHOR> https://www.theguardian.com/us-news/2018/jun/02/las-vegas-workers-strike-automation-casinos <comment><USER2> Ha. Seems like dealers can be automated rather trivially.

<comment><USER3> Part of the charm of gambling is interacting with the dealer.</comment></comment>


What's So Special About Low-Earth Orbit? - <AUTHOR> http://www.wired.com/2015/09/whats-special-low-earth-orbit/ <comment><USER2> Beyond the information presented, this is really neat to see interactive visualizations with code that can be edited inline.

It reduces the apparent distance between "wow cool visualization" and the code that generated it.

The more people see code regularly, and can connect it with something they understand, the more likely they are to be code fluent.

<comment><USER3> Prolly off topic but: I show my code to my sister who is a professional artist every chance I get. It demystifies the process in her mind and makes me better at understanding what is happening around her.

</comment></comment><comment><USER4> From an energy perspective, as Heinlein pointed out "Once you get to earth orbit, you're halfway to anywhere in the solar system."

<comment><USER5> Except if you want to go somewhere - and then stay there.

<comment><USER6> Once you go somewhere, you're halfway to staying there.

</comment></comment></comment><comment><USER7> I was delighted to learn recently you can get a nanosatellite to orbit for not very much money, and at the same time disappointed to find out that it would not stay up there for very long. Somehow I thought satellites could stay on orbit permanently

<comment><USER8> Yep there's still a good bit of atmospheric drag even at the ISS's altitude. Once you get beyond the atmosphere you get solar wind creating drag too. The ISS is periodically reboosted into a higher orbit by engines on one of the Russian modules.

<comment><USER9> Also, small satellites have much higher surface area to mass ratios.

<comment><USER8> Yeah that's another big factor in how fast they decay. The fact that they deorbit easily is good though, if they stayed up for a long time it'd be harder for NASA to approve so many so easily because they'd become a threat to LEO operations. A lot of them are launched from the ISS.

<comment><USER7> How do the approvals work if you are not in the US?

<comment><USER8> The ESA and JAXA both seem to run approvals too. I'm not sure about their current plans/proposal request statuses the only ones a surface googling finds are several years old so they might be in between proposal cycles.

</comment></comment></comment></comment></comment></comment><comment><USER10> I'd be interesting to see some example orbits re-scaled so that instead of showing physical distance, their "altitude" is actually the energy necessary to establish and maintain them for a reasonable period of time.

<comment><AUTHOR> Someething like xkcd's Gravity Wells ([https://xkcd.com/681/](https://xkcd.com/681/)) but for orbits would be neat indeed.</comment></comment>


AI system for granting UK visas is biased, rights groups claim - <AUTHOR> https://www.theguardian.com/uk-news/2019/oct/29/ai-system-for-granting-uk-visas-is-biased-rights-groups-claim <comment><USER3> Data-based AI, rather than being neutral, seems to be a tool for amplifying and extending human bias. Whereas a person might occasionally treat a member of a discriminated against class neutrally, whether by accident, oversight, training, human decency, moral bias or a different form of discrimination - <USER2> the other hand, a computer will always do the same thing, and amplify the "people from poor backgrounds are refused entry to places that might increase their wealth".

Police get more arrests where there's more police, so if you use an AI to work out where to send cops, it's going to find you the place your discrimination wants it to.

The modern data-ocracy is just the old eliteocracy with a new coat of paint.

</comment><comment><USER4> The only claim in the article is that nationality affects the process. In what universe is it wrong to take this into account? For example, the considerations when admitting an American into your country are different than admitting an Azerbaijani. There are linguistic barriers, concerns about self- suffiency based on personal financial resources, and even edge-case concerns like the prevelance of militant groups. There are also unequal costs to the UK to do follow-up investigations in fully modernized countries vs. others, and greater costs to countries with less common official languages, given that the number of staff who are fluent in Azerbaijani is lower than those speaking English, Spanish, French, etc.

And all of this is separate from the objective fact that, on average, a random American will be better educated, more culturally in sync, more able to participate in Civic society, and capable of making a larger net positive contribution economically.

So, it is easier and more beneficial to admit an American, why are we trying ourselves in knots, engaging in humanist handwringing about the fact that the process acknowledges this fact in practice? Immigration isn't welfare.

The goal isn't, or shouldn't be, the welfare of all human beings. It should be the best outcome for the nation whose policy is in question.

In other words, one man's 'bias' is another man's common sense.</comment>


Mac App Store in dire need of in-app purchase ability - <AUTHOR> http://alexknight.net/blog/2011/1/8/mac-app-store-in-dire-need-of-in-app-purchase-ability.html

<comment><USER2> Agreed. It was something the iOS app store needed as well. Obviously it's in the works, I'm sure it won't be too long.

<comment><AUTHOR> It certainly would be incumbent upon Apple to do it. It's just early days for the Mac App Store. Next 6 months should be interesting!

</comment></comment><comment><USER3> Need something like gamecenter, and the ability for current customers to transfer a license or something like that.

It would be nice on both app stores (ios/mac) to have upgrading pricing for apps.</comment>


Simon Peyton Jones – Elastic sheet-defined functions [video] - <AUTHOR> https://www.youtube.com/watch?v=jH2Je6wUvPs <comment><AUTHOR> For those more interested in reading than watching a video, here is the detailed 67-page paper about this subject that was recently accepted for publication:

 _Elastic Sheet-Defined Functions: Generalising Spreadsheet Functions to Variable-Size Input Arrays_

 _Sheet-defined functions (SDFs) bring modularity and abstraction to the world of spreadsheets. Alas, end users naturally write SDFs that work over fixed- size arrays, which limits their re-usability. To help end user programmers write more re-usable SDFs, we describe a principled approach to generalising such functions to become elastic SDFs that work over inputs of arbitrary size._

 _We prove that under natural, checkable conditions our algorithm returns the principal generalisation of an input SDF. We describe a formal semantics and several efficient implementation strategies for elastic SDFs._

 _A user study with spreadsheet users compares the human experience of programming with elastic SDFs to the alternative of relying on array processing combinators. Our user study finds that the cognitive load of elastic SDFs is lower than for SDFs with map /reduce array combinators, the closest alternative solution._

[https://www.microsoft.com/en- us/research/publication/elastic...](https://www.microsoft.com/en- us/research/publication/elastic-sheet-defined-functions-generalising- spreadsheet-functions-to-variable-size-input-arrays/)</comment>