Ask HN: Expat founders in the UK? - <AUTHOR> Does anyone have experience starting something as an Expat in the UK? Are the hurdles substantial? How do the entrepuener visa requirements end up working? Does the government need to see business plans and financials?<p>Im about to embark on a project, and trying to figure out if i should try where i live (London) or return to my country of citizenship (US). The project's target audience is not country specific, so i dont think that matters terribly. <comment><USER2> Honestly you'll probably be best served reading the border agency's website and consulting with an immigration solicitor.

Immigration law in the UK changes every year. What was true for someone else may not be true for you. Never mind that your extact immigration circumstances will most likely not be the same as anyone elses.

<comment><AUTHOR> I've been in the UK for a few years, and completely recognise that. I also have found that most solicitors gthat i've contacted are geared towards handling applicants from large corporates, or fight battles for people looking to immigrate from low income countries (and do so in mass quantities). i have yet to speak to someone who has any experience dealing with entrepeuner visas. Hence -- looking to find someone in a similar situation who has read all the same documents to sanity check my views.

<comment><USER3> What visa are you currently on, do you have indefinite leave to remain (ilr)? Up until a couple of years ago if you had been in the UK for five years you could apply for citizenship depending on if you had ilr and then of course you would have no limitations to what you do as a founder.</comment></comment></comment>


Elon Musk Invites Hackers to Break Into Tesla Cars - <AUTHOR> http://versus.com/en/2014/08/11/elon-musk-invites-hackers-to-break-into-tesla-cars

<comment><USER2> This article ends on a tech doomsday/fear message? What is the point of that? Outside of that, this is non-surprising - the man thinks. I'm not sure he does much publicly without some sort of plan.</comment>


Show HN: Visually Analyze Your Pages and Elements with Chrome Extension - <AUTHOR> https://browsee.io <comment><USER2> What are the main differences between Hotjar and Browsee, apart from pricing?

<comment><AUTHOR> Thanks for the question. Hotjar is a very good product. In Browsee, we are attacking the unsolved problems like

1\. Too many sessions to watch - We mark sessions with AI tags like rage clicks, n/w errors, and slowness, u-turns, repeated events, etc to give you pinpointed sessions to watch

2\. Easy querying for Page URL and Elements - You can use our Chrome Extension to watch sessions and analytics for any URL or button or link just by selecting it within our Chrome Extension instead of searching for complex CSS elements.

3\. Identify Your Users - You can identify your users with your Database ID etc to know exactly how your users are using your product.

4\. Actions - You can also target your users with segmented notifications.</comment></comment>


Vagrant Share - <AUTHOR> http://www.vagrantup.com/blog/feature-preview-vagrant-1-5-share.html?utm_content=buffer080fe&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer <comment><USER2> Sort of neat.

We already share our vagrants within our office as we port forward from the host computers to the guest. Thus to access bob's vagrant I just go [http://bob-computer:3120/](http://bob-computer:3120/), etc... We use this feature periodically and it is useful in those cases.

I guess this new feature is mostly for remote collaborations where you are not in the same secure network.

PS. Vagrant is one of the most reliable parts of our [http://clara.io](http://clara.io) development infrastructure. We absolutely love it. I was trying to think of issues I have with it and I can not think of any.

<comment><USER3> I know this isn't the right place for it but here we go: it's a total PITA when somehow the VM connection gets severed and "vagrant ssh" won't work and "vagrant shutdown" won't shutdown cleanly, so you have to kill it by hand and then reboot MANUALLY in virtualbox because the VM gets stuck in the grub menu after a non-clean kill. So then you have to wait for it to boot completely, then shut down normally and then run vagrant up again.

That connection dies for me periodically, like a few times a week. Anyone manage to solve that one?

<comment><USER4> If i somehow can't connect to the VM anymore "vagrant halt" followed by "vagrant up" usually fixes that. Doesn't happen alot though.

<comment><USER3> Halt breaks too, though!

<comment><USER5> I run into similar problems. Seems to happen when I disconnect and then reconnect to a different network.

<comment><USER3> I think it may be this for me too, I code at work and then carry the laptop home to finish up sometimes.

</comment></comment></comment></comment></comment></comment><comment><USER6> It _looks_ like the SSH option sends the encrypted private-key to anyone who asks (via the "vagrant connect --ssh" command)... so they could do an offline brute-force attack.

Thus, even though there's some public-key encryption being used, this is ultimately _less_ secure than even allowing remote logins with just a password (because you can't observe and throttle failed attempts).

Even with the keypairs being regenerated each time, or the "\--ssh-once" option, there's likely a tendency to use the same password over and over with the same collaborators. If so, once a third-party captures one of the ephemeral keypairs, the offline brute-force could be used to discover the reused password, effective for compromise in the future despite the key- refreshing.

So be aware of this, and don't think the ephemeral-keypairs and public-key aspects mean you can use/reuse weak passphrases over long periods with the feature. The passphrase is the major security measure, and appears vulnerable to unthrottled offline brute-force attack.

<comment><USER7> Yes, you're right. But this is assuming a lot of things which are still really hard:

* The person would've had to retrieved the private key before. This private key is only ephemerally written to disk to `ssh` and is deleted after, otherwise being stored in memory only a brief period of time. Additionally, when it is sent down over the network, it is done so over a TLS connection. So this means they would've had to somehow intercept that.

* The person, as you said, would have to brute force the password. PERHAPS the easiest part if people are using easy passwords (we can assume they are, for this comment).

* A subsequent share name with SSH enabled. The share names created (happy-panda-1234) are in a set of 40,000,000 possibilities (for now). They probably won't guess that, especially since shares are only active 1 hour. They could try to intercept the share name, but we send it down over TLS, so they'd have to do a MITM there somehow. Unless the person is running OS X 10.9.{0,1}, not easy.

* And, if the sharer is using --ssh-once, they'd have to do all the above and connect in before the intended connector is in.

This is all to say that while you're correct, I imagine there are much easier ways to attack someone.

And this is just the state of things as Vagrant Share is right now! We already are working on and completed many additional features that we're simply not launching Vagrant 1.5 enabled with:

* You'll be able to restrict access to a share based on organization or username. In this case, to retrieve the private key, you'd need to have an API token for the user in question.

* We're adding detailed audit logs of connections, connection attempts, and more. The person sharing will be able to see who connects and when.

* You'll be able to run a private installation of the server-side component. In this case, you'll be able to run it within a firewall that requires perhaps a corporate VPN to access. Another layer to the onion.

And that is just changes that affect the SSH portion. We have more coming for HTTP and others.

Again, I want to say that given all the right prerequisites, you're right. But we did a lot to thwart this sort of thing, and we're not stopping there. Sharing anything over the internet always has a risk attached to it, and we're trying to mitigate any potential threats as best we can.

<comment><USER6> Thanks for the extra info!

The unpredictable share-name likely adds the most important short-term barrier – essentially an extra random passphrase that's not offline-brute-forceable – especially if your server is logging/throttling scans of the share-name space.

Do I understand correctly that the 'share' command doesn't exit until the session ends? If so, it might be nice to report all the connections/attempts at each session end, so checking another log at a remote console isn't required to notice an anomaly.

The security may be fine for typical uses, I just wanted to help people understand that the password chosen is potentially leaked (in encrypted form) to third-parties... so shouldn't be something too weak/common/widely-reused.

</comment></comment></comment><comment><USER8> I noticed the share funcionality uses a Vagrant URL (vagrantshare.com). Sure enough, when I scrolled to the bottom there was mention of a web service. I imagine this web service will be monitized as it just makes a lot of sense, but hoping that single devs can still demo their projects to friends without having to pay a fee.

If my above assumption is true, would a Vagrant Share Enteprise be in the works/future, similar to Github Enterprise?

Thanks for the awesome tools HashiCorp!

<comment><USER9> It's most likely being used to generate a nice URL that does dynamic DNS forwarding to the actual server.

One could probably get away with just using an IP and a port, but within an organization that's behind a firewall it's not always immediately obvious which IP and port to use.

</comment></comment><comment><USER10> This is exactly why widespread adoption of IPv6 would make everyone more productive. Why doesn't every Vagrant box have an IPv6 address that you can expose by opening up firewall rules? Well, aside from ISP's being slow and developers not spending the 40 minutes necessary to set IPA tunnel, VirtualBox does not support IPv6 addresses on bridged interfaces if the host interface is a wireless card. Sad day all around.

Edit: in fact I wonder if instead of building yet another IPv4 traversal tool, new applications that require P2P connectivity should actually start building it as an IPv6-in-something tunnel. There are plenty of ways to do this including AYIYA which while not fantastic would actually work really well here.

<comment><USER4> Why? Even now, you can just forward the guest port to the host and expose it to give people access.

Vagrant share makes the process of doing this super easy which is great, because you don't need to do any of that manual trickery.

<comment><USER10> Several reasons. First, you are likely behind a NAT. Sure, you can share your Vagrant box with people in your office, but you can't share them if you are working remotely. This has actually bitten me on my team (I work remotely) where I needed to debug a Puppet issue on a coworker's VM and had to try to ssh to his box. What a pain.

Second, port forwarding is fun and all, but why bother when you can do things properly and directly. Your host machine can still control everything and you won't need to do anything manual. All the manipulations that are required can still be hidden behind the simple `vagrant share` command. You can even combine this with a dynamic DNS entry so that you could share not just an IP address but a simple name.

Third, it would be less effort do support this, than to maintain yet another NAT traversal service.\

Fourth, this would speed up adoption of IPv6, something we will all benefit from in the long run.

</comment></comment></comment><comment><USER11> So basically localtunnel ([http://progrium.com/localtunnel/](http://progrium.com/localtunnel/)), but integrated into Vagrant? Plus some SSH magic.

<comment><USER12> [https://localtunnel.me/](https://localtunnel.me/) does the same thing but with ssl for free.

Likely the first to market was [http://showoff.io](http://showoff.io) ( pivoted to [https://forwardhq.com/](https://forwardhq.com/) ), promoted by [http://pow.cx](http://pow.cx)

</comment></comment><comment><USER13> Just the other day I was telling a friend of mine how having the ability to share my Vagrant dev environment was the only missing from what I would consider my dream development workflow.

Edit: I got so excited I even forgot to thank Mitchell for all the hard work. Vagrant completely changed my (work) life. Thank you a thousand times, Mitchell!

<comment><USER14> How would you use this? I'm racking my brain trying to figure out what this would be used for. I just shared my vagrant file.

<comment><USER13> Well, this came to my mind because I'm developing a new responsive web app and wanted to test it on both my iPhone and iPad. There are several ways around this but I guess this makes it so much easier.

Also, as noted, seems great to let your colleagues see what's going on with your work.

<comment><USER2> > I'm developing a new responsive web app and wanted to test it on both my > iPhone and iPad.

This is already easily done just by configuring port forwarding from your host to your guest in your vagrantfile.

e.g.:

config.vm.network :forwarded_port, host: 8080, guest: 80

<comment><USER13> I'm not sure that's the way to do it. I mean, I do use forwarded ports on my projects and I also configure my private network for each one but at least on my current machine it doesn't allow outside access unless I mess with the router and the firewall. A little bit easier would be to configure a network as "public_network" on the Vagrantfile. I assume this works for most people but, again, my machine doesn't vagrant up if I have a public_network configured. I never tried to figure out why, it's not something I had to care about that much to be honest.

So, as you can see, if vagrant share works as announced it would make it much easier for me for that particular task.

</comment></comment></comment></comment></comment><comment><USER15> Feature request for the Mac OS Vagrant port: please don't copy your executable into /usr/bin when running the installer (and that seems to be the only reason the installer exists at all — it would be better distributed just as a .zip containing the Vagrant directory; I can take it from there). /usr/bin is for system-provided executables, not for applications. I understand the wish to provide an easily-accessible command line executable, but Vagrant users likely know how to modify the path or make an appropriate symlink.

<comment><USER7> We used to do that, but we received SO MANY complaints that it wasn't instantly available on the PATH, that we symlinked into /usr/bin.

An alternate approach: Mac has some way to hook into shell profile loading so we should be able to inject a script somewhere that amends the PATH. This turns out to be a little crazy on Mac for some reason I don't remember, but should be possible.

Surprisingly, things are better here for Windows: Windows has an API call to both set environmental variables AND send messages to applications to refresh their environmental variables! So, Windows people have somewhat nice things.

Anyways, I'd love to improve this. The installer generators are open source, would be happy for any contributions: [https://github.com/mitchellh/vagrant- installers](https://github.com/mitchellh/vagrant-installers)

<comment><USER16> Well I mean what's wrong with adding a file to /etc/paths.d which is picked up by the default shells using /usr/libexec/path_helper

It works for Mactex without fucking with the system folders.

</comment></comment></comment><comment><USER17> Cool addition to vagrant. It reminds me a lot of another useful tunneling utility written in Go: [https://ngrok.com](https://ngrok.com). Maybe an inspiration for it? And also, an interesting related deck: [https://speakerdeck.com/inconshreveable/gosf- decentralizing-...](https://speakerdeck.com/inconshreveable/gosf- decentralizing-the-web-again)

</comment><comment><USER18> I helped build something like this for developing Facebook apps. I'm happy to see that the service is still running at [https://tunnlr.com/](https://tunnlr.com/). It's a good tool, and I think it makes sense as a Vagrant plugin but I'm surprised this is becoming part of core.

</comment><comment><USER4> This is great! What i am still missing is sharing a vagrant vm across multiple machines i work with easily. Basically to set this up i have to set up a base box and bring that up on another machine and then manually switch the hd uids so that they are in sync. Otherwise dropbox sharing my vagrant workspace will sync a different id and on the other pc vagrant up wont find the existing vm and start a new one.

Also when i install a new service inside the vm on one machine, i have to repackage the vm and reinit it on the other machines later.

Kind of a pita, has anyone an elegant way to solve this ? Can't believe no one is working with multiple machines at work/home.

</comment><comment><USER19> Will this also be open source so I can run my own proxy?

</comment><comment><USER20> I was using ngrok before, but I guess it's much easier to use vagrant share.

</comment><comment><USER21> that is an awesome idea, cant wait.

</comment><comment><USER22> There are lots of vagrants, you don't have to share, everyone can have one or two all of their own. But if you choose to share I'm sure this is the best way.

Hopefully they expand in to Hobo's, Bums, and the underemployed.</comment>


Hailo brand drives off into sunset as MyTaxi arrives - <AUTHOR> https://www.irishtimes.com/business/technology/hailo-brand-drives-off-into-sunset-as-mytaxi-arrives-1.2996766#.Wjp4-EBEnns.twitter <comment><AUTHOR> With all the chatter around Uber today, I thought it poignant to remind people about how things could be done differently.

Apropos of this article, the _MyTaxi_ rebrand amounted to the complete destruction of what was a very healthy experience for both customer and driver.

Hailo did what Uber did, but stopped short of trying to eat the whole cake and instead sought to enhance the market that was there by joining up customers to drivers.

This worked amazingly well since you had all of the delightful customer friendly features you get with Uber without the cognitive dissonance that goes with making an established driver's family go hungry.

It was good for the drivers as well because it connected them to a "certain type" of customer and they were even allowed to discriminate based on geography or perhaps even some undisclosed customer rating feature (I honestly don't know if this would be a thing but I believe Air B&B has something similar so why not)?

It was a win win all around. A non-zero sum of technology and business that unlocked an increase in growth in the market without inconveniencing anybody.

I miss Hailo and I hope somebody comes along with a worth successor soon. Maybe [http://www.whistletaxiapp.com/](http://www.whistletaxiapp.com/) is that App? I don't know. I have no affiliation with these guys but I really want somebody to come along and rebuild this market.</comment>


Glorious silicon globes could hold key to elusive PERFECT kilogram - <AUTHOR> http://www.theregister.co.uk/2012/11/20/quantum_kilogram/

<comment><USER2> I can't believe this encyclopedic article fails to mention the fact that one of the reasons for wanting to abandon the physical standard mass is that its mass has measurably changed over the years, and that the most likely reason for the change is that cosmic rays impinge on the standard mass and collide with particles within it, which turns a tiny amount of mass into a tiny amount of radiated energy, which is radiated away.

The wish to abandon a physical artifact is well-founded regardless of the proximate reason, but the most commonly stated reason deserves mention.</comment>


Building an Ingress table using flow-based programming - <AUTHOR> http://bergie.iki.fi/blog/ingress-table/

<comment><USER2> Source code at [https://github.com/c-base/ingress- table](https://github.com/c-base/ingress-table)

Feel free to ask any questions about the project!

</comment><comment><USER3> Would you eventually make a BOM of all the hardware used? I can see a lot of local Ingress communities being interested in making their own version of the table.

<comment><USER2> Yeah, we already got some contacts about that. I think we'll add it to the repo README

</comment></comment><comment><USER4> Neat. Is there any chance to see it during the Recursion event? I'm planning to visit Berlin :)</comment>


Cross shard transactions at 10M requests per second - <AUTHOR> https://blogs.dropbox.com/tech/2018/11/cross-shard-transactions-at-10-million-requests-per-second/ <comment><USER2> Ok, while we're at this, I think that the most important part of the post was this:

> Edgestore data was already well-collocated, which meant that cross-shard > transactions ended up being fairly rare in practice—only 5-10% of Edgestore > transactions involve multiple shards.

What if performant cross-shard transactions is a red herring, and the thing that we should be looking more into is reliable automatic data colocation to avoid performing cross-shard transactions as much as possible? There's decent amount of academic research around this, with projects like SWORD [1] and Schism [2] that study shard load balancing as a problem of hypergraph partitioning. It seems like this might be worth incorporating into commercial distributed database projects.

[1] [http://www.cs.umd.edu/~amol/papers/edbt13.pdf](http://www.cs.umd.edu/~amol/papers/edbt13.pdf)

[2] [http://www.vldb.org/pvldb/vldb2010/papers/R04.pdf](http://www.vldb.org/pvldb/vldb2010/papers/R04.pdf)

<comment><USER3> Thanks for the references -- will take a look.

Edgestore's API is set up to shepherd users into good collocation patterns by default, and a lot of work over the past year or two went into improving collocation and educating users about best practices. The collocation efforts were actually orthogonal to implementing cross-shard transactions, but they were obviously very beneficial.

<comment><USER2> Thanks -- just curious, am I correct to interpret this to basically mean that thus far, the performance of the system basically relies on users to explicitly define colos nicely within their application-level data model?

For some reason this reminds me of something like the entity group concept in Google's Megastore [1].

[1] [http://cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf](http://cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf)

<comment><USER3> There is somewhere between where we are today and a completely uncollocated free-for-all where the system would fall over. There's a separate axis of the rate at which users request transactions (with locks) versus non-transactional writes using optimistic concurrency control that would come into play sooner. Our guidance is therefore that users try to reserve transactions for when there's correctness critical reasons why two objects need to be updated together and rely on asynchronous primitives to handle "eventually consistent" mutations.

</comment></comment></comment></comment><comment><USER4> I have witnessed first hand companies who decide to do cross-shard transactions when in fact they don't really need it.

For example, imagine that you have a distributed key-value store. You want to users (on different shards) to either both be able to see a piece of content or neither. You can achieve this by allocating a key on either shard (or a third shard), writing a reference to each user's shards. If all of those writes succeed, you can write to the new key which was allocated. If any writes fail, you can bail and your data is consistent. Wrap the above logic in a nice library and your service will scale horizontally.

<comment><USER3> For us (Dropbox), the threshold ended up being multiple product teams having to implement ad-hoc two-phase commit over their datatypes and burning engineer hours not to implement it but to prove that they had gotten it right and to handle the clean up after any unsuccessful writes.

You're right that most systems probably don't need 2PC, which is why Edgestore didn't include it until now. As mentioned in the post, we finally felt that we had reached the right balance of tradeoffs to justify the API-level primitive.

<comment><USER5> When you get to big enough scale, you need cross shard transactions. Google reached it with Spanner.

If I look at Dropbox, i am sure features related to sharing folders between people/organizations cross-shards. You can't avoid them if you want to offer a fully featured product.

After a quick skim of 2PC in Edgestore (sorry, no time), it is unclear if there is a single transaction coordinator (TC) or not. I assume it is a single TC - that you can scale it to 10m trans/sec is impressive. The really hard part is have multiple TCs and to design protocols to coordinate their recovery after failure. Here is a good example in the open-source NDB (MySQL Cluster) system - [https://drive.google.com/file/d/1gAYQPrWCTEhgxP8dQ8XLwMrwZPc...](https://drive.google.com/file/d/1gAYQPrWCTEhgxP8dQ8XLwMrwZPcXHW_1/view)

<comment><USER3> We have a routing tier of a few hundred machines. Any of those can serve as transaction coordinator. This is one of the places where having a transaction record is useful -- in general, our locking and non-transactional read scheme will wait nicely for a 2PC transaction to complete, but in the face of failures, any actor in the system can abort the 2PC transaction by marking the transaction record. There's also a really nice optimization for non- transactional reads that allows them to read even in the event of a staged but not yet committed 2PC transaction (you can prove to yourself that a linearized read can occur on either side, if the 2PC transaction is still pending when the read begins).

</comment></comment></comment></comment><comment><USER6> The idea of creating an abstraction layer over thousands of MySQL nodes is a novel one, and should be commended. I like the idea a lot, and by DropBox's scale and success, it must work well.

I would be interested in a granular point-by-point comparison between this layer and native MySQL replication. In other words, if someone were to rewrite MySQL replication such that it did everything that this abstraction layer did (in addition to replication), what would it need to do?

Now (and this is strictly academic/theoretical), I'm curious what would be necessary to modify in the abstraction layer if the abstraction layer was to support a whole bunch of disparate SQL databases underneath it, i.e., Postgres, SQLite, SQL Server, Oracle, etc. (No, that wouldn't be practical, but it would be an interesting exercise to really learn where the gotchas might be where working with different SQL dialects...)

<comment><USER2> Vitess [1] is a similar kind of abstraction layer that's compatible with the native MySQL wire protocol. It also supports atomic cross-shard transactions using 2PC.

[1] [https://vitess.io/](https://vitess.io/)

<comment><USER7> Does anyone know of a vitess equivalent for Postgres? I'd love to use it

<comment><USER2> Citus seems to be doing pretty well.

</comment></comment></comment></comment><comment><USER8> Dropbox is very easy to shard though, the users are mostly independent. Take something like Facebook, and it becomes much much harder - anybody can friend anybody, anybody can message anybody, anybody can like or comment on public posts.

<comment><USER3> That's what we thought, too. (Un)fortunately(?) that's not the case anymore given our product offerings :)</comment></comment>


Ask HN: The new "back to HTML" trend - <AUTHOR> This year, we can see a renaissance of (mostly) JavaScript-less websites and web apps, as an answer to the "Website obesity crisis" [1].
A recent prominent example is that "Hey.com UI is 100% HTML" [2]. Other compelling examples that I've seen are: "It's perceivably faster to load 8.5 megabytes of HTML than it is to load a single tweet with a client-side React app." [3] or LOW←TECH MAGAZINE, a "solar-powered website" [4].
The numerous examples show that this is a trend now. I am looking for other compelling examples and explorations of this trend.
Do you think it is an important trend to copy, or mostly irrelevant gimmick (similar to code golf excercises)?
[1] https://idlewords.com/talks/website_obesity.htm [2] https://news.ycombinator.com/item?id=23536517 [3] https://twitter.com/zachleat/status/1169998370041208832 [4] https://solar.lowtechmagazine.com/about.html <comment> open-source-ux UK government websites and associated services have been designed to work without JavaScript ([https://www.gov.uk/](https://www.gov.uk/)).

They have blogged about this a few times:

 _Building a resilient frontend using progressive enhancement_ (Dec 2019)

[https://www.gov.uk/service-manual/technology/using- progressi...](https://www.gov.uk/service-manual/technology/using-progressive- enhancement)

Here is a much earlier (2013) blog article but still relevant

 _How many people are missing out on JavaScript enhancement?_ (Oct 2013)

[https://gds.blog.gov.uk/2013/10/21/how-many-people-are- missi...](https://gds.blog.gov.uk/2013/10/21/how-many-people-are-missing-out- on-javascript-enhancement/)

</comment><comment><USER2> It's tough to say. Many teams are going on a Graphql adventure, and I'm not sure how we're going to avoid things like React when Apollo is being seen as the default client framework.

These things are not set in motion based out of need from the average developer, we have to examine the trends in conference talks to pinpoint who is controlling the zeitgeist here.

The JavaScript Illuminati is real.

<comment><AUTHOR> While having a GraphQL API, it is totally doable to render a React app on an edge node and send SSR web pages to the client. But such setup rarely seems to be a goal for businesses.

</comment></comment><comment><USER3> Wow.. any resources for this type of architecture?</comment>


Bringing the best of Desktop and Web together - <AUTHOR> http://blog.pullur.com/2007/03/12/combining-the-best-of-desktop-and-web-dekoh/ <comment><USER2> Java Web Start does this. That is why I used it for juwo.

<http://juwo.com></comment>


The Dataist: An Ethical Approach and Best Practice of Data Science - <AUTHOR> https://medium.com/the-dataist/towards-data-industry-best-practice-946c547b001d <comment><AUTHOR> Hi everyone, this is the publication I've just started on making ethics part of the data industry best practice. The link leads to my first article in which I shared the back story of why I started this and why I think it's important for everyone to be aware of the issue and its consequences. As of this writing, I haven't put up the official landing page at thedataist.org. It's gonna be live in 2 days. If you're interested in this discussion, please follow our publication and leave some comments to help us improve. Thanks.</comment>


China Clamps Down on Web, Pinching Companies Like Google - <AUTHOR> http://www.nytimes.com/2014/09/22/business/international/china-clamps-down-on-web-pinching-companies-like-google.html <comment><USER2> More and more I feel like this is pure protectionism and not internal security or censorship.

The Chinese government already has a vice-like grip on the internet and has no qualms about locking up dissidents. We know from the Harvard social network study that they really only care about activism and not criticism. Most of the population has been through the Chinese education system and they are generally speaking apathetic and passive when it comes to politics or the possibility of change. And, don't forget, just like most Western netizens, users are mostly interested in non-political aspects of the web - chatting to friends, getting work done, watching movies online....

The only sense for this blanket ban/slowdown/disruption of big foreign online services seems to be to shelter Chinese companies from foreign competition.

<comment><USER3> That does seem to be the case. This July I travelled to the Hubei and Jiangsu provinces. I couldn't get on Gmail or Dropbox without going through Tor.

But I had no problem opening the zh.wikipedia.org (the Wikipedia in Chinese), which has content that the Party decidedly would not like to see, such as the account of the Tiananmen events. And I could do that with HTTP or HTTPS, without VPN or Tor. I'm guessing they don't block Wikipedia because nobody in the population really cares anymore, and Baidu is not making enough money out of Baidu Baike (Baidu's online encyclopedia) for it to matter.

<comment><USER4> Or perhaps they just edit the chinese version of the wikipedia article to be more sympathetic.

It's not like chinese don't know that tiananmen square happened. They just (generally) don't believe it was wrong.

Also studying the event, it's pretty grey in general. The west's portrayal of it is not without bias.

<comment><USER3> "They just (generally) don't believe it was wrong."

That's not exactly true. The deeper issue is that the Party wants the people to _forget_ the incidence or make it look trivial and insignificant, so that any discussion about right or wrong does not even surface.

In fact, there isn't much that the Party needs to do about it nowadays. The population is literally drowned in entertainment, amusing themselves to death, which is part of the reason the government can start to repurpose the GFW towards protectionism.

</comment></comment></comment></comment><comment><USER5> I've been living in China, and I can definitely feel a shift here. These days, as an English-speaker in China it's almost impossible to get anything done online without VPN. Google, for most practical purposes, is blocked. Using other search engines, a large percentage of sites won't load when you click on them. Considering the scale and importance of China's economy, the Chinese gov. is moving in an ominous direction.

<comment><USER6> My family has done business in China pretty much my whole life, and my take on it is that the Chinese gov. is more or less a bloated system. As the economy grows, greater access is definitely needed and I think top gov. officials realize this, but really nothing can be done in a timely fashion there because of how their politics work.

<comment><USER5> If they realize it, then why is access becoming MORE restricted than ever before? Ever since Chairman Xi came into power, all I can see is more restrictions and tightening of gov. policy.

<comment><USER7> > If they realize it, then why is access becoming MORE restricted than ever > before?

Believe it or not, network filtering is a huge business in China, especially selling products to gov.

</comment></comment></comment></comment><comment><USER8> I'm a Chinese software developer. In China, VPN or other proxy tools are essential for anyone who need look up serious technical materials online. We call it "翻墙"(Go Over the Wall), which is a basic skill.

And about the reason why these sites are blocked: just BECAUSE THEY CAN! (They even blocked github for mysterious purpose once!) There is no public rule/law/principle to tell us why a foreign site is blocked. The just kill sites silently and pretend nobody cares.

Fortunately enough, HackNews has not been blocked, yet.

(Sorry for my bad English)

<comment><USER9> Do you think this harms new engineering students in China when they don't have access to tools and knowledge that the rest of the world does? I can't imagine having Github blocked.

Does there seem to be a pattern to the blocks? Or is it just completely random?

<comment><USER10> Maybe it helps weed out the lesser engineers. Step one is trying to work out how to get the information you need.

<comment><USER11> > Maybe it helps weed out the lesser engineers.

Problem is VPNs require money, not just skills and tor can be dead slow.

</comment></comment></comment></comment><comment><USER12> This isn't really news is it? Putin and Xi both defining what is and isn't allowed to be shared/said/stored in machines on their soil. The next step can start pretty much at any point, its going to look like the old ARPAnet, where a set of protocols, defined by group sharing a common interest in communication, creates a network that empowers new work and new industry. Back in the 70's the big issues were bandwidth and signalling technology, today the big issues are interference from the government. But from an engineering standpoint its just getting signal through the noise. As I mentioned earlier, UUCP through Tor has a lot of potential for creating a pretty decent source routed email network without central control. Add homomorphic encryption on routing/packet management and you have the basis for building the first network within the network.

<comment><USER13> Tor does not work in China. Since their servers' addresses are publicly available, it is quite easy to block.

To access real Internet inChina, it is not only about encryption, nor anonymity. you have to hide the fact that you are accessing a foreign server. GFW has this feature: if a node inside GFW access a foreign address too long/frequently, the foreign address will be temporary blocked. This is a very rude but powerful feature.

</comment></comment><comment><USER14> How is this allowed from a free trade perspective? Not to diminish the human rights portion, but even from a purely financial standpoint how does China get away with this?

<comment><USER15> The same way the US flouts free trade agreements: [http://www.npr.org/blogs/money/2014/01/17/263101422/why- u-s-...](http://www.npr.org/blogs/money/2014/01/17/263101422/why-u-s- taxpayers-started-and-stopped-paying-brazilian-cotton-farmers)

When you're big enough, the laws don't apply to you.

</comment></comment><comment><USER16> Probably because a lot of people use jquery and other frontend libraries hosted by google, and those (of course) have been blocked by some moron in China. And most sites don't properly handle cases when third party assets fail to load.

</comment><comment><USER17> VPNs are sometimes unreliable too, they seem to be getting blocked/shut down quite often, so lately I pay for two VPNs to try and get around these problems. >.<

</comment><comment><USER18> Human beings... We think we are so evolved culturally and intellectually. Just take a look at what's going on around the world. We are but a step away from cavemen. Sad.

</comment><comment><USER19> sites using google fonts are basically unable to load or takes too much time to load too.

</comment><comment><USER20> Spy on the Chinese and this is what you get.

</comment><comment><USER21> The large-magnitude international relationship structure is changing. The new cold war is approaching.

Therefore the internet -- built in 1990s -- will be splitted. Again.

</comment><comment><USER22> There's a tiny bit of substance here, but what's amusing is how out of touch with reality they are on some comparisons. For example:

"You've got people spending their time figuring out how to send a file instead of getting their work done."

This happens in every company with an internet connection.

"That has led to the rise of a number of locally run application stores, which analysts say will sometimes market pirated copies of software or charge extra to promote a new application."

So the piracy problem doesn't happen so much in the west, but the objection here is that simply advertising on Facebook or Google isn't going to get you a global audience because someone has split off part of the market. This is like saying how terribly inconvenient it is that not everyone in the world watches US based TV channels.

<comment><USER23> I live in China (I'm from Europe) and the article is a really good representation of the daily work life here. Without a good VPN service the internet feels like half of the stuff is missing. Ironically the NY Times is also blocked in China.

\- "This happens in every company with an internet connection." Yes, that's right. Now multiply this by 100. As described in the article, the internet is not only censored but also unreliable. Some sites work only sometimes, others break silently. If you work for an international company, the websites and services your western colleagues use, will not work for you (dropbox, fb, google, xing, etc).

\- "This is like saying how terribly inconvenient it is that not everyone in the world watches US based TV channels" No, the whole point of the internet is to be accesible from everywhere. Advertisers on US based TV channels target an US audience. If I release an Android app on the Play Store, I expect it to be downloadable by everyone with internet access.

<comment><USER24> NY Times has been blocked in China ever since the former released information about the ex-premier Wen Jiabao's family fortune.</comment></comment></comment>


Our Stripe Billing implementation and the one webhook to rule them all - <AUTHOR> https://blog.checklyhq.com/our-stripe-billing-implementation-the-one-webhook-to-rule-them-all/ <comment><USER2> The problem I had with implementing subscriptions with Stripe is that the batches of messages from the webhook could come back in clumps in random order. So they would have references to objects that I hadn't been notified about their creation yet.

Theoretically and rarely a webhook could fail and be retransmitted arbitrarily later due to bad weather on the internet, so you have to be able to tolerate that, but practically and often it sent bunches of messages all at effectively the same time, which caused them to be processed by my web server in random order.

I finally gave up trying to structure the code so it could create objects in any order, and deal with objects it hadn't heard about yet, and just treated the webhook callback as a sign that I should soon make a request back to Stripe and ask them for ALL the events that had transpired.

So I'd log all the information in the webhook just for chuckles, then schedule a task that polled Stripe for batches of events, and dealt with them all at once without anything slipping through the cracks because of random reordering in transit.

<comment><USER3> I have run into this also using subscriptions + webhooks. For example, I am using the "charge.succeeded" webhook event to send a custom email receipt on my backend. But, for a few new customers there seems to be a race condition where the "charge.succeeded" webhook event will arrive, before the API call in my code returns a success, so there is now an event but I have no idea how to tie this back to a user. I am using the API to create a new customer and updating their user DB record with the customer token from stripe. So, I get into a situation where I do not know what customer to send the email too (because I don't have the customer token yet). I ended up just adding a HTTP 503 error (Service is Unavailable), for this specific webhook event when I cannot find the user, so the Stripe API will retry that event. This is a hack but it works. This just started happening a few weeks ago. There are a few of these things popping up here and there that I need to deal with but generally it works really well.

<comment><USER4> We just use a database lock for this. Lock on the charge/order/user/whatever, then the stripe webhook delivery is blocked until the transaction commits.

This might seem like a pain, but Stripe have no idea how fast their clients are, there's not much point in waiting, say, 1 second minimum before delivering the webhook, as that only alleviates the problem for clients fast enough to "finish" within 1 second, whatever their definition of finish is.

For this reason, I think Stripe are doing the right thing, and I'm not sure there's much they can do to make it much easier. Once you know it needs to be done, a solution is pretty straightforward.

<comment><USER2> Yikes!

You really should respond to the webhook handler immediately, by queuing it, and handling it asynchronously in another process.

It's bad enough to perform slow operations like creating lots of objects in the database, calling other web apis, sending email, before you return a success to stripe's webhook.

But for one webhook to lock out all other webhooks while it did all that work would only compound the problem.

Stripe tends to send a whole flurry of events related to the same transaction, when there's actually only one thing for you to do in one swoop (create a user and start their subscription), so to processes them one by one is very inefficient, especially if you have to call back to stripe for each event.

<comment><USER4> Sorry, maybe should have clarified, webhooks are typically processed in a queue, we reply to Stripe immediately, but we block the processing of the webhook on that lock. As we're processing in a queue with multiple workers this typically doesn't block much work from happening.

There are some cases where we specifically want to propagate the error to a webhook provider (not in the case of Stripe, so we work inline, but that's rare). There are also some cases where we want to process webhooks on a serial queue, one at a time, to ensure in-order delivery (again not in the case of Stripe).

</comment></comment></comment></comment></comment><comment><USER2> One thing about the Stripe API that I loved was that I could intertwingle it with my own admin interface, by pushing descriptions and metadata properties into stripe objects that had url links back into corresponding objects in my admin interface (like users, products, coupons, subscriptions, transactions, etc).

Though it may not be explicitly documented, Stripe's web site is smart enough to make the urls in metadata be clickable links, which is a godsend.

So I didn't have to duplicate stuff you could do on stripe's site in my own admin interface, I could just link back and forth between them!

</comment><comment><USER5> (Yikes) I feel compelled to remind the folks complaining about webhooks arriving out of order of the fallacies of distributed computing.

[https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing)

For goodness sakes, log webhooks to a queue and rebuild the object graph later by probing their API. I would be absolutely blown away if Stripe recommended that folks attempt to process these inline (simple tutorial-level example code notwithstanding).

EDIT: And for your own sanity, assume 50% of the webhooks you expected to arrive didn't. Schedule a periodic task to scrape transactions from their API.

<comment><USER6> > EDIT: And for your own sanity, assume 50% of the webhooks you expected to > arrive didn't. Schedule a periodic task to scrape transactions from their > API.

I'm pretty sure the Webhooks keep retrying until they get a success response back from your server.

<comment><USER7> They do that but not forever: [https://stripe.com/docs/webhooks/best- practices#retry-logic](https://stripe.com/docs/webhooks/best-practices#retry- logic)

<comment><USER6> They do it for 3 days though which seems more than enough... and you can even manually trigger them from the dashboard. So I'm not really seeing why you can't just rely on Webhooks?

<comment><USER7> You can rely on them. I just wanted to correct what was said in that comment.

</comment></comment></comment></comment></comment><comment><USER8> You just leaked your customers email addresses by improperly obfuscating them.

See [https://web.archive.org/web/20190409072021im_/https://blog.c...](https://web.archive.org/web/20190409072021im_/https://blog.checklyhq.com/content/images/2019/04/image-3.png) With knowledge of the font used (which is very easy to figure out as its the Stripe dashboard) everyone can reconstruct these email addresses.

You should now notify these customers as well as your supervisory authority.

<comment><AUTHOR> Thanks for bringing this to my attention. I will have my legal help check it and take the appropriate steps.

</comment></comment><comment><USER9> Cool post! I don't have a ton of experience using Stripe, but shouldn't you at least be handling some sort of payment_failed webhook?

It looks like you call _createSubscription and set the initial value for currentPeriodEnds before you know the payment actually succeeded, and since you don't ever check or listen for failed payments, anyone could get a free month (or year) of Checkly by using a bad card, or if the payment just randomly fails.

Maybe this isn't a huge deal in the early days, but you and your customer might not even notice the failed payment for quite a while unless you happen to check your Stripe dashboard!

<comment><AUTHOR> This is a great comment. And it's stupid I left this out of the post, as I made a conscious decision to not deal with that now. I should at some stage.

I actually had one failing credit card already, but my customer base in in the 30+ under 100 range, so I easily caught it. Also, it was totally benign from an early customer that just had bodged renewal for their card.

</comment></comment><comment><AUTHOR> OP here. I'm super curious about other folks using Stripe Billing and their experiences. My SaaS is fairly young, so I probably have missed some things.

<comment><USER10> Great article!

Here was a related discussion, last week: [https://news.ycombinator.com/item?id=19556579](https://news.ycombinator.com/item?id=19556579)

<comment><AUTHOR> Oh wow that's a cool post. The comments kinda echo the spirit of my post. Yes, Stripe has a good API and SDK. The actual code is not the problem.

The problem is the whole workflow, how it interacts with your customer and with your business backend processes.

</comment></comment></comment><comment><USER11> I wrote a Stripe subscription integration a couple years ago in Scala. The product never worked out and it's running in free mode now.

For fun, here's the main controller code that processes the webhook events (obviously there is more logic in other files)

I had never worked with a payment service before so there may be mistakes, but it worked well for us for the time it was in use :)

[https://gist.github.com/bschwind/1371a196920981c1dea2b0e4f02...](https://gist.github.com/bschwind/1371a196920981c1dea2b0e4f028e442)

</comment><comment><USER12> Great post! The number of subscriptions that Stripe sends out is really overwhelming at first, and especially the order they can be received in. I ended up coming to the same conclusion that you did, and only end up listening to the invoice payment succeeded and invoice payment failed webhooks.

Doing so greatly simplified the process, rather than listening for customer or subscription updates and changes.

Great to read others are taking a similar approach!

</comment><comment><USER13> Thanks for sharing. Very timely for my team.

</comment><comment><USER14> Why does this one webhook "rule them all"? Because you can scrape by by ignoring Stripe's other webhooks?</comment>


History of T - <AUTHOR> http://www.paulgraham.com/thist.html <comment><USER2> We should point out that Emacs Lisp now has lexical scope. It took about 30 years, but it's here.

[http://stackoverflow.com/questions/7654848/what-are-the- new-...](http://stackoverflow.com/questions/7654848/what-are-the-new-rules- for-variable-scoping-in-emacs-24)

</comment><comment><USER3> Olin mentions the Markup language, which I extended so that I could write my thesis and generate both HTML and Latex. Here's the resulting web version: [http://draves.org/cmu-research/diss/main.html](http://draves.org/cmu- research/diss/main.html)

All the diagrams and charts were made by calling out to external programs like dot and gplot, with the source all held in one big text file, edited with emacs of course.

that was finished in 1997.

<comment><USER3> heh i found it's still available for download with samples: [http://draves.org/cmu-research/markup/markup.html](http://draves.org/cmu- research/markup/markup.html) bonus points for getting it to run.

</comment></comment><comment><USER4> "You could also program the -10 in a beautiful, roughly-C-level language from CMU, called Bliss. I see C and I remember Bliss, and I could weep."

I see C and I wonder if that's the best we can do for a systems programming language. Maybe C is just a local optimum that we have just invested too much manpower into to switch to anything else?

<comment><USER5> > I see C and I wonder if that's the best we can do for a systems programming > language. Maybe C is just a local optimum that we have just invested too > much manpower into to switch to anything else?

At this point the ecosystem around C is too big to justify the cost of switching for the domains it's used in. Any new contender is going to have to offer something new, without giving up any cost in performance or control over the machine.

With Rust (disclaimer: I work on Rust) we're shooting for safety to give us that edge—having the compiler reliably prevent use-after-frees and buffer overflows instead of discovering them through 0-days, without the traditional approaches that sacrifice performance and control over the machine (GC), seems pretty compelling to me. Even if we do succeed in getting enough new projects to use a safe language to make a difference, though, C will still be immortal; once a language has that much staying power it'll be around forever.

<comment><USER6> Let me first say the I applaud Rust and the effort you're making in developing a new, safe systems-programming language. But I do have one concern.

I like this quote by an Intel engineer, which appears on the back of _Java Concurrency in Practice_ : "For the past 30 years, computer performance has been driven by Moore's Law; from now on, it will be driven by Amdahl's Law."

I think that this is the biggest challenge facing software development in this age. Now, I know concurrency is core to Rust, and I know that Rust adopts message passing as the principal concurrency construct. Message passing is terrific, extremely useful, and as close as we've got to a model of programming multi-core that's simple and easy to grasp. But at the end of the day, we do need a way to develop efficient mutable, concurrent data structures. Languages that rely on message passing, like Erlang, usually pass this problem down to a database, but the problem still has to be solved.

You say that languages resorting to GC "sacrifice performance and control over the machine". But GC already provides better throughput than manual memory allocation in practically all circumstances, and suffers mostly from latency issues (pauses), but even those are being worked on with some good progress (like the work done by Azul on their JVM). The only major true sacrifice GC requires is extra RAM, which is becoming ever cheaper.

GC is currently the best way to develop efficient concurrent (shared) data structures. There are ways of writing those without a general-purpose GC (RCU and hazard pointers), but those wither require kernel cooperation, and/or suffer from worse worst-case performance, and/or quickly transform into GC when generalized.

I know the plan is for Rust to have a GC as part of the libraries, but this problem (of developing efficient concurrent data structures) must be addressed. It's possible that hardware has become too complicated for us _not_ to give up some control over to some runtime. Maybe we're at a stage where low-level full-control programming is incompatible with fully utilizing the hardware for best performance. It is possible that low-level systems programming languages will excel in resource constrained environments and have several other advantages, but sheer performance/scalability won't be one of them (at least on server-class hardware).

<comment><USER5> > But at the end of the day, we do need a way to develop efficient mutable, > concurrent data structures. Languages that rely on message passing, like > Erlang, usually pass this problem down to a database, but the problem still > has to be solved.

Rust fully supports concurrent data structures with shared mutable state, and there are several in the libraries.

> But GC already provides better throughput than manual memory allocation in > practically all circumstances

This is far too broad of a statement. I can certainly come up with cases in which manual memory management will outperform GC. For example, if you have an arena-like pattern like the binary-trees benchmark, I think it's impossible to outperform manual memory management. Even if you bump allocate in the nursery then you still have to copy to the tenured generation, reducing throughput over a bump allocator.

> and suffers mostly from latency issues (pauses), but even those are being > worked on with some good progress (like the work done by Azul on their JVM).

Azul C4 generally requires a kernel extension to perform well, reducing its applicability in practice (desktop/mobile software). It also suffers from somewhat reduced throughput over the HotSpot garbage collector, according to the paper. This is not to bash Azul C4, of course—it's a really exciting piece of technology—but I feel that it's often held up as a solution to all of the problems of garbage collection when it, too, has tradeoffs.

> It's possible that hardware has become too complicated for us not to give up > some control over to some runtime. Maybe we're at a stage where low-level > full-control programming is incompatible with fully utilizing the hardware > for best performance.

I don't see this the case in practice quite yet. Java HotSpot, which features the best widely used GC, is routinely outperformed by low-level C++. At this point I see the burden of proving that garbage collection outperforms manual memory management _in practice_ is on the proponents of pervasive concurrent GC. It may well happen, but I don't think we're there yet.

<comment><USER6> > Rust fully supports concurrent data structures with shared mutable state, > and there are several in the libraries.

Are they lock-free? If so, how do you do it without a GC?

> Java HotSpot, which features the best widely used GC, is routinely > outperformed by low-level C++

This is true mostly in single-threaded computations. Also, as another commenter points out, Java's main problem is the lack of arrays of structs which makes locality difficult. This is being worked on (and almost completely orthogonal to the issue of GC), and will hopefully be at least partially resolved in Java 9[1].

My point is that while we need new languages now, we also need to prepare them for a many-core future. Once you have over 100 cores, many locking schemes stop scaling[2]. I'm not saying Rust specifically should think ahead, but I think a new systems programming language should, especially if it's goal is to replace C for the next 40 years. Unless, that is, what I said turns out to be true, and low-level programming will give us good control over resources, but not the best performance; or if the many-core CPU future isn't coming (I say CPU because we're already in the many-core SIMD present with modern GPUs).

[1]: [http://openjdk.java.net/jeps/169](http://openjdk.java.net/jeps/169)

[2]: [http://www.infoq.com/news/2008/05/click_non_blocking](http://www.infoq.com/news/2008/05/click_non_blocking)

</comment></comment></comment></comment></comment><comment><USER7> BTW, everyone should read Olin Shivers' dissertation acknowledgements:

        http://www.scsh.net/docu/html/man.html

</comment><comment><USER8> The irony is that no one wishes to port it to "modern" AArch64 and x86_64 with, due to proper modular design and GC and compiler written in high-level language (same as MIT scheme) it at least seems like a manageable effort. The goal is that it could be comparable with Haskell in code quality.

Compact and clean _native_ Scheme compiler (such as Gambit) could be a very nice tool. The problem, as usual, is funding and ignorance of the investors who know no other words but Java.)

</comment><comment><USER9> And here is the manual for the T language [http://web.archive.org/web/20060925104715/http://mumble.net/...](http://web.archive.org/web/20060925104715/http://mumble.net/~campbell/t/tman.pdf)

</comment><comment><USER10> So much of this sounds interesting, so what happened to T? Is there any code for the T3 variant? I presume you need T1, T2, and T3 to bootstrap the former, but I would love to see some of this stuff!

</comment><comment><USER11> >There was, for example, a snooty French paper that sort of dismissed Lamping as an "autodidact," before proceeding to build (with, let me be careful to note, proper credit given to John) on his work

Is there a reference to this paper available?

<comment><USER12> I think it might be this paper:

[http://users.soe.ucsc.edu/~abadi/Papers/proofs.ps](http://users.soe.ucsc.edu/~abadi/Papers/proofs.ps)

The reference to Lamping being 'autodidactical' is at the top of the second column.

<comment><USER11> >"Girard is a logician and Lamping is an autodidactic engineer"

Ouch! I can't believe that went to print.

</comment></comment></comment><comment><USER13> @pg: Your blog would be _MUCH_ more readable if you made this change:

<font size="4" face="verdana" style="line-height: 2em">

Right now, a great sadness comes over me when I think about reading one of your articles :-)

<comment><USER14> Is ctrl+ really that hard to do?

<comment><USER13> Ctrl+ doesn't fix leading (spacing between lines).

Is it really that hard to make sure you understand a comment before posting a frivolous reply?

</comment></comment></comment><comment><USER15> Original source: [http://people.csail.mit.edu/gregs/ll1-discuss-archive- html/m...](http://people.csail.mit.edu/gregs/ll1-discuss-archive- html/msg00650.html)

</comment><comment><USER16> I assumed I might find this article in [Essays], but I couldn't find it except on the 8th page of [Index]. Does this article really only exist in the index?

<comment><USER17> It's on the page Lisp Articles: [http://paulgraham.com/lispart.html](http://paulgraham.com/lispart.html)

which is linked to from the page Lisp Links: [http://paulgraham.com/lisplinks.html](http://paulgraham.com/lisplinks.html)

which is linked to from the top level page Lisp [http://paulgraham.com/lisp.html](http://paulgraham.com/lisp.html)

</comment></comment><comment><USER18> Does anyone have any references for the Clark GC? I'm writing a toy lisp compiler and would find it interesting to read about it.

<comment><USER17> I think it might be An empirical study of list structure in Lisp [1], by Douglas W. Clark and C. Cordell Green. I haven't read it because it is behind the ACM paywall. I'm pretty sure that the Clark he mentions must be Douglas W. Clark, who has at least five citations in the Jones and Lins Garbage Collection book (1st ed), and is the only Clark listed (under his own name, at least) in the bibliography (of the 1st ed).

If you are looking for a simple GC algorithm that might be suitable for use in a toy lisp language, you might check out Simple Generational Garbage Collection and Fast Allocation by Andrew W. Appel [2]. I think it has many of the same characteristics and is also pretty simple. For a while I had thought that this must be the one that Shivers meant, and that he just misremembered the author.

Edit: The reason I think it is this particular paper by Clark and Green is due to a discussion of it on page 140 (section 6.8) of Jones and Lins (this page is visible on Amazon in search inside):

> Experiments with a recursive copying collector by Douglas Clark and Cordell > Green produced a cdr-cell linearisation -- the property that a cell that > points to another will be next to each other in Tospace after collection -- > of over 98 percent [Clark and Green, 1977]. The incidence of off-page > pointers was also low (between 2.7 and 8.4 percent).

[1] [http://dl.acm.org/citation.cfm?id=359427](http://dl.acm.org/citation.cfm?id=359427)

[2] [http://www.cs.ucsb.edu/~ckrintz/racelab/gc/papers/appel88sim...](http://www.cs.ucsb.edu/~ckrintz/racelab/gc/papers/appel88simplegen.pdf)

</comment></comment><comment><USER19> why does this not have a date?

</comment><comment><USER20> It took me a while to figure out Shivers wasn't talking about the conventional diss - the narrative lives and dies on the technical one-ups.

</comment><comment><USER21> Fascinating read.

"[..] I had never been to California before, so I was discovering San Francisco, my favorite city in the US and second-favorite city in the world. [..]"

Out of curiosity does somebody know what is PG's first-favorite city in the world?

"[..] It was also a massive validation of a thesis Steele had argued for his Master's, which was that CPS was a great intermediate representation for a compiler [..]"

What does CPS mean? Further down in the article

"[..] Richard Kelsey took his front end, which was a very aggressive CPS-based optimiser, and extended it all the way down to the ground to produce a complete, second compiler, which he called "TC" for the "Transformational Compiler." His approach was simply to keep transforming the program from one simple, CPS, lambda language to an even simpler one, until the language was so simple it only had 16 variables... r1 through r15, at which time you could just kill the lambdas and call it assembler. [..]"

So I guess CPS means Continuation-Passing-Style? But then I wonder what PG means by "CPS was a great intermediate representation for a compiler"?

Even further down we can see PG's own opinion:

"[..] So the lineage of the CPS-as-compiler-IR thesis goes from Steele's Rabbit compiler through T's Orbit to SML/NJ. At which point Sabry & Felleisen at Rice published a series of very heavy-duty papers dumping on CPS as a representation and proposing an alternate called A-Normal Form. ANF has been the fashionable representation for about ten years now; CPS is out of favor. This thread then sort of jumps tracks over to the CMU ML community, where it picks up the important typed-intermediate-language track and heads to Cornell, and Yale, but I'm not going to follow that now. However, just to tell you where I am on this issue, I think the whole movement from CPS to ANF is a bad idea (though Sabry & Felleisen's technical observations and math are as rock solid as one would expect from people of their caliber). [..]

Fascinating stuff, though I do not what IRs are used by the modern compilers like LLVM, GHC, Java, .Net?

"[..] Jim Philbin, like Kelsey, also went to NEC, where he built an operating system tuned for functional programming languages, STING (or perhaps it was spelled "STNG" \-- in any event it was pronounced "sting"). He built it in T, of course, and one could see that it had its roots in his work on T3. (Implementing the runtime for a functional language, in some sense, requires you to implement a little virtual OS on top of the real, underlying OS.) [..]" \- Does anybody know what happened to this OS? The only link that I found is [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.11.4...](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.11.4692)

The summary of this article then goes to say "[..] Sting is currently a prototype implemented on an eight processor Silicon Graphics PowerSeries 480 running Unix. The next step is to integrate Sting into the micro-kernel of an operating system such as Mac [..]" \- Would be interesting if this fruited in some new works for VMs for functional languages.

<comment><USER22> PG didn't write this article, Olin Shivers did.

<comment><USER23> And Jonathan had some small amendments:

[http://mumble.net/~jar/tproject/](http://mumble.net/~jar/tproject/)</comment></comment></comment>


John Carmack on id Tech 6, Ray Tracing, Consoles, Physics and more - <AUTHOR> http://www.pcper.com/article.php?aid=532

<comment><USER2> Really neat. While not as technical, on Shacknews they just posted the HD feed of the Epic Games presentation on the engine enhancements that they've done for Gears of War 2. Looking good: <http://www.shacknews.com/onearticle.x/51906></comment>


MongoDB vs. Clustrix: Fault Tolerance and Availability - <AUTHOR> http://sergeitsar.blogspot.com/2011/02/mongodb-vs-clustrix-comparison-part-2.html

<comment><USER2> I'd like to correct some factual errors from this article.

1) Failover of a MongoDB Replica Set is totally automated and requires no manual intervention. The replica set remains available for writes as long as a quorum can be established between remaining members. See <http://www.mongodb.org/display/DOCS/Replica+Sets> for more info

2) MongoDB does support different consistency models through Write Concerns and Safe Mode. The client can choose to wait for the transaction to be written to multiple replicas if it wants. See [http://www.mongodb.org/display/DOCS/Verifying+Propagation+of...](http://www.mongodb.org/display/DOCS/Verifying+Propagation+of+Writes+with+getLastError) for more info

Disclaimer: I work for 10gen

<comment><AUTHOR> 1\. Say I have a 2 node replica set. Now a replica dies, permanently. How is the recovery automated? These are quotes directly from your docs:

[http://www.mongodb.org/display/DOCS/Resyncing+a+Very+Stale+R...](http://www.mongodb.org/display/DOCS/Resyncing+a+Very+Stale+Replica+Set+Member)

"1. Delete all data. If you stop the failed mongod, delete all data, and restart it, it will automatically resynchronize itself. Of course this may be slow if the database is huge or the network slow.

2\. Copy data from another member. You can copy all the data files from another member of the set IF you have a snapshot of that member's data file's. This can be done in a number of ways. The simplest is to stop mongod on the source member, copy all its files, and then restart mongod on both nodes. The Mongo fsync and lock feature is another way to achieve this. On a slow network, snapshotting all the datafiles from another (inactive) member to a gziped tarball is a good solution. Also similar strategies work well when using SANs and services such as Amazon Elastic Block Service snapshots.

<http://www.mongodb.org/display/DOCS/fsync+Command> "Lock, Snapshot and Unlock

The fsync command supports a lock option that allows one to safely snapshot the database's datafiles. While locked, all write operations are blocked, although read operations are still allowed. After snapshotting, use the unlock command to unlock the database and allow locks again

2\. Really? Is this wrong then?

[http://www.mongodb.org/display/DOCS/Replica+Set+Design+Conce...](http://www.mongodb.org/display/DOCS/Replica+Set+Design+Concepts)

"Writes which are committed at the primary of the set may be visible before the true cluster-wide commit has occurred. Thus we have "READ UNCOMMITTED" read semantics. These more relaxed read semantics make theoretically achievable performance and availability higher (for example we never have an object locked in the server where the locking is dependent on network performance).

<comment><USER3> 1\. You really need a minimum of three replica set nodes, one of which can be a lightweight arbiter. If the primary fails, the secondary node will be promoted to primary automatically. In the case of a network partition, the old primary will come back up as a secondary with no problems. In the case of a true hardware failure, you can resync very quickly from a snapshot. For extra peace of mind, add more nodes to the replica set. You can have up to seven.

2\. If you're reading from both primary and secondary nodes, then the view may not be consistent. In most cases you simply read from the primary for fully- consistent reads. You get to decide whether reads from secondaries are consistent or not by setting the write concern (i.e., the minimum number of nodes to replicate to before returning each write.)

<comment><AUTHOR> 1\. Yes, I recognize that MongoDB will automatically fail over when we go from N nodes in the set to N - 1. But how do I get back to N nodes? That's completely manual.

2\. What happens when I read an update that succeeded on the master but then later fails on the slaves?

<comment><USER3> 1\. It depends on how the node fails. If there's just a network partition, then you still have N nodes, so no issues. If you're running with durability enabled, and you experience, say, a power outage, then the member should rejoin the set and resync with no issues. If a node's drive crashes, then you'll need to restore from a recent snapshot (within a day or so) or perform a complete resync if you don't have snapshot. But this can all be done without taking the replica set offline. In that last case, there is some manual work involved. But your post, unless you've corrected it, implies that replica set failover is completely manual. That's certainly not true.

2\. Outside of some kind of hardware failure, you won't have situations where writes succeed on the primary but fail on a secondary. And as I stated on your blog post, if you're really concerned about it, you can specify a write concern on insert, and if the write fails to replicate in the desired way, you'll know about it.

<comment><AUTHOR> Sorry, but "hardware failure" is a fault, and when you can't deal with it, you're not tolerant. And with larger clusters, you see hardware faults on a regular basis. So saying we're ok in the nominal mode is not fault tolerance.

</comment></comment></comment></comment></comment></comment><comment><USER4> These posts are written by one of the Clustrix founders.

<comment><USER5> Are you trying to imply that the post has wrong information because of this fact? If so, attack the wrong data. I don't care who posts facts, as long as they really are facts.

<comment><USER6> I haven't read their analysis yet (I will try to when I have some free time), but in general, I would argue that trying to compare a document database to a SQL one is always going to be somewhat misleading. I'd care more if they were comparing Clustrix to MSSQL, MySQL, PostgreSQL.

If you are using MongoDB in a way that is similar to the way you would have used a SQL DB you are probably doing something wrong. Specifically, you are trying to place normalized data in a database designed for denormalization.

<comment><USER7> Sergei compares Clustrix to MongoDB because their target markets are very similar--not because the technology is similar.

As a startup, it behooves them to attack the low-end database market, but I suspect they've found that the primary market for a highly scalable low-end database lies on the web, but that market has chosen to go cheap-and-dirty with NoSQL. So now they're in the middle ground between fast-and-and-loose- and-free and my-enterprise-uses-Oracle.

I think a lot of web development is of a highly speculative, winner-take-all sort, so devs. want to be as cheap as possible until they win the web lottery. For all the flaws of NoSQL, software only solutions do allow developers to make very efficient use of their hardware by running multiple services on the same machine, or run them in the cloud. Once they hit the jackpot, they can afford to either go Oracle, hire software developers to work around deficiencies in their data store (e.g. Facebook), or use a data store from Amazon or Google or Microsoft.

That's a shame, because I think Clustrix is ultimately the right approach. The web has a history of doing the shittiest-and-easiest thing first (ColdFusion, anyone?) only to repent years later to the second-shittiest solution. Rinse, repeat.

<comment><USER8> Clustrix does give you the option to start with MySQL and then do a drop-in upgrade when your idea gets traction.

<comment><USER9> That's a huge risk. What if the way I use MySQL does not go well with the way Clustrix is supposed to scale?</comment></comment></comment></comment></comment></comment>


A Rabbithole I fell into recently that changed how I work - <AUTHOR> https://rsapkf.netlify.com/blog/the-rabbithole-series-part-1 <comment><USER2> tl;dr Author used Windows+Kali as a teenage webdev, then discovered tiling window managers and Arch.</comment>