Announcing GVFS: Git Virtual File System - <AUTHOR> https://blogs.msdn.microsoft.com/visualstudioalm/2017/02/03/announcing-gvfs-git-virtual-file-system/ <comment><USER2> This is similar to what Google uses internally. See [http://cacm.acm.org/magazines/2016/7/204032-why-google- store...](http://cacm.acm.org/magazines/2016/7/204032-why-google-stores- billions-of-lines-of-code-in-a-single-repository/fulltext):

"Most developers access Piper through a system called Clients in the Cloud, or CitC, which consists of a cloud-based storage backend and a Linux-only FUSE13 file system. Developers see their workspaces as directories in the file system, including their changes overlaid on top of the full Piper repository. CitC supports code browsing and normal Unix tools with no need to clone or sync state locally. Developers can browse and edit files anywhere across the Piper repository, and only modified files are stored in their workspace. This structure means CitC workspaces typically consume only a small amount of storage (an average workspace has fewer than 10 files) while presenting a seamless view of the entire Piper codebase to the developer."

This is a very powerful model when dealing with large code bases, as it solves the issue of downloading all the code to each client. Kudos to Microsoft for open sourcing it, and under the MIT license no less.

<comment><USER3> Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly created in response to Perforce's inability to scale and be fault tolerant. Until Piper came along, they would have to periodically restart The Giant Perforce Server in Mountain View. Piper is 24x7x365 and doesn't need any restarts at all. But the key bit here is not Piper per se. Unlike Microsoft, Google also has a distributed, caching, incremental build system (Blaze), and a distributed test system (Forge), and they are integrated with Piper. The vast majority of the code you depend on never actually ends up on your machine. Thanks to this, what takes hours at Microsoft takes seconds at Google. This enables pretty staggering productivity gains. You don't think twice about kicking off a build, and in most cases no more than a minute or two later you have your binaries, irrespective of the size of your transitive closure. Some projects take longer than that to build, most take less time. Tests are heavily parallelized. Dependencies are tracked (so tests can be re- run when dependencies change), there are large scale refactoring tools that let you make changes that affect the entire monorepo with confidence and without breaking anyone.

Google's dev infra is pretty amazing and it's at least a decade ahead of anything else I've seen. Every single ex-Googler misses it quite a bit.

<comment><USER4> > Google's dev infra is pretty amazing and it's at least a decade ahead of > anything else I've seen. Every single ex-Googler misses it quite a bit.

This may be naive but why not recreate it as an open source project?

<comment><USER5> Blaze has been: [https://bazel.build/](https://bazel.build/)

Forge and Piper are built on Google's internal tech stack and designed for Google's production infrastructure, so open sourcing them would be a very big project. I think it would be a lot more likely for them to be offered as a service -- and that might be more useful to users anyway, since you'd be able to share resources with everyone else doing builds, rather than try to get your own cluster running which might sit idle a lot of the time. Of course, there are privacy issues, etc.

(Disclaimer: I'm purely speculating. I left Google over four years ago, and have no idea what the tools people are up to today.)

</comment></comment></comment></comment><comment><USER6> There is a discussion thread on r/programming, where MS folks, who implemented this answer questions. A lot of questions like why not use multiple repos, why not git-lfs, why not git subtree, etc. are answered there

[https://www.reddit.com/r/programming/comments/5rtlk0/git_vir...](https://www.reddit.com/r/programming/comments/5rtlk0/git_virtual_file_system_from_microsoft/)

<comment><USER7> Thanks for bringing this up, it was actually a more interesting read than this thread. Less trolling, more facts and also interesting to read stuff I didn't happen to know. Like

 _One of the core differences between Windows and Linux is process creation. It 's slower - relatively - on Windows. Since Git is largely implemented as many Bash scripts that run as separate processes, the performance is slower on Windows. We're working with the git community to move more of these scripts to native cross-platform components written in C, like we did with interactive rebase. This will make Git faster for all systems, including a big boost to performance on Windows._

<comment><USER8> > "We're working with the git community to move more of these scripts to > native cross-platform components written in C"

Sad. Rather than fix the root problem they rewrite the product in a less-agile language and require everyone to run opaque binaries.

They probably even think they're doing a good thing.

<comment><USER9> C is portable, bash scripts are not.

<comment><USER8> Bash is portable across other OSes... They could work on a good port. Or, remove some bash-isms from the code so it would work in another shell if that was an issue.

I understand they took the initially easy route. But it'll be harder for everyone to use that code now, including them.

</comment></comment></comment></comment></comment><comment><USER10> It's interesting how all the cool things seem to come from Microsoft these days.

I still think we need something better than Git, though. It brought some very cool ideas and the inner workings are reasonably understandable, but the UI is atrociously complicated. And yes, dealing with large files is a very sore point.

I'd love to see a second attempt at a distributed version control system.

But I applaud MS's initiative. Git's got a lot of traction and mind share already and they'd probably be heavily criticized if they tried to invent its own thing, even if it was open sourced. Will take a long time to overcome its embrace, extend and extinguish history.

<comment><USER11> Maybe something that has the data models of git but has a more consistent interface? Today on Git Merge there was a presentation about [http://gitless.com/](http://gitless.com/)

For example one of the goals is to always allow you to switch branches. Stash and stash pop would happen automatically and it would even work if you're in the middle of a merge.

<comment><USER12> I'm still waiting for a decent GUI that takes full advantage of the simplicity of git's underlying data model. The CLI is okay and I've gotten really good with it, but fundamentally I think git's DAG is something that would be best represented and manipulated graphically.

[Reinventing the Git Interface][1] was written almost 3 years ago now and yet to my knowledge nobody's implemented anything quite like that yet.

[1]: [http://tonsky.me/blog/reinventing-git- interface/](http://tonsky.me/blog/reinventing-git-interface/)

<comment><USER13> Git Kraken has some neat ideas around dragging bits of the DAG around to manipulate them.

<comment><USER12> Yeah, I've been meaning to try that for a while now. Unfortunately I can't use it at work because Kraken still don't support connecting to the internet through a proxy, and they won't let you use it offline.

</comment></comment></comment></comment></comment><comment><USER14> Using git with large repos and large (binary blob) files has been a pain point for quite a while. There have been several attempts to solve the problem, none of which have really taken off. I think all the attempts have been (too) proprietary – without wide support, it doesn't get adopted.

I'll be watching this to see if Microsoft can break the logjam. By open sourcing the client and protocol, there is potential...

Other attempts:

* [https://github.com/blog/1986-announcing-git-large-file-stora...](https://github.com/blog/1986-announcing-git-large-file-storage-lfs)

* [https://confluence.atlassian.com/bitbucketserver/git-large-f...](https://confluence.atlassian.com/bitbucketserver/git-large-file-storage-794364846.html)

Article on GitHub's implementation and issues (2015): [https://medium.com/@megastep/github-s-large-file-storage- is-...](https://medium.com/@megastep/github-s-large-file-storage-is-no- panacea-for-open-source-quite-the-opposite-12c0e16a9a91)

<comment><USER15> I think Joey Hess' attempt at "solving the problem" deserves a mention.

It is open source (GPLV3) licensed. [not proprietary]

Written in Haskell. [cool aid]

Currently has 1200+ stars on Github and is part of at least Ubuntu ([http://packages.ubuntu.com/search?keywords=git- annex](http://packages.ubuntu.com/search?keywords=git-annex)) since 12.04. [shows something for support and adoption]

edit: Link to Github [https://github.com/joeyh/git- annex](https://github.com/joeyh/git-annex) \-- thanks dgellow

<comment><USER12> For the problem of large files I think Git LFS has largely won out over git annex, mostly because it's natively supported by GitHub and GitLab and requires no workflow changes to use.

<comment><USER16> Atlassian's Bitbucket and Microsoft's Visual Studio Team Services both also support Git LFS.

</comment></comment></comment></comment><comment><USER17> It's disappointing that all the comments are so negative. This is a great idea and solves a real problem for a lot of use cases.

I remembering years ago Facebook says it had this problem. A lot of the comments were centered around that you could change your codebase to for what git can do. I'm glad there's another option now.

<comment><USER18> It's because the 'problem' it solves is a corner case that's rarely encountered. I love their absurd examples of repos that take 12 hours to download. How many people have that problem, really?

All they did is create a caching layer.

<comment><USER19> If you deal with code, the case is marginal for you.

If you deal with graphics, audio assets, etc, the binary-blob type of data, the case is central.

<comment><USER20> This is about code, and code history. Just insane volumes.

</comment></comment></comment></comment><comment><USER21> I'm immediately reminded of MVFS and clearcase. Lots of companies still use clearcase, but IMO it's not the best tool for the job. git is superior in most dimensions. From what this article says, it's not quite the same as clearcase but there's certainly some hints of similarities.

The biggest PITA with clearcase was keeping their lousy MVFS kernel module in sync with ever-advancing linux distros.

I really liked Clearcase in 1999, it was an incredible advancement over other offerings then. MVFS was like "yeah! this is how I'd design a sweet revision control system. Transparent revision access according to a ranked set of rules, read-only files until checked out." But with global collaborators, multi-site was too complex IMO. And overall, clearcase was so different from other revision control systems that training people on it was a headache. Performance for dynamic views would suffer for elements whose vtrees took a lot of branches. Derived objects no longer made sense -- just too slow. Local disk was cheap now, it got bigger much faster than object files.

> However, we also have a handful of teams with repos of unusual size! ... You > can see that in action when you run "git checkout" and it takes up to 3 > hours, or even a simple "git status" takes almost 10 minutes to run. That's > assuming you can get past the "git clone", which takes 12+ hours.

This seems like a way-out-there use case, but it's good to know that there's other solutions. I'd be tempted to partition the codebase by decades or something.

<comment><USER22> I used Clearcase (on Solaris) in 1999 and was not a fan. It slowed our build times by at least 10x. I'm sure it was probably set up wrong, but this was a Fortune 100 company with lots of dedicated resources.

<comment><USER23> Clearcase performance for many builds was specifically impacted by the very poor performance of stat(). You could make very real improvements on build times by reducing the number of calls to stat(). It was sort of amazing.

Clearcase also suffered, at least in my experience, from a clumsy and ugly merging process and deeply unintuitive command set which meant everyone who "used clearcase" actually tended to use some terrible homegrown wrapper scripts.

Still, considering it was the last remaining vestige of the Apollo Domain OS, not bad.

</comment></comment></comment><comment><USER24> I think they could have picked a name that doesn't conflict with GNOME Virtual File System (GVfs).

<comment><USER25> They used to choose very bad names like .NET or COM [1] (this predates Internet) makes searching information very tricky. MSDN doesn't help.

[1] [https://en.wikipedia.org/wiki/Component_Object_Model](https://en.wikipedia.org/wiki/Component_Object_Model)

<comment><USER26> A few years ago I had to do some interfacing between python and some modelling software. I went through a COM interface, and it was a bloody nightmare to find docs.

I later found out I could have looked for "ActiveX" and found similar results.

<comment><USER25> A few years ago your best friend would have been [https://www.codeproject.com](https://www.codeproject.com) . The issue of searching difficult questions using another keyword (e.g. ActiveX) is that you can miss the only answer available. For common questions (with answers!) you can find an answer with all the variations.

</comment></comment></comment></comment><comment><USER27> The article doesn't directly say it, but are they migrating the Windows source code repository to git? That seems like a big deal.

I seem to recall that Microsoft has previously used a custom Perforce "fork" for their larger code bases (Windows, Server, Office, etc.).

<comment><USER28> Yes, Windows is migrating to Git.

<comment><USER29> Do you have a citation for that?

<comment><AUTHOR> It was stated in Saeed Noursalehi's talk "Scaling Git at Microsoft", held at Git-Merge 2017. Until the conference recordings are available, here is the closest thing to a "source": [https://twitter.com/no_more_ducks/status/827479795185364993](https://twitter.com/no_more_ducks/status/827479795185364993)

</comment></comment></comment></comment><comment><USER30> If I understand this correctly, unlike git-annex and git lfs, this not about extending the git format with special large files, but changing the algorithm for the current data format.

A custom filesystem is indeed the correct approach, and one that git itself should have probably supported long ago. In fact, there should really only be one "repo" per machine, name-spaced branches, and multiple mountpoints a la `git worktree`. In other words there should be a system daemon managing a single global object store.

I wonder/hope IPFS can benefit from this implementation on Windows, where FUSE isn't an option.

<comment><USER31> The blog post does mention that some changes have been made to git (in their fork)

<comment><USER32> I did a quick comparison of Microsoft's fork and it appears they have done quite a bit with it.

Microsoft's fork contains 67,522 commits. The official Git repo contains 45,810. It appears the bulk of the work started in 2010, with significant ramp up of development in 2015.

[https://gitsense.com/mgit-vs-git/history.png](https://gitsense.com/mgit-vs- git/history.png)

Looks like Microsoft only really introduced about 100 more new files.

[https://gitsense.com/mgit-vs-git/files.png](https://gitsense.com/mgit-vs- git/files.png)

Microsoft's repo contains 1712 contributors. Git's repo contains 1685 contributors. So it looks 20 - 30 employees worked on Microsoft's fork.

[https://gitsense.com/mgit-vs-git/mgit- contributors.png](https://gitsense.com/mgit-vs-git/mgit-contributors.png) [https://gitsense.com/mgit-vs-git/git- contributors.png](https://gitsense.com/mgit-vs-git/git-contributors.png)

</comment></comment></comment><comment><USER33> This is pretty big news. I know that when I was at Adobe, the only reason that Perforce was used for things like Acrobat, is because it was simply the only source control solution that could handle the size of the repo. Smaller projects were starting to use Git, but the big projects all stuck with Perforce.

</comment><comment><USER34> I love this approach. From working at Google I appreciate the virtual filesystem, it makes a lot of things a lot easier. However all my repos are large enough to fit on a single machine so I wish there was a mode where it was backed by a local repository, however the filesystem allows git to avoid tree scans.

Basically most operations in git are O(modified files) however there are a few that are O(working tree size). For example checkout and status were mentioned by the article. However these operations can be made to O(modified) files if git doesn't have to scan the working tree for changes.

So pretty much I would be all over this if:

\- It worked locally.

\- It worked on Linux.

Maybe I'll see how it's implemented and see if I could add the features required. I'm really excited for the future of this project.

</comment><comment><USER35> Assuming that the repo was this big in the beginning, I wonder why the ever migrated to git (I'm assuming they did, because they can tell how long it takes to checkout). At least when somebody "tries" do the migration, wouldn't they realize that maybe git is not the right tool for them? Or did they actually migrate and then work with "git status" that take 10 minutes for some time until they realize they may need to change something?

Also, it would have been interesting if the article mentioned whether they tried other approaches taken by facebook (mercurial afaik) or google.

<comment><USER36> To me it sounds like these numbers are from a migration-in-progress. So they are trying, but instead of giving up and saying "not the right tool for us" they are trying to improve the tool.

</comment></comment><comment><USER37> > repos of unusual size

Sounds like they've almost solved the secrets of the fire swamp!

<comment><USER38> Repos of Unusual Size? I don't think they exist.

</comment></comment><comment><USER39> Did they really need to make a name collision?

[https://en.wikipedia.org/wiki/GVfs](https://en.wikipedia.org/wiki/GVfs)

</comment><comment><USER40> This sounds like a solid use case and a solid extension for that use case - but definitely not the end-all-be-all.

For one, it's not really distributed if you're only downloading when you need that specific file.

But that doesn't change the merrits of this at all, I think.

</comment><comment><USER41> My sysadmin: "we won't switch to git because it can't handle binary files and our code base is too big"

Our whole codebase is 800MB.

<comment><USER42> Our codebase (latest tree) is similar, but switching to git it's the total history size that is the problem. Our history is well over 25GB which git doesn't handle very gracefully.

<comment><USER34> History shouldn't be a problem, you can do a shallow checkout. But you will have to store the working tree at least on your workstation.

This solves the next scaling problem of avoiding managing the whole working tree. (without requiring narrow clones which have significant downsides)

<comment><USER42> Yeah, the working tree works well to have locally, and that's what's done with svn currently.

The problem is that I also want a fast log/blame for any file back to the beginning of time - but I'm ok with that requiring devs connecting to the server containing the history (as with svn).

I also haven't found a way to make git work smoothly in shallow mode as the default, e.g can I make checkout of a branch always remember it must be shallow? Can I make log use remote history when necessary etc? I don't want to fight the tool all the time because I'm using a nonstandard approach.

</comment></comment></comment></comment><comment><USER43> I appreciated the Princess Bride reference with "repos of unusual size"

<comment><USER21> I don't believe in them.

</comment></comment><comment><USER44> Just to make sure I have this right, this has to do with the _amount_ of files in their repo and not the _size_ of the files? So projects like git annex and LFS would not help the speed of the git repos?

<comment><USER16> That's how I read it, that this is about monorepos with file trees with large numbers of files where users don't necessary need every single file in their local worktree to get work done.

I'd assume this GVFS would work hand in hand with Git LFS for the use case of large files.

</comment></comment><comment><USER45> > _when you run "git checkout" and it takes up to 3 hours, or even a simple > "git status" takes almost 10 minutes to run. That's assuming you can get > past the "git clone", which takes 12+ hours._

How on Earth can anybody work like that?

I'd have thought you may as well ditch git at that point, since nobody's going to be _using_ it as a tool, surely?

        git commit -m 'Add today\'s work - night all!' && git push; shutdown

<comment><USER7> _How on Earth can anybody work like that?_

Since it's look like they are still migrating I don't think a lot of people actually did work like that. Maybe just a couple of times to figure out how long it would actually take. Or maybe those who really use it are actually doing shallow clones which would probably take much less time. Actually shallow clone is nice but doesn't seem to be known very well. I use it often if I know I won't ever need the full history anyway. Also great to shave time of CI builds.

<comment><USER45> Shallow clones are great, until they're not. I don't think I've ever (having tried a few times) cleanly cloned 'below' the graft point when I've needed to, or a different branch.

</comment></comment></comment><comment><USER46> Or how about we start some compartmentalizing your codebase so that you can like. You know, organize your code and restore sanity to the known universe.

I think when the powers that be said that whole thing about geniuses and clutter, they were specifically talking about their living spaces and not their work...

</comment><comment><USER47> Does anyone know Microsoft's open source policy works internally? I'm thinking from a governance perspective, as I'm involved in a similar effort at $WORK.

</comment><comment><USER48> I had a medium sized project in Ruby on Rails as git repo inside vm.

It was slow to do 'git status' and other common commands. Restarting RoR app was also slo. I've put repo on RAM disk which made the whole experience at least few times faster.

Since all was in vm that I rarely restarted I didn't have to recreate files on ram disk all that often. I was syncing changes with the persistent disk with rsync running periodically.

</comment><comment><USER49> "For example, the Windows codebase has over 3.5 million files and is over 270 GB in size."

Okay, so this is a networking issue. Or is it a stick everything in the same branch issue?

Whatever the reason here the issue is pure size vs. network pipe, pure and simple. Hum, when can I get a laptop with a 10GBaseT interface?

One of the issue with the way they are doing this (only grab files when needed) is you cannot really work offline anymore.

</comment><comment><USER50> I'm no expert but if most single developers only use 5-10% of the codebase in their daily life, wouldn't it make to maybe break the project into multiple codebases of about 5% each and use a build pipeline that combines them together when needed?

Although I could definitely be wrong but this sounds a lot like monolith vs microservices to me.

</comment><comment><USER51> Microsoft is moving away from source depo to git it seems. I think its fantastic that a company like Microsoft is adapting git for its big king and queen projects such as office and windows. Also open sourcing the underlying magic tells a lot about the new Microsoft. They're really moving away from not-invented here syndrome

</comment><comment><USER52> Does this article imply that Microsoft itself is also moving towards Git? Instead of e.g. using their own product like TFS?

<comment><USER27> TFS has first-class support for Git repositories (in addition to the classic TFSVC repositories). So yes, they're moving more and more to Git. But no, they're not abandoning TFS.

Interestingly, however, most of their "open source" efforts (.NET, C#, and related) are all on GitHub rather than their own hosted offerings: CodePlex (which is basically dead) or "Visual Studio Team Services".

<comment><USER28> Not sure why Visual Studio Team Services is in scare quotes -- that's the product's name. And it's not an open source hosting service, which handily explains why Microsoft's open source isn't hosted there.

Disclosure: I'm a PM on VSTS/TFS, and I own part of version control.

<comment><USER53> Are those scare quotes or just regular old quotes? TFS and associated technologies have been through a lot of names (Visual Studio Team System, TFS, Team Services, and probably a few I can't remember).

Disclaimer: used to work on TFS team.

<comment><USER28> Ha, fair point :)

</comment></comment></comment></comment></comment><comment><USER54> MS has been doing really neat stuff lately. I never worked on a project that takes hours to clone. The largest repository I regularly clone is the Linux repo. It still takes only a few minutes. Yet I can see the GVFS being beneficial for me as I spend most of the time just reading the code (so no need to compile) on my laptop.

</comment><comment><USER42> Could this also help a smaller repo but with long history, making the total repo size too large?

The whole repo is needed for every developer - i.e it's not possible to do a sparse checkout but many gigs of old versions of small binaries I would prefer to keep only at the server until I need it (which is never).

</comment><comment><USER55> And for all those who still try to stick to anything older:

[https://github.com/Microsoft/gvfs](https://github.com/Microsoft/gvfs)

"GVFS requires Windows 10 Anniversary Update or later."

</comment><comment><USER56> I remember few years ago Git under Windows was very slow, is it still true?

<comment><USER16> Git on Windows has gotten very fast and stable in the last few years. Microsoft employees themselves, among others of course, have directly contributed to a much better Git experience on Windows.

<comment><USER57> The reddit thread has quite a few people with opposing opinions, fwiw. Mostly "stuff that's ~instant on unix takes many seconds on Windows" and the like. It's true that Microsoft has contributed a lot (to the benefit of all), but from what I'm seeing it sounds like it's still lagging quite a bit.

I haven't touched Windows in quite a while, so I can't really make a claim either way.

<comment><USER16> I'm at least speaking from daily use in my anecdotes. Apples for apples, yes Windows is going to lag behind Linux. [1] That doesn't mean it isn't fast and stable from the perspective of day-to-day Windows usage, and definitely as I stated in the previous comment, it is much faster and more stable on Windows today compared to Windows a few years ago.

Several of the anecdotes on the reddit thread don't even seem to take account what version the offending slowness was happening in, and anecdotally every time I've helped a Windows user experiencing slowness enough to complain about it, they've been years behind on their git version and installing the latest removed the complaints.

[1] ...and is just about guaranteed to in the many places in git where a command is still built as a tower of bash scripts calling perl scripts calling more bash scripts... If you read the changelogs, a lot of the performance optimizations that are helping _every platform_ are the places where entire commands are getting replaced with C versions of themselves.

<comment><USER57> Ah, you're right, you were referring to on-windows progress.

And "years behind on their git version" is I think the norm for git users :) I pretty regularly have to recommend that coworkers / etc upgrade from git 1.7 (or 1.8 or something similar) to an even-remotely-modern version.

</comment></comment></comment></comment></comment><comment><USER58> Check out the GVFS back story and details here: [https://news.ycombinator.com/item?id=13563439](https://news.ycombinator.com/item?id=13563439)

</comment><comment><USER59> Quite nice use of C# and C++/CX for a virtual system implementation.

<comment><USER60> looks like C++/CLI (C++/CX reused its syntax and maybe parsing code, but they're still distinct)

<comment><USER59> Yes, hence why Microsoft had lots of trouble to convince developers that don't read documentation, that they are distinct and C++/CX gets compiled to just pure native code, as they were spreading misinformation about it.

In any case, when C++/WinRT gets feature parity, I imagine it will eventually be deprecated, depending which one gets more developer love.

</comment></comment></comment><comment><USER61> Is it really that fucking hard to check if your package name is unique?

Here is another virtual filesystem with the exact same name: [https://wiki.gnome.org/Projects/gvfs](https://wiki.gnome.org/Projects/gvfs)

Debian package for it: [https://packages.debian.org/jessie/gvfs](https://packages.debian.org/jessie/gvfs)

</comment><comment><USER62> So... what happens when one runs "git grep foo" on it?

<comment><USER34> It will be slow. Small steps. But in practice companies with large repos have other search solutions so that each user doesn't have to do a raw search on the entire working tree.

</comment></comment><comment><USER63> Anybody knows what does Linus think about it ?

</comment><comment><USER64> Can we use this together with git LFS?

</comment><comment><USER65> Couldn't they use git over IPFS?

<comment><USER34> No. The problem isn't only the storage or fetching of the files (this is the easy bit :) ), it's the operations that detect changes in the working tree. If you have a large tree scanning it becomes slow.

Using a vfs allows you to track which files have changed so that these operations no longer need to scan. Now they are O(changed files) which is generally small.

Now IPFS has a vfs, but it is just a simple read/write interface. This vfs needs slightly more logic to do things like change the base revision and track changes.

<comment><USER65> IPFS clearly does a lot more than storing and fetching files. Seriously, go have a read. A single hash can represent an arbitrarily large subtree of data (Microsoft's entire repo). Using an IPLD selector (in its simplest form, a path beyond the hash) an arbitrary sub component can be addressed. This can be used to avoid scanning entire subtrees (maintaining your O(changed files)). To commit your modifications is O(changed files + tree depth to the root of your modifications) you never need to do anything with the rest of the repo.

For tracking changes (i.e. mutable data) you can use IPNS and create a signed commit history. This will be built on IPFS eventually so it's only a matter of time.

</comment></comment></comment><comment><USER66> Don't believe in modular development with smaller repos?

<comment><USER57> Yeah, I see things like this, and I always wonder why they don't make a submodule tree.

It wasn't an option a couple years ago, but submodules work fine now. With a little bit of scripting to wrap common uses, they're practically pain-free.

<comment><USER67> Could you elaborate a little what has changed there? My understanding is that submodules are still considered a mess, but would be really nice if some actual improvements have happened.

<comment><USER57> [https://github.com/blog/2104-working-with- submodules](https://github.com/blog/2104-working-with-submodules) is a decent overview of what was available a couple years ago (things have improved a bit since then too), though it needs a tl;dr. So here's an attempt.

1) When you cd into a submodule, it's the same as if it you just cloned into there, all normal git commands work. need to update your submodule-lib? cd, do stuff, git push, _at worst_.

2) `git clone --recursive` instead of just `git clone`, no need to `git submodule init --update` / etc.

3) `git pull` will automatically pull submodules when the parent repo changes which commit it's using. `git push` should push any changes too, though the manpage isn't explicit (there's an identical flag/config value for push as for pull to control this). also solvable with `pushall` and `pullall` aliases, which is a very minor re-education.

4) submodules can track submodule-repo branches, not just commits. auto- updating ftw? if you want it.

5) there are some somewhat-unhappy defaults / you probably want `git diff --submodule=log` and `git config --global status.submoduleSummary true`, etc. these (and aliases) are easily fixed the same way as you probably already have for templated .gitignore / etc - just generate some company-wide defaults, and move on with your life.

\---

A lot of the "you have to git submodule command everything all the time" is a thing of the past, the difficulty now is largely related to it being a minor conceptual difference from a monorepo. It's a repo in a repo, and you're manipulating the pointer to the version. There are more options because of this, but they exist for good reasons, and they're not too hard to wrap your head around.

[https://git-scm.com/book/en/v2/Git-Tools-Submodules](https://git- scm.com/book/en/v2/Git-Tools-Submodules) also has some nice examples, and e.g. `git submodule foreach` can simplify a lot if you actually dive into submodules and make changes across multiple simultaneously (big refactor maybe?).

</comment></comment></comment></comment><comment><USER68> Why is that so hard to believe? America is run by Donald Trump.

The problems with these companies is that developers aren't making technical decisions, it's executives who know nothing about computer science. That's why Windows 10 is such a mess with spyware and adware.

Now they have some FOSS advocate who doesn't really know anything about software or VCS but saw that an internal problem they were trying to solve was making their code base work with git. So he decided it would be really cool for Microsofts image to develop an open source extension of git, instead of actually solving the underlying problems (because he didn't recognize them). Now he's probably got a promotion at Microsoft for "fixing" their problem with git.

<comment><USER69> We detached this subthread from [https://news.ycombinator.com/item?id=13559893](https://news.ycombinator.com/item?id=13559893) and marked it off-topic.

</comment></comment><comment><USER70> Interesting M$ is moving to Git and the rest of the world is pretty much Github & alternatives while Facebook and Google are going with Mercurial. I actually liked Mercurial apart from its name being little hard to pronounce, but it doesn't seems to get used anywhere.

So are the DVCS converging to Git and Git only?

<comment><USER71> In the open source areana, it certainly seems like the game is over. But as others mentioned, both FB and GOOG are big Mercurial users and contributors.

Our shop uses Mercurial becuase of its Python basis and the amount of time and effort it takes to master Git makes me draw strong and uncomfortable parallels to emacs.</comment></comment>


The Chutzpah of Israeli Startups - <AUTHOR> http://radoff.com/blog/2009/12/05/chutzpah-of-israeli-startups/ <comment><USER2> _How can we encourage people to take more risks in the USA? Clearly, we've got a great startup culture here–particularly in regions like Silicon Valley and Boston, but could we be doing more elsewhere in the country?_

Decouple health insurance from employers.

</comment><comment><USER3> I think there may some misunderstanding of the "Chutzpa" ingredient. As an Israeli living in NYC for the past 3 years, I don't think Americans are less entrepreneurial or more risk averse than Israelis. What is _very_ different is the manner of communication - which I think has as much to do with the fact that modern Hebrew is extremely young, and simply hadn't yet evolved the "syntactic sugar", so to speak, that makes American communication somewhat more abstract.

The Hebrew mode of communication vs the American English one can be likened to executives at a board meeting vs engineers working on fixing a bug.

This has positive & negative implications: communication in Israel, translated literally to English, would seem blunt and sometimes plain rude to Americans (and remember, I'm in NYC ;)). On the other hand, the same directness (and almost total lack of rank - you call your army officer, or cabinet ministers, by their first name) tends to get assumptions questioned, ideas presented and problems recognized quite faster.

<comment><USER4> _"This has positive & negative implications: communication in Israel, translated literally to English, would seem blunt and sometimes plain rude to Americans."_

That is quite interesting! Especially so when one takes into account that Americans i) tend to lack manners and ii) take pride in being direct.

<comment><USER5> My people (the Germans) have this reputation for being impolite. I tend to think this is because we have the tendecy to be brutally direct when we want to be honest. We like to be honest, in that sense anyway, regarding matters of work. This comes off as rude to members of the civilized world (say, the British or Japanese). (I am writing this after +6 years of experience living in foreign countries.)

Another observation: the heroism around doing ones work is strong in Germany, we have that in common with the Americans (in the West, don't ask certain Asians). Yes, certainly there are many French (for instance) fellows, who like theire work. But the Germans (and Americans) have that tendency to get "macho" about it.

<comment><USER3> The first time I visited Germany as a Sales Engineer, the company's local German salesperson told me "Be direct. People here appreciate a straight, yes/no reply". I was relieved to hear this :)

I suppose some might consider communication in German less direct due to issues like Sie vs Du separation. But since I wasn't speaking German it didn't really register with me.

</comment></comment></comment></comment><comment><USER6> _Israel has the highest density of tech start-ups in the world._

Comparing the density of startups in Israel vs. the entire US or the entire European region is misleading, because the US and Europe are so much larger and more heterogenous. It would be more accurate to compare the density of startups in Israel vs. Silicon Valley (comparable geographic area and population).

</comment><comment><USER7> You should be careful in the way that you use the word 'Chutzpah'—in English people usually use it to mean ballsiness, but in Yiddish it still means impudence, affrontery, gall. Not a good quality. I don't know about modern Israeli but given its negative cognates I wouldn't be surprised if it is still not a nice thing to accuse someone of there, too.

</comment><comment><USER8> "Chutzpah". Very Gladwellian. The simplest explanation is that Israel has lots of Jews, who tend to be smart and hard-working, as demonstrated consistently for 1000s of years. The need-a-poker-face-to-utter-it explanation is that they have "Chutzpah".and that's the key to success.

<comment><USER7> Wait a minute, you think it's reasonable and patent to assert that there's an inherent racial quality (we know it's not cultural, since Israeli culture can barely be said to be a hundred years old) that makes Jews smart and hard- working, whereas the assertion that Israeli culture is less risk-averse than American is utterly absurd? I don't particularly agree with him but I think I agree less with you.

<comment><USER8> Standardized testing, historical influence, Nobel prize results, Fields metal prize results, Senate and House election results, Ivy League enrollment results, Forbes 400 membership, chess grandmaster positions, etc, etc, all point to unusual abilities on the part of Jews. There are only a few categories where they aren't represented to a wild disproportion. To say that "Chutzpah" (aka Assholism) is the reason, is absolute nonsense. Human endeavour deals in intelligence and little else.</comment></comment></comment>


Now Opensourced: Project Code_Swarm, an experiment in organic software visualization - <AUTHOR> http://code.google.com/p/codeswarm/ <comment><USER2> Can't wait to see the code_swarm of code_swarm after it's been open-sourced for awhile.

</comment><comment><AUTHOR> A big thanks to Michael Ogawa, who originated the project, and undoubtedly put in long hours cleaning up the code so as to be a palatable base for an open source project.

</comment><comment><USER3> Don't get too excited - it's not quite available just yet. <http://code.google.com/p/codeswarm/source/list> shows only the directory structure at the moment.

<comment><AUTHOR> /source/list is configured to list the contents of trunk. The existing code is in /svn/prototype. To checkout a read-only copy, try pulling down the entire repository:

svn checkout <http://codeswarm.googlecode.com/svn/> codeswarm-read-only</comment></comment>


96% suppression of the dengue mosquito in Brazilian trial - <AUTHOR> http://www.oxitec.com/oxitec-newsletter-june-2013/

<comment><USER2> They were allowed to release something like this _in the wild_? This is like playing god. If you kill all the mosquitos, nature cannot recover. They may be pests for us, but they are still a critical part of the biosphere. The true impact of this activity will not be felt for some time. It harks back to Mao's devastation of the birds.

<comment><USER3> They are probabbly targeting _Aedes aegypt_. It's original location in Africa, so if they can eradicate it from Brazil they will restore the natural balance.

<comment><USER2> _Moscamed announced that they had achieved a 96% reduction of the adult mosquito population in the target area._

That sounds like all mosquitoes to me.

<comment><USER3> A few lines below that line, a more complete quote appears:

 _> The effect on the local population of the _dengue vector* was dramatic: analysis of eggs laid in specially designed 'ovitraps' showed that the number of viable 'wild-type' eggs laid in the trial areas was reduced by 91-95%, while adult trapping showed a reduction in the local adult population of an estimated 96%.*

They are releasing OX513A ( [http://www.oxitec.com/health/our-products/aedes- agypti-ox513...](http://www.oxitec.com/health/our-products/aedes-agypti- ox513a/) ). It's a variant of Aedes aegypti, so it mates only with Aedes aegypti and only that type of mosquitoes is affected. The first assertion is misleading, but it's only a simplification / edition error / marketing hype.</comment></comment></comment></comment>


Show HN: Noocaf, enhanced coffee for better focus - <AUTHOR> https://noocaf.com <comment><USER2> Those testimonials looked suspicious.

It took me all of 2 minutes to do an image search on Google against Natasha West and Kyle Stevens

Natasha West : [https://unsplash.com/photos/QXevDflbl8A](https://unsplash.com/photos/QXevDflbl8A)

Kyle Stevens : [https://www.bestdentistguide.com/whitening- veneers/](https://www.bestdentistguide.com/whitening-veneers/)

Apparently [https://www.magicmoonlamps.com/](https://www.magicmoonlamps.com/) thought this was a great idea too.

They used a similar photo but called the guy Floyd instead of Kyle.

If you're putting fake testimonials for your products, please don't. It's disingenuous and people can find out really quickly.

<comment><AUTHOR> Thanks for pointing this out!

These are actually remnants from before we launched and are placeholders from before we had any reviews.

We only launched a month ago so will be changing these out today for the actual reviews you can see on the product page!

<comment><USER3> Hmm, this excuse sounds familiar...

[https://www.wsj.com/articles/health-startup-ubiome-used- stoc...](https://www.wsj.com/articles/health-startup-ubiome-used-stock-photos- for-website-testimonials-11558016423)

<comment><AUTHOR> Call it what you will, but I stand by what I said.

Testimonials have been changed and are no longer stock.

</comment></comment></comment></comment><comment><USER4> The ingredients look very similar to most energy drinks.

The only one I don't recognize is Alpha GPC, the rest are standard components of a multivitamin with amino acids.

Crush one of those into your coffee if you really must.

$32 for a 340g bag of ground coffee + vitamins? The goose won't lay eggs forever.

<comment><AUTHOR> There are some similarities, yes.

However what I believe to be very powerful is the caffeine + l-theanine combination ([https://noocaf.com/blogs/articles/caffeine- theanine](https://noocaf.com/blogs/articles/caffeine-theanine)). On top of this we added ingredients that help to further reduce any jitters so you don't get pangs of anxiety with coffee (as a surprising number of people do, even to the point of not drinking coffee anymore) and some that are in general good for cognition and brain health.

As for the price: we aren't a mega corporation. It's not cheap to start something like this, but I aim to reduce prices as we scale.

</comment></comment><comment><USER5> There is very little evidence for the efficacy of any of these compounds in humans.

<comment><USER6> Supplements are unregulated and this marketing page takes advantage of that and other unregulated areas such as the use of the word "natural" (anything can legally be called natural, all natural or 100% natural) as well as fake testimonials. At an absurd price this is cashing in on the high-cost wellness fad. Compared to buying fancy beans and some supplements you can get the caffeine from the coffee and whatever you're convinced the supplements do for much less.

</comment></comment><comment><AUTHOR> Hi I'm Jack, Noocaf founder.

Thought I'd provide a bit of a background into what (and why) Noocaf is:

As with a lot of businesses, Noocaf started as a problem I wanted to fix for myself. I've always been a big coffee fan, both of its taste and its effect profile. I've also always been into self improvement through mediums like working out, meditating, and nootropics.

My routine with coffee used to be that I would make a cup and either put l-theanine powder into the drink or take an l-theanine capsule so that I was getting what is by far my favorite and in my opinion the most underrated nootropic combo.

As someone who likes to experiment and optimise I soon wanted to add more nootropics to my routine, though quickly realised that I'd be that guy who was drinking coffee and having 5 or 6 pills at the same time (or dumping a whole bunch of white powder into my coffee) to get the effect that I wanted. This may not be an image many people want to paint of themselves and definitely not the simplest way to get the effects I was looking for.

So I thought, as long as the taste of the coffee isn't tainted by extra ingredients: why not just find a way to infuse these nootropics into the coffee blend and make my own great tasting coffee enriched with the nootropics I wanted to consume on a daily basis?

What's super interesting to me is the idea of coffee with purpose. There are so many nootropics and supplements out there that can be utilized to make a coffee for focus, for working out, fasting, and more. I think the idea of getting more out of a simple cup of coffee and having it be tailored to an effect profile is something definitely worth exploring.

Our first product is the Focus blend, which is designed with high performance in mind. We carefully selected these nootropics to complement the caffeine, enhancing the brain-boosting effects. They also improve brain functionality and health, as well as providing benefits to memory and recall. This special mix of attention-sharpening and calming ingredients results in a relaxed and focused state of mind.

I think a big market for Noocaf is those who are put under heavy cognitive strain for long periods of time, so wanted to post here and see what you all think and hopefully come away with some good feedback for improvement.

<comment><USER7> Does it come with electrolytes?

<comment><AUTHOR> It does not currently but pre/post workout blends in the future likely will.

</comment></comment></comment><comment><USER8> Have you considered adding creatine? It reduces fatigue in cases of sleep deprivation, and for vegetarians it boosts IQ a little bit.

It dissolves well and doesn't have much taste. It's inconvenient to take as capsules or powders, because the volume you'd want is so high.

[https://examine.com/supplements/creatine/](https://examine.com/supplements/creatine/)

</comment><comment><USER9> Sorry bub but fancy marketing and branding is going to convince that your coffee is anything more than beans + a couple common supplements.

</comment><comment><USER7> It's got what geeks crave.</comment>


Breaking the MintEye image CAPTCHA in 23 lines of Python - <AUTHOR> http://www.jwandrews.co.uk/2013/01/breaking-the-minteye-image-captcha-in-23-lines-of-python/ <comment><USER2> For those interested, the minteye captcha has been broken by other methods as well.

Speech recognition: <https://gist.github.com/4520930>

Laplace: <https://gist.github.com/4564489>

Fourier transform: [http://nbviewer.ipython.org/urls/raw.github.com/rjw57/mintey...](http://nbviewer.ipython.org/urls/raw.github.com/rjw57/minteye- captcha/master/captcha-test.ipynb)

<comment><USER3> Could someone explain the general idea behind using FFT in these situations (excusing my ignorance)? I recall it from my university days but that was completely out of context. I know you can use it to decompose audio signals into frequency components, I don't understand how it applies to images though.

<comment><USER4> The basic idea is that just as an audio signal can be decomposed into frequency components, an image can be decomposed into frequency components, but in two dimensions. (Imagine stripes of varying frequency and direction instead of sine waves of various frequencies.)

Sharp edges in an image have high frequency components, just like a sharp transition in an audio signal. Filtering the high frequencies will blur the image, while filtering low frequencies will enhance edges.

This is very oversimplified, but will hopefully give you a bit of an idea. I encourage people to learn more about Fourier theory, because it explains a lot.

One introductory page that you may find useful is: <http://cns- alumni.bu.edu/~slehar/fourier/fourier.html>

<comment><USER3> That makes perfect sense. Thanks for the link - that's given me a much better intuitive understanding. I'll have a dig around some of the other transformations mentioned now.

</comment></comment></comment></comment><comment><USER5> Hhahahaha, comment on the blog

"Please post the Visual Basic codes for this. The language you post in the article is not Visual Basic.

Thank you."

xD God !

<comment><USER6> It's totally sarcasm

<comment><USER7> Wasn't there something about how 'high' sarcasm is undistinguishable from the real thing.

Unfortunately I can't make the distinction, and I think it's most likely the commenter was serious

<comment><USER8> Sounds like Poe's Law to me. <http://en.wikipedia.org/wiki/Poes_law>

</comment></comment></comment></comment><comment><USER9> This is nitpicking. Well, maybe not.

This code does not take a swirled image and solve it. It takes a set of images with various swirl levels and finds the one with shortest sum of edge lengths.

The code does not do any un-swirling of the image. It also uses external libraries to convert to grayscale, apply the Sobel filter and the sum of edges.

I other words, it is far from breaking the swirled CAPTCHA in 23 lines of Python. If you had to write the un-swirling, gray-scaling, Sobel filtering and summation code in Python you'd be looking at a much larger pile-o-code.

The demo and the intent is good. I just wish the title didn't say "23 lines of". I don't think the article would have suffered at all if the title was "Breaking the MintEye image CAPTCHA with Python".

Again, the intent and what is being demonstrated are excellent. The title, in my not-so-humble opinion, is hugely misleading.

<comment><USER10> >it is far from breaking the swirled CAPTCHA in 23 lines of Python

Except it does break the captcha in 23 lines. I don't get your point.

Why would he need to do any 'un-swirling' in the first place? That's not the point of the captcha. The captcha provides you with multiple images and you have to pick the least swirled one - which is exactly what his code does.

And you argue that because he did use external libraries it's not 23 lines of python. Of course it is. _He_ had to write only 23 lines that make up the logic to break the captcha. If you'd argue that one would have to add any previous work (or lines) done by other people to the linecount of your own project, I could also argue that 'print("Hello")' is not one line of python, because it uses some standard python library, which in turn calls native code, that native code would make a few syscalls and at some point resulting in multiple hundred lines of compiled kernel C code being executed.

So the linecount of his 23 lines python script would actually be: 23 lines of python + various libraries + the entire Linux kernel = roughly 20 million lines of code. Good job.

And yes, that was nitpicking. And I'm angry for some reason.

<comment><USER9> No need to be angry. None of this is going to remove food from your table or affect your life in any way whatsoever. Take it easy. I am just opening the topic for conversation. We can discuss things without pulling out semi- automatic weapons, right?

When someone publishes code and says something like "solved <problem> in <n lines> of <pick your language>" there generally is an implied "my language is better than yours" subliminal message that, for some strange reason, is based on line count.

Let's not debate the merit of line count as a measure of how good a language may or may not be. If we are going to go there I'll write the same solution in APL and we'll start counting characters instead. Again, line or character count, in my opinion, is not a measure of the "superiority" of a language.

The 28 line solution is not pure Python. How do we define pure Python. Let's just say that "solved <problem> in <n lines> of Python" to me means that you download and install Python:

<http://www.python.org/download/releases/2.7.3/>

Install nothing else whatsoever and write code that solves <problem>.

OpenCV, which is what the author used for a couple of aspects of this code, is an extensive (and really cool) C++ library that is being accessed from Python. Here's the source:

<https://github.com/Itseez/opencv>

I'll leave it up to the reader to find the source for the Sobel method.

A much more honest title could have included something like "solving <problem> with <n> lines of Python using OpenCV".

As I said before, the intent of the article and what is being demonstrated are good, no, great. I just wish the title was a little more reflective of reality. That's all. No need to get worked-up. This is not that serious of an issue. Just a comment.

<comment><USER11> Someone has disagreed with you -- which you anticipated would happen. Therefore he must be angry?

I agree with him. If it's wrapped up in a library and it's distributed by a package manager then it doesn't count towards the total SLOC of your project.

<comment><USER9> Read the last line of his post please. He said he was angry, no me.

>If it's wrapped up in a library and it's distributed by a package manager then it doesn't count towards the total SLOC of your project.

Think about what you are saying. With that logic I can write a library in C++ that evaluates all images in the current working directory for least edge length and returns the name or index of the winning file. My Python program, then, might look something like this:

        import magic
        print magic.evaluate()

And then I claim that I have written a program that solves a swirly CAPCHA in two lines of Python.

C'mon.

I you want to count true Python lines, download the language from python.org and write a solver without the use of any add-on libraries. Then we can talk about Python lines.

<comment><AUTHOR> [http://www.jwandrews.co.uk/2013/01/breaking-the-minteye- imag...](http://www.jwandrews.co.uk/2013/01/breaking-the-minteye-image- captcha-in-34-lines-of-python/)

OK, I didn't write the JPEG decoder, but how far do you want me to go?!

<comment><USER9> That's great.

Please note that I did not have any issues with the method you used. My issue was only with the title of your post not being accurate. You used Python + OpenCV in addition to pre-un-swirled images.

To me at least, solving the problem with n lines of Python means that I email you a single swirled image and you, using Python as downloaded from python.org and nothing more, write a solver. The image doesn't even have to be JPG. It can be an easy to read non-compressed format. That'd be fine. But you'd have to un-swirl and do everything else, which is a lot more code.

Don't loose any sleep over this. It isn't important. Your original post was fantastic and very informative. My comments were only about the title and how, in every language camp, there's sometimes a tendency to look down upon other languages by quoting such nonsense as line counts.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> That's a really clever way of detecting the swirl effect. As for simply measuring overall sharpness, that could be thwarted by normalizing the FFT after swirling. That wouldn't help against the sum-of-edges technique though (in fact, it would make it worse).

</comment><comment><USER13> remembered me of how Interpol was willing to keep the reverse twirling capability as confidential: [http://thelede.blogs.nytimes.com/2007/10/08/interpol- untwirl...](http://thelede.blogs.nytimes.com/2007/10/08/interpol-untwirls-a- suspected-pedophile)

</comment><comment><USER14> Its actually a super fun excercise to do. I did that years ago with a browser game that showed images on login, some years ago. Also with python. It was a fun learning experience because it's so much more visual then your typical "todo list tutorial" or "hello world".

</comment><comment><USER15> Since the algorithm can only detect the relative swirled amounts of a set of the same images, a solution could be to, rather than using a set of the same image, use a set of different images swirled different amounts (or with some other transformation), and the user must select the unmodified image.

</comment><comment><USER6> I'd love to see MintEye respond.

<comment><USER2> "Like all CAPTCHAs, sliding CAPTCHA can be cracked . Still, it's more usable and mobile friendly"

<https://twitter.com/EyeOnTheMark/status/291837470486704129>

</comment></comment><comment><USER16> Very interesting article, and an insightful realization. I had no idea Sobel could do that.

<comment><USER17> What? Edge detection is pretty much the only use for the Sobel operator.

<http://en.wikipedia.org/wiki/Sobel_operator>

</comment></comment><comment><USER5> Breaking captcha is bad, mkay :)

<comment><USER18> If you're using a CAPTCHA, you're doing it wrong already.

<comment><USER19> Yeah, all these CAPTCHA stories are definitely making me think there's no refuge left of simple tasks humans can do that machines can't.

What's the replacement? Tying every account to a mobile phone or credit card?

<comment><USER18> The thing is, CAPTCHAs are used to validate it's a human operating the interface, not a robot.

That's a stupid idea, based on some weird assumption that it's somehow safe to give access to your system for a user, but not for a robot. If the only thing stopping people from messing your system is automation, you should rethink it from the get go.

<comment><USER19> > weird assumption that it's somehow safe to give access to your system for a > user, but not for a robot.

Nice point.

> If the only thing stopping people from messing your system is automation, > you should rethink it from the get go.

Are there systems that can't be abused through automation, or where that's much less of a concern? (Serious question, not intended as snark.)

Maybe throttling activity is a better approach... it's volume that is harmful to the system, not interactions with programs per se.

Don't know how you throttle spammers distributing activity across a number of accounts and IP addresses though...

<comment><USER18> > Are there systems that can't be abused through automation, or where that's > much less of a concern?

Let's analyze. One place CAPTCHA has been used a lot is in comment forms. It increases the opportunity cost of posting a comment via automation by introducing a complicated challenge.

Fair enough. But that doesn't solve the _underlying_ problem (abusive comments). You'll _still_ have abuse and spam, it will just come from humans instead of robots. If I'm a blackhat SEO with a budget, I pay people to solve CAPTCHAs all day and spam the internet (that's what happens).

If you design the comment system to fight abuse from the ground up (e.g., markov chain spam filters, user flagging), you don't need to care anymore wether it's a human or a robot behind the POST, because all the spam actually _improves_ your training set. You defeat spammers with their own data.

</comment></comment></comment></comment></comment></comment><comment><USER20> Love it. +1.

But wouldn't an obvious answer to this be to use a background full of swirl, then add people onto it, then swirl again (so the background swirl is swirled twice).

Then when you'd be "deswirling" with you code, you'll be finding something always swirled?

maybe even use a background who's "swirled right" and then "swirl left" the "backgroung + people" (or nyan cat or whatever)?

<comment><USER18> > Then when you'd be "deswirling" with you code, you'll be finding something > always swirled?

It wouldn't change what he is detecting, which is the sum of the edges.</comment></comment>


Scientists Deconstruct the Harry Potter Sorting Hat - <AUTHOR> http://time.com/4809884/harry-potter-house-sorting-hat-quiz/ <comment><AUTHOR> They used research-backed personality inventories to crowd-source the traits of each House, then made a statistical model to match your affinity to all four.</comment>


An Upcoming Authentication Solution for Elixir Phoenix - <AUTHOR> https://dashbit.co/blog/a-new-authentication-solution-for-phoenix <comment><AUTHOR> Some discussion on Reddit about this - [https://www.reddit.com/r/elixir/comments/fsgj3x/an_upcoming_...](https://www.reddit.com/r/elixir/comments/fsgj3x/an_upcoming_authentication_solution_for_phoenix/)</comment>


Haskell's Niche: Hard Problems - <AUTHOR> http://cdsmith.wordpress.com/2011/03/13/haskells-niche-hard-problems/ <comment><USER2>
       Haskell shines at making frequent fundamental changes.

my experience of haskell is completely the opposite of this. haskell is terrible when you want to make big changes---it's often quicker to just rewrite completely from scratch. fixing all the little inconsistencies is a real pain. want to add an extra parameter to a type class? want to remove an attribute from a data type? sure, if you picked the right abstraction from the start, this is a non-issue, but that's not how it is with a hard problem.

the type system gets in your way; this is why i now use python (colleagues have noted the same problems when using c++; we prototype in python, scale up in c for the slow bits). often when i'm fleshing out ideas or trying a new approach, i don't need everything to be consistent---just the bit i'm working on, even then i don't care too much. if things don't quite work, i'd like a quick way to find an inconsistency (often i write a consistency checker function for each class that i use when debugging, that checks types and structures).

i find haskell fundmentally useless for tackling hard (research) problems for this reason alone.

<comment><USER3> > The type system gets in your way

I think that anyone saying this suffers from a fundamental misunderstanding of how to use a type system like Haskell's. I encounter it often with people coming from C/Python who start writing typeclasses, data types etc. and only afterwards start adding functions and then struggle trying to get the right types for their functions.

Fundamentally, programming is a search problem. We are searching for a sufficiently good solution (program) in the space of all possible solutions (all possible compiler inputs). I am of the opinion that the type system can significantly help in searching for such a "good solution". In the words of Conor McBride: "Sometimes it's easier to search for good programs in the space of well typed programs, rather than in the space of ascii turds."

Personally, I start by writing down type names (just names, no implementation or anything) and function names of what I want to do with said types and most importantly what the type of said functions should be (inventing new type names and functions as I discover I need them).

Say, for example, a game would start with GameWorld type and an update function of type "GameWorld -> some events -> GameWorld".

This way I start fleshing out the skeleton of WHAT I want to do, without any regards yet as to HOW I plan on accomplishing it. Once I think my skeleton is somewhat complete I define all my functions to be "undefined" (actually, I do this as I write them down, of course). undefined has all possible types in Haskell, so it always typechecks, but throws an exception when evaluated.

Then I start replacing the undefined's with actual code that matches the signature and start implementing typeclasses/datatypes as required while writing the functions. I will occasionally compile the code to see if everything so far actually type checks.

<comment><USER2> that's how you write haskell the first time. that's fine. easy, nice, simple. you've written a lot of code; it compiles, it works, no problem.

then you realise that you want to change a data structure or a function needs some information that you want to weave through it or you used the wrong abstraction. then it's a pain in the ass.

<comment><USER4> There are two points to be made here:

1\. With a static type system, you can make the key change at the data type declaration level, and then chase down the compiler type errors to figure out all the place you need to change it. You need to do extra legwork in dynamically typed languages to get the compiler to tell you this sort of info: how many times have you changed some variable name in Python and forgotten to update the reference somewhere obscure?

2\. Yes, if you have some pure, functional code, and then you decide you want to have a feature that uses state, fixing things up can be a bit tedious. I think there's an interesting space here for refactoring tools that help take out the tedium. I will also note that this restriction is a good thing sometimes: you know how you add a global here and a global there and suddenly you have an unmaintainable blob of global state? Haskell asks you to "slow down" and re-think about what you actually want to do. It encourages you to use the minimum amount of language features to get the job done. This leads to more maintainable code.

<comment><USER5> I think there's a space for partial compiles that Haskell doesn't take advantage of well. Correct me if I'm wrong.

Assume that you've got a simple dependency tree on defined names in a program and you want to refactor the type of one of the leafs. The type system is _great_ here in that it'll track those mismatches straight up to the root.

The trouble appears if you want only an experimental type change at the leaf. It would be excellent to get successful _partial_ compiles threaded up the tree. In other words, it should be possible to make a cut in the type dependence tree in order to test a functional subset of your program under more rapid experimental changes.

Once you've iterated enough to decide on a new formulation of that branch of the dependency tree, you can continue attempting typechecking up to the root node.

<comment><USER4> One thing I've seen done here is copy paste the method into a new name (usually with a leading underscore) and then make the appropriate changes. It's not hooked up to the rest of the system (but that wouldn't have worked anyway), and you can still compile it and poke at it.

<comment><USER5> That works as long as it's a simple dependency tree like I was talking about and there is a single cut with that property. It's pretty likely you'll have convergent nodes (like changing a datatype's interface late in the game) and there's no simple cut that can let the thing compile.

I mean, sure, this can all be manually avoided, but I still think that having GHCi "compile as much as typechecks" instead of stopping before loading any symbols would be nice.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> I find that Haskell shines at a specific _kind_ of hard problem, namely those involving significant math. It's particularly good for problems involving abstract algebra. Why?

1) The Haskell community tends to rely heavily on math—it's part of the culture. Part of this comes from Haskell's functional nature, which favors mathematical approaches, and part of it comes from a community tradition of using _more_ math to work around the limits of functional programming.

2) A typical Haskell library is usually an algebra defined over an abstract data type. Haskell has excellent abstract data types, and good pattern matching, which makes this a natural way to work.

3) The QuickCheck library generates random input data for functional programs, and makes sure that certain invariants always hold. For example, you can use QuickCheck to verify that (reverse (reverse xs)) is equal to xs.

4) If your problem domain has a certain mathematical structure (specifically, a topos or a closed Cartesian category), then there are some really slick ways to map it onto a Haskell DSL. See my notes at [http://www.randomhacks.net/darcs/probability- monads/probabil...](http://www.randomhacks.net/darcs/probability- monads/probability-monads.pdf) (PDF) for an example involving Bayesian filtering and particle systems.

5) Haskell's strong type system make it feasible to work at _very_ high levels of abstraction without going completely insane. For ordinary CRUD apps, I actually prefer dynamically-typed languages, but when it comes to scary math, a good type system makes life a lot easier.

6) Because Haskell is so steeped in math, it tends to force my brain into "math mode" well before I reach the mathematical heart of a problem.

So I'm not convinced that Haskell is the ideal language for small DSLs. (Ruby shines, here, for almost exactly opposite reasons.) But when there's enough math involved, Haskell is absolutely a wonderful language.

<comment><USER7> Have you tried F#? If so, could you say the same things about F# as well?

Also could you explain the last point "Because Haskell is so steeped in math"?

</comment></comment><comment><USER8> "Reason 1: Haskell shines at domain specific languages.

"[...] if you can embed a domain specific language into the programming language you're using, things that looked very difficult can start to appear doable. Haskell, of course, excels at this. [...]"

Says who? Just because I can write `f(x)` as `f x` with full featured operator precedence and associativity tables doesn't mean I can introduce new syntax into the language. I can't even find a moral equivalent of ELisp's rx package for Haskell.

"[...] Lisp is defined by this idea, and it dominates the language design, but sometimes at the cost of having a clean combinatorial approach to putting different notation together."

This is either weird nonsense or the old variable capture argument. You can write crappy functions in any language too, that doesn't mean we should take away the programmer's ability to write new ones.

"Reason 2: Haskell shines at naming ideas. If you watch people tackle difficult tasks in many other areas of life, you'll notice this common thread. You can't talk about something until you have a name for it."

I applaud the willingness to appeal to natural language, but this isn't even close to true.

This blog post seems like an argument for Haskell's value based on specific times the author has enjoyed programming in Haskell. I call post hoc ergo propter hoc shenanigans.

<comment><USER9> "Just because I can write `f(x)` as `f x` with full featured operator precedence and associativity tables doesn't mean I can introduce new syntax into the language."

There's two definitions of "DSL" and I find mixing them to be a bad idea. I prefer to confine "DSL" to a fully-fledged actual _language_ , with its own parser and evaluator. Haskell is just about the only language I know that makes this easy enough to actually consider as the solution to a problem.

Then there's "glorified API to look like a language", a definition I don't like but is clearly in current use, and the one used in this post. Haskell is pretty decent at that, but of course you can't escape from the fact you're using Haskell. But then, people pretty freely use the term "DSL" in Ruby where you can't escape the fact that you're in Ruby. In either case, "adding new syntax" is not actually part of the general definitions of DSL nowadays. The "adding of new syntax" is an illusion, and I guess that's part of why I don't like this definition of DSL, there's actually an element of deception in it and in practice I have found people not 100% comfortable with the base language the DSL is in often end up deceived and less effective. And either way, if you really, truly want some new syntax, Template Haskell can mostly keep up with Lisp anyhow, but not entirely. There may not be, at this moment, a precise match for the rx package already on Hackage but I believe Haskell has all the necessary functionality to create one, up to and including compile-time regular expression compilation. Though given the way Haskell works, it is also the case that compile time RE compilation is way less useful, because it's easy to make it compile only once even at run time with the way Haskell works. As is so frequently the case, what takes a macro in Lisp does not require a macro in Haskell.

"This is either weird nonsense or the old variable capture argument."

No, it's a functional purity argument. Lisp in practice has the same problems with mutation and state as pretty much any other language. Insert debate about importance of functional purity here. (That is, I'm not making the argument now, I'm just letting you know that's what was being referenced.)

<comment><USER10> > _No, it's a functional purity argument._

I'm pretty sure it's a reference to the fact that combining macros can be risky; one of the risks is variable capture.

<comment><USER9> That's a subset of the functional purity problem. It's a relatively-commonly- understood one, but Haskell has taken the argument far further than most, such that that is now only a subset of a larger argument about state and mutation. The argument is not merely that you've got some state being mutated in a bad way, the argument is that the fact you've got state mutating _at all_ is a bad thing. In this case, it's state in the compiler phase, but it's ultimately the same argument. The argument would be that combining any two things that mutate state is inferior to trying to combine two things that don't, at any level.

I'm personally still skeptical of this general argument, but I mean that in the "true" sense of skeptical. I think in a lot of ways the benefits have panned out as promised but there's some costs still swept under the rug. Rather a lot of the development ongoing even now in the core Haskell community can be viewed as various ways of lowering or eliminating the costs associated with their approach.

(For instance, the still-ongoing, but rapidly converging, changes associated with trying to create a decent string library that does not treat strings as linked lists of integers. On the one hand you can see it as normal feature work, but on the other it's a way of trying to go from having an unusually bad string story induced by functional purity to potentially having an exceptionally _good_ one. It's this sort of work that actually keeps me interested in Haskell; I don't know of any other community doing so many actually new things to prove out their philosophy, instead of rehashing mutation-based OO in yet another way.)

</comment></comment></comment></comment><comment><USER11> haskell's niche: PL theory research. as a professor of mine said, "it's an incubator for PL theory". that said, are there merits to non-PL researchers learning it? i think so. for one, the functional paradigm is increasingly being integrated into the mainstream languages and the "genpop" is starting to use FP concepts to solve problems. haskell is supposedly "the purest" functional language. therefore, if you like FP, studying haskell is like tapping into "the source".

while i find functional constructs aesthetically more pleasing than imperative ones, a reason i often hear given in support of using functional languages is the promise they offer w.r.t concurrency/parallelism. in the words of the same professor i quoted earlier, "concurrency/parallelism is currently broken in most other languages. it's like building skyscrapers atop matchsticks". i believe this now. i also believe haskell's approach to be quite a big step in the right direction (STM and atomically). an outstanding paper on the topic exists here [http://research.microsoft.com/en- us/um/people/simonpj/papers...](http://research.microsoft.com/en- us/um/people/simonpj/papers/stm/beautiful.pdf)

regarding haskell's type system...in my opinion, hands down the best currently in existence. yes, it's hard to work with at times. i strongly disliked it at first. yes, the type errors can be a bit cryptic at times, at least until one gets used to them. the upside? there be real magics there. and i'm not talking about devices that begin with the letter i. the majority of the time, if i can get code to type check it works flawlessly.

working with it over the last 6 months has really changed my thinking about programming languages and has actually swayed me to go further in that direction in my studies.

don't mean to sound like a fool in love, but ...</comment>


Trym app help you view, optimize and convert SVG icons - <AUTHOR> http://kontentapps.com/trym <comment> angry-hacker I'm not the target audience, but are people really paying for that kind of apps in the Apple's ecosystem? Assuming OP is also the author of the app and has some knowledge about the market.</comment>