Levenshtein Distance - <AUTHOR> https://en.wikipedia.org/wiki/Levenshtein_distance <comment><USER2> We wrote a generalization of the Levenshtein distance algorithm for regular expressions in Hyperscan ([https://github.com/intel/hyperscan](https://github.com/intel/hyperscan)), which allows you to do Levenstein distance matching over regular expressions. It is not necessarily a cheap thing to do, given that incautious use of edit distances and permissive regular expressions can result in automata that match almost anything.

I believe that the first widely available approximate regex matcher was Tre ([https://laurikari.net/tre/](https://laurikari.net/tre/)). However, while everybody wants to talk about approximate matching like they have something to say, most algorithmists act like they forgot about Tre.

<comment><USER3> agrep is from 1988. I heard about it some 10 years later.

[https://en.wikipedia.org/wiki/Agrep](https://en.wikipedia.org/wiki/Agrep) says "A more recent agrep is the command-line tool provided with the TRE regular expression library. TRE agrep is more powerful than Wu-Manber agrep since it allows weights and total costs to be assigned separately to individual groups in the pattern."

<comment><USER2> You're right. I can't believe I forgot agrep (so addled was I with my "forgot about tre" pun). Our departmental Unixen machines had agrep in '91, I think.

</comment></comment></comment><comment><USER4> Here's a helpful tip in case it comes in handy one day:

If you need to run Levenshtein Distance over tens of thousands of strings, perhaps as part of a fuzzy search, and you only need to match on distances of 0 or 1, not any distance in general, then there's an optimization on Levenshtein Distance where you hard code the function just for distances of 0 or 1. It's not difficult to do and the payoff might enable new applications of what would otherwise be too costly.

<comment><USER5> If you're fuzzily looking up members in a fixed corpus, you can also employ the trick from [https://github.com/wolfgarbe/SymSpell](https://github.com/wolfgarbe/SymSpell), which is essentially just to, for each string in your corpus, enumerate all of the variants of that string with one letter removed, and put both the original string and each variant in a hashtable as a key, with the original string as the value. To do a lookup, you do the same enumeration (original query plus each variant with one letter dropped), and look all of them up in your hashtable. The values you get out of that are a list of all of the strings in your corpus at edit distances 0 or 1, and potentially some at edit distance 2; you can do a subsequent Levenshtein calculation on each to weed out the distance-2 strings, but you only have to do it on this massively reduced set rather than on your whole corpus.

So like, to index a string "hello", you'd add all of {"hello":"hello","ello":"hello","hllo":"hello","helo":"hello","hell":"hello"} into your table, and for the query "gello", you'd look up all of ["gello", "ello", "gllo", "gelo", "gell"], and get a match on "ello"->"hello", then do a Levenshtein calculation dist("gello","hello") to confirm it's within ED=1 (it is), and be done. (Bonus: the same method works with Damerau-Levenshtein distance as well.)

<comment><USER6> In which case do you wind up with distance=2? I figured it'd jump out at me in an obvious way, but it hasn't.

<comment><USER7> Can't think on a real word example, but abc would match bcd in the above algo, as both permute to bc, but they are distance 2. So, any case where one letter is deleted and another is added at a different position.

<comment><USER8> Boat and Oats is a real world example with Levenshtein distance 2 that would pass through the algo

<comment><USER5> Yeah, that's the crux -- two different real words storing the same variant but having omitted different letters.

</comment></comment></comment></comment></comment></comment><comment><USER9> Is there a comprehensive guide that explains the differences between the different libraries for fuzzy/approximate string matching?

There is SymSpell, agrep, fuzzywuzzy, closestmatch, fuzzysearch, fuzzyset, SimString, ukkonen and others in Github.

The problem is I don't have a clue which one to use for each case, which are the cases and what are the differences between the libraries.

A writeup illustrating several cases with a recommendation of a library and a practical example for each one would be very helpful.

Btw, if anyone has experience with this type of stuff I would be interested in talking to you (mail is in profile).

<comment><USER10> There's also metaphone[1] for words that might sound similar but are spelled differently for when levenshtein might be too sensitive.

For example, someone might have mistyped "berry" as "barry" but you don't want to match it against "terry" (both have a levenshtein distance of 1)

[1] [https://en.wikipedia.org/wiki/Metaphone](https://en.wikipedia.org/wiki/Metaphone)

<comment><USER11> Thanks, I was searching the comments to see if anyone suggested a phonetic variation. The default Linux spelling checker is usually good enough but sometimes I only remember how a word sounds and the close letter variations are clearly wildly incorrect, while asking a search engine or phone is able to figure out exactly the word I mean nearly all of the time.

</comment></comment></comment><comment><USER12> At my first professional programming gig the company I worked for was selling items on Amazon. The listings were big ticket items that had several sellers competing on them. Any seller could change some of the details of the listing, so suddenly all their competition would be shipping the item with wrong specs. I wrote a program that periodically grabbed the titles of our live Amazon listings and compared them to our database of what we actually wanted the title to read. I used Levenshtein Distance as a sort of "severity of change" metric, the program would sort the changes accordingly and send an email to a person on the sales team. It was fun implementing Levenshtein Distance, but it wasn't a perfect metric for this use case. Some of the most severe changes would be a single digit changing; e.g. "2gb" vs "8gb"

<comment><USER13> You can modify the basic algorithm to weight changes differently. I had the opposite case--differing numbers were more likely to be what the user was looking for. I made digit changes cost 1 point, one string being longer cost 1 point per character (suffixes were another case where it was likely the objective) and everything else cost 2.

<comment><USER14> Someday I want to write a diff utility specifically for code. Whitespace addition/removal cost like .1, addition/removal of a curly brace .5. There are a handful of other things default diff does that make it really annoying for code.

<comment><USER15> I did this for a school project, because students would "cheat" on group projects by exclusively changing whitespace in diffs, or moving code around, thereby making it look like they had large commits.

We tracked code "moves", whitespace changes, and "trivial" changes via Levenshtein distance--which isn't great for capturing most simple renamings-- and then marked them up separately in an "augmented diff".

We briefly looked into doing AST edit-distance, but that turned out to be infeasible for many reasons.

Over-all it helped the prof grade faster, at least in the degenerate cases.

<comment><USER16> Measuring a student's impact by a project on the number of lines of code they committed can be very misleading.

Sure, in the extremes it will give sensible results (someone wrote only 3 lines, or wrote 90% of the code base).

But minor, questionably motivated refactorings can create huge commits, while someone may come up with some great design while walking around for hours and sketching on paper, which all ultimately boils down to a short function.

Dumb metrics like this cannot replace actually talking with the students, asking questions about their choices and alternatives, having them do presentations or lab notebooks/logbooks etc.

<comment><USER17> > Dumb metrics like this cannot replace actually talking with the students, > asking questions about their choices and alternatives, having them do > presentations or lab notebooks/logbooks etc.

As said in the parent:

>> Over-all it helped the prof grade faster, at least in the degenerate cases.

</comment></comment></comment></comment></comment></comment><comment><USER18> I was once asked in an interview how I'd find a row in a relational database where the value in one column was the shortest levenshtein distance away from a search key, and I was absolutely stumped when I realized I didn't really know a sort/search that optimized that way. Of course, by the time I realized I could email the interviewer for the answer an awkward amount of time had passed.

So now I ask HN: is there any "good" way to do this short of some kind of answer like "find a database that natively offers levenshtein distance search"? Are there actually sorting algorithms for sorting by levenshtein distance (other than just using L as a sort function in a traditional sort function)? Are there search algorithms that work well for searching by L distance?

<comment><USER19> Depending on the RDBMS and the target text (ideally, you're looking to compare an entire column to the search text, and in a known character set, and known casing, etc.), I'd take one of two approaches:

1) write a UDF—in C, or for SQL Server, in pretty much any managed language—that returned the Levenshtein distance between a passed param and a column name, with a "max distance" param to aid in optimizing (e.g., maybe it doesn't bother calculating the distance if the length difference exceeds the max Levenshtein distance)

Then you could write a query that looks something like:

        SELECT product, description WHERE levenshtein(`fobar`, product, 2) <= 2

-or-

2) if you only care about a distance of 1 or 2, create a set of secondary tables where you'd pre-calculate the permutations of every word, with a FK pointing back to the PK of the actual row.

Alternately, a combination of a Levenshtein distance function and an RDBMS that already supports fuzzy text matching seems (to me) to be ideal.

<comment><USER20> I think sometimes it is easy to forget that SQL is but an abstraction over algorithmic processing and shaping of data. A UDF is only a way to extend the built in algorithms, albeit, one which may not be fully integrated with a given query engine architecture and therefore cannot be fully optimized without having access to the engines internals.

I would argue your answer is the best as as far as doing an actual Levenshtein distance calculation.

That being said, sort orders are implemented algorithmically by the query engine. ORDER BY in SQL Server, as I recall implements up to 7 different algorithms built in, depending on the case, as far as I recall. But those algorithms are optimized for the engine and have access to the internals.

I can think of two interesting ways to look at this, one being how one could use an open source database to extend the built in sort algorithms in the query engine, perhaps controlled by collation settings, to get a native levenshtein sort implemented that has access to internals. Collations defining the rules for sort order, but this would probably be within a column, not against a value (although if you're extending the query engine, why not). Example of collation settings [1] and how they control the rules of sort order - perhaps a SET command to define levenshtein order and then a native extension to the Postgres query engine, and now you can sort a column by Levenshtein distances.

The other thing I was thinking was word vectors (but this is similar to Levenshtein distance) and still in a relational database requires something special. Maybe some of the full text search implementations in database could do this out of the box.

[1] [https://docs.microsoft.com/en-us/sql/relational- databases/co...](https://docs.microsoft.com/en-us/sql/relational- databases/collations/collation-and-unicode-support)

</comment></comment></comment><comment><USER21> My first job at Microsoft was to make a system that could catalog the "contracts" or "extension points" installed in Windows. I'd define a category of extensions, such as a COM object registration and my program used normalized levenshtein difference to warn about potential typos. After removing all matching fields names from the instance & class I compared the edit distance in the remaining cross product. If any had a normalized distance < N it produced a warning. It even helped me find a mistyped GUID in a prerelease copy of Windows.

</comment><comment><USER22> We use the Damerau-Levenshtein edit distance for fuzzy matching in our data prep tool. Here is a post about how it works: [http://blog.easymorph.com/2017/08/fuzzy-matching-and- merging...](http://blog.easymorph.com/2017/08/fuzzy-matching-and-merging.html)

Damerau-Levenshtein [1] is more practical then just Levenshtein as it includes transposition - a common cause of typos.

[1] [https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_di...](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)

</comment><comment><USER23> I love this algorithm -- it's so elegant and succinct.

Slightly off-topic, but one of the algorithms I'm most proud of coming up with as an engineer was an offshoot of Levenshtein distance where I had to find all pairs of strings in a set of ~1m strings which were fairly similar. 1m x 1m x O(n^2) is very slow. There turned out be a good way to figure out a lower bound on distances using some precomputation, and that eliminated almost all of the pairwise O(n^2) Levenshtein computations.

I'm anyone's interested, here the description: [https://www.quora.com/What-is- the-best-programming-algorithm...](https://www.quora.com/What-is-the-best- programming-algorithm-that-you-have-ever-created/answer/Leo-Polovets)

<comment><USER24> Your sample use case in your 'At Google' response is eery.

> "At Google, say we had 1 million plaintext passwords..."

<comment><USER23> That wasn't the actual use case and we didn't store passwords in plaintext. But I didn't want to give away the actual use case. In hindsight, passwords are definitely a bad fake example.

</comment></comment></comment><comment><USER25> Only application I've used this for so far is a technique for defining and recognizing gestures. They're called "flash gestures" (or that's what the page where I read about them called them) where you find the closest of 8 directions to the gesture direction and encode each gesture as a sequence of digits 1-8 corresponding to these closest directions, then use the Levenshtein distance algorithm to find the closest matching pattern.

EDIT: Found the reference I used to implement this.

[https://books.google.com/books?id=_1rSBQAAQBAJ&pg=PA85&lpg=P...](https://books.google.com/books?id=_1rSBQAAQBAJ&pg=PA85&lpg=PA85&dq=gesture+recognition+algorithm+%2B%22levenshtein%22&source=bl&ots=lW4f4MvuV0&sig=ACfU3U2psj9IXvQHlnhakiWYFIF-7ONTLw&hl=en&sa=X&ved=2ahUKEwiP5bDimpfkAhUESN8KHeWkDyo4ChDoATAAegQICBAB)

</comment><comment><USER26> Here's my attempt to implement Levenshtein Distance in Go, with lots of comments:

[https://github.com/cbartley/diffy/blob/master/src/diffy/diff...](https://github.com/cbartley/diffy/blob/master/src/diffy/diff/diff.go)

<comment><USER27> Very nice! If you're looking for one more variant to add: LD with a limit on the acceptable distance can be done more efficiently, and is nearly always what you want in practice.

I wrote a commented version years back that might be a handy reference; it's since been deprecated because we moved it out of StringUtils, but the original code is here [https://github.com/apache/commons- lang/blob/master/src/main/...](https://github.com/apache/commons- lang/blob/master/src/main/java/org/apache/commons/lang3/StringUtils.java#L2297)

<comment><USER26> > _This implementation only computes the distance if it 's less than or equal > to the threshold value, returning -1 if it's greater. The advantage is > performance: unbounded distance is O(nm), but a bound of k allows us to > reduce it to O(km) time by only computing a diagonal stripe of width 2k + 1 > of the cost table. It is also possible to use this to compute the unbounded > Levenshtein distance by starting the threshold at 1 and doubling each time > until the distance is found; this is O(dm), where d is the distance._

That's pretty cool, especially the doubling scheme. I'm using a modified form of Levenshtein Distance for comparing lines when diffing files, and that's pretty expensive since code files that are thousands of lines long are not uncommon. Since you are usually comparing one file to another version of itself, the differences are often small though, so an incremental approach would really pay off.

</comment></comment></comment><comment><USER28> We used this when building an app to determine 'meaningful' discrepencies between prelimary radiology reports (done by Residents) and final reports (done by attending radiologists). It's a fun algorithm to implement that isn't as simple as it seems. I think there's lots of uses for it.

<comment><USER29> +1

</comment></comment><comment><USER30> is Levenshtein distance a "metric" in the sense of "metric spaces"? For example, the TaxiCab metric.

[https://en.wikipedia.org/wiki/Taxicab_geometry](https://en.wikipedia.org/wiki/Taxicab_geometry)

[https://en.wikipedia.org/wiki/Metric_(mathematics)](https://en.wikipedia.org/wiki/Metric_\(mathematics\))

Also, this looks like could be related to the Hamming Code ? Error correcting codes.

[https://en.wikipedia.org/wiki/Hamming_distance](https://en.wikipedia.org/wiki/Hamming_distance)

<comment><USER31> Yes, it is a metric. The following is BS "proof by restatement of requirements".

1\. Always >= 0 (what is a negative character edit?)

2\. It satisfies identiy in that if no edits are required to make them equal, metric distance is 0 and they are equal (applies to either ordering of strings A->B and B->A)

3\. It is symmetric (run the edits backwards to get B->A instead of A->B)

4\. It satisfies triangle inequality. exercise left to reader, but intuitive since the edit distance is always the "shortest path" through character changes between two strings A and B, and deviating to visit an intermediate string C would not decrease number of edits from A->C->B vs original path A->B

I'm sleep deprived due to "offspring induced insomnia" so take this with a grain of salt.

</comment></comment><comment><USER32> I became intimately acquainted with Levenshtein's distance for my summer internship project, and have implemented it myself a few times for interviews. It was central to an event de-duplication system I built. Amazing how such a seemingly trivial idea can actually be a) non-trivial to implement and b) be so powerful.

EDIT: By 'seemingly trivial', I mean that if you explain it to a lay person it _seems_ easy at first blush - not that it is actually trivial.

<comment><USER33> The problem with "trivial" is that it's in the eye of the beyolder. And everything can seem "trivial" after you've studied it long enough ;-) For instance, I'd say the matrix implementation of Levenshtein is quite trivial...

</comment></comment><comment><USER34> I've used Levenshtein distance to sort fuzzysearch results by order of relevance. Worked very well combined with a special-case to sort exact matches first.

<comment><USER35> Why would a special case be necessary? The edit distance for two identical strings is zero.

<comment><USER34> It's an autocompletion for emoji, so several letters will be missing regardless.

I'm that case, a long word like "helicopter" will be placed at a greater distance than "elipsis" despite being what I want "heli" to actually match, so I just added a "startsWith" check to override such issues.

<comment><USER36> Have you experimented with n-grams for approximate matching? I've found it works very well, especially if you add special characters to the alphabet to mark "start of string" and "end of string": like "heli" would have trigrams {"^he", "hel", "eli", "li$"}, which would certainly be a better trigram match to "helicopter" than to "elipsis" (sic).

[https://en.wikipedia.org/wiki/N-gram#n-grams_for_approximate...](https://en.wikipedia.org/wiki/N-gram#n-grams_for_approximate_matching)

<comment><USER34> I haven't, mostly because it's fast enough already and I don't really feel like implementing this stuff myself.

<comment><USER37> Postgres has support for trigrams and indexes on them. I used them to help match addresses at a previous company, worked fantastically for the part we used them for.

<comment><USER34> It's a client-side library.

</comment></comment></comment></comment></comment></comment></comment><comment><USER38> I use the levenshtein distance for NLP data cleaning/spell correction. I also use a bloom filter that can validate if the word does not exists in my corpus, so I don't have to run the algorithm for every word.

</comment><comment><USER39> This link was gray for me because I recently used Levenshtein distance to sort an arbitrary list of strings by finding a "comparison word" that gave the ordering that I wanted. Definitely more of a cutesy (and difficult to explain) way of doing things rather than practical but Case statements are so boring. It was nice that Presto had it built in.

Did some more digging to see if there was a unique comparison word that would enable me to sort any group of strings but found that it was impossible. Still a fun afternoon of learning.

</comment><comment><USER40> The names 'edit distance' and 'levenshtein distance' are a bit unfortunate.

When I was trying to match batches of OCR'd strings to known strings, I knew some algorithm like this had to exist, but I could not find it. My searches kept turning up Jaccard similarity and Hamming distance again and again.

(Once I found it and began working on an implementation, I started noticing my compiler would find miss-typed variable names and suggest the correctly- spelled variable for me.)

<comment><USER41> There are a lot of unhelpfully named metrics, algorithms, etc. but I actually think "edit distance" is fairly illustrative as these things go.

Another good one is Manhattan distance (or better yet, city block distance) in place of Hamming distance. It gives a nice mental picture of the number of road segments you'd need to travel in a city to get from point A to point B.

<comment><USER42> One step further (multi-layer Manhattan distance) will also get you optimal DNA, RNA and protein sequence alignment (Myers, Gotoh and others, 1977-1981). The edit path (often discarded) can be repurposed for simulating RNA and protein folding, as well as calculating DNA melting points (used for DNA-DNA hybridization studies).

</comment></comment></comment><comment><USER9> What type of algorithm does Algolia use for its search (HN search specifically)?

Is it anything like this (typeahead example)? [https://rawgit.com/jeancroy/FuzzySearch/master/demo/autocomp...](https://rawgit.com/jeancroy/FuzzySearch/master/demo/autocomplete.html)

What is the name for this kind of approximate string matching?

<comment><USER43> Algolia's engine relies on an optimised Damerau-Levenshtein + prefix matching algorithm.

</comment></comment><comment><USER44> For some reason the wikipedia article does not mention the more efficient Myers diff approach. I have recently implemented it here: [https://jasonhpriestley.com/array-distance-and- recursion](https://jasonhpriestley.com/array-distance-and-recursion) (for edit distance but Levenshtein should work similarly)

</comment><comment><USER45> I'm using Levenshtein distance combined with Trie/DAWG in my iOS keyboard, to detect typos and offer suggestions

</comment><comment><USER46> If you consider strings as list of characters you can abstract levenshtein to work on any two lists. For example i use this with list of types to provide a hoogle-like search for some programming languages stdlib. Ironically i don't know what hoogle actually uses but i think they atleast considered such an approach.

</comment><comment><USER47> I'm surprised this page and the page on "string metrics" don't say more about the mathematical implications of defining a distance metric on strings. Something interesting must pop up if you apply theorems about metric spaces from analysis and topology to the set of strings with edit distance.

<comment><USER48> This metric induces the discrete topology, which isn't very useful. Regarding its metric structure, you could apply the contraction mapping theorem but I can't think of any interesting contractions off the top of my head.

I'm just spit-balling here... but when a natural metric induces the discrete topology then it's probably more useful to look at connectivity graphs induced by the metric. For instance the graph on all strings with two strings connected by an edge if their edit distance equals 1. Restrict to words in a dictionary and you have a basic spelling suggestion algorithm.

Edit: the analyst in me really wants a translation-invariant abelian group structure ... then we can do Fourier analysis...

<comment><USER49> > Restrict to words in a dictionary and you have a basic spelling suggestion > algorithm.

Yep.

[https://norvig.com/spell-correct.html](https://norvig.com/spell-correct.html)

</comment></comment></comment><comment><USER50> Had to use this to improve the speed of our admin user search after the table got fairly large (>1mil). The biggest problem was that it's a general search that searched display_name as well as email. We ended up cleverly omitting the email search unless '@' was included in the search string.

</comment><comment><USER51> This was the first algorithm I ever searched for on wikipedia, back when I first learned to code. Good memories.

<comment><USER52> And then there's that strange yet oddly persistent certainty that back then it was called the Levenshtain distance.

</comment></comment><comment><USER53> Ugh. At a past job I was tasked with matching records, based on street addresses and business names, from various systems. Levenshtein Distance was pretty instrumental in helping find potential matches once we had cleaned up the address format and business name as much as possible.

<comment><USER54> I did the same thing. I was merging our billing db with that of a business we had just bought. Due to the nature of our business, customers could exist in both.

So I had a routine that normalized the address, as best I could anyway, didn't help that it was all just one varchar field. Then implimented Levenshtein Distance, messed with the weighting a bit to fit our particular data and away it went. Saved a bunch of headaches. It wasn't perfect, but it was better than hand matching a couple thousand accounts.

</comment></comment><comment><USER55> Another cool metric that's used commonly when measuring quality difference between two texts is a BLEU score ([https://en.wikipedia.org/wiki/BLEU](https://en.wikipedia.org/wiki/BLEU))

<comment><USER56> Yup, it's used a lot in machine translation to compare different systems, and to a lesser extent in natural language generation (where it's completely inappropriate...)

</comment></comment><comment><USER57> For numerical sequences, see dynamic time warping (DTW) algorithm.

[https://en.wikipedia.org/wiki/Dynamic_time_warping](https://en.wikipedia.org/wiki/Dynamic_time_warping)

</comment><comment><USER58> What other languages have it built in?

[https://www.php.net/manual/en/function.levenshtein.php](https://www.php.net/manual/en/function.levenshtein.php)

<comment><USER59> D does [https://dlang.org/library/std/algorithm/comparison/levenshte...](https://dlang.org/library/std/algorithm/comparison/levenshtein_distance.html)

</comment></comment><comment><USER60> We have implemented and using this in MarkLogic:

[https://docs.marklogic.com/spell.levenshteinDistance](https://docs.marklogic.com/spell.levenshteinDistance)

</comment><comment><USER61> I like using levenshtein distance for building out bayseian networks. Its works well in conjunction with other distance/ similarity metrics that are better for continuous data.

</comment><comment><USER62> I stumbled across the concept of Damerau-Levenshtein about two years ago. Applying it to a a large database of very messy consumer data in Pandas returned near miraculous results!

</comment><comment><USER63> OT: Why are so many mathematical objects named after some dudes and not after what they represent, i.e. in this case edit distance? Never understood this obsession.

<comment><USER17> Historically, at least to my eyes, this seems to derive from how mathematics has been cited by mathematicians.

Before a thing is called 'Levenshtein Distance' it will be used by someone saying 'calculating distance between these two strings the way Levenshtein did in [0]'. Eventually, when enough people do this, or someone gets particularly bold, the method will instead be given its name (occasionally, the original author will be so bold, but that seems rare). If I were looking to procrastinate more than I already am I would go find some examples of this, examples that unfortunately don't spring immediately to mind though I'm sure I remember their shadows.

As to why they are not named for what they represent - it's only after the thing has been invented, tested, and named that we realise that it is the correct or best way to do a thing. Often there are multiple ways of doing things (this is Levenshteins edit distance) and so they are named to distinguish them from each other. The survivor is not renamed just because the others are forgotten.

[0] not a real reference

<comment><USER64> I believe Turing Machines are an example. Turing originally called them "automatic machines" or a-machines.

</comment></comment></comment><comment><USER65> 103 comments and not one mention of line coding!

This distance metric is exceptionally useful when picking the states to map to in line codes with error correction/detection.

</comment><comment><USER66> [https://en.wikipedia.org/wiki/agrep](https://en.wikipedia.org/wiki/agrep)

</comment><comment><USER67> This is a major part of how [https://safepass.me/](https://safepass.me/) works.

</comment><comment><USER68> I used this to implement a spam detection system a few years back. It was super helpful.

</comment><comment><USER9> Anyone know of a library that is good for OCR post processing (correcting errors)?

</comment><comment><USER69> What's the difference between Levenshtein distance and Hamming distance?

<comment><USER70> [https://en.wikipedia.org/wiki/Edit_distance#Formal_definitio...](https://en.wikipedia.org/wiki/Edit_distance#Formal_definition_and_properties)

> Given two strings a and b on an alphabet Σ (e.g. the set of ASCII > characters, the set of bytes [0..255], etc.), the edit distance d(a, b) is > the minimum-weight series of edit operations that transforms a into b. One > of the simplest sets of edit operations is that defined by Levenshtein in > 1966 Insertion of a single symbol. If a = uv, then inserting the symbol x > produces uxv. This can also be denoted ε→x, using ε to denote the empty > string. Deletion of a single symbol changes uxv to uv (x→ε). Substitution of > a single symbol x for a symbol y ≠ x changes uxv to uyv (x→y).

...

> Other variants of edit distance are obtained by restricting the set of > operations. Longest common subsequence (LCS) distance is edit distance with > insertion and deletion as the only two edit operations, both at unit cost. > Similarly, by only allowing substitutions (again at unit cost), Hamming > distance is obtained; this must be restricted to equal-length strings.

</comment></comment><comment><USER71> ripgrep [Discussion] Fuzzy Search

      https://github.com/BurntSushi/ripgrep/issues/1053

</comment><comment><USER72> I've been playing around with the data made available [1] by the states that are part of the Streamlined Sales Tax system [2].

The boundary file address records include, among other things:

      address range start
      address range end
      odd/even/both flag
      directional prefix
      street name
      street suffix
      directional postfix
      zip code

The address range specifies the range of numerical addresses that the record applies to, and the odd/even/both flag indicates if it is for just the odd address, just the even addresses, or both.

The direction prefix and suffix are supposed to be one of {blank, N, S, E, W, NE, NW, SE, SW}, but some states botch that and I've seen EA, EB, NO, SO, and WE end up in there.

The suffix field is for things like LANE, AVE, ST, and the like. It is a 4 character field, so a street that residents thing of as ELM AVENUE would be name=ELM suffix=AVE in the boundary file. Or it would be name="ELM AVENUE" with no suffix. (There are even some with AVENUE in the name and something else in the suffix, such as ROOSEVELT AVENUE ALY in Albany GA. The ALY is the suffix). (There are a 318 distinct suffixes across all the SST states, for the curious [3]).

Anyway, my point is that the data is kind of loose.

I want to be able to take what a user puts on an address form:

      address
      city
      state
      zip code

and find a good match in the boundary files. Most people manage to put something reasonable in the city, state, and zip code fields, and manage to put their street address at the start of the address field, so I can pretty reliably get the zip code and the street number. It's the rest of the address line that is problematic.

At first I was trying to figure out what in there might be prefix, postfix, and suffix to isolate just the street name. But because of both the aforementioned looseness of the boundary data, and differences between how the user might think of their address and how their state thinks of it, this didn't work too well.

I was considering keeping an index that maps words that occur in street names to those street names, and then looking for the longest word in the user supplied address and looking up all streets in the user's zip code that contain that word and that include the user's street number. If that returned more than one match, the idea was then to look at the prefix, postfix, and suffix and see how well those matched with things in the user address. Looking at a few addresses that had given me trouble, though, it looked like this plan to resolve them would still be tricky and unreliable.

So then I decided to try a Levenshtein distance approach. Given zip, street number, street, where street is whatever the user included in the "address" field after the street number, I pull up all records in my boundary DB that match on zip code and street number. I don't even include street name in the query.

Then for each result, I take the prefix, name, suffix, and postfix and join them separated by space. I then simply take the one with the smallest Levenshtein distance from what the user supplied. With this, I no longer care if something is name=ELM, suffix=AVE or name="ELM AVE", suffix="".

This has worked remarkably well. Even if the user throws on stuff that doesn't belong (some people manage to put the city, state, and/or zip in the address box) it usually doesn't throw it off. It raises the Levenshtein distance of the correct match by sometimes a lot...but it also raises the incorrect match distances too, so the correct one still wins.

The boundary files also have secondary address fields, for things like apartments, unit, suites, and the like. These are an abbreviation (APT, e.g.) and low and high ends of a range, and an odd/even/both flag. Unfortunately the ranges for these aren't always numeric. E.g., there is 695 E MORRILL AVE in Columbus, OH, which has secondary fields of "AP", "A", "M", and the odd/even/both flag set to both. I assume that means Apartments A-M.

So far, I've ignored the secondary address fields. Every time I've seen more than one result for the lowest Levenshtein distance, the matches have been close enough to each other to be in the same taxing districts, so would have the same rate and the tax would be allocated to the same entities, so further resolution is not necessary.

At some point I'll take my scripts that clean up the raw rate and boundary files and transform them into a form more suitable for DB use, and that make sqlite and mysql databases from those, and that do the address matching and also sales tax lookups by address, and put 'em on Github, but that will probably not be very soon.

[1] [https://www.streamlinedsalestax.org/Shared-Pages/rate-and- bo...](https://www.streamlinedsalestax.org/Shared-Pages/rate-and-boundary- files/rate-and-boundary-file-updates)

[2] [https://www.streamlinedsalestax.org/](https://www.streamlinedsalestax.org/)

[3] [https://pastebin.com/tQLGTGS9](https://pastebin.com/tQLGTGS9)</comment>


Windows Package Manager CLI (a.k.a. Winget) - <AUTHOR> https://github.com/microsoft/winget-cli <comment><USER2> In other words, the prospective APT or YUM of the windows world.</comment>


The Most Clever 'Zip Bomb' Ever Made Explodes a 46MB File to 4.5 Petabytes - <AUTHOR> https://www.vice.com/en_us/article/597vzx/the-most-clever-zip-bomb-ever-made-explodes-a-46mb-file-to-45-petabytes <comment><USER2> Are there documented examples of these things out in the wild, in viruses etc? Just wondering what the actual threat profile is.

<comment><USER3> I used to work on commercially available proxies for scanning email and web traffic. A large amount of time was spent unpacking compressed content so that someone couldn't smuggle in naughty pictures by hiding them in a zip file, for example.

Zip bombs were designed to DOS companies using such products by wasting disk space (or just CPU time) while they were unpacking and scanning the contents.

The mitigation is easy though. We kept track of the compression ratio as we were unpacking the archives and tossed out anything where it got too large. No legitimate content compresses 1000:1.

<comment><USER4> > Zip bombs were designed to DOS companies...

Maybe today - but back in the day, Zip bombs were done as pranks on users of BBSs; upload one named something intriguing, and the hapless user would download it and unzip it to their tiny (then) hard drive, and a file that was supposed to 4 MB blows up into several hundred MB to gigs, filling their drive if they weren't paying attention.

> The mitigation is easy though. We kept track of the compression ratio as we > were unpacking the archives and tossed out anything where it got too large. > No legitimate content compresses 1000:1

Interesting and neat solution!

</comment></comment></comment><comment><USER5> Ouch.

Though this begs the question: Would you rather: 4.5 Petabytes or 9 Petanibbles?</comment>


Speaking of tattoos – Interview with Micah Baldwin - <AUTHOR> http://techcrunch.com/2010/08/31/speaking-of-tattoos-interview-with-micah-baldwin-ceo-co-founder-of-graphic-ly-tctv/

<comment><AUTHOR> I count myself lucky to know Micah. He is a genuinely inspirational guy, and his blog learntoduck.com is great too.</comment>


Amazon's Winner-Take-All Approach to Small Business - <AUTHOR> https://www.bloomberg.com/opinion/articles/2019-02-19/amazon-uses-search-to-undercut-small-businesses-on-its-site <comment><USER2> It's getting tougher than ever to disrupt the previous industry disruptors. How does one compete these days?

</comment><comment><AUTHOR> [https://archive.vn/dgioh](https://archive.vn/dgioh)</comment>


The Whittling Part Of the Brain - <AUTHOR> http://jeff-vogel.blogspot.com/2011/03/whittling-part-of-brain.html <comment><USER2> "I have found that, in general, our brains our smarter than we are. They want what they want, and if my brain wants to spend a while in front of a computer screen stabbing orcs, who am I to tell it it shouldn't?"

I have several problems with this statement, starting with the fact that it was derived from a rationalization of the author's own habits, not a meaningful empirical observation.

Further, this is a formula for addiction. Games, porn, gambling, food cravings and drugs are all similar in that they self-enforce a habit through perceived short-term rewards.

Self-discipline is what allows one to accomplish goals. And, it means adapting your brain so that it can supress these hormone driven urges and take action that measurably moves your progress towards a rationally derived goal.

I'm all about some R&R, and fun / entertainment / addictions all have their place. Personally, I become a gamer for about 1 month every 2 years. I even think this author probably has more self-disciplne than he lets on judging by his writing and the history of games he has created. However, the message this article conveys is destructive.

Rationality and self-discipline use a different part of your brain than cravings for short-term rewards. Before you rationalize a time wasting habit, make sure you have adapted your mind to have the self-discipline to switch back and forth. Some people with particularly addition-oriented genetics really should fully abstain from mind-whittling activities if they want to accomplish anything in their life.

My point of view also assumes you are someone who wants to accomplish some substantial goal(s) in your lifetime. If you choose to live a lifestyle of fun / bliss, by all means, whittle away..

<comment><USER3> The takeaway for me wasn't so much that he was trying to rationalize wasting time instead of being productive, but rather that we all do waste time sometimes one way or another and that playing games isn't a "bad" way of wasting time.

<comment><USER2> I would agree with you if not for the last 2 paragraphs. Combine the quote I referenced above with "it's good and healthy and productive for us to like to play computer games".

Its just a reckless way to say 'we all waste time..' - not without making the distinction of the need for self-discipline and noting that there are some people that are very prone to addiction. I have an extended family member who's life outside of games has been completely destroyed by a sustained addiction to video games. He has stated repeatedly that he would rather be dead than give up gaming.

This is a real problem for a percentage of the population, and that percentage is increasing as video games are being engineered to be more and more addictive.

<comment><USER3> The whole "who am I to tell my brain otherwise" seemed to meant to be tongue- in-cheek, and you kinda sound like you're twisting that to try to push your agenda... (no offense intended, I do think addictions in general are bad).

FWIW, just to put things into a bit of perspective, I recall reading some stats saying that the average american spends something like 5 hours per day in front of a TV. (And numbers for, say, surfing the web during work hours weren't particularly exemplary either.)

Regarding social withdrawal, and things like hikikomori (<http://en.wikipedia.org/wiki/Hikikomori>), perhaps the issue isn't so much of self-discipline, but one of social environment, e.g surely such gamers must develop above average game playing skills, but they might not want to put similar effort into real life challenges because of fear of the unknown, or out of spite of parental/social pressure, etc.

</comment></comment></comment></comment><comment><USER4> The argument that "the brain knows better" seems very weak. The brain also wants people to over consume fast food, reward itself with alcohol and drugs, and several other things which obviously aren't rewarding in the context of modern society.

<comment><USER5> Right, I second that. Also, I used to "take breaks" by playing games or putzing around on the internet, until I realized that it doesn't actually help me feel relaxed and energized. But stretching, walking around, getting a cup of tea, and most other things in meatspace get me ready to go back to work.

</comment></comment><comment><USER6> Im always puzzled by the superiority complex people harbour about their 'traditional' pasttimes, especially reading newspapers, watching the 6 o'clock news as if it's important, watching wildlife documentaries, going for runs, etc.

 _Anyway_ , in one of those half asleep dazes the other day, I swear I caught my brain 'compiling' or 'defragging' or something. I could feel my eyes doing REM, and I could see or sense a whole lot of information I'd been studying being chunked, parsed and filed away in a more appropriate part of memory. Of course, that itself was probably just a dream...

<comment><USER7> > I swear I caught my brain 'compiling' or 'defragging' or something.

That might be your brain telling you to switch to a task that doesn't involve one of those activities for a day or two. :P

</comment></comment><comment><USER8> I resent the suggestion that whittling is not productive. I've made utensils in the past while in the mountains when I forgot my fork or spoon.

<comment><USER9> In my neck of the woods, the common word for making something out of wood is "carving." "Whittling" generally means destruction of a piece of wood with a knife.

<comment><USER10> The only use I've seen for it is making trinkets (paper knife anyone) out of wood. Like carving but less refined, only using ones general duty knife not carving tools (chisels, etc.). Like scrimshaw but using wood only and not spending so much time and care.

</comment></comment></comment><comment><USER11> Related: <http://www.tlb.org/busywork.html>

</comment><comment><USER12> Whittling sounds like an evolutionary advantage. Imagine if humans hated to whittle; they would hate spending time collecting little things for a big reward. This could be as simple as collecting root vegetables to feed a stoneage family or creating a arrowhead. To me it seems whittling was an important part of evolving to where we are today. Small progress/steps to lead up to something great.

<comment><USER13> Sure, but the neurobiology of whittling is the polar opposite what we see in gamers. I speak as a doctor, kid who bought his own nintendo (back in the early 90s) and still proud owner of my original cub scout pocket knife.

</comment></comment><comment><USER14> I don't buy it. As far as I can tell the this thing Jeff calls Whittling is simply energy preservation behavior. Brains take up 30% of your overall energy consumption. Thinking has a relatively high energy cost to it and people are hardwired to prefer limiting their energy expenditure that does not generate a return (wage, food, etc).

<comment><USER4> I'm wondering if it's ever been shown that doing something mentally difficult consumes more energy than doing something where the brain is active but that isn't perceived as difficult?

Another explanation could be that the precursor to grinding in EQ is the urge to do things like searching for edible plants or animal tracks, carving tools, cleaning the home, and other things which have obvious evolutionary benefit.

<comment><USER15> The brain consumes more sugar when it's thinking harder. Before functional MRIs were invented, brain imaging required injecting irradiated glucose and scanning for radioactivity in the brain. Tasks requiring attention and self- discipline deplete glucose, and people demonstrably have higher vigilance for repetitive attention-heavy tasks when they've been injected with sugar.

This has been known to science for at least a century. The slightest glimpse at Wikipedia would have sufficed.

<comment><USER4> > people demonstrably have higher vigilance for repetitive attention-heavy > tasks when they've been injected with sugar

That's not at all the same as saying that more difficult tasks actually consume more sugar, any more than programming must consume caffeine because coffee makes some people more focused on it.

<comment><USER15> Meanwhile, in reality, where evidence extends further than whatever I happen to mention, it does deplete glucose.

<comment><USER4> I don't know why you need to be rude and condescending. I don't know much about these things, but cursory research shows that these are at least good questions: <http://www.epjournal.net/filestore/ep08244259.pdf>

"It does deplete glucose" seems to be in question, doesn't it?</comment></comment></comment></comment></comment></comment>


Ask HN: Resources for how to be a first-time dad - <AUTHOR> Tech forum, I know. But I've seen some of the most earnest advice here. So what books or materials should an about-to-be-dad consume? <comment><USER2> There is a blog post on the Art of Manliness about how to fall asleep in 2 minutes based on research done during WWII to help pilots rest [0]. You can begin training now so the middle of the night feedings, etc. can be handled easier.

Beyond life saving things, like prevention of SIDS for example, there are a ton of books out there with different tips or techniques on raising a child. Treat them like you would someone telling you about a new software development process: there might be some stuff that might apply, some of it just doesn't matter or work for your situation.

As time goes on realize your child is their own person. They may not have the same interests or beliefs as you do and that should be okay.

Other practical things: instead of toy/clothes gifts from relatives, setup a college/savings fund and an easy way for them to put money in it as gifts. Don't go overboard on clothes for the first 18-24 months. Keep a couple sentimental things if that's your style but they outgrow clothes sometimes surpassing sizes before they even wear them.

[0] [https://www.artofmanliness.com/articles/fall-asleep- fast/](https://www.artofmanliness.com/articles/fall-asleep-fast/)

</comment><comment><USER3> Look,

The advice I say for the new parents and seems to be helpful is:

"Hang on there until 3-4 month"

What I mean.

The baby is adapting to the parents and the parents are adapting to the baby. Until you learn to take "shifts" [1] and learn how the baby communicates[2] and its sleep pattern[3] set will take some time.

[1] - At beginning both parents will be awake but then you realize that you can't be awake all the time and there is no need to _both_ be awake and tired then you start to make some "shifts".

[2] - Is cry for change diapers? hungry? scared?

[3] - Mostly at 3rd month the baby is already sleeping 6-8h continuous then you will start to have a "full" sleep.

The important thing is: the firsts months are hard adaptation and you may fight with your spouse because of stress.

"Hang in there" it will get better.

You will be amazing father

</comment><comment><USER4> I never found any decent resources, just lots of click-bait. These were my key takaways though:

\- You will freak out about everything as a new parent (feeding, prams, weight, learning). By your second you will take everything in your stride. With good reason. Just bear it in mind.

\- Dont be overly protective, it will backfire, kids will get scrapes, fall, will be constantly sick. So did you. Dont worry. Also dont worry about being a perfect parent / finding the perfect school etc. This tends top backfrire too. Just be a good dad, -put in the time, put in the effort.

\- Let kids play with other kids. Kids are learning, so dont freak out when another kid is mean or whatever. Make time to play with them yourself. Encourage them but never force them to be your definition of a success.

\- Kids learn FAST and they ENJOY it. Seriously. So feed them with knowledge! You can teach latin to a toddler. Kid loves mops? Show him how to make a mop at home. Loves getting dirty? Let them do do your garden. Loves the motorbike, teach em how to change the oil! Dont shy away from complex learning, build hobbies.

\- Phones are crack to kids, especially youtube. Which is a shame as its also great for learning. Best to severely limit.

\- Use basic psychology tricks. Eg you can limit reading to 30 mins before bed when very young so its a treat thats looked forward to. Use diversion tactics.

\- Dont fall for basic psychology tricks. Dont override your spouse, present a joint front. Dont let them use embarasment as a weapon in public.

\- When you first buy a car seat internalise the way to tie it to the seatbelt properly so you dont have to try and remember every time you put it back.

Kids are great fun. Enjoy

</comment><comment><USER5> Having 3 children: There is no magic secret. Sometimes you do great at something and sometimes you fail.

\- read the book "Baby Wise"

\- home is much more important than work or money. Do Not forget this.

\- you wife will have a lot going on physically, mentally, and emotionally. Take as much of the "burden" as you can when the baby is first born.

\- Do Not forget weekly date nights. I repeat...make sure you ALWAYS have weekly date night with your spouse. The child and your marriage will thank you for this.

\- your wife has A LOT going on with her during pregnancy. Tell her daily that you love her, appreciate her, and thank her for being your wife. And...tell her she is beautiful every day - this is important.

\- Your life will get A LOT busier. Share the house chore, and make time for each other.

\- Children are a lot easier to raise compared to managing your marriage with a child.

\- and last...have FUN! They grow SO FAST!

</comment><comment><USER6> Do a search, there was a thread on this yesterday and another a couple days before that. The summary is read a book or two but be ready to adapt to your babies individual needs. Prepare to lose any semblance of hobbies or personal time, and say goodbye to a good nights sleep. Oh also your significant other and yourself will almost definitely experience a significant mental health decline within the first 6 months, so I hope you guys already have figured out how to support each other because thinking rationally and building new relationship habits with only 5 hours of sleep a night is a struggle.

Edit: It is also super cool to watch a new person discover the world. Enjoy your time with them!

</comment><comment><USER7> Congratulations! Short answer ... kids do not want stuff nearly as much as they want to spend TIME with you.

The 'what to expect ... ' series is great techie style. Plain spoken. Explains limits on both sides...

They have one for before the baby.

And another for the first few years. Common problem, my child is age x months but has not started walking / talking.

Great for the calming the nerves

</comment><comment><USER8> I mostly just googled stuff when I thought either, "I should learn more about [topic, like SIDS or how to setup nursery]" or "Is [action, like how much food they eat] normal for [age] baby". Googling and using resources like the Mayo clinic or similar can give you multiple perspectives to consider.

</comment><comment><USER9> I posted to a similar question a few days ago. There is one book I would recommend.

Read Bringing Up Bébé: One American Mother Discovers the Wisdom of French Parenting. The chapter on sleep cycles is important.

</comment><comment><USER10> [https://fatherhood.gov/](https://fatherhood.gov/). Ideas in there to expand on.

</comment><comment><USER11> How to think like a Roman Emperor.</comment>


Uber's Online-Only Restaurants: The Future, or the End of Dining Out? - <AUTHOR> https://www.npr.org/sections/thesalt/2018/10/23/658436657/ubers-online-only-restaurants-the-future-or-the-end-of-dining-out <comment><AUTHOR> > "When we see people searching for something but not finding it, that signals > to us that there's an opportunity and there's unmet demand," says Elyse > Propis, who leads Uber Eats' virtual restaurant initiative across North > America.

>

> So the company approaches an eatery and suggests creating a virtual side > restaurant, with those dishes people are craving but can't get.

<comment><USER2> Many restaurants in downtown areas have separate order desks / staff / kitchens for online, it's a significant and growing percentage of their business. Space is almost wasted on a single floor of tables, generally it's more efficient because of expensive real estate and cheap people to just order.</comment></comment>


Ask YC: An analysis of Java's Class Library - any ideas on what it means?  - <AUTHOR>

For a while I've been working on an opinion piece on Object Orientation for my company blog (http://www.jodoro.com).
I decided to try and generate a few facts and figures to flesh my article out. With that in mind, I did a quick analysis of the Java Class Library (JDK1.5 to be precise). What I found was curious, so I decided to share some of the raw figures whilst I mulled over my article.
If you take a look at the class library for Java, you'll find that the majority of the libraries either inherit directly from java.lang.Object, or are just one level of separation away (In fact this 1,2 level is ~60% of the class library). If you take the Swing package out of the equation, almost half the library (47%) directly inherits from Object.
The maximum level of inheritance depth is 7. This depth amounts to less than half a percent of the total. Perhaps not surprising to anyone who has worked with Swing, but this maximum level only occurs in the javax.swing package. It includes deeply nested things like javax.swing.JCheckBoxMenuItem.AccessibleJCheckBoxMenuItem. Moving upwards - Swing is also 65% of the 6th level of depth, with java.util being the other majority. Swing also represents 25% of the 5th level of depth. Again, if you're familiar with Java and Swing this is not a big surprise.
Exceptions take up around 13% of the class definitions. If you take exceptions out of the previous examples, the figures go up. 48% of non-exception classes are at level 1, and 73% are at level 2 or below. Excluding Swing again these figures leap to 57% and 80% respectively.
Turning to interfaces. Interfaces only nest to a level of 4 - and 70% of all interfaces are either root (34%) or one level of extension (36%). Of the roughly 2,700 classes that ship with JDK 1.5, around 40% of them implement interfaces. Interestingly, this is reasonably consistent across most packages. Even quite hefty packages such as AWT and Swing clock in at about 45% each. Aside from outliers, most packages fall in the range of 30% to 50% (classes that implement interfaces).
Some other facts - Around 20% of classes are what I'd call "The most basic" - both at depth 1 (directly inherit from Object) and do not inherit from any interfaces. In other words, of the 60% of classes that don't have an interface, a third also do not inherit from anything aside from Object.
I've got a dozen theories on this - Some controversial, most mundane. The most logical conclusion to leap to is that the JDK is made up of a lot of concepts that come from systems programming, and aren't necessarily well suited to OO. However, right now I'm mostly just curious rather than particular on an agenda or soapbox.
Any before I went off on any crazy tangents, I wanted to ask YC/HN what they thought of all this? <comment><USER2> What would be interesting is to slurp a bunch of OSS java and see how people's use of classes (either by imports, or by method calls) correlates to implementing an interface and/or not inheriting directly from Object. Maybe a bunch of stuff is legacy?

</comment><comment><USER3> i just think that as it developed from the hot new thing to the ubiquitous language, things got tacked on and wrenched in just to get the functionality in there.

<comment><AUTHOR> That might be borne out by the stats - these figures can vary greatly between packages. For example, Swing is quite significant.

However, that implies that these figures would be higher (greater levels of abstraction) if things had been designed holistically...</comment></comment>


Bram Moolenaar responds to Neovim - <AUTHOR> https://groups.google.com/forum/m/#!topic/vim_dev/x0BF9Y0Uby8 <comment><USER2> _Total refactoring is not a solution._

To Diego: Prove him wrong. I suspect that Bram has been maintaining the code base for so long, that he might need a bit more than a mere message to warm him up to the idea of a refactoring to the scale that you're undertaking. Also, you might not be the first person to contact him with such a proposal. I wonder how long it took for others before they gave up. Build something tangible and contact him again later with proof of concept, something that would make this more than just a fleeting dream.

To people questioning Bram's position: You need to look a bit further than your nose. The fact that you want to create plugins or extensions does not put you in the mainstream as a Vim user. It's an extremely popular editor of almost religious proportion, with lots of existing plugins and extensions. So we know that people use it and making plugins is possible. With all of its annoyances it gets the majority of the job done for most users. One can't just wake up one day and say that they'll change things, not even Bram at this point.

The most desirable outcome in my opinion would be to have Neovim be a continuation of Vim (Vim 10.0?) rather than just another fork, and for that, having Bram on board would be a boon. His longevity as a maintainer is impressive.

<comment><USER3> Good points but I'm not convinced there exists a compelling reason to fork vim. From a user perspective there are no points of differentiation between NeoVim and vim. As an outsider looking in, it appears to me that NeoVim exists only to scratch the itch of some developers writing vim extensions.

Is that reason enough to exist? Maybe. But what's the long-term aim here? Without a strong point of differentiation to attract users and thus developers NeoVim will be left forever playing catchup with Bram's original creation -- constantly rewriting new features to remain relevant and forever fixing both old vim bugs and new NeoVim-specific bugs.

<comment><USER4> "From a user perspective there are no points of differentiation between NeoVim and vim."

This is shallow and short-sided opinion. If neovim achieves goal of "easier to maintain codebase" then it will have much faster feature development, bug fixing, and extension development. All extremely user facing and exciting.

<comment><USER3> > This is shallow and short-sided opinion. If neovim achieves goal of "easier > to maintain codebase" then it will have much faster feature development, bug > fixing, and extension development. All extremely user facing and exciting.

I think this is a bit naive. There's no plan of any kind laid out for NeoVim, not even a goal against which progress can be concretely measured. New and compelling features must be well thought-out, well designed and well executed; they don't just fall out of the sky as a consequence of "easier to maintain code".

</comment></comment></comment></comment><comment><USER5> To be fair to the authors of neovim, there were patches they wrote and submitted to vim, which added exciting new features I would like, which were rejected.

People who basically want him to never change (and that is fine) can be happy with mainline. There are others who would like to see some significant updates, particularly allowing better threading support.

<comment><USER6> What were the features they submitted? Stuff that's mentioned on the github page or something else?

<comment><USER7> At Floobits we tried to submit set_timeout/async stuff, and couldn't get anywhere with Bram. Months of work wasted.

[https://groups.google.com/forum/#!topic/vim_dev/-4pqDJfHCsM%...](https://groups.google.com/forum/#!topic/vim_dev/-4pqDJfHCsM%5B1-25-false%5D)

<comment><USER8> Your patches were broken. People spent plenty of time trying to help you fix them.

Your response is to tell the Internet that Vim is wrong for not accepting broken patches?

<comment><USER9> Did you read the same thread I did? Floobits addressed all the issues.

It came down to a fundamental design decision—and after months of hard work, they got a non-answer.

<comment><USER10> Bram's last post was 10/21/13, and a month later Bjorn confirmed in his fork that he could reliably freeze VIM and had segfaults that he had yet to fix. The final word seemed to be:

"There are enough vim users out there (myself among them) to warrant supporting a fork for as long as we're around. We'll try to keep it up to date with the latest changes."

I don't think anyone should expect Bram to chase them down for updates to the patch. Reading through the discussion, it seems clear to me that while good- natured, the feature was not ready to be merged into VIM.

<comment><USER7> We actually continued to work on the fork until it was clear Bram wouldn't be responsive to any more changes. Once it became clear he wouldn't accept the patch we stopped wasting more time as we are trying to create a viable business. We weren't going to sink our startup for the sake of vim, as much as I love the editor. We spent a long time, it was a frustrating experience, so yes we tell the internet when it comes up. Any of the freezes we found we would have loved to patch back into vim had this gone differently, and it would have been a lot of work.

<comment><USER8> My only takeaway from the thread was that you approached the process of patch review begrudgingly, and seemed to assume that everyone should understand/appreciate your node-derived vocabulary and design.

Each iteration of the patch was provided with a whine about whether it was ready for inclusion _yet_ ... when the reality is, that as the patch author, it's _your_ job to make sure there aren't more bugs and the patch is ready for inclusion.

Every time someone found another issue was a red flag to any sane maintainer that your patch wasn't ready for inclusion. Every time you whined about having to iterate, you made it clear that you couldn't be trusted to have written something stable and maintainable.

If you can't deal with working on mature software, stick to startup code.

<comment><USER7> Nothing you've said is true.

<comment><USER8> And nothing you've said is true. Fortunately everyone can compare our subjective realities against the actual thread.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> Perhaps this was mentioned the other day, but Neovim did make me think of this old chestnut from Joel Spolsky:

[http://www.joelonsoftware.com/articles/fog0000000069.html](http://www.joelonsoftware.com/articles/fog0000000069.html)

I realize Neovim is being characterized as an "aggressive refactor," but I'm sure that's the way Netscape thought of it as well.

tl;dr Things you should never do: rewrite the code from scratch.

<comment><USER12> With that thinking we never get Linux (NeoMinix), tmux (NeoScreen), Clang (NeoGCC), WebKit (NeoGecko), Subversion (NeoCVS), or Vim itself (NeoVi/Elvis/Stevie/etc).

<comment><USER13> Huummmmm no

Neovim is being sold as a VIM refactor, but apparently with no new functionality

 _Very_ different from Linux (had an aim of being a playground for new features), tmux is different from screen as Clang is different from GCC, Subversion is a totally different beast from CVS

<comment><USER12> > Neovim is being sold as a VIM refactor, but apparently with no new > functionality

Where did you get that idea? When I read the project page ([https://github.com/neovim/neovim](https://github.com/neovim/neovim)), two of the explicit goals are:

"Enable the implementation of new/modern user interfaces without any modifications to the core source.

"Improve the extensibility power with a new plugin architecture based on coprocesses. Plugins will be written in any programming language without any explicit support from the editor. "

It is exactly designed to be a playground for new features, but with the goal of them not having to be in the core.

<comment><USER14> The original mailing list message says:

"There's a new project that is called Neovim that seeks to refactor and modernize the codebase"

<comment><USER15> That's the necessary step 1 to allow further enhancements

</comment></comment></comment></comment></comment></comment><comment><USER16> well of course. Vi was specifically written by Bill Joy to optimize for every character, see

[http://www.theregister.co.uk/2003/09/11/bill_joys_greatest_g...](http://www.theregister.co.uk/2003/09/11/bill_joys_greatest_gift/)

And there is no reason we could ever improve on that 35 years later. we need those optimizations...

"It took a long time. It was really hard to do because you've got to remember that I was trying to make it usable over a 300 baud modem. That's also the reason you have all these funny commands. It just barely worked to use a screen editor over a modem. It was just barely fast enough. A 1200 baud modem was an upgrade. 1200 baud now is pretty slow.

9600 baud is faster than you can read. 1200 baud is way slower. So the editor was optimized so that you could edit and feel productive when it was painting slower than you could think. Now that computers are so much faster than you can think, nobody understands this anymore. "

<comment><USER17> Whenever I use vim through a lagging SSH connection, I'm very happy that it's still usable (at least more so than other editors).

<comment><USER18> If you can stand the fact that it's only got the original feature set, ex- vi.sf.net provides a version of the original vi that runs on modern unices.

I use it on a daily basis and have found that it copes with conference wireless far better than vim does.

</comment></comment></comment><comment><USER19> Why aren't there other vim committers?

Bram's conservatism makes perfect sense for someone who has to maintain this huge, consequential, intimidating codebase. Why hasn't he gotten some help?

<comment><USER20> Big old code bases, have a lot of what I call as 'tribal knowledge' associated with them. People go out, and do experiments only to burn their hands. Over years, the old maintainers generally accumulate a lot of ideas and 'tribal knowledge' to know why 'pie in the sky' ideas won't work.

Nevertheless, people still try. A lot of them fail, a few succeed.

The vim code base is around 300K lines of C code. I'm sure its nothing like Java where 90% of the code is get/set methods, high level delegating classes and exception handling code. 300K lines of code which does the actual work is simply too much code. And one should exercise great caution while dealing with it. Regression is just one thing, I'm sure there are a lot of places in the code which can silently break things you can't detect only to pop up in some strange case in the wild.

<comment><USER21> I can't speak for vim, but the following smacks of code that _needs_ a refactor:

 _300K lines of code which does the actual work is simply too much code. And one should exercise great caution while dealing with it. Regression is just one thing, I 'm sure there are a lot of places in the code which can silently break things you can't detect only to pop up in some strange case in the wild._

</comment></comment></comment><comment><USER22> This reminds me to the origin of Inkscape, as a fork from Sodipodi. The (only) Sodipodi committer didn't want too many patches, and he wanted to stay with C (no C++ code).

Some people forked Sodipodi, and created Inkscape. I can remember that one of their first tasks was to make the code compatible with a C++ compiler.

Nowadays, Inkscape is a very good tool, and I hope that Neovim has a great success too.

</comment><comment><USER23> Vim is not a healthy project. Example: They support transparent encryption for documents, but refuse to address or document the shortcomings of their implementation. It's of course a fully custom implementation of a half- homerolled protocol.

Main flaws: Apart from implementation issues, non-authenticated encryption susceptible to bitflipping.

</comment><comment><USER17> It's very easy to look at code base full of weird special cases and think "this could be such much easier if I started from scratch", but in the end if often turns out that all this cruft is there for a reason. It's essential complexity that you wrongly identified as artificial complexity. And the rewrite ends up being much more work than originally anticipated.

I don't think there's a way to learn that except by making the same mistake yourself, possibly more than once.

<comment><USER5> That is an overly simplistic viewpoint. Vim is written in an inedibly old style of C which is not taught anymore, or used by most people. The code if fill of typedefs for AMIGA and OS/2\. Useful to a small number of people certainly, but making progression harder too.

This is not a total rewrite, it is a massive overdue refactoring.

<comment><USER17> > The code if fill of typedefs for AMIGA and OS/2\. Useful to a small number > of people certainly, but making progression harder too.

Sounds like the code of pretty much every non-trivial C program that is meant to run of many platforms, not just the five or ten most popular ones.

I'm not saying that this is a good thing, just that I don't know any good alternatives that don't sacrifice compatibility with lots of platforms.

<comment><USER23> It should say ifdefs, not typedefs. Inline ifdef directives everywhere is a real nightmare.

The very first commit in neovim includes this log:

         - Process files through unifdef to remove tons of FEAT_* macros

<comment><USER8> Which is exactly the sort of pointless "refactoring" that someone does instead of actual refactoring.

It also makes merging back to vim difficult to impossible.

Expect to see 'neovim' development dissipate rapidly.

</comment></comment></comment></comment></comment><comment><USER24> My $20 is totally worth the "risk". Here's some dude from Brazil, that has at least some demonstrable skill, willing to take a month and have a go at improving a tool I use every day. For the price of four coffees.

As for Bram, he can, and most likely will continue his good work on vim. I am kind of disappointed that he is not more supported of this effort though.

</comment><comment><USER25> That seems like a fair response. I tend to agree that sweeping refactorings tend to take a lot of time and it's better to break off small bite size pieces. That being said I really like some of the proposed improvements into vim. I wonder if a similar approach could be taken as when vi was improved to create vim, or perhaps that's what added all the complexity Neovim is trying to solve.

</comment><comment><USER26> _As vi was a popular editor amongst programmers and system administrators, there initially was doubt whether Bram 's 'improved' version could achieve the quality and fan following of the original. But since its first release for Unix systems in 1992, Vim has effectively eclipsed the original Vi, having won several awards[3] and has been referred to as one of the most popular text editors._

\- Excerpt from [http://en.wikipedia.org/wiki/Bram_Moolenaar](http://en.wikipedia.org/wiki/Bram_Moolenaar)

</comment><comment><USER27> The old Unix ways are dying...

Or, at least, I think that's the best way to summarize the essential miscommunication between these two camps. Vim is, in the spirit of Unix, a single purpose tool: it edits text. Why would it need threading? async? background jobs? You can't type in two places at once, right?

NeoVim wants to be something like an IDE-lite. Text editing, yes, but with the ability to do background compilation/syntax checking, a bit of debugging here, some REPL-ish things there. For that you need a better async story, better process control, etc.

So, in the end, I don't actually see a problem here. Vim will cater to one audience, NeoVim to another. For my money (or donation to Uganda), I'm happy using tmux splits, sending text between vim windows and REPL windows, and keeping Vim primarily as a text editor with syntax highlighting. I would appreciate if _some_ events could be backgrounded (e.g. syntax check on file save), but I don't think this needs a complete refactor.

<comment><USER28> In some ways, neovim keeps to the Unix spirit better than vim. People already use vim as an IDE, but it doesn't work well in many cases. And the alternate user interfaces are not as good as they could be.

If you've got a clean separation between the editing engine and the rest of the IDE suite of processes, that is the Unix way in my view.

</comment></comment><comment><USER29> I get the idea that Bram thinks it's going to be more of a rewrite than _I_ think it's going to be. As in, I assumed that neovim would not be a complete rewrite or something, but more targeted refactorings.

Of course, Bram knows the codebase and I don't, and I may well have misunderstood the neovim plan, so I'm not sure which of us understands.

</comment><comment><USER30> Abrash wrote in his graphics programming black book, and I quote, Carmack's law ->

"I'll take this opportunity to coin Carmack's Law, as follows: Fight code entropy. If you have a new fundamental assumption, throw away your old code and rewrite it from scratch. Incremental patching and modifying seems easier at first, and is the normal course of things in software development, but ends up being much harder and producing bulkier, markedly inferior code in the long run, as we'll see when we discuss the net code for QuakeWorld. It may seem safer to modify working code, but the nastiest bugs arise from unexpected side effects and incorrect assumptions, which almost always arise in patched-over code, not in code designed from the ground up. Do the hard work up front to make your code simple, elegant, great—and just plain right—and it'll pay off many times over in the long run."

<comment><USER31> If you watch any of Carmack's recent keynotes you'll see hes advocating doing the opposite now. Whats changed is the multi million dollar risk profile associated with modern game releases, the huge increases in the sizes of the teams, the large scale complexity of modern AAA code-bases, etc. And also the fact that most games now are data-dominant rather than being sold entirely on some new game-engine feature(s).

</comment></comment><comment><USER32> Every 10 years or so I think some energetic programmers should set about writing God's Own Text Editor. There have been a lot of really useful technologies that have come from those efforts, whether it was EMACs, Gosling EMACs, vi, vim, pfe, visual slick edit, sublime text, or even wordpad. An editor is a fairly complete system, well within the capabilities of a single motivated individual, and like cocktail gowns entirely fashion/taste driven.

</comment><comment><USER33> > It's going to be an awful lot of work, with the result that not all systems > will be supported, new bugs introduced and what's the gain for the end user > exactly?

That may be a problem for vim but not for neovim. Neovim doesn't have a legacy user base, they can focus on the currently used systems.

</comment><comment><USER34> The project has already surpassed its requested value in the first 48hs. The enthusiasm reminds me of git-annex.

I really love this business model, the developer works on something he is passionate about, we can support the project and influence its future, all while strengthening the community around it.

</comment><comment><USER35> He's right. With every project, there's a tradeoff between achieving the goals of the project and maintaining compatibility with existing systems.

As a Go programmer, I feel a bit sad/ashamed to be saying this, but look at Plan 9 - it failed to usurp Unix (and its descendants) as the dominant OS, because the latter was "good enough", and was already more widely supported and used.

Plan 9 vs. Unix is a very different comparison than Vim vs. Neovim, and there are a lot of other factors at play, but the same principle actually holds. Improving or extending an existing system in a compatible way is a lot easier than trying to establish mindshare with a completely new tool.

The tradeoff is that extending existing systems leads to cruft and bloat. So _if_ you can establish mindshare with a new tool, you have an opportunity to make a much more elegant one. But that's a big "if".

 _EDIT_ : I think a better analogy may be an attempt to refactor the Linux kernel into a microkernel. I'd certainly love it if this magically happened - microkernels are much easier to work with, and much more elegant. But it'd be hard to make the case that that'd be a worthwhile endeavor at this point, given the costs of doing so.

<comment><USER36> I think the other factors in play are the dominant ones in the Unix/Plan9 issue, though. The original "upstream" of Unix was Research Unix, the stuff the Bell Labs researchers were working on. V1 came out in 1971. But V7, released in 1979, basically got forked into a bunch of commercial Unixes (and indirectly, some free ones via the BSDs). Research Unix V8, V9, and V10 got virtually no uptake, despite being incremental evolutions of the original upstream, due to a mixture of inertia and licensing restrictions from AT&T. Instead, different incremental evolutions of V7, developed by other developers outside Bell Labs, became the dominant strains of Unix.

Plan9 was a pretty significant departure from Research Unix V7, but not _as_ significant a change compared to V10... which didn't matter because nobody outside Bell Labs was using V8/9/10 anyway, since everyone was working on various forks of V7 (BSD, SunOS/Solaris, Xenix, etc.).

<comment><USER37> I agree Plan 9 is not a good example in this story. Plan 9 didn't failed because of the problems associated with a re-write as described in Joel's post, but because of political and social issues.

> Research Unix V8, V9, and V10 got virtually no uptake, despite being > incremental evolutions of the original upstream, due to a mixture of inertia > and licensing restrictions from AT&T.

I'm not sure what you mean by inertia. v8-v10 were not widely released to avoid another BSD incident.

v8-v10 were 4.3BSD derivatives, so I'm not sure I'd call them incremental evolutions on the original upstream. They removed and/or reworked all the new BSD stuff though, and the basic userland was very custom. Some of it survives (in some form) in Plan 9.

</comment></comment></comment><comment><USER38> I won't move off of Vim personally. I don't push it really as I use a pretty stock setup. However, good luck to the neovim devs, it will be cool to watch.

</comment><comment><USER39> Sorry to point out to the ironic here. But his signature includes the link to a "new programming language". And it reads much like "Neo-C".

</comment><comment><USER21> Refactoring is _hard_. I'm learning this with delving into the LibreOffice codebase. It's got a lot of issues - about 25 years of accreted code and design decisions that need unpicking, tangled code that needs refactoring, etc.

However, it's sure easier to refactor than start from scratch! At least LO has a working product. Best of luck to the Neovim guys :-)

</comment><comment><USER40> What would you call going from vi to vim? Regardless, anything that fixes the mess that is vimscript gets a thumbs up from me.

</comment><comment><USER41> Am I the only one thinking why C?

<comment><USER42> Nope. [http://talks.golang.org/2014/go1.3.slide#23](http://talks.golang.org/2014/go1.3.slide#23)

</comment></comment><comment><USER43> maybe if Bram even commented on the two patches that Thiago had posted, things could have been different.

All Thiago got was: being ignored, one of the worst ways of disdain. Not just even a 'thanks but not interested'.

Other two guys who tried to provide patches to be merged on the same area also suffered from being mostly ignored (he replied to them but just once I think).

</comment><comment><USER44> Streamlining and simplifying a code base by introducing JSON-RPC for internal API? Success is practically guaranteed!

</comment><comment><USER45> I haven't looked at vim code in ages but if it is really 300K lines of code, I wouldn't refactor it in any major way. If I absolutely wanted to scratch that particular itch, I'd rewrite it from scratch. In another language (not C or C++). And structure it differently.

</comment><comment><USER46> Was Vim a complete rewrite of Vi?

<comment><USER47> Lots of things were a complete rewrite of vi. Vi was a complete rewrite of ex. ex was a rewrite of em, which was in turn a rewrite of ed, which was a rewrite of qed.

Trying to rewrite vim is the wrong approach. Trying to replace it in a backward-compatible way, the way vim did to vi might received a lot better.

<comment><USER48> Good thing then that they aren't rewriting vim.

</comment></comment></comment><comment><USER49> Refactoring is not a big deal. In fact I don't even want total compatible with vim.

</comment><comment><USER50> Total refactoring without regression is rare. Especially in the wild.

</comment><comment><USER51> Clang v. GCC all over again?

</comment><comment><USER52> 1\. make kickstarter campaign to "revolutionize" open source project vim.

2\. :%s/vim/neovim/g

3\. profit!?!?!?!?!?

</comment><comment><USER53> Well that does it I'd say. With such a negative attitude, who wants him on the team anyway? What the advantages are? They're clearly laid out in the project.

Seriously, screw that guy.

edit: Yes I know who 'that guy' is and it's great he's been working on Vim for all these years, but he certainly has his head up his ass if he thinks Vim users would not benefit from some big refactorings, a more accessible code base and some dropped platforms. If all those Amiga programmers love Vim so much, why would they be disappointed with an eternal Vim7.4?

It's not too late for Vim to lose the editor wars.

<comment><USER54> I still have yet to see these concrete user benefits.

<comment><USER55> The benefit is indirect rather than direct, as in the user benefits from the increased availability of plugins/guis since they would be easier to create. Also vim lacks lots of very simple but essential features of modern code editors simply because it lacks the graphical capabilities to do so (an example would be displaying the documentation of some method you are autocompleting, something which I have never seen in vim and which I actually believe is impossible).

<comment><USER56> > an example would be displaying the documentation of some method you are > autocompleting, something which I have never seen in vim and which I > actually believe is impossible

For python at least, having 'preview' as part of your completeopt has it display the docstring of what you're completing. And :h completeopt suggests that's intended behaviour for all types of completion (where it makes sense), too.

<comment><USER57> Yes, but currently retrieving that info is done synchronously, so it might slow down things. Under neovim's model, this can be done more efficiently.</comment></comment></comment></comment></comment>