In search of the perfect URL validation regex - <AUTHOR> http://mathiasbynens.be/demo/url-regex <comment><USER2> If you're going to allow dotted IPs you should really allow 32-bit IPs too, e.g., [http://0xadc229b7](http://0xadc229b7), [http://2915183031](http://2915183031) and [http://025560424667](http://025560424667). (The validity of this last one was news to me I must admit.)

<comment><USER3> None of those is a URI, so a URI validator most certainly should not accept them. Just because browsers tend to understand them as a matter of a historical accident does not mean those are valid URIs, just as tag soup that browsers also tend to understand isn't valid HTML either.

<comment><USER4> Exactly. The goal was to come up with a good regular expression to validate URLs as user input. There's no way I'd want to allow alternate IP address notations.

</comment></comment></comment><comment><USER5> Why use a regex? It's much simpler to write a URL validator by hand, speaking as someone who wrote a URL parser,[1] and fixed a bug in PHP's.[2]

Or, you know, use a robust existing validator or parser. Like PHP's, for instance.

[1] [https://github.com/TazeTSchnitzel/Faucet-HTTP- Extension](https://github.com/TazeTSchnitzel/Faucet-HTTP-Extension) \- granted, this deliberately limits the space of URLs it can parse, but it's not difficult to cover all valid cases if you need to

[2] [https://github.com/php/php- src/commit/36b88d77f2a9d0ac74692a...](https://github.com/php/php- src/commit/36b88d77f2a9d0ac74692a679f636ccb5d11589f)

<comment><USER6> Exactly. Isn't this as bad an idea as trying to parse HTML with regular expressions? [1]

[1]: [http://stackoverflow.com/questions/1732348/regex-match- open-...](http://stackoverflow.com/questions/1732348/regex-match-open-tags- except-xhtml-self-contained-tags/1732454#1732454)

<comment><USER3> No, it's not, URIs are regular, so using _regular_ expressions is perfectly fine.

<comment><USER6> Good point. You're right, it's not _as_ bad an idea.

(I still think it's not a great idea. Being regular isn't necessarily the same as being parseable with a maintainable regex.)

</comment></comment></comment></comment><comment><USER7> Why are [http://www.foo.bar./](http://www.foo.bar./) and [http://a.b-- c.de/](http://a.b--c.de/) supposed to fail?

The @stephenhay is just about perfect despite being the shortest. The subtleties of hyphen placement aren't very important, and this is a dumb place to filter out private IP addresses when a domain could always resolve to one. Checking if an IP is valid should be a later step.

<comment><USER8> The trailing dot makes it an invalid URL, which defines hostname as *[ domainlabel "." ] toplabel.

"b--c" seems to be a valid domainlabel, though, so I'm not sure why it's on there.

<comment><USER9> > The trailing dot makes it an invalid URL, which defines hostname as * [ > domainlabel "." ] toplabel.

I disagree. But, this is a tricky one. The relevant specs are:

        Spec               |          | Validity | Definition
        URL      (RFC1738) | obsolete |  invalid | hostname = *[ domainlabel "." ] toplabel
        HTTP/1.0 (RFC1945) | current  |  invalid | host     = <A legal Internet host domain name
                           |          |          |             or IP address (in dotted-decimal form),
                           |          |          |             as defined by Section 2.1 of RFC 1123>
        HTTP/1.1 (RFC2068) | obsolete |  invalid | ; same as RFC1945
        HTTP/1.1 (RFC2616) | obsolete |    valid | hostname = *( domainlabel "." ) toplabel [ "." ]
        URI      (RFC3986) | current  |    valid | host     = IP-literal / IPv4address / reg-name
                           |          |          | reg-name = *( unreserved / pct-encoded / sub-delims )
        HTTP/1.1 (RFC7230) | current  |    valid | uri-host = <host, see [RFC3986], Section 3.2.2>

The only way that URL is invalid is if we are in a strict HTTP/1.0 context.

As a note about RFC1738 being obsolete: these days a URL is just a URI (1) whose scheme specifies it as a URL scheme, and (2) is valid according to the scheme specification.

As the given URL is a valid URI, and is valid according to the current http URL scheme specification (RFC7230), that URL is valid.

<comment><USER4> You forgot the most relevant spec, the URL Standard: [http://url.spec.whatwg.org/](http://url.spec.whatwg.org/)

</comment></comment></comment></comment><comment><USER10> At best this lets you conclude that a URL _could_ be valid. Is that really useful? Is the goal here to catch typos? Because you'd still miss an awful lot of typos.

If you really want your URL shortener to reject bad URLs, then you need to actually test fetching each URL (and even then...)

As an aside, I'd instantly fail any library that validates against a list of known TLDs. That was a bad idea when people were doing it a decade ago. It's completely impractical now.

<comment><USER4> My exact use case was the following: the user clicks a bookmarklet that passes the current URL in the browser as a query string parameter to a URL shortener script. The validation is then performed before the URL is shortened.

In that scenario, and with the given requirements, I can't think of a case where the validation fails. There's no need to worry about protocol-relative URLs, etc.

(Keep in mind that this page is 4 years old — I very well may have missed something.)

> If you really want your URL shortener to reject bad URLs, then you need to > actually test fetching each URL (and even then...)

I disagree. [http://example.com/](http://example.com/) might experience downtime at some point in time, but that doesn't mean it's suddenly an invalid URL.

> As an aside, I'd instantly fail any library that validates against a list of > known TLDs. That was a bad idea when people were doing it a decade ago. It's > completely impractical now.

Agreed.

<comment><USER10> I still don't quite follow the purpose of the validation. Is it against malicious use? In normal use, I would think that pretty much any URL that's good enough for the browser sending it would be good enough for the link shortener.

<comment><USER4> But then you might end up shortening things like `about:blank` by accident.

</comment></comment></comment></comment><comment><USER3> WTF? When will people finally learn to read the spec and implement things based on the spec and test things based on the spec instead of just making up themselves what a URL is or what HTML is or what an email address is or what a MIME body is or ...

There are supposed URIs in that list that aren't actually URIs, there are supposed non-URIs in that list that are actually URIs, and most of the candidate regexes obviously must have come from some creative minds and not from people who should be writing software. If you just make shit up instead of referring to what the spec says, you urgently should find yourself a new profession, this kind of crap has been hurting us long enough.

(Also, I do not just mean the numeric RFC1918 IPv4 URIs, which obviously are valid URIs but have been rejected intentionally nonetheless - even though that's idiotic as well, of course, given that (a) nothing prevents anyone from putting those addresses in the DNS and (b) those are actually perfectly fine URIs that people use, and I don't see why people should not want to shorten some class of the URIs that they use.)

By the way, the grammar in the RFC is machine readable, and it's regular. So you can just write a script that transforms that grammar into a regex that is guaranteed to reflect exactly what the spec says.

<comment><USER4> The goal was to come up with a good regular expression to validate URLs in user input, and not to match any URL that browsers can handle (as per the URL Standard). I am fully aware that this is not the same as what any spec says.

> By the way, the grammar in the RFC is machine readable, and it's regular.

The RFC does not reflect reality either (which, ironically, is what you seem to be complaining about). If you're looking for a spec-compliant solution, the spec to follow is [http://url.spec.whatwg.org/](http://url.spec.whatwg.org/).

> If you just make shit up instead of referring to what the spec says, you > urgently should find yourself a new profession, this kind of crap has been > hurting us long enough.

I am aware of, and am a contributor to, the URL Standard: [http://url.spec.whatwg.org/](http://url.spec.whatwg.org/) That doesn't mean there aren't any situations in which I need/want to blacklist some technically valid URL constructs.

<comment><USER3> > The goal was to come up with a good regular expression to validate URLs in > user input, and not to match any URL that browsers can handle (as per the > URL Standard).

WTF? What is "validation" supposed to be good for if it doesn't actually validate what it claims to? Exactly this mentality of making up your own stuff instead of implementing standards is what causes all these interoperability nightmares! If you claim to accept URLs, then accept URLs, all URLs, and reject non-URLs, all non-URLs. There is no reason to do anything else, other than lazyness maybe, and even then you are lying if you claim that you are validating URLs - you are not. If you say you accept a URL, and I paste a URL, your software is broken if it then rejects that URL as invalid.

This does not apply to intentionally selecting only a subset of URLs that are applicable in a given context, of course - if the URL is to be retrieved by an HTTP client, it's perfectly fine to reject non-HTTP URLs, of course, but any kind of "nobody is going to use that anyhow" is not a good reason. In particular, that kind of rejection most certainly is something that should not happen in the parser as that is likely to give inconsistent results as the parser usually works at the wrong level of abstraction.

> The RFC does not reflect reality either (which, ironically, is what you seem > to be complaining about).

Well, or reality does not match the RFC?

> If you're looking for a spec-compliant solution, the spec to follow is > [http://url.spec.whatwg.org/](http://url.spec.whatwg.org/).

A spec for a formal language that doesn't contain a grammar? The world is getting crazier every day ...

> That doesn't mean there aren't any situations in which I need/want to > blacklist some technically valid URL constructs.

Yeah, but blocking IPv4 literals of certain address ranges seems like a stupid idea nonetheless. Good software should accept any input that is meaningful to it and that is not a security problem. And as I said above, such rejection most certainly should not happen in the parser.

<comment><USER4> > > The RFC does not reflect reality either (which, ironically, is what you > seem to be complaining about).

> Well, or reality does not match the RFC?

Doesn't matter – if there's a discrepancy between what a document says and what implementors do, that document is but a work of fiction.

> And as I said above, such rejection most certainly should not happen in the > parser.

This is not a parser.

<comment><USER3> > Doesn't matter – if there's a discrepancy between what a document says and > what implementors do, that document is but a work of fiction.

Yes and no. When there is a de-facto standard that just doesn't happen to match the published standard, yeah, sure. Otherwise, bug compatibility is a terrible idea and should be avoided as much as possible, many security problems have resulted from that.

> This is not a parser.

Well, even worse then. Manually integrating semantics from higher layers into parsing machinery (which it is, never mind the fact that you don't capture any of the syntactic elements within that parsing automaton) is both extremely error prone and gives you terrible maintainability.

edit:

For the fun of it, I just had a look at the "winning entry" (diegoperini). Unsurprisingly, it's broken. It was trivial to find cases that it will reject that you most certainly don't intend to reject. For exactly the reasons pointed out above.

</comment></comment></comment></comment></comment><comment><USER11> Another important dimension when evaluating these regexes is performance. The Gruber v2 regex has exponential (?) behavior on certain pathological inputs (at least in the python re module).

There are some examples of these pathological inputs at [https://github.com/tornadoweb/tornado/blob/master/tornado/te...](https://github.com/tornadoweb/tornado/blob/master/tornado/test/escape_test.py#L20-29)

<comment><USER12> From experience, the python re module does weird things sometimes. There is a better third-party regex module, [https://pypi.python.org/pypi/regex](https://pypi.python.org/pypi/regex).

<comment><USER13> Does it use NFA?

[http://swtch.com/~rsc/regexp/regexp1.html](http://swtch.com/~rsc/regexp/regexp1.html)

Because the issue with the URL regex mentioned is with backtracking.

</comment></comment></comment><comment><USER14> Use a standard URI parser to break this problem into smaller parts. Let a modern URI library worry about arcane details like spaces, fragments, userinfo, IPv6 hosts, etc.

       uri = URI.parse(target).normalize
       uri.absolute? or raise 'URI not absolute'
       %w[ http https ftp ].include?(uri.scheme) or raise 'Unsupported URI scheme'
       # Etc

<comment><USER4> That was not an option in this case, as the goal is to validate URLs entered as user input and blacklist certain URL constructs even though they're technically valid.

<comment><USER14> So check if uri.host matches your blacklist.

</comment></comment></comment><comment><USER15> Why no IPv6 addresses in the test cases?

</comment><comment><USER16> Why not put in some of the new TLDs as test cases... ;)

</comment><comment><USER17> John Gruber (of daringfireball.com) came up with a regex for extracting URLs from text (Twitter-like) years ago, and has improved it since. The current version is found at [https://gist.github.com/gruber/249502](https://gist.github.com/gruber/249502).

I haven't tested it myself, but it's worth looking at.

Original post: [http://daringfireball.net/2009/11/liberal_regex_for_matching...](http://daringfireball.net/2009/11/liberal_regex_for_matching_urls)

Updated version: [http://daringfireball.net/2010/07/improved_regex_for_matchin...](http://daringfireball.net/2010/07/improved_regex_for_matching_urls)

Most recent announcement, which contained the Gist URL: [http://daringfireball.net/linked/2014/02/08/improved- improve...](http://daringfireball.net/linked/2014/02/08/improved-improved- regex)

<comment><USER18> Isn't it the @gruber v2 column from the page? It looks to have no false negative, but many false positives. The only one which does perfect on the tested set is Diego Perini's [https://gist.github.com/dperini/729294](https://gist.github.com/dperini/729294)

<comment><USER17> Hrm, you're right. I managed to miss seeing that while skimming the page.

</comment></comment></comment><comment><USER19> Interestingly it seems [http://✪df.ws](http://✪df.ws) isn't actually valid, even though it exists. ✪ isn't a letter[1], so it isn't allowed in international domain names. I was looking at the latest RFC from 2010 [2] so maybe it was allowed before that. The owner talks about all the compatibility trouble he had after he registered it [3]. The registrar that he used for it, Dynadot, won't let me register any name with that character, nor will Namecheap.

[1] [http://www.fileformat.info/info/unicode/char/272a/index.htm](http://www.fileformat.info/info/unicode/char/272a/index.htm)

[2] [http://tools.ietf.org/html/rfc5892](http://tools.ietf.org/html/rfc5892)

[3] [http://daringfireball.net/2010/09/starstruck](http://daringfireball.net/2010/09/starstruck)

<comment> X-Istence It's the same way with [http://💩.la/](http://💩.la/) it gets turned into the following using IDNA: xn--ls8h.la.

That is a valid domain name and should be treated as such.

<comment><USER19> I guess you could argue the definition of "valid". According to the RFC it's

DISALLOWED: Those that should clearly not be included in IDNs. Code points with this property value are not permitted in IDNs.

</comment></comment></comment><comment><USER20> There is no perfect URL validation regex, because there are so many things you can do with URLs, and so many contexts to use them with. So, it might be perfect for the OP, but completely inappropriate for you.

That said, there is a regex in RFC3986, but that's for parsing a URI, not validating it.

I converted 3986's ABNF to regex here: [https://gist.github.com/mnot/138549](https://gist.github.com/mnot/138549)

However, some of the test cases in the original post (the list of URLs there aren't available separately any more :( ) are IRIs, not URIs, so they fail; they need to be converted to URIs first.

In the sense of the WHATWG's specs, what he's looking for _are_ URLs, so this could be useful: [http://url.spec.whatwg.org](http://url.spec.whatwg.org)

However, I don't know of a regex that implements that, and there isn't any ABNF to convert from there.

</comment><comment><USER21> This is a good lesson why you want to avoid writing your own regexes. Even something simple like an email address can be insane:[http://ex- parrot.com/~pdw/Mail-RFC822-Address.html](http://ex-parrot.com/~pdw/Mail- RFC822-Address.html)

<comment><USER22> This always comes up in these discussions and is a terrible counter example. RFC 822 is the format for email messages (ie headers) and not a form you'll ever find email addresses in "in the wild" (eg on web forms).

</comment></comment><comment><USER23> What's wrong with IP-address URLs? If they are invalid because it says so in some RFC, this is still not the ultimate regex. If you redirect a browser to [http://192.168.1.1](http://192.168.1.1) it works perfectly fine.

And why must the root period behind the domain be omitted from URLs? Not only does it work in a browser (and people end sentences with periods), the domain should actually end in a period all the time but it's usually omitted for ease of use. Only some DNS applications still require domains to end with root dots.

<comment><USER24> This is the context of a URL shortener, and one of the goals seems to prohibit bad IPs, not _all_ IPs. Bad seems to be defined here as ones in the private IP spaces, and those in multicast space, etc.

<comment><USER23> Okay, that makes sense!

</comment></comment></comment><comment><USER25> I've put the test cases into a refiddle: [http://refiddle.com/refiddles/53a736c175622d2770a70400](http://refiddle.com/refiddles/53a736c175622d2770a70400)

</comment><comment><USER26> I just validate with this regex '^http' :P

</comment><comment><USER27> It has to match this valid URL: [http://موقع.وزارة- الاتصالات.مصر](http://موقع.وزارة-الاتصالات.مصر)

</comment><comment><USER28> What's wrong with /([\w-]+:\/\/[^\s]+)/gi

It's not fancy but it will essentially match any url

<comment><USER3> /./ will also match any URL. The point is to _reject_ non-URLs.

<comment><USER4> Yes, to reject non-URLs and also some URLs that are technically valid but that I want to explicitly disallow anyway.

</comment></comment></comment><comment><USER29> When you have a hammer, everything looks like a nail.

</comment><comment><USER30> What does the red vs green boxes mean?

<comment><USER30> Oh I get it now. Got confused between 1s and 0s.

</comment></comment><comment><USER31> What flavor of regex are we do making this in?

</comment><comment><USER32> You do realize that RFC 3986 actually contains an official regular expression, right? [http://tools.ietf.org/html/rfc3986#appendix-B](http://tools.ietf.org/html/rfc3986#appendix-B)

<comment><USER8> That doesn't work for validation.

<comment><USER32> Of course it does, if that regular expression matches we have a valid URI. It's silly what gets downvoted on HN these days.

<comment><USER3> No, it doesn't. This regular expression matches strings that are not URIs, and that should be quite obvious if you compare the grammar in that same RFC to the regex.

<comment><USER32> Yes, it does. And it is quite obvious if you compare the grammar to the regular expression.

<comment><USER3> Please show how to produce the following string (which is matched by that regex) from the grammar (without the quotes):

"%x"

(edit: actually, feel free to do it with the quotes included if you like, that would still be matched by that regex)</comment></comment></comment></comment></comment></comment>


An industrious mouse tidies up a retired electrician's tool shed each night - <AUTHOR> https://petapixel.com/2019/04/20/trail-camera-catches-mouse-secretly-cleaning-mans-shed-every-night/ <comment><USER2> I saw this on reddit a few weeks ago and it actually had useful comments: [https://www.reddit.com/r/AnimalsBeingBros/comments/dwj7i3/a_...](https://www.reddit.com/r/AnimalsBeingBros/comments/dwj7i3/a_72_year_old_retired_man_from_britain_found_out/f7jnzcy/)

tl;dr; the mouse is nesting

<comment><USER3> Relevant top comment:

> _So this looks really cute and all, but what she 's doing is nesting, and > once she fills that box with hard stuff, she adds tufts of her own fur and > possibly other soft stuff like fiberglass insulation or the stuffing from > car seats, then she has her babies in there, all the while peeing all over > those parts and causing them to rust and smell like mouse piss for ever > more._

</comment></comment><comment><USER4> I couldn't find it with a few Google searches, but I recall an interview with a Bansky-like character who 'created' this video and story as a sort of viral art project. Her interview also included a few other examples of strange phenomena she was behind. Since I'm unable to find evidence of said interview, however, I'm doubting myself...

<comment><USER5> Sounds like Zardulu

[https://www.vox.com/culture/2017/4/24/14912316/zardulu- viral...](https://www.vox.com/culture/2017/4/24/14912316/zardulu-viral-videos- mythmaking-surrealism-pedro-lasch)

<comment><USER6> I was going to flag this as not really being a HN story so much as something I'd expect to see go by on Facebook, but it did produce an interesting article link there.

I could see this going either way; a mouse could probably be trained to do this for a viral video. On the other hand, you can see animals get mental diseases known to occur in humans sometimes as well; an obsessive-compulsive mouse is also plausible to me.

(A disturbing number of "Look at my dog/cat/whatever doing this funny thing!" videos appears to my eyes to be the dog/cat/whatever having a seizure, for instance.)

<comment><USER7> Maybe the mouse escaped from a behavioral lab.

<comment><USER8> Yep, this one is Mr. Frisby. The rest were rats.

</comment></comment></comment></comment></comment><comment><USER9> Reminds me of a book I had as a kid about Anatole the Mouse .. he would sneak into a cheese factory at night and taste the samples and leave tasting notes, which benefited the cheese factory .. [Anatole (mouse) - Wikipedia]([https://en.wikipedia.org/wiki/Anatole_%28mouse%29](https://en.wikipedia.org/wiki/Anatole_%28mouse%29))

<comment><USER10> Are there any disney films which aren't plagiarised

<comment><USER11> This one clearly is not...

Eve Titus (the author) is listed as a producer on the Disney movie "The Great Mouse Detective", which is derived from her characters.

[https://bakerstreet.fandom.com/wiki/The_Great_Mouse_Detectiv...](https://bakerstreet.fandom.com/wiki/The_Great_Mouse_Detective)

</comment></comment></comment><comment><USER12> This mouse would make some bucks if it had a Twitch channel with ads

</comment><comment><USER13> In Finland, there's a lot of old folk tradition around "house elves" ("tonttu"), that sometimes perform useful tasks, sometimes not, but are best not angered in any case.

On the one hand, it's easy to see that mice rattling around could lead to such stories, but on the other hand, it's difficult to imagine people intimately familiar with mice being very mystified by them. However, if they do things like cleaning up in the night, I can imagine that might give rise to some myths!

</comment><comment><USER14> Reminds me of a strange story of mysterious 'modern art' arrangements, of leaves and twigs, that appeared every morning on a workshop stoop (spoiler, it was a fieldmouse): "Elena's Messages: From Her Big Sur Sanctuary" [https://www.amazon.com/Elenas-Messages-Her-Big- Sanctuary/dp/...](https://www.amazon.com/Elenas-Messages-Her-Big- Sanctuary/dp/069232951X)

</comment><comment><USER15> I would think that if this mouse came by every day, he would have seen some tell-tale "droppings."

<comment><USER16> Right, this is staged b.s.

Anyone with the slightest experience dealing with rodents knows there'd be droppings everywhere.

</comment></comment><comment><USER17> I bet a lot of people's haunted houses can be explained by animals (raccoons, rats, crows etc)

</comment><comment><USER18> That's a determined mouse. Favorite quip in that article's comments:

"Probably not a teenage mouse." \- ha!

</comment><comment><USER19> The mouse appears to be doing "cargo cult" type of behavior, placing junk in the bin and hoping peanuts will appear. Kind of like managers who hope that requesting documentation will help a better product appear on time. Fascinating.

<comment><USER20> Your comment is delightful.

</comment></comment><comment><USER21> cute!

</comment><comment><USER22> Url changed from [https://laughingsquid.com/mouse-cleans-up-tool- shed/](https://laughingsquid.com/mouse-cleans-up-tool-shed/), which points to this.</comment>


Most Popular Coding Languages of 2015 - <AUTHOR> http://blog.codeeval.com/codeevalblog/2015#.VOkpIHUX_0o <comment><USER2> Good to see Tcl on the list</comment>


Urbit user guide, hosted on Urbit - <AUTHOR> http://urbit.org/docs/user <comment><USER2> From[1]: "Anyone can run the Urbit VM, of course. But the %ames network is officially invitation-only. Not that we're antisocial -- just that we're under construction."

So, is there support for running a separate p2p network? Or is the only option to wait for urbit to stabilize?

[ed: Never mind, it's covered under "Launch Instructions": If you don't have an invitation, pick a nickname for your comet, like mycomet. Urbit will randomly generate a 128-bit plot]

[ed2: Hm, looks like the ability to spawn an alternate universe would make playing with urbit more interesting, as I understand it - without an invite, most of the best parts are inaccessible: "The fanciest way to control your urbit is through Urbit itself: a moon, or satellite urbit. Sadly, only planets can have moons."]

[1] [http://urbit.org/docs/user/intro](http://urbit.org/docs/user/intro)

<comment><USER3> It's entirely possible to spawn an alternate universe. In fact, it's recommended to spin up an offline version of ~zod to develop on so you don't end up crashing you actual planet. The `-F` flag makes it so your ships only connect over localhost instead of looking up the top-level galaxies from *.urbit.org

It's also possible to hard-fork the network. The galaxies' public key hash is hard-coded in the source, so if you change it your ship will accept that the new key is the actual galaxy. Of course, there is no guarentee that any OTHER ship you talk to will agree with you...

<comment><USER2> Oh, thanks for the update. But this doesn't appear to be documented anywhere? So far I've only been reading the documentation, not the source code -- perhaps it's more obvious from the source code that such functionality already exists (eg: the -F flag)?

<comment><USER4> Our documentation is still very young, but it is mentioned in our contributing guide:

[http://urbit.org/docs/dev/contributing](http://urbit.org/docs/dev/contributing)

There's much more to cover in terms of how the network works. It's coming! For the time being some things are just folk knowledge. We're always happy to answer questions though.

</comment></comment></comment></comment><comment><USER5> I am not convinced that urbit is not just some sort of ironic performance-art criticism of computer science. Perhaps it would be elucidating if someone would be so kind as to explain what the point is.

<comment><USER6> Urbit is a reinvention of the Old Internet with a lot of the principles from ancient computer lore thrown in. (eg. Lisp Machines)

It reintroduces the social computing environment, emphasis on the word _computing_. The Old Internet and timesharing systems like ITS and Unix were based around this model:

[https://medium.com/message/tilde-club-i-had-a-couple- drinks-...](https://medium.com/message/tilde-club-i-had-a-couple-drinks-and- woke-up-with-1-000-nerds-a8904f0a2ebf)

So you have a standard identity layer, instead of a million little fiefdoms of identity competing for sign ups. You have proper one-to-one connections between individual users on the system, with modern cryptography inbetween.

You have a functional, very very small kernel out of which the rest of the system is built. This kernel and core concept is a sort of distillation of the ACID concept from database systems, so that your computer has transactions and forgives mistakes.

It bakes the social layer and the community layer of the Internet into the protocol. Urbit tries to be community-aware and politics-aware and handle this gracefully. This sort of formal acknowledgement of necessary human factors in technology is certainly in line with Curtis Yarvin's previously expressed views on social organization. (And is mostly what people are talking about when they say he 'baked neoreaction' into Urbit. This accusation strikes me as intellectually dishonest on multiple levels, but I'd rather not digress on it.)

If nothing else it's a very interesting piece of new research in computer science, you should be excited about it.

EDIT: Curtis I know you're lurking, can I get an invite?

<comment><USER5> Why do you think it is a good idea to make things "social" at the OS level? That seems like a very strange abstraction policy.

And how is this system "politics-aware"? I'm not even sure what that would entail from what basically seems to amount to a collection of esolangs.

<comment><USER6> >Why do you think it is a good idea to make things "social" at the OS level? That seems like a very strange abstraction policy.

Social means a bit more than just Facebook. It also means primitives for things like collaborative editors and coworking spaces, video conferencing, etc. IMO the ideal social system would look a lot like a mix between the old timesharing systems, the early Internet and Douglas Engelbarts "Mother of All Demos": [https://archive.org/details/XD300-23_68HighlightsAResearchCn...](https://archive.org/details/XD300-23_68HighlightsAResearchCntAugHumanIntellect)

>And how is this system "politics-aware"? I'm not even sure what that would entail from what basically seems to amount to a collection of esolangs.

Well it comes with the community. The way Urbit is structured your personal cloud/ship/etc is intended to be linked up to a community of other users. Politics are part of being social and part of communities. If you don't have a structured way of handling it it'll happen regardless and can be made less ugly with official support.

"The part that is stable we are going to predict, and the part that is unstable we are going to control." \- John Von Neumann, 1948

</comment></comment></comment></comment><comment><USER7> I was late last time, so this is my chance to ask again. Is the Urbit team familiar with Ted Nelson's work? For some context, see my question from the previous Urbit thread at [https://news.ycombinator.com/item?id=10286521](https://news.ycombinator.com/item?id=10286521).

<comment><USER8> How could anyone not be familiar with Ted Nelson's work? In the broad sense, anyway.

I think the crucial layer that we need to implement... a lot of things... is a global immutable (aka referentially transparent) namespace. Urbit is one project building such a thing; another one is IPFS. (Urbit names are addressed by identity; IPFS names are content-addressed; so they're complementary and not competitive.)

One of the reasons the Web seems like such a poor imitation of Xanadu is that it rests on this rickety foundation of a mutable binding from name to resource. Once global immutable namespaces -- Urbit, IPFS, anything -- are more widely deployed, I think Xanadu would be wise to use such a thing as a layer.

But to paraphrase a famous saying: grant me the serenity to accept the code I cannot rewrite, the courage to rewrite the code I can, and the wisdom to know the difference :-)

<comment><USER2> > How could anyone not be familiar with Ted Nelson's work? In the broad sense, > anyway.

Computer Science is not renowned for it's diligent study of history. And that's just for those that actually study it in some form of institution, not all the people who "practice" it without any formal training.

That said, the fact that Ted Nelson/the publisher have stubbornly refused to just publish for example "Computer Lib/Dream Machines" free on the (inferior) web, or at least as a DRMed ebook or merely a dead-tree re-print -- makes it unnecessarily hard for people to read up on the concept(s).

<comment><USER9> "Computer Lib/Dream Machines" was in print once upon a time. I bought a copy circa 1992, the copy I have is the revised edition which was published in 1987, these are photos of my copy (it's a wee bit aged looking now):

[http://imgur.com/a/ghtEb](http://imgur.com/a/ghtEb)

You can still find copies on Abe Books but they're not cheap.

[http://www.abebooks.com/servlet/SearchResults?isbn=091484549...](http://www.abebooks.com/servlet/SearchResults?isbn=0914845497&sts=t)

Glad I held on to this little bit of dead tree history even if it is the revised version.

</comment></comment></comment></comment><comment><USER4> We also put up a new homepage that's hosted on Urbit: [http://urbit.org/](http://urbit.org/).

</comment><comment><USER10> Suggest adding a reminder that when saying "urbit" as "herb it", you need to not say the "h". Unless you _are_ actually supposed to say the h, of course. Perhaps it's the opposite of a silent letter.

<comment><USER8> Sorry, this is a ridiculous parochialism on our part. In America, which is where English was invented and is also of course the biggest bestest country in the world with giant atomic bombs and stuff, we don't say the "h."

But apparently there's some little islands or somewhere where they do. Will fix.

<comment><USER11> Honestly, i was a little sad to see you mention latin out of the corner of my eye, which would mean you'd properly pronounce it oorbit, to then find out you're being as english about it as the internet usually is.

</comment></comment></comment><comment><USER11> It would be nice if i could read the docs without needing JS.

It is nice to see sanely written documentation though. Cheers on improving that. :)

Edit: Useful, thanks! vvvv

<comment><USER12> Or for that matter [http://urbit.org/home/pub/docs/user](http://urbit.org/home/pub/docs/user), which is a simple-html rendering of the OP link. Some of the pages require JS to list children components, however.

<comment><USER13> Thanks, because something is not working with the main site. You go to the user introduction, click on the install manual or dojo or something, and nothing happens. PaleMoon, latest.

</comment></comment></comment><comment><USER14> Isn't this one of those Mencius Moldbug things.</comment>


Introducing JIRA Software: the #1 software development tool used by agile teams - <AUTHOR> http://blogs.atlassian.com/2015/10/introducing-jira-software/ <comment><USER2> Good move by Atlassian to expand beyond just software development. Another key area they are lacking are mobile products.

JIRA Board for iOS addresses some of those needs: [https://itunes.apple.com/us/app/jira-board-kanban-scrum- agil...](https://itunes.apple.com/us/app/jira-board-kanban-scrum- agile/id934196108?mt=8)

</comment><comment><AUTHOR> Looks like JIRA & JIRA Agile have merged into JIRA Software.

<comment><USER3> Yes, in a way: JIRA Software includes JIRA Agile's functionality and some other goodies. It runs on top of JIRA Core, which is now available as a standalone offering for non-software teams in an organization that don't want the extra complexity that Software features introduce.

source: I'm a developer at Atlassian

<comment><USER4> Besides customer segmenting, how does JIRA Core compare to the old JIRA vanilla offering? Is it missing any features?</comment></comment></comment>


Ask HN: Who's buying all the big data? - <AUTHOR> With every service, platform, software package, god knows what collecting data on you, I can't help but ask: who's buying all this data?
It's clear in the case of massive players such as Facebook and Google that don't have to sell their data. They can just keep their data and directly offer targeted marketing to advertising.
But what about smaller players, that don't have a strong 'big data' presence and product? Examples such as:

  - online only NVIDIA drivers
  - anti-virus software
  - chat apps such as Skype, Discord, WhatsApp, etc
  - online sites such as Github, StackExchange, etc
  - <shady mobile app #1402>
 All of the above collect data about you in some form of another, it's a staple nowadays. Who's buying all this data? For how much? <comment><USER2> Mostly hedge funds and market research companies. Less frequently, companies looking to bootstrap computer vision or natural language processing systems without sourcing their own data.

If you productize the data as a forecast, it can cost between mid four figures to low six figures per quarter per client, depending on the data and its signal. Raw data is not particularly lucrative or useful, but developing a predictive forecast with a very low margin of error based on an exclusive source of data is both.

Smaller companies who find themselves with marketable data tend to partner with well-networked financial research companies such as 7Park. Sometimes they sell directly to clients if they have the network for it. In my experience, sophisticated quantitative hedge funds mostly internalize their data sourcing initiatives instead of purchasing from external vendors. For example, Renaissance Technologies and Two Sigma both have massive data sourcing and analysis operations internally, off the top of my head. They very rarely engage with outside data vendors (particularly the former).

All of this falls under what is usually termed, "alternative data." There are basically two classes of data vendors. First, you have Foursquare, Meraki (prior to its acquisition), Airmail, etc. Basically, any company which provides a free service for things like geolocation, inbox decluttering or financial account/statement aggregation is most likely reselling your data. For example, Yodlee resells customer receipt data. These companies source their data from a public facing product with a sufficiently strong moat to make the data exclusive.

The second class of vendor curates data from a wide variety of sources and has no public product from which to source their own data. For example, SecondMeasure provides sophisticated analysis of the kind of receipt data from Yodlee. YipitData is very active in this work (though they're not actually very good at it). Most of the very good companies doing this work are pretty under the radar - they tend not to be VC-funded or well-publicized. This class is more easily replicated as an internal initiative in a hedge fund, because all of the data is technically public, just very hard to find and analyze.

<comment><USER3> Why is YipitData not very good?

<comment><USER2> There are several reasons, in my opinion:

1\. They are very "loud" \- VC funding, visibility and large headcount are antagonistic features for a firm specializing in boutique data mining. It's not impossible, it just makes it harder to maintain a competitive edge in e.g. how long your data remains exclusive. On the other hand, these qualities are excellent for developing a moat around a product from which you can source data. Their website gives away quite a lot of information.

2\. I have personally heard from analysts who mentioned that various forecasts of theirs were rushed, superfluous or simply incorrect.

3\. I have personally "beaten" their team in finding, curating and analyzing data to produce a forecast which was in high demand by various clients.

It's not uncommon for funds to buy redundant forecasts from several vendors, so research firms tend to be at least peripherally aware of one another. Keep in mind that the more exclusive a source of data is, the more lucrative a forecast will be if it has a high signal. The two extremes are data which essentially only you are in possession of, and data which has no signal due to widespread diffusion in the market. Eventually your forecast slides from the first sort to the second sort, and letting other companies know what kind of data you have is inviting them to compete.

<comment><USER3> Care to share type some examples of data you've found that was used to produce financial forecasts?

</comment></comment></comment></comment><comment><USER4> I did a grocery/food app. Grocery stores buy massive amounts of product data. They need to know what to put on the shelves, what to put on what shelf, what not to put on the shelf, which products are trending. Since they make very low margins, this is one of the main differentiators in income. A single store also has cash flow in the millions, so they're willing to spend a lot.

Another major use is that it builds it as a wall around the product users. It helps to recognize user patterns, build a better product, and keep them from going to similar products, e.g. Netflix.

I think a lot of people look at "big data" as some kind of gold mine, but really it's just a rock mine. You have to know which rocks are in demand by whom before you start mining all of it.

<comment><USER5> > I think a lot of people look at "big data" as some kind of gold mine, but > really it's just a rock mine. You have to know which rocks are in demand by > whom before you start mining all of it.

I think this hits the nail on the head. Knowing how to extract value is important. Not just the raw data.

</comment></comment><comment><USER6> One person's metadata is another's data! You (may be unintentionally) characterise it as a market where you check boxes on "datasets" to buy, but it's far from that. Monetizing by licensing/selling some kind of log data isn't really interesting--but understanding what signal can be derived and how it can be applied to an opportunity (arbitrage, value added analysis, financial product/insurance etc.... is where it gets interesting.

I've written a bit about derivative uses of data if you are interested: [https://post-employment.com/category/business/](https://post- employment.com/category/business/)</comment>


Watch out for millions of IPv6 DNS AAAA requests - <AUTHOR> http://blog.boxedice.com/2010/09/28/watch-out-for-millions-of-ipv6-dns-aaaa-requests/

<comment><USER2> We have the exact same problem with Dyn, and it's pretty much just a way for them to make more money off you. Leaves a sour taste in your mouth, and I doubt most people even notice the extra req/s they're being billed for.

If there was an alternative, we'd switch. DNSMadeEasy _does not_ charge for those AAAA requests, but it's not quite at the same level UI/functionality wise.</comment>


A new antibiotic kills pathogens without detectable resistance - <AUTHOR> http://www.nature.com/nature/journal/vaop/ncurrent/full/nature14098.html

<comment><USER2> So let's use this one WITH EXTREME CAUTION, i.e. only in very limited, justifiable circumstances with careful medical supervision to make sure that the patient finishes the course of treatment instead of just taking it until they feel better. Let's not give it out for, oh say, viral infections 'just in case' as was the case with doxycycline or cipro in the past. It would be nice to have one emergency antibiotic that actually works when something serious hits. Learning from our recent history in provoking antibiotic resistance should be priority 1.</comment>


(1) Design  Quora (Web2.0 Expo Presentat... by Rebekah Cox - Quora - <AUTHOR> http://www.quora.com/Rebekah-Cox/Design-Quora-Web2-0-Expo-Presentation <comment><USER2> So, I'm going to go against the grain here...

I think there is a lot of great stuff about Quora. Design isn't one of them, here is why:

The site and its team have shown a specific arrogance over voices of dissension about their design. Visually there are aspects of the design that make it difficult for people to use - the choice of colors for the text, the size of the fonts, and the contrast make it tough to quickly sort the information provided.

Their topic management system is difficult to use. The search box hijacks what you're typing and provides no easy way to get the cursor back.

The topics and threading of comments has issues as well. Topics lump a large number of content into a singular bucket, and though you can apply multiple topics to a question, you cannot filter responses in a given topic.

Commenting is limited to singular responses, non threaded conversations and are far too easily buried.

Architecturally the site suffers from sever performance problems when there are a large number of answers to a give question. See this post on companies hiring in the SF Bay, which on many machines will crash the browser.

[http://www.quora.com/Which-startups-are-hiring-in-the-San- Fr...](http://www.quora.com/Which-startups-are-hiring-in-the-San-Francisco- Bay-Area)?

Typical browser behavior gets hijacked/broken as well - HOME and END keys no longer work, and at times, scrolling seems to break.

The Quora Search Bar, though, does provide good snappy returns of questions that have been asked from all topics that contain the words you enter. So I like that feature.

Overall though, I don't know what gasses fill the echo-chamber these folks design in, but I surely think they are high if thy think the site is stellar.

Further, why do the facebook alum feel like facebook blue is the best color to use? I hate it - and simply adopting design elements from your previous employer doesn't show me that you're a great designer.

Also, in this presentation she says "Concentrate on what matters most: purpose and goals" yet they seem to shoot down any criticism of their design decisions as quickly as possible, as opposed to hearing what others think. So, if the purpose and goal is to make a site that is instantly useful, responsive and navigable - isn't it a good idea to see what users are saying about it and take criticism?

I really like Quora, but it is in its infancy - I refuse to jump on the bandwagon that believes this thing came out of the gate mature and tested over the years.

You want to see a site that has an unbelievably good user experience? Spend some time on Reddit. Damn good design -- oh and guess what, they implement pretty much every great UI/UX idea that users submit -- for example, submitter highlighting in the comments thread is just one.</comment>


Hacker News Shuts Down Account Creation to Keep Bored Reddit Users Out - <AUTHOR> http://siliconfilter.com/as-reddits-bored-users-are-set-adrift-hacker-news-shuts-down-account-creation-to-keep-them-away/

<comment><USER2> This is retarded and flardinois is retarded.</comment>


Announcing the 2nd Hacker News Cologne Meetup - <AUTHOR> http://www.meetup.com/Hacker-News-Cologne/events/36946902/

<comment><USER2> Awesome, missed the last one but I can hopefully make it this time.

<comment><AUTHOR> Cool! You were missed the last time! :)</comment></comment>


EAutist project - <AUTHOR> http://eautist.com/

<comment><AUTHOR> We want to know what people think about project</comment>


Entrepreneurs blow their exit opportunity 75% of the time - <AUTHOR> http://www.exits.com/blog/only-25-percent-of-saleable-companies-exit/?Hackernews

<comment><AUTHOR> I think this is depressing - anyone disagree that companies blow their opportunity to sell about 75% of the time?</comment>


Amazon Wants Alexa to Move into Your Apartment Before You Do - <AUTHOR> https://www.vice.com/en_us/article/qj45kx/amazon-wants-alexa-to-move-into-your-apartment-before-you-do <comment><USER2> Developing skills and integration for IOT devices against Alexa is quite convenient and a lot of fun from a technical perspective. I think Amazon did a good job on it and is actively promoting/enforcing secure information exchange between devices and the Alexa service. I would believe them that they delete user data and shield it from third party ambitions as best as they can.

That said, I would not want to have a smart speaker in my home. There are too many ambitions to get a hand on this data. We know that mass surveillance from state actors is pervasive, advertisers are experts in removing their own decency and the overall risk is just too high. Get rid of them or maybe offer Alexa offline options and it could be a nice tool.

It could help with logistical problems of delivery and make it even more convenient aside from interfacing a lot of devices. In theory all fine and dandy.

</comment><comment><USER3> If Alexa is part of your lease, how long before disabling it going to be grounds for eviction? If your landlord owns Alexa can they buy the data from it?

</comment><comment><USER4> Renting isn't difficult enough, now we also have this, which would have to be removed for most of us to be able to move in.</comment>


Ask HN: What is your favorite desktop Linux distribution these days? - <AUTHOR> most stable/usable <comment><USER2> Fedora is my choice.

</comment><comment><USER3> Arch.

<comment><USER4> +1

</comment></comment><comment><USER5> Debian</comment>


Revealed: How Monsanto's 'Intelligence Center' Targeted Journalists & Activists - <AUTHOR> https://www.theguardian.com/business/2019/aug/07/monsanto-fusion-center-journalists-roundup-neil-young <comment><USER2> FTA :

>>> Monsanto paid Google to promote search results for "Monsanto Glyphosate Carey Gillam" that criticized her work

I understand Google may not have caught the maneuver. But this is quite problematic.

How much does Google shape our vision of the world ? (even if we assume Google is benevolent)</comment>