Why is the Internet so Slow? - <AUTHOR> https://blog.apnic.net/2017/06/19/why-is-the-internet-so-slow/ <comment><USER2> _If the 3x slowdown from the infrastructure were eliminated, each round-trip- time being 3x faster would affect all the protocols above, and we could immediately cut the latency inflation from ~37x to around 10x, without any protocol modifications._

The page this article is published on makes 87 separate network requests to 19 different domains. Even with a great deal of the content cached, it takes 7 seconds to reload over my WiFi today. This is because I'm using an ad blocker; without one, the page makes more requests to more domains and transfers more stuff, slower.

It seems likely that cutting bloat would have a much bigger impact on responsiveness than infrastructure upgrades. Cutting bloat would produce benefits even (especially) in times and places where signals or infrastructure are marginal. Cutting bloat could mean your old phone or tablet computer could browse the web tolerably instead of ending up in a landfill. Cutting bloat from a website today produces a benefit, for all visitors, worldwide, _today_. Improving infrastructure may be a good investment, but cutting bloat is a force multiplier.

The responsiveness of the web has been adjusted to optimize the number of eyeballs that see ads. If a site is too sluggish, people leave, but a site that's too fast is leaving money on the table. How many advertising, tracking, and affiliate domains would a web page like this connect to, if latency were cut by two thirds?

<comment><USER3> As much as ads are annoying, this has nothing to do with the article. One man's bloat is another man's content. No need for this nag to be top comment on HN for an article about why data isn't moving at the speed of light.

<comment><USER4> It's not ads! Jquery.com, bootstrapcdn.com, cloudfare.com, cloudfront.net, google-analytics.com, fontawesome.com, Wordpress, Twitter, Facebook, Pocket, rating-widget.com, and typography.com are all used to put the content on this website, and none of them are ads.

<comment><USER5> but this comment was

<comment><USER4> Buddy I hate all that shit. I use uMatrix and block as much as I can while still seeing the real content. In fact that's how I know what the domains are.

<comment><USER5> I was joking. I think it wasn't a terribly good joke though, so sorry.

<comment><USER3> Jokes don't work on HN for some reason (I liked/upvoted it)

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> Since the OP compared Internet latencies to the speed of light in glass...look up 802.1CM/IEEE1904.3 or "CPRI over Ethernet"/"Radio over Ethernet"! Those describe techniques for conveying LTE _baseband_ signals (digital I/Q-samples) over Ethernet -- so the radios can live far away from the equipment that speaks the physical / MAC / radio-resource-control layers of LTE. There are strict latency/jitter requirements involved (the phone and the eNB run a very fast automatic-repeat-request protocol between each other, too much latency breaks the loop) -- and large (but predictable) data rates involved.

Read [https://drive.google.com/file/d/0B6Xurc4m_PVsZ1lzWWoxS0pTNVE...](https://drive.google.com/file/d/0B6Xurc4m_PVsZ1lzWWoxS0pTNVE/view) for a general introduction about latency in Ethernet/IP networks and [http://www.ieee802.org/1/files/public/docs2016/cm-chen- front...](http://www.ieee802.org/1/files/public/docs2016/cm-chen-fronthaul- bandwidth-analysis-0716-v01.pdf) for an introduction to the "radio over ethernet business"

Even if you're not trying to send radio baseband data over ethernet networks -- there's lots of noticeable queuing going on, _everywhere_. It's not just the unmanaged and pathological "bufferbloat" queues (that occur when a device has a network link much faster than its other network link) -- there's little queues that are used for tasks like:

* frame aggregation

* holding onto data until a shared medium is idle

* for scheduled systems like DOCSIS or LTE, holding onto packets until an uplink grant has been given -- or holding onto packets until a station's downlink grant/timeslot becomes active

* inter-device communication (usually with ring buffers or the like)

* any processing/forwarding that isn't of the cut-through flavour usually involves queuing at least a single frame

* context switching / memory copying / interrupts

Retransmission mechanisms can introduce latency as well -- if there's no special channel for acknowledgements, they'll use up airtime, and if data needs to be retransmitted, it might induce delays for never-transmitted data that's waiting for its first transmit opportunity.

In general, unless you're actively being paid to stomp out every imaginable source of latency, there'll be little queues living everywhere in your system -- from TCP's queues right on down to the CPU's store buffers-- because queuing is the natural way to cope with subsystems that produce/consume at different rates.

<comment><USER7> If you are interested in bufferbloat and TCP, Google's BBR is a really interesting attempt at solving part of the problem. See [http://queue.acm.org/detail.cfm?id=3022184](http://queue.acm.org/detail.cfm?id=3022184)

BBR is a sender-only modification to TCP that basically solves the problem of large buffers.

<comment><USER8> There was a great article + discussion about BBR a few weeks ago: [https://news.ycombinator.com/item?id=14298576](https://news.ycombinator.com/item?id=14298576)

</comment></comment></comment><comment><USER9> If you're thinking, "it's fast enough!" \-- remember that a system's performance determines what you're able to build on top of it. If GPU designers had said "it's fast enough" ten years ago, for example, modern neural-net AI may never have happened. It's easy to imagine doing the same things faster, but it's harder to picture the new things that a lower-latency Internet would make possible.

<comment><USER10> I don't think anyone that has ever used Skype, VoIP, Facetime, etc has thought "this is fast enough".

<comment><USER11> Actually, I think ~10mbit is fast enough. You can easily use Hangouts/Skype with good quality video on that kind of link. The problems are elsewhere: it's not stable enough, and it's not responsive enough.

Anything faster is just a cherry on top. Sure, files download faster, you can stream 4k, etc. But I'd say that's luxury which you could choose. Lower speeds are "enough" for a lot of people.

<comment><USER12> That's like saying 640K ought to be enough for everyone. Once we have stable 1GBit connections on all landlines and mobile devices, we might eg stop hoarding data on our disks altogether. Apps (if they still exist) could be downloaded on the fly when the user chooses to start them. Lifelike VR/AR communications could be enabled by this. Teleworking could become a no- brainer, from any location imaginable (perhaps with telepresence). I will be able to run a dev machine on some EC2 instance and VNC into it with an unnoticeable lag, so I won't need serious CPU horsepower in my laptop anymore. That's what I could think of off the top of my head..

<comment><USER11> I don't think that comparable to 640k. Things you mention are are either what I define cherry on top (VR/AR meeting? Why? Teleworking is already present and completely doable on low bandwidth, mid latency - I'm doing it every day). VNC lag issues are mostly exactly what I mentioned. You already have enough bandwidth - it's the latency that affects you.

Besides, high detail remote desktop are a small, specialised segment, relatively speaking. If you're a developer, most of the time you work with text. If you're an office worker, most of the time you work with fairly static apps like office. Sure, you can't really work with remote blender at the moment, but we're getting into improvements for the long tail now - which is great. But once we're past 10mbps, improving latency will help more people than improving bandwidth.

</comment></comment></comment></comment></comment><comment><USER13> ETHZ is developing a new kind of internet architecture: [https://scion- architecture.net/](https://scion-architecture.net/)

It's deployed in some places and working already. Citing from the webpage:

> SCION is the first clean-slate Internet architecture designed to provide > route control, failure isolation, and explicit trust information for end-to- > end communications.

Also it will reduce latency, increase throughput and more, as mentioned in the papers.

The project has really been getting traction in recent years, it may just become our new internet base layer.

Currently the people at ETHZ are working on verifying router hardware mathematically to guarantee SCION's properties.

<comment><USER14> Very interesting approach. I've read about SCION before. I believe, however, that its goal is primarily security. The reduced latency claim simply comes from the fact that they try to restrict routing in such a way that, for example, traffic with source and destination both in the US will never leave the US because of some weird routing policies. I'm not sure about this. Somehow, it seems to be against the principles of a truly free internet.

<comment><USER13> Yeah, speed isn't the primary focus, more of a byproduct

I don't understand what this has to do with a free internet, routing doesn't have any affect on that

<comment><USER15> I think the poster means "free" as in data travelling freely regardless of State/national borders, as the internet largely does today.

</comment></comment></comment></comment><comment><USER16> Because every time we get faster connections, we download more the client and force more client side processing. Angular and friends are just causing a worse user experience, not a better one.

We also don't take advantage of DNS enough. We should/could use it to pass along information about the client's ISP to locate a server one or two hops away from the client. DNS could easily solve the IPV4 with SRV records, but instead, we allow service location to happen at the TCP level.

The internet _could_ be fast, but it's not :/

<comment><USER17> How is angular making the web slower. An angular project can sit somewhere in the 130kb range, with lighter frameworks like vue sitting at 30kb. These are smaller than most images or gifs used on the web. So the trade-off is one image on your page, or a robust RIA experience, with much more maintainable code. If you build it correctly, you're decreasing the amount of bytes on the wire significantly when using client side frameworks, over re-rendering each and every page on the server for every interaction. I also maintain a large vanilla javascript application that was written before I arrived, and the javascript weighs in at almost 5MB, and we have a general no touch policy on it because of how fragile the vanilla javascript is. It was poorly written, but even a poorly written vue app, would come in at much less than the 5mb JS. I just don't buy this as the reason the web is slow.

<comment><USER18> > So the trade-off is one image on your page, or a robust RIA experience

If I had the choice, I'd take an additional image in that trade, thanks. The 'rich' 'experiences' I've been offered so far are inferior to bog-standard web pages. I'll grant you that it is friendlier to coders whose applications fit the model. Beyond that, they don't work without Javascript[1]. The "richness" is usually useless animation and similar, frequently employed because it is there, rather than actually adding any value (Anyone remember the Jquery animation explosion?). It breaks a lot of automation; see the JS comment. These are all things that are important to me.

I get that folks are fine with losing me as a user/customer, and that's their choice. But they should know why, thus my explanation.

[1] Which means I usually go elsewhere when I encounter it, because I default to leaving it off.

<comment><USER19> If I understand you right, your contention seems to be more with animation and 'fluff' rather than with MV* frameworks like Angular specifically. To which I'll simply say: in some cases animation is as important to the user experience as an additional image would be, and in some cases the additional image just contributes more 'fluff' than the animation or interactivity would. There are many dimensions to the problem (and the article only hints at some of them on the protocol side), and pinning JavaScript frameworks as the cause strikes me as unfair.

What you're really asking for is for the components being used, be they images, animations of frameworks, to be used with a deliberate purpose. Which I think is what everyone is asking for, and what good developers and designers strive for.

<comment><USER18> You're actually picking up the smaller of my two complaints. If everything else about Angular were sensible, I'd probably grumble about dumb overuse of trendy visual primitives, but that's about it. And I absolutely agree that, used sensibly, animations and whatnot are very useful. This complaint basically boils down to wishing all web designers were actually as good as a lot of them think they are.

You missed my complaints about the uselessness of Angular apps without a JS interpreter. That's important, because it means they're basically worthless for consumption by nonhumans. (Sure, I can run Chrome headless, but that's an entirely different can of worms that play poorly with pipelines, not to mention an enormous, absurd runtime for what should be a trivial file transfer.)

Web automation on a personal scale is enormously useful. Angular breaks the underlying assumptions that make it work for sites that don't go out of their way to accommodate it. I get that may be an unintended benefit to some people who want to be control-freaky about how their offerings are used , but it enormously reduces the value of the web.

A connection just hit me - it is similar-ish to way back when, when some misguided designers wanted to publish PDFs on the web instead of HTML. They wanted full layout control, at the cost of basically everything else. Angular is similar, in that doing anything the creator didn't anticipate is very difficult, despite complying with the letter of web standards.

And also, as I said, I default to browsing with JS off. When I hit a blank page, I curse "Angular" and go back to the search engine. So that's annoying, but I've yet to encounter an Angular site I can't live without.

</comment></comment></comment></comment></comment><comment><USER20> If you want an easy speed up and you use a modern version of nginx, which you should for many reasons but chiefly performance and security.

Add these lines to your nginx config file:

        # Only support HTTPS and enable http2
        # which will allow you to multiplex requests.
        # In a separate config do a 301 from normal HTTP
        # for all paths / subdomains.
        listen 443 ssl http2 default_server;
        listen [::]:443 ssl http2 default_server;

        # Make TCP send multiple buffers as individual packets.
        tcp_nodelay on;

        # Send half empty (or half full) packets.
        tcp_nopush  on;

It reduced our page load time by around 75%, and more for people accessing the site from the other side of the world.

<comment><USER21> I'm not very experienced with these nginx settings. Could you explain how this improves speed so much for you?

It seems that you've followed [1] by the similarity of what that suggests to what you've suggested. tcp_nodelay seems to force it to not wait 0.2 seconds, which nginx does so that it doesn't send lots of really small packets. I can see this increasing speed by 200ms (a lot) but increases network traffic as a cost.

tcp_nopush seems to be about optimising each packet sent, reducing the total size of them.

Interesting stuff though!

[1] [https://t37.net/nginx-optimization-understanding-sendfile- tc...](https://t37.net/nginx-optimization-understanding-sendfile-tcp_nodelay- and-tcp_nopush.html)

<comment><USER20> So I'm more of a machine learning / cyber security guy than a sys admin, but with that proviso, here is my understanding:

Of course you'll have to check your own site for your own performance issues, but while travelling from Toronto to Vilnius I found our site _much_ slower and investigated why. We have a webapp running Ember with mix of largish assets, like images, and smallish assets, talking to a JSON API standard API written in rails and proxied through nginx (responses are usually around 1kb of JSON, but this gets compressed with HTTP2).

In reality waiting 200ms is an enormous cost and carries a hidden cost: Larger packets are easier to lose. If you're going mostly to browsers that is more and more often going through wifi or cell tower networks with at least 1% packet loss and when the resend happens you're stuck with that 200ms delay again.

That's my understanding, but the real truth is "it just works for me when testing around the world with real-world use cases, try testing it out yourself and please let me know if you learn something new".

<comment><USER22> Nagle's algorithm is not a network level delay. If data is lost then it is retransmitted as soon as the sender learns about the loss. Nagle only delays application level sends.

The problem with webapps is a compounding effect of many requests.

a) dependency chains (asset A loads B loads C) mean the browser only knows to load C once B has been loaded, so the latencies (including nagle) add up. http/2 push is supposed to help with that.

b) statistics. your 95th percentile latency is irrelevant if you're loading hundreds of assets and want to know when the site has finished loading.

</comment></comment></comment></comment><comment><USER23> Did I read this correctly? Is it saying that a general internet connection, without specialist low latency connectivity, is only 37x slower than its theoretical minimum?

That's astonishingly good. It's great that they think they can do better but bloody hell.

It's indistinguishable from magic.

<comment><USER24> You're saying that achieving 2.7% of the theoretical efficiency is good?

Many car engines have an efficiency of around 20% for example, as compared to the maximum theoretical efficiency of 67%. If car engines performed only well enough to meet your standard of amazingness, your gas costs would be 7 or 8 times higher. Or if you used a modern Toyota engine for comparison, something like 14 times higher.

Obviously I took an example from a different sphere for comparison... if you really find this to be an astonishing result can you elaborate?

<comment><USER22> Your car is not operating at 20% efficiency of theoretical limits if you take the whole energy pipeline into account. You know... fusion, sunlight, plankton, oil, refining, combustion, mechanical energy. On the other hand speed-of-light to actual ping times is an end-to-end efficiency calculation. Plankton in particular is terrible at converting energy.

sun -> mj-PV -> battery -> electric motors run circles around those.

<comment><USER25> > Plankton in particular is terrible at converting energy.

Says the person who can't even photosynthesize!

</comment></comment></comment></comment><comment><USER26> Let's add to that all the fuss that occurs to deliver us ads according to our recent navigation ... Improving physical delays will not speed up browsing!

<comment><USER27> I get amazed every time I'm configuring a new install of an OS, because the Internet is shit until I install uBlock. Then suddenly it's back to normal.

</comment></comment><comment><USER28> No mention of bufferbloat, it's in mainline linux since a few releases now. If all devices used these patches or applied the same logic, could it also work ?

</comment><comment><USER29> Javascript frameworks. Try timing the loading of a text-only or text/html/css site, then your favourite site built with React or Angular.

</comment><comment><USER30> The idea that browsing the modern web transfers "tens of kilobytes of data" is pessimistic. I think you can argue bandwidth constraints in the real-world are real. Improving the underlying stack is one thing and would obviously bring improvements, but browsing speed won't be one of them. If you make responses faster, they'll be able to stuff more garbage in those responses.

</comment><comment><USER31> Tangentially, I have been struggling, for several years, with Safari being extremely slow at opening sites, compared to almost instant loading in Chrome. Anyone seen this?

I'm always on fast networks (100mbps or so), but I find myself waiting 4-5 seconds or more before Safari even shows signs of connecting, even to things like Google and YouTube. It seems to be limited to HTTPS sites, and seems to have been introduced when Apple started using a new MacOS TLS/SSL framework called something like "Apple secure transport" (I forget exactly) a few MacOS versions back, and it's almost certainly not DNS (I get the same issue with Google DNS as with my provider's).

I don't even know how to debug this, since I don't have a command line client that goes through the same TLS/SSL framework code as WebKit. Curl is fast, for example.

<comment><USER32> I've used Safari since the very beginning, and I haven't really noticed this, no. Extensions, perhaps?

</comment></comment><comment><USER33> TL;DR: TCP and DNS, which is what you would expect. If you want light-speed internet, send UDP packets with a laser.

<comment><USER34> Or any form of electromagnetic wave passing media with low dielectric constant. Like microwave communication in athmosphere.

Speed of propagation in typical cable is around 66% of c. That is not too bad. More latency is probably caused by buffering.

Speed of light is just too low. We should get a lobby-group working on it and the limiting laws of physics.

<comment><USER22> You really shouldn't be complaining about the speed of light. We're leaving a lot on the table by sending the data along the surface of earth instead of through it.

With neutrino-based signalling you eliminate both the lower propagation speed and the suboptimal path.

<comment><USER35> Has neutrino-based signalling been achieved or is it still a research topic? Last time I read about it it was more a case of "if we could do this it would be great, but we have no idea how to do it" and a quick Google search only brought a Forbes article from 2012 which had a similar sentiment.

<comment><USER36> > Has neutrino-based signalling been achieved or is it still a research topic?

An accelerator- or reactor-based neutrino experiment can detect whether the source is on or off, so in principle you can send binary information. This has been demonstrated using NuMI/MINERvA [1]. It's more of a novelty than anything practical.

[1] [https://arxiv.org/abs/1203.2847](https://arxiv.org/abs/1203.2847)

</comment></comment></comment></comment></comment><comment><USER37> "Why are we so far from the speed of light?"

Because we don't know how to build an optical router with the same capabilities as the ones we use today and the alternative is to give up on net neutrality. I'm ignoring endpoint performance and protocol bloat.

There is work being done in quantum computing that might eventually lead to some CPU like device operating on photons alone and hollow core fiber optics also seem to be an active field. The work on upgrading the various protocols together with limiting network requests seem to be the way to go until our technology catches up.

</comment><comment><USER38> "Slow" internet is not so bad (especially when it is not really THAT slow). Limitations in connectivity force designers and developers (therefore large companies) to optimise things and save resources. Internet speed growth rate could be 5% (compounding) would be enough in order to provide a friendly environment for new frameworks, technologies in the field of network and web to benefit all.

<comment><USER39> Speedtest.net claims an increase of average internet speeds by 40% in 2016: [http://www.speedtest.net/reports/united- states/](http://www.speedtest.net/reports/united-states/)

<comment><USER40> I'm curious how much of that is actual infrastructure improvements and how much is ISPs increasingly prioritizing traffic to speedtest.net.

</comment></comment></comment><comment><USER41> because applications are written with the "if it is not un acceptable it is acceptably fast" mindset

</comment><comment><USER42> You should move to Romania. [https://motherboard.vice.com/en_us/article/jp5aa3/why- romani...](https://motherboard.vice.com/en_us/article/jp5aa3/why-romanias- internet-is-so-much-faster-than-americas)

<comment><USER43> A lot of latency inflation over past 10 yeara came from increase in L3 switchings over L2. Nowadays, you can have over 10 IP switching in DCs, and within the last mile.

You also get more hops thanks to some backbone networks abandoning MPLS in favour of native routing/packet labeling.

In Romania, as I understand, networks are largely free of L3 switching overhead thanks to large portion of networks running pure L2

<comment><USER44> Most switches are routing switches and the same pipeline is used for both L2 switching and L3 routing. There is no sensible difference in switching and routing unless the switch has the ability to completely disable L3 routing. The typical latency is 500ns in store-and-forward mode (cut-through fallbacks to store-and-forward in most situations). If L2 processing could be done in 300ns, this is 200ns. With 10 routers, this is 0.002 milliseconds. Not something noticeable. Even if you compare with cut-through switching which could lower the latency to 50ns, that's still only a 0.005 milliseconds penality.

<comment><USER43> My measurements give me from 10ms to 50ms for IP routing on mainstream hardware (for the whole route).

L2 switching is faster than IP routing because even cheap "smart" L2 hardware come with hardware acceleration, but a typical non-core router for a mid-tier ISP is usually a consumer grade, PC hardware based router.

<comment><USER45> Do you mean microseconds here? ms is milliseconds, and that just does not make any sense. E.g. I'm 7 (visible) hops away from the nearest Google DNS, and roundtrip pings to 220.127.116.11 are between 7ms and 8ms.

<comment><USER43> Yes, milliseconds. The difference there is because routers above you most likely are a carrier grade equipment like Cisco gear with ASIC based switching. This is what you can expect from big ISP in a big city, and less likely in networks serving more remote suburbs.

<comment><USER44> An EX2200, a routing switch you can pay for about 500$, is also ASIC-based. Linux on a server has no problem routing with a latency in the order of 100ns. I don't see what kind of equipments would route in the order of 10ms. That's just too huge.

<comment><USER43> I meant 10ms for the whole route.

>100ms

Under load with anymuch sizeable routing table?

<comment><USER44> Route lookup with 1 core with a full view on Linux is on the order of 100ns. [https://vincent.bernat.im/en/blog/2017-ipv4-route-lookup- lin...](https://vincent.bernat.im/en/blog/2017-ipv4-route-lookup- linux#performance_1)

Linux is able to handle about 5 Gbps of traffic of small packets (64 bytes) with one core. This gives a latency of 100ns to keep the pace. You need to find a Linux router 10000 time slower than this to get a 1ms latency.

<comment><USER43> Then, I must be getting something wrong

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER46> I appreciate the big picture they've presented. I think an important next step is to measure more precisely. Specifically to look at how QUIC and HTTP/2 impact real world performance. After reading the paper it's not clear they've looked at that at all.

</comment><comment><USER47> Answer: Advertising

<comment><USER48> The current common implementation of advertising. You can serve ads that don't noticeably slow anything down.

</comment></comment><comment><USER49> High-Performance Browser Networking is a phenomenal resource. Highest possible recommendation. [https://hpbn.co](https://hpbn.co)

</comment><comment><USER50> it it really interesting but simply "ignore the technical challenge for now" makes the Whole discussussion pointless, as the problem here ARE the technical challenges that a zero latency (above the theoretical limit) network implies... Anyway it is good to know the numbers: in my opinion a 37x factor, in this case is not so bad

</comment><comment><USER51> CDF = cumulative distribution function

People who care about how fast the internet is are not the same set as people who know statistics acronyms, and they don't show up in a simple web search. You know how else we can make the internet better and faster? Publishing articles with software and licenses that make it easy for people to improve and share your article.

<comment><USER52> If you are unfamiliar with basic statistical concepts it is hard to believe you will be able to provide any compelling contributions to making the internet faster.

<comment><USER53> No, that really doesn't follow.

Some classes of issues will be better solved if the solver understands stats. M/M/1, distributions, percentiles, etc, yes. If only I understood them...

By my local major London paper somehow needs 4MB of crap and >60s to display a 2 para story. I don't need anything as subtle as "stats" to tell them that they're holding it wrong and why I avoid their site like the plague.

</comment></comment></comment><comment><USER54> ISPs should come out of the stone age, and should start using fiber optics everywhere.

</comment><comment><USER55> Why are you in such a hurry?

</comment><comment><USER56> If the Internet moved as fast as the fastest thing in existence it would be faster. That's brilliant.

<comment><USER57> Internet Explorer might face hard times! :D</comment></comment>


Black – Uncompromising Python code formatter - <AUTHOR> https://github.com/python/black <comment><USER2> Against my better judgment I'll bite.

I super dislike black's formatting, and I think it's really rare to actually see it in codebases. It wraps weirdly (sometimes not at all). I'd prefer to use yapf, but last I checked it still crashes on "f-strings".

Here's a small example:

        basket.add({
            apple.stem
            for satchel in satchels
            for apple in satchel
        })

Black formats this as:

        basket.add(
            {
                apple.stem
                for satchel in satchels
                for apple in satchel
            }
        )

I've never seen Python code like that.

I totally believe using a formatter is good practice. Black is in a challenging position of coming into a community with a lot of existing code and customs, and I get that. But I also think that's an opportunity, rather than having to guess at what is good, there's a wealth of prior art to look at. I wish it had done this, rather than essentially codify the author's style.

<comment><USER3> I completely agree that black does not format all examples in a nice manner, and your example is something I see pretty often. Whenever black adds too many indentation levels and line breaks in what should be a simple-enough statement, I simply refactor into multiple statements, e.g.

        stems = {
            apple.stem for satchel in satchels for apple in satchel
        }
        basket.add(stems)

I concede that without black it wouldn't have made sense to make that refactoring, but since I _really_ want to use an autoformatter, and black is the best autoformatter I have, I'd rather restructure my code slightly so that I can just use black 100% all the time.

<comment><USER4> It's an interesting meta thought that using the autoformatter changes the way you write your code. Not that that's implicitly a good or a bad thing, I just find it interesting.

<comment><USER5> I think it's under-appreciated when comparing tools like this. I've noticed that a bit since adopting Prettier and Black and one thing I've noticed during code review is that usually when I end up changing code structure to get better results from the tool, it's a refactor that I probably would felt like I should have done but might have put off due to laziness even though it's really not that much work.

I'm kind of thinking of it as something like baking some exercise into your day by putting a walk into your commute: the little nudge of not fighting the tool on something minor means you don't skip little things, and over time that adds up more than it seems at first.

</comment></comment></comment></comment><comment><USER6> I often dislike autoformatter output too, but then I remember that while no- one likes what the autoformatter does to _their_ code, everyone likes what the autoformatter does to their coworkers' code, and then I chill out about it.

Having a standard is more important than the standard being excellent.

<comment><USER7> > Having a standard is more important than the standard being excellent.

Neither is very important, though. It's just formatting and your code will run the same regardless.

<comment><USER8> You could say that about anything, though. Why use punctuation and capitalization when typing forum comments? Its just syntax after all.

<comment><USER9> Why use many word when few word do trick

<comment><USER10> Because you don't know if they've done the trick. That's why natural language is padded with extra words to add context and help rule out misunderstandings. The grammatical rules of agreement are one mechanism to do this.

And you can see the phenomenon of over-abbreviation by reading a few debates on twitter and repeatedly seeing, "how can you not understand what I tweeted?!"

<comment><USER11> FYI you're replying to a joke from The Office (US) [https://www.youtube.com/watch?v=_K-L9uhsBLM](https://www.youtube.com/watch?v=_K-L9uhsBLM)

</comment></comment></comment></comment></comment></comment><comment><USER12> The way you can tell that black is good is that everyone mildly dislikes a few things about it, but they're usually different things.

That's usually a good clue that you've hit real middle-ground.

I blackify my projects once we hit 3 contributors.

<comment><USER3> Why wait? I run black on all my new single-author projects.

<comment><USER12> i have my personal tics about formatting that i enjoy. As long as it's just me, i'll persist them.

<comment><USER3> I can see the point of that. For me though, I just changed my personal tics to match the tics of black :-)

</comment></comment></comment></comment><comment><USER13> DEP 0008 ('Formatting Code with Black') was accepted by the Django board last week. They're going to wait until Black has a stable release before reformatting all the code, as there's at least one thing they don't like about how Black handles things.

[https://github.com/django/deps/blob/master/accepted/0008-bla...](https://github.com/django/deps/blob/master/accepted/0008-black.rst)

[https://groups.google.com/forum/#!topic/django- developers/7G...](https://groups.google.com/forum/#!topic/django- developers/7GZE3fIVVvo)

<comment><USER14> Seems like there are two things they want to see fixed:

> Several developers report that, in their experience, Black made code > formatting worse and decreased readability. Concrete examples shown in the > discussion were short lists, which Black reformats when they fit on a single > line, and vertically aligned comments, which Black is unable to preserve. > This is being addressed [0] in Black and is expected to be resolved before > Black becomes stable.

[0] [https://github.com/python/black/pull/826](https://github.com/python/black/pull/826)

</comment></comment><comment><USER15> This is a case where I'm completely decided. Everybody should use an autoformatter. The minimal benefit you get from custom formatting is completely outweighed by the uniformity, the consistency and readability of autoformatted code.

<comment><USER16> I think a nontrivial portion of Go's success can be attributed to having a standard format and treating deviation as compiler errors.

It is immensely valuable to be able to look at any Go code, whether in the toolchain, the standard library, or a random stackoverflow snippet, and not have to think about formatting at all.

<comment><USER17> I think you're conflating two concepts a little bit - deviation from `gofmt` isn't a compiler error, but lots of things (like unused variables or imports) are. I think they're both great.

<comment><USER16> You are corrrect. I guess I've learned to treat deviation from gofmt as error because of the go vim plugin. It runs goimports on save, which fixes a bunch of real errors (like unused imports).

</comment></comment></comment></comment><comment><USER18> I've been using it for a few months now, but I don't really like it. It does remove quabbles about formatting, but I've personally never felt that as a problem and it just replaces them with constant frustration.

I think code formatting is very important for readability, and I think many (subtle) choices about how to make a piece of code more readable are very subjective. These types of tools are just incapable to make those choices because they have no concept of intention.

And some rules make this painfully obvious even in less subjective cases. Eg: black formats dictionaries into a single line if they fit one line, which makes nested structures unreadable if they combine bigger and smaller sub- dictionaries.

In the end, I don't have problems with other peoples' formatting choices; as long as they are sufficiently consistent within their own style, which they usually are, I can live just fine with them even when they go against how I'd do things.

I do have problems with tools that reduce code readability for consistency's sake and argue that these provide little benefit, and may even be detrimental to code quality.

<comment><USER19> You could configure your IDE to re-format files into your favorite format and just have a pre-commit hook that formats them back into the official format. That's how I work. No arguments about style and formatting, yet I get to make the code look how I personally prefer it.

<comment><USER20> No, we can't. My (and, by the sound of it, CrLf's) favorite format relies on information that your pre-commit hook has _artificially removed_ from the code. Eg:

      munge(gidget,
        thing1,thing2,thing3);
      cmplt(dtypeA,valueA,
            dtypeB,valueB);

(Most cases are more subtle (and thus less amenable to "oh, you just need to build a complete static type checker into the formatter") than this, but I wanted an obvious example.)

<comment><USER21> It seems like the signature of the functions should be different, especially the second example. I would have written the cmplt function to take tuple pairs:

        cmplt(
            (type_a, value_a),
            (type_b, value_b),
        )

That's much more clear about the relationship between each pair of values either way, and would get formatted nicely by Black.

<comment><USER22> No it wouldn't; black will put that all on one line if it can. The semantic information in vertical space is lost.

A better example is a matrix laid out as a grid, which auto-formatters always destroy

        mat3x3(
             0,  a, -b,
            -a,  0,  c,
             b, -c,  0,
        )

<comment><USER20> To be fair, I suspect RussianCow would write that as something like:

      mat3x3(
        rowvec3( 0, a, b),
        rowvec3(-a, 0, c),
        rowvec3( b,-c, 0),
        )

Which is probably better, especially with tuples instead of struct{Scalar[3]}. Doesn't fix

      mat3x3(( 0, a, b),(-a, 0, c),( b,-c, 0))

, though.

</comment></comment></comment></comment></comment></comment><comment><USER23> I use black because it is idempotent. Google's yapf lacks this property.

Edit: Previous discussion: [https://news.ycombinator.com/item?id=17155048](https://news.ycombinator.com/item?id=17155048)

<comment><USER24> Unrelated: thank you for using "idempotent" correctly. Too many people use it to mean referential transparency or having a functionality property

<comment><USER25> wooow, so can we think of Black as outputing a projection of your source code in some space ?

<comment><USER26> wut? idempotent is any operator T such that T^2=e the identity. for example ring elements can be idempotent [https://en.m.wikipedia.org/wiki/Idempotent_(ring_theory)](https://en.m.wikipedia.org/wiki/Idempotent_\(ring_theory\)). it's not just characteristic of linear operators.

<comment><USER27> Idempotence means T^2=T. Are you thinking of involution? [https://en.wikipedia.org/wiki/Involution_(mathematics)](https://en.wikipedia.org/wiki/Involution_\(mathematics\))

<comment><USER26> you're right i'm wrong. but my saving grace is that they're related [https://en.wikipedia.org/wiki/Idempotent_(ring_theory)#Relat...](https://en.wikipedia.org/wiki/Idempotent_\(ring_theory\)#Relation_with_involutions)

</comment></comment></comment></comment></comment></comment><comment><USER28> I've been using this for a couple of weeks and its a been really nice to remove the constant mental overhead of structuring code. it also makes you think about structuring functional blocks into more manageable, bite-sized chunks which is easier to understand. 10/10 would recommend

<comment><USER29> Yes, the side effect of using black seems to be a more reasonable factoring within function bodies.

</comment></comment><comment><USER30> Played around with it in the sandbox and immediately disliked its insistence on putting every item in a sufficiently-long array on its own line. For example, something like:

        foo = [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 ]

Black formats it to something more like:

        foo = [
            1,
            2,
            3,
            4,
            5,
            6,
            7,
            8,
            9,
            10,
            11,
            12,
            13,
            14,
            15,
            16,
            17,
            18,
            19,
            20,
            21,
            22,
            23,
            24,
            25,
            26,
            27,
            28,
            29,
            30
        ]

That's, in my not-so-humble opinion, absolutely atrocious. Compare with how Emacs formats if it I press M-q (with python-mode and EditorConfig):

        foo = [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
                19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 ]

More readable (again, IMO) and way more compact.

The default line length of 88 also seems weird, but at least that's readily configurable to my preferred 80.

<comment><USER31> Because it's optimised for producing the smallest diff once it's already been formatted and a change is made.

If you add a single item to your preferred style, the entire thing has to be reflowed.

<comment><USER30> Only if I add an item in the middle. If I'm always appending, then only the last line will show a diff, and said diff will be for the added entry.

Even in a case where I need to add an item in the middle, the diff will - again - only start where I actually added an item, and any human can readily see that the rest of the downstream diffs were from the reflow.

Of course, it'd be even better if diff tools were more aware of the format/language of the code being diff'd, at which point it'd be able to say "new element added to foo" instead of reporting on a bunch of reflowed lines.

</comment></comment></comment><comment><USER32> Black is amazing. I've added it to all my work projects and will be adding it to all my personal projects over time. My own opinion is that black is the future of Python formatting and it's a long time in coming. I disagree with many of the ways it formats things, but I mostly am just ecstatic to no longer have to discuss these things and just write code. If you are an isort user: isort's primarily goal for the next release is to add a mode that gaurentee's 100% compatibility with black (it can be achieved currently via config options). And of course, I'll be formatting the isort project itself using both isort and Black.

</comment><comment><USER33> I have just started using it and love it. It sounds funny but while formatting is not always a deal breaker it still kills me for 1) readability and 2) the faith that when someone reads the code base later one they know I was not an idiot.

I know 2 is kind of subjective but when I look at someone elses code and I see obvious mistakes against pep8 I start judging both the person and the code harshly. Perhaps unfair but for the very simple pep8 rules I see it as someone who is not as well versed in Python and perhaps does not care about their code as much.

Overall I just concede that consistency is better than everyones personal tastes. For a startup I think you can get by with just some custom linting but in a larger org I find it much harder to agree on a consistent format unless you are a Python first org. Much easier just to roll Black into the project and go with it.

<comment><USER34> That's pretty lazy thinking, imho. "Person x doesn't care about their code because they didn't use the formatting I like"

Worrying about or judging code based on PEP8 is missing the forest for the trees[0]

[0] [https://www.youtube.com/watch?v=wf- BqAjZb8M](https://www.youtube.com/watch?v=wf-BqAjZb8M)

<comment> frosted-flakes It's not that, it's that they use _inconsistent_ and non-standard formatting.

For example, in JavaScript, there is always a space before a function's opening curly bracket. This is pretty much a global standard, and it's rare to see code that doesn't follow this rule.

However, I'll occasionally see code on Stack Overflow that randomly excludes these spaces. It honestly makes it so much harder to read, and I'm less willing to put in the effort to help them, because they didn't even go to the effort of properly formatting their code. Inconsistent indentation especially kills me because it makes code impossible to read.

<comment><USER35> Technically, one could spend his life learning and perfecting JS or Python skills, without ever learning how other people format code. If you don't have a reference, you cannot align.

Also, imagine blind people coding. I imagine they have a completely different stance on how code should be formatted.

</comment></comment></comment></comment><comment><USER36> We use Black in a pre-commit hook for all our python code. It makes legibility and formatting a non-issue in the team. Yes it can look strange at first if you're not used to it, but after a while it makes formatting invisible (and it's odd reading non-black python). Would recommend.

<comment><USER26> how do you do this? do you have a jenkins server running that does the formatting? i would love to set this up because while i use black religiously i cannot for the life of me convince my boss get in the habit of using it (which is crazy because your commits are always fighting on whitespace).

<comment><USER37> Use pre-commit [0] so people can run it client side for quick feedback, and then use whatever CI you have (Jenkins, Travis, Circle, Azure) to enforce this. The important parts are the majority of people agree to do it, and that you have CI to be the "bad cop"/fall guy so it never becomes personal.

(edit: black has pre-commit support [1], and it's easier than setting up and maintaining your own hooks IMO)

[0] [https://pre-commit.com/](https://pre-commit.com/)

[1] [https://github.com/python/black/blob/master/README.md#versio...](https://github.com/python/black/blob/master/README.md#version- control-integration)

</comment></comment></comment><comment><USER38> I use black every day. It's pretty good. The hours you save on pointless arguments e.g. ' or " is better as default for strings. I wish it'd come with standard python distribution `python fmt`

<comment><USER39> yeah, someone above mentioned that with Go as the example. It's pretty nifty in some cases but think there are two sides to the story. I know folks who write lovely python code everytime. I know others at my gig who even after 3 years are not putting spaces after equals sign (and other PEP/flake8 blunders) and every PR is littered with syntactic errors. The code formatter is brilliant for this case.

<comment><USER40> I don't think making code written by hacks look like code written by a competent programmer is a good thing.

<comment><USER41> Making the code more clear to read helps its quality come through. Or its lack thereof.

<comment><USER40> Nothing stops a reviewer from reformatting the code before reading it (probably using an autoformatter).

But I'm not sure if making the code easier -- and thus faster -- to read is necessarily a good thing. If the brain parses and analyzes the code in parallel, making it harder to parse could give the analyzing process more time and thus make the review more thorough.

Of course, nothing stops a reviewer from being thorough either way. But making it the path of least resistance makes it more likely.

</comment></comment></comment></comment></comment><comment><USER42> Beside the benefit that black removes any discussion regarding formatting, my favourite benefit otherwise is the speed-up it yields by having it run everytime I save a file. I don't have to care about line lengths or single/double quotes or stuff like that anymore, since black will fix it for me anyway. And if black fails to trigger, it means there is something wrong with my code somewhere.

This is what I've stuffed in my VS code settings:

        "python.formatting.provider": "black",
        "editor.formatOnSave": true,

</comment><comment><USER43> > Black is the uncompromising Python code formatter. By using it, you agree to > cede control over minutiae of hand-formatting. In return, Black gives you > speed, determinism, and freedom from pycodestyle nagging about formatting. > You will save time and mental energy for more important matters.

I run into this philosophical debate at work. In my mind, I don't care much what the resulting code looks like as long as

\- it's somewhere in the reasonable spectrum of readablility

\- it's consistent and unambiguous

\- it never fails for a syntactically valid source file (if it's doing something for syntactically invalid source files, even better, formatting helps me find the error)

There are other opinions though that emphasize minimizing diffs or extensive customization to fit somebody's favorite style (I'm looking at you, rustfmt). Black seems to be right down my alley.

</comment><comment><USER44> We have been using this for almost 7-8 months and even though I don't personally like the style of it very much, it has helped keep everything consistent. You can set it up as a pre-commit hook to automatically format the code too

</comment><comment><USER45> We recently started using a lot of auto-formatters across our Python/iOS/Android code, as the team has grown (there are 17 people touching code at the company now).

I like auto-formatting, because it makes PRs less stressful to commit, and makes review comments more focused on stuff that actually matters. How exactly the code gets formatted is not something I care much about, just that it happens consistently, and I don't have to think about it.

We use pre-commit with Black and other formatters, such that it formats the code, warns you stuff changed, prevents the commit, and makes you recommit with the formatting patch included.

<comment><USER17> Because pre-commit can be tough to google sometimes, depending on your search history, here's the direct link: [https://pre-commit.com/](https://pre- commit.com/)

<comment><USER46> I assumed the poster meant the git concept of pre-commit hooks, which is just a shell script - [https://git-scm.com/book/en/v2/Customizing-Git-Git- Hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks)

<comment><USER45> Your parent is correct. Here's our pre-commit config too FWIW:

      more .pre-commit-config.yaml
        repos:
          - repo: https://github.com/pre-commit/pre-commit-hooks
            rev: v2.1.0
            hooks:
              - id: end-of-file-fixer
              - id: check-json
              - id: check-yaml
      - repo: https://github.com/asottile/reorder_python_imports
        rev: v1.3.4
        hooks:
          - id: reorder-python-imports
            args: [--application-directories=gaia]
            language_version: python3
      - repo: https://github.com/ambv/black
        rev: stable
        hooks:
          - id: black
            language_version: python3
      - repo: local
        hooks:
          - id: eslint
            name: eslint
            entry: ./frontend/node_modules/.bin/eslint --fix
            language: node
            language_version: system
            files: \.(js|jsx|ts|tsx)$
      - repo: https://github.com/prettier/prettier
        rev: "1.15.3"
        hooks:
          - id: prettier
            files: \.(yml|yaml|md|json)$
            language_version: system

</comment></comment></comment></comment><comment><USER47> Feature request: Add a flag that allows you to specify which lines to format.

I'm an auto-formatting true believer, but I _hate_ doing code reviews where the author has run a formatting tool that introduced diffs all over parts of the file that they didn't actually touch. It makes it really hard to review the actual change, and you have to worry about missing something.

With something like ClangFormat's '\--lines' flag, you can configure the tool to only modify the lines that the author actually changed. This removes most of the pain that comes with adopting an auto-formatting tool, and your codebase will gradually converge to the fully auto-formatted state over time.

This also makes it easier for to introduce improvements to the formatting tool itself, because the improvements won't immediately force a bunch of diffs on code that was already "blessed" by the tool.

<comment><USER48> Sounds like you're using it wrong. The autoformatter is for the whole repo, not for individual contributors. The point is that the whole repo has the same style, so you should only ever get diffs on the lines that were changed or the first time you run the autoformatter on the repo (and then you shouldn't be making manual changes that will be hidden amid the autoformatter noise).

<comment><USER47> Most larger orgs/projects aren't going to be willing to reformat all the code in their repo in one big bang. Regardless of what promises the tool claims to make, you need to be concerned about behavior changes or breakages, and most projects don't have high enough test coverage to cover everything. Depending on the size of your repo, it might not even be technically possible to make this change all at once. It's also much harder to get organizational buy-in on "let's change all existing code in one swoop" vs. "let's require all new code changes to be formatted using this tool".

As I mentioned above, formatting only the diffs also removes most of the pain of any changes to the formatting tool itself that aren't diff-stable.

<comment><USER48> I shouldn't have said repo, but rather some agreed upon unit of code. It could be a single package or file as long as all contributors touching that file adhere. You should not use black for any unit of code that you can't get buy- in on. Black is a technical tool; it doesn't solve political problems.

</comment></comment></comment></comment><comment><USER49> "Gofmt's style is no one's favorite, yet gofmt is everyone's favorite"

Happy Black user over here!

</comment><comment><USER50> Since I've started using black as a pre-commit hook and an editor extension, I have noticed that I tend to use inline-if expressions and lists comprehensions much more liberally, because I do not have to worry about formatting all the time. Begone temporary variables!

</comment><comment><USER51> This is a trend that I believe started with Go's code formatter, but such formatters differ in how much reformatting they will do.

Go's formatter adjusts whitespace within a line and will add or remove blank lines, but it doesn't change line breaks and has no opinion on the maximum length of a line. Other formatters work differently.

</comment><comment><USER52> There are some things in Black that I don't particularly like, for example the preference for double quotes over single quotes...

But the part that drives me crazy is this...

        # in:

        ImportantClass.important_method(exc, limit, lookup_lines, capture_locals, extra_argument)

        # out:

        ImportantClass.important_method(
            exc, limit, lookup_lines, capture_locals, extra_argument
        )

instead of

        ImportantClass.important_method(exc, limit, lookup_lines,
                                        capture_locals, extra_argument)

which to my eyes is way more readable and understandable. I understand what in some corner cases (if the call is too long that all arguments require its own line) it may be weird, but in those cases, it is showing that you try to do something strange in the first place (too long of a method name or too many elements in the call)

</comment><comment><USER53> How do you enforce the use of a formatter in a project -- pre-commit hook? pre-receive hook? both? And is there a convention to put a specific dotfile at the project root to hint at your IDE/editor that it should use a certain tool for automatic formatting?

<comment><USER54> CI runs a linter and fails the build if the output is non-empty. PRs can't be merged unless the build passes on the branch.

<comment><USER55> Yeah no because this changes the code and does not merely complain

<comment><USER56> One, that shouldn't matter on CI anyway. Two, you can tell black to just print the difference with --diff and tell it to just check the files with --check.

<comment><USER55> Yeah, true, though ideally you would do it before the commit so you wouldn't have one extra lint commit per code commit

</comment></comment></comment></comment></comment><comment><USER57> My current "problem" with Black is actually a 5-year-old bug in how pylint checks for bad-continuation issues with hanging indentation. Sure, you can ignore the pylint rule, but I imagine pretty much everybody who tries to use black and pylint on the same codebase runs into this issue:

* [https://github.com/PyCQA/pylint/issues/289](https://github.com/PyCQA/pylint/issues/289)

* [https://github.com/python/black/issues/48](https://github.com/python/black/issues/48)

</comment><comment><USER58> Every language should just have a go fmt. Go showed us a better way. All languages should copy this design.

</comment><comment><USER59> I love black: formatting isn't a _big_ thing to think about, but it's always there. Until black. For [current project] it's hooked into VS Code as the default formatter, and (inspired by simonw) we've incorporated a black check into the test suite. It's simple:

[https://github.com/simonw/sqlite- utils/blob/master/tests/tes...](https://github.com/simonw/sqlite- utils/blob/master/tests/test_black.py)

</comment><comment><USER60> Black is essential for maintenance of Pandas code which uses method chaining.

If you use Pandas you should be using method chaining and therefore also black.

<comment><USER61> However the pandas source is not blackened, and I really miss not be able to just fix all formatting issues while working on a PR. After just a few months I am so used to black it just feels weird to not have it everywhere

</comment></comment><comment><USER62> And in Stack Exchange's "The Workplace", people are complaining that their employees strongly object to Black [https://workplace.stackexchange.com/questions/136742/virulen...](https://workplace.stackexchange.com/questions/136742/virulently- against-code-formatting)

<comment><USER63> I have some issues with aesthetic decisions used in Black but the degree of head-nodding on the top answer in the SE post you linked is really disconcerting. The person basically says "well you're the boss so you should kindly remind your employees that its your way or the highway" in several different sentences.

It's pretty disturbing that people think this is how a manager behaves. Turns out there are reasons beyond aesthetics that people should apply strict code formatting practices.

</comment></comment><comment><AUTHOR> I like the idea of auto-formatting code, like go does.

Though I think I prefer autopep8

<comment><USER19> Black goes much further than autopep8. Black is an auto-formatter. autopep8 tries to automatically fix PEP-8 issues.

</comment></comment><comment><USER64> I initially used autopep8 and had set vscode to autoformat region on paste. When I changed to black I started getting error messages about region format not supported everytime I pasted.

As I mainly use other languages, with formatters with working region support, I went back to autopep8 so I could leave it format region enabled.

<comment><USER65> Doesn't it cache everything that it's formatted, meaning you could paste + save to achieve the same effect? I had the same annoyance as well but stated doing it this way and never had any complaints.

</comment></comment><comment><USER66> Auto formatters are great but the problem with Black is that Python is fundamentally incompatible with them, at least if you're only willing to insert whitespace. For example a line of code like this:

        x = y[1]['a'] + z[2]['b']

can be broken like this in Black (if it's over the line length limit):

        x = y[1][
            'a'] + z[2]['b']

But it needs brackets inserted so that it can be formatted something like this:

        x = (
            y[1]['a']
            + z[2]['b']
        )

You can argue the formatting I've chosen e.g. whether the binary operator should go on the previous line (the above line is what PEP 8 suggests). But surely it's clear that breaking in the indexing bracket is atrocious.

<comment><USER67> Black does add brackets in some situations.

<comment><USER66> Frankly, that is a really annoying comment to make. Can it add brackets in the particular situation I just said? If so, I'd actually be interested, because it's while since I last checked so it could will have changed. But if not, you've made it look like my comment's invalid for no reason.

It reminds me of the last time Black came up on a Hacker News. I posted the same objection (but hypothetically) and the actual creator of Black replied with a comment along the lines of "you should check for yourself before making a comment like that". By the time I finally did check, the discussion had died down (like this one now has) and not many people saw the reply. As the creator, he must have known that my objection was correct but he posted a low- effort misleading comment that invalidated it. It doesn't give me a great feeling about the project. I realise you couldn't have known that happened but you can see why I find your reply so annoying.

</comment></comment></comment><comment><USER68> Automatic code formatting should be the norm in programming. Once you get used to format-on-save and CI enforcement (especially on a team) it's hard to imagine going back to having to think about how to format your code, or pestering collaborators to follow some nickpicky code style guide.

</comment><comment> john-radio I like black. As hprotagonist pointed out, everyone has a few things about it they don't like. What I like to do:

        $ black --skip-string-normalization my_python_script.py
        $ git add --patch my_python_script.py

Then I accept or reject all its suggestions (other than the string "normalization" idea which is not our convention) on a case-by-case basis, then `git commit; git reset HEAD --hard`.

<comment><USER69> Seems like a lot of work just to satisfy some arbitrary aesthetic sensitivity.

Especially when working in teams, I couldn't care less about the style itself. As long as it's consistent, super easy to adhere (`black .`) and fairly readable, I'm happy.

</comment></comment><comment><USER70> If you use PyCharm, you can use cmd-alt-L to autoformat a file (and also reorder/fix imports). By default, I believe it just uses the pep8 standard, which I find generally sufficient, but you can evidently integrate Black via:

[https://black.readthedocs.io/en/stable/editor_integration.ht...](https://black.readthedocs.io/en/stable/editor_integration.html)

<comment><USER71> FWIW gg=G formats in vim (whole doc), with lots of variations available .. more examples: [https://stackoverflow.com/questions/506075/how-do-i-fix- the-...](https://stackoverflow.com/questions/506075/how-do-i-fix-the- indentation-of-an-entire-file-in-vi)

</comment></comment><comment><USER72> pre-commit with black and flake8 makes for a pretty low overhead setup to have a pretty consistent codebase

</comment><comment><USER73> Black is an awesome tool. You might not like one or two of their decisions, but it completely eliminates discussions related to taste on a given team and the fact it is not configurable is its biggest value proposition.

</comment><comment><USER74> I hope to have something similar for every programming language, no need for configurations, no petty discussions on this anymore.

        If you're paid by the line of code you write

:-)

</comment><comment><USER75> I haven't used Black in production; does it work well with a fully explicit .pylintrc file, or does it do its own thing? I think interop with Pylint would make or break such a tool for me.

</comment><comment><USER76> Neat. I've been using an autopep8 pre-commit hook for a while, even though I kinda hate it (but nobody complains about noncompliant code anymore). I'll have to give this one a shot

</comment><comment><USER77> Why on earth would it convert this:

printable_chars = { 33, 34, 35 [...and another few hundred ]

to printable_chars = { 33, 34, 35 [...and another few hundred lines ]

<comment> crooked-v Those look the same to me.

<comment><USER78> Perhaps your parent failed to add double-newlines and HN's formatting ate their newlines, but notice the extra word "lines" at the end.

</comment></comment></comment><comment><USER79> How do people use autoformatters in CI pipelines? I dislike the fact that precommit hooks have to be set up for each git clone. Is there a better way?

<comment><USER80> At my previous job, "Formatting" task was the first thing pipeline did. It was really simple - it ran the same "make format" target you could run locally, except it was configured in a way to fail if running "git diff" afterwards resulted in any changes when running on the build server (git diff -s --exit- code). Of course, when this failed, none of the other tasks in a pipeline ran so it forced you to format your code correctly if you wanted to see build results. You could just commit the next commit with "formatting" message but the idea was to just change the history since commiting was happening on dev branches - only when everything in the pipeline passed things could be merged on stable branches.

</comment></comment><comment><USER81> i have been using the black integration with vscode and i have come to the conclusion that:

1\. black is not perfect, but it is pretty great

2\. vscode is a much better IDE than Atom

</comment><comment><USER82> I started using black with one of my personal projects. Now I find myself wishing if was set up in every other python project I work on!

</comment><comment><USER83> I sincerely hope that adding this to your project is referred to as "taking the Black." Too good an opportunity to miss.

</comment><comment><USER84> My friend (who played keyboard in my jazz quartet until he moved to Europe) is the author of this! It's super cool.

Hi Lukasz!!

</comment><comment><USER85> Is there anything like this for ruby? Would love using it as a plugin for sublimetext, atom or visual studio.

<comment><USER86> [https://github.com/testdouble/standard](https://github.com/testdouble/standard)

</comment></comment><comment><USER87> Black is my favorite auto formatter. I use it for python scripts and blackcellmagic for jupyter notebooks.

</comment><comment><USER88> More languages should have this. Does anybody know of a c# autoformatter that works on Linux?

</comment><comment><USER89> The ford logo is a nice touch.

<comment><USER90> For anyone who doesn't get it, the name and logo comes from the famous story where Henry Ford, maker of the Model T -- one of the first affordable automobiles, because it was made on an assembly line -- said "Any customer can have a car painted any color that he wants so long as it is black." This quote and assembly-line idea translate into the branding of this project. (The project logo is visually similar to the modern Ford company logo, but with the word Black instead of Ford.)

</comment></comment><comment><USER91> perhaps the only missing feature (like in flake) is to be able to bypass the formatter with a comment (to be confortable with edge cases, where readavility matters)

something like: # nofmt

<comment><USER31> This exists now :)

</comment></comment><comment><USER92> Let me know when it's available as a Vim plugin.

<comment><USER93> It's been available that way for over a year:

[https://github.com/python/black/blob/master/plugin/black.vim](https://github.com/python/black/blob/master/plugin/black.vim)

</comment></comment><comment><USER94> >uncompromising

But does it have an #ignoreformatter directive?

<comment><USER95> Yes.

      # fmt: off
      unformatted code goes here
      # fmt: on

</comment></comment><comment><USER96> Having worked in Python for many years, I can tell you that nobody actually writes code that looks like Blacks formatting.

It's a major step backward for the Python community.

<comment><USER48> Having worked in Python and other languages for many years, I can tell you that 1) this doesn't matter and 2) many of the ways people write Python are much uglier than the conventional formatting in other languages. Most notably:

        this_is_my_very_long_function_name_and_here_come_the_args(self,
                                                                  foo,
                                                                  bar,
                                                                  baz)

<comment><USER97> Yay! This. Where else do we recommend arbitrary indentation levels? Nowhere. So...which is the "foolish consistency" then? Is it consistent indentation rules? Or consistently lining up arguments with function/method opening parentheses?

</comment></comment></comment><comment><USER98> Bikeshedding: The Thread.</comment>


Breaking your mind currying functions in ruby - <AUTHOR> https://blog.browntreelabs.com/breaking-your-mind-currying-functions-in-ruby/ <comment><USER2> I think you have a few JS/Ruby examples mixed up, which makes the article kind of hard to follow. But on topic--I absolutely love Ruby, but I don't think I'll ever get used to Ruby's call syntax, e.g. foo.(), foo[], foo.call(), with the latter being my preferred method.

...it's like Matz purposefully made the language feel and look terrible when used functionally. :)</comment>


Secret Society of Soul Painters (neural style applied to video) - <AUTHOR> https://www.youtube.com/watch?v=0uCdJKVjPmQ <comment><USER2> This would make a great movie or TV show intro sequence.</comment>


How a Blog Might Have Stopped Madoff - <AUTHOR> http://exchanges.nyse.com/archives/2009/02/madoffblog.php <comment><USER2> The SEC wasn't interested in investigating documented allegations. I don't see how a blog could have made anything different except to get Markopolos attacked by syncophants of Madoff. And further, based on the sort of people who were in the previous administration, I suspect that Markopolos would be the one behind bars if he had published his allegations in a blog.

</comment><comment><USER3> Communication doesn't always have to be vocally spoken. Good link.</comment>


When App Engine Went Wrong - <AUTHOR> http://www.agmweb.ca/blog/andy/2286/ <comment><USER2> DeadlineExceededError is a really stupid problem in App Engine and the simplest fix would be to just kill the instance after such an error and start a new one. Google doesn't want to do that because it wastes resources. So, the current solution is to write Python modules which, when they're imported, will work correctly even if the execution stops in the middle of the module and then is restarted from the beginning (i.e., when a new request is received).

It's possible that with the upcoming pre-warming requests this bug will disappear in practice because your whole project gets a chance to import and initialize all modules before the first request is received. If these pre- warming requests don't have a 30s timeout your instance will always have enough time to finish the initialization in a non-broken state.

Anyway, it's your own mistake if you stay with the Django helper. The bug can be worked around. It's fixed in Django-nonrel. I still see people who try to use the helper for mission-critical projects. Don't do that. The helper is buggy. It uses monkey-patches all over the place. This makes it extremely vulnerable to bugs caused by DeadlineExceededError. Just use Django-nonrel, even if you only want to use App Engine's models.

</comment><comment><USER3> I don't have anything running on App Engine, in large part because Google is not a service provider I can trust. There's a really tough dichotomy between selling their magic scaling beans and the fact that anyone who needs said scaling beans also needs real support—something which is simply not in Google's DNA.

<comment><USER4> I'm surprised that Google isn't just selling a raw, white-box cloud-services layer (like AWS) that other companies can then resell with convenience and support on top.

<comment><USER5> Why are you surprised by this?

AWS was grown out of Amazon's internal systems. They productized what they used everyday because they thought it would probably be useful to others.

Google doesn't offer anything like a virtual server internally. Everything is built on their various abstraction layers. They are productizing what they used everyday because they thought it would probably be useful to others.

</comment></comment></comment><comment><USER6> I had a very similar experience, unfortunately it wasn't feasible for me to migrate the site to another vendor, we just had to sit tight. It almost ruined the launch, which had a hard deadline. Fortunately we too had an understanding client, but being hamstrung was no fun at all.

The issue of the App Engine status page not reflecting reality is a very real one - we had contact with a Google engineer who told me that "this is affecting all our stuff" and I know that it went on for several days, but this wasn't reflected in the status chart; the severity of the issue was definitely not conveyed well enough.

I like the promise of App Engine, particularly the "instant" scalability which is great for sites that you know are going to get hammered as soon as they go live. But it really highlights the risk you take when you allow yourself to be locked into such an all-encompassing platform.

</comment><comment><USER7> Shameless plug: Djangy.com fixes all of these problems. We're focusing on customer service + all of the other things that suck about app engine.

In private beta now, but if you're in need of something better than appengine right NOW, email me and we can get you in.

<comment><USER8> Can you explain your infrastructure a bit (or is it a trade secret)? Even though I sorely need a service like that, I don't like black boxes...

That said, I'd love to join the beta.

<comment><USER7> I'm reluctant to give up too much detail, but our infrastructure is very, very scalable and secure. It's pretty solid, and allows for very fine-grained control over many instances of any running application. Load balancing is taken care of, as well as redundancy.

Once we go public, it's possible we'll be more open about the architecture, but right now we're still iterating pretty rapidly, trying to make the user experience dead-simple without sacrificing stability or scalability.

<comment><USER8> I see, thank you.

</comment></comment></comment></comment><comment><USER9> django + appengine is something I am suspecting. Personally, I had good luck with sticking with tornado + appengine. Tornado is pretty raw and atleast keeps an option to port the code over if I have to. Datastore and dns are still suckers in the performance.

</comment><comment><USER10> I'm looking forward to the current crop of ongoing errors being resolved. I strongly prefer App Engine for a variety of reasons (built in versioning, pay only for what you use, easy deployment).

Right now I'm just happy that some of the features that would have caused me to put all my eggs in the app engine basket weren't available, because if they had been I'd have felt a lot more pain from the outages.

</comment><comment><USER11> Has google posted anywhere on these issues and what they are doing to resolve them? I'm working on a site for App Engine and this has me a bit concerned.

<comment><USER12> There's a good amount of fear mongering going on here. I've been using App Engine+Python exclusively for side projects since its release and never run into these problems.

</comment></comment><comment><USER13> It's never been clear to me why people use the AppEngine for anything other than learning and playing around.

Why not just get a $40/mo. VPS?

<comment><USER2> It's because a little VPS won't give you a fail-safe system. For a SaaS startup you'll need to have several machines in a redundant cluster and ideally you'll also have machines in at least two data centers. The people who use App Engine for their business all have the same dream: Build the app, click deploy, and have Google handle all the annoying server stuff. Obviously, App Engine isn't quite there, yet. It's not fast enough, it's not stable enough, and it has a limited feature set. Clearly, we're early adopters. However, App Engine is going in the right direction and once the problems are gone people will ask "Why not just use App Engine?" (or whatever other PaaS) instead of "Why not just use a VPS?".

<comment><USER14> Because I'm feeling pedantic today: you meant to say "a fault-tolerant system". A fail-safe system is one which fails in a safe way. An example would be a CPU that shuts itself down if it's overheating, rather than risk permanent damage. If anything, a fail-safe system may be _more_ prone to failure because of its cautious approach to dealing with faults.

There are some pretty cool examples on the Wikipedia page:

<http://en.wikipedia.org/wiki/Fail-safe>

<comment><USER2> OK, thanks. Will try to remember that. :)</comment></comment></comment></comment>


Comcast has agreed to sell its stake in Hulu in 5 years - <AUTHOR> https://www.cnbc.com/2019/05/14/comcast-has-agreed-to-sell-its-stake-in-hulu-in-5-years.html <comment><USER2> Biggest thing I took from the article is NBC gets half a billion dollars a year for it's content catalog, WOW.</comment>


FBI to ACLU: Nope, we won't tell you how, when, or why we track you - <AUTHOR> http://arstechnica.com/tech-policy/2013/01/fbi-to-aclu-nope-we-wont-tell-you-how-when-or-why-we-track-you/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+arstechnica%2Findex+%28Ars+Technica+-+All+content%29 <comment><USER2> People are able and should be tracking the F.B.I. it's all in the numbers.</comment>


Blackberry's Net Neutrality Plan - <AUTHOR> http://blogs.blackberry.com/2015/01/blackberry-net-neutrality/

<comment><USER2> They'll get five minutes of newsworthiness and prompt many to throw their two cents in about how asinine their argument is, but I imagine it's a bit late to rock the boat for them.

</comment><comment><USER3> Have a failing business? Not to worry. Just lobby for legislatures that can render your business relevant. Problem solved.

<comment><AUTHOR> Honestly, this is the most ridiculous move from a company to try to save their business that I've seen in a long time. It's utterly absurd on Blackberry's part

<comment><USER4> Not to mention they're not even a US company. Why on earth would Congress pass a law favoring a Canadian company over the US industry leaders?</comment></comment></comment>


Is ISIS actually related to Islam? - <AUTHOR> http://istanaislam.tumblr.com/post/143684802595/isis-didnt-originated-from-islam-debunked-once <comment><USER2> _" As far as I know, the original name of this terrorism entity is in Arabic. So why must you you focus, highlights and link it to Islam?."_

From Wikipedia:

 _" The group has referred to itself as the Islamic State (الدولة الإسلامية ad-Dawlah al-Islāmiyah) or IS ever since it proclaimed a worldwide caliphate in June 2014"_[1]

So _they_ call themselves the "Islamic State" in Arabic, and thus _they 're_ the ones who are focusing on Islam. Others are just accurately translating their name.

[1] [https://en.wikipedia.org/wiki/Islamic_State_of_Iraq_and_the_...](https://en.wikipedia.org/wiki/Islamic_State_of_Iraq_and_the_Levant)

</comment><comment><USER3> Let's put it this way: could ISIS actually function without Islam? If you think it can't, then it's related to Islam. And I think the OP should at least try to say to what is ISIS related and why.

I file this under the No true Scotsman fallacy.</comment>


Samsung and NTT Docomo jointly developing new smartphone OS - <AUTHOR> http://www.yomiuri.co.jp/dy/business/T121230002722.htm

<comment><USER2> OMG!!!</comment>


Certain bacteria may help ward off obesity - <AUTHOR> http://www.latimes.com/science/la-sci-gut-bacteria-obesity-20130906,0,194428.story

<comment><USER2> Very cool to read ... by the end of the article thought I found myself saying, "well?", as if expecting them to tell me what I could eat to foster the "correct" type of bacteria. Of course, it seems that we're still at the beginning of this kind of research, so there are no well established answers yet.

<comment><USER3> I'm glad there's finally some evidence to support this theory, but it's not exactly a new line of thinking. There's already been literature published based around the correlation between gut flora and obesity, and one book that stands out and has actionable tips is 'The Diet Cure' [1]. I admittedly haven't got around to implementing it yet myself, but the concepts make sense. Either way, it was originally written in the late 90's, so I wouldn't expect it to be thoroughly modern in its reasoning and nomenclature with regards to these microbiome developments, but it's something...

[1] [http://amzn.com/0143120859](http://amzn.com/0143120859)

<comment><USER4> I've subconsciously been putting this into practice probably, i.e., eating a 'clean diet', or foods that promote 'good' bacteria. I never talk about it, though, because it sounded very much like pseudoscience (until recently, I guess?). Just not enough samples or controlled groups for meaningful data to be interpreted. Strangely enough, I find this to be true for a lot of things in nutrition/diet.

Madonna is one of the biggest proponents of this diet who is famous: [http://www.bloomberg.com/news/2012-05-08/madonna-s-secret- fo...](http://www.bloomberg.com/news/2012-05-08/madonna-s-secret-for- longevity-seen-aiding-bacteria-boom.html)

The article mentions probiotics, and so does the LA Times article, but there are also 'natural'/whole foods Madonna consumes, which are recommended by her nutritionist, who she works closely with.

<comment><USER5> _I never talk about it, though, because it sounded very much like pseudoscience (until recently, I guess?)._

Changing the gut bacteria doesn't, but trying to do it through diet probably does.

<comment><USER4> Admittedly, I'm not too well read on it. So I have a lot of questions unanswered. I'm sure many feel this way: there are times when I wish I could read and understand so many things, but between work, learning to program and my family duties, I feel like I fall behind all the time in other areas.

Thanks for the feedback, by the way.

</comment></comment></comment></comment></comment><comment><USER6> Poop transfers! Get your poop transfers!

Pick from a catalog of svelte donors at FloraFlash!

All your fat got you down? Well, don't wait another minute! You deserve to be living a life of supreme-fucktitude! Get laid being the tight tiger you always knew you were.

A study in a journal by some doctors who are trying to be silenced by THEM have discovered natures miracle! POOOOOOOOOP! But don't settle for that off- the-shelf shit. Get yourself down to FloraFlash for the best in rectal remedies!

Our certified specialists each study under master transfer technologists using the latest techniques to ensure that not even a drip of the slimming poo juice leaves your body before it has rebooted your fat-fighting gut with enough underwear model crap to ensure that you will be wearing a tight banana-hammock in no time!

FLORA FLASH!

</comment><comment><USER7> Plenty of opportunities for start ups with catchy brand names. Overheard on Twitter:

* Skinny the Poo * FitShit

</comment><comment><USER8> So will tapeworms. Be careful what you wish for.

<comment><USER9> That's a very good point. It's awfully similar, though hopefully the lean- bacteria won't have as many bad side-effects :)

<comment><USER10> It's not similar at all. Your body already has several pounds worth of bacteria in it. Hopefully, your body has zero tapeworms.

This is talking about bacteria already in a healthy person's body.

<comment><USER9> tapeworm vs bacteria, parasite vs parasite, symbiotic or otherwise. and you have plenty of other parasites.

edit: maybe a simpler example. Try to describe why tapeworms are different from bacteria, without using "multicellular". both can cause harm. both eat your food. both can result in weight loss.

<comment><USER10> [http://en.wikipedia.org/wiki/Gut_flora](http://en.wikipedia.org/wiki/Gut_flora)

Bacteria that aid in weight control don't just "eat your food," they actually help your body break down food correctly. That's why you take a pro-biotic after going through a round of anti-biotics - you need to re-grow your gut flora to aid in proper digestion.</comment></comment></comment></comment></comment>


Joe Armstrong: "In my opinion Erlang is brilliant at handling text" - <AUTHOR> http://groups.google.com/group/erlang-programming/msg/a5c0e9585d148294 <comment><USER2> Erlang now has a full-featured standard regex module called re that handles utf8 strings and binaries. It now has a standard module for handling unicode.

Yes, regexes aren't part of the language syntax, but then that's not true for, say, python, either. Having to use a library to do regex is not going to be the difference maker in productivity for your app.

Besides, while I love regex, I still use them as a last resort. You're going to want to use a real parser for reliable structured text processing.

Generally, erlang programmers keep strings in binaries, which are compact. Most modules for handling string type tasks allow you to do this, e.g., the re module understands a unicode_binary type.

In 6 months of programming erlang professionally, in a domain dominated by scripting languages like python and ruby, I've certainly never been tempted to bolt over string handling issues. Erlang's flexible distribution, concurrency, and reliability model is just too compelling.

To get competitive performance in a reasonable amount of programmer time for concurrent applications in other languages you're limited to the subset of tasks that, say, twisted or tornado makes easy. The program I'm writing now couldn't have been done with either of them.

Frankly, if it's the choice between built-in support for regex and built-in primitives for distribution, concurrency, and fault tolerance, there's no question in my mind which is more important.

</comment><comment><USER3> He compares Erlang's string handling with C... And C sucks at string handling. He should compare Erlang to all the modern languages where strings are a first class citizen. Comparing Erlang's string handling to Java, Python or Ruby might change his mind.

<comment><USER4> I think the main (and almost the only) reason to have distinct string type is performance, not for the ease of programming. Many string operations are useful as a general list operations as well, including regular expression matcher. (Note: Some people emphasize importance of O(1) access of string access by index, but using integer index is also a performance hack. If search operations can return some way to point to the substring you don't need integer indexes.)

Another minor reason is to display; people prefer reading sequence of characters in a string syntax. If you have a statically typed language it is easy to display a list of chars in string syntax instead of list syntax. For a dynamically typed language with heterogeneous lists, it can be a performance penalty to check whether a list entirely consists of characters or not at runtime. So, in a sense, it is also about a performance. (Note: Having a syntax for strings has nothing to do with having distinct type for strings. The string syntax can be just a syntax sugar.)

But performance is important, of course. One thing very common in string (a list of characters) but not very common in general lists is concatenation. To be precise, lazy language programmers use list concatenation without a guilt, but eager language programmers tend to avoid it since it may cause unnecessary copying of lists. So for the eager evaluation languages, it makes sense to have a string type that has very cheap concatenation operation (e.g. using tree representation) internally.

<comment><USER3> Performance and _ease of use_ are the main reasons why strings should be first class citizens. Strings are one of the most used data structures and most of today's popular languages have very good support for them, encodings of them and manipulation of them. C does not have that good support for them. Ignoring strings and labeling them as "a list of integers" is a step backward, since a lot of the data we have today is textual and will continue to be textual in the future.

<comment><USER4> Having dynamic typing makes the discussion complicated, so let's assume we have a cheap way to know the type of a given object.

In one world, you get an object of type [Char] ---means a list of characters--- and you can apply all sorts of list operations on it, and all sorts of operations specialized to [Char]. You can add a type alias String to the type [Char]. In the source file you can write "string" and it is read as a list of six characters. On the output the same list is printed as "string".

In another world, you get an object of type String, which is distinct type from a list of characters. Type String has all sorts of useful operations. But if you want to apply a generic list algorithm, you have to either duplicate the code, or coerce the string into a list. Conversely, if you have a list of characters and want to pass it to a string library function, say, regexp matcher, you have to coerce it to a string.

Which is easier?

As nostrademons commented, one way is to implement a generic interface so that you can write a generic algorithm on top of both list of characters and Strings, but that's actually the same thing I'm saying. I say "list" as some data structure on which you can peek the head, the tail, and you can add an element in front of existing one. I don't care how it is represented---if the runtime or the compiler can find out the list only contains ASCII characters, it can freely store the entire list in an octet array. In a sense, I say "list" as "data structures that implement the list interface".

Now, suppose if you have such a smart runtime/compiler. Suppose you can have specialized functions on [Char], apart from generic list. Do you still think having distinct string type is for ease of use?

In reality we don't have such sufficient smart runtime/compiler, so we compromise. That's the distinct string type.

<comment><USER3> I would prefer if strings were treated as a list of characters and NOT as a list of integers (as they are in Erlang)...! I.e. your reasoning does not really apply to Erlang.

<comment><USER4> Oh, I thought you were arguing with my proposition: Distinct string type is for performance and not for ease of programming.

I agree that conflating [Char] and [Integer] is not good. That's a different story.

<comment><USER3> I disagree with your proposition. Languages should have a string type and not only for performance, but also for ease of programming, since strings are an important and special data type. I don't really care how this string type is implemented - if strings are an array of characters (like in Java) or a list of characters (like in Haskell). What I think is important is that a language treats strings as first class citizens - and not just for performance, but also for ease of use. And currently, strings aren't a first class citizen in Erlang (due to Erlang's limited support for custom datatypes).

<comment><USER4> Let's forget Erlang. We've agreed that conflating [Integer] and string is bad.

I can't find the reasoning to back up your claim in your posts; if I miss it, could you point it? I think I explained a few points that a language does not need _distinct_ string type, except from performance reasons.

Note that I've never said that strings shouldn't be a first class citizen. A list of characters _is_ a first class citizen. You can have rich string library on top of lists of characters, plus generic list operators works on them. So, why do you want a string type _disjoint_ from lists?

<comment><USER5> It seems the key is having a good "Char" type, so that Unicode bytes turn into something matching the intuition of what a character should be. So the bytes for "a umlaut" or whatever become a single Char in the list. So the Char type is responsible for storing a representing characters at the right level of granularity, and then string operations can all be implemented as list operations.

Is there a case where this still breaks down?

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> I don't know very much about erlang but I can understand why it has the stigma that it doesn't handle text well as almost every introduction I have seen has said this is the case due to it being created by telecommunication companie[s].

However my familiarity is limited and whether this is true or not I cannot comment on, but comparing erlang to C in performing text based operations is probably silly if the other person has an option such as say perl available to them.

<comment><USER7> String processing is just one of those things you can't offload to a second process. Having Perl in the same box doesn't mean you have the luxury to open some IPC channel and pass your texts to Perl. String processing is one of those types of problems that just benefit from a little thought and 10 seconds of planning.

C itself is bogged down by its own horrible string representation, the nul termination, where most operations need to traverse the string up to the terminator before they know where it ends. This causes all sorts of horrible buffering tricks, conditional tests on every character, and other unpleasant things. Pascal had length-prefixed strings from day one, and DJB created Netstrings to make network programming easier on people. See <http://c2.com/cgi/wiki?LeasedString>

</comment></comment><comment><USER8> Even though he has a good point: strings are lists, and erlang is good at lists — erlang doesn't have native regexps like perl, js, or ruby, nor is the support for them that good.

<comment><USER9> This is true - looked at making a very concurrent webcrawler in Erlang, and the regex bit was painful.

<comment><USER10> From 12Bx that changed, regexps are enjoyable but still not as rich as Perl.

<comment><USER11> Maybe someone should write a parse transform for dealing with them?

</comment></comment></comment></comment><comment><USER12> How does Erlang fare with Unicode in practice?

<comment><USER13> If I understand it correctly, Erlang fares well here because it stores each character as a 32-bit integer in a list. The implication of that approach is more memory overhead for each character, but that allows you to treat strings as a list of characters and gives you all the benefits of the standard list- manipulation libraries.

<comment><USER14> > Erlang fares well here because it stores each character as a 32-bit integer > in a list

Which is silly because if you make a list of 32 bit integers that have ASCII equivalents then Erlang assumes it is a string.

<comment><USER15> The main difference is the way it's displayed by default in the shell. Strings are handled with list operations, and Erlang is _great_ for list operations.</comment></comment></comment></comment>