Europe reverses course on net neutrality legislation - <AUTHOR> http://arstechnica.com/business/2015/03/europe-reverses-course-on-net-neutrality-legislation/?r <comment><USER3> We don't have the same problem in europe with net neutrality.

The problem in the states is very very different. In the states you have three corporations providing shit, slow service. Th

Traffic shaping, which stops one person on a contented link (ie ADSL) ruining everyone else's day by torrenting every last drop of bandwidth. Its been a fact of life for many years.

THe problem in the states can be characterised in two ways:

* A lack of competition

There is often only one ISP in town, and they don't have to try hard. This means that service is terrible, and so is innovation.

* A lack of bandwidth

In Europe bandwidth exchanges, or peering exchanges are normally run by a cooperative. This means that its in everyone's interest to talk to everyone else as fast as possible. Its really really cheap to peer in london, ([http://www.lonap.net/fees.shtml](http://www.lonap.net/fees.shtml))

In the states, exchanges are run by either AT&T, verizon or level3. Which means everything costs squillions. As there is no need to provide more bandwidth, because customers can't move between ISPs so no one peers.

For example in the EU, ISPs are more than keen to have CDN leaf nodes in their infrastructure, it lowers costs and improves customer experiences. However they can't charge Netflix et al to install them because there is too much competition.

Its not like they can limit access to netflix for a few months to make a point, because their customers would change ISPs. Something not possible in the States.

So net neutrality is really a side issue to competition.

<comment><USER4> > So net neutrality is really a side issue to competition.

No, net neutrality is not a side-issue to competition.

Competition would mitigate some of the harms of non-neutrality, but it does not address the full picture. Non-discrimination is essential both for consumers and for content producers[0]. Non-neutrality has an incredibly chilling effect on business, even if it only applies to a portion of the population[1].

We need both neutrality and competition; we can't sacrifice either one for the other.

[0] ie, web companies, startups

[1] [https://avc.com/2014/01/vc-pitches-in-a-year-or- two/](https://avc.com/2014/01/vc-pitches-in-a-year-or-two/)

<comment><USER3> As someone who works for a content producer, the only place to make money is in the gatekeeping to content.

Once again, with competition that sort of practice doesn't work. If a service is popular it will cost customers to ban it.

Nokia phones came with music for the best part of 3 years, it didn't stop the rise of spotify. (back when data cost £1 a megabyte)

Spotify is bundled with many contracts here in the UK, but people still use other services.

None of those VC ideas are businesses, it won't be net neutrality that stops them it'll be common sense. Who invests in clones of already loss making services?

</comment></comment></comment><comment><USER5> This is why I fear net neutrality (and the internet as we know it) don't stand a chance.

A lot has been accomplished with public awareness and public outcry has repeatedly averted disaster - _so far_. But the proponents of this come back with another attempt at least once a year, in both Europe and the US, and they have money on their side. It's amazing net neutrality has gone on as long as it did, against vast corporate and political interests.

Whereas net neutrality advocates have to beat this back _every time_ everywhere, their opposition has to be successful _only once_ , and only in one of these regions, for the whole thing to fall.

It is, I'm afraid, just a matter of time.

<comment><USER6> It's very true it's a war. Each win we get against idiot corporations is a battle won, but the war rages on. Hard to say if it'll ever be "won."

It's especially hard when so many people are spouting stuff about how the free market will solve this (if I got a dollar every time "free market" was the drooled out answer to a complicated problem...). It's like we have to convince a critical mass of our peers of the benefits of free speech every time the issue comes up.

However, it's still important we fight each battle, each time one rears its ugly head, wherever it happens, with whatever resources we have available. The internet is too powerful a tool to be left in the hands of greedy slimeballs. It's the great equalizer. Anyone can have a voice, and anyone can listen to that voice, whether you're a giant corporation or a 12 year-old kid with a good idea.

It's worth the struggle.

<comment><USER7> The free market would solve it. ISPs do not operate in a free market.

<comment><USER8> I'm gonna drop a train on you. Unless you're seriously suggesting living in an anarchic society where might means right, there is no such thing as a "free market." There are only differing levels of regulation. At a very minimum you need some body to enforce contract law. More specifically to this issue, you have the problem of running lines to peoples' houses. With no regulation and no sharing of poles/lines, you'd have very high barriers to entry (read: monopolies, bad for consumers) and dozens or hundreds of poles and lines in every yard (bad for the environment and the people who live in it). So okay, you build a few lines and enforce the line owners to share the lines equally to create a "free market" in the service provider space. But now you've introduced regulation and destroyed the "free market" in the line and pole provider space.

Boy, this is complicated, isn't it? It's almost like "free market" isn't a magic incantation that fixes everything.

<comment><USER7> Hence my replying to the parent

</comment></comment></comment></comment></comment><comment><USER9> Net neutrality is very important, but it occurs to me that when we finally figure out that we don't need separate telephone lines, cable tv, etc - i.e. when we realise _it 's all just data_ and one-pipe-fits-all - then it seems fairly obvious that your neighbours 999/911 VoIP call should perhaps not be drowned out by 4K Netflix. And neither should your burglar alarm. Or panic button. Or self-driving car. Or robotic doctor.

It's an issue in its infancy now, sure, but it will become a problem.

<comment><USER10> The issue has stayed in its infancy for 35 years so far (Jon Postel wrote about it in January 1980), so do you think it will become a bigger problem in the next five? Ten? Twenty? If so, why?

<comment><USER11> The answer to that should be pretty obvious. How many different services relied on internet in 1980? How many will in twenty years?

And I'm not sure the issue still is in its infancy. It's becoming a problem and it's stopping further innovation. I think we really need to think about "internet 2.0".

<comment><USER10> "Many" and "even more". That's not the point.

The internet has added hosts, users and services every year since 1980. It didn't cross a threshold to need priority blah in 1981, not in 1982 either, not in any of the 33 following years. What's so special about the 36th year? Or the 40th?

</comment></comment></comment></comment><comment><USER12> TL;DR The only problem is that ISPs love to oversell their capacities. And important services don't need that much bandwidth. NN has nothing to do with this.

The "911 calls should not be interrupted by torrents" is such a load of crap argument. It has nothing to do with net neutrality but is all about service level agreements.

With landlines, the main problem here is not that your neighbour could prevent you from making a 911 call by torrenting, but that his torrenting affects your bandwidth at all.

When you buy 10mbit from your ISP, you should _always_ be provided with that speed, and the ISP should take measures to ensure that. Not on a per service level, but on a per customer basis.

But instead ISPs like to sell "up to 25mbit" offers which actually means "0mbit guaranteed, the rest is lottery" because you share that "25mbit" connection with everybody on the street.

Mobile contracts are a bit trickier, because there the topology is unknown and can change anytime, but even here we can calculate a lower bound if we take into account that each lte cell can only support a few hundred [(maybe a few thousand, if you have really fancy tech and own a very wide frequency spectrum)]([https://www.linkedin.com/groups/Plz-anyone-help-me- How-11807...](https://www.linkedin.com/groups/Plz-anyone-help-me- How-1180727.S.217555082)) connections at the same time.

Even if we assumed that an lte cell can have an utopian 10.000 _active_ users using every available frequency, we would still be left with 300mbit/10.000=30kbit per user, [which is still enough for that goddamn 911 call.]([http://www.erlang.com/bandwidth.html](http://www.erlang.com/bandwidth.html))

</comment><comment><USER13> >The proposals did not go unopposed. More than 100 MEPs signed a letter to the Telecoms council that accused it of "lacking ambition". "Weakened proposals on net neutrality go against the European Parliament's repeated calls for clear definitions," it read. "We call on you to have clearly defined net neutrality rules for Europe."

I think this is the important part, yes some services probably should be considered special, many people here mention VoIP as an example, but it must be clearly defined in the law, we must not give ISPs the power to decide what is special because the law would be become effectively worthless.

<comment><USER14> And then a new service that requires low latency appears. Have fun waiting for the European Parliament to vote on it, for the EC to ratify the change, and for each member state to turn it into law.

But hey, I'm sure the providers of the services that are already in the law (like VoIP) will be very grateful for this rent you've granted them.

<comment><USER13> What would be the alternative, should we just get rid of the laws then? I think that would be an order of magnitude worse than having to wait.

Also, in my mind special would mean that the service is of great public importance, it doesn't seem likely to me for such things just turn up every other day.

<comment><USER14> _What would be the alternative, should we just get rid of the laws then? I think that would be an order of magnitude worse than having to wait._

We don't have net neutrality laws in Europe right now to get rid of. The alternative is doing nothing, which in my experience is working pretty well. Are you European? Are ISPs in your area trying to slow down some traffic? If not, what problem exactly are you trying to solve?

And even if you _are_ having problems, why do they must be solved at the EU level?

 _Also, in my mind special would mean that the service is of great public importance, it doesn 't seem likely to me for such things just turn up every other day._

Services don't appear and are suddenly recognized as of great importance. They start small and unimportant until they gain enough momentum to be recognized by the public at large. And they'll never will if you enshrine into law the privilege of existing ones.

<comment><USER13> >... what problem exactly are you trying to solve?

Look at what happened in the US, net neutrality was not an issue, just like it isn't really in the EU at this time, then services like Hulu/Netflix/Youtube appeared. In the US like in Europe most ISPs are either Cable companies or Telecoms which offer their own IPTV. Now you as a ISP have a competing service using your network, why would you just tolerate that if you don't have to?

Well they didn't, Netflix is now paying off AT&T, Comcast and Verizon so they won't throttle its service. Now in order for your service to compete with Netflix you'll have to pay up too.

How does that further competition?

>And even if you are having problems, why do they must be solved at the EU level?

Why not? How is the EU's common market supposed to work if we just keep implementing things like this on a country by country basis.

> Services don't appear and are suddenly recognized as of great importance. > They start small and unimportant until they gain enough momentum to be > recognized by the public at large. And they'll never will if you enshrine > into law the privilege of existing ones.

Again I don't think this will be an issue, even if you allow for special cases, _Netflix_ should never be special, and neither should probably 99% of services available today. VoIP is special (and I guess is already treated as such by ISPs) because it is a replacement for PSTN which was special and you can't have voice communication be degraded because people are watching the World Cup final or whatever.

Let's put it this way, what would be the harm of having a common net neutrality law in Europe?

Wouldn't that just mean that if you want to do business inside the EU you won't have to worry about being blackmailed by ISPs in 28 different countries?

And yes I live in Europe.

<comment><USER14> _Look at what happened in the US, net neutrality was not an issue, just like it isn 't really in the EU at this time, then services like Hulu/Netflix/Youtube appeared. In the US like in Europe most ISPs are either Cable companies or Telecoms which offer their own IPTV. Now you as a ISP have a competing service using your network, why would you just tolerate that if you don't have to?_

For the same reason I'm paying much less for higher speeds than the average US household. Unlike most places in the US, there's competition in the ISP market in my country. I live in a city of less than 100K, and I have four ISPs offering me fiber.

 _Why not? How is the EU 's common market supposed to work if we just keep implementing things like this on a country by country basis._

How is the common market hampered by this particular regulation being implemented at a national level?

 _Again I don 't think this will be an issue, even if you allow for special cases, Netflix should never be special, and neither should probably 99% of services available today. VoIP is special (and I guess is already treated as such by ISPs) because it is a replacement for PSTN which was special and you can't have voice communication be degraded because people are watching the World Cup final or whatever._

What about real time video communication? What about real time 3D environments, when they appear? Say real time VR becomes feasible, and a school wants to implement remote education for the underclass?

And that's just an example I came up with in 30s. Thinking that all important services that can be invented have been invented already is just silly.

 _Let 's put it this way, what would be the harm of having a common net neutrality law in Europe? Wouldn't that just mean that if you want to do business inside the EU you won't have to worry about being blackmailed by ISPs in 28 different countries?_

But do you already?

Laws are important, but they impose costs. Financial, administrative, and potentially worse. It's fine for us to say "No discrimination for traffic except for VoIP!", but a law will be a 30 page document that will still leave plenty for well paid lawyers to discuss and find loopholes over.

Trying to start a small ISP? Better make sure your QoS tables are exactly like the law says, or you might get shutdown if your competitors find out - <USER2> when was the last time you saw a law that could be objectively translated into a technical spec?

And then there's the fact that a EU-wide net legislation is a great opportunity for the lobbyists of the media industry to stick their clauses into.

And finally, EU officials have made some worrying claims about censorship "for the children!" for me to trust them with such things.

No, thanks. Let's talk again if and when it becomes a problem, not before.

</comment></comment></comment></comment></comment></comment><comment><USER15> Does anyone have more information on what qualifies a service to be "special" enough to qualify for the loophole? It's amazing to me that no qualifying factor is included in the legislation.

</comment><comment><USER11> Sorry guys but reality is that on a technical level some packets must be treated as higher priority. Think voice communication, for example. It must be prioritized over bulk data transfers. It is already built into LTE.

The future holds more use cases. Remote surgery, autonomous vehicles etc etc.

<comment><USER16> LTE voice communication is not a very good example, since it is provided by your mobile carrier, even with LTE. It's not an internet service.

I can't see how autonomous vehicles necessarily require any kind of priority access to the network. Remote surgery is a pipe dream - it has been discussed 20 years ago and nobody cares. It's certainly not worth destroying the network just for that.

<comment><USER17> LTE is a packet switching solution, so without traffic prioritization regular cellular voice calls could easily be trumped by a few heavy torrenters.

</comment></comment></comment><comment><USER18> This loophole will definitely not be abused. /s

</comment><comment><USER19> I can already see the future: IP packets with prioritization and billing information in their headers.</comment>


Scalability concepts: read-after-write consistency - <AUTHOR> https://avikdas.com/2020/04/13/scalability-concepts-read-after-write-consistency.html <comment><AUTHOR> I've learned a ton working at scale, and I wanted to share some of that knowledge with others.

To be honest, this is the type of knowledge I actually had to unlearn when I decided to try my hand at early-stage startups. Unfortunately, this is also the type of knowledge that's helpful to go from a startup to a big tech company, where you get the chance to learn this in the first place! Hopefully my writing can help bootstrap that knowledge for others.

</comment><comment><USER2> See Google's Spanner Database.

<comment><AUTHOR> Absolutely, it's something I researched while writing this article (since I knew someone would bring it up :)). Ultimately, I wanted to talk about concepts more than specific technologies, so I avoided going into detail on Spanner.

I don't claim to understand the underlying tech for Spanner, since I haven't read through the research paper fully. But my understanding is that write latencies are increased, which is a trade-off that has to be made. No matter what, _some_ trade-off is necessary, unfortunately.

<comment><USER2> Yes, "write latency" exists if the paxos group is down.

[https://storage.googleapis.com/pub-tools-public- publication-...](https://storage.googleapis.com/pub-tools-public-publication- data/pdf/45855.pdf)</comment></comment></comment>


How we ended up with microservices - <AUTHOR> http://philcalcado.com/2015/09/08/how_we_ended_up_with_microservices.html? <comment><USER2> Might be a clearer as "How Soundcloud ended up with microservices". I was expecting a piece on the general trend within the industry.

<comment><USER3> The title originally contained [about SoundCloud] at the end, which is the only reason I clicked the link (I'm a front end dev and frequent SC user). Not sure why it was removed.

<comment><USER4> Some HN mods are anal about not editing original titles, even when it makes for nonsensical HN titles.

</comment></comment></comment><comment><USER5> It's easy and obvious to set up queues and hand offs. It seems efficient and logical and it looks good on a whiteboard. The throughput of work in flight is of course amazing.

Then, much later, you realise latency sucks. People don't care that you have 100 features coming in 2 years. They care that you have the killer feature _now_.

So that's what counts here: _latency to customer value_.

The time between picking the next most valuable feature and putting it in front of paying customers, is the one loop that needs to be optimised.

I'm lucky to work Pivotal Labs. Our archetypal team is one product manager, one designer, two engineers from us and two engineers from the client. Currently we are experimenting with building teams with embedded data scientists as well. It works because we can, as an autonomous unit, do all the work that needs to be done to take a feature from ideation to production.

When you have that, it's amazing what you can get done.

I know I sound all-knowing and clever and stuff. But if I'd been setting up a software project before working at Labs, I'd have done a lot of the same things as this article describes as the starting point.

[edited to try and remove the impression that I would've Done It Perfectly From The Beginning, which is the opposite of what I was trying to say]

<comment><USER6> > Our archetypal team is one product manager, one designer, two engineers from > us and two engineers from the client.

This organizational system is also referred to as "squads"[1]. We also use it at DSC and it works remarkably well.

1\. [https://labs.spotify.com/2014/03/27/spotify-engineering- cult...](https://labs.spotify.com/2014/03/27/spotify-engineering-culture- part-1/)

<comment><USER7> I think "The Mythical Man Month" (1975) calls it a "surgical team". If this practice works, I wonder why it isn't common?

Just postulating here, is it because most established businesses cling to whatever they currently do, and the next generation get their start at those big companies, and copy what they're used to? Or, is it because the people who have the responsibility for organizing teams do not come from a technical background, so miss out on reading material that discusses this?

<comment><USER8> Could partly be because you can't have any B players on a squad.

<comment><USER4> That's not what TMM advocates for surgical teams. IIRC, the surgical team is an A/A+ lead, and a few directee/assistants, some highly specialised (TMM advocated for each team to have a designated tool-maker responsible for creating and maintaining the team's tooling like scripts or codegen)

</comment></comment></comment></comment></comment><comment><USER9> Great read.

I found it interesting that they basically replicated the XP playbook: cross- functional teams, continuous code review with pairing, collective team ownership of code and results, bounded contexts.

It makes me wonder if the "microservice" part of it was necessary. What if they had produced "microlibraries" rather than "microservices?"

<comment><USER10> >It makes me wonder if the "microservice" part of it was necessary. What if they had produced "microlibraries" rather than "microservices?"

I've wondered that every single time somebody has touted the benefits of microservices.

The people who have really 'succeeded' in it seem to conflate the benefits of looser coupling between dependent software systems (which is always a good thing) and making those systems talk to one another over a network socket (which isn't necessarily a good thing).

<comment><USER11> The benefits of microservices _aren 't_ architectural. The point re. library APIs vs services APIs is completely correct, because that's actually not what the microservice style addresses.

The point is rather organizational/operational - Being able to independently deploy, scale, monitor and manage different parts of the system.

That allows you to distribute ownership of the system, and also allows partitioning, so that a failure in one component need not effect the whole system.

<comment><USER12> Could you give an example of an architectural benefit vs operational benefit?

<comment><USER11> When I say 'architectural' there I mean that as in the underlying design. How the system is modeled and and what different objects and interfaces exist in that model.

Eg. Say you have a billing component in your system. You may have the same underlying billing component in either a monolith or as microservices. It may be a nicely decoupled in both cases (either existing as an independent module/library in the monolith case, or as an independent service in the microservices case.)

The (potential) benefit that's afforded from the microservices approach is not that there's a better underlying design there, but that a single team or developer can properly take ownership of deploying, scaling, monitoring and managing the service, independently of the rest of the application. (Plus scope for partitioning and graceful partial failures vs whole system failures)

</comment></comment></comment></comment></comment><comment><USER13> I've never been convinced about splitting developers into front and back end.

I see no reason why a decent developer can't build the front and the back end for each piece of new functionality. This removes all the communication issues.

<comment><USER14> Maybe on a small line of business app.

On a big, widely used consumer facing app however, front end developers really come into their own.

They have cross browser, cross device responsive design issues to contend with. Accessibility and graceful degredation. Performant JavaScript. They also need design skills.

Very few people have deep skills in back end and front end.

<comment><USER10> I've seen plenty of projects with separate front end and back end developers fuck up because the front end developers try to solve "back end problems" on the front end and vice versa.

This is the cause of an awful lot of technical debt, and "big, widely used consumer facing app" is the natural habitat of large technical debts.

<comment><USER15> I vividly remember a case where a really simple page in a system had become incredibly slow - on asking what had changed I was told "all we did was add a count of the number of users".

Turns out the back end (they were micro-services - even though the term hadn't been invented at that point) didn't have a call for "total number of users" so the front end code (this was server side HTML generation - 10+ years ago) was getting a list of _every_ user in the system and iterating over all of them in chunks and counting.

<comment><USER16> Probably some last minute feature that was forced with a same day deadline. If I had a dollar for each of those I'd be able to retire.

</comment></comment></comment></comment></comment><comment><USER17> The Reddit post has some pretty good comments too:

[https://www.reddit.com/r/programming/comments/3k8sb2/how_we_...](https://www.reddit.com/r/programming/comments/3k8sb2/how_we_ended_up_with_microservices/)

</comment><comment><USER18> This is really great. The micro services debate has just kicked off on my teams, and people are starting to make their cases. While the details of the article differ from our systems (less web, for one thing) the problems they experienced sound pretty familiar to me right now.

I'll be sending this around the office tomorrow.

</comment><comment><USER19> Ok, so when are you going to fix search though?

There is not way to search by tags & filter by length.

Consider this query: [https://soundcloud.com/search/sounds?q=*&filter.duration=epi...](https://soundcloud.com/search/sounds?q=*&filter.duration=epic&filter.genre_or_tag=dub%20techno)

It includes shows tracks that are clearly short.

Alternative is this: [https://soundcloud.com/search/sounds?q=dub%20techno&filter.d...](https://soundcloud.com/search/sounds?q=dub%20techno&filter.duration=epic&filter.genre_or_tag=dub%20techno)

But then not all dub techno tracks have it in the title.

Also, default grid view when viewing your likes is super annoying. Any chances to change into list?

<comment><USER20> The guy is no longer working for soundcloud.

<comment><USER19> Hope he can relay though, esp. if he is in a good relation with them.</comment></comment></comment>


Your terminal is not a terminal: An Introduction to Streams - <AUTHOR> https://lucasfcosta.com/2019/04/07/streams-introduction.html <comment><USER3> This is a good article, but this bit irks me:

      $ printf "Look, I have something inside" > /tmp/careful.txt
      $ cat /tmp/careful.txt
      Look, I have something inside
      $ printf "\nNow I have one more thing" >> /tmp/careful.txt
      $ cat /tmp/careful.txt
      Look, I have something inside
      Now I have one more thing
      $

What really happens is:

      $ printf "Look, I have something inside" > /tmp/careful.txt
      $ cat /tmp/careful.txt
      Look, I have something inside$ printf "\nNow I have one more thing" >> /tmp/careful.txt
      $ cat /tmp/careful.txt
      Look, I have something inside
      Now I have one more thing$

And what you want is:

      $ printf "Look, I have something inside\n" > /tmp/careful.txt
      $ cat /tmp/careful.txt
      Look, I have something inside
      $ printf "Now I have one more thing\n" >> /tmp/careful.txt
      $ cat /tmp/careful.txt
      Look, I have something inside
      Now I have one more thing
      $

Newlines are line terminators, not separators. This article missed what could have been a really good section on line buffering that could have explained some of the merit of having line terminators as opposed to line separators.

<comment><USER4> One of the reasons that I like zsh,

      $ printf "Look, I have something inside" > /tmp/careful.txt
      $ cat /tmp/careful.txt
      Look, I have something inside%
      $ printf "\nNow I have one more thing" >> /tmp/careful.txt
      $ cat /tmp/careful.txt
      Look, I have something inside
      Now I have one more thing%
      $

Zsh will add the line break and mark its absence with a % with reverse video).

I follow the "terminator-not-separator" rule religiously in my own source code, but my impression is that this is somewhat a Unix-ism.

<comment><USER3> First of all, that's a hack zsh really shouldn't have to do. It's not like it can see the output of the commands that it runs. That's the job of the terminal. To make that '%' effect work, when a command it's waiting for dies, it outputs a '%' and then as many spaces as it knows there are columns in the terminal and then a carriage return '\r'. So it uses line-wrapping to move to the next line if there's content in the current one. That's really out of scope of what a shell should be doing. It also will not work if you're not working with the shell through a terminal, but something else like a tcp connection or a serial device. What's going to happen then is that that last "line" is going to disappear in zsh. The prompt will be rendered on top of it.

Anyway, it's not just about the shell prompt. Things are just simpler and more composable when using it as a terminator instead of a separator.

Consider programs that might every now and then have to warn you about errors, like this:

      cmd() {
        >&2 printf "%s-error-1" "$1"
        sleep 1
        >&2 printf "\n%s-error-2" "$1"
      }

and them being used in pipe:

      cmd foo | cmd bar

using newlines as separators causes them to merge lines like so:

      foo-error-1bar-error-1
      foo-error-2
      bar-error-2%

if one used line terminators:

      cmd() {
        >&2 printf "%s-error-1\n" "$1"
        sleep 1
        >&2 printf "%s-error-2\n" "$1"
      }

then the output would be like so:

      foo-error-1
      bar-error-1
      foo-error-2
      bar-error-2

Now consider what would happen if you need to conditionalize the first line:

      cmd() {
        if (( $RANDOM % 2 == 0 )); then
          >&2 printf "%s-error-1" "$1"
        fi
        sleep 1
        >&2 printf "\n%s-error-2" "$1"
      }

Now, if that first line doesn't output anything, you'll get a extra, blank line. So, you need to include the newline in the conditional body:

      cmd() {
        if (( $RANDOM % 2 == 0 )); then
          >&2 printf "%s-error-1" "$1"
          >&2 printf "\n"
        fi
        sleep 1
        >&2 printf "%s-error-2" "$1"
      }

Now what if you need to conditionalize the last line?

      cmd() {
        if (( $RANDOM % 2 == 0 )); then
          >&2 printf "%s-error-1" "$1"
          >&2 printf "\n"
        fi
        sleep 1
        if (( $RANDOM % 2 == 0 )); then
          >&2 printf "%s-error-2" "$1"
        end
      }

should the newline go in the first or second conditional body? If it's on the first, then if the first condition is true and the second is false you'll get a trailing newline (something you don't want if you're trying to have separators).

What's the solution here? The newline will have to be added only when BOTH conditional bodies run:

      cmd() {
        if (( $RANDOM % 2 == 0 )); then
          first_was_true=1
          >&2 printf "%s-error-1" "$1"
        fi
        sleep 1
        if (( $RANDOM % 2 == 0 )); then
          if [[ $first_was_true ]]; then
            >&2 printf "\n"
          fi
          >&2 printf "%s-error-2" "$1"
        end
      }

What if you have a 3rd conditional error? Well, that's a new newline that will have to be output iff the second and third ran or the first and third ran:

      cmd() {
        if (( $RANDOM % 2 == 0 )); then
          first_was_true=1
          >&2 printf "%s-error-1" "$1"
        fi
        sleep 1
        if (( $RANDOM % 2 == 0 )); then
          second_was_true=1
          if [[ $first_was_true ]]; then
            >&2 printf "\n"
          fi
          >&2 printf "%s-error-2" "$1"
        end
        sleep 1
        if (( $RANDOM % 2 == 0 )); then
          if [[ $first_was_true || $second_was_true ]]; then
            >&2 printf "\n"
          fi
          >&2 printf "%s-error-2" "$1"
        end
      }

Can you see how this quickly becomes unwieldy? Each new body becomes coupled to the ones before it. The third conditional needs to know about the first and second when semantically it shouldn't need to care at all! If you use line terminators, things are much simpler:

      cmd() {
        if (( $RANDOM % 2 == 0 )); then
          >&2 printf "%s-error-1\n" "$1"
        fi
        sleep 1
        if (( $RANDOM % 2 == 0 )); then
          >&2 printf "%s-error-2\n" "$1"
        fi
        sleep 1
        if (( $RANDOM % 2 == 0 )); then
          >&2 printf "%s-error-3\n" "$1"
        fi
      }

This isn't just an arbitrary Unix decision. It's objectively good design.

<comment><USER5> When a job completes, it should end with a newline (if it printed anything). That's good UX. But you _cannot mandate this_ for two big reasons:

* You can't enforce this at the process level, because you may want to use multiple process invocations to build up a single line of text, or want to accept input after printing a prompt. So this logic has to live at the job level.

* Programs that print data that exist somewhere else should not be adding unexpected newlines. `cat foo > bar` should produce an identical copy of foo, not a copy with a newline added.

Since this has to happen at the job level, the only place this can be done is in the shell. Ideally there'd be some way to coordinate with the terminal emulator to say "if I'm not at the beginning of the line already, please add some indication of this and move the cursor to a new line", which not only means the terminal is responsible for the indicator instead of the shell, but it also enables smart behavior like being able to copy the output without including the line terminator indicator

Unfortunately, terminal emulators don't have a way to do this, so the shell has to hack it together.

<comment><USER3> I'm not sure what your stance is.

If it's "individual programs should use line terminators as convention and if the shell can help improve UX in the face of programs that break convention then it should", then I agree with you. What I meant by how zsh shouldn't be doing this is that if programs and people working on the command line followed the convention of terminating newlines, then the shell wouldn't need to do that.

If it's "individual programs should use line separators and the shell should add the terminating newline", then I disagree. I don't think shells and terminals are so fundamental that this behavior should depend on them. It also does nothing to address the example of `cmd foo | cmd bar` that I gave because that's 1 job.

<comment><USER5> It's, as I said, that programs should use line terminators, but we cannot mandate that all programs do use line terminators, so it's still the responsibility of the shell to deal with this. Specifically, it's a reaction to

> _First of all, that 's a hack zsh really shouldn't have to do._

In an ideal world, zsh wouldn't need a hack, but it would still need to send a signal to the terminal emulator to say "please ensure the cursor is at the beginning of a line; if it's not, mark this somehow so the user understands". Slightly less ideally, a simple way to say "am I at the beginning of a line?" (this is less ideal only in that now the terminal can't be smart about copying program output). But either way, it's still the responsibility of the shell to ensure its prompt ends up starting at column 0 without overwriting the previous program's output. Because no matter what conventions you establish, there will be programs that exit without printing a line terminator (even the best-behaved of programs can still do this if they use unbuffered output and you send a signal to kill the process in the middle of a line).

</comment></comment></comment></comment></comment></comment><comment><USER6> Was expecting this to be about [https://en.wikipedia.org/wiki/STREAMS](https://en.wikipedia.org/wiki/STREAMS)

<comment><USER7> I wasn't aware of the issues behind SysV STREAMS. Thank heavens that Alan Cox flat out refused.

[http://www.groklaw.net/articlebasic.php?story=20060703095643...](http://www.groklaw.net/articlebasic.php?story=2006070309564374)

<comment><USER8> The original idea behind streams is decidedly more Unix-like than the whole BSD sockets hack. Also it is only partially relevant to networking, conceptually it is generalization of the concept of line discipline, IIRC the original implementation of PTYs in SysV was layered on top of streams. Well, then there was X/Open and another standardization bodies who though that ISO OSI stack is a good thing (you have to be standards body to be capable of believing that) standardized STREAMS (note the capitalization) and build ISO OSI API in terms of that, complete with userspace layer in between that hides all the complexity in order for the thing to be actually usable.

</comment></comment></comment><comment><USER9> This is interesting to compare to the 3270 terminals in mainframes. On those the terminal inherently understands labels and fields, and typing doesn't necessarily go to the host until you submit. It's closer to a web browser than a Unix style terminal.

<comment><USER10> When I was working on one open source terminal library, I wanted to get a 3270 family terminal (with appropriate bridge for TCP/IP or RS232), just to look for ideas that were lost to the Unix and other communities I already know.

I understand some other, more generic serial terminals, like were used on Unix, also had block modes (local editing of screen regions), but I'm not aware of Unix software taking advantage of that.

</comment></comment><comment><USER11> Awesome article!

Handy maximum-portability redirection cheatsheet (write, append, discard):

STDOUT (no number needed for file descriptor)

      command >stdout.txt
      command >>stdout.txt
      command >/dev/null

STDERR (2 is required file descriptor)

      command 2>stderr.txt
      command 2>>stderr.txt # Append errors to file
      command 2>/dev/null # Discard stderr

STDOUT + STDERR (separate destination)

      command >stdout.txt 2>stderr.txt
      command >>stdout.txt 2>>stderr.txt

STDOUT + STDERR (duplicate destination, order matters)

      command >shared-output.txt 2>&1 #stderr follows stdout
      command 2>shared-output.txt >&2 #stdout follows stderr
      command >>shared-output.txt 2>&1 #append

Practical example:

      npm install >npm-output.txt 2>npm-errors.txt #two destinations
      npm install >/dev/null 2>&1 #all output discarded

</comment><comment><USER12> [https://www.linusakesson.net/programming/tty/](https://www.linusakesson.net/programming/tty/) mentioned at the bottom of the article is a must read as well.

</comment><comment><USER13> Recently I broke my terminal emulator's config while I didn't have an internet connection so I had to use a virtual terminal to fix it. It made me wonder, does anybody use virtual terminals as their daily drivers?

<comment><USER3> As in not starting an X server? No.

As in most of the time the window focus is on a terminal? Yes.

In my computer, 90% of the time there are only 2 types of windows open. 1 is the web browser, and the other is the terminal. Sometimes I open up Emacs to read my mail or write some notes in org-mode, but my text editor of choice is vim in a terminal, not even gvim.

<comment><USER13> Not that (which is similar to my setup), I mean tty[0-6] which come up when you press ctrl-alt-F[1-6]

<comment><USER3> For a daily driver... it's hard to imagine the conditions where that would be better than using X. Maybe if one's stuck with a machine so old (like 90's old) where even light window managers lag, then maybe. You normally wouldn't install X on servers or raspberry-pi's or other purpose built machines like routers, but you also wouldn't use them as a daily driver. I imagine even people that need to work on those extensively would rather log-in via ssh from their real daily-driver.

</comment></comment></comment></comment><comment><USER10> The article doesn't get into the few ways that terminals are more than normal stdio streams, but it turns out you can do a lot without knowing any of that. For one terminal library I made, I was able to avoid any extra native code in the process, using only unbuffered streams and a kludgey call out to `/bin/stty` to set raw input on that end. (Next time, I'd probably just do FFI to native code.) [https://www.neilvandyke.org/racket/charterm/](https://www.neilvandyke.org/racket/charterm/)

</comment><comment><USER14> Does an OS/terminal/shell exist that allows one to apply a different interpretation to streams than as a stream of characters? For example JSON objects, or even images?

Not sure how useful it would be, but it could look like this:

        $ cat car.png | grep "traffic-light"
        Illegal operation: stream is of type image

        $ cat car.png
        (image appears)

<comment><USER15> Powershell does pipelining with streams of objects instead of only characters.

<comment><USER16> Unfortunately it comes with risks. With byte streams you can have an arbitrary max buffer size after which the program has to wait to write. With objects you can run out of memory trying to pass everything in a single object. I hit this trying to generate large CSVs in powershell.

</comment></comment></comment><comment><USER17> Recently I needed to multiplex stdin to both stdout and stderr and was surprised that none of the coreutils tools supported that. The closest is `tee /dev/stderr` which only works on system where stderr is bound to that virtual device. It seems such a basic thing to do when you want to debug a pipe.

</comment><comment><USER18> This is a great article and very beginner friendly as well. While I didn't learn anything new from this, it helped me organize the concepts in a way that would be much easier to communicate with someone else.

Thanks for posting. I bookmarked this incase I ever need to explain streams to someone.

</comment><comment><USER19> I can see some Monads in his examples. One particular example being [https://lucasfcosta.com/assets/fortune-and-cowsay- streams.pn...](https://lucasfcosta.com/assets/fortune-and-cowsay-streams.png)

</comment><comment><USER20> So `cat /tmp/hello.txt` does not read stdin from `/tmp/hello.txt`? Why does `cat /tmp/hello.txt` set stdout to `/tmp/hello.txt` this is confusing

Also his example ` exec 5<> /tmp/lines.txt` fails with ambiguous redirect.

<comment><USER21> > _So `cat /tmp/hello.txt` does not read stdin from `/tmp/hello.txt`? Why does > `cat /tmp/hello.txt` set stdout to `/tmp/hello.txt` this is confusing_

While stdin, out and err are files, they're not really the same thing as a /tmp/hello.txt type file you're thinking of. They're file system objects with special permissions. And while the idea is you'd use normal file read / writing APIs on them they're really more like FIFO buckets (technically it's a symlink to a file - for reasons that will make more sense shortly)

Take the example:

        cat /tmp/hello.txt | grep world

(ignoring for the moment the "abuse of cat" for the sake of this example)

So what would happen there `cat` would write to this bucket (the location of it varies from one OS to another) and another process in the pipe (eg `grep`) would read from that same bucket. But from the perspective of `cat` it's STDOUT and from the perspective of `grep` it's STDIN - even though they're the same file.

Things get a little more complicated when you start talking about redirection but essentially because each programs STDIN, STDOUT and STDERR file is actually just a symlink, when a program starts it's symlink points to another file instead of the normal one that would be next (or previous) in the pipeline. However this is all done automatically by the OS when you make the syscalls.

Things get even more complicated when you start talking about stuff that is read from or written to your terminal because they will be slightly different file system objects again but with a bit set to identify them as a pseudo-TTY. It's not really worth getting into the mechanics of it here. However ostensibly the principle is the same as the above.

edit: rewritten for clarity

<comment><USER20> I see, and thank you for the clarification. My main problem then is the blog post for the redirection section is poorly written and plain wrong with regards to the examples and pictures.

Their example is $ cat /tmp/root_content.txt

And then in the picture the show that the `cat` program's stdout is `/tmp/root_content.txt`, which is incorrect according to your description. This would only be correct if it was then redirected or piped.

<comment><USER21> Their description was correct. What they missed off the /tmp/root_content.txt example was that the shell would create that file first, then launch the program pointing it's STDOUT to that file.

Because those streams are file system objects that use the same reading and writing APIs as regular files, it means you can also substitute them for regular files too. Which is what's happening in that specific example.

Essentially UNIX supports a whole array of special files that might behave slightly differently in the background but all can be interfaced with using the same principles as any regular file. It's quite a clever solution - <USER2> not without it's problems, kludges and mistakes too.

</comment></comment></comment></comment><comment><USER22> I thought this was something related to [https://github.com/matz/streem](https://github.com/matz/streem)

</comment><comment><USER23> great stuff! minor editing note: the first time > is mentioned there is no 1 with it. a bit further down you say "we can drop the 1 from >".

</comment><comment><USER24> The cat example is missing an arrow. Clearly, you see everything twice. So somehow the input also goes around cat and directly to the output.

</comment><comment><USER25> I've shared this with my team, it's super useful as an introduction and reference to terminals for a wide range of folks.

</comment><comment><USER26> There are too many mistakes and typos in this to recommend it. Better if you go straight to the end and click on "TTY demystified" and the other links he recommends.

</comment><comment><USER27> I'm frankly tired of the less is more, simple > complex fad. It's understandable this mindset is so trendy: it's easier.

And then what ? Showcase another pointless note taking app ?

<comment><USER28> The idea is not to be driven by simplicity _per se_ , but by composability. Build software wherein you take it for granted that it will be used in many contexts, and that it is only ever able to solve 'part' of the problem because other parts may not yet be understood.

If you're not focussed on building simple systems, then you're unlikely to be able to compose them, and if you can't compose them, they will likely become obsolete quickly. So the goal is really to avoid having to implement the same things over and over again.

<comment><USER27> Yes sorry, I totally agree with this and I am digressing. I'm reacting to the first words of the article:

> I always try to build less software. Less software means you have to spend > less time updating it, less time fixing it, and less time thinking about it. > The only thing better than "less software" is no software at all.

The problem is that I have been harassed on this very basis (i.e. that of a __philosophically __stated point of view) by people that are less able than me and this lead to writing over and over again the same dumb code for 15,000 lines while it could have been kept under 1000 (in fact I did it) (and as you may have already guessed it, this is resentment speaking here).

Actually you don't make things simpler, either you move complexity around, often concentrating it somewhere to alleviate its weight on other parts of the system, and they can then be described in shorter terms, enhancing the pace at which you can tweak them – or you get rid of that complexity, moving it out of the code, i.e. delegating it to future implementations or to manual maintenance support tasks.

This is the bit that is problematic to me: > [you have to spend] less time thinking about it

Absolutely not. In some parts yes, definitively, but some other parts will be incredibly dense, and will require a very slow reading pace. When you want to change such code, you have to sit in front of your screen, thinking hard about what happens for 30 minutes, and when you get it it's a 5 lines change. Maybe the author doesn't consider this in this article because he doesn't have to deal with implementing these piping heuristics, he already has those of bash.

Now why am I pulling the cow's horn like this ? Because this also encourages people into __simpler thinking __and they mix this up with whatever noble ideal simplicity actually refers to, while, as Hickey pointed out, it 's a lot more about the tradeoff that populate the quadruality between easy/simple vs complex/complicated. Nonetheless, here they come, shitting on my tree- traversal – not a simple thing, not in their realm, and here they are, "flattening" it in a 200 files long pull request.

As for how software should be done, err ... I think I never heard what I'd say, i.e. that software should be developed by programmers or in other words teammates, not pupils nor slaves, and that these people, as thinking beings, shouldn't be treated as things that conform and must be, quite to the contrary, they should let be.

</comment></comment></comment><comment><USER29> What an attention grabbing first line but ultimately contradictory. If you hate software you hate streams because streams are software.

Guess what? I hate computers. If you do less streaming across computers and put everything in a monolithic code base you need less computers. Is what I just said pointless, obvious and attention grabby? well so is the first couple lines of this article.</comment>


FVWM3 - <AUTHOR> https://github.com/fvwmorg/fvwm3 <comment><USER2> Wow that this is still being developed. I used this back in '93 on a sequent Unix box. It's nice but a bit barebones by today's standards. Having to edit config files just to add items to a menu.. cool though!!

<comment><USER3> You'd be surprised at just how popular this is still. I think people appreciate the flexibility, even if that means things being on their terms.

Plenty of applications still operate with text files (just look at emacs/vim) and the people that use those have no problems editing text files.</comment></comment>


Simplicity Before Generality, Use Before Reuse - <AUTHOR> https://medium.com/@kevlinhenney/simplicity-before-generality-use-before-reuse-722a8f967eb9 <comment><USER2> > People do not on the whole pay for — or need — generality: they tend to have > a specific situation, and it is a solution to that specific situation that > has value

Yeah, but otoh, most progress in science and tech has been made by _sort-of- tricking_ people into _paying for stuff that solve a slightly more interesting /general problem than the original boring problem they wanted solved._

So at least when it doesn't drown everyone in paralyzing complexity, we need to remember to keep that "hacky rascality" alive and _slightly_ "screw" our clients at the expense of developing some general and reusable solutions (hopefully as open-source libraries :P) that will probably matter more than whatever "important" problems they want solving ;)

<comment><USER3> "slightly more interesting/general problem"

I'd say there's no great sin in solving a _slightly_ more general problem than what you currently face. In fact, that's often net simpler almost immediately; it's not long before a function to format a URL is simpler than simply manipulating a URL every time. (Even ignoring the simpler issues.) The problem arises when you have something like "Let's see... I need a database... _let 's write an ORM_." Whoops. (Speaking from harsh experience there.)

</comment></comment><comment><USER4> This article really hits home in terms of how people sometimes think about React components, and how I'd like them to think about them.

I sometimes hear, from other engineers working on a complex React app, that the point of React's components is that every component is written to be "reusable." Every component. This is used as a justification for "going general" immediately when writing a component. In reality, there is real work to make a one-off component into a reusable component, and that work can only be done in the context of having (or at least realistically imagining, not dreaming up as an exercise) multiple use sites.

What React components actually provide are the same things that modules and functions provide: encapsulation; limited dependencies; a calling convention. When you write a function, you try to keep the arguments to a minimum; you might say it makes the function more "reusable," but there is still a big difference between a library function that is _designed_ to be used from anywhere, and a private helper function that is not. React components are "reusable" the way functions and modules are reusable. A better word might be "usable." :) You can reason about the use, even if there is only one use.

Making something "reusable" means making it general, and we've all had the experience that the article points to of taking some code that was intended to be general, but in fact is used in only one place, and remaking it so that it's actually possible to use in a second place.

That said, I have many times in programming succeeded in boiling something down to its essence in advance, and writing the general thing first, followed by the specific thing. I suspect this is actually more about simplicity than generality, however. I'm thinking of cases of "bottom-up" development where you write the primitives and then the higher-level parts in terms of the primitives. Actually, maybe it is about generality too, and when you are testing out your primitives in different combinations, you are doing the work required for good generality (seeing lots of different cases and basing your decisions on that).

</comment><comment><USER5> This is especially true for libraries. A minimal, simple library is much better than a sprawling, "convenient", but more complex library.

<comment><USER6> Or -God forbid- framework.

</comment></comment><comment><USER7> Sorry for ranting. I've been reading articles like this one for many years now, and I've certainly learnt a lot through them. Lately though, it feels as though we as a profession are just going around in circles, not making any real progress.

More and more, these articles remind me of schoolyard wisdom, the kind people love to throw around because they sound wise. _" Winners never quit."_ " _Quitters never win "_. " _Quit while you 're ahead"_. Taken individually, they all sound perfectly reasonable and wise. Put them together, and clearly there's something wrong.

This article strikes me in the same vein. " _YAGNI. Keep your code as simple as possible. Focus on your code as it is right now, and not what 's going to happen in the future"_. " _The only constant is change. Business requirements are always in flux. The mark of good code is the ability to change its behavior without overhauling everything "_. Taken individually, they both sound perfectly reasonable, but put them together, and something is clearly wrong.

Maybe it's time we accepted that there is no one right way. No one all- important metric or rule that trumps everything else. That simplicity is important, that conciseness is important, that DRY is important, that being able to adapt to future changes is important. That good code finds a balance between all of the above, without sacrificing one on the altar of another. That instead of writing articles trying to elevate one over another by attacking strawmen, we should instead present realistic code examples, thorough which we can have a grounded discussion on how best to find the balance between all the different qualities that comprise good code.

It's not going to be as easy or fun as simply declaring _" X considered harmful"_, but it's going to help elevate the quality of our discussion, and help our profession progress to the next level.

<comment><USER6> Id say that this article is trying to be too general and give advice for a too broad case. It would be better to focus on real world cases, solve them and try to find some slightly generalised advice </meta>

</comment></comment><comment><USER8> I'd like to take this opportunity to flesh out some ideas I've had regarding the difference between the concepts "generic" and "general". "Oh, I think we need the ability to do this kind of thing someday" falls under "general", giving more and more functionality to a component. "Oh, there are several things about this piece that we don't actually need to know" falls under "generic", allowing the code to generally be simpler. In Haskell

        traverse :: (Traversable t, Applicative f) => (a -> f b) -> t a -> f (t b)

is more _generic_ than

        traverse :: Int -> IO Float -> [Int] -> IO [Float]

and therefore strictly better (unless there are reasons the latter could be optimized but not the former).

<comment><USER9> They are definitely some downsides to the first definition. For instance, the error messages will often be much harder to read. For the second one you might get something straight forward like it expected `[Int]` but the actual type was `Int`. For the first one you will probably get something about not being able to deduce some long constraint for some long type. That might be possible to improve with better tooling, but for now it is a very real downside.

I think that especially for application code (as opposed to library code), it is a mistake to make your type signature far more generic than you need right away, especially because it is usually easy to switch to the more generic type declaration later without changing much existing code.

<comment><USER8> This is a good point. I think there's some merit in doing

        traverse :: (Traversable t, Applicative f) => (a -> f b) -> t a -> f (t b)

        mySpecialTraverse :: Int -> IO Float -> [Int] -> IO [Float]
        mySpecialTraverse = traverse

in some instances.

</comment></comment></comment><comment><USER10> This really hits home:

> _How much of what software engineers now do is still devoted to the > accidental, as opposed to the essential?_

Far, far too much

</comment><comment><USER11> So true. Premature generality is a big problem.

<comment><USER12> IMHO this is one of those things that only experience can help with. Nobody wants to be the guy adopting a single-use codebase to a somewhat different problem and having to rewrite half of it, but overgeneralizing from the start leads to complex and bloated programs that are hard to deploy anywhere--where the effort to deploy it is about half as much as it would have required to write a custom solution from scratch...

My personal philosophy is to keep the dependencies to a minimum and make judgement calls on simplicity versus generality. I probably err on the side of simplicity a little too often, but I also get asked to deploy solutions on non-internet connected devices fairly often. Solutions that you can deploy out of a tarball on a base CentOS 5.5 machine are good sometimes.

</comment></comment><comment><USER13> Great article! We need more "practical rational knowledge" infused arguments in a sea of over-engineered "download and configure" nightmares!</comment>


World's oldest blogger makes final post - <AUTHOR> http://www.theage.com.au/news/web/worlds-oldest-blogger-makes-final-post/2008/07/14/1215887490954.html R.I.P Olive <comment><USER2> I find it very compelling that this otherwise ordinary lady's words may very well be with us _forever_. (Or at least so long as our society lasts.)

I sometimes find myself wishing more older people would start blogging so that what knowlege they carry will not be lost.

</comment><comment><AUTHOR> R.I.P Olive</comment>


Three of the top Anonymous leaders allegedly captured in Spain - <AUTHOR> http://www.cnn.com/2011/BUSINESS/06/10/spain.hackers/index.html

<comment><USER2> "The suspects encoded their communications and penetrated nearby secure Wifi networks, since two of the suspects did not even have an Internet connection at home, the police said."

Spain probably has a pretty rigorous internet/telephony monitoring aparatus as a result of it's internal terrorism problems, but I'm curious how they were tracked down after the wifi network was located. Did the police just search all nearby properties? Is there hardware that can locate the source of individual wifi broadcasts?

<comment><USER3> Any law enforcement authority is likely to have hardware for triangulating radio signals.

</comment></comment><comment><USER4> Anonymous has no leaders. Anonymous is not a hacker network. It may be that some people who belong to hacker networks use Anonymous as their name, but that doesn't involve Anonymous as a whole.

The reason Anonymous is called Anonymous, what makes it different from another human group is that anyone can be part of it, anyone can claim to represent it.

<comment><AUTHOR> That's the theory, but in reality there are likely members that are more influential than others. Not official leaders perhaps, but a form of leader nonetheless.

<comment><USER5> There are!

But Anonymous is kind of disjoint. It's feudal. You'll realize this if you try to hop on to one of the IRC networks -- there are a million different little channels each representing its own fiefdom. They mostly have some common beliefs, and they all fly the same banner -- but when one group rises to prominence they will often start doing things under their own name. Some recent examples include LulzSec and Goatse Security. There are also the people who run the websites and the IRC channels and who make the DDoS tools and similar skiddie stuff; these guys are by any measure the _real_ "leaders" of the operation, but they prefer to sit on the sidelines and don't usually get their hands dirty. The argument could be made, for example, that Julian Assange is an unwitting leader of Anonymous.

It's an emergent structure, not an intentional one. This is key to its resilience.

I used to post regularly on /i/ a few years ago; it grew into the "Anonymous" of today after the website owner shut it down. There was a Fox News report about the "Anonymous" threat way back in 2007; the "informant" mentioned in the report is Alex Wuori, a self-proclaimed "defector", and I'm not exactly breaking his privacy because literally everyone knows who he is and what he did. The Fox News report centered on the /i/ board of the time.

The other thing about Anonymous is that it is, as you may have noticed, rather old, in Internet terms. The first true Anonymous raid was YTMND vs. Ebaums back in 2006, in which 4chan and 7chan ran DDoS attacks against ebaumsworld.com in retaliation for stolen content.

I don't do any of that anymore, in case anyone's wondering. Being 17 only lasts one year.

</comment></comment></comment><comment><USER6> They've already been released with charges. One of them hosted the IRC server in his own house.

They were not involved in the Playstation network attack but on national Spanish attacks to the Minister of Culture and some other local websites.

He, at least these last two weeks between the cucumbers crisis, and now this misunderstanding on the PSN, the focus is diverted from bonds speculation :)

</comment><comment><USER7> > a computer server that was used to coordinate and execute the attacks

lol, they arrested a kid with a laptop running LOIC. Nothing to see here, move along.

</comment><comment><USER8> I'm waiting for their response (Anonymous), it's usually comical and entertaining.

</comment><comment><USER9> You can't just arrest the leaders of Anonymous, because it has no leaders, this is the whole point of it.

These goverment people fight back the only way they know: propaganda.

</comment><comment><USER10> I think the term "leaders" is used very loosely here since last I checked they weren't an organisation with "leaders"

I think they're trying to put the blame on anonymous. They found this and thought, we'll blame it on them:

"police found a large quantity of software programs specifically designed to infect third-party computers. The suspects encoded their communications and penetrated nearby secure Wifi networks, since two of the suspects did not even have an Internet connection at home, the police said."

</comment><comment><USER11> Lol. They think they know what Anonymous is. Lawfags can't even triforce, etc etc....

<comment><USER11> Minus points for satire? It's a good job HN doesn't run a country.

<comment><USER12> >Minus points for satire?

Yep. Be useful or move on.

<comment><USER11> Satire isn't useful? Are you for real? How useful can someone's opinion on the Internet be exactly? Is there a scale of usefulness on which satire ranks near the bottom? I doubt it. You move on, the world need less of you. Pedants.

<comment><USER12> Substantive exchange of knowledge especially that which is based on experience is the most favored kind of post here.

Satire, at least of the internet variety, rarely has anything to contribute except in the form of noise.

I'm not a pedant, I'm helping you understand the people here. Your ungrateful and angry attitude only proves my point.

I've been here for nearly three years now, I'm not going anywhere and neither is the rest of community which has the values I've just described.

You might enjoy reddit.com/r/programming more, they seem to value humor more.</comment></comment></comment></comment></comment>


Finland makes broadband a 'legal right' - <AUTHOR> http://news.bbc.co.uk/2/hi/technology/10461048.stm <comment><USER2> Great intentions - completely wrong way to make it happen.

Regulating the broadband internet market will just make people in metropolitan areas subsidize the cost for people in rural areas. If it ended there...sure it wouldn't be fair, but at least it would be a zero sum game (some win, some equally lose). But with regulation that's never the case because:

1\. you introduce additional costs for government oversight

2\. you discourage new entrants to the market, thereby reducing competition

As Milton Friedman said, you can never judge a government program by it's intentions, only the results it gets.

Free markets and unbridled competition have a much better track record of driving prices down, making new technologies available to the masses.

Cost of flat panel TV's (unregulated) have cut in half every 18 months since they came out. What was once a toy for the rich is now available even to poor people.

Cost of health care (regulated, and increasingly a technology product) has steadily increased each year.

<comment><USER3> _Free markets and unbridled competition have a much better track record of driving prices down, making new technologies available to the masses._

In industries with a high barrier to entry, the market naturally tends away from competition. Broadband is one such industry, due to the enormous initial infrastructure costs.

<comment><USER4> I don't think so, if you look at the united states the broadband industry is heavily regulated.

<comment><USER5> That's partly because it's nearly impossible to build a telecommunications network in the private sector, due to the fact that you don't have eminent domain power, can't dig up roads, etc. And if we're going to have the government assisting private businesses, granting them local monopolies, etc., then sure, they should be regulated.

</comment></comment></comment></comment><comment><USER6> It's nice to see some governments recognizing that something which was once a luxury is now becoming a necessity as offline alternative for mandatory services (paying bills, job searching, etc) get stripped down or removed entirely.

<comment><USER7> A car used to be a luxury, now it's a necessity.

Do you think owning a car is a legal right?

<comment><USER8> Irrespective of whether you believe governments should play a role, this is the wrong comparison to make. Governments don't provide the cars, they provide the _roads_.

Similarly Finland's government isn't proposing giving everyone computers - they want to make sure everyone has access to the _infrastructure_.

<comment><USER9> This is also important in Australia where the population is mostly clumped in cities along the coasts. Without similar thinking, broadband in "the bush" would simply be unavailable or prohibitively expensive.

</comment></comment></comment></comment><comment><USER10> This really implies other rights for the requirements to access the broadband connection. Shelter, electricity, computer, etc, would all be needed for someone to realize this legal right. Will these also be provided by Finland to its citizens?

</comment><comment><USER11> Not necessarily related to this article, but doesn't Finland also require every citizen to enlist in their military for a few years?

I remember reading about it in some Linus Torvalds interview, seems like an interesting policy.

<comment><USER12> In Finland, all _men_ are subject to a mandatory service. Women can join in voluntarily but the numbers have been really low number, to no surprise.

The service can either be military training (at least 6 months) or civil service (pacifist option but it's twice as long, which is criticized internationally by Amnesty). About ten percent of young men are dismissed each year: some for a real reason (medical, psychological, other), some for "other" reasons.

While officially never admitted, the system is heavily geared towards re- enacting Finland's WW2, that is, defence against Russia, Finland's eastern neighbour. It's a collective wartime trauma in constant replay.

It's an archair system that will turn into something more modern in a couple of generations or the coming decades, if not for anything else then for the lack of funding.

</comment></comment><comment><USER12> If they just stopped censoring the internet as well, too, we could be talking about something likening a "right".

<comment><USER12> Why was this downmodded? There's active internet censorship in Finland, see <http://www.effi.org/blog/kai-2008-02-18.html> or google for more information.

I think it brings a quite striking contrast to the way the "internet rights" were bragged about in the news.

</comment></comment><comment><USER13> Interesting. I wonder if every Finn has the right to food, clothing and shelter.

<comment><USER14> Yes. Finns who cannot afford the above get government support that is equivalent to an estimated minimal reasonable cost for such expenses.

</comment></comment><comment><USER15> I keep telling people it's my legal right to have a spaceship, but noone seems to believe me. :(</comment>


A Possible Polymath Project?  - <AUTHOR> http://rjlipton.wordpress.com/2011/05/12/a-possible-polymath-project/ <comment><USER2> TIL about [http://michaelnielsen.org/polymath1/index.php?title=Main_Pag...](http://michaelnielsen.org/polymath1/index.php?title=Main_Page)

I cant help but think that a wiki isnt the best platform for this. It needs something more analogous to pen-and-paper proofs. ie. discrete steps with the possibility to fork any step and combine identities...</comment>


Big-O: how code slows as data grows - <AUTHOR> https://nedbatchelder.com/text/slowsgrows.html <comment><USER2> I have a CS degree, and while I feel it has been quite helpful, I by no means think less of those without one or that it makes me a better dev than someone else. In fact, most of the best developers I've worked with have not had a CS degree. My running theory is that the same skills and talents that enable somebody to learn software engineering/programming without the structure of a degree program are the same skills and talents that make a great engineer. Long story short, there is no reason to feel like less of a dev, programmer, or engineer because you don't have a piece of paper.

</comment><comment><USER3> Also related: [https://accidentallyquadratic.tumblr.com/](https://accidentallyquadratic.tumblr.com/) examples of where, due to abstraction or otherwise, a quadratic or even worse algorithm emerges, and is not obvious from a quick glance since the multiple nested loops are separated out from each other.

<comment><USER4> I ran into an excellent example of this once. A build system was performing a bunch of operations on a very large (hundreds of megs) xml file. There was a little javascript utility that was created to perform operations on the xml file. That script was called externally with hundreds of separate operations. This took about half of a day. Because each time the xml document was loaded and saved by the script took about 4 mins of overhead. I rewrote the script so that it could do multiple operations in one pass, bringing the time for that step back to about 4 mins total instead of 12 hours.</comment></comment>


Watch 37signals talk about server infrastructure - <AUTHOR> http://webpulp.tv/post/573991716/37signals-mark-imbriaco

<comment><USER2> From the summary:

      - 37signals has 40 servers, and a total of 125 OSes running.
      - 3 Server admins work at 37signals.

That is insteresting. Most orgs would consider that a pretty low ratio of servers/admin.

<comment><USER3> It's much easier to have a higher ratio when the absolute numbers are higher. Fewer than 2 admins means no vacations, and a team of 3 is often more comfortable.

<comment><AUTHOR> I agree with rarrrrrr, having been the only admin or in a team of two, vacations can be tough. I would gladly work in a team of three if I had the choice.

</comment></comment></comment><comment><USER4> this guy must be the worst interviewer ever. sometimes, he seems to be so bored...

interesting insights, tho.

<comment><AUTHOR> I really wasn't bored, just had a bad head cold. I have to get used to video too, I've always don't audio only.

</comment></comment><comment><USER5> _Bare metal servers showed a 15% increase using production traffic, over KVM._

KVM?

<comment><USER6> That's an interesting data point to keep in mind. Because KVM is boasting better efficiency than Xen, and Xen boasts ~2% overhead. Well, that might be theory...

<comment><USER7> Virtualization overhead numbers are basically useless since they depend on hardware generation, virtualization mode, configuration, workload, etc. Unless you're running _exactly_ the same configuration as 37s, your mileage will vary... a lot.</comment></comment></comment>


The Unknown Hackers (2000) - <AUTHOR> http://www.salon.com/2000/05/17/386bsd/ <comment><USER2> I was a regular salon reader and remember reading this article seventeen years ago. Specifically, I remember thinking that it was interesting that Salon of all places would cover a relatively obscure operating system that wasn't even the second or third most popular open source Unix port on PCs.

</comment><comment><USER3> Linux won because the copyrights to BSD hadn't been sorted out yet. If the AT&T lawsuit weren't still happening, we'd be looking at a very different OS landscape.</comment>


Speech-To-Speech Translator iPhone App - <AUTHOR> http://www.ddj.com/mobile/221100130

<comment><USER2> Think about how flaky translation of written text already is, and then imagine doing a speech-to-text step first.

I strongly suspect this isn't ready for prime time, yet. And I'm guessing the potential for hilarious mishaps is fairly high.

</comment><comment><USER3> would it be possible to take this same engine and use it for a live sentence by sentence translation of spoken voice conversations on the android, since there the platform is more open? that's a scarily awesome proposition.

<comment><USER4> You mean, translating a phone conversation? I believe Android's current API doesn't allow you to access phone call audio -- similar to the iPhone, if the article is correct. (A translating app using IP telephony might work, though...)

[http://developer.android.com/guide/topics/media/index.html#p...](http://developer.android.com/guide/topics/media/index.html#play)

</comment></comment><comment><USER5> It is a good effort, but i doubt it will work. This is the professor who is notorious in the academic/funding circles for faking demos, getting fined and so on.</comment>


The Night They Drove the Price of Electricity Down - <AUTHOR> http://www.slate.com/articles/business/the_juice/2015/09/texas_electricity_goes_negative_wind_power_was_so_plentiful_one_night_that.html <comment><USER2> I don't think the author understands the electricity market. Negative spot prices for energy are not unique to Texas, or wind.

Every energy production mechanism has a limited ability to scale production (either up or down) on demand. If you overproduce electricity, it needs somewhere to go, and the electric company is willing to pay consumers for the service of dumping the excess electricity.

This is not a market inefficiency, it is a technical inefficiency, and is essentially inherent to the problem of power generation.

If the cost was negative for a prolonged period of time, you would have a real story.

<comment><USER3> Worked on a project to address the problem of predicting wind power production over the short term (as in the next couple of hours). It is a very hard problem, and when I left it we had declared it successful because we were something like 10% better than the current method, which is just to assume that current production will continue.

Wind power is a strange beast. A 300MW wind farm can go from 250MW to 0MW to 100MW over a 1-2 hour period. Trying to properly plan for that or ramping up/down your coal/gas plants to cover the changes is very difficult and expensive.

<comment><USER4> I'm inclined to think that wind will only work at bigger scales if we can find industrial processes that are dominated by energy costs rather than capital costs and which can scale up and down in energy consumption fast enough to track the wind output.

I can think of a couple of possibilities. Water desalination is an obvious candidate. It might be economical to massively overbuild desalination plants and then only run them at peak capacity when wind is at peak as well. If the cost of electricity is much greater than the capital cost of the desalination plants, then it might be a net win when amortized over the lifetime of the plants.

Generating hydrogen by electrolysis and then immediately converting it to ammonia by the Haber process might be another. The production of ammonia for use in fertilizer using methane as the hydrogen source is a major green house gas contributor. There would be a significant advantage if we could move to carbon-free production.

<comment><USER5> The usual candidate for this is aluminium smelters.

They are extremely energy intensive: usually operators will only turn them on full when electricity is below e.g. $50 a MW.

The have the added bonus that they can act as a temporary battery: you can smelt extra aluminium to use up the negative energy rates, but then let it convert back again.

<comment><USER6> I thought aluminum smelters couldn't really shutdown production.

I remember it was big news in New Zealand when Tiwai Point (which uses 15% of the power nation wide) shut down some of it's production during the 2008-2009 power shortage (caused by a drought in the hydro power lakes.

Graph: [https://upload.wikimedia.org/wikipedia/commons/e/e2/Electric...](https://upload.wikimedia.org/wikipedia/commons/e/e2/Electricity.NZ.TiwaiPoint.png)

You can also see the other major drought in 1993.

<comment><USER5> They can't just turn the whole plant off; but we were taught they can just be left with a trickle charge to keep them in liquid state.

The large energy cost is in driving the reaction forward.

<comment><USER6> Oh right, you can adjust the speed of the reaction throughout the day to adjust to market fluctuations.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> This happens occasionally in the PJM control area, even without a subsidy. PJM has a lot of nuclear and hydro plants in their territory. The nuclear plants don't like to cut output at 4 AM when they'll have to crank it up at 5:30 AM; the thermal time constants in the system are too big. So they'll take an occasional hour of negative prices.

Hydro Quebec sometimes has too much water, in which case they run all their generators at capacity. They, too, will take an hour of negative prices rather than closing huge steel intake gates and taking a generator out of service for an hour, only to have to go through the startup and resynch process an hour later.

Wind output in the PJM area varies about 4:1 over a typical day. When peak wind lines up with minimum power demand, prices go negative, the automatic bidding system communicates back to the wind turbines, and the blades go to zero pitch, slow, and stop. The wind guys grumble about this, because some of them financed their operations without allowing for this. But it's not frequent. Wind turbines are good at shutdown and startup resynch, because they have to do it every time the wind slows down below useful speed.

</comment><comment><USER8> Interesting topic; silly article. The author really wanted to use the "only in Texas" tag, but if you read closely you notice the main driving factor was a _Federal_ subsidy; nothing at all to do with Texas. (And as other commenters have noted, other regions and countries also have occasional negative spot prices.)

I'd say something snarky about "only in slate", but of course, they're hardly unique.

</comment><comment><USER9> So, Texas constructed a market that allows it to take advantage of Federal dollars being credited to other market participants (energy providers) in the most efficient way possible.

The author didn't clarify the "complicated" market, but my assumption of the highest-price-to-everyone mechanism is that ERCOT only selects a limited subgroup of all bidders each auction period, according to whatever is most economical for ERCOT. So there is some incentive to keeping your bid low.

What I don't understand is how anyone who isn't benefiting from the tax credits could bid at or below zero. The author seems to indicate that wind did not meet 100% of the need during the negative price period and did not indicate anyone else was receiving tax credits beyond wind producers. So how was the _highest_ price in the negatives? _Someone_ had to have costs associated with the electricity they were producing within the 70% that wasn't wind-powered.

<comment><USER10> >What I don't understand is how anyone who isn't benefiting from the tax credits could bid at or below zero.

It's also a pretty cheap way of advertising a market opportunity and driving investors' expectations: your money would be better spent on a new aluminum smelting plant being our customers rather than a new coal or natural gas plant being our competitors.

Believe it or not, in the early days, the oil industry also had this problem of "excess supply". In the UK in the 1900s they dealt with this by lobbying Winston Churchill (before he was prime minister) to shift the navy from running largely on coal to running on oil.

<comment><USER9> So, intentional loss leaders? That's an interesting theory.

</comment></comment></comment><comment><USER11> As someone who interned at the New York ISO, the organization that operates the bulk electricity market in New York State, this article doesn't seem well researched:

> Third, Texas has a unique market structure....

The article then goes on to describe a market structure that is very similar to New York ISO and ISO-New England. It is called Location Based Marginal Pricing.

<comment><USER7> The article is oversimplifying, of course.

PJM has most of their training materials on line.[1] Those give a sense of how PJM does it. Start with "PJM 101", "Generation Basics", and "How PJM operates and dispatches".

PJM has a day-ahead market, where most of the capacity is booked a day in advance, and a real-time market, where differences from the predicted load are handled. This allows everybody involved to plan ahead a day. There's also "non-economic operation", where, in the event of big problems, the PJM control center just tells generators what to do. That's for emergencies.

All generators post not just a single price, but a price/output curve. Generators also post a "ramp rate"; how fast they can ramp output up or down on request. PJM's job is to optimize all this, which happens on about a five minute cycle. There are also lots of physical constraints - transmission lines and substations can only handle so much power, and policy is to run the system so that any single failure won't take it down. After a failure, the transmission system is reconfigured to become single-point failure tolerant again. There's a control room in Valley Forge, PA, (and a backup control room somewhere else) where about ten people control the wholesale power grid.

California had, briefly, a system with an spot power auction every half hour, with dealers who weren't generators or power consumers, but that was abused and produced blackouts, along with the bankruptcy of PG&E. California now has a day-ahead market and a spot market for fine tuning, like PJM, which resulted in much saner operation. So does ERCOT, the Texas grid manager. It's not a raw auction every half hour.

[1] [http://www.pjm.com/training/course- catalog.aspx](http://www.pjm.com/training/course-catalog.aspx)

</comment></comment><comment><USER12> As an aside, it annoys me when people straw-man economists. It is of little consequence, but obviously any economist who studied the situation would understand at once why producers are incentivized to do this.

</comment><comment><USER13> This is happening occasionally here in germany[1], most recently 2 weeks ago[2]. Here it's not just oversupply but also the lack of overland lines to distribute wind power from the north to the south, so we have to push it to neighboring countries instead.

[1] [http://energytransition.de/2014/05/german-power-prices- negat...](http://energytransition.de/2014/05/german-power-prices-negative- over-weekend/) [2] [https://www.epexspot.com/en/market- data/intradayauction/char...](https://www.epexspot.com/en/market- data/intradayauction/chart/quarter-auction-chart/2015-09-20/DE/30d)

</comment><comment><USER14> Could this price make it to ordinary consumers? Could a consumer who was monitoring the prices consume a large quantity of power during the period to drive their on electricity bill down?

<comment><USER15> I live in Texas and I've never seen a plan more complex than a couple of tiers of prices depending on consumption, or different prices depending on the time of day (e.g. nights and weekends free).

So no I don't think any residential customers have access to electricity priced in any quantity besides a month of usage.

</comment></comment><comment><USER16> So this isn't a free market. This negative price is the result of an artificial market created by government subsidies.

If someone is selling something at negative prices, then there is a business opportunity for someone to buy that product and throw it away. Someone could setup a giant space heater to do nothing more than absorb electricity. The windfarms would pay this entity to keep buying power at no cost until the price approached the subsidy. Perhaps a steel mill or aluminum smelter could be paid to fire up another furnace for no other reason than to suck up power. I'm all for free markets and green energy subsidies, but I am totally against waste.

Hopefully some market actor will appear with the ability to store and resell energy. That might be homes with batteries and net metering.

<comment><USER17> No. The article states that electricity is sold at the highest bid to meet the demand, and that wind was only supplying up to 40% of the demand. I.e. It is the coal and nuclear power plants that are also bidding negative.

The market is working, to such an extend that during times of extreme oversupply power generators have to assess if they will temporarily turn down production (which has a cost, especially for coal/nuclear, as they are difficult to switch), or if instead they are willing to pay people to take the electricity.

Yes, without subsidies, wind would probably minimum bid at a $0/MWh, but that would still leave fuel based producers running at negative marginal cost.

What's missing in this market is flexible demand. I'm pretty sure if our appliances would spin up real time at low prices that it would be almost impossible to hit negative prices again, as demand will rise much more rapidly in response to lowering electricity prices.

</comment></comment><comment><USER14> Is there a reason the electrical plants don't short / burn off their excess electricity rather than paying consumers to use it?

<comment><USER2> The same reason people do not take their own trash to the dump, it is cheaper for electrical plants to outsource the wasting of electricity.

<comment><USER18> Presumably in a wind turbine you can adjust the pitch of the blades and bring the rotor to a stop.

I agree that in the case of a nuclear power plant it's hard to stop generating power if the price goes negative - but if there were no subsidy presumably wind generators could switch off easily if they wanted to.

</comment></comment></comment><comment><USER19> Germany with its switch to renewables is seeing the same issues. They pay the Netherlands and other neighbors to dump German surplus electricity in peak wind hours.

Germany has a goal of 80% energey out of renewables by 2050, but it is entirely unclear if this is even physically possible because of this issue of spikes.

Norway has it easier as it mountaineous enough to allow hydro everywhere.

</comment><comment><USER20> Completely inaccurate headline, and they buried the lede. The real reason power producers were willing to 'sell' electricity for negative $8 per MWh, is because of a $23 federal tax credit they received per MWh. So the negative sale wasn't negative at all.

</comment><comment><USER21> _" Impossible, most economists would say. In any market—and especially in a state devoted to the free market, like Texas—makers won't provide a product or service at a negative cost. Yet this could only have happened in Texas, which (not surprisingly) has carved out its own unique approach to electricity."_

Solar revolution drives negative electricity prices in Australia - [http://www.businessgreen.com/bg/news/2353894/solar- revolutio...](http://www.businessgreen.com/bg/news/2353894/solar-revolution- drives-negative-electricity-prices-in-australia)

German power prices negative over weekend - [http://energytransition.de/2014/05/german-power-prices- negat...](http://energytransition.de/2014/05/german-power-prices-negative- over-weekend/)

Negative pricing in the GB electricity market: is the outlook positive? - [http://www.baringa.com/our-thinking/baringa- blog/july-2015/n...](http://www.baringa.com/our-thinking/baringa- blog/july-2015/negative-pricing-in-the-gb-electricity-market-is/)

Why windy days lead to negative electricity prices - [http://gelookahead.economist.com/why-windy-days-lead-to- nega...](http://gelookahead.economist.com/why-windy-days-lead-to-negative- electricity-prices/)

I sometimes wonder if it is a point of pride for journalists to be ignorant of the subject of their articles.

edit - also, in a regulated market, one of the things that is very commonly regulated is companies offering goods or services below cost, as it is a very common tactic for manipulating markets or driving your competitors out of business.

</comment><comment><USER22> tldr: Government subsidies for wind power production exceeded the cost of "selling" at a negative price so it was profitable for wind farms to do so.

</comment><comment><USER23> This is why I bought Tesla stock. Tesla Energy is aiming to own this market.</comment>