Why Does Software Rot? - <AUTHOR> http://www.overcomingbias.com/2016/06/why-does-software-rot.html <comment><USER2> In most of the cases discussed in the article, it's not the software that rots but instead the users and/or organization that decays. When a system is created, it is generally designed to solve a known set of problems encountered by an existing population of users. Over time, the tool creators, original users, and the current problem domains change but often the old system is modified to solve these changes. A flexible system may be adaptable to certain amounts of change, but only if the current populace of creators/maintainers as well as users understand the limits. If they do not, the software often ends up less useful than if it had never been touched.

<comment><USER3> _"...it's not the software that rots but instead the users and/or organization that decays."_

Orgs decay, sure.

But I believe the OC's point was that orgs become more brittle over time as they specialize. Conway's Law would suggest that software architecture mirrors org structure, so it too would become more brittle.

Applicable cliches: "victim of one's own success", "the complexity catastrophe".

Going the full meta here, I'm okay with the death-rebirth cycle for orgs. Yes, a lot of knowledge (experience) is lost. But forgetfulness is also crucial for learning, adaptation.

<comment><USER4> It was nice up to this: "But forgetfulness is also crucial for learning, adaptation."

In IT, it's the opposite: history repeats itself endlessly with same flaws, same missed opportunities, and same techniques recreated due to often-willful ignorance of the past. One of the things I do here is get old or even current work to people it might benefit. The number of times the old stuff applies to current problems, but was never handed down to those people by predecessors, shows the problem we need to work on is maintaining, packaging, and delivering prior wisdom. Interestingly, the other engineering disciplines already do that a lot better with IT remaining pretty stubborn. To put loss into perspective, it took some groups 50+ years to reinvent benefits of Burroughs B5000 and ALGOL. Just one example among many.

<comment><USER3> I agree with you.

By "forgetfulness", I also mean the stubborn clinging to the old ways, because reasons. The necessary dying of the old guard, a la Kuhn's Structure of Scientific Revolutions.

But I also mean that each new cohort apparently needs to rediscover the deep truths (first principles) for themselves, to earn the wisdom vs blindly accepting received knowledge. I have no idea why this seems to be the case.

      When I was a boy of fourteen, my father was so ignorant I could
      hardly stand to have the old man around. But when I got to be
      twenty-one, I was astonished at how much the old man had learned
      in seven years.

        -- Mark Twain (disputed)

Perhaps forgetfulness is the wrong word. I'd don't yet have another. Some trite phrase for the culling of the herd, gouging out the dryrot, burning off the grasslands, aggressive RIFFing of the tentured... That also conveys that we must give people the room and opportunity to learn things for themselves.

Maybe what I'm advocating is the Socratic Method. Versus telling people "just because".

Thank you.

<comment><USER4> "By "forgetfulness", I also mean the stubborn clinging to the old ways, because reasons. The necessary dying of the old guard, a la Kuhn's Structure of Scientific Revolutions."

Well, that's true. I think we might need to distinguish between what ideas are lost or are not what a new group is clinging to. Often comes with new people, young or old, coming into an organization. Maybe also distinguish between "current thoughts on what's best method" vs "known methods along with their benefits and problems."

"Maybe what I'm advocating is the Socratic Method. Versus telling people "just because"."

Well, there's how we publish the knowledge and how they learn it. I'm for traditional reporting with empirical focus, less silos, easier search, and easier distribution. Far as _learning_ , Socratic Method is one among many methods that might have potential. It's too big a topic for me outside brief work I did on using various parts of brain simultaneously to increase recall, incremental problem solving of increasing complexity, matching training to realistic scenarios, and side-projects that are open-ended to match fun & curiosity. Don't think I did anything else in research on the topic. Well, a M.C. test generator and some expert system prototypes but they were crap honestly. :)

</comment></comment></comment></comment></comment><comment><USER5> Note, article is not talking about bit rot, I think that's confusing a lot of people.

The point might be clearer as:

"Adaptability and efficiency are opposing priorities."

Ecosystems face this too. A stable environment will lead to adaptations that improve efficiency, while creating new dependencies on everything staying the same. In a sense, species are constantly competing to make the ecosystem more fragile.

If this holds, there are broad impacts to information systems outside of software. broader impact of this. We like to fantasize about the mind being immortal. Maybe we could fix the telemere thing, figure out cancer, hop our brain to a clone, or upload our consciousness to some cloud.

But in my experience, being mentally alive involves some mix of plasticity and progressive refinement. You can't have both forever.

<comment><USER6> There's a similar relationship between "regularity/predictability of an environment" and "how much optimization is possible".

<comment><USER7> Not sure which way you say the relationship goes.

My first guess would be that the more regular/predictable an environment is, the more optimizations are possible.

<comment><USER6> That's correct. Optimization isn't really possible in a "white noise universe".

The less regular the universe is, the more general and less specialized your solution can be.

</comment></comment></comment></comment><comment><USER8> Software rots if you can no longer set up the exact tool chain and environment (compiler, build tool, OS version, external database, etc) required to build and run it. Even interpreted languages suffer from this problem as features are subtlely changed - by design or accident. It doesn't matter if your project is using an ancient version of VC++ or a two year old version of NodeJS. If it doesn't keep up with the latest releases of the tools then it's already got one foot in the grave.

<comment><USER9> Through the miracle of virtual machines, this is much less so. VMs are the NAT of computer systems.

<comment><USER8> Less so, I agree. But give it time - older VM image and container formats will probably rot as well.

<comment><USER10> VM formats are easier to migrate than the actual software they run.

On the other hand this is a valid concern for containers.

</comment></comment></comment></comment><comment><USER11> Unless your sandboxed in a self contained hardware and software environment (such as an embedded system) you will eventually be screwed. (rot sounds like a gradual degradation its generally not in my position)

Standards changes and OS Updates are the biggest culprit; The Vista update broke a couple of my old programs (written for Windows 95) due to the user access control changes, another was broke due to the fact it interfaced to a piece of hardware on the parallel port and the parallel port and the manufacturer went the way of the dodo.

I have seen a stand-alone DOS 6 program running a machine at a factory. The PC has been replaced three times now is only a few years old but the operator says it still does the job, I also have a 8051 powered clock I built in 1987 that still happily ticks along if I plug it in.

<comment><USER9> This is because there's little glory in simply making things work. There is glory in making things that look shiny and new. As they say, "Absolutem Obsoletum".

And congratulations on understanding exactly why I prefer embedded systems - you don't have to keep chasing the dragon of the fashionable new thing ( which is invariably old wine in new skins ). It is not that there is no new wine it is just that proportionally, it's less than all new things on offer.

FWIW, if you look carefully there will always be a motherboard available with a working parallel port. They're not cheap, and at some point the thing plugged into that parallel port wears out.

<comment><USER12> semi-related: a couple months ago I found a large industrial shipping scale in my parents basement. I grabbed an FTDI USB -> RS232 cable and a DB25 -> DB9 cable, plugged it into a Raspberry Pi, and hacked this together:

[https://gist.github.com/sielickin/8cc79f0cb6a4b4c229b9786dff...](https://gist.github.com/sielickin/8cc79f0cb6a4b4c229b9786dffcabdbe)

There are some nasty hacks in there and probably a few bugs that I could fix up at some point, but it works just fine!

I love how in a little more than 100 lines of Go code, I can so easily have a program with a dedicated thread to reading data from the serial port, a dedicated thread to serving up the initial http page to clients, a dedicated thread for sending data to websocket clients, and all of it works together so cleanly with messaging through channels.

I copy-pasted some d3.js code from around the web, retrofitted it, and in maybe 2 hours of work I have something pretty cool.

My Mom runs an ecommerce website and uses another scale for shipping, I wonder how hard it would be to write a driver for Windows that consumes data over the network and would interface with her shipping label software.

<comment><USER9> That's amazing - thanks for showing me that. I have much to learn that-a-way.

</comment></comment></comment></comment><comment><USER13> Three reasons:

1\. The code is the same but the people using it forget / never knew the reasons it was built that way - and so it looks rotten for the job

2\. Because requirements change and people try to make the old code do new things without cleaning up / refactoring correctly - so it now does neither job well, and looks rotten.

3\. Because the environment / platform changes, the FTP server is moved to a new data center and the timeouts kill the jobs etc. It looks rotten.

"Rot" more accurately is just not keeping the code inside the code base up with entropy _outside_ the code base

<comment><USER14> Or when the operating system or build tools change or are no longer available. Your code hasn't changed and suddenly it no longer compiles.

</comment></comment><comment><USER15> An interesting simile here is that software rots in the same way that Encyclopedias do.

(This is admittedly a fitting simile partly because the simile itself is being rotted by software like Wikipedia.)

In the world where new Encyclopedias (and Almanacs and Recipe Books) were printed and sold on an annual basis, the question was often why do we need "this year's Encyclopedia" when the old one is still perfectly valid. Books in general decay pretty slowly and have a long shelf life, but the facts and the views in the world inside them are frozen and possibly. Changes from year to year of an Encyclopedia are somewhat hard to notice, but in Middle School in the 90s I recall having to compare articles from a tobacco yellowed Encyclopedia set from the 70s to trips to the same articles from very early predecessors of Wikipedia. The worlds contained in those two sorts of Encyclopedias were very interestingly diverging. The yellowed Encyclopedia's facts were almost all still valid and "worked", but there were things that didn't hold up and lots of new facts that needed to be inserted in various places. If I were to edit an Encyclopedia, I'm not sure I would start from the version in that yellowed Encyclopedia if I could find a more recent set. Some of the predecessors to Wikipedia were direct descendants of that yellowed Encyclopedia and yet for various reasons historical and technical, Wikipedia itself did not inherit directly from that set in any meaningful way.

(It's interesting to note too that the physical media of software to date has a much shorter shelf life than the pulp medium of books, tobacco-smoke-filled library aging included, so argument exists that software rots worse than Encyclopedias physically, at least.)

<comment><USER16> It's a nice analogy but there are important differences here.

The thing is that the process of encyclopedias roting/going-out-of-date is obvious because we understand that an encyclopedia is a collection of assertions about the world. It is not as obvious that software is also a set of assertions about the world - and it's not as easy to determine _which_ assertions about the world the software depends on.

Just as much, an encyclopedia is a static list of supposed facts and its change is somewhat predictable by in terms of how our understanding of the world changes. We have some idea what will be valid or invalid in an encyclopedia X years old.

Software depends on obscure facts about OSes, about networks, about UI expectations, about time, and so-forth and moreover, software involves chains of dependencies so it's harder to predict what would or wouldn't break software X years old.

<comment><USER15> Thanks Joe. I think the difference you point out only strengthen analogy, and which I tried to colorfully explore in the original post, but expand here because it is interesting.

«It is not as obvious that software is also a set of assertions about the world - and it's not as easy to determine which assertions about the world the software depends on.»

I like that definition: "Software is a set of assertions about the world". It may be less obvious than an Encyclopedia, but again, perhaps that only makes it a better analogy.

«An encyclopedia is a static list of supposed facts and its change is somewhat predictable by in terms of how our understanding of the world changes.»

Ah. This is where Wikipedia actually strengthens the analogy. An Encyclopedia has always been loosely hyperlinked with cross-references. Entire complex webs of assumptions and assertions about the state of the world. Refactors of Encyclopedia content always could have interesting complex relationships with other content.

Certainly there is an extent to which Encyclopedias are known to predictably change: new elected officials each cycle, for example. But there is still plenty of unpredictable changes: new red letter days in history, new evolutionary scientific shifts in consensus, new lenses in which a culture views itself.

An example of the lattermost, in that yellowed 70s Encyclopedia you could still spot fragments in the process of being excised of what we now consider to be racist views of culture and cultural history. That among other things was a source of fascination to this teen in the 90s.

The often unspoken truth is that as much as Encyclopedias try very hard to be objective, they are still a product of their culture. (So too do we often want to think that Software is objective and nothing but mathematical assertions about facts about our world, but here too they are products of our [corporate] cultures and moored in ways both obvious and not obvious to the culture that built them.)

Wikipedia shows us that complexity of Encyclopedias much more directly, but those bones were always there in Encyclopedias even when we couldn't see them. A link to click is much faster than shuffling between volumes of an encyclopedia to follow cross-references, but the cross-references have always been necessary to build up a view of the assertions about the world.

More interestingly, Wikipedia strives to give a better, more transparent view, into the cultural biases of its contents; all of the [citation needed] tags and "More Research Required" templates and Talk pages of endless discussion. The novelty here is that they've opened up all of that to the common user, but the essence of that has always been in Encyclopedias, just locked away by the librarians and researchers in their own notes.

</comment></comment></comment><comment><USER8> I find that integrated tests in software projects go a long way to reducing rot. When something eventually invariably breaks due to some external factor, the test suite greatly reduces the time to identify and fix the problem.

<comment><USER17> If you can build a piece of software and run unit and integration tests on it, it is more or less rot proof. That's because all those tests lets one delete or replace components of the system without having to worry too much that the whole thing is going to break in some unknown way that won't be caught until it's in production.

Without a good suite, the larger the system gets, the more people get scared of doing radical things to it because they are worried about breaking some critical functionality. As far as the not being able to build aspect goes, the main threat here is relying to much on closed source software and/or libraries that go out of support that no one can find the license, or critical documentation for, or worse manifests some critical unfixable bug.

</comment></comment><comment><USER18> "... the arrogance and self-indulgence of youth."

s/youth/younger programmers/

How can they make work for themselves?

1\. Do what's already been done, not even knowing it's already been done. 2\. Declare "rot" or some similar claim of obsolescence and proceed to redo what's already been done.

There's nothing necessarily awful about this unless they fail to do a better job than the earlier effort.

Alas, this is too often the case. For a variety of reasons.

In the early days, portability was a higher priority. Not to mention longevity. Because everything was expensive.

Today's software "rots" a lot faster than the software from the early days of computing, IMO.

And so the younger programmers have lots of "work" to do.

Yet I do not see much progress being made. Because I do not measure progress by productivity alone.

Programmers who can churn out code in a dozen different languages to do the same old things are a dime a dozen.

As a user, I do not want software that needs to be updated every week. Poorly written software and gratuitous use of network bandwidth.

But I can see how programmers who love writing code would enjoy this state of affairs.

<comment><USER19> Replace younger programmers with older programmers and people would scream ageism.

</comment></comment><comment><USER20> I think _rot_ is not quite the correct metaphor. In my experience it's more likely to ossify, become sclerotic, build up scar tissue. As features are added or performance is tweaked, individual pieces become more complex and the connections between them multiply. If specific action (refactoring) isn't taken to fight this tendency, later developers will react to one piece being maintainable by making even more spurious connections and workarounds in adjacent pieces. That fixes the immediate problem, but makes things worse overall in the long term. Ultimately everything turns into the kind of tangled mess that everyone who has worked on an old multi-person project can recognize.

Unfortunately, a good refactoring requires understanding greater than the original author's[1], and therein lies another whole essay. ;)

[1] Related to [http://www.linusakesson.net/programming/kernighans- lever/](http://www.linusakesson.net/programming/kernighans-lever/)

</comment><comment><USER21> Wheres the connection to software rot?

</comment><comment><USER22> "Software rot" stems from the world around the software changing in a way the software is unable to adapt to, and breaking it.

<comment><USER23> I feel like "rot" is the wrong term, and the right term might come from that old parable about the man who built his house on the sand rather than the rock.

</comment></comment><comment><USER24> Dependencies.

Big old hairy mainframe apps have no dependencies, and last decades. Anything new expects constant updating.

</comment><comment><USER25> Software rots because everything does.

</comment><comment><USER26> > Apache, the most important web server software today, is an old piece of > technology whose name is a play on words ("a patched server") indicating > that it has been massively patched.

Is this true? I have never heard that Apache was a play on of words.

<comment><USER27> Apparently its a myth, and the original intention was based on the name of the tribe. [http://www.linux-mag.com/id/472/](http://www.linux-mag.com/id/472/)

</comment></comment><comment><USER28> There is an analogy to be drawn between software and societies, and they way their early adaptations to one environment block their later adaptions to another.

</comment><comment><USER29> What does rot even mean, in terms of software?

As far as I can tell, he means "Why is complex software hard to _change_ " which is a reasonable, though fairly easy to answer, question.

Software doesn't rot. New features or refactors screw it up. New needs or technologies may make it obsolete. But it doesn't rot, and people who talk that way are often busy-body rewriters who want to pitch the existing implementations, with all their Chesterton Fences[1], and begin anew.

[1] [http://www.chesterton.org/taking-a-fence- down/](http://www.chesterton.org/taking-a-fence-down/)

<comment><USER30> > Software doesn't rot.

On the face it, that's true. But the reality is that the environment that software runs in is constantly evolving and software that doesn't also change slowly starts working worse until it stops working entirely. We have plenty of software at my work that hasn't changed fundamentally in 10 years and can no longer be used.

Lots of really old software requires equally unchanged old environments and old hardware and that hardware slowly rots too.

<comment><USER31> ... add to this that user expectations change. Applications not matching that feel rotted. Just look at websites from 10 years back to get an idea only on very surface levels.

</comment></comment></comment><comment><USER32> Stewart Brand wrote a book called "How Buildings Learn" and in many ways it's a better version of this post.

[https://en.wikipedia.org/wiki/Shearing_layers](https://en.wikipedia.org/wiki/Shearing_layers)

</comment><comment><USER33> > Newer programming languages are often interesting, but they are typically > less flexible at first than older languages. Everything else being equal, > older languages perform better and are faster

At the cost of more difficulty in writing, or other tradeoffs (e.g. security).

> Programmers, especially young programmers, often prefer to start from > scratch. .. In part because it is much more fun to write code than to read > code, while both are equally hard.

No way! You need to NPM literally everything.</comment>


Easy Steps to a Complete Understanding of SQL - <AUTHOR> http://tech.pro/tutorial/1555/10-easy-steps-to-a-complete-understanding-of-sql <comment><USER3> _SQL JOIN tables should be used rather than comma-separated tables_

Having used SQL since long before the ANSI JOIN syntax was well supported (first Sybase, then MS SQL and then Oracle) I resisted it for a long time out of habit, and also because at first the syntax was buggy when used in Oracle.

But I have come around to being in favor of it. The bugs have been fixed, and the main advantages are that: inner and outer joins are more clearly stated than by using ' __*= ', or '(+)' suffixes on one side of a predicate; and the join criteria are clearly separated from the WHERE clause. It makes it much easier to see how the tables are being joined vs. how the results are being limited.

<comment><USER4> They stop Cartesian products from accidentally occurring. That alone should force the use of ANSI joins. cross joining two million row tables is not a pleasant experience!

</comment></comment><comment><USER5> The way I see SQL in my mind is programming Venn Diagrams ( [http://en.wikipedia.org/wiki/Venn_diagram](http://en.wikipedia.org/wiki/Venn_diagram) ) followed by creating a projection that uses one or more of the areas within that diagram (contrasting strongly with imperative languages, i.e. logic programming). Unfortunately it's not straight-forward to do that because SQL is a superset of Venn Diagrams, but that line of thinking is where you need to be.

<comment><USER3> Yes, exactly. To use SQL well you need to think in terms of sets and set operations. Venn Diagrams are a good idea to help visualize this. If you find yourself writing a lot loops over cursors there's a good chance you're doing it wrong.

</comment></comment><comment><USER6> I thought this was a very good article, unusually so for a blog post about databases. I do take issue, however, with the idea that there's a beauty to SQL. There's a beauty in the conceptual model, but you have to squint sideways to see it through the deeply hostile syntax.

<comment><USER7> I'm not sure the "Relational Model" is even a great conceptual model for common business or analytical processing. I think SQL is ugly because the model is a poor fit; if the model was solid, a clean syntax would follow naturally. As for alternatives, MDX has superseded it for core "pivot table" analytics. For application processing, it seems modern document/hierarchical databases are good for many transactional needs.

What modern relational databases have... is a very intuitive abstract storage model. That's awesome. We've been working on an alternative "navigational model" (inspired from CODASYL and ORMs) based on this storage model. Our experimental query language is at [http://htsql.org](http://htsql.org)

Our critique of SQL is at [http://htsql.org/doc/overview.html#why-not- sql](http://htsql.org/doc/overview.html#why-not-sql)

<comment><AUTHOR> I'm sure you'll agree with Stonebraker's recent (actually, not so recent) claims that the relational model is "all wrong":

[http://blog.jooq.org/2013/08/24/mit-prof-michael- stonebraker...](http://blog.jooq.org/2013/08/24/mit-prof-michael-stonebraker- the-traditional-rdbms-wisdom-is-all-wrong/)

htsql seems very interesting. I should soon blog about your product and critique. Have you been publishing elsewhere?

<comment><USER8> Thanks for linking to that talk, it deserves a submission all of its own.

<comment><USER9> Absolutely. Very interesting talk. Definitely deserves a submission.

BTW, the person asking the last couple of questions is Ed Bugnion, one of the co-founders of VMWare. He is a faculty now at EPFL.

<comment><AUTHOR> Interesting, I didn't know that!

</comment></comment></comment></comment></comment></comment><comment><USER10> Another thing to understand is how NULLs are handled in different contexts, using either the familiar two-valued logic or the more exotic three-valued logic. It's kind of messy but really worth knowing if you're working with SQL. The wikipedia page actually gives a pretty good account of the issue: [https://en.wikipedia.org/wiki/Null_%28SQL%29](https://en.wikipedia.org/wiki/Null_%28SQL%29)

</comment><comment><USER11> The most beneficial thing you can do when you start with SQL is to NEVER EVER put join clauses in the WHERE section. ALWAYS put the join clauses as close to the table you're joining as possible.

eg : never do SELECT * FROM A JOIN B WHERE A.ID = B.a_id and B.id > 10 but always JOIN B ON A.ID = B.a_id AND B.id > 10

the second way of doing "scale" much much better when you add more tables, and start mixing left and right joins.

</comment><comment><USER12> Great article! I wish all SQL libraries would adhere to this structure instead of forcing devs to write glorified (string) SQL. Ie., let me

        from(join(table1, table2))
        .where(table1.col1 > 10)
        .groupBy(col1)
        .project(col1, avg(col2))

and only in that order.

<comment><USER13> I wrote my own library in Java recently for something similar. It was inspired by LINQ and SQL grammar.

         Select statement = table1.select()
           .join(table2,[join condition])
           .where(table1.col1.gt(10))
           .groupBy(table1.col1)
           .selection(col1, Fn.avg(table1.col2))

         Select s = new Select()
            .from(table1)
            .join(table2, table2.t1_col1.eq(table1.col1))
            .where(table1.col1.gt(10))
            .groupBy(table1.col1)
            .selection(col1, Fn.avg(table1.col2));

<comment><USER14> have you seen empire db? [http://empire-db.apache.org/](http://empire- db.apache.org/) no-one much seems to use it, but it's actually quite neat.

</comment></comment></comment><comment><USER4> Be careful with point 6. If you use a subquery through an IN clause and it returns a NULL as one of the rows, nothing will return.

<comment><AUTHOR> That's only true for NOT IN predicates (which is mentioned in the article)

Example: [http://sqlfiddle.com/#!4/d41d8/16785](http://sqlfiddle.com/#!4/d41d8/16785)

        with a(x) as (
          select 1    from dual union all
          select 2    from dual union all
          select null from dual
        )
        select 1 from dual where 1 in (select x from a)

<comment><USER4> Doh! Of course. :-)

The reason being is that IN uses an disjunction and NOT IN uses a conjunction. When you compare the value to NULL it returns UNKNOWN, which ultimately causes the Boolean expression to be false.

I hope that redeems my reputation a little :-)

Btw, great article!

<comment><AUTHOR> > which ultimately causes he Boolean expression to be false

Strictly speaking, the boolean expression is also UNKNOWN ;-) I've tried to explain this here: [http://blog.jooq.org/2012/01/27/sql-incompatibilities-not- in...](http://blog.jooq.org/2012/01/27/sql-incompatibilities-not-in-and-null- values/)

... but to this date, I'm still not 100% sure if I correctly understood SQL NULL. A quite worrying thing is documented here: [http://blog.jooq.org/2012/12/24/row-value-expressions-and- th...](http://blog.jooq.org/2012/12/24/row-value-expressions-and-the-null- predicate/)

It's about row value expressions and NULL predicates. The following are not the same!

            (A, B) IS NOT NULL
        NOT((A, B) IS     NULL)

> I hope that redeems my reputation a little :-)

You're forgiven ;-) Thanks!

<comment><USER4> Darn it, right again. Sigh.

Those expressions can be worked out via first order predicate logic. Damned tricky, I can definitely see how that would trip up pretty much anyone at first glance. If you weren't aware of logical equivalences then you might be scratching your head for some time!

Another great article, btw.

</comment></comment></comment></comment></comment><comment><USER15> Fantastic article. I clicked on it, hoping to find a way to help some of my less SQL-savvy co-workers to understand SQL a little better. I instead came away with a better understanding of my own.

<comment><USER16> Stuff in here they nevr taught me all those years ago either! And a smart geek is never be too proud for a refresher.

<comment><USER17> Where do you go to keep up with things like this.

I was taught at a reputable university to put joins in the where clause. Now it seems that is frowned upon, and I only find out 13 years later.

</comment></comment></comment><comment><USER18> I like SQL. The relational model is often underestimated. All of the complex structure that goes beyond individual rows is orthogonal and is itself first- class data. Contrast with document databases where hierarchies are part of the low-level storage mechanism. By making the structure first-class data, the relational model keeps you from painting yourself into a corner.

</comment><comment><USER19> > _If a has 3 columns and b has 5 columns, then the "output table" will have 8 > (3 + 5) columns. The records contained in this combined table reference are > those of the cross product / cartesian product of a x b. _

This is where the easy steps become hard for real noobs.

<comment><USER20> A graphical example of a Cartesian join - [http://web.utk.edu/sas/OnlineTutor/1.2/en/60477/m71/m71_3.ht...](http://web.utk.edu/sas/OnlineTutor/1.2/en/60477/m71/m71_3.htm).

</comment></comment><comment><USER21> I think the author has the order wrong. Consider:

        SELECT WEEKDAY(created_at) wkday, COUNT(1)
        FROM orders
        GROUP BY wkday

The above query works fine, thus GROUP BY, ORDER BY, and HAVING are all aware of SELECT.

<comment><USER4> Those are expressions. They only work once you have picked the table(s) - <USER2> - and determined the rows you want returned - WHERE. The query then groups and orders the results and calculates the scalar value of COUNT(1) which it repeats on every row, then it determines the weekday for every row for the column created_at and uses that value for output.

<comment><USER21> My point is that a better explanation of the execution ordering is:

FROM, JOIN, WHERE, SELECT, GROUP BY, HAVING, DISTINCT, UNION, ORDER BY, LIMIT.

The author has SELECT after the GROUP BY and HAVING, which doesn't make sense because GROUP BY is aware of expressions in the SELECT statement. Grouping and filtering happens _after_ selection.

<comment><AUTHOR> (I had to correct myself)

SELECT is evaluated _after_ GROUP BY, at least in most databases that implement SQL correctly. SQL Server's documentation nicely explains this:

[http://technet.microsoft.com/en- us/library/ms189499.aspx](http://technet.microsoft.com/en- us/library/ms189499.aspx)

Skip to "Logical Processing Order of the SELECT statement".

MySQL, PostgreSQL, and SQLite behave differently, though!

</comment></comment></comment></comment><comment><USER22> Not a bad article but it needs a step 11:

 _Know thy database engine for if you do not, your local DBA will be mightily pissed off when you do a cross join across 15 tables and watch his /her IOPS go through the roof..._

<comment><USER3> This is why you have a test instance for developers (or more likely now, every developer runs his own engine locally or in a VM).

<comment><USER22> That's irrelevant as it usually only goes bang in production under production load and data volumes.

</comment></comment></comment><comment><USER23> Thanks for this article very beneficial, I'm learning about Databases at the moment so this definitely comes in handy while learning SQL.</comment>


NFL Games Have 11 Minutes of Action - <AUTHOR> http://online.wsj.com/article/SB10001424052748704281204575002852055561406.html <comment><USER2> I've long compared an NFL game to a game of chess - it's much more about incremental position playing than, say, soccer where a freak move can swing a game rapidly.

This analysis seems to bear out my comparison - watch a grand master chess game, and there's a _lot_ more time between moves than actually moving. Maybe we need a speed version of the NFL?

<comment><USER3> Chess? Well, you're making the wrong comparison. Football is like war. War is two things: Extended, pointless boredom, and short bursts of energy and terror and injury, all of which is encompassed by years of unseen strategic, economic, logistical, and tactical planning.

Chess is some freak memorizing combinations of opening moves from past games.

<comment><USER4> Saying: _"Chess is some freak memorizing combinations of opening moves from past games."_

Is about as accurate as saying: _"NFL Games Have 11 Minutes of Action"_

And incidentally, chess also happens to be a game about war.

<comment><USER3> _chess also happens to be a game about war._

So is the game RISK. But the actual relationship of both games to war is exactly nothing.

</comment></comment></comment></comment><comment><USER5> I bet it seemed like a lot more than that to Brett Favre, last Sunday evening.

The beauty of Football (I've only really followed the game for about a year now, coming from a place where soccer & basketball are the major sports) is that these short bursts of action are extremely concentrated, physically and mentally. The game may seem simple, but when you see the plays explained later on it's incredible how many variables go into each move.

BTW, one thing I think is unique to the NFL are these clips where they actually put a mic on a player for the duration of the game. Haven't seen it in any other sport yet, pretty cool: [http://www.nfl.com/videos/nfl-films- sound-efx#orderBy:allTim...](http://www.nfl.com/videos/nfl-films-sound- efx#orderBy:allTimeViews)

<comment><USER6> I think it's cooler that in Rugby broadcasts they actually have a mic on the referee for the entire match.

<comment><USER7> Interestingly, the Australian Football Leagues three field umpires are mic'ed up as well.

It was also tried on a number of players but the constant grunting (from exertion) was seen as a distraction to the enjoyment of the game.

<comment><USER8> I remember reading that a center in the NFL had been mic'd and that he said "fuch" an absurd number of times during the game (three or four hundred).

</comment></comment></comment></comment><comment><USER9> Football fans should really check out the sport of Rugby.

If you want to see an all encompassing video of the sport, watch this: <http://www.youtube.com/watch?v=kRpPf6rCyW4>

Rugby, 80 minutes of prowess and determination. The camaraderie is unparalleled in any other team sport I have played, that includes soccer, hockey and baseball. The game is definitely underrated in North America.

<comment><USER10> Specifically rugby union, rugby league is a pretty stop. go. stop. go. but not as bad as American footy.

I don't really like any footy and i'm not from NZ but just about any All Blacks game is worth a watch.

<http://www.youtube.com/watch?v=OsXTa7UCGlk>

<comment><USER7> If you are also into high speed, high action, full contact sports. I'd highly recommend youtubing Australian Football or Aussie Rules.

Teams to watch out for who tend to play a very entertaining game is the Geelong Cats (last years AFL Grand Final winners). They are going through a golden era at the moment.

<comment><USER10> Tight short's... arial ping pong... but at least you didn't say collingwood.

That said i'm always up for a kick in the park :D

</comment></comment></comment></comment><comment><USER11> In related news, marathons have a minimum of ~2 hours of action.

It would seem that length and/or amount of action does not necessarily correlate with average spectator enthusiasm/excitement.

Who knew?

</comment><comment><USER12> I'd be interested to see a similar breakdown of a baseball game. If you count from the pitcher's release to the end of a play (basically, when the ball is in motion), I bet the total would clock in somewhere around 15 minutes.

Edit: 15 minutes on average. Of course baseball isn't timeboxed like most other sports.

<comment><USER13> When the pitcher steps on the rubber would be a better starting point for the start of the play. Many plays don't even involve a pitch, and wouldn't be counted in your definition.

<comment><USER12> That's a great point and I agree with you. However by that logic, shouldn't the amount of time counted for football include when the quarterback comes under center? There's a lot of football going on when Peyton Manning comes to the line of scrimmage. Or when a quarterback is trying to draw the defense offsides.

</comment></comment></comment><comment><USER14> There was a guy a while back who would put up torrents of NFL games with all the non game stuff cut out. I seem to recall them weighing in at around 20-25 minutes. I though they where great, and I love watching football.

</comment><comment><USER15> This comes as no surprise to anyone who's attended an NFL game, _especially_ a playoff game.

</comment><comment><USER16> Thats like saying that olympic powerlifters work for 3 minutes every 4 years.

<comment><USER17> No one's denying that NFL players train heavily throughout the year. But their games are pretty short on action. Pretty similar to your powerlifter comparison, actually.

<comment><USER16> What I'm trying to say is that just because you dont see the action, that its not there.

Every football fan knows the average length of a football play is 6 seconds or less. But within those 6 seconds there is soooo much: 1\. Offensive - Defensive line play 2\. Offiensive coordinator versus the defensive coordinator in play calling, formations... (tactical) 3\. The safety verus the quarterback. 4\. The corners versus the wideouts. 5\. Coach strategy. (strategic) 6\. Specific Matchups ... 7\. QB versus defensive coordinator.

Its all in there ... You have to be able to see it. I was going to write a document on understanding all this from the position and number of players at the line of scrimmage - ie what they show on TV during the play ... but I've not yet had time :( ...

The thing is in a team sport you can either be long on action and short on strategy or vice versa. The main tradeoff is the amount of communication required between players.

I love American Football more than any sport because of the amount of strategy + the great quality of players that you get over here.

</comment></comment></comment><comment><USER7> 11 minutes of action in the NFL would very compare poorly with a full contact outdoor grass field sport like Australian Football (aka Aussie Rules).

I suspect that the amount of action time in a full 100 minute game of Aussie Rules (taking the professional Australian Football League as an example) would be close to 30-40 minutes action time (with some of the midfield players racking up around 20 kms running per game).

It would be interesting to do a similar analysis to the one undertaken here.

NB: Aussie Rules in the US has around 60 clubs and 3000 or so players. Their website can be found here: <http://www.usfooty.com/>

</comment><comment><USER18> So, what is the hourly pay rate of the players that are only really involved the plays, not planning/strategy? What is the caloric output during play of the typical NFL player? How does that relate to how much money they make and how does that compare to other sports, say basketball or hockey? I'd love to find some stats on this kind of thing mostly just so I could be more sick of football :)

I understand they make so much because the public 'demands' and they 'supply' but it seems almost grotesque how much some of them make and how little they apparently do.

<comment><USER19> I would argue that their pay rate isn't high enough because of what we're learning about head injuries. I still love watching football though.

</comment></comment><comment><USER20> Probably why a lot of the world doesn't like the game, I don't mind it every once in a while but certainly don't follow it.

Random question, they can pass more than once right? If so why don't they?

<comment><USER21> The restriction on passing is that if you have moved the ball past your own line of scrimmage, you cannot throw the ball forward relative to the field, only sideways or backwards. It is not often tactically advantageous to do this, although it does happen on occasion[1]. Since getting 10 yards in 4 downs suffices to retain possession, it's not worth the risk of turnover unless there's 4 seconds left on the clock and you're down by one point (as in [1]).

[1] <http://en.wikipedia.org/wiki/The_play>

</comment></comment><comment><USER22> And unless they're in a no-huddle offense, there is usually very close to 30 seconds between the tackle at the end of one play and the snap of the ball at the start of the next. So, if you enable the 30 second skip button on your TiVo, you can watch an entire game in a half hour.

</comment><comment><USER6> This is why I prefer to watch soccer. You consistently get about 90 minutes of action without commercials. I know when I sit down to watch a game that I'll be wasting no more then 110 mins (halftime included) of my life.

<comment><USER23> Likewise hockey. Lots of highly-engaged play for about an hour. With a DVR I only spend about an hour and a half per game including some interviews, maybe overtime/shootout, and replays.

<comment><USER6> This is true. After watching soccer almost exclusively for the last four years I find myself more attracted to hockey each time I catch part of a game.

</comment></comment></comment><comment><USER24> I suppose Highlightcam for football and baseball games would be sacrilege.

</comment><comment><USER25> It would be cool if somebody could edit together a 20/30 minute version that showed all the action plus time for a few replays and breaks. This would also be useful be done for Baseball and Cricket. I wonder if it would be exciting or if the continuous action would just be too much to handle.

<comment><USER26> Not sure about baseball, but pretty much every sports channel showing cricket in India has a 'highlights package' at the end of the day which does just this.

<comment><USER25> Highlights packages only show the most interesting balls. Usually with a lot of replays and filler. I'm talking about showing every single ball without all the fluff.

</comment></comment></comment><comment><USER27> I've never watched a NFL game, but my impression from German TV is that the same ratio of content to idleness could apply to every single program out there. Perhaps the human brain has a desire to be numbed?

</comment><comment><USER28> For years, I've thought it was under 5 minutes. oh, well.

</comment><comment><USER29> ... which explains why football games are so boring.

<comment><USER30> Better yet, this explains why football games are so exciting. Constant anticipation for the next burst of action. Enough action within those brief bursts to review, replay and analyze, just in time for the next burst of action.

<comment><USER31> _this explains why football games are so exciting_

I agree, and often ponder what play I would call were I a coordinator.

Note this is also the argument political columnist George Will used to enshrine baseball as the "thinking man's sport", where there's no hard limit between plays. But I prefer football precisely because of the time pressure; it's rather like chess without clocks vs. chess with clocks. (Pitcher checks the runner, steps off the rubber, runner retakes base, pitcher steps on the rubber, gets sign from catcher, winds up, checks the runner again, batter steps out, grabs some dirt, rubs bottom of bat, reseats helmet, steps in box, pounds plate with bat, pumps a couple times, pitcher gets sign from catcher ... lather, rinse, repeat. I've had my fill long ago.)

Back when I was a tyke, televised baseball games (with Dizzy Dean and Buddy Blattner, sponsored by Falstaff, Pabst or Hamms beer) usually took about two hours. Now it's three hours for the same amount of action, and the beers are only slightly better.

</comment></comment></comment><comment><USER32> So if there's 11 minutes of action total, the defense might play 5 minutes, the offense 5 minutes, and special teams 1 minute. So star quarterbacks play 5 minutes per game. With 16 games, that's 80 minutes of action per season. Add three more games for Super Bowl bound teams and you've got 95 minutes. The Colts pay Peyton Manning $14 million per year. Assuming that he is paid for his _actions_ on the field, he is making a wage of $8.85 million per hour.

<comment><USER33> In addition to the five minutes of action, you're not counting: watching film, practice, training, off-season conditioning, and showing up for the job.

<comment><USER32> You don't get it. All those other things you mention could be done by any amateur quarterback. They require no special skills. What makes Peyton Manning unique and consequently expensive isn't his ability to show up, watch film, practice, work out, etc. Rather, it's what he does in those 5 minutes per game that separates him from thousands of journeyman quarterbacks and determines his $14M salary. So yes, he is making $8.85M per hour. All the other stuff he does could be done equally well by countless others.</comment></comment></comment>


CyanogenMod – Android Community Operating System - <AUTHOR> http://www.cyanogenmod.org/

<comment><USER2> "Download the CyanogenMod Installer for Windows Vista/7/8"... . No love for Linux? Wow!</comment>


Wikipedia Has Raised in a Week What Took a Month in 2009 - <AUTHOR> http://www.readwriteweb.com/archives/wikipedia_has_raised_in_a_week_what_took_a_month_i.php

<comment><USER2> Even if Wikipedia gets ads on it, I think we editors will continue editing.

</comment><comment><USER3> Ads, when not overdone, really just don't phase me honestly.

I tune them out.</comment>


Feds: $100,000 'Low Income' in Parts of Bay Area - <AUTHOR> http://sanfrancisco.cbslocal.com/2017/04/24/bay-area-low-income-100000-san-francisco-san-mateo-county-hud/ <comment><USER2> very interesting!

[https://news.ycombinator.com/item?id=14220925](https://news.ycombinator.com/item?id=14220925)</comment>


The Stockholm Investment Landscape - <AUTHOR> http://www.thenordicweb.com/blog/the-stockholm-investment-landscape <comment><USER2> Great overview always seems that Sweden/Stockholm is doing a better job then Copenhagen, Denmark. We need to promote our self because I believe we have a great community and investment opportunities!</comment>


The CEO of a Failed For-Profit College Started a Coding Bootcamp - <AUTHOR> http://motherboard.vice.com/read/the-ceo-of-a-failed-for-profit-college-started-a-coding-bootcamp-brensten-devcodecamp-milwaukee <comment><USER2> $14,000 for a 3 month program with no accreditation?

If someone is not in a hurry, spending $20,000 or so for a Harvard Extension School Masters in Software Engineering seems like a better investment. I have a Bachelors (non-STEM) and an MBA, and am taking CS50 right now in preparation for the Masters.

The problem I see with a lot of bootcamps is they focus on specific technologies which may become outdated (thus necessitating re-learning) without giving students a solid grounding in CS. In the long run, if you're willing to make a more gradual transition to programming, a solid grounding in CS is, IMO, a worthwhile investment.</comment>


Show HN: Use Meetup.com Data to Find Your Ideal City - <AUTHOR> https://cole-maclean.github.io/meetupcityfinder/ <comment><USER2> This is cool and a nice idea / UI, but it seems to bubble up populations centers. Needs to normalize for population somehow. ([https://xkcd.com/1138/](https://xkcd.com/1138/))

<comment><AUTHOR> Yup that's definitely true, population centers rank highest for most topic lists (especially New York where meetup.com was founded)

The original use-case I envisioned for something like this is someone deciding which cities they might like to move to by choosing cities that have a lot of people with shared interests. So should the metric for that be total people that are part of a topic, or the % of total city population engaged in that topic? I've currently chosen the former, but interested in what others might think.

Thanks for the feedback,

-pH+

<comment><USER3> I agree with erex78. The former measurement doesn't tell me much; I already know what cities have the most people.

The latter measurement better indicates the probability of a chance encounter with someone who has your shared interests.

<comment><AUTHOR> Yeah that's a good point. I like scribu's point that it can be somewhere in between. I'll have to think on that a little.

<comment><USER4> Usually (taken from erf/error based on sample size), something grows proportional to the square root of the sample size. If using raw sample sizes, you can apply a square root to that sample size before using it (meters its impact).

</comment></comment></comment></comment></comment><comment><USER5> I think meetup.com is a fairly poor data source for these kind of questions - even if the data was normalized. Interested in Coffee? Seattle is better then NY, but is way less likely to have meetups dedicated to it since they are just baked into the culture - so theres a higher probability that the unrelated meetup you will be hosted at a local coffee shop.

<comment><USER6> This is probably one of things where you only count positive results - presence of meetups on your topic is probably a good thing, but the lack of them doesn't necessary mean anything.

<comment><USER5> Well my point is that positive, doesn't actually mean positive - there could be more meetups about coffee because there are less good coffee places to consume it. To illustrate this point: I would intuitively expect there to be more pro life meetups in a liberal area and more pro choice in a conservative one - since the norm is baked into society and people don't feel the need to meet up about what everyone agrees to. A much better approach would be to use existence of business that support said aspect of a city.

</comment></comment></comment><comment><USER7> Very cool. I haven't looked at what Meetup has available for an API but I've rarely ever seen its data used as a way to find out about a city, even though it's a great way to (manually) look up interests. You'd miss out on things that are on Eventbrite (but no reason why you couldn't combine their data...), but otherwise, it seems like a solid way to at least find hotspots for interests, particularly regional cities (i.e. not New York/SF/Chicago) that have, for whatever reason, an especially strong group.

</comment><comment><AUTHOR> I published a quick writeup to accompany the visualization here: [http://cole- maclean.github.io/blog/Meetup.com%20City%20Finde...](http://cole- maclean.github.io/blog/Meetup.com%20City%20Finder/)

<comment><USER8> I feel like this should be the link instead and you can have a fairly prominent link on the post pointing to the app

<comment><AUTHOR> I'm curious what the interest would have been had I linked to the article over the graphic. Unfortunately I'm not sure how I could test that without having a time machine.

Thanks for reading the writeup!

</comment></comment></comment><comment><USER9> I put in "fishing, hiking, & skiing" and the #1 recommendation is NYC.

</comment><comment><USER10> Doesn't work very well for people who don't want to live in the US. For every topic there are exponentially more American meetup groups so even if I insert tons of topics related to my country and only my country the top picks are still in the US.

</comment><comment><USER11> Actually that's part of what I do manually. However, in Berlin (my favorite city so far) meetup seems to go down. Fewer events in general, fewer interesting ones, and more events where nobody shows up. 2012 it was awesome, but in 2016 fun is hard to come by. I don't know, maybe they increased the payment in the meantime or something or maybe I'm just getting too old.

<comment><USER12> Funnily enough, Berlin is at the top of my list. (And it's also my favorite city so far.)

</comment></comment><comment><USER13> If I double-click on the globe, it zooms in, and then I can't rotate the globe anymore. I also don't see how to zoom back out!

</comment><comment><USER14> Cool! I've used meetup to meet people while traveling and it's gone really well. I've thought about making a tool for that. Did you scrape meetup, use their api, or is there a data dump available?

<comment><AUTHOR> I used their API, it's actually quite well documented and their rate limits seemed generous to me. Here's a writeup I did for collecting the data: [http://cole-maclean.github.io/blog/Meetup.com%20City%20Finde...](http://cole- maclean.github.io/blog/Meetup.com%20City%20Finder/)

</comment></comment><comment><USER15> Looked up my home city of Dublin, Ireland. Apparently we're number one for "sober activities". Which both surprises and doesn't surprise me at the same time (Plenty of people tired of Irish pub culture using it as an alternative place to find social events)

</comment><comment><USER16> I can't tell if this is getting hammered right now or I'm not using it correctly. I put a topic in and nothing happens...

It did work for one topic after a while, but I'm not sure if I was just lucky or did something different.

Great idea though, I hope I can get it working later!

<comment><AUTHOR> It's a pretty large dataset for d3 to be loading, so it can take quite a while. It also seems to struggle on firefox. I have alot to learn about web app optimizations and building for multiple browsers and devices. And generally anything frontend :p.

Try waiting quite a while before interacting with it. An example topic "data science" should initialize the map data eventually.

<comment><USER17> In my case, Mac El Cap, FireFox will briefly load a topic and immediate reset the page. Safari appears as a giant no-op.

</comment></comment></comment><comment><USER18> If I put in anything mah jong related it removes all the previous topics and resets the page. I've tried with 'mahjongg' and 'mah jong'. Though 'mahjong' seems ok, it just doesn't pull anything.

</comment><comment><USER19> I put in all my interests (and current meetup groups) and Brisbane was in the top 10 list. The funny thing is I moved FROM Brisbane to Sydney (for better jobs)

</comment><comment><USER4> Usually (taken from erf/error based on sample size), something grows proportional to the square root of the sample size. If using raw sample sizes, you can apply a square root to that sample size before using it (meters its impact).

Also, perhaps you could do something like establish an overall average and standard deviation "percentage interested" in each topic, then compare the percentage in a given location with the expected. The farther the percentage from the expected (in either direction), the more that location gets pulled up or down. For example, maybe every location is equally good if you're interested in "breathing air", but then maybe one has a slightly higher concentration, making it more relevant.

Also, as you are combining several interests, you are trying to maximize coverage and uniqueness (maximum number of interests present in the maximum amount, with stronger interests given more weight (though you don't ask anyone to rank interests), such that locations having them getting a boost). That is, one location shouldn't dominate the rankings due to a much higher likelihood of having one interest, while having the other interests being "averagely represented" or worse.

That is, maybe the proper way to combine the standard deviations for each place is through multiplication (take the absolute value, then "multiply in" if standard deviation is positive or "divide in" if standard deviation is negative). This will ensure that below average "satisfaction of interests" divide/lower the ranking score and that positive "SOIs" multiply/increase it.

Also, it's good to pull the ranking score from a mean/center/expectation with each "interest score", rather than just blindly averaging/kludging them in. The standard deviation approach achieves this, but I'm mentioning it explicitly such that you can consider it in the event something other than standard deviations are used.

There could also be something done to boost a city with meetups for a rare interest (the smaller the overall percentage, the more weight a location gets). For example, if 2% overall are interested in Cricket and 20% in design, then a city with an overall percentage of 3% cricket should be boosted more than a city with an overall percentage of 30% design, even though the proportion "above the norm" is the same, as Cricket is "hard to find" or a rarity.

Also, if you could factor in the area of a city, then that could enhance the scores further (5k out of 5 million people in New York means something entirely different than 5k out of 5 million in a sprawling suburbia, as population density makes it more likely the 5K in New York will be accessible).

See what I'm getting at?

<comment><AUTHOR> Thanks for the detailed response! I think you and others on here are right, there's a better way to do the ranking so the results closer represent what we're looking for. I think I need to better define what question we actually want to answer with this data and then establish the metric that best measures that, and your ideas definitely give some hints for some directions that measurement can go. I especially like the idea of "boosting" cities that have a rare topic the user is interested in. Thanks again for the interest!

-pH+

</comment></comment><comment><USER20> amazing representations of the midwest, from that data</comment>


Power Save Mode in PhpStorm - <AUTHOR> https://blog.jetbrains.com/phpstorm/2017/11/power-save-mode-in-phpstorm/ <comment><USER2> "Hector" icon, cute :) This is a long-standing feature that few people know about, I think.</comment>


The abandoned college campuses of Second Life - <AUTHOR> http://fusion.net/story/181901/we-took-a-tour-of-the-abandoned-college-campuses-of-second-life

<comment><USER2> The university where I did undergrad conducted "Virtual Biology Lab" in Second Life as recently as Spring 2015. A friend of mine got stuck in one of these fake classes. His description made it sound like a terrible excuse for a learning environment, clicking a single button to magically perform dissections while constantly battling connectivity issues.

Of course, they still charge students the full lab fee, as if they were actually purchasing supplies.

</comment><comment><USER3> I'm not a Second Life apologist, but it seems like (from comments on the OP) that some of these environments are successful, just maybe not on a full-time basis.

</comment><comment><USER4> Eastern Carolina University sports teams are "The Pirates", hence the pirate ship. Their color is purple so I am sad to see they didn't make a Purple Pirate Ship.

An hour down the road from there: if NC State, UNC Chapel Hill, and Duke had had Second Life campuses, I can only imagine the level of trolling that would have taken place since rivalries are not mellow.

</comment><comment><USER5> I always felt that these environments could be ported and utilized for some other type of VR game. A cross between Lost and Westworld. Trapped in a virtual world, your being hunted by some type of rogue AI.... (something like that).

<comment><USER6> You mean Reboot?</comment></comment>


Show HN: Automatic VPN Generator – Protect Yourself at Sochi and Starbucks - <AUTHOR> https://www.tinfoilsecurity.com/vpn/new <comment><USER2> A couple of things that I'd like to see in this script:

1\. Go ahead and apply all of the software updates. The RAX Ubuntu images are behind on a lot of security updates.

       apt-get -q upgrade
       apt-get -q dist-upgrade
       reboot (at the end)

2\. While you're at it, enable automatic security-related upgrades. It's likely that the user of your service is less technical and not Linux-saavy. Let's keep their cloud server from being owned.

3\. Force public key auth (disable password auth) on their OpenSSH server and (preferably) disable root logins entirely. Create a user account for them if needed, with sudo access.

4\. Set up IPtables as default-deny with holes punched for OpenVPN and OpenSSH.

5\. Configure OpenSSH to listen on port 443 in addition to 22. Some hotspots block port 22. Almost nobody blocks port 443/tcp. This is super-handy if you're ever working from some place with a restrictive or filtering firewall. SOCKS over SSH is awesome, especially when you're dealing with censorship or malevolence at the DNS level. I used this technique when I was in the Army and our Army housing had horrible DNS servers that censored a lot of legitimate sites.

6\. I would prefer you not ask for people's cloud provider API keys. This has huge potential for abuse. Instead, give them a script that they can run on any Mac or Linux box that takes the root password and IP and provisions their server for them.

I suppose I should make a PR for you; maybe later this weekend

<comment><USER3> I'd also install fail2ban. I looked at the auth logs for my DigitalOcean VM, and it's amazing how many bots/people try to log into it, even though it doesn't contain anything necessarily valuable.

<comment><USER4> I've noticed the same on DO. Lots of weird proxying attempts trying to use the webserver too.

</comment></comment></comment><comment><USER5> The script is currently generating a 512-bit (EDIT: 2048-bit after merging my pull request) DH modulus. DH over a finite field modulo a 512-bit prime is weak sauce, about as hard to break as a 56-bit or 64-bit key for a symmetric cipher.[1] You're using DH over a finite field, not ECDH.

Please upgrade your script to generate a 4096-bit DH modulus. EDIT: A 2048-bit safe prime provides over 100 bits of security and is much faster to generate.

I'm not sure why OpenSSL hasn't upgraded their default modulus size, but to have the same strength as a 150-bit symmetric cipher key, against the best attack techniques 2004 had to offer, you'd need about a 4575-bit DH modulus.[1] AES-128 is about as hard to break as a 3200-bit DH modulus given the best techniques of 2001.[2]

EDIT: Times to generate different sized safe primes on my MBP maxing out one core:

512 bits = 0.5 sec

1024 bits = 0.8 sec

2048 bits = 2 min

3072 bits = more than 30 minutes

4096 bits = more than 60 minutes

[1] [https://tools.ietf.org/html/rfc3766](https://tools.ietf.org/html/rfc3766) (see table in section 5)

[2] [http://tools.ietf.org/html/rfc3526](http://tools.ietf.org/html/rfc3526)

<comment><AUTHOR> Would you mind submitting a pull request? We're happy to take a look, and if we merge it it will update for all future VPN provisions.

That would be awesome.

</comment></comment><comment><USER6> Need secure internet? Just drop your API key in here. Yes, I know they link to the script, but you don't have any assurance that they're running the same script when you give them your creds.

Promoting the use of VPNs and secure browsing habits is awesome, and I applaud them for open sourcing the script. But asking people to trust them to do the work negates much of the benefit they're trying to provide.

<comment><AUTHOR> We hear you. We think that getting more people using VPNs is more important, so we made that trade-off. We built this for people who are mostly non- technical, but we wanted to provide the script such that those who are more technical are able to run this on their own. :)

I get that you can make the argument that we're training people to stick their API keys in random textboxes on the internet, but we thought getting more people on a VPN was worth the risk.

<comment><USER6> Can you possibly link the script more prominently, with a suggestion that folks run it themselves? I found it easily enough, but I was also browsing with the primary intention of finding out what it was you were running on the servers, not as someone looking to make a VPN.

<comment><AUTHOR> Pushing to production as we speak. :)

</comment></comment></comment></comment><comment><USER7> If you just need secure browsing and you have a shell somewhere (like a VPS, EC2 instance, a Linode, etc.) just use ssh.

ssh -D <port> user@host

Then configure your browser (I use a plugin called FoxyProxy) to use localhost:<port> as SOCKS5 proxy.

This is also very cool: [https://github.com/apenwarr/sshuttle](https://github.com/apenwarr/sshuttle)

Edit: I should add that I do not think your DNS requests will go over the proxy. You might be able to configure your browser to do that. Caveat emptor.

<comment><USER8> Am I correct in thinking that browser plugins like flash, java and silverlight will not use the browser proxy and will leak data?

<comment><USER9> Flash will honor browser proxies for HTTP connections initiated within an app (e.g. via getURL()), but Flash apps can also open arbitrary sockets, which go directly. For Flash video, recent versions will first try a direct RTMP connection, but will fall back to RTMPT (RTMP tunneled over HTTP) if that fails, so they'll successfully go via the browser proxy if you block other outgoing connections at your firewall.

But yes, if you allow plugins that have the ability to initiate arbitrary connections, there's no way to guarantee they aren't making un-proxied connections, unless you either use firewall rules to block outgoing un-proxied connections, or you transparently proxy everything (VPN). Same as with running arbitrary non-browser apps that might open socket connections.

</comment></comment></comment><comment><USER10> If you're looking for an L2TP setup (iOS, etc), check out this script for EC2:

[http://www.sarfata.org/posts/setting-up-an-amazon-vpn- server...](http://www.sarfata.org/posts/setting-up-an-amazon-vpn-server.md/)

I'm still trying to find a VPN solution that can stream 1080p video across the Pacific Ocean, but I still haven't been able to get something working with enough bandwidth.

<comment><USER11> That's awesome. We used OpenVPN because that's what we were familiar with, but your L2TP script looks great. Would you mind if we tried to integrated that at some point?

<comment><USER10> I didn't create it; I just linked to it, lol.

</comment></comment></comment><comment><USER12> Note that deep packet inspection will be able to identify this as OpenVPN traffic the way its configured right now. You can configure OpenVPN to use a fixed key [1] at which point the traffic is indistinguishable from random noise and no longer has any protocol data. The big tradeoff here is that this disables perfect forward secrecy; you can't add this as an extra layer on top.

You may also want to specify "cipher AES-256-CBC" in both client and server config to upgrade from the default AES-128 it uses.

[1]: [https://openvpn.net/index.php/open- source/documentation/misc...](https://openvpn.net/index.php/open- source/documentation/miscellaneous/78-static-key-mini-howto.html)

<comment><AUTHOR> Would you mind making a pull request? We're happy to take a look. :)

</comment></comment><comment><USER13> This is awesome, but there's always something that makes me feel uneasy when asked for API keys like this. They could probably list and delete any and all of my droplets with that kind of credentials, couldn't they?

<comment><USER14> Definitely agree. We hopefully answer some of those questions on our FAQ ([https://www.tinfoilsecurity.com/vpn/faq](https://www.tinfoilsecurity.com/vpn/faq)). We also link to the script we run if you want to set it up yourself.

<comment><USER13> Thanks for the info! That's definitely helpful. Maybe it should be a bit more prominent, like adding that "Why should I trust you?" section to the main page below(or above) "What is a VPN?".

<comment><USER15> On a site claiming to be about security, the FAQ should explain how and why it works even if the user doesn't trust the server. That's the whole point.

<comment><AUTHOR> We already do? [https://www.tinfoilsecurity.com/vpn/faq](https://www.tinfoilsecurity.com/vpn/faq)

Do we need more detail? :)

</comment></comment></comment></comment></comment><comment><USER16> Good timing to piggyback on the Sochi hate. You may also like to put up this NBC story on the site :)

 _NBC: All Visitors to Sochi Olympics Immediately Hacked_

[http://www.youtube.com/watch?v=waEeJJVZ5P8](http://www.youtube.com/watch?v=waEeJJVZ5P8)

<comment><USER17> It's better not to lie to people about what really happened. They were owned because they executed mysterious applications.

</comment></comment><comment><USER18> Is anyone else getting their DigitalOcean IPs banned from certain sites? I use it as a personal VPN sometimes and lately any application hosted on Google App Engine is inaccessible. Sometimes Gmail, Google Analytics and Google Drive are also inaccessible.

The strange part is that the server I am using should not be sending email or doing much really except hosting some git repos and a basic website.

<comment><USER14> I've run into that before on VPNs using AWS. Likely the public IP my server is using was used for abuse before I got it, especially as they usually only stick very ephemerally unless you upgrade to an Elastic IP.

</comment></comment><comment><USER19> This is cool. Thanks.

The site talks about deleting or pausing servers, then going back to the TinFoil page to start over in the future. However, it looks like DigitalOcean charges a flat $5 per month for the lowest tier. Is there any harm in leaving it running 24/7 and connecting when I'm in public? The most I'd be charged is $5/month, right?

<comment><AUTHOR> There's no harm in that at all, and you're correct. But keep in mind it's $5/mo/server at the lowest tier, so if you have multiple running simultaneously you'll get charged more.

If you don't mind paying the $5, by all means leave it up 24/7\. :)

</comment></comment><comment><USER20> I've been kicking around a similar idea for some time now, only that it would be a standalone iOS/Android app. On iOS, it'd output a .ovpn file that could be directly loaded into the OpenVPN app and started up. I'm sure Android would have a similar process, though I admit to not being as familiar with it.

<comment><USER21> Android is basically the same, you have a .ovpn file that gets put into the OpenVPN app. I think you have to put it in /sdcard/openvpn/ or something, but im not 100% sure

</comment></comment><comment><USER22> What I would love is one of these that lets you chain your VPN.

E.g. I want a 10 chain VPN proxy, here is my API keys for N servives, please distribute the VPN across these.

Obviously this is a slightly different use case than just protecting against passive monitoring, but I think it'd be cool.

</comment><comment><USER23> Very nice! You could make the linked script a bit more prominent, but the API key way is a good tradeoff between security and convenience. I'd still use the script myself, but I know that most people wouldn't bother with it.

<comment><AUTHOR> We're totally OK with you using the script. :) Most people, especially those just looking for a way to be secure without knowing anything about the command line, was whom this tool was built for. Glad you liked it!

</comment></comment><comment><USER24> Since the script is meant for less technical people I would advise to add a guide on how to use the VPN

<comment><AUTHOR> There is one, actually, later in the process. This is intended to help you through the process in real-time, I suppose.

</comment></comment><comment><USER25> New droplets are now being created in San Francisco datacenter instead of Amsterdam.

</comment><comment><USER26> Can you use these services' VPN (Rackspace et al) for using BitTorrent?

<comment><USER14> This VPN won't anonymize any traffic, just encrypt the traffic between you and the server. The Rackspace account would be tied to you, so any piracy-type violations will go to them first, which they will pass along to you.

</comment></comment><comment><USER27> You're abusing your refcode in the DO link. DO prohibits this linking.

<comment><AUTHOR> We got it cleared with them first. In fact, they're the ones who told us to do it; originally that link contained no refcode.

</comment></comment><comment><USER28> Why not just use Sidestep?

<comment><USER29> [http://chetansurpur.com/projects/sidestep/](http://chetansurpur.com/projects/sidestep/) :

>"Sidestep is an open-source application for Mac OS X that sits quietly in the background, protecting your security and privacy as you browse the web."

>[...]

>"When Sidestep detects you connecting to an unprotected wireless network, it automatically encrypts all of your Internet traffic and reroutes it through a secure connection to a server of your choosing, which acts as your Internet proxy. And it does all this in the background so that you don't even notice it."

</comment></comment><comment><USER30> Did they take the site down? The link is 404'ing.

<comment><USER14> Sorry about that! Things should be working again now :)

<comment><USER30> Thanks! I was able to get to it via the blog post link. Just configured my own VPN. Been meaning to do that for a while and was already planning to use DigitalOcean. Thanks for the boostrapping!

<comment><USER30> Okay, followed the script but the VPN won't connect to the internet. :( Even blew away the first build of the VPN server and rebuilt from scratch, no dice. :(

</comment></comment></comment></comment><comment><USER31> Isn't setting up a ssh tunnel easier?

<comment><USER12> This relies on properly configuring all relevant software to use the SSH tunnel as a proxy. That's very difficult to do in a way that you don't end up leaking information over the real connection.

OpenVPN works on a lower level and just tells the operating system to use it as a gateway (as configured here) and every software will magically start routing traffic over it. This is generally what you want for security, but can be annoying for bandwidth or latency sensitive applications.

</comment></comment><comment><USER32>
        ssh -D port host

</comment><comment><USER33> sochi sells.

</comment><comment><USER34> "vpns are too painful to set up for everyone else"

Its been a long time i did not not see as much bullshit. In linux, its as simple as going into the vpn tab of your connections, entering your username, password and crt file, and you are done.

<comment><USER35> They are referring to the server side of the setup process, which is painful.

<comment><USER36> Just use tinc

[http://www.tinc-vpn.org/](http://www.tinc-vpn.org/)</comment></comment></comment>


Mssql-Ng – MS-SQL Interface for Next-Generation JavaScript - <AUTHOR> https://www.npmjs.com/package/mssql-ng

<comment><AUTHOR> HN Mods: should have prefixed with "Show HN: " ... sorry.

It's based on another article that I saw here, and thought it would be cool to implement something similar for MS-SQL, since I am actively writing code to access MS-SQL the method will make that effort a lot simpler.</comment>


Professional astronomers are calculating the cost of the Starlink constellation - <AUTHOR> https://twitter.com/cgbassa/status/1132555579988303872 <comment><USER2> This issue astronomers have with Starlink is not with increased light pollution per se, where the effect is negligible, but with the fact that when there are 10,000 of these in the sky at one time, it will be much more difficult to point a telescope and not have one of your exposures contaminated by a Starlink satellite passing through. Some telescopes will sit on a source for hours for the faintest objects and the odds of encountering one of these becomes worrisome.

Anything visible to the naked eye or even binoculars is honking bright to modern astronomical telescsopes, which are sensitive to sources millions of times fainter. Even if they're not directly in the field, off-axis reflections and glints can contaminate exposures. A train of these moving through the field would be a total disaster.

For those saying let's just build space-based telescopes, sure, but it's 10-100X more expensive, and even modest ground-based facilities produce 100GB/night these days, and you can't do that from space (yet).

<comment><USER3> On the bright side Starlink is something that can help to solve both of this issues by helping to lower the price of rockets and by improving the antennas.

</comment></comment><comment><USER4> The twitter thread (and most of the discussion here) is focused on the optical impact of these satellites. As @watersb mentions in a reply[0] these satellites will likely have an impact on radio telescopes. I've found conflicting information on the frequency bands starlink will use, but all of them would put the emission in the 10s of GHz regime, which is in the middle of commonly used bands by current radio telescopes (e.g., the Very Large Array), future expansions of existing telesopes ("Band 1" for the Atacama Large Millimeter Array), and future radio telescopes (High-frequency Square Kilometer array, Next Generation Very Large Array). Surely these are outside of the "protected" radio frequency bands, but they may still affect the measurements made by the sensitive electronics of radio telescopes.

The linked twitter feed notes that 84 satellites would be in the sky at any time. I don't know how colimated their radio transmissions would be, but it's possible more of them would be "visible" in the radio, compared to the optical. So it could be a significant effect on radio astronomy.

[0] [https://news.ycombinator.com/item?id=20016619](https://news.ycombinator.com/item?id=20016619)

</comment><comment><USER5> I don't think I've seen anyone simulate the full Starlink constellation including the ~7500 VLEO satellites. For your amusement, here it is: [https://youtu.be/bKj4GDNhH0Q](https://youtu.be/bKj4GDNhH0Q)

No-one can say that SpaceX lacks ambition.

<comment><USER6> Wow, at first glance, that seems like a nightmare scenario, but then one must realize that the animation is not to scale. For it to end up seeming as crowded as that animation looks, each satellite would likely have to be the size of a small city.

The vastness of just the Earth is hard to comprehend sometimes. In comparison, if you put mountains, to scale, on a table top globe, Mount Everest would be about the height of a grain of sand.

</comment></comment><comment><USER7> Let's be thankful we don't have giant flying illuminated billboards up there (yet).

<comment><USER8> There is a startup trying to do just that: [https://astronomy.com/news/2019/01/billboards-in- space](https://astronomy.com/news/2019/01/billboards-in-space)

<comment><USER9> Wow, I sincerely hope their project fails miserably, what an awful idea.

</comment></comment></comment><comment> Paul-ish Is the concern here that they will add to light pollution and make astronomical observations more difficult?

</comment><comment><USER10> Can we make satellites with materials which don't emit or reflect in frequencies which pass the atmosphere? So those satellites won't send energy which would reach ground-based observers?

<comment><USER11> Radio astronomy relies upon internationally-recognized protection from transmission on certain frequencies. Even when satellites comply with these restrictions, these things have gotten so powerful they often overwhelm the telescopes.

</comment></comment><comment><USER12> The article doesn't provide any luminosity estimates. It is entirely possible that when they claim up to 70 satellites would be "visible" that they mean in a low light pollution area when you can see thousands of stars. On the other hand if they mean the satellites would be visible with high luminosity, I would be skeptical of that until they show their estimates of the size and matterial since these are very small and mostly under Earth's shadow.

</comment><comment><USER13> Was hoping this would be a calculation of a financial cost. Any idea how much that could be and how much would the consumer have to pay so that Starlink is profitable?

</comment><comment><USER14> The only reason we can see them is the solar panels - right? Well, let this generation of Starlinks go up as they are, and once they are out of commission in about 5 years, send up the next generation. Instead of using solar panels, they will use small nuclear reactors that will enable much higher power, potentially higher orbits, higher bandwidth, longer life etc. If you want to go NIMBY on the light, you should embrace nuclear power sources.

<comment> semi-extrinsic I don't see any reason why we should allow these satellites, if we're talking about literally mass polluting the night sky.

What they provide is a service that's already widely available in first world countries, and is seeing rapid increase in availability in third world countries (well, most of them, the US seems to be stagnant).

<comment><USER15> Starlink could provide global connectivity that can't be censored or regulated in most countries - and that's huge.

<comment><USER16> Starlink would have to comply with what the country they are transmitting to permits or doesn't if they don't want to get in all sorts of trouble. An effect that Starlink could have though, is, that people might protest censorship more because all that stands between them and freely accessing the internet is their government allowing import of receivers (instead of no cable or mobile available at all, ISPs, government, ...).

</comment></comment></comment></comment><comment><USER17> People in cities can't see stars anymore, so this complaint about the new satellite constellation is almost academic.

In decades past, people began to be concerned about light pollution. The city of Phoenix, Arizona passed its first lighting ordinances around 1983. One of the goals was to reduce sky glow.

After all those politicians retired LED lighting came out. Because the politicians who worked on the sky glow problem were two generations back, the current politicians didn't realize there are very good reasons for using amber low pressure sodium bulbs for the street lights, and replaced those lights with white LEDs. The quality of light from the LEDs is terrible, but the city saves a few pennies a year (electricity, maintenance), so they probably won't be fixing the problem.

My friend told me of how she takes her grandkids out to look at the phases of the moon, and how sometimes they find a single star in the sky.

<comment><AUTHOR> > _People in cities can't see stars anymore, so this complaint about the new > satellite constellation is almost academic._

There are designated areas of dark skies, and around observatories there are limits as to the lighting that can be used around.

But all that is being rendered irrelevant - there are no areas that won't be affected by these satellites - that's the whole point, they will cover pretty much everywhere.

And these are professional astronomers saying that they are affected. The fact that you, and others, don't really care and aren't affected isn't the point. There's a very real risk that these satellites will kill Earth-based professional astronomy, and large scale space astronomy isn't cost-effective.

<comment><USER17> The point of my post was supposed to be that people who care about earth-based astronomy really need to resist the proliferation of white led streetlights. Kids who grow up without ever seeing stars aren't likely to care about the night sky.

<comment><USER18> Neil DeGrasse Tyson has a story about going to a planetarium as a child and seeing a realistic depiction of what the night sky looks like in the countryside. His immediate reaction: Fake!</comment></comment></comment></comment>


Code Don't Change the World - <AUTHOR> https://medium.com/@lnalex/code-dont-change-the-world-8bfd9487bc39 <comment><USER2> It's all good and makes sense that the code or projects you work on should in the long run contribute to something meaningful that others can use. But choosing projects and developing software with just "that" aspect in mind leaves me with no motivation to consistently work on it. Perhaps other people find it easier to focus on work with this "good for humanity" ideology but personally I can't identify with it. I need that "intriguing" drive in the short run to build a sustainable motivation.</comment>


Dropbox Teams Officially Opens - <AUTHOR> http://blog.sfgate.com/techchron/2011/10/27/dropbox-goes-after-business-customers/

<comment><USER2> I know our work with a couple developers would love something like this, but that price we just can't afford, using personal accounts and shared folders works fine, especially if you don't need a lot of space.

</comment><comment><USER3> Am I the only one that thinks the price point is astronomical?

From 100 to 800. That is quite a commitment to what is a premium share folder.

<comment><USER4> I was thinking that too. $66/month is a bit steep for a 2 person team currently using their personal Dropboxes.</comment></comment>


Ask HN: Why hasn't visual programming taken over? - <AUTHOR> I should hope it's obvious I don't want everything to be written in scratch. But consider this: the base unit of code is the character. I use vim because it's the best at manipulating characters. Your IDE (visual studio, intellij, etc.) understands your code on the AST level, but when you edit with it, you're still editing on the character level. It still performs character-to-character transformations, even if it uses that knowledge.  Your code is still saved as a plain text file.
And this has caused all sorts of problems. For instance, you can make a flat text file which doesn't map correctly onto AST; this is called a syntax error. There are disputes about how the text file should be formatted (tabs or spaces?). Parsing is slow and difficult. Sometimes you misalign something; that only occurs because tokens have no boundaries.
Now consider the alternative: code is stored as a binary AST. Your editor is an interface for interacting with that AST, your compiler deserialises that AST directly into memory. An entire class of problems is eliminated. Whether your editor understands your code on an AST level is no longer a question. Yet, at the same time, the barrier to entry for writing an intelligent editor is significantly lowered. Same for writing a compiler. You get a lot more choice in terms of how you want your code formatted. Want your python code with braces, or your c code without? Done. The language itself doesn't even have braces or colons anymore. Any tool that parses code runs a lot faster. And we don't even need a new language! All we need do is define an AST format for a given pre-existing language.
I'll readily admit, my title was a bit clickbaity. There is some work on such editors.  And the answer to my question is trivial: inertia. But the time has come to overcome that inertia, for the sake of all the wasted electricity and lost productivity. So here's what my post title should have been: 'Ask HN: will you help me write tomorrow's IDE?' <comment><USER2> > you can make a flat text file which doesn't map correctly onto AST;

if you operate on AST, you still can create invalid AST, so not much of a change there.

> There are disputes about how the text file should be formatted

Solved by autoformatters.

> Parsing is slow and difficult

that's the only drawback so far

> Your editor is an interface for interacting with that AST

So now you can't use your text editor, and must use something that understand specific AST (and even specific version of that AST). Any change to that AST will require all the tools to be updated, while currently any text editor or processor can be used to edit any programming language in any version. So you just created massive problem with the tooling. Your grep, documentation generator, sed and countless other text tools just stopped working.

</comment><comment><USER3> What problem are you trying to solve? You spend a lot of time talking about code formatting. The more mature teams I've worked with have settled on a format and use linters to identify deviations. From my perspective, that is a non-issue.

I can type characters much faster than I can drag a mouse, so I'm not seeing any increase in productivity. In fact, the coding part of my job is perhaps the most streamlined. It's the design, planning, and people ops that consume the most brainpower.

<comment><AUTHOR> > You spend a lot of time talking about code formatting. The more mature teams > I've worked with have settled on a format and use linters to identify > deviations. From my perspective, that is a non-issue.

Formatting is not a major part of the result, it is just a nice side effect; I didn't mean to give it so much attention. I will not say, however, that formatting is a non-issue. It isn't an _important_ issue, but it is one that a disappointing number of man-hours have been spent thinking about. Just look at all the comments and controversy on this ticket about zig not supporting tabs: [https://github.com/ziglang/zig/issues/544](https://github.com/ziglang/zig/issues/544). Consider how much time has been spent making auto-formatters, and making them preserve things like comment alignment. There are 'less mature teams'; what will you do for them?

> I can type characters much faster than I can drag a mouse, so I'm not seeing > any increase in productivity.

Like I said, I don't want to rewrite everything in scratch. I have no intention for the future to be anything but keyboard-driven; an editor which is aware of tokens and syntax trees will be _better able_ to facilitate editing, no matter what your inputs. Consider fructure[1], which is not dissimilar to what I propose. I don't think it uses the mouse at all.

1: [https://github.com/disconcision/fructure](https://github.com/disconcision/fructure)

</comment></comment><comment><USER4> I've always thought someone should make a Lisp editor that is essentially visual like Scratch and AgentSheets/Cubes. The pages (or desktops or whatever you want to call them) and on-screen position and size can simply be held in ; comments using the familiar X windowing layout method.

That way, you can have different screen/page/workspace environments tailored to whatever it is you are working on at any given time.

The way Scratch lays out functions in graphical C-shaped objects makes it pretty much impossible to do incorrectly, and ensures your inputs are always correct. Except I'd make mind collapsible, since you don't always need to see the code itself, but the functions are always right where you want to see them.

I'd also incorporate ideas from Blender and Logic Audio's MIDI patching interface, which are sort of like dia on steroids.

It's something I've always wanted to do, but it is, at least currently, outside the perimeter of my available skill set, and I haven't bumped into a team that would be interested in creating it.

</comment><comment><USER5> Have you tried it? It's nice for very small projects and demos, but when the program gets longer it's painful. Perhaps a new editor can fix it, but I doubt it is possible.

Also, the conversion of text to AST is not straightforward. What do you do with infix operators? In the AST you loose the hability to write nice math formulas. What about f'' strings in python? I'm 99% sure they are desugared, specially the new version with the = sign.

<comment><AUTHOR> I'm not talking about scratch or UE blueprints. The thing I'm talking about doesn't really exist yet; there is _some_ research into it, but it's nowhere near mature enough to draw conclusions yet.

The AST would be designed for human consumption and it would include information like f-strings. Infix operators would not be lost in the first place.

</comment></comment><comment><USER6> Visual programming has certainly taken off on digital arts, electronic engineering and introduction to programming fields.

<comment><AUTHOR> Sure. But it hasn't taken as the dominant way of editing c++, python, or javascript.</comment></comment>