Is Software Engineering Possible? - <AUTHOR> https://vanemden.wordpress.com/2017/06/07/is-software-engineering-possible/ <comment><USER2> Edsger Dijkstra famously derided software engineering as "how to program when you cannot".

But if being able to program means being able to write large programs and get them perfectly correct, then no one has ever been able to program, the great Prof. Dr. Dijkstra not excepted. We all have puny brains next to a million lines of code. That being the case, the study of "how to program when you cannot" is critical for those of us who actually need to write these large programs.

I like to say that software engineering is _the art of managing complexity_. Designing abstractions to be as general as possible, with interfaces as simple and clean as possible, and minimizing coupling, makes it much easier to think about how the major components of the system interact. And that's the key: if it's too hard to think about the system at a high level of abstraction, because the interfaces have too many special cases in them, more mistakes will be made.

I don't think I can agree with the OP that formal verification is the only thing that can fairly be called software engineering. I do agree that the more verification we can figure out how to do, the better. But I would point out that well-designed abstractions are still important even when you're doing formal verification; indeed, possibly more so, because without them, the proofs get to be even more complicated and difficult than they need to be. It's really the same phenomenon: systems that are hard to think about are also hard to prove things about.

<comment><USER3> _Designing abstractions to be as general as possible, with interfaces as simple and clean as possible, and minimizing coupling, makes it much easier to think about how the major components of the system interact._

I think this is a false statement. In fact I think you're confusing modularity with abstraction.

Abstraction hides complexity, and abstractions are leaky; how the whole behaves can be strongly influenced by behaviours that leak from the abstractions. It took a couple of goes, but eventually we realized that DCOM and CORBA were poor ways to design distributed applications.

Abstractions should be as general as necessary, where necessity is subject to judgement.

Make code too abstract, and people may not understand it because it's not concrete enough, full of indirections and compositions and factory-factories; make it too concrete, and people may not understand it because it's too complex.

Creating an abstraction to share some code may pay off if there's sufficient clients for the code and they all have a similar usage profile, but it may be a false economy if the clients are few or the usage varied.

Modularity isn't an unalloyed good either. Too many modules and you may write code that acts optimally in a local way, but suboptimally globally because it has no access or awareness of other modules' behaviour. Modules designed around an abstraction may be substituted, but designing for module replacement is more often than not over-engineering. It's strongly encouraged by modern OO testing approaches, but IMO it puts the cart before the horse.

I'd like more people to realize that abstractions represent unknowns, and that any given implementation needs to be tested in an engineering materials sense, to find out what its force limits are, how it reacts under scale and pressure, so people using the abstraction are _not_ just thinking about the abstraction, they are considering the real thing they're dealing with as well. It saves a lot of time going down dead ends.

<comment><USER4> That we're not all programming in machine code is evidence that some abstractions are useful.

The most vital abstraction is that most software is blithely unaware of the physical construct embodying it.

Abstraction isn't the enemy. I think much of what we call good "software engineering" is identifying, designing, and implementing the "right" abstraction.

</comment></comment></comment><comment><USER5> Calling the discipline Software "Engineering" was the worst thing that could be done, at least given the current state of the art. Because of that, people thought you should write software the way you build a bridge or a car. The waterfall method came out of that, and it took decades to realize and recover.

Software development is more reliant on discrete math (and more generally the ability to reason abstractly) than math that helps model the physical world. Hopefully, Professional Engineering bodies will one day realize this and will adapt their program accreditation criteria accordingly.

<comment><USER6> > _the worst thing_

On the contrary, the worst thing was the abandoning of the sound engineering principles and the obligatory professional certification. Since then, the trade has been open to, and flooded by, know-nothing lazy half-wits who, should they find themselves in any other industry or medicine, would be immediately disqualified (or, rather, would not be admitted in the first place).

<comment><USER7> Why do you think they are prevalent and successful?

<comment><USER8> Because the market doesn't reward useful, quality work as much as it rewards throwing crap together and investing in marketing.

<comment><USER7> But do we mean quality and usefulness as technical attributes of a program's construction at a point in time?

Or as beliefs of a customer about a product, or a company?

The skills to get to the latter are often different to the first, and both are expected in many modern software engineers. The later is much more about iteration, experimentation, empathy, and judgement. It's not worth polishing areas of a system which you anticipate abandoning based on the results of an A/B test. Writing it and abandoning some meh code can still be the best engineered choice in a product development _process_.

The engineering of the development process matters more than the program.

I'd argue that success in modern software engineering (at least in Silicon Valley) is at least as dependent on understanding social processes, as it is on computational ones.

<comment><USER9> Oddly enough I think there's an analogy in mainstream engineering. If you build a supermarket, it has to be engineered for safety. But the engineers are not responsible for whether the mall serves a purpose, such as selling products for money.

</comment></comment></comment></comment></comment></comment><comment><USER10> Weeks of coding can save you hours of planning.

<comment><USER11> But then it wouldn't be called "hacking", and so wouldn't pass as software development these days. Instead, let's have no planing whatsoever, no design and no requirements, have minimal time for actual development, "hack" code together and see where bugs take us. Aka "scrum", "agile" and whatnot.

<comment> Joe-Z I feel like the problems you describe here (constantly changing requirements, minimal time for development, "hacking" code together) have less to do with agile methods and more with people-stuff.

I think agile methods can help a great deal in software development, especially breaking apart your projects into more discreet steps and supervising progress or giving the possibility for correcting course via sprints. Also using eXtreme programming tools like pair programming or code reviews to avoid having people slip into a frenzy of their individual domain / help keep obvious bugs out.

Having minimal time and changing requirements is just part of the game. And people "hacking" code together is just... well bad craftsmanship really

<comment><USER12> >>using eXtreme programming tools like pair programming or code reviews to avoid having people slip into a frenzy of their individual domain / help keep obvious bugs out.

Wtf does this even mean? What are you talking about. If the bugs are obvious why do you need extreme programming to find them? What frenzy? Extreme programming is garbage.

<comment><USER11> Agreed.

From my experience, when a proper preparation is done and there are clear requirements, most of the "obvious" bugs do not even come into play. They do, when "hacking" a feature and / or working in "POC" mode, which, for me, is not software engineering.

<comment> Joe-Z For me it isn't either and I'm certainly not advocating for "hacking" features together. The obvious bugs I mentioned were not design bugs, but simple slips that creep in when you've already worked for a few hours and your attention starts lacking.

</comment></comment></comment></comment></comment></comment><comment><USER13> "Whither Software Engineering?", discussed in the OP, sounds like an interesting historical piece about how long it took for mathematics to be integrated into engineering work. The author is John Allen, who wrote _Anatomy of Lisp_.

Can anyone find that paper online? [https://news.ycombinator.com/item?id=962733](https://news.ycombinator.com/item?id=962733) points to a blog post about it from 2009, but none of the links seem to work.

<comment><USER14> You can find papers more easily one "Google scholar", I found it there:

[https://news.ycombinator.com/item?id=14528514](https://news.ycombinator.com/item?id=14528514)

<comment><USER13> Ah of course. Thanks!

</comment></comment></comment><comment><USER15> FSCQ is a really great example of a large system with proofs of correctness using extraction from Coq.

Another well known project is CompCert the certified C compiler [1]. Which has seen a fair amount of external testing and use in verification of GCC and Clang as a reference for checking invalid compiled semantics [2] (to say nothing of compiling programs).

1 [http://compcert.inria.fr](http://compcert.inria.fr)

2 [https://blog.regehr.org/archives/1052](https://blog.regehr.org/archives/1052)

</comment><comment><USER16> The problem with this line of thinking is that a piece of software is ultimately built to solve a problem in our real, messy, uncertain world. There is no way to "prove" that an application served its intended purpose. As Fred Brooks wrote: "Even perfect program verification can only establish that a program meets its specification. […] Much of the essence of building a program is in fact the debugging of the specification."

<comment><USER17> Exactly by the time you could formally prove the application is correct, the market will have moved on to something different. Right now the combination of quick to develop and mostly correct seems to be more desirable to companies then slow to develop and formally correct. The success of languages like JavaScript and python show this.

</comment></comment><comment><USER18> So what would such a curriculum look like? I've tried diving into this a few times and it often feels like hurling yourself against a wall in hopes of picking up bricks. You actually do make progress after a time but it's difficult to determine the curriculum flow.

For me I think it started with reading a throwaway aside on "type verification" and wondering what the heck that was. After a bunch of wikipedia searching into different types of type calculus (not fun), I saw a reference on dependent types, which led me to Coq tutorials, and then, in an effort to be more practical, learning more Scala and a some Haskell - reasoning being that even if they didn't have full support for dependent types, at least their type systems were on the right track.

Failed efforts so far have been learning TLA+ and Idris (I haven't yet found a way to apply any learning), and trying to get through some youtube lectures on homotopic type theory. Maybe next time.

<comment><USER19> I don't know, maybe something like this, degree in informatics engineering (sorry Google translate):

[https://translate.google.com/translate?sl=pt&tl=en&js=y&prev...](https://translate.google.com/translate?sl=pt&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.di.fct.unl.pt%2Fensino%2Foferta- formativa%2Festrutura-curricular&edit-text=&act=url)

Check the "Consolidation Units" and "Specialization Units" required for earning enough credits.

Also the degree is validated by the Portuguese Engineers Society.

Just one example, most Portuguese universities offer similar degrees.

I am a strong proponent of Engineering in our work and how software should be under the same quality regulations as other products in general, not only when there are human life's at stake.

<comment><USER20> Looks a lot like mine (also western European but with more sysadmin topics and less graphics and AI).

</comment></comment></comment><comment><USER21> There exists a body of theory about how to do software engineering as described here, but there doesn't exist a sizable community of software engineers willing to _do_ this kind of work on a reasonably large scale. Judging by HN (which itself probably represents the upper-end of 'engaged' software developers), most people fidget uncomfortably when forced to explicitly declare derived types, and gnash their teeth when faced with Haskell monads; who among you would make a career out of writing Coq proofs about system correctness?

Optimistically, the community might come around to realizing that this sort of development is at least _desirable_ , and we might wind up with the basic components of popular OSes, or of firmware for dangerous things like cars and medical devices, being produced via provably-correct processes. But that will almost certainly wholly exhaust humanity's reserves of strongly math-educated software engineers (not to mention, industry's willingness to pay for it.) Software engineering may be impossible because _most humans don 't like math_.

<comment><USER22> > there doesn't exist a sizable community of software engineers willing to do > this kind of work on a reasonably large scale

also on the other side there's a very limited pool of employers willing to pay for this kind of engineering - aereospace and military mostly, and the second is cheapening out too.

</comment></comment><comment><USER23> I think a significant issue in software is the comparatively unconstrained input space.

I would imagine the average natural gas power plant would have more than a few issues if you fed it pure hydrogen instead of whatever methane based mixture it was designed for. Similarly I imagine there are other inputs that have tight specification, lubricants, etc. But, these inputs are relatively easy to control via both human and machine processes and there aren't that many combinations you would consider valid. There are complex inputs in the physical world, for example the impact of complex wind patterns on bridges, but I still feel this is simpler than many input spaces in software.

In software, at some level you're almost always dealing with human or machine generated input which may be very complex in nature such that that even after all reasonable efforts at validation the size of the possible input space is massive if not effectively infinite. We're getting a bit better at dealing with this, e.g. fuzzing, but there is a long way to go for fuzzing to be viable on more complicated inputs.

</comment><comment><USER24> Computer Science is a science field, therefore computer scientists usually see themselves as something more than an engineer - you don't want to apply the same knowledge over and over to make something exactly as the known knowledge tells you to do, you want to try new things and experiment with new ideas. That's the general mentality of the degree by itself.

Real "software engineering" would have to be done with Agda, Coq, or similarly tools - theorem provers that guarantee your solution is sound. It is done that way for software that absolutely cannot fail (eg, aerospatial). Unfortunately, engineers are usually too far away from this kind of knowledge - and sometimes even doubt this is possible or doable, even if you direct them to the research (!).

By the way, software engineering disciplines are usually depressive OO garbage, frameworks and all that non-sense that comes with excessive Java usage, and frankly never teach anything about reliability or even proving programs correct - which should have been their primary goal.

<comment><USER19> Having a degree in informatics engineering I strongly disagree with that, given the lectures we had as part of our curriculum.

</comment></comment><comment><USER25> When you build a bridge, a power station, or a wind turbine, you're creating something of known economic value for several decades, even centuries. The whole process is expected to have little to no business deliverables for years. That lends physical engineering nicely to an established set of standards. In software, we're often innovating entirely new products, and people want to see working proofs of concept, and see organic growth with the user base without getting locked in to preexisting concepts of what the product _should_ do the business figures out what the market is.

The nexus in products like vehicular and medical equipment software might be a good place to start implementing something like a discipline, but those systems are so much more singular and simplistic in their product function (not the implementation, I realize) than what the average software dev is working on that I don't even know how much downstream influence it would have.

<comment><USER26> Your hot new gamified Pets.Com for mobile doesn't need to be engineered. A web server handling payments absolutely should be.

</comment></comment><comment><USER27> Nice title, although I don't think the proposed solution (formal proof methods) is the answer. My own view is that building software is more of a design process than an engineering process, for the simple reason that software rarely has the kind of fixed requirements that make engineering solutions possible. Most software is constantly evolving and the factors that make it possible to evolve well include things like conceptual clarity and modularity, about which formal methods have nothing to say.

There was a movement around 20 years ago to establish a field of practice called"software design" but I don't think it went anywhere. The problems remain the same though. [http://hci.stanford.edu/publications/bds/](http://hci.stanford.edu/publications/bds/)

</comment><comment><USER28> There is a movement going on from weak typing back to strong typing which is the same idea approached from the opposite (practical) end. Have the computer do more work to verify program correctness by rejecting invalid expressions.

Maybe the two will meet at some point?

<comment> Joe-Z I never got the excitement that some of my college classmates had for JavaScript because they didn't have to 'deal with types'. I always much preferred a compiler telling me that this won't work instead of having to find some obscure type bug at runtime. Don't get me wrong, I still very much liked programming in JavaScript for other merits, especially the functional aspect or flexibility of extending your program simply in a browser command line window.

EDIT: I kind of got distracted by my JavaScript-story here. What I originally wanted to post was: I always had an interest in philosophy and the logical thinking you can apply in software development (i.e. you can save yourself so much trouble by just thinking through your system thoroughly before writing even one line of code). However, the general sentinment that I also got from a lot of my classmates was that they didn't want have anything to do 'with any of this stuff' (this stuff being the humanities) but didn't have a problem with cracking hard math problems day in day out (which for me at least is pretty similar to using logical thinking / philosophy on a less abstract level).

Sorry, if by my large edit I confused some people!

<comment><USER29> You wouldn't be alone as a philosopher-developer, there are a decent number of us out there.

However - shipping code is often very organic, cultivated stuff which foils attempts to establish a coherent philosophy, a through-line of "how it works" \- it works because it worked the day before, all that changed was that a little bit more was added. Repeat till broken, then refactor back to sanity(else abandon ship). It's Sisyphean hill-climbing.

This in turn plays into the falsity of "engineering" as a way to describe what's going on with a lot of code. You can do it in the small, but at any scale the codebase habitually becomes a living organism. And that tends to get puzzle-solver types excited, because if it's alive and keeps changing, then they will have endless problems to solve forever!

So I believe the philosophers of the crowd ultimately tend to move away from the applications coalface and look for something relatively smaller that does allow some time for reflection and distillation of the problem.

<comment> Joe-Z I find your last paragraph very interesting. May I ask, did you make that move yourself and, if so, where to?

In my daily work I often refactor an old, organically grown, codebase and that's exactly what I do:

I try to think of what the system actually is / should be and try to put this in code in as concise terms as possible (that's kind of a general description of software development, but I just wanted to contrast this to simply hammering in the quickest fix you can think of). A lot of times I wonder though, if that is really what is needed and a simple 'prototyper' kind-of- developer would get the job done faster, although maybe not as elegantly (slight humble-brag there).

</comment></comment></comment></comment><comment><USER30> Teams that are 100% invested on having clean, tested and reliable code _can_ have software engineering, absolutely.

The problem is that outside of the tech industry, most of the people with the pursestrings don't see the value in it _most of the time_.

Why? When their main widget makes $100M/month (for example) and its current maintenance, as fucked as it might be, is ($1-10M/month, or 1-10% of revenue), spending, say, $1-10M to drop that OpEx to half of current costs, prima facie, doesn't make a lot of sense.

What changes that equation is when some event that _could have_ been prevented by good software engineering principles occurs. Good examples: the latest British Airways IT meltdown, LinkedIn's massive data breach, 100's of millions of CC's stolen from TJ Maxx and others.

You see, when something happens that threatens that $100M/month cash machine, many millions of dollars of bonuses, raises, hirings, parties, expansion plans, and other general symbols of growth and progress get threatened as well. THAT is what people with the pursestrings don't want to see messed with.

The overall problem with this is that it's _reactionary_ , and every reactive event has a lifespan. So it takes finding someone with connections that can either capitalize on an unfortunate event trailblazing a path for software engineering to happen, or tell a good enough story of previous woes and misfortunes that can make that happen all the same.

TL;DR: If you want "software engineering" to happen at your company, you need to tell a compelling story of how NOT making it happen will lead to great losses, and then you need to be extremely patient.

</comment><comment><USER31> I've looked into Coq and like a lot of others here tried to learn the theory and application but it is so far removed from my daily work that the benefits are basically non-existent. I've gotten way more mileage out of trying to learn systems theory and cybernetics than anything related to dependent types.

People think this is a tool problem but it's not. Well, the tools kinda suck but fundamentally it comes down to culture and mind share. My background is in pure math but my programming knowledge is steeped in the hacker tradition. It is very hard to overcome all that momentum.

There is no killer application for verified programming. Whatever you can do in Coq in 10 weeks you can do in C in 2 weeks with a few extra buffer overflows. We'd be further ahead if there were more tools for validating C code than trying to rewrite the world with dependent types.

<comment> Joe-Z You're right it does come down to culture and mind share. That's why the author already admits this would be a long process. But I think he's right and to truly call what we're doing software _engineering_ there needs to be a more formal basis for how to do things and especially to discern between this is right and this is wrong.

No offense to you, but often times people that entered programming originally from related fields and taught it themselves give me the hardest time working with them. Like, no it's not enough that the software now provides this feature but you copied 1.200 lines of code for it and just squeezed in your if-branches and for-loops somewhere. Your job as software developer is to think how you can wed these new requirements with our existing codebase for others to build upon.

Sorry, but as a bit of a pedantic and clean code enthusiast this is kind of a passion topic for me :)

</comment></comment><comment><USER32> There's a terrible trend in the San Francisco tech scene where "engineer" is viewed as a synonym for "software engineer", which is itself a synonym for "programming". It isn't.

That said, I tend to disagree with people who believe the solution is "use Coq".

</comment><comment><USER33> We have existence proofs that Software Engineering is _possible_. For example, look at what NASA did with the Space Shuttle flight control software. But for most business domains, real engineering simply isn't cost effective.

[https://www.fastcompany.com/28121/they-write-right- stuff](https://www.fastcompany.com/28121/they-write-right-stuff)

</comment><comment><USER34> I'd like to contribute a definition of the words "engineer" and "engineering" from an old dictionary I had lying around.

From Webster's (1948):

engineer: n. 1. [Rare], a person who makes engines. 2. a person skilled or occupied in some branch of engineering: as, a mechanical _engineer_ , an electrical _engineer_. 3. the operator of an engine; especially the driver of a railroad locomotive. 4. in _military science_ , a member of that branch of the army which is concerned with the construction and demolition of bridges, roads, and fortifications, the laying and sapping of mines, etc. Abbreviated E., e., eng., engin., engr. v.t. 1. to plan, construct, or manage as an engineer; hence, 2. to contrive; manage skillfully; superintend; guide (a measure, action, etc. _through_ ).

engineering: n. 1. the planning, designing, construction, or management of machinery, roads, bridges, buildings, fortifications, waterways, etc.; science, profession, or work of an engineer: abbreviated E., e., eng., egin. 2. a maneuvering or managing.

</comment><comment><USER35> Feels like some stability in computing generations (CPU/GPU) or interaction paradigms (mobile) will be necessary to formalize software engineering. Until bridges/etc had a relative standard building each one was a craft. Not to say there hasn't been generational tech in traditional engineering, but generally not on the order of every few years.

<comment><USER36> It'll help, but it's not essential. There are practices that we can adopt today that will formalize software engineering. We should look to systems engineering and modeling (as used by other engineering disciplines: prototypes, mockups, math models, etc.). We already have many of the necessary formalisms, either from the fields of Math and CS, or from other engineering disciplines (notably systems engineering).

The problem for software engineering is that it's really easy to model with code, and those have an unfortunate tendency to become the final deliverable. We have short iteration cycles that make the benefit of systems engineering and large scale engineering approaches less obvious. In CE it's quite obvious that a model of a bridge (physical and mathematical) will have benefits towards the final bridge construction. It's also obvious that it's just a model.

In software, the model is too often functional enough to pass as the real thing, but fails in various ways: was constructed haphazardly allowing security vulnerabilities or stability issues; wasn't designed to run quickly or to scale effectively across multiple servers; not developed with maintenance in mind.

</comment></comment><comment><USER37> I think it's probably not possible. Here's a way of thinking about why that may be: what we're doing when writing software has as much in common with doing science as it does with engineering. By 'science' I mean coming up with a formal description of how some physical system behaves. If we're willing to generalize this activity to include non-physical systems then, I'll argue, software development would be included as the same kind of thing.

Fundamentally writing software is about constructing a formal description of a less formal 'specification' of some abstract system. The architecture used in this formal description corresponds to what's ordinarily called theory in science; when we execute our formal description, it corresponds to performing an experiment, which may lead to the discovery of anomalies (i.e. bugs); sometimes this anomalies are significant enough that we are led to revise our theory (i.e. refactor); if our re-conceptualization is significant enough and we decide to use an entirely new architecture, you're running into something like a paradigm shift with all the resistance and other social factors this entails.

Coming up with architectures/theories to formally describe systems in the foregoing manner is not something we know how to make reliable. If that goal is clearly out of sight for science, why would we expect it to be achievable in the sibling discipline of software development?

The exception I see to this (which admittedly is a large exception) is that probably most programs which need to be written are very similar to programs that have been written before. If we had some unified process for evaluating architectures for these programs that we write over and over again, then we'll probably eventually stabilize on 'true' architecture for that class of program. We see this happening to some extent in the creation of 'frameworks,' but maybe we could come up with better measures for comparing the aptitude of competing frameworks?

It could also be that the physical systems described in science tend to be amenable to simpler description than typical software specs, which may be loaded with non-unifiable features. In this case, frameworks/theory only gets you so far because most of the work is just describing special cases...

Anyway, maybe the 'science' we need to found software engineering on top of would be a formalization of this general process of coming up with formal descriptions of either abstract or physical systems. I bet there are regularities there which could be exploited for systematization—and if nothing else, there are likely insights to be had just through seeing software development as a thing having significant relations to the practice of science.

Edit: phrasing, typos, etc.

<comment><USER38> I think really hits the nail on the head. It also may be the foundation for why we find things like neural networks more usable for things like vision than directly formalizing the approach. In other words, you could use an approach like coq to verify the neural net software calculates correctly, but you couldn't use to formalize what it calculates.

</comment></comment><comment><USER39> I hope things go this way. I'm betting my future on it.

<comment><USER40> The best way to predict the future is to invent it ;)

Good luck!

</comment></comment><comment><USER41> In my opinion, "Software Engineering" has two main problems:

\- Incompetent management.

\- Incompetent programmers.

Incompetent management, because of "sky is the limit" effect, pretending being some Steve Jobs or similar, instead of doing the real job: risk management, resource allocation, put competent people in charge, and being involved so the project(s) are kept on track.

Incompetent programmers, because not being qualified enough, not having enough experience, or whatever reason making the programmer being nihilist, or a mix of the previous.

If you have competent management, and competent programmers, you'll deliver. The problem comes when you have incompetent and/or corrupt management, taking decisions from emotional impulses, or thinking in their bonus first, hiding the problems, and making the "bomb" explode years later. Then, you'll receive the excuses: legacy code, architectural problems, blaming contractors, processes, methodology, etc.

</comment><comment><USER42> No, it's necessary</comment>


Anybody Read "Technologies of the Future" by Alec Ross? - <AUTHOR> It's a must for any person who's fascinated by tech and economics. The technologies mentioned are well known, but just imagine what they will do to the world economics once become mainstream. Scary and exciting at the same time. <comment><USER2> Yes, I did a few weeks ago. I highly recommend it to anyone reading this post.</comment>


You probably don't need ReCAPTCHA - <AUTHOR> https://kevv.net/you-probably-dont-need-recaptcha/ <comment><USER2> "Many developers vastly over-estimate the likelihood of customized spam."

I run 100s of small random low traffic low priority sites. Without some form of form control, the ALL get hit with customized and random other crap spam. I don't have decent experience with many things in life, but I can say this is one topic I have YEARS of experience with. I've never over-estimated the amount of any type of spam any form can get after being on the web for just a few months. Doesn't matter how big they are or what they do.

I'm not saying ReCAPTCHA is the only thing out there or even the best, but having an open form is just asking for trouble.

<comment><USER3> Have an input element that can't be seen. If it has something in it, ignore the submit. Works for all my sites so far.

<comment><USER4> Doesn't work as soon as you're big enough to target.

The company I work for makes a SaSS forum product, and while we do have multiple spam prevention methods (akismet, stopforumspam, honeypot, a hidden input), there's enough stuff out there that has targeted our platform that a Recaptcha on the registration form is needed.

We haven't need it on any other forms yet though. After registration it's all handled by the other methods and various moderation tools.

<comment><USER5> Did you try randomizing the 'name' and 'ids' of the inputs? (including the invisible one)

<comment><USER6> I really dont know how well that will work against a dedicated attacker.

I am much more confident in ReCAPTCHA of stopping bots compared to any roll your own solution.

I dont want to hope that an alternative is good enough for my needs. I want the best when it comes to protecting my site.

Any alternative needs to have a proven track record and support to make consider replacing ReCAPTCHA.

<comment><USER7> So you force your users to consent to sharing all of their data with Google? That'll teach 'em.

<comment> shaki-dora "all their data" is a bit much, isn't it? ReCAPTCHA gives Google exactly one datum, namely the user's visit to the one page it is on.

And I would even hazard a guess that the TOS specify that Google will not retain/link that information, considering that's how Analytics is run.

<comment><USER8> No, that's how it used to be. Now with ReCaptcha v3 the recommend you load it on all your pages, not just the forms you are trying to protect, so they can predict friend vs foe more accurately.

<comment><USER9> So how does one block this?

<comment><USER10> Firefox and uMatrix[0], and then never go to those sites again, because you won't be able to use them anyway. Whether or not you want to contact the owner of the site and tell them what's up is up to you.

[0]: [https://addons.mozilla.org/en- US/firefox/addon/umatrix/](https://addons.mozilla.org/en- US/firefox/addon/umatrix/)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> Most websites probably don't. If you're one of those people, congratulations! Stick a honeypot input into your form and call it a day.

However, if you're working on anything with non-insignificant amounts of traffic, you'll get hit with some customized spam.

I've been dealing with these spammers, and if you do nothing, your forum will be filled with korean ads. We implement Akismet, StopForumSpam, Project Honeypot, and ReCAPTCHA, with the latter being the most effective (sadly). I'm pretty sure some of these spam agencies have customized tooling to handle NodeBB (they're using websockets to submit the posts, instead of HTTP POST).

Outside of these strategies, the most effective by far is reputation restrictions. Post queues if you're new or don't have enough upvotes, etc. However it does require manual effort, of course.

Would definitely love an alternative.

<comment><USER12> _> We implement Akismet, StopForumSpam, Project Honeypot, and ReCAPTCHA_

Did you tried some techniques from the article? Like hidden form fields, simple javascript checks or simple captcha?

<comment><USER13> I used to run a forum hosting service. After a while you try everything. Honeypot fields, incorrect field names, etc. You will notice things like CSRF tokens generated by one IP and used by another. When stuff fails, they send 100 real people and record what worked, and it fixes their script. It's all pretty automated. IP reputation can be helpful for some players, but most snowshoe.

Spam attempts would grow exponentially. So every time we cut it down by 90% via one of these tricks, it only gave us a bit of time.

None of this stops the determined troll though.. they can have all day to manually add offensive content. Shadow banning was good for this (1999) and group shadow banning was the best (bifurcated forum posts so all the banned people saw each other, but no one else did). Ah, memories. So good.

<comment><USER14> > all the banned people saw each other

What did happen with that strategy?

</comment></comment></comment></comment><comment><USER15> In my experience, the biggest issue I run into is targeted botnet brute force attacks.

In cases like these, someone loads up a huge botnet, a downloaded list of hacked usernames and passwords, and tries every single combination hoping to find a reused username/password combination.

In these cases, it is almost always extremely targeted. Log correlation has helped quite a bit, but it is still very painful since they alternate IPs with every request.

Automatically adding a blanket ReCAPTCHA on all login pages during a distributed brute force attempt is one of the few things that actually stops an attacker like this with minimal negative consequences.

I'm sure it is frustrating to users, but I think service disruption from what is effectively a DDoS is a worse user experience.

<comment><USER16> and you can easily count the number of failed attempts from a particular IP, and just show captcha for those over X failures, rather than every login. Normal users don't fail _that_ many times, and so are non-the-wiser.

<comment><USER17> As the commenter said, they rotate IPs. It is not that easy. I've also been on the other side of a sophisticated attack like this. The really savvy adversaries do the following, at least:

1\. Rotate through several thousand to several hundred thousand noncontiguous, geographically distributed, residential IP addresses,

2\. Associate each IP address with a single user agent and suite of cookies,

3\. Associate each IP address with a particular target username,

4\. Only attempt a few incorrect logins at a time, and a somewhat random (albeit realistic) number at that, within a given time interval,

5\. Use random, apparently human delays between successive requests,

6\. Issue requests using extremely high fidelity simulacra of web browsers, customized to the sequence and structure of HTTP requests on the website.

When the stakes are high this is the kind of opposition you'll get. Bank account takeover, social media account takeover, ticket scalping, automated sneaker buying, financial research, market research, etc.

Recaptcha introduces unpleasant user friction, but it usually works well. To invert a popular turn of phrase, it makes stopping simple attackers easy and hard attackers possible. The most sophisticated attackers will still lease reputable Google accounts and mechanical turk time to bypass Recaptcha challenges, but it will be expensive for them.

Technical sophistication is only one dimension of this game. The other is making adversaries spend more money than they can gain from being successful.

<comment><USER18> > automated sneaker buying

???

Please ELI5. I mean, why are sales bad, even if automated? Are they using stolen cards?

<comment><USER19> Most likely related to high-demand, limited run sneaker "drops", which people then resell on the secondary market. Sneaker-scalpers, if you will. It's a problem because it prevents legit buyers from getting in on the sale.

<comment><USER18> Why don't they just sell more of them?

Or as ALittleLight says, auction them?

Edit: OK, I know, limited editions. Like numbered and signed prints. But it's arguable that people who want them the most will get them. Even if it's just for resale. Doesn't seem like the seller's responsibility.

<comment><USER16> > Doesn't seem like the seller's responsibility.

sounds like to me that the seller doesn't want the scalper to sell outside the official channels imho. It might dilute the brand as well.

<comment><USER20> Exclusivity agreements are anti-competitive tbqh

The real problem is supply. Popular tickets are scalped because there's only so many tickets. Then unpopular tickets are scalped because it was so easy to scalp the popular ones.

There's only so many sneakers that can be made: making more chews up the supply chain for something which isn't _truly_ being consumed.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> ReCAPTCHA has crossed into the domain of cattle-corralling users and thus should be considered harmful. If the system decides it doesn't like you (most likely because you're "too anonymous," but you don't really know) you will be presented with slower-loading images to click and more click-all-the-things rounds. To pretend this is about slowing down bots is disingenuous as best. On top of that, usage of ReCAPTCHA perpetuates the very problems I just described.

Why isn't there a solid alternative offering yet?

<comment><USER22> >Why isn't there a solid alternative offering yet?

The latest version of recaptcha doesn't even prompt users. It loads on the front-end and uses a scoring system. It's likely you've used it but didn't even know because it's invisible.

It's the older implementations that have the slow loading images.

<comment><USER23> On Google Chrome, with adblock on, without my Google account signed in, in a new incognito tab with no extensions, I have the experience of it being invisible. When I go back to the same site on Firefox, logged onto my Google account, no adblock on, no privacy options on, I have to identify dozens of photos.

As far as I can tell, it just checks to see if your browser is Google Chrome to give you your score.

<comment><USER22> Again, that's still not the latest version of recaptcha, which never prompts users for photos.

</comment></comment></comment></comment><comment><USER24> I think we have to take a step back, and consider why we want to separate humans from computers in the first place.

Humans can do a lot of bad things that computers can do. Think of armies of low-wage people in Asia, that are paid to click on ads, spread spam, or write reviews.

And also consider that computers can actually do _good_ things, for example, allowing humans to automate their work on certain websites, or providing better accessibility for certain users.

Therefore, instead of introducing CAPTCHAs, why not focus on the actual threats. If you want to protect against spam, then build a spam filter. If you want to prevent bots from bulk-downloading your data, then build a rate- limiter, etc.

<comment><USER25> But that's something captchas are used for. Prevent fake signups.

<comment><USER26> It doesn't do that, though. Humans also create fake accounts. It does make mass creation of fake accounts impractical, though.

I solved that in past by actually charging for my service. I think the internet would benefit from having more paid content and less ads driven stuff.

One thing that captchas do protect from is brute force attacks on user passwords. Although there are other possibilities (like making the connection slow after a number of attempts).

<comment> m-p-3 This is why you need a multi-layered solution to fight spam. Captchas reduce the amount of fake accounts, which can then be taken care of by the additional layers.

It's easier to fight against a bigger opponent (botnets, etc) if you mitigate their superiority in number first.

</comment></comment></comment></comment><comment><USER27> Literally none of those alternative methods listed worked on my moderate traffic wiki. Recaptcha (and before it went away, identify the dogs or cats from Microsoft) is literally the only solution that stopped us from getting spammed. I wonder how much experience the author of this article really has in this domain.

Recaptcha has saved the internet as far as I'm concerned.

<comment><USER28> For my small wiki, refusing all submissions with external links eliminated virtually all spam. Yes, it's drastic, but in my case external links were not essential. I still use ReCaptcha to cut down on spam account signups.

<comment><USER27> We do that as well, outside of a small whitelist of allowed external links. Turns out it's not enough and we still need recaptcha for a few reasons.

We have a small set of anonymous edits every day, which go through recaptcha to be allowed.

</comment></comment></comment><comment><USER29> I fought an interesting implementation of customized spam on a web app registration form a few months back. Suddenly, every 2-3 seconds, we would get a sign up from a random email address @qq.com (it really clogged our sign up Slack channel). I didn't want to go full CAPTCHA so I dropped in a simple honeypot and the spam stopped for a good 4-5 hours. Then it picked right back up like normal. I then implemented a randomized honeypot, e.g. <input name="lastname478482"> and again, it ceased immediately and picked up a few hours later. Finally I just blocked all submissions with emails @qq.com and it stopped completely and hasn't returned months later.

Sometimes I wonder if there was a real person on the other end writing code to combat the code I was writing at the same time, and finally gave up on the 3rd iteration.

<comment><USER30> There is someone else on the other end. I had one of them previously get on the forums I worked on and complain about how quick I was to blocked them. I wrote an entire system for learning the patterns and automatically blocking them as they occurred. Eventually I determined that the vast majority of the IP addresses originated from Bangladesh so I just banned the entire country from accessing the web sites. That was the only solution, unfortunately for those legitimate users there, that made the spam stop for good.

<comment> frosted-flakes I have to ask, why do people do this? What do they get out of it?

<comment><USER30> I assume SEO targeting and associating phrases with various products. For example, people on Reddit will name and shame by dropping people's names accused of crimes next to the thing they are accused of in a complete sentence. This is an attempt to promote the relevance of those two facts in Google Search to each other.

<comment><USER31> Doesn't the nofollow attribute [1] work to remove that incentive? IIRC that was the intended purpose of nofollow?

[1] [https://en.wikipedia.org/wiki/Nofollow](https://en.wikipedia.org/wiki/Nofollow)

</comment></comment></comment></comment></comment><comment><USER32> "ReCAPTCHA relies extensively on user fingerprinting, putting emphasis on the question of "Which human is this user?" rather than the ordinary "Is this user human?". "

Classic example of collecting more information than what is needed.

<comment><USER33> Depends. The traditional techniques that automatically establish humanity without determining identity are more and more vulnerable to AI, so the only way to keep CAPTCHAs effective is to integrate identity. From that perspective, ReCAPTCHA isn't collecting more than "needed". On the other hand, the cutting-edge cryptographic technique used by Privacy Pass does supposedly preserve anonymity by making it impossible for CloudFlare to link "who solved the CAPTCHA" to "who wants to access X site", but it still involves information being collected in some form.

<comment><USER34> > _The traditional techniques that automatically establish humanity without > determining identity are more and more vulnerable to AI, so the only way to > keep CAPTCHAs effective is to integrate identity._

Is it? What's stopping AI from developing an identity in the eyes of Google? An AI that might behave exactly like Google's ideal user. Searches random stuff on their search, looks at and clicks their ads, logs in to various Google services, and when it encounters a ReCaptcha, it clicks the "I'm not a robot" checkbox.

At some point, caring about privacy might turn out to be the distinguishing feature of humans.

</comment></comment></comment><comment><USER35> I've noticed a number of major cryptocurrency exchanges using this slide puzzle as a form of captcha [1]. I dont exactly know how it works but I assume the jittery nature of a human sliding a mouse is enough to discern the bots from the nots. Are there any major downsides to this form of captcha that I may be missing? [1] [https://www.geetest.com/en/](https://www.geetest.com/en/)

EDIT: I now see the article does actually mention this, though I still do wonder how far the fingerprinting goes.

</comment><comment><USER36> Requiring to solve ReCapcha __after __may be more sensible than the usual use case where ReCapcha is required before any interaction.

Let the user create account, let the user create first comment/post, save it somewhere but hide it, then require ReCapcha to make the post visible and activate the account.

The issue with requiring ReCapcha __before __is that the website owner will never know whether a legitimate user was turned away or whether it was spam.

<comment><USER37> And how does the timing change things? You want to look at first comments/posts to determine whether it was spam or not?

<comment><USER36> After the post is saved (but hidden until the user solves ReCapcha) you can look at the database and analyse what's happening. Did ReCapcha turned away an interesting post by a legit user? What is the ratio of spans versus legit users? Etc.

If you turn both spammers and annoyed users with ReCapcha before any interaction,you will never know how many were legit and you won't be able to manually accept interesting legitimate content written by a user that is annoyed by Recapcha.

</comment></comment></comment><comment><USER38> I had this same thought in 2013 and created this library which uses a few simple tests to protect from comment spam. It definitely could be improved/tuned (and oof this code I wrote is baaaad), but for many sites it is good enough. [https://github.com/mccarthy/phpFormProtect](https://github.com/mccarthy/phpFormProtect)

</comment><comment><USER39> I used to use a script to combat uncustomized spam. A hidden input would get populated with my birthday in hex and then checked on the server. Every JS based client would be able to use the forms and we went back to math captcha for noscript.

Literally thousands of spam comments per day would stop instantly on deployment. We put this on over 400 sites and never had an issue of customized spam.

Definitely agree with recaptcha not being necessary. However, it's probably needed for popular sites which you're more likely to use. Do we see more recaptcha because of that?

</comment><comment><USER40> Here is an idea I thought of for a captcha. Render your webpage and form and include a hidden "password" field. Use a javascript hashing algorithm to hash the password on the client browser (preferably a very slow one that uses a lot of CPU). When you submit the form check the calculated hash the client did with a pre-calculated hash on the server. If they don't match reject the form. You can pre-generate a list of password/hash combinations to avoid slowing down your servers. Could it work?

<comment><USER41> It could work... If you want to set minimal system requirements to visit your website.

It will also annoy users of password managers with auto-filling capabilities. "password" is normally used for actual passwords.

Besides, nothing stops the attacker from replacing your code with a faster implementation.

<comment><USER42> There are password hashing algorithms out there (like bcrypt) that specifically take a long time to compute using the fastest method that we can think of.

</comment></comment></comment><comment><USER43> I've developed an alternative to captcha for blog comment/forum spam a while ago.

There are many methods that are easily dismissed with "but not all spam is like that" and "someone _could_ work around it easily":

• blocking of links (if you don't need them, or in fields that are not for them).

• blocking of obviously spammy keywords (or bayesian filter)

• invisible fields and syntax to trip up dumb implementations

• requiring JS, properly-functioning cookies

• blocking of IP ranges that belong to VPS providers and a couple of 3rd world telecoms that allow spam

Each of them is surprisingly effective and combined they block 99.8%.

You really shouldn't flatter yourself thinking that a spammer will even look at your page. They have literally millions of sites to spam, and they couldn't care less if they get yours or not. A bot will find a 1000 other sites to spam quicker than it takes a human to click "View Source".

Most of spam is done by amateurs who take shitty off-the-shelf spam software, seed it with a target list copied off some forum, and run it on a couple of spam-friendly or incompetent VPS hosts. By volume, this is the vast majority and it's very easy to block.

</comment><comment><USER44> > The comment form of my blog is protected by what I refer to as "naive > captcha", where the captcha term is the same every single time. This has to > be the most ineffective captcha of all time, and yet it stops 99.9% of > comment spam.

This is what we did on one of our sites. 5 minutes to implement it using a few lines of code. Same result, couldn't be happier.

</comment><comment><USER45> "It's worth noting how much easier it is to successfully solve ReCAPTCHAs when the user is logged into their Google account". Well to me it makes absolute sense as Google knows that the logged in user is a human. This article is just following the current trend all Google is bad.

<comment><USER46> Right, someone can't register a google account and make their spambots bypass CAPTCHAs with it. Just knowing that a human was present once doesn't mean they are always human. Googlen probably lets you pass easily once or twice in a certain amount of time but keeps track and starts giving you more difficult solves as time goes on. But then, someone might register a thousand google accounts and rotate through them to allow cool off time, so I'm not sure how this is handled. It is not a given that rewarding people with a google account is a good thing.

</comment></comment><comment><USER47> I understand when Recaptcha gets used before registration forms, but why, oh why, does Discord do it before any login?

If I know my email address and password I should be able to login without being logged in to Google as well ...

<comment><USER48> On the plus side you've likely become fairly adept at identifying street signs, light posts, bicycles, storefronts, bridges and buses in order to access Discord, especially if you were trying to access via Tor to gain a little privacy.

<comment><USER49> Can't speak for OP, but personally I simply stopped using Discord.

<comment><USER48> Ditto. Don't forget to close your account as it will not self-destruct.

</comment></comment></comment></comment><comment><USER50> We did everything to avoid using CAPTCHA because spammers on our platform could defeat CAPTCHA anyway. Oftentimes, they're real, actual humans.

Hidden inputs, honeypots, etc. None of them worked long-term.

Our solution: focus on the content.

All content now runs through the following filters, and takes care of 99.9% of spam: 1) Auto-approve any and all content immediately, unless: a) The content contains any HTML (url included). If it does, send it to a moderation queue. a1) If the submitter had submitted content previously and gotten approved, auto-approve their content, even if it contains HTML.

</comment><comment><USER51> Aren't hidden form elements are a major issue for accessibility?

Admittedly, so is ReCaptcha, so the trade-off may be necessary as much as it sucks. But, it's probably at least worth a mention?

<comment><USER52> They are definitely an issue for accessibility. I make sure to put "Hey! Don't put anything in this field!" as a placeholder.

<comment><USER51> That sounds like something bots could easily adapt to if the practice become widespread.

<comment><USER53> It's already widespread. But, it requires customization to overcome for many spammers. Depending on the popularity of your site, your threat model may require you to do more. But it is still a useful tool.

</comment></comment></comment></comment><comment><USER54> Microsoft's implementation is the worse. I sometimes have a hard time deciphering the captcha. Why do they need that in an iOS app? Are robots emulating people from an iPhone?

<comment><USER55> The iPhone app probably communicates to some servers over an API of some kind - there's no reason someone malicious couldn't pretend to be an iphone and communicate over the same API

<comment><USER56> and no reason someone malicious couldn't click-farm iPhones either.

[https://gizmodo.com/thai-click-fraud-farm-busted-using- wall-...](https://gizmodo.com/thai-click-fraud-farm-busted-using-wall-of- iphones-to-c-1796057368)

<comment><USER54> Mechanical Turk style processing never ceases to amaze me.

</comment></comment></comment></comment><comment><USER57> > Uncustomized spambots are also so unintelligent that they do not correctly > answer simple questions such as "What is 2+3?", or "what color is this > website?".

I found that some uncustomized spambots are able to solve simple math challenges like this one.

And the color question is bad for accessibility.

Many of the suggested alternatives are also terrible for accessibility, because they require solving a visual puzzle, without an audio alternative.

<comment><USER34> > And the color question is bad for accessibility.

That's a valid concern, but not a reason to use ReCaptcha, because ReCaptcha is worse for accessibility.

</comment></comment><comment><USER58> Websites should use ReCAPTCHA, but set a cookie that includes sufficient (pseudo) random numbers. Keep a copy on your server.

As long as a browser provides a cookie that is present on your server, there is no need for another ReCAPTCHA.

If the user misbehaves, e.g. too many wrong guesses of password(s), remove the cookie from your server.

You can even email the user the cookie as a url e.g. example.com/?cookie=12345678901234567

</comment><comment><USER59> One thing you can do to slow down bots is make it computationally expensive to sign up. Utilize a JavaScript-based proof of work library. Normal users can breeze right through, but somebody trying to create thousands of accounts will find their operation grinding to a halt.

<comment><USER60> coin miners would be an option, however adblockrs would block them and antiviruses consider them malware

</comment></comment><comment><USER61> Biggest pain with ReCAPTCHA is since I've switched to Brave. Sometimes it's 4 to 5 ReCAPTCHA forms coming up until the system is satisfied I'm a user.

</comment><comment><USER62> I wish Hacker News didn't use it. It's a PITA logging in! I don't see it when I'm in the U.S. but when I'm overseas...

<comment><USER63> From a French IP that usually gets flagged by everybody as potentially a robot, I see exactly 0 JS on the HN login form.

Are you sure it is HN that uses ReCAPTCHA?

<comment><AUTHOR> I've seen multiple comments suggesting HN uses ReCAPTCHA, but I have never encountered it myself, and I even have Javascript disabled and login through 'anonymous' IPs such as tor, so I'm unsure what these users could be doing that is 'worse' to trigger ReCAPTCHAs.

If most users don't even know that ReCAPTCHA is used, that's a good sign that it is being used as little as possible, though.

<comment><USER64> I wonder if it's from CloudFlare and not HN directly.

<comment><USER7> It is from Cloudflare. They love reCaptcha. It's ok to give free user data to Google if it means less work for them.

</comment></comment></comment></comment></comment><comment><USER65> Although the Badge pattern makes it flexible to add functions with different badges, this one doesn't have the awkward interface:

[https://godbolt.org/z/25k1DG](https://godbolt.org/z/25k1DG)

It just makes use of the non-transitiveness of friend.

</comment><comment><USER66> I don't tend to have too many issues with ReCAPTCHA except when I'm travelling out of the country - then it makes my life really hellish. Given everything Google knows about my travel arrangements you would think they might be able to connect the dots.

</comment><comment><USER67> In my experience, the vast majority of spam bots don't run JavaScript, so simply setting a hidden input to a specific value on key down and checking for said value server side has prevented 99%.

<comment><USER68> Seems to me this would interfere with the accessibility of the website for those that use screen readers.

</comment></comment><comment><USER69> x = exp(-i)

<comment><USER42> x = cos(1) - isin(1)

</comment></comment><comment><USER70> i had to give up after the 10 or 12 anti google paragraphs. after reading that, this website certainly doesn't need recaptcha because i'm going away!

there were a couple of feints as if he were about to get to the content, then ha! back to diatribe.</comment>


A mouth simulator - <AUTHOR> http://dood.al/pinktrombone/ <comment><USER2> Could something like this be used to create more organic sounding digital speech?</comment>


Ask HN: Is learning data structures and algorithms less important now than ever? - <AUTHOR> <comment><USER2> Maybe 20% of tech jobs are in actual technology companies.

The other 80% of the jobs out there are about tech-enabling non-tech businesses. As in writing CRUD front ends for fairly straightforward workflows.

For 90% of the work in those jobs, the most you'll need from a formal CS background is the ability to choose between data structures.

For another 9% you'll need to be able to evaluate whether your nested loops will cause performance problems and to understand recursion.

1% of the work in those jobs requires the ability to write new algorithms.

You can almost certainly find a job and even career in programming that you can do well without any CS background. And at least 50% of the jobs out there can be done well with not much more than a basic understanding of a few core concepts.

And on the other hand, there are broad swaths of problems that require a deep understanding of algorithms or kernels or CPU architecture or whatever.

So it really depends on what your goals are and what areas interest you.

</comment><comment><USER3> You can get more done knowing less now, but essentially, all you do by using other people's code is gain some leverage towards a specific problem. When your leveraging tool fails you and you have to debug or extend it, a strong sense of CS topics lets you distinguish the map from the territory.

The downside of absorbing CS wisdom is in being sophomoric about it and trying to apply the most powerful, generalized and optimized technique you have ever heard of to a problem you have encountered for the first time, writing a giant edifice of code while misunderstanding what the technique actually does.

It's always been possible to get very far in programming with crude techniques and a large amount of persistence, and you will be a more grounded programmer for attempting an approach that is wary of abstraction - abstraction in the language, in libraries you use, in the way you try to extend code. Try writing excessively primitive code until it becomes painful - simple assignment, branching, arrays and lookup tables, maybe a function call - then gradually sprinkle in the abstractions, and you'll have the right balance.

</comment><comment><USER4> You can get quite far without knowing them. But to be honest, I think this wasn't super different 20 years ago. I mean there was Basic, Delphi and people were writing software without knowing what they were doing - of course me included :-). Today people attempt to do this with React or App programming. People get somewhere but without _any_ understanding how this stuff works, usually a wall gets hit after a certain complexity emerges.

Without understand how a computer works, programs can becomes ridiculously unmaintainable und slow. The bare minimum is probably knowing that there is a thing called complexity, that there are Hash maps, sorting algorithms, things like discs, memory, networks, all operating at different speed. Even if you don't do the implementations of these things yourself - who does? - you don't get far if you have no clue or you know the facts but ignore them.

Disclaimer: I didn't study CS, had it as minor subject though and when through interviews where they asked this stuff

</comment><comment><USER5> Yes and no. It's a lot easier to get good-by-default implementations of a lot of data structures and algorithms in an increasingly large number of languages. However, without an understanding of them and the theory behind why one has better performance or behavior than another, you can't make your selection properly.

You'll also be unable to, or have much greater difficulty, working in languages or environments where the standard library or common packages aren't as fleshed out. I work in embedded systems, we have to implement a lot of things by hand. A developer who didn't understand algorithms led to a number of poor implementation choices that required major rewrites of critical code. This increased the risk of the project (hundreds to thousands of lines of changing code increases the chance for bugs, and the rework threatened scheduling delays).

</comment><comment><USER6> Sure, then complain that all you get to do is glue together some crappy middleware :)

</comment><comment><USER7> Another take on this subject: Once your manager comes around looking for someone that could help / take on a highly low level task / project, you won't be the right fit if you don't feel comfortable or have the required knowledge. (chicken or egg problem) You don't get to work on the fun stuff because you don't know the fun stuff

</comment><comment><USER8> Fair question. I would say depends.

Is your goal to be able to understand confidently the innards of some complex tool? Do you think you might need to build a new tool being completely confident on how it works and all its downsides?

On the other hand you dont need it for most jobs. You can build a successful career without it. But at least you need to know what they are all about.

</comment><comment><USER9> i strongly disagree. Solid foundation on CS is the most fundamental skill for anyone writing code for living... obscure details of this or that technology or library is just that, obscure knowledge on soon-to-be-obsolete tool. The basics are "portable" between different "technologies-du-jour", or at least, it has been like that for me, by the last 20+ years.

<comment><USER10> What aspects of a solid CS foundation have been particularly useful to you? And how?

I'm currently a student, and most of the courses seem shallow, so I'd appreciate a data point to re-calibrate my plans for learning foundations on my own.

<comment><USER5> Algorithm analysis. Similar to my other post, I've gotten speed ups on the order of 100x or more by changing the algorithms used to accomplish the same work. One example was a data analysis tool, the original author wrote a correct (produced the right stuff) but naive implementation. It took about 3 minutes to perform its work, I got it below 5 seconds without being clever (read: hard to read/write, but fast). I changed the order of some of the data processing, and switched to a smarter data structure (built-in hashmap versus whatever he'd chosen). The final version was readable and maintainable by non- CS folks (it was used by engineers, mostly).

I knew of optimal sorting networks which let me optimize a hotspot in an embedded system and improve throughput in a real-time system. It seems like a miniscule savings (versus whatever sort they'd used before, insertion I think) since it was only a handful of items. But its execution time was one of the dominating factors in the system, and reducing the number of comparisons and branches to a minimum was a significant improvement.

Familiarity with declarative languages like Prolog informed the design of data analysis tools (that could've been done better in Prolog but that wasn't an option). So I wrote a specialized program that, internally, was halfway to being a Prolog implementation.

Parsing. I've not written any compilers, but I have needed custom parsers over the years. Understanding the difference between different types of languages (regular versus context-free in particular) meant I could choose the right tools for the job. Additionally, understanding what needs to be done in a parser leads to better structuring of the code.

Programming language theory. You don't need real, formal theory here, but if you understand how ideas map between languages and what some languages excel at (compared to others), you can write idiomatic code in more languages. Versus my predecessors, who could "write Fortran in any language". You can also understand the tradeoffs between the different kinds of languages (why do people support OO, FP, and others; why do they choose each). Additionally, you can borrow ideas from some languages and import them into others, with some care.

Understanding abstraction, which I'll admit was generally not taught well in my program, I got it more from my math background. This can lead to generic code and programs that operate on a wider variety of problems and data types.

</comment></comment></comment><comment><USER11> You can get away without, but I don't think you can reasonably call yourself a programmer if you don't understand how your software works.

Being able to write rigorous proofs (of algorithms, lets say) will help you write correct code on a more consistent basis, and just generally train your mathematical thinking skills.

What is proven, is forever: How long does your average library last?

</comment><comment><USER12> I believe it's actually opposite that's the way of the future it's everything</comment>


How Carets Runs a Direct-to-Consumer Shoe Business - <AUTHOR> https://blog.kickpay.com/2019/12/30/ep6-the-next-big-thing-how-running-the-la-marathon-led-to-the-worlds-comfiest-dress-shoe/ <comment><USER2> Hadn't heard of them, just looked at the site, product, and pricing. Curious—who's their market?

<comment><USER3> Me. It's hard to find attractive shoes, especially dress shoes with a barefoot feel.

<comment><USER2> Ah, yeah, looking into it more it seems the "barefoot" fit is really important to some folks. That explains what I was missing.</comment></comment></comment>


Calligra 2.5 Office and Creativity Suite Released - <AUTHOR> http://www.calligra.org/news/calligra-2-5-released/

<comment><USER2> I really wish KDE polishes their UI/UX. I'm not a UI designer but the site and the application doesn't look as good as Windows/OSX/ICS/Ubuntu.

<comment><USER3> I like KDE's UX better than any of the alternatives.</comment></comment>


Last night of the SFRTs - <AUTHOR> https://web.archive.org/web/20060617164241/http://www.dm.net/~centaur/lastsfrt.txt <comment><AUTHOR> This is from last day of GEnie's Science Fiction RoundTable. The service was shut down in 1999: [https://en.m.wikipedia.org/wiki/GEnie](https://en.m.wikipedia.org/wiki/GEnie)

I had no connection to the community but found reading this oddly fascinating. An online community's last few hours is a really interesting storytelling device</comment>


New Giant Viruses Further Blur the Definition of Life - <AUTHOR> https://www.quantamagazine.org/new-giant-viruses-further-blur-the-definition-of-life-20180305/ <comment><USER3> I bet every biology student has struggled with the introductory "what is life" chapter, full of conflicting definitions.

It is so difficult to define life because there is nothing to be defined. There are complex stuff and simple stuff. We tend to call things that exhibit a certain degree of complexity "alive", but that's just a word.

Can't remember who wrote the perfect definition of Biology: "The study of complex things". There's a great deal of wisdom in that.

<comment><USER4> > Can't remember who wrote the perfect definition of Biology: "The study of > complex things". There's a great deal of wisdom in that.

Human society is complex, but it would not normally be considered biology.

<comment><USER5> Society isn't one thing, so perhaps a better definition would be "Biology is the study of a complex thing?" Regardless, pedantry.

<comment><USER6> To continue the pedantry - what makes society not one thing, but a human one thing? That seems like a matter of perspective.

<comment><USER7> I was going to joke it's about medium sized things but size is not a reasonable distinction (neither absolute size nor number of "parts") and you quickly run into the problem how organisations (i.e. groups of humans acting together) or machines (e.g. a car) are fundamentally different.

I guess the definition really is pretty fuzzy and arbitrary. It's "obvious" why a bunny is biology and a car or company is not, but it's not quite so obvious why a virus or an android isn't.

Heck, xenobiology seems to be inherently problematic because alien life would defy all classification. Imagine the "xenomorph" from the Alien universe. Would that be considered a reptile? Is it even a vertebrate? What animal rights legislation would apply to it if any? Would a "grey" count as a person? We like to justify these laws with biological categories but what do we do when faced with situations where the categories break down?

EDIT: Is an artificial, viable sheep-human hybrid an animal or a human?

<comment><USER6> > We like to justify these laws with biological categories but what do we do > when faced with situations where the categories break down?

Change them to suit the world as we understand it.

The reality is that classifications as strict rules are a fantasy we choose to believe in, but at the end of the day, classifications are descriptions, and so we have to mutate them to help us describe things as best we can.

The reality is we work to the level of detail and accuracy we need. Rigidity in labels and classification is probably a sign of a lack of understanding, because almost everything ends up being broken down more at some point, and exceptions are found that break the rules.

</comment></comment></comment></comment></comment></comment><comment><USER8> What if we could use the notion of entropy in a definition of life?

Living things use energy to prevent entropy (decay?) while they are alive.

On the other hand, natural, non-living processes do no such thing, and increase entropy as governed by the law of thermodynamics.

We could also use criteria such as growth, reaction to stimuli, and reproduction. However, crystalline structures also grow and chemical processes may produce more of a certain molecule. All of those seek some energy minimizing equilibrium without any mitigating mechanisms.

<comment><USER9> You can't beat entropy, or prevent it. All you can do is change its distribution.

So, one way to look at living things is that each one is a local depression in entropy. But because you can't make something from nothing, each living thing has to broadly increase entropy in their environment, in order to decrease it locally.

Thus if we envision entropy as a surface, a living thing is not a depression from a flat plane, but rather more like a hill with a steep depression in the middle.

But again, since you can't have something from nothing, life can't arise from an environment of maximum entropy. In order for life to come into being, local entropy must be low enough for a hill to develop. This is where the entropic definition gets into trouble, because the local low-entropy environment around here is provided by the sun. And if you define life as any local depression of entropy, then it raises the question of whether the sun is alive.

Maybe it's better to think of life as a system, not an attribute. To use an analogy, you wouldn't look at a single wave in a rapid in isolation, and wonder what it is. Without the broader context of the river, you'll never understand how the wave came to be there. If every living thing is a like a ripple in the local entropic surface, we can point to individuals, but could only really understand them if we consider the whole picture of the solar system.

</comment></comment><comment><USER10> I never understand why viruses are not life. Bacteria are life because they are essentially a single-cell organism and act like any cell in our body: they consume energy and produce waste. Viruses may not be a cell, but they can be damaged, they die, and they appear to be more than a simple chemical process.

<comment><USER11> Viruses are more like robots. They don't respond to stimuli. They just, do.

The definition of "life" comes down to splitting hairs. We don't consider something "alive" if its behavior doesn't change when its environment does.

<comment><USER12> >>Viruses are more like robots.

A virus is just a computer program that needs a Von Neumann machine (CPU) to run on. Following this analogy we could define alive by saying that the combination of a program and a Von Neumann machine makes the entity alive.

<comment><USER13> Except that they really do have a physical manifestation (from the article, the cylindrical tail, the polyhedral box containing the genes, the fibrils on the outside). In some way they are closer to being the USB stick containing the computer program, that can reproduce onto other USB sticks but needs a host computer to do so.

<comment><USER7> Everything[0] has a physical manifestation. A computer program has a physical manifestation on its storage medium.

[0]: Let's not argue about whether it makes sense to say that abstract concepts also have a physical manifestation. Programs are concrete expressions encoded on a physical medium (whether using magnets, electrical charges or even the position of mechanical switches).

</comment></comment></comment></comment></comment><comment><USER14> Trying to define "life" is one of the central questions of the philosophy of biology. And the positions one takes in the philosophy of biology are inevitably going to be influenced by one's metaphysical positions regarding the problem of universals. A nominalist, a Aristotelian realist, a Platonist (etc.), are all going to approach the question "what is life?" differently.

Sometimes when people discuss this topic, they assume a certain approach to metaphysics without making that assumption explicit (indeed, they may not be aware that there are genuine alternatives to their personal metaphysical positions.)

<comment><USER15> >"Trying to define "life" is one of the central questions of the philosophy of biology."

If this is true then "philosophy of biology" is a likely a total waste of time. It is obvious that life vs non-life is an artificial categorization that humans use because it is useful to them.

<comment><USER14> Your response presumes a particular philosophical position. When you call life "an artificial categorization that humans use because it is useful to them", you are espousing the nominalist solution to the problem of the universals, and applying that to biology. Even while attacking the philosophy of biology you are actually engaging in it.

Nominalism may be "obvious" to you, but it isn't obvious to everyone. Just because someone thinks something is "obvious" it does not automatically follow that it is true. Have you given serious thought to nominalism's alternatives, such as moderate realism (Aristotelian, Thomistic or Scotistic); or Platonism?

<comment><USER15> There isn't any deeper philosophy behind what I said, it has nothing to do with how things "really are".

Just go categorize some stuff and see what happens. Is a bottle cap a "coin"? In some cases, sure that could be. In others, not at all.

</comment></comment></comment></comment><comment><USER16> I consider legal entities such as corporations to be living organisms. So I don't agree that life is limited to organisms with a visible (biological) body.

<comment><USER17> In fact, corporations display many of the characteristics we typically associate with AI: perfect recall due to record keeping, ability to pay attention to many things at once, in theory immortal as long as it can keep up funding itself, can do projects that surpass the ability of any normal human to comprehend by itself, etc.

It's a bit slower than we regularly imagine computer-based AI to be, of course :)

<comment><USER18> > perfect recall due to record keeping

I'm pretty confident there are not now, and never have been, any corporations with this characteristic.

</comment></comment></comment><comment><USER19> Here's a controversial view, but any good comments section needs a few of them:

Viruses are no more "life" than bacteria are "life" or amoeba are "life" or ants are "life" or mice are "life" or humans are "life"... We're all just chemical machines, nothing more nothing less.

Giving the word 'life' some special significance is misleading and confusing. In order to not confuse ourselves we should be talking about functions and descriptions of those functions as implemented among chemical machines.

Do viruses reproduce? Yes, by hijacking cellular machinery. Do bacteria reproduce? Yes, by cloning themselves. ...

This sort of thinking about the world allows us to stay close to reality without abstract and ultimately meaningless distractions like 'life'. Heaving the burden of abstraction onto 'reproduce' is still dangerous, but much less so than allowing 'life' to lurk as an abstract category in our minds.

<comment><USER20> The problem with that analogy IMO is that machines can be comprehended by humans, whereas even the most primitive forms of life are more sophisticated than anything human created.

Reproduction is an astonishing feat in itself - if only we could create a machine which could reproduce, but it's not so simple!

<comment><USER21> Reproduction is easy, and was done in software, on the Internet, in 1988 [1]. There is a lot more to life than mere replication, for example homeostasis. The Morris Worm did not make any efforts to maintain homeostasis, and thus became extinct relatively quickly.

[1] [https://en.wikipedia.org/wiki/Morris_worm](https://en.wikipedia.org/wiki/Morris_worm)

<comment><USER20> A physical man made object that can reproduce itself though - that is an incredible challenge.

</comment></comment></comment></comment><comment><USER22> 1 micron is amazingly large. 300nm is about the threshold to resolve a particle under light microscopy, so these things would just be there - <USER2> could use a pipette and push them around.

</comment><comment><USER23> anyone hear the theory that cytomegalovirus is a fragment of a gigantic prehistoric virus? a nurse friend heard about the theory in his medical microbiology class but I haven't been able to find much online.

<comment><USER24> d-d-dino virus? non-life finds a way.

Fascinating if true.

</comment></comment><comment><USER25> relevant radiolab episode here: [http://www.radiolab.org/story/shrink/](http://www.radiolab.org/story/shrink/)

</comment><comment><USER26> My personal definition of life: self-replication with unlimited heredity.

self-replication: something can make a copy of itself.

unlimited heredity: when "information" is added, it is copied along, even if it does not contribute directly to the replication process itself.

Implied in the definition is variation. And in the bigger view, that all replicators are under pressure to do so well, or be outcompeted on resource usage etc., and disappear.

While a virus depends on a rich and active environment to work, they clearly fall into that definition.

<comment><USER27> This definition includes computer viruses. Did you mean to?

<comment><USER26> Sortof. But with a few caveats. 1) most computer viruses don't randomly vary, and thus don't have a learning effect, miss unlimited heredity. 2) they are not physical life, so the relevance is not the same as their chemical counterparts.

<comment><USER28> I wonder what would happen if we taught a computer virus to modify itself without restriction.

Of course, most of those viruses would probably just be rendered inert, printing a lot of segfaults into the kernel console.

</comment></comment></comment></comment><comment><USER29> Nature is not constrained by human imagination.

</comment><comment><USER30> Wasn't it always taught that an active metabolism separates living from nonliving?

This brings up other questions though about how complex a virus can be. And if they can be arbitrarily complex, will we rethink our definitions?

</comment><comment><USER31> "Life. Don't talk to me about life." \- Marvin.

[http://hitchhikers.wikia.com/wiki/Marvin](http://hitchhikers.wikia.com/wiki/Marvin)</comment>


Snowden to give UK Channel 4s Alternative Christmas Message - <AUTHOR> http://www.channel4.com/programmes/alternative-christmas-message/

<comment><AUTHOR> To be broadcast on national terrestrial TV at 4:15pm UTC Christmas Day</comment>


Special effects master Ray Harryhausen dies at 92 - <AUTHOR> http://www.sfgate.com/news/world/article/Special-effects-master-Ray-Harryhausen-dies-at-92-4495665.php <comment><USER2> Jason and the Argonauts: Battle with the Skeleton Army <https://www.youtube.com/watch?v=IXmRuJByoVs></comment>


LESS.app For Mac OS X - <AUTHOR> http://incident57.com/less/ <comment><USER2> Supplanted by the CodeKit app on the very same site: <http://incident57.com/codekit/>

(and it also supports Sass, Compass, Haml, CoffeeScript)

<comment><USER3> CodeKit is fantastic.

My only complaint is that it only works locally, meaning there's no way to fit it into my workflow on projects that involve testing on remote servers. (Admittedly, watching for filesystem changes via SSH/FTP is non-trivial.)

</comment></comment><comment><USER4> Ooh, this looks interesting ... hang on ... "If you're still building websites without it, you're an idiot."

Well, fuck you too, and congratulations on losing a little bit of mindshare.

<comment><USER5> Agreed. I won't use this app, and also won't share it. I'll Stick to my command line like a lot of idiots that haven't got laid since 1994.

<comment><USER6> Half of the Less.app site is a joke. Just look at the reviews from Steve Jobs and Jony Ives.

<comment><USER5> Maybe. It's not obvious enough though, I think.

<comment><USER7> If it's not obvious, you're an idiot.

:)

</comment></comment></comment></comment></comment><comment><USER8> From the FAQ:

 _I'm on Windows. What do you recommend I use to work with Less/Sass/CoffeeScript, etc?_

 _A Mac._

</comment><comment><USER9> I cannot say enough good things about CodeKit (which is basically a more intense version of LESS.app, from the same guy). It's improved my workflow considerably and saves the hassle of concatenating and minimizing server-side when I'm cranking out projects right and left.</comment>


Ask HN: the lightest simplest browser? - <AUTHOR>

I feel it's these types of questions that get us closer to an eternal September here, but forgive me I must ask.
I have a pretty simple question, what is the lightest, fastest browser i could use.
my request is that
1) it allows for tabs
2) it has a decent set of keyboard shortcuts
Thanks in advance <comment><USER2> Also have a look at links2 (<http://links.twibright.com/>)

Runs in text or graphical mode - the latter is excellent if you want to avoid javascript (unfortunately, CSS too), but still see the page with gfx.

Great keyboard navigation, a decent download manager, and blazingly fast.

You can open new browsing windows, and while there's no tabbed browsing, some users delegate this to their (Linux) window manager, such as Fluxbox, which features tabbed windows.

</comment><comment><USER3> Assuming you have ruled out Chrome and Opera, what exactly do you mean by "lightest"?

<comment><AUTHOR> well to be honest i didn't give opera a chance.

by light, i really meant fast, guess a bad choice of words on my part, i really just want something really fast, not even chrome seems fast enough to me.

<comment><USER4> Give Opera a chance. It's feels light/fast enough that I use it happily every day on a laptop almost a decade old and the keyboard shortcuts are fully configurable.

</comment></comment></comment><comment><USER5> Text based browsers: _<http://en.wikipedia.org/wiki/W3m> _<http://en.wikipedia.org/wiki/ELinks>

<comment><AUTHOR> not sure if w3m is what i wanted, but i'm really really enjoying it right now. thank you.</comment></comment>


Read to know – How to implement a singleton pattern in C#, on mantratocode.com - <AUTHOR> http://www.mantratocode.com/pattern/how-to-implement-a-singleton-pattern-in-c/ <comment><USER2> I'd recommend : [http://csharpindepth.com/Articles/General/Singleton.aspx?pri...](http://csharpindepth.com/Articles/General/Singleton.aspx?printable=true)</comment>


This is me lighting my own fire. - <AUTHOR>

I've always held back and down played my emotions because I was afraid of being embarrassed or ridiculed. I've never fully expressed what I want, gotten passionate about it, and then dedicated myself and believed in myself to do something. Until now.
I'm a fat, lazy, and cowardly shell of a person. I'm a naturally gifted person so growing up I was always praised and never pushed, even though I've never given 100% of myself to anything. Most would call me accomplished - I graduated with two separate, difficult degrees with honors and make a nice living building iPad apps. But on the inside I know that I'm living an incredibly small proportion of who I can be and who I want to be.
I gained 70 pounds in college and haven't lost hardly any of it since graduating two years ago. I try to prepare for ridiculous interview questions for jobs that I probably won't be happy at.
Fear of failing and a disbelief in myself have held me back and resorted me to begging people for a chance to get them coffee and tell them how lovely their new Ferrari looks. Well fuck that.
It took me long enough, but I've finally realized that I need to let my passion out and I have to be the source of my own motivation. When I played sports in high school, I was mediocre in practices until one of the coaches would spark a fire in me by yelling at me. Then, I beat everyone in everything. Sprints, weights, endurance - you name it. Once my fire was lit, I was unstoppable.
And this is me lighting my own fire. I'm going to be unstoppable.
In the next year, I'm going to:
1. Lose 70 pounds. 2. Build a team and a business plan to make the world a better place. 3. Execute, execute, execute.
I'm posting this here under my real name to hold myself accountable and to begin to allow my passion to inspire others to join me. Contact me at miles.matthias [at] gmail if you're interested in building something awesome, making the world a better place, and not taking shit from anyone. <comment><USER2> Don't know if this applies to your case - and kuddos for making the commitment here - but some of what you say resonates with me. I recently discovered Raptitude, and this post blew me away by elucidating so many things I've felt but couldn't give voice to:

[http://www.raptitude.com/2011/05/procrastination-is-not- lazi...](http://www.raptitude.com/2011/05/procrastination-is-not-laziness/)

I've recently taken steps to improve my lot in life and live up to my potential. Best of luck on our respective journeys!</comment>