Gorgeous Unreal Engine 4 brings direct programming, indirect lighting - <AUTHOR> http://arstechnica.com/gaming/2012/06/gorgeous-unreal-engine-4-brings-direct-programming-indirect-lighting/

<comment><USER2> Here's the video of the developer walktrough: [http://www.gametrailers.com/video/exclusive-development- unre...](http://www.gametrailers.com/video/exclusive-development-unreal- engine/731871)

It's stunning that this runs on a single Nvidia GeForce 680GTX in an editor without any visible lag at all. I wonder why tesselation wasn't enabled though, it may have been because of performance issues.

<comment><USER3> Wow. When he popped out of full screen and revealed that he was in his editor - _mind blown_.

<comment><USER4> I was more impressed by the live recompilation. It didn't interrupt his gameplay at all.

</comment></comment></comment><comment><USER5> Coming from a film background, it is clear there is a lot of conflation happening in these articles between light that 'behaves' like it would in the real world, and 'good' lighting. That is why you get quotes like in this article 'I light my scene by dropping a sun in'.

In reality, there is a whole lot more to lighting then simply having lights and materials that behave naturally. Sure, this gives you more realistic looking results out of the box, but it doesn't mean you don't have to sculpt and massage and tune your lighting to achieve the artistic result you want, especially for interiors, or scenes with characters in the foreground.

<comment><USER6> > In reality, there is a whole lot more to lighting then simply having lights > and materials that behave naturally.

For example? (genuinely curious)

<comment><USER7> Achieving a specific mood or "feel" for an environment requires someone who is very sensitive to lighting, color, and architecture, and who has knowledge of techniques to achieve certain looks.

For example, if you want an ominous feel to a cathedral, you need to know what light colors are more ominous than others, what direction of lighting (maybe bottom up, rather than from the top), what types of lights...many dim point lights? or single directional lights?, if light scattering through a medium like fog would look good, etc etc. And also keeping in mind functional aspects like, can the player still see items or things he needs to do, even with all of this specific lighting.

Another example might be that while lighting and materials might behave naturally, sometimes they still behave in undesireable ways. Suppose the angle of the sun reflected off of a reflective surface right into the player's face right while he was at a specific location where he needed to do something. If you can't move the surface, can you shift the light? Will that effect the scene much? Maybe now that you shifted the light, the rays of light that were going through a tree branch beautifully are not there any more.

<comment><USER8> Getting back to UE4, if its lights behave more like real-world lights it may allow game artists to translate real-world lighting knowledge into games or let them spend less time working around kinks in the engine.

<comment><USER7> Definitely true. We draw our experience on lights from the real world, so the closer an engine is to the real world, the faster the artists can prototype and test lighting setups.

<comment><USER9> As a still photographer who sometimes works with multiple off-camera lights (strobist-style) I would love to be able to experiment in software to see a very close approximation of how a setup would look.

<comment><USER7> I'm not aware of any photographer-centric photorealistic 3d apps, but that sounds like a potential niche business! Might be worth exploring.

<comment><USER10> I have a plugin for Cinema 4D called "HDRI Studio Kit" that is basically a set of predefined studio lights and lighting setups. Makes it very simple to light objects using traditional photographic methods. You have various tent setups, soft-boxes, umbrella lights, etc...

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> The bit where he made a change to the source code, let it recompile on it's own, got a little alert in the editor when it finished and then had the change apply all without restarting the engine...WOW!

<comment><USER12> Unity does this now. You can recompile a file while your game is playing in the editor, and the running code is replaced. And I almost never use it -- I should figure out why that is.

<comment><USER13> For C++? That's the impressive part.

</comment></comment></comment><comment><USER14> I find it hilarious that (presumably) my computer could render that smoothly but the video itself stutters.

<comment><USER15> "The UE4 demo is running on PC, specifically an Intel Core i7 processor with an NVIDIA GTX680 and 16GB of RAM"

[http://www.eurogamer.net/articles/digitalfoundry-vs- unreal-e...](http://www.eurogamer.net/articles/digitalfoundry-vs-unreal- engine-4)

</comment></comment><comment><USER16> Incredibly stunning tech. Now, I'm gonna go fantasize about an open-source version of this with updates every month (instead of every 3 years) -- as a compromise, I could do without the whole hot-reloading IntelliEditor shebang and consoles support -- a concentration on PC hardware that "will be commodity in 2 years, on mobile in 4 years, but can already be bought as state of the art now" (aka Kepler GPUs) would be enough.

To the pros: in your opinion, which of the many FOSS engines out there (many of which carrying a large legacy code-base of supporting soon-outdated modes of operation such as DX9 or lower or GL < 4.0) is the most likely candidate to offer a deferred pipeline incorporating lighting / particles such as we see here in this UE4 demo, at that level of performance and (expected but also demoed) robustness? Again, their "industry" tack (smart editor / console support, "we license only to pro developers" stance etc) would _not_ be required ... only the "metal".

<comment><USER16> Also I'm wondering about the lighting. Some folks on Twitter called it dynamic Global Illumination but here they just call it indirect lighting. I wonder -- are they still using an ambient light? More specifically, what's their GI algorithm? Might be something like the voxel approach sported by @icare3D but then it seems that one's just barely interactive, not as "real-time" as this stuff -- or well it might be on a GTX680 with a really-lo-res voxelization.

<comment><USER17> Voxels would be too heavy. I have worked with volume rendering before and it's way too expensive to do it at this scale (although their scene with the light passing through the volumetric smoke column kind-of defied this).

Some real-time version of PRT? "RTRT"?

<comment><USER16> Turns out it IS voxels! <https://twitter.com/Icare3D/status/211404654062481408>

(Of course the (fairly lo-res compared to your screen) voxelized geometry representation is not rendered directly to screen ala Atomontage or Gigavoxels -- the the GI is based on RT scene voxelization and approximate cone-tracing.)

Well that's awesome because at least there's (A) a paper for this (grokking it is another matter) and (B) seems like it runs really smoothly on next-gen GPUs!

<comment><USER17> lo-res being the key :) I can see how the voxel representation would be computed and made resident in GPU memory. What's amazing to me, that such a highly branched computation algorithm is now possible in real-time on a GPU! (we had branching available since SM 3.0 - but was "not recommended")

</comment></comment></comment></comment></comment><comment><USER18> This stuff is really fascinating. Does anyone have any recommended reading or general advice as to how to learn more about game/graphical engine implementation? I mean, from the ground up. (I'm a CS undergrad and the closest we've gotten to learning about this has been the computational geometry chapter in CLRS, but that was a mildly esoteric introduction to everything that is possible in this field.)

<comment><USER17> NeHe's OpenGL tutorials used to be a good (although now kind of outdated) starting point. Shaders didn't use to exist back then. But still a good start:

<http://nehe.gamedev.net/>

A game engine is much more complex than a graphics engine. Maybe somebody can point to guides for creating physics/sound engines and AI/gameplay stuff as well?

<comment><USER19> When I started learning I spent a lot of time reading NeHe tutorials. But modern OpenGL feels so far removed from those tutorials that it feels like learning a completely different API.

The best tutorials I've found for modern OpenGL is "Learning Modern 3D Graphics Programming" by Jason McKesson. The only problem I have with these tutorials is that there aren't enough! <http://www.arcsynthesis.org/gltut/>

<comment><USER18> I really appreciate all of the responses and have already put many of the recommended texts on my wishlist. (And I just bought Game Engine Architecture as well, too bad you didn't post an Amazon affiliate link for yourself!) But I just wanted to point out that since this tutorial is free, I immediately checked it out, and I am absolutely blown away. This will certainly keep me busy until GEA arrives! Thanks so much, it looks to be a great starting point for learning more.

Edit: I also wanted to point out that in the author's preface, he discusses what he considers the downside to most introductory texts and tutorials in 3D graphics programming - presenting "fixed functionality" that allows newcomers to more quickly use the tools at hand by abstracting much of the foundational information away.

While this is certainly useful for experienced developers learning a new derivative technology on top of what they already know, I have always found this approach for introductory stuff frustrating. At the end of the day I may come away pseudo-understanding a "higher level" concept, but ultimately much has been abstracted away and I am left ignorant and, as the author says, "Programming thus becomes akin to magical rituals: you put certain bits of code before other bits, and everything seems to work."

I wish I knew of more texts like this for other fields where over or premature abstraction could endanger comprehension. (For example, I know I would like to see a similar approach taken to other complex topics, like networking.)

<comment><USER19> Hey, glad I was able to give you some useful information! If you ever find yourself in Toronto, say, for an indie game jam (<http://www.tojam.ca/home/default.asp> hint hint), you can buy me a beer ;)

If you're looking for something lower level, the book you probably want is the white book - Computer Graphics: Principles and Practice. I have the 2nd edition from 1992, which is the still the standard intro graphics textbook for many CS departments. Though Amazon says there will be a 3rd edition coming out at the end of this year!

[http://www.amazon.com/Computer-Graphics-Principles- Practice-...](http://www.amazon.com/Computer-Graphics-Principles-Practice- Edition/dp/0321399528/ref=sr_1_2?s=books&ie=UTF8&qid=1339214116&sr=1-2&keywords=computer+graphics+principles+and+practice)

Another highly recommended book (also recommended in another comment here) is Real-Time Rendering, but I've only used bits and pieces from this one, so I don't know how good it is for folks just starting out. Still probably one you'll want to add to your shelf if you continue on in the field.

[http://www.amazon.com/Real-Time-Rendering-Tomas- MOller/dp/15...](http://www.amazon.com/Real-Time-Rendering-Tomas- MOller/dp/1568814240?tag=realtimerenderin)

Oh, and also also, head over to YouTube with some snacks and a drink, sit down, and watch the weekly Overgrowth game developer videos from Wolfire Games. It's both inspirational to see what other people are doing, and a great demo of concepts that you'll read about in GEA, such as animation blending: [http://www.youtube.com/playlist?list=PLA17B3FAA1DA374F3&...](http://www.youtube.com/playlist?list=PLA17B3FAA1DA374F3&feature=plcp)

</comment></comment></comment></comment></comment><comment><USER20> The graphics stuff is amazing, but I'd be also interested to know how they achieve hot-loading with C++. It seems to me that Erlang's philosophy (or FP in general) would be of great advantage here - just have the engine handle entity state and make all of the game world manipulating functions referentially transparent. This way one can swap code between world "ticks" (or update cycles) without worrying about breaking something.

I don't know how efficient something like that would be though; somebody mentioned cache issues with lumping heterogenous properties together, but maybe this could be optimized behind the scenes.

</comment><comment><USER21> At this point, seeing this kind of extremely high production value animation used to enact this kind of fantasy scene instantly makes me think of hackneyed backstory and plot focus-group optimized to death to appeal to slightly dull 14-year-olds. You don't throw out that sort of development budget to try out something with even a hint of narrative experimentalness, and you can always rely on there being kids with disposable income who haven't yet been saturated with cliches to the point of fatigue.

"This looks very well done, it's probably bad" is a strange heuristic to have.

<comment><USER22> _"This looks very well done, it's probably bad" is a strange heuristic to have._

It's the same for Hollywood movies, isn't it?

When the producers want to focus your attention on how expensive and uniquely complicated a game or movie production was, that's a sign that the actual content was a secondary concern.

</comment></comment><comment><USER23> We've certainly come a long way since Demon Attack and Pitfall! :)

I remember 23 years ago, trying to do 'ray tracing' on my Amiga would take... hours for one frame. It looked pretty good, but... wow. This is jaw-droppingly cool.

</comment><comment><USER24> Why are people getting so excited about realtime global illumination and code hot-swap? CryEngine 3 already supports both, and has done so for the past year or so.

Realtime global illumination has been possible for the past few years. I believe Crytek was the first studio to make a game engine with support for the same (www6.incrysis.com/Light_Propagation_Volumes.pdf)

And the code hot-swap feature in the freely-available CryEngine 3 SDK isn't just for Lua, there's the CryMono project which adds support for hot-swapping C# scripts.

</comment><comment><USER25> I still have Unreal III in a box new that I bought in 2007 because they said they would have a Linux Client for it.

</comment><comment><USER17> Very Beautiful! The demo has a nice Skyrim/LOTR feel to it and the lava flows were just gorgeous! If that lava flow generation is completely procedural then I'm extremely impressed.

Writing a highly parametrized engine like that with JIT script compilation is plain awesome!

<comment><USER26> There is no script. It's all done in C++ with the project broken up cleverly into DLLs which can be reloaded at run-time without interruption. And I believe the lava flow is done with animated meshes.

<comment><USER17> I thought Kismet was pseudo-script (albeit with a GUI).

Edit: <http://www.unrealengine.com/features/kismet/>

<comment><USER26> UE3 had Unreal Script which was a custom scripting language and that has been removed from UE4. Kismet is more data than script. There's no JIT-compilation.

<comment><USER27> I have mixed feelings about this. UnrealScript had a lot of interesting ideas; particularly its module system (a kind of sideways inheritance where you could override an ancestor in a package, but not have to override all the descendants to have them inherit the behaviour - really handy for third-party plugins); the synchronous animation playback, effectively using continuations behind the scenes; how it handled synchronization of variables for client/server; and to a certain degree the "modes" that objects could be in, a way of switching behaviour of a whole slew of methods en masse.

On the other hand, object-oriented approaches are not great for simulating huge numbers of entities, where column-major array-based layouts are a lot more efficient.

I wonder if Kismet data-flow graphs may end up getting too complicated for their own good. They look like they could do with a textual representation. The obvious manual iteration involved in the creation of the orrery in the gametrailers demo video posted in the comments elsewhere here shows how painful this approach can be.

<comment><USER28> > a kind of sideways inheritance where you could override an ancestor in a > package, but not have to override all the descendants to have them inherit > the behaviour - really handy for third-party plugins

This sounds like the virtual classes that Tim were talking about ten years ago. (For the curious, the first programming language with virtual classes was BETA from Aarhus University.) They never made it into UnrealScript. His language research after that time was less incremental and sought to uproot almost everything about programming games. He moved away from object-oriented programming and more towards functional programming, specifically type theory. When I arrived in 2004, he was all about dependently typed languages, the big inspiration being David McAllester's Ontic. For a long time his plan was that his new language would be used to implement most if not at all of UE(n+1). Eventually that ambition had to be tempered by reality and thrown aside; I'm not sure if he's still working on programming language design.

<comment><USER27> Yes, it's been 10+ years since I last seriously looked at UnrealScript; I must have misremembered that feature, virtual classes.

</comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> In a way, this system reminds me of Light Table[1] in its ability to change some code/settings and get instant feedback. It totally changes the way people create content because it becomes so much more accessible and so much faster to see changes -- you can play around with a lot of different approaches more quickly, resulting in more room for experimentation.

Pretty amazing stuff! Really curious to see the tools that power all of this, as well as hear the architect's (or architects') vision for all of this

[1] <http://www.kickstarter.com/projects/ibdknox/light-table>

<comment><USER29> Being able to compile code and reload it while the engine is running is something that has existed in many engines for several years. What's cool here is that the engine keeps running while editing and compiling, then it auto- reloads and pushes the changes when compiled.

<comment><AUTHOR> Yeah definitely -- I guess what I meant was that there's now an interface to manipulate code, without having to touch code, and then it automatically goes through the process you mentioned

</comment></comment></comment><comment> mck- What a leap forward since Unreal/UT .. Hope the gameplay stays at least as good..

</comment><comment><USER30> I just have to say: Realtime rendering has finally become interesting to me. o_O

</comment><comment><USER31> I read about this is the latest issue of Wired. This looks amazing.</comment>


Python Performance Tips - <AUTHOR> http://wiki.python.org/moin/PythonSpeed/PerformanceTips

<comment><USER2> These tips are ok. Particularly the ones about profiling. I have a few more that I have found over time. First tho, as always a good algorithm improvement can help over any of these. Trick 0 is about that, the rest are more about performance.

0\. Generator comprehensions and list comprehensions are you friends. Sometimes one will make things like woah fast -- generators are lazy evaluation at its finest (if you don't follow that find a Haskell fan and ask them), this can be a real boon to your app. Sometimes you need the list comprehension. If you can't reason out which would be better for you, or reasoning suggests it shouldn't matter, try both anyway, just to be sure :).

1\. Function calls are expensive. Sometimes when you call a function a lot (on the order of 1000's of calls in a normal run) its better to bite the bullet and just unfactor it. I have seen improvements of 50% from just this.

2\. Dictionaries are fast. Classes are great, objects make life easy on programmers, but when dealing with large datasets, sometimes nested list/dict combos is way better than Objects (which are just dicts, but the syntactic sugar of objectness can eat up quite a few method calls, see #1 above). Also, sets are extremely useful and based on dicts (therefore also fast), so if the semantics of a set are ok with your needs, use them.

3\. Python regex is fast. It is usually frowned upon, but sometimes a complex regex is way better than pyparsing or python string methods. This saved my butt on one very memorable occasion -- #1 and #2 above didn't really do enough, so I replaced the core processing bits with a very complex regex and got the speedup.

4\. The gc module can really do a LOT of good. Short running scripts with lots of data can really be sped up by turning off gc (if you have the memory capacity). Even tuning generational parameters can really affect performance -- almost shockingly so.

5\. Any type of bit twiddling sucks. Take the time to do it in C and make the extension. This includes most IP ops -- if you are working with large lists of addresses look into dnet. Similar libs exist for a lot of other projects.

6\. __slots__ prove very useful sometimes. So does the struct module. Learn about both, they can really do wonders for your code in both readability and performance if used carefully.

7\. All of the above are wrong in some contexts -- they are not hard and fast rules, but guidelines, in some cases they work great, in others they don't help at all.

<comment><USER3> _5\. Any type of bit twiddling sucks. Take the time to do it in C and make the extension._

I recently had to do something like this and ended up using SWIG to generate the glue code. However, I noticed that my code had to spent considerable time (many iterations) in the C++ library to reduce overall execution time (but then it did by a factor of 20x-40x).

Do you happen to have some advice on how much speedup could be gained by replacing SWIG with handwritten wrappers?

<comment><USER2> My personal preferences are: Boost.python for C++[1], and Pyrex/Cython for C wrappers. Both make things pretty nice. I never really got into swig, so I'm not sure if there is noticable speedup betwen any of these. As for handwritten wrappers, I have not had any personal experience trying to eke the extra speed from not using a code generator type wrapper. hth

<comment><USER4> Swig solves a somewhat different problem than Boost.python/pyrex. Swig works better when you have an existing C++ codebase you want to call from python while making as few changes to the C++ side as possible, while Pyrex/Boost work better when you are writing a C or C++ module from scratch to be called from python.

</comment></comment></comment></comment><comment><USER5> I wonder if any of them still are current.

"At the time I originally wrote this I was using a 100MHz Pentium running BSDI."

<comment><USER6> Most of the tips here seem to be based on "do stuff in such a way that less Python C API calls making lookups get involved". Python API is the same Python API more or less so I'd assume most of the tips are current.

</comment></comment><comment><USER7> The timeit module (<http://docs.python.org/library/timeit.html>) is an invaluable tool for taking and comparing performance measurements.

</comment><comment><USER8> Python performance tip #1: don't use it if performance is a design goal.

<comment><AUTHOR> I wouldn't go that far as to avoid it altogether.

Python makes a great glue language even in applications where performance is important. You can write all the performance critical stuff in C and then glue it together in Python.

It's a pretty common approach and it works really well. Game engines often do this where the main engine is in C++ with scripting in Lua.</comment></comment>


What Compsci textbooks don't tell you: Real world code sucks - <AUTHOR> http://www.theregister.co.uk/2012/12/21/financial_software_disasters/print.html <comment><USER3> Good grief, what a vast amount of supposition backed up by very little evidence or insight. "People often write bad code." Yes, we know that. "There's more than one reason for this." Yes, we know that too.

The rest of the article is just random presumptions thrown together, for what reason I couldn't say. The best developers work in finance because the money's highest? Yeah, OK then... (To be fair, I think there is a qualifying "considered to be" in there somewhere) K is impenetrable? Hmmm. Haskell is an offshoot of ML and unreadable? Umm, factually wrong.

There's lots wrong with software at scale, and the pressures of financial institutions along with high budgets etc do produce an interesting set of problems, but this article offers nothing new about anything at all in the field.

As an aside, I already don't trust an author when they start off with a claim that textbook code is good code! It's almost universally accepted to be trivialised and simplified to the point of "don't do this in the real world" simply because it has to be. Some of the worst code I've seen has been in textbooks (and that's worst as in "dangerous" rather than as in "badly written and in need of refactoring/replacement/enclosing in concrete).

Edit to note: I think it's actually beyond incompetent journalism and in to plain lying to present a piece of code as evidence of Haskell unreadability when the link to that code on SO refers to it clearly as "obfuscated Haskell code".

<comment><USER4> > _Good grief, what a vast amount of supposition backed up by very little > evidence or insight. "People often write bad code." Yes, we know that. > "There's more than one reason for this." Yes, we know that too._

 _YOU AND ME_ know that. The title of the article is "What Compsci textbooks don't tell you", so I guess it is targeted at people still in college, to show them how things are in the "real world".

<comment><USER5> If they are in college and don't have enough interest to already know that, then they are the problem..

<comment><USER6> Yes. And there are a shitload of them.

</comment></comment></comment></comment><comment><USER7> Alan Kay classifies most real world code as pop culture (one of the most helpful categories for thinking about it):

[http://www.drdobbs.com/architecture-and-design/interview- wit...](http://www.drdobbs.com/architecture-and-design/interview-with-alan- kay/240003442)

 _Pop culture is all about identity and feeling like you're participating. It has nothing to do with cooperation, the past or the future — it's living in the present. I think the same is true of most people who write code for money. They have no idea where [their culture came from] — and the Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free?_

I appreciate this categorization on many fronts:

\- It explains how non-computer science majors have frequently succeeded. In music, you will never find a classical musician without formal training, but frequently find pop musicians who have little or none.

\- It shows how some of the same ideas keep being "discovered" by one group while another group sighs and exclaims "we knew about that years ago..."

\- Pop culture is inherently youthful, prone to extremes, sensitive to new trends. Sound familiar?

\- It explains a vast amount of decision making that goes on in industry. Rather than choosing the absolute best solution from a somewhat abstract technical perspective, a solution that is new and hip (or fits what I was doing in the hey-day of the bosses youth) is selected.

It is over simplistic to simplify real world code as "sucking" (though of course much does). It is like comparing Bach and <fill in your favorite rock/pop star here>. Both produce valid creations that accomplish something, but were made with vastly different intentions and for different purposes.

<comment><USER8> Real world code? All code sucks. And Alan Kay says something more damning than that: he says that the whole industry is a pop culture. We are living in the murky anterenaissance and precious few seem bothered by it.

<comment><USER9> You're right. Alan Kay does come across as rather bitter.

<comment><USER10> Alan Kay _has every right to be_ bitter. He's a great software architect, and came up with OOP as a perfectly reasonable policy for managing complexity. Now most people associate OOP with commercial garbage and that VisitorSingletonFactory nonsense.

If you came up with a set of really good ideas that were later bastardized and mutated into the sort of business horseshit that's destroying software, wouldn't you be pissed off?

</comment></comment></comment></comment><comment><USER11> > Haskell, an offshoot of ML. According to a friend in academia who's studied > it, it's "the Taliban version of ML," in which it's all but impossible to > write readable code

Can we stop posting this crap, please? TheReg was kinda edgy and cool circa 1998, but now it just seems a forum for trollish and content free articles.

</comment><comment><USER12> It apparently didn't occur to David that the reason why the code in banks suck might be because they don't actually employ "the very best".

I live in London, arguably the current world financial capital, but I'd have to be fairly desperate to work for a bank. (Not suggesting I'm the very best, of course, but rather taking banker money is like being a whore.)

<comment><USER13> > rather taking banker money is like being a whore

What if you're a barber who cuts a banker's hair, is that "like being a whore?" What if you sell software to a banker, or you're a nanny for the children of a banker? Also: do you have any respect for actual women who engage in prostitution?

<comment><USER14> _Further: do you have any respect for actual women who engage in prostitution?_

No.

Ditto for drug dealers.

<comment><USER15> I can understand having little respect for drug dealers as they can negatively affect communities.

But how on earth can you justify blanket-ly having no respect for the women themselves that engage in prostitution?

<comment><USER14> Are you under the delusion that prostitution doesn't negatively affect communities?

<comment><USER15> Well "illegal prostitution" certainly brings about things that negatively affect communities, but that's not the point I was making, nor the question I was asking you.

You said you have no respect for the WOMEN who engage in prostitution. Why?

My point about drug dealers is that I can see having no respect for the person that deals drugs as they themselves produce harm (they could sell to kids).

</comment></comment></comment></comment></comment></comment><comment><USER16> The little bit of haskell FUD in this article made me largely disregard the rest (although I've heard my share of horror stories about the software s big financial institutions). Once you get past the different syntax I find haskell code to be extremely readable.

<comment><USER17> I recently learned Haskell and am writing code in it on a daily basis, although I'm still a novice.

In my limited experience, Haskell is like any other PL, you can write clean, readable code or a horrible mess.

My major concern with Haskell right now is with maintainability/changability of code. One recent example that bit me in the ass was: I wrote an elegant solution to check whether a list of files exists, using sth like foldr (&&) True doesFileExist. Then I realized I really need to print out which file is actually missing, but only the first N to avoid flooding the user with error messages. At this point FP becomes a pain IMO, and it'd be much easier to add a counter variable and printfs in the imperative program and move on. Actually this is fairly common, think about adding printfs for debugging: not so easy in Haskell, as IO changes the type signatures of the function and its callers. So, in my limited experience, changing code in Haskell is expensive and its effect are not localized.

<comment><USER18> Regarding your example, I think that one part of the problem (which you mentioned in the first paragraph of your comment) is that you are still a novice. I cannot claim to be an expert in FP or Haskell, but here's how I would go about your problem.

First, I think your problem really has two tasks to perform: getting a list of files that don't exist, and selecting only a part of those inexistant files. It would make sense to decompose these two tasks into two different functions, and since Haskell has non-strict evaluation, we don't have to worry about generating too much data (list of inexistant files) for nothing. Starting with the function to select inexistant files:

        inexistantFiles :: [FilePath] -> IO [FilePath]
        inexistantFiles files = filterM (\f -> doesFileExist f >>= return . not) files

Some Haskellers may prefer to write their functions completely point-free, but I like to have my variables in the code if it helps me read the coder better. And then we can write another small function to select only the `n` first inexistant files:

        takeInexistantFiles :: Int -> [FilePath] -> IO [FilePath]
        takeInexistantFiles n files = takeM n (inexistantFiles files)
            where takeM n = liftM (take n)

(Surely, we'd prefer to just write `takeM` and call it directly on the result of `inexistantFiles`, but I prefered to go with the specialized function.) Someone who knows Haskell better than I do could certainly come up with a cleaner solution, but I think the code is readable, solves the problem you mentioned and isn't any harder to understand than the equivalent imperative code.

Have a Merry Christmas!

<comment><USER19> _and since Haskell has non-strict evaluation, we don't have to worry about generating too much data (list of inexistant files) for nothing._

Be careful here. Because you're in the IO monad, the entire list of files will always be computed first (ie. doesFileExist will be called for every file given).

Non-strict doesn't really buy you much here. The "expensive" operation here will always be executed. At most, you're avoiding converting some list- consing-thunks to a list.

</comment></comment></comment></comment><comment><USER20> It's not just bad coders, management is also very responsible. If management doesn't make sure code is at least getting documented and shown to co workers, then a lot of bad code gets written only one person understands and can develop into the shape of his own fetish, and become a problem when he leaves. The second point is that I have seen very senior programmers, as in years, e.g. the guys with multiple decades of experience, that are quite used to dealing with this. When they take code over they can read through it, add the necessary comments, and document the structure, and convey it to others.

</comment><comment><USER21> Well, not necessarily. There is plenty of good code out there. The dirty little secret of software development is that bad code is an effort sink. Let me run through a few truisms about bad code and see where they lead us, development wise.

Bad code is buggier, so it will require fixing more often.

Bad code is more difficult to understand and sometimes has non-intuitive side effects or method of operation and so working on it will take longer.

Because of the convoluted way that bad code often works, adding features or doing maintenance on it often makes it even more convoluted and difficult to work on.

Bad code typically has lower performance, so it will require more attention to improve performance. Due to the above 3 factors this becomes a vicious cycle of sinking effort into the code only to make it that much harder to work on in the future.

For these simple reasons bad code becomes a huge time sink. Programming tasks involving good code end up taking up a much smaller portion of the developer- effort budget because it's so much easier to work on that code, while the bad code takes so much more effort and often the code base just ends up even worse than before, perpetuating the phenomenon through to the next cycle of development.

The only way out of this cycle is to identify the code that is the main source of bugs and the main sink of effort then budget out more than enough time to tackle it and improve it. But most development shops aren't proactive enough, introspective enough, or set aside enough dev-resources to be capable of doing that. Even though such efforts tend to have an enormous impact on future development.

</comment><comment><USER22> I think this article sucks, but of course they are right; by far most real world code is horrible. That goes for academic as well by the way; I cringe when I see the code people get As & Bs for in universities. "It has to work" they say and the prof seems to agree. So you can see, yes also in Python, the most incredible constructions which 'work' but are hardly real code and do not work outside of the required examples.

Code in real world companies is mostly horrible, especially when the company gets bigger and older. Spolsky has a bunch of articles about that. To counteract or prevent that you need heaps of money put into it and most companies really don't see the need or actually don't have the funds.

</comment><comment><USER23> In my own experience, the development process is usually to blame, more than the developers themselves, their capabilities (or lack thereof) or their paycheck.

The project I work for depends highly on usually inexperienced developers and we're using a development process that is task-oriented but at the same time based on continuous feedback and regular code reviews. Should it be any other way, things would quickly degenerate. That's what happened before I arrived, and we're still (years later) recovering from the damage that that has caused to the code base.

Outside the IT industry, most bosses don't want to hear about code. Coding is actually seen as a pretty trivial step in many cases. It's OK to hire an intern to add new feature X. Writing good code is not properly rewarded - <USER2> long as it works, it's stable, no-one complains, it's OK.

Not to mention stuff such as niche technologies, that usually create a barrier to entry that would-be employees have to overcome, thus concentrating the know-how, power and pay in the hands of a few bored people that won't bother learning the next thing.

</comment><comment><USER24> This begs the question, which textbooks do include or at least address real- world code?

The closest I can recall are those with heavy emphasis on refactoring, because gradually moving from bad to good is usually the best one can hope for in the real world. I'm thinking in particular of Bob Martin's Agile Software Development, because I liked how he made the point in the book that he didn't just arrive at this beautiful code, but achieved it through TDD and refactoring.

I think "bad code" is actually what the most talented developers excel at (see also Duct Tape Programming <http://www.joelonsoftware.com/items/2009/09/23.html>). A talented programmer knows when it's worthwhile breaking the rules and, by necessity of getting things done, does so on a frequent basis.

<comment><USER21> Well, this brings up another problem. And that is that software development is not applied computer science. There are plenty of books which will teach you skills on dealing with real-world code (Refactoring and Working Effectively With Legacy Code being two of the biggies) and there are books that will teach you about other important aspects of development processes (such as Code Complete, Rapid Development, Design Patterns, The Pragmatic Programmer, etc. although those books are showing their age a bit these days) but none of those would be considered computer science textbooks.

<comment><USER24> True, though I don't think this article is really talking about pure CompSci. That would be about as relevant to Register's audience as books on physics.

</comment></comment></comment><comment><USER25> Most of the time, when stuff I write ends up as 'bad code', it started out as good code, and then got mangled over time due to a series of requirements changes (not necessarily due to bad processes - you often just understand your requirements better as you go along).

This is a fairly natural series of events that can be helped by periodic refactoring. Unfortunately, because each change is small, none of them on their own justify the time required for the refactor. That's when you've got to hope you've got a management team that trusts your judgement, and is willing to carve out the time necessary for this stuff. Most people, alas, don't have that.

<comment><USER26> Well, first thing, in the Computer Science part of programming, the requirements are very simple and straightforward. For example, implementation of hashing algorithm or HashMap.

You just can't compare the requirements of "science" part with requirements of "real world business system". The "science" requirements are almost laughably trivial. And the documentation of business system will run into thousands of pages.

Now, you say if you just refactored the code, if you saw the "grand design" and rewrote the code to accommodate every little requirement elegantly and with proper abstractions and so on... I will say you would just end up with system so abstract and general, no-one would understand exactly how it is meeting those requirements. Because you would had generalized every special case (where it should've stayed a special case, and that's that).

So in a way, the simple hacked-together awful program might actually make more sense than the elegant and grand abstract program. Because you can follow the hacked-together program, and you can't follow the arhitected program. It's this "abstractions all the way down", where you can hunt abstractions down for "how exactly does it do this one thing" all day, and then find out it's a little config setting in obscure .properties file that actually covers this one business case. (And if it was there right in code, you would be done in 5 minutes. But alas - it's a config option, so to change behaviour, now you must write all infrastructure to support another possible config option.)

I say - bad code is just fact of life in real-world business. You should strive to clean it up, make it elegant, abstract and generalize... but really business just needs to to chug along.

<comment><USER25> Don't get me wrong, I don't strive for 'perfection'. I passed through the architecture astronaut phase quite some time ago now :-).

Equally, I don't think that the opposite end of 'hacked together' is some kind of AbstractAbstractFactoryFactory extravaganza - that's just a different kind of bad code. To me, good code is code that is simple for other people to understand, and takes basic precautions with respect to being not-too-brittle to change. Every so often, I find that once the code has received enough alterations, it needs a bit of TLC.

</comment></comment></comment><comment><USER27> The worst[1] code I've seen is code produced by academics with the purpose of teaching undergrads.

[1] In terms of needing refactoring, poorly named variables and functions, just doing The Wrong Thing you don't ever do in real life.

<comment><USER28> I'd say academics are very good on theory, but tend to lack the real world experience of working in teams bound by company processes. They make code for machines and students to read, not for other engineers, I suppose.

</comment></comment><comment><USER29> On this entrepreneurially-focused website I'm going to offer that you can't afford to be an idealist about both business and engineering- probably neither. Good-enough at the right time in the right market trumps perfection without all the outside business forces lining up every time. If you're an artist at heart, no-surprise, it's far more charming to the world if you're financially successful.

</comment><comment><USER30> The problem highlighted by this article and the ensuing debate here is that most people in our industry (management and developers alike) talk about good code and bad code in purely subjective terms. Until we learn to talk objectively about the systems we build (where code is just one part of that system) in terms of function (what a system needs to do), performance (how "good" the system has to do the functions) and economics (the resource constraints in building the system) - you know, like proper engineers do, we'll never move forward. The ugly truth is that would appear that most banking systems are "good enough" - i.e. they are competitive in the environment operate in - despite the horrors that undoubtably exist in their implementation. I see a lot of talk about the consequences of "bad code" but few have the evidence to back it up. It is our responsibility as engineers to measure the performance of our systems in the _right_ way so it becomes painfully obvious where we should focus our efforts to improve things.

</comment><comment><USER31> An assumption: code is designed [to get things done].

Another assumption: the [thing to get done] will likely eventually be transformed via evolution or destroyed, as all natural things like rocks and species and hydrogen atoms and gluons and the entire known universe starting[?] with the big bang transforms via evolution or gets destroyed.

A conclusion: code that can not be easily changed and evolved will be conveniently destroyed via a complete re-write. This is why "facades" and opaque APIs are so popular - it is theoretically possible to rewrite different components of the code without rewriting the whole entire dang thing.

Case in point: [http://stackoverflow.com/questions/810129/how-does- whiles-t-...](http://stackoverflow.com/questions/810129/how-does-whiles-t- work)

>>>while ( _s++ =_ t++);

This is code that is hard to understand(hence the stackoverflow question), making it not easily changed. Not a good sign for the survivability of the code.

JoelonSoftware.com makes a sparse point with this example, in that a "good" coder _must_ understand what this does. He leaves the reasoning up to the reader, likely because he is clever enough not to make any claims that can be disagreed with. Luckily I am not this clever and I will make a claim that follows from my assumptions and conclusion above:

Claim: a "good" coder needs to understand the above type of obfuscated code so that they can evolve it or destroy it safely when they see it.

Note that my claim does not say whether a "good" coder would write code like that. A "good" coder gets the job done, whether it's a write-once-use-once- read-never perl one-liner or a big part of a big program. I only claim that a "good" coder must be able to read it, so that they can change it or destroy it safely.

<comment><USER32> > while(s++ = t++);

is _absolutely trivial_ to understand if the reader has taken the time to actually learn the language in which the code is written. Saying this is "hard to understand" is like saying that "Ich liebe dich" is hard to understand. I'm sure it might be if the reader has never encountered German, but that's not real complexity, and it's not unreasonable for someone writing in German to assume that her reader has some competency with the language.

<comment><USER33> I disagree. I have 16 years' experience programming in C++ (and substantial experience in other languages before that), and I find that an important factor in code clarity is not writing stuff like this.

Yes, you can understand it, but it takes more brainpower to do so than it should, especially once you get beyond trivial cases. It is much better just to write it the long way.

<comment><USER32> > I have 16 years' experience programming in C++ <

Well, there you go; this is absolutely _not_ idiomatic C++; in C++ (or even in application-level C code) one would (correctly) use language features or a library call to perform this copy.

I should have been more specific; this code is completely idiomatic for standard-library-level C (though standard libraries will generally use more sophisticated copy implementations). If one is fluent in that particular dialect of the C language, then this construct is natural and correct.

</comment></comment></comment></comment><comment><USER34> Trollish but true. Things will only change when we start to treat code as _knowledge_ and not _just a tool_. And to have _knowledge_ and not just data that can incidentally be compiled or executed as a program that does something, you also need: meta-data (comments), documentation (always sucks, probably because writing good docs is as expensive as writing the code itself...), history (vcs solves this only if the commits have meaningful comments, bug-tracker data helps a lot but it would be great to be able to easily get from a piece of code to either the written requirement or the bug- request that the code satisfies/solves) and meaningful organization (the UNIX philosophy of small one-task tools, coupled with exposing these tools through web/services APIs is still one of the best ideas in the field - just don't overengineer the interfaces and protocols, keep them thin, keep them dumb).

</comment><comment><USER35> This is why I don't worry about Kurzweil's singularity. The putative Godlike AI will be littered with catch (e) {;} and non reentrant signal handlers and shit. It's the nature if complex systems.

</comment><comment><USER36> Yet another reason to work at Google. They're anti- both obscure or clever code (see their coding standards), and so far in my own experience, you get the time you need to get the code right. There are periodic 'testing on the toilet' pieces to remind you to refactor ruthlessly as you go, and, on how to bring up and gather support for fixing technical debt, with your team. I haven't seen this at any other place I've worked, quite the opposite. It's a luxury.

</comment><comment><USER37> Real world code doesnt suck in my opinion. The issue is that a lot of the time it solves problems that these books don't need to. Its easy to build an alogirthm and print it when you know every edge case.

If you would take the first version of most production code put it in a book and write the problem it solves, I'm sure you wouldn't think it sucks. My exp developer 15years, finance industry 7.

</comment><comment><USER38> Not that textbook code is that much better ;)

</comment><comment><USER39> "You are what you eat", when applied to software is something along the lines of "software's eating the world, and the world is turning to crap".

We've all seen it, at least those in the audience that have worked at some corporate of any size.. maybe "start-ups" face similar issues, it's difficult to get a consensus there.

</comment><comment><USER40> A more revealing and accurate look at software in the real world is 'Big Ball of Mud': <http://www.laputan.org/mud/>

As a Haskell programmer, I thought the Taliban comment was funny.

<comment><USER41> As a non-Haskell programmer (who's had to deal with code in Haskell) I think it's spot-on.

</comment></comment><comment><USER42> Note to OP. Linking to the print version might hide some crud but it makes the line length too wide to be comfortable reading.

<comment><USER43> I usually use the print version + Readability for the best of both worlds.

</comment></comment><comment><USER44> More important: The code you write in the real world will also suck.

</comment><comment><USER10> This guy shits all over his credibility by turning it into a language war. He clearly is out of his depth on the Haskell issue, and I know credible people who think highly of q and kdb (which are rare outside of finance, but ubiquitous in finance).

That said, he's got a point, but the only solution he gives is "Include some comments!" I've seen so many shitty comments that I really think this is no advice at all.

Four levels of software maturity. (There are others. This is a simplified model.)

Level 1: Your work doesn't exist until it's in version control. It might be amazing to 2012-era programmers that this could be controversial, because even the laggards have reached L1 maturity, but at one time, it was. (See: the Joel test.)

Level 2: Your work doesn't exist until it has tests at multiple levels (e.g. unit and integration). Otherwise, it has no enforced semantics.

Level 3: Your work doesn't exist until it has well-documented semantics _in addition to_ the tests, which cannot cover all cases. Otherwise, how does one even decide what a "bug" is?

Level 4: Your work doesn't exist until you've provided the resources (e.g. a code walk or an interactive module) to _teach people_ how to use it.

If you're at Level 1, you won't lose source code. If you're at Level 2, your breakage rate will be reduced, and you can run CI. At Level 3, your system integrity will remain decent because you don't end up with those software-as- spec modules that, over time, corrupt the whole system.

Guess what? It takes Level 4 to get good code. Unless there is a culture of teaching (incremental code reviews are not enough) within the company, you will not have good code for very long. Since engineers are intolerant of context switches and in-person teaching doesn't scale, the teaching should be automatic. Interactive modules or, at the least, code walks should come with the technical asset being demonstrated. If people don't know _what_ they are looking at, then how can they be expected to review or maintain code. With the worst code, it's not even clear _why_ it's bad. It's just incomprehensible.

<comment><USER15> Come on! He makes __one __reference to Haskell, links to the SO post on it, admits he has never used it, and all of a sudden he "shits all over his credibility by turning it into a language war"? It's an example of how ugly code can get for goodness sake.

<comment><USER45> The Haskell code snippet is from a _parody_ of Wikipedia, and is purposefully obfuscated code. Imagine someone holding up a piece of code written in C from an IOCCC entry, as an example of how C is a language to be avoided. Additionally, bringing it up is a total non sequitur, because Haskell doesn't have much to do with the problems the article discusses, and neither does APL.

I think it does ruin the author's credibility. They didn't do the most basic research before including that code snippet, and are just holding it up as a distracting side show. The rest of the screed might have some value, but I'm pretty skeptical of that after reading the last paragraphs.

</comment></comment></comment><comment><USER46> The problem of bad software is too big for one post, and there are multiple causation, of course. Mostly it is incompetence, over-management, pay per line of code in a 9 to 5 shifts, while the resulting product is not anyone's problem. It has something to do with forced collectivization and wage labor, while everyone concerned only with oneself.

The good analogy is with writing a poetry and professional writing in general. There are many examples of technical and scientific books produced by two or free authors (they could list ten, but actual work was done usually by no more than a couple). But we haven't seen any decent book produced by a mediocre writers on 9 to 5 shifts. The idea that this will work for programming is very naive one.

Another analogy is engineering, which is also done by few capable individuals with help of others. There are nothing but failures when mediocre groups trying to engineer anything. On the contrary, we had lots of things invented by capable individuals, things that later were polished by later generations.

So, if one looking for an example of good software look at individuals, or small groups lead by one capable visionary, and avoid anything created in sweat-shops. Most of the really good software we have (Lisp, C, Unix, Plan9, Emacs, vi, nginx, redis, postgres, Informix, etc) was created by an individual effort of men of unusual capabilities, and then polished by community.

Good software engineering is as rare as good poetry, and it cannot be produced by any amount of a manual labor.

There are people who can sell you products and solutions, even ready processes which, they say, will guarantee that you can create good software employing mediocre coders in 9-to-5 shifts. It is all Java or Scrum scams is all about.

It is not tools that create software or invent things, it is minds, the same way no typewriter or fountain pen could make one a poet.

</comment><comment><USER47> What a terrible article. That line of APL is sheer beauty (<http://www.youtube.com/watch?v=a9xAKttWgP4>) - being ignorant of a language's syntax and not willing to invest the time to learn it is no reason to claim it is 'impenetrable'. I'm sure sheet music, replete with quavers and semi-breves and treble clefs is impenetrable the first time you see it, but once learned is far superior to playNote(double hertz, double duration).

Worse, it goes against his point just a paragraph earlier. The whole advantage of Q/K/J/Haskell style languages is that once skilled in them, the code is much (as in 10-20 times) denser than the equivalent OO-style code, and the entire flow can be grok'ed with ease. There is much less danger of over- engineering in those languages.</comment>


How to [Actually] Reduce Sexual Harassment in Tech - <AUTHOR> https://medium.com/@anonymous_99198/how-to-actually-reduce-sexual-harassment-in-tech-8111836c1587

<comment><USER2> Is it sexual harassment to be a man in tech?

<comment><USER2> You can downvote me all you want but it is a valid question.

Legally the definition of a crime must be clearly and objectively defined or it is unfair and invalid. So what constitutes sexual harassment? Is it the feelings of the woman? If that is the definition then eventually no man will work for any company. Is that the goal perhaps?</comment></comment>


Ask HN: What do founders of failed startups do next? - <AUTHOR>

I was wondering if any founders of now defunct startups could share what happened next?<p>Was it easy to get back onto the startup wagon, or did you realise it wasn't for you and move onto something different? <comment><USER2> I was one of the founders of Abacus Concepts, a pre-internet Macintosh startup. For thirteen years we were in start-up mode as all of our competitors were much larger than us. After I sold, I was truly burnt out and took some time to see the world and start a family. The money was holding out good until the crash of dot-com and a property purchase I made -- then I had to start looking for a job again.

For me, it was difficult because the advent of the internet changed everything, and my resume was filled with dinosaur skills. I knew it would be mistake to jump into the internet with my skills as they were, so I was able to find a job doing java. Nothing exciting, but I'm getting my hands dirty with Linux and net stuff.

Working for people sucks. The regular money is nice, but it's a pittance and the tasks are boring. I spend my spare evenings diving into python and django to build my own site. I'm getting kinda old for the crazy hours, with kids and all, but if you want real financial security you either have to put in forty years with the government and get a nice pension, or do your own thing.

I thought about a career change, but I love programming too much. Your on the edge of a revolution they will be talking about for a thousand years, and the money's good. Besides, what other careers can you have where you can drop out for nine years and then jump back in again?

</comment><comment><USER3> In the case of YC startups, a lot of them get picked up by other YC startups. This was one of many unanticipated advantages of funding startups on a large scale.

</comment><comment><USER4> After I resigned from my startup early last summer, I spent about two weeks moping in the evenings/on weekends (I had a dayjob by then), then came up with another idea that I started to work on.

</comment><comment><USER5> this is too situation-based to answer, really. they do what they want or need to do. maybe that's start another, or maybe its work at a more normal job.

<comment><AUTHOR> It would be great if (ex)founders could share a little bit of their situation too, for context.

<comment><USER5> well, i'm currently working on something that i do after my 9-5. if it goes under, its not much of a loss -- i'll probably do another. however, if i was working on it AS my 9-5 and it went other, i imagine i could be in a bad financial situation and need to get a solid job.</comment></comment></comment>


How My Illness Forced Me to Leave My Job and Start a Business - <AUTHOR> http://notthegolfer.com/2015/01/15/year-in-review-2014-part-2/?new

<comment><AUTHOR> This is the second part of my long year-in-review post. You can read the first part about how my health caused me to up and quit my job here: [http://notthegolfer.com/2014/12/31/year-in- review-2014-part-...](http://notthegolfer.com/2014/12/31/year-in- review-2014-part-1/)</comment>


Marzipan: The False Mac iPad Dilemma Resolved - <AUTHOR> https://mondaynote.com/marzipan-the-false-mac-ipad-dilemma-resolved-fa7003a7d2fd <comment><USER2> There is no need for a Mac Pad, the iPad Pro does it all.

iOS is OSX ported to ARM, if Apple wanted to sell ARM based Laptops they would be the iLaptop and run iOS.

iOS is for mobile devices, OSX is for desktops and laptops.

</comment><comment><USER3> Mmm I don't think so.

The Marzipan apps on macOS are so terrible in every way that I don't see why any developer would go down this path.

Keep the two OSes separate.</comment>


How cloud computing impacts the environment - <AUTHOR> https://thecloud.christmas/2019/15 <comment><USER3> Disclaimer: former OVH employee

> Although many cloud providers have pledged to decarbonize their data > centers, none have ditched fossil fuels entirely

Most datacenters in France are located close to nuclear power plants and only run on fossil fuels in case of a prolonged power outage. To me this is as carbon neutral as it gets.

<comment><USER4> I don't mean to nitpick, but wouldn't "as carbon neutral as it gets" be zero reliance on fossil fuels, even for backup?

<comment><USER5> >as it gets

It's "as it gets" that's doing the heavy lifting here.

<comment><USER4> Agreed. And "it" is the real workhorse - is "it" carbon neutrality or datacenter power supply?

</comment></comment></comment></comment><comment><USER6> While yes, datacenters will be a significant consumer of energy in the foreseeable future, I honestly don't feel bringing "cloud" into the picture gives relevance other than the buzzwords.

Like what the article actually says, cloud is basically datacenter hosted by another guy. In fact, I would argue that cloud is more energy efficient due to the shared infrastructure and higher efficiency due to the scale involved. Imagine the amount of redundancy an organisation have to have to handle peak demand, vs a cloud provider aggregating the demands together.

The graph is probably even more telling. With (assumed) increased need for technology going forward, we actually have almost static demand for energy due to these technologies. It's actually a good news!

End of the day, yes technology is expected to consume more energy in the future, but in a good way, they will be somewhat aggregated to cloud. That means we can look into solutions targeted to cloud providers that actually have significant impact to the environment (eg. more incentives to use solar panels in cloud datacenter)

<comment><USER7> > cloud is basically datacenter hosted by another guy. In fact, I would argue > that cloud is more energy efficient due to the shared infrastructure and > higher efficiency due to the scale involved

While I agree with this, I think it is precisely because of this commoditisation of cloud computing that we can have this conversation at all, as we have only 3 (AWS, GCP, Azure) companies to look towards to give a very accurate and measurable environmental impact of each CPU/TPU cycle or GB stored/transferred.

Prior to this "cloud" movement, I'd argue it was too fragmented to measure, other than as a cost of electricity.

My takeaway is that there are negative externalities to using computers beyond the direct costs, and that it is a good time to take stock of these and consider how to reduce them.

Edits:

From the article:

> However, there lies a responsibility on all of us developers as well to > utilize libraries, coding techniques and compression algorithms which > consume less storage and less energy. Mobile developers are well aware of > the power restrictions due to batteries. It's time the rest of us follows > and give their contribution to lowering data storage and processing for our > workloads. The benefit? Reducing storage, memory and CPU for your workloads > has an economic benefit - you pay less money to the cloud vendor. Win-win.

The issue will be in pushing an agenda that conflicts with the cloud providers motives (revenue & profit).

<comment><USER8> Disclaimer: I'm the author of the article

I don't really believe the suggestions conflict with cloud provider motives. If all cloud customers reduce their workload requirements, it's actually a win for the cloud provider as well. Because they are then able to put more customers on less hardware. Then they pay less money for electricity and pay less money on hardware investments. This is just a repeat of the virtualization / Dockerization of workloads, just on much higher scale.

</comment></comment></comment><comment><USER9> Google at least claiming to buy 100% of their usage from renewable sources, they are pretty open that it doesn't mean that is what feeds their electrons at all times of the day though. Better than most traditional computing / on premises sources though.

[https://www.google.com/about/datacenters/renewable/](https://www.google.com/about/datacenters/renewable/)

<comment><USER10> I buy 100% wind power at home, but its pretty much a scam I think. Of course when there isn't any wind I get coal/gas/nuclear generated power. The 99% of people who dont care dont care and its just a way to get some extra charge to some people who try to be green. I keep doing it to try to send some signal but might give up soon.

<comment><USER11> I've wonderd about buying "eco power", it's not like electrons have markings on them that say "I came from a generator attached to a wind turbine!". It's just that the wind turbines put some Megawatts into the pool and their buyers took the same amount of Megawatts.

</comment></comment></comment><comment> yoz-y Ok but how about all the carbon removed by cloud computing? Less commutes necessary thanks to video conferencing tools and work from home, less cars on streets due do deliveries (disputable, depends on how the city is laid out I suppose)...

<comment><USER12> Possibly less energy consumption for the same cloud service thanks to shared hosting, on demand instances etc

</comment></comment><comment><USER13> Well more and more computing power in one place won't be carbon neutral. I disagree with this article. It's like writing that 100 meter building can be more efficient than 100 buildings. Yes it takes less space horizontally but it's more complex to power it and support people then make 100 same buildings and provide each with wind / solar power. That's what decentralisation is about and maybe somewhere in future we start building self sustaining villages with own small networks that barely need to communicate with outside world instead of building giant networks that are inefficient. P2P FTW

</comment><comment><USER14> Low Tech Magazine's piece "Why We Need a Speed Limit for the Internet: The energy use of the internet can only stop growing when energy sources run out, unless we impose self-chosen limits." [https://solar.lowtechmagazine.com/2015/10/can-the- internet-r...](https://solar.lowtechmagazine.com/2015/10/can-the-internet-run- on-renewable-energy.html) covers this situation from a systemic perspective, realizing problems in meeting every energy need without considering alternatives.

</comment><comment><USER15> When you consider the impact of "cloud computing", you need to consider the offsets as well.

Enterprise datacenters have much higher PUEs than a hyperscale datacenter, and the hyperscale folks have an incentive to use energy as efficiently as possible. Hyper-scale datacenters often tend to be placed in locales where space and access to electricity is better.

There is always alot of navel gazing about anything environmental, and it's easy to throw rocks at what's new, because the facts are known and easy to find. Ultimately, I don't think that what the author is suggesting that we all do (ie. developing with an eye on efficiency) could meaningfully happen without cloud computing, because the "old way" of making big capital purchases for infrastructure is too disconnected from the operational costs.

</comment><comment><USER16> Sure but I think in a way they're a necessary evil.

I don't think we're gonna get to a futuristic carbon neutral world without a hell of a lot of processing power

</comment><comment><USER17> Since we don't know for sure what the capacity utilization of say AWS is, we do not know if they are running physical servers which are lightly loaded or not.

It might be that AWS runs a tight ship or it might be that they have entire aisles of servers spun up but which run for only 1 hour per day.

<comment><USER18> I would think they would turn off computers they don't need, since that is a huge saving on power and cooling.

They presumably can predict when they need to be turned on again, ready for use.

</comment></comment><comment><USER19> Data centers are more efficient than people managing that same amount in their own homes or office.

Also, "1%....might come from burning coal....if it did....it would result in 1.2 billion tonnes of CO2". "Might"? "If" it did? How is that unreal theory useful to anyone?

<comment><USER8> Disclaimer: I'm the author of the article.

I'm not sure which part of it you believe is unreal, but I'll try to clarify; \- the 1% electricity consumption of total world consumption is real - <USER2> on research done by IEA ([https://www.iea.org/reports/tracking-buildings/data- centres-...](https://www.iea.org/reports/tracking-buildings/data-centres-and- data-transmission-networks)) \- The calculation on the effect of generating 200TWh of electricity from burning coal is 1.2 billion tonnes of CO2 emission is real - the link in the article points to the estimation \- the fact that most of world's energy comes from fossile sources is real. While it's not 100% from coal, 67% is from fossile fuels; [https://www.iea.org/reports/electricity- information-2019](https://www.iea.org/reports/electricity-information-2019)

And some countries are worse than others. [https://edition.cnn.com/2019/09/10/asia/china-data-center- ca...](https://edition.cnn.com/2019/09/10/asia/china-data-center-carbon- emissions-intl-hnk/index.html)

<comment><USER19> Thanks for clarifying

</comment></comment></comment><comment><USER20> There is a lot of migration from smaller enterprise data centers to the cloud which may be net positive.

</comment><comment><USER21> Reading the title, I was thinking this would be about cloud cover and not data centers to be honest…

</comment><comment><USER22> proposed solutions:

> Increased energy efficiency on component level

> Increased energy efficiency on data center level

Why no "Increased efficiency on software level"? Is it simpler to throw hardware (and money, energy, carbon emissions) at the problem than find better solutions?

<comment><USER8> Disclaimer: I'm the author of the piece.

If you read the very last paragraph, I actually propose increased efficiency on software level as a way to lower energy consumption. The two other (component and data center level) are things us as developers usually have little or no control over, they were merely used to illustrate that the industry as a whole is working on the problem, and that you as a developer actually can contribute.

<comment><USER22> Thanks for clarification and for the article! First time I only skimmed over headings and missed that part.

</comment></comment></comment><comment><USER23> Here's the number to keep in mind. A medium-sized cloud datacenter uses about as much power as a single passenger airliner, but emits no carbon. That's your comparison.

</comment><comment><USER24> Would be great if cloud providers added how much energy a service/server/etc is consuming and where that energy comes from in the dashboard.

</comment><comment><USER25> This is dumb, the environmental impact is the same as any other electrical appliance but we don't ask "What is the environmental impact of life support?" because it doesn't matter.

The problem and solution have been clear for over a decade we all know it, we need clean renewable electricity.

edit: I guess if you are releasing 12 articles a day they can't all be insightful.

<comment><USER8> Disclaimer: I'm the author of the article

I'm sorry to hear that you didn't like the article. I believe there is a fundamental difference between data centers and other electrical appliances because data centers consume energy 24/7/365, while electrical appliances do not.

It is a well known fact that electricity consumption from other sources, like heating or cooling buildings is much larger in comparison. Or even data networks and internet traffic in general; [https://www.iea.org/reports/tracking-buildings/data- centres-...](https://www.iea.org/reports/tracking-buildings/data-centres-and- data-transmission-networks) And there is a lot to do in those fields as well.

Unfortunately, renewables to overtake fossile are far away. Even with the development the last decade, wind and solar still count for only a fraction of world production; [https://www.iea.org/reports/electricity- information-2019](https://www.iea.org/reports/electricity-information-2019)) Until then, we as developers can actually make a small contribution by offsetting getting the consumption from this particular source higher than it is today, even if number of data centers is exponentially increasing.

Hope you found some of the other articles insightful.

<comment><USER25> > because data centers consume energy 24/7/365, while electrical appliances do > not.

But life support isn't? It really doesn't matter that its on 24/7, the source of the energy is the problem not where its going.

In my country renewables are 50% of our electricity and growing.

The cost of installing a new windmill will be a lot cheaper than hiring a team of coders to optimise a single piece of code. You also have the benefit of solving the problem instead of the symptom. Again this is very dumb sorry.

</comment></comment></comment><comment><USER23> A lot of people seem to cling to this number, 3-5 year lifecycle for servers. But, much to the disappointment of software developers, machines in large clouds are often in service for 8-10 years. When you build your own computers there's nobody to tell you that the warranty has ended, so you just leave them in the rack as long as they are TCO-positive.

<comment><USER26> > But, much to the disappointment of software developers, machines in large > clouds are often in service for 8-10 years.

Why does this matter to software developers?

<comment><USER23> Because they can't easily use AVX or whatever.

</comment></comment></comment><comment><USER27> I tried to share this link with someone on Facebook Messenger and it said it could not be shared due to violating their community standards. Only thing I can guess is it's something to do with the .christmas tld

</comment><comment><USER28> I once read the suggestion to build a powerful cloud datacenter in orbital space, powering it with solar panels. That would solve any overheating problem too.

<comment><USER29> That would work for tasks that are not latency sensitive (i.e. long compute tasks, not interactive ones) and that don't require oodles of bandwidth (or aren't time sensitive so it doesn't matter that getting the data up there for processing takes a while).

IIRC typical packet round trip times for generic networking a single satellite is at least half a second, two or three times that being typical RTTs, and bandwidth is going to be orders of magnitude lower than can be achieved with land/sea wire based networks.

Also dissipating heat is more problematical in space than most people intuit do to vacuum conditions.

And getting the kit up there in the first place will have a large environmental impact, and upgrades will be somewhat more time consuming and otherwise resource intensive.</comment></comment>


WiFi-enable USB printers with a Raspberry Pi and share it over your network - <AUTHOR> https://www.balena.io/blog/wifi-enable-usb-printers-with-a-raspberry-pi-and-share-it-over-your-network/ <comment><USER2> The article maybe should explain why not just apt-get install cups and wifi- share the USB printer (AirPrint included) without the need for downloads from github or an account somewhere which presumably can get a copy of your every page.

I did exactly that at home, and also installed a second printer driver which prints into a PDF file, piled into a HTTP-served directory, so I can easily "print" stuff from my iPhone to reference later or to send to a friend, without any paper.

Very happy with the results.

Raspberry Pi Zero W (10 euro IIRC).

Drawback: it's not very fast.

<comment><USER3> One advantage of Balena seems to be that the logs in the cloud reduce wear on the sd card, so you can just unplug the raspberry pi (I can't find the source right now - I think it was in the issues of balena sound's GitHub Page)

<comment><USER4> Why you would need persistent logs for printer?

I use Alpine with CUPS and splix installed in "diskless" mode.

</comment></comment></comment><comment><USER5> I've Samsung SCX 3201G bought 6 years ago and still running on the starter toner cartridge, I've printed 739 pages so far with it as per the statistic.

Printer was $180 and new cartridge on HPs website is listed for $85 for standard cartridge with 1500 prints.

There are $10 Chinese cartridge available for same printer

What do you guys recommend? I am already using it with Pi+Cups.

Well the printing issues started appearing when I noticed my pages in printer had a lot of dust, I've no idea of its dust which killed my print quality or 70% used cartridge.

</comment><comment><USER6> There's already a good solution to this: usbip is supported in all major Linux kernels and has drivers for Windows as well.

The source code of usbip was merged into the staging tree, and finally has been moved to the mainline since Linux-3.17. For Windows, the precompiled client-side program is available from the project page. You need a Linux machine (could be raspberry pi) for the server side.

Link: [http://usbip.sourceforge.net/](http://usbip.sourceforge.net/)

<comment><USER7> How is tje security situation? This post from few years paints a sad picture: [https://sourceforge.net/p/usbip/mailman/message/35003716/](https://sourceforge.net/p/usbip/mailman/message/35003716/)

<comment><USER6> I would tunnel the connection through SSH, which if is good for X sessions (and their keystrokes) I believe is good enough for this

</comment></comment></comment><comment><USER8> Interesting project. I did something similar with a raspberry Pi I had laying around and a receipt printer. I installed the serial print driver, then just apt installed the avahi and cups daemons. It was then accessible to everything else on my network. I later made a nice little web UI that allowed you to write (or upload) whatever and get it printed. (Along with a messaging bot that allowed you to send it anything cups will ingest like .png or .pdfs or raw text. It was a fun little project that also let me print files to my real printer when I wasn't even home!).

</comment><comment><USER9> How long does this OS take to boot? This seems like a way too simple task for a Linux equipped Raspberry Pi.

<comment><USER10> People are really bad at gouging exactly how powerful an rpi is. I have met people that bought several to do a lot of things that would barely put any stress on a raspberry pi 3.

I have a rpi4 4gb that acts as a torrent box, a unifi controller, IRC machine, print server and a music player.

I just put an aluminium pipe filled with copper coins on the CPU. The thermal mass and extra heat dissipation area makes sure it rarely goes over 50c. Most of the time (98%?) the workload is just barely discernable from idle.

</comment></comment><comment><USER11> Cool idea I'd do it for the fun of it but my last wifi printer was $80NZD a Pi would cost me over $50NZD. I'm not sure this makes any economic sense. Not that fun projects need to.

<comment><USER3> It might be nice to have a printer that doesn't call home or an order one that is easier to refill.

Also with the cheaper pi zero, why waste a functioning printer - you might also have other services running on the Raspberry pi.

<comment><USER12> Ya, I have used a setup like this for an older thermal label printer. You can buy a good used zebra label printer for $75 on eBay where a more modern wifi enabled version would cost several hundred dollars.</comment></comment></comment>


Ask HN: When to buy vs rent servers? - <AUTHOR>

I am wondering, at what point does it make sense to own your own servers (and switches, routers, whatever) as opposed to rent them monthly through a VPS service?<p>Also, which Windows VPS provider do you recommend? <comment><USER2> _I am wondering, at what point does it make sense to own your own servers (and switches, routers, whatever) as opposed to rent them monthly through a VPS service?_

When your cost of capital is less than the hosting company's cost of capital plus the hosting company's markup.

New startups have a very high cost of capital -- because there's a high risk of them not paying money back, investors demand high rates of return -- so for a young startup renting is generally better than buying. Large well- established companies have a very low cost of capital -- banks will happily loan them money at 5% interest -- so for them it's worth buying hardware.

Figure out where you are on the spectrum and do the math.

</comment><comment><USER3> In general, you should rent as long as possible. This allows you to focus on your product, unless, your product is web hosting.

There are however, three reasons, why you might consider buying over renting.

1\. Cost -- Renting is more expensive than buying. At some point it probably make sense to buy. You can buy the individual machine and co-lo or you can buy the datacenter. Depends on your needs.

2\. Control -- For whatever reason, you need more control. Maybe it's specialized hardware or something that you can't easily do on a rented machine.

3.Pride -- This is one that basically is asking to shoot your self in the foot. A lot of guys want to have the server closet. Once you are established, maybe it's worth it. Not from a cost or control standpoint, but maybe its worth it.

</comment><comment><USER4> I wouldn't necessarily agree with renting as long as you can.

You can buy a very cheap server that's much more capable than a cheap VPS server. Here are some decent servers for about $300; you can often find good servers for $150:

<http://www.geeks.com/products_sc.asp?Cat=821>

Colocation at a good facility costs $60-100.

So for $150-$300 upfront and $60-$100 monthly, you can have a capable server hosted at a good facility at a much cheaper cost than the equivalent "rented" server.

You can use a VPS or something like EC2 as a hot backup, setup replication, and have a very capable setup for not a lot of cost.

This assumes that you have sys admin skills.

<comment><USER5> If you are considering buying one of the servers listed at the above geeks.com link, take care in doing so.

The ones for < $890 are running NetBurst Xeons ... which means, at the youngest, they're nearly 4 1/2 years old.

The one at $890 uses a Xeon 5120 which at least gets you into the Core Xeon era, but you're still looking at a more than 4 year old processor.

If you are going to cheap-out, at least do yourself a favor and buy something low-end-but-new from somewhere like Dell. You can pick up a new PowerEdge T110 from Dell for $400 which comes with a only-released-a-year-ago Xeon X3430 which is going to be about 3 times faster than that Xeon 5120. Plus you're going to have a faster FSB, memory, drives, networking, modern interface to upgrade with, etc.

If you're looking at spending $1k/yr for colocating your server, make it worth it by co-loc-ing a server that's not 5 years old.

<comment><USER4> Fair enough, but either option is going to be much more than you'd get in a basic VPS.

</comment></comment></comment><comment><USER6> I'd say when you're spending > $5k/mo on hosting you should sit down and evaluate the cost of bringing it in-house.

If you can live on a VPS or two you shouldn't even be thinking of buying.

</comment><comment><USER7> I've been using UltimaHosts (Windows VPS) for a while and its been great so far.</comment>


Show HN: Moving My HTTP Website to HTTPS Using LetsEncrypt, HAProxy and Docker - <AUTHOR> https://blog.georgejose.com/moving-my-http-website-to-https-using-letsencrypt-haproxy-and-docker-deb56ff6be9b <comment><USER2> Wat.

You have take your website offline AND rebuild the haproxy container every time you need to renew a cert?

There is a use case for containers. This is not it.

<comment><AUTHOR> So in my actual production website, I'm using a ssl certificate that's valid for quite a while that I bought off namecheap.

You could still get around having to take your website offline if you're using letsencrypt by using a docker volume and having it read the certificate from your host machine. I wanted to keep the post short & didn't want to get into volumes. There are several ways to automatically renew a letsencrypt certificate. The last part in [https://www.digitalocean.com/community/tutorials/how-to- secu...](https://www.digitalocean.com/community/tutorials/how-to-secure- apache-with-let-s-encrypt-on-ubuntu-16-04) explains how you can automatically renew using a cron job.

The advantage of using Docker for my setup is being able to plop the whole thing on a different service and having it run exactly the same way (as long as it's provided an ssl certificate).

<comment><USER2> That still sounds like you need downtime for cert renewals. I personally keep HAproxy bound to 80/443 and use rules in it to send the http/dns-sni verification checks from LE to certbot in standalone mode.

I still don't understand what docker is giving you here though. nodejs and haproxy run on more platforms and a wider range of linux kernels than docker. You could 'plop the whole thing' onto many more servers if it was just nodejs and haproxy than you can like this.</comment></comment></comment>


Creating new fonts from classic Detroit signs - <AUTHOR> <comment><USER2> FONTS: Preserving Detroit's visual cultures - [http://www.modeldmedia.com/devnews/GratiotAndRiopelle.aspx](http://www.modeldmedia.com/devnews/GratiotAndRiopelle.aspx)

[https://www.kickstarter.com/projects/bluucat/gratiot-and- rio...](https://www.kickstarter.com/projects/bluucat/gratiot-and-riopelle- type-foundry)

</comment><comment><USER3> I think we have a bug right now where some invalid URLs get replaced by an empty string, instead of letting you know there's an error. Want to try to repost this?</comment>