Real-Time Global Illumination by Precomputed Local Reconstruction - <AUTHOR> https://users.aalto.fi/~silvena4/Projects/RTGI/index.html <comment><USER2> As far as I can tell this is mostly the same capability as the Enlighten middleware used by e.g. Unity and Frostbite (i.e. recent Battlefield titles) with some small improvements on top (e.g. the opaque occluder approximation at the end). A demo for reference: [https://www.youtube.com/watch?v=ARorKHRTI80](https://www.youtube.com/watch?v=ARorKHRTI80)

Is there some improvement in the technique beyond that - e.g. some clever speedup across the board?

<comment><USER3> I feel I should point out that Frostbite is used by a lot more than battlefield. Including Anthem, Dragon Age Inquisition, the latest Fifa, Mirror's Edge Catalyst and the Need for Speed games by Ghost Games (which is where I work).

<comment><USER4> And don't forget about Star Wars: Battlefront and the upcoming Battlefront 2. Ridiculously good looking games.

</comment></comment></comment><comment><USER5> You know it's a good paper when you reach the end and ask "is that it?" and yet no one else has thought of it before.

Side note: local-global approximation seems to do well in graphics and visual tasks. For example in the field of alpha matting the state of the art for a while was KNN Matting which sampled locally and globally. Most methods since then have taken a similar approach.

</comment><comment><USER6> The article mentions that they used a Nvidia Titan X Pascal, and that it took < 5 ms to compute. 5 ms is still a large part of the time budget for each frame, and most users have a slower GPU than that.

<comment><USER7> I remember buying a new GPU everytime a new edition of Quake came out. Those were the days...

<comment><USER8> Has anything changed, other than the particular game?

<comment><USER9> Yes. Quake 2 was December 1997, Quake 3 was December 1999. In that span, the game went from "use a GPU and you'll get colored lights and a higher resolution" to "use a GPU or else you don't get to play the game at all".

GPU power seems to be increasing as fast as it always was, but the things we can do with that extra power are less impressive. So if you're playing games, it's not so big a deal any more if your graphics card is five years old.

<comment><USER10> Up to a point; obsolescence happens by games requiring graphics APIs that didn't exist 5 years ago.

<comment><USER9> There aren't many games that fall into that category. For example, look at Direct3D 12. You can count on your fingers the number of games that require D3D 12, and it's supported by graphics cards made in 2011 (although not all of them).

Just look at the new API features we've been getting, they aren't really the kind of things you'd rewrite your engine for, with the exception of Vulkan / D3D 12 / Metal / etc. But Vulkan / D3D 12 are really just API changes that don't reflect underlying hardware changes.

On top of that, a large percentage of games are made on top of engine middleware with good support for mediocre hardware (like Unreal and Unity, both of which run on phones if you want).

<comment><USER11> Not to mention that lots of neat effects can be produced by modern shaders, and do not need further API support.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> Does this support dynamic moving lights? That's the 'wow' moment that's going to feel like a move to next generation graphics - when all/most lights are fully dynamic and contribute to GI.

<comment><USER13> Haven't read the paper yet, but it looks like yes. The caption at the top says "Real-time global illumination rendered using our method in 3.9ms with fully dynamic lights, cameras and diffuse surface materials."

It's a couple of years old now, but Crassin's voxel cone tracing paper is a neat one: [http://research.nvidia.com/publication/interactive- indirect-...](http://research.nvidia.com/publication/interactive-indirect- illumination-using-voxel-cone-tracing)

</comment></comment><comment><USER14> Link to the paper (pdf):

[http://research.nvidia.com/sites/default/files/pubs/2017-02_...](http://research.nvidia.com/sites/default/files/pubs/2017-02_Real- Time-Global-Illumination/light-field-probes-final.pdf)

</comment><comment><USER15> The approximate dynamic occluders is the most impressive part of the demo.

Games etc. really need that stuff in real time.

<comment><USER16> Games need more. The occluders from the paper completely absorb light. A more realistic occluder would also reflect light.

<comment><USER17> > _The occluders from the paper completely absorb light. A more realistic > occluder would also reflect light._

My understanding, "approximate occluders" aren't visible-geometry but stand-in replacements (very coarse to represent much finer-grained visible-geometry) during the GI pass(es). Probably even geometric primitives (sphere, box, torus etc pp) to, indeed, simply approximate occlusion.

Physically-based _reflections_ across _all_ geometry already works brilliantly well for the last few years in real-time.

</comment></comment></comment><comment><USER18> That is incredible.

Finally, realistic-looking GI in real time.

</comment><comment><USER19> It took me a minute to appreciate it, didn't know what to look for. But wow!

<comment><USER8> What should I look for?

<comment><USER11> Light bouncing around, lighting up things that aren't in direct light.

<comment><USER20> This kind of lighting (global illumination) is typically "baked in", i.e. lights don't move around.

</comment></comment></comment></comment><comment><USER21> Anyone have links to papers describing how GI works in Unreal and Unity?

<comment><USER22> Enlighten is used in UE4 and Unity as I understand it. You can read about enlighten in Frostbite circa 2010 here:

[http://advances.realtimerendering.com/s2010/Martin- Einarsson...](http://advances.realtimerendering.com/s2010/Martin-Einarsson- RadiosityArchitecture\(SIGGRAPH%202010%20Advanced%20RealTime%20Rendering%20Course\).pdf)

<comment><USER23> UE4 doesn't use Enlighten out of the box. It uses its own system called Lightmass, which generates very nice static lightmaps, but is not that great for dynamic GI (it has some basic support through what's called "indirect lighting caches" [https://docs.unrealengine.com/latest/INT/Engine/Rendering/Li...](https://docs.unrealengine.com/latest/INT/Engine/Rendering/LightingAndShadows/IndirectLightingCache/)). However, you can get Enlighten as a licensed plugin.

</comment></comment></comment><comment><USER24> Does anybody know where the Gallery scene is from?

</comment><comment><USER25> Isn't it frustrating? All of this effort, and it looks nothing like real camcorder footage.

That's not dismissive -- no one has ever made any program that outputs a string of images indistinguishable from a real camcorder. It's just that hard.

I think whatever the next leap forward looks like, it will come from a nontraditional approach. Something strange, like powering your real-time lighting model by an actual camcorder -- set it up, point it at a real-world scene, then write a program that analyzes the way the light and color behaves in the camcorder's ground truth input. Then you'd somehow extrapolate that behavior across the rest of your scene.

That last step sounds a lot like "Just add magic," but we have deep learning pipelines now. You could train it against your camcorder's input feed. Neural nets tend to work well when you have a reliable model, and we have the perfect one. So more precisely, you'd train your neural net against the camera's input video stream: at each generation, the program would try to paint your scene using whatever it thinks is the best guess for how the colors should look. Then you move your camcorder around, capturing how the colors actually looked, giving the pipeline enough data to correct itself. Rinse and repeat a few thousand times.

The key to realism, and the central problem, is that colors affect colors around them. The way colors blend and fade across a wall has to be _exactly_ right. There's no room for deviation from real life. Our visual systems have been tuned for a billion years to notice that.

There are all kinds of issues with this idea: the real-world scene would need to be identical to the virtual scene, at least to start. The program would need to know the camera's orientation in order to figure out how to backproject the real-life illumination data onto the virtual scene. But at the end of it, you should wind up with a scene whose colors behave identically to real life.

It seems like a promising approach because it gets rid of the whole idea of diffuse/ambient/specular maps, which don't correspond to reality anyway My favorite example: What does it mean to multiply a light's RGB color by a diffuse texture's RGB value? Nothing! It's a completely meaningless operation which _happens_ to approximate reality quite well. There are huge advantages with that approach, like the flexibility of letting an artist create textures. But if the goal is _precise, exact_ realism as defined by your camcorder, then we might be able to mimic nature directly.

(Those dynamic occluders looked incredibly cool, by the way!)

<comment><USER26> We have spectral renderers that simulate light transport extremely accurately and in fact _are_ indistinguishable from a photograph (see [http://www.graphics.cornell.edu/online/box/compare.html](http://www.graphics.cornell.edu/online/box/compare.html))

The problem isn't solving light transport itself (the rendering equation has been known for years, and is asymptotically exactly solved using unbiased numerical integration techniques); it's modeling the interaction of light with complex physical materials and doing so quickly that poses the challenge.

<comment><USER25> Note the important distinction: no one, anywhere, has ever created a real-time video simulation indistinguishable from the output of a real camcorder. Real- time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example.

Our visual systems are tuned to notice problems with real-time outputs, whereas a still frame is ambiguous. There are all kinds of transformations you can do to a perfect still frame where it still seems perfect afterwards: You can tweak the contrast, brightness, hue, throw a softpass filter on it, etc, and a human observer still won't notice a big difference. In other words, there are many ways to "cheat"! Even though we're doing operations that don't correspond to how nature behaves, a human observer is still fooled by them.

Yet the moment that you string a bunch of still frames together into a video, that human observer will pounce on you immediately and call it out as fake. I'm not sure why motion makes such a massive difference, but it does.

To put it another way, if it were possible to make a real-time video whose ground truth was identical to a camcorder, we'd see it in hollywood. But no one has been able to; Avatar was the best we could do.

I think it should be possible to train a neural net to paint the scene, similar to a human painter. DaVinci didn't make diffuse/ambient/specular textures; he simply painted the scene according to how it would look in real life.

And in real life, color changes are gradual. It's crucial that the colors change in _exactly the right way_ , but the changes themselves are rarely discontinuous. That means a neural net might be able to catch on pretty quickly.

Bots are already doing similar work:

[https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71g...](https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71grax/?context=3)

Before: [https://i.imgur.com/EqMLNFo.jpg](https://i.imgur.com/EqMLNFo.jpg)

After: [https://i.imgur.com/omYmn7Q.jpg](https://i.imgur.com/omYmn7Q.jpg)

If it's possible to clean a puppy, it might be possible to extrapolate how light should behave in the general case.

<comment><USER27> Hollywood is not optimizing for perfect realism. They are operating under many constraints such as budget (yes, even on Avatar they had budgets), schedule, and the need for directorial control (which often conflicts with realism). It would absolutely be possible to produce prerendered videos of mostly static scenes using today's technology that would be indistinguishable from reality, although the expense would be high.

<comment><USER25> _It would absolutely be possible to produce prerendered videos of mostly static scenes using today 's technology that would be indistinguishable from reality, although the expense would be high._

And yet, no one has. :)

It'd be a fun trophy to claim.

My theory is that it's not merely a matter of computing power. We have massive horsepower now, but our algorithms are still wrong at a fundamental level. We still approach it by trying to simulate the underlying physics of light: the transport algorithms, radiance sampling, and so on. But between the RGB textures that have nothing to do with nature and the physics approximations that we're forced to settle with, something ends up lost in translation.

No matter what we try, that human observer ends up looking at a final product that looks nothing like a real camcorder video.

Back during ~iPhone 4 days, someone forged some pictures of the "next gen iPhone." It was a hoax, but a lot of people were fooled at the time. Remarkably, it was a computer rendering. The reason it looked real is because he took a picture of the rendering using a real camera, which muddied up the colors to the point that you couldn't tell it was a rendering.

The idea here would be similar: analyze the actual pixels coming out of a real camcorder and try to mimic the nuances as closely as possible. In the best case, it wouldn't be a physics simulation at all: the program might be able to guess how the scene should look based on partial data and past experience.

<comment><USER28> > We still approach it by trying to simulate the underlying physics of light: > the transport algorithms, radiance sampling, and so on. But between the RGB > textures that have nothing to do with nature and the physics approximations > that we're forced to settle with, something ends up lost in translation.

You keep talking about rendering being the problem and yet you acknowledge that photos can be indistinguishable from reality and motion being the hard problem. Do you not see the disconnect in your own argument? Light simulation isn't the issue, motion simulation is. Movement and model and texture complexity all have a long way to go.

The people practicing graphics in research and in production know what the problems are, they are aware of the approximations they are making, they know what they need to make graphics more realistic, and they are making conscious choices of where to spend their limited budgets. The big problems just aren't due to light transport or radiance sampling, your pet theory seems ignorant of what's going on in CG practice today. If you would just talk to (and listen to) some production people and researchers...

You're still ignoring the fact that realism in CG in increasing. There's a clear trend, and we are closing the gap on the details that make CG look unrealistic. The standards are higher now than last year, which was higher than the year before. Realism has always increased every year, and yet at no time was the increase due to a fundamental change in multiplying colors together.

<comment><USER25> _Light simulation isn 't the issue, motion simulation is. Movement and model and texture complexity all have a long way to go._

FWIW I completely agree, and this is a key observation. DaVinci devoted several chapters to the problem of motion in art, and it will always be with us.

That said,

 _You 're still ignoring the fact that realism in CG in increasing. There's a clear trend, and we are closing the gap on the details that make CG look unrealistic. The standards are higher now than last year, which was higher than the year before. Realism has always increased every year, and yet at no time was the increase due to a fundamental change in multiplying colors together._

This is the same argument that's always trotted out. Graphics are improving, but if you diff 2018 to 2013, it's nothing like 2013 to 2008. The fundamental leaps we've been accustomed to seeing are simply not happening anymore. The rate of progress is very clearly slowing down.

It depends which axis you measure, of course. We're able to render more things each year, which is nice. But the visual quality from a fundamental perspective is more or less the same as it was a few years ago.

The quality issues stem largely from light transport -- the colors are all wrong! If you compare them to a photo, you'll see that we don't end up remotely close. You can see this vividly in the YT link above (the Unreal engine walkthrough). If you try to picture yourself _in_ the video, you'll get a strange feeling of being in a candy world, or a shrinkwrapped house.

That's certainly a promising axis to explore, and there are hundreds of papers published each year solely about light simulation.

<comment><USER28> >> Light simulation isn't the issue, motion simulation is. Movement and model and texture complexity all have a long way to go.

> FWIW I completely agree, and this is a key observation. DaVinci devoted > several chapters to the problem of motion in art, and it will always be with > us.

Now we are getting somewhere! Let's talk about those things instead of rendering! Which parts of motion are killing realism? Fluids and rigid body dynamics are pretty good these days. Facial animation is still in the uncanny valley. Why?

> Graphics are improving, but if you diff 2018 to 2013, it's nothing like 2013 > to 2008. The fundamental leaps we've been accustomed to seeing are simply > not happening anymore. The rate of progress is very clearly slowing down.

The rate of progress of _rendering_ is slowing down, that goes right to my point that rendering isn't the main problem anymore, it's approaching good enough for realism. Movement and model and texture complexity progress is increasing. Papers on fluid simulation and facial animation and foliage and texture synthesis and multi-dimensional textures are on the rise.

> The quality issues stem largely from light transport -- the colors are all > wrong!

No. The way we handle colors and light transport is fine, those pieces of human knowledge are ready for the jump to realism. You've already acknowledged that by refusing to consider still photos, because they already look realistic. One bad example from a game engine doesn't prove anything. I can show you lots of bad examples from game engines.

You would have a stronger argument if you gathered the very best examples on earth and we talked about those. By pointing at known bad examples and claiming that they say something about the state of the art, it makes me feel like you either don't know what the state of the art is, or you're taking cheap shots.</comment></comment></comment></comment></comment></comment></comment></comment>


Red Announces EPIC-M Monochrome Black & White Movie Camera  - <AUTHOR> http://www.reduser.net/forum/showthread.php?85214-EPIC-M-Monochrome

<comment><USER2> Leica do a monochrome camera too <http://en.leica- camera.com/photography/m_system/m_monochrom/> with a similar filter (only necessary to capture colour images) removed. Must be an incredibly small market for these, though...

<comment><USER3> Huh. I guess the benefit is increasing the resolution while improving light sensitivity by removing the filter. For a couple of years, manufacturers tried to increase resolution size by packing more sensor elements (which end up as image pixels) on the same area, leading to a higher resolution. This ended up making each sensor element smaller, thus reducing light sensitivity.

So this is quite cool. I have to wonder, though, that even if you don't affect light sensitivity, if a higher resolution is going to be very useful. For instance, the glass you put in front of the sensor might affect sharpness in a way that 3x 18 MP won't look better than a Bayer pattern in front of 18 MP. Does anybody know how much light a Bayer filter eats?

</comment></comment><comment><USER4> What is it and why is it so expensive?

<comment><USER5> It's a video camera that only shoots in black and white.

The reason you would want to shoot in black and white (rather than converting to B&W in post) is because it gives a sharper image, because it allows every pixel to be used for luminosity data rather than trying in interpolate. Read on [1][2] in the announcement thread for more details.

It's expensive because of the market it's aimed at, because it isn't hugely expensive for them. It doesn't cost much more than the colour version of that camera. Spending tens of thousands on bodies, and similar amounts on lenses is more or less the norm for very high end video. If it existed a couple of years ago, it might have been used to film The Artist.

1\. [http://www.reduser.net/forum/showthread.php?85214-EPIC-M- Mon...](http://www.reduser.net/forum/showthread.php?85214-EPIC-M- Monochrome&p=1061850&viewfull=1#post1061850) 2\. [http://www.reduser.net/forum/showthread.php?85214-EPIC-M- Mon...](http://www.reduser.net/forum/showthread.php?85214-EPIC-M- Monochrome&p=1061865&viewfull=1#post1061865)</comment></comment>


Major banks team together to launch real-time P2P payment system Zelle - <AUTHOR> https://www.zellepay.com/ <comment><USER2> A tactic to counter the emergence of the blockchain?</comment>


Show HN: Kodiak – a GitHub app to automerge pull requests - <AUTHOR> https://kodiakhq.com <comment><AUTHOR> Hello HN,

Kodiak is an open source GitHub app that can update, approve, and merge pull requests.

At work we were getting frustrated waiting for CI to merge pull requests, so I built Kodiak to automate this tedious work.

With Kodiak you don't need to wait for CI checks to finish, you simply add a label and Kodiak will merge your pull request when ready. It's super helpful when you have long running CI or "Require branches to be up to date before merging" configured via GitHub branch protection.

It's really simple to get started: [https://kodiakhq.com/docs/quickstart](https://kodiakhq.com/docs/quickstart)

github repo: [https://github.com/chdsbd/kodiak](https://github.com/chdsbd/kodiak)</comment>


Why the Culture Wins: An Appreciation of Iain M. Banks - <AUTHOR> http://sciphijournal.org/why-the-culture-wins-an-appreciation-of-iain-m-banks/ <comment><USER2> As the article says, Banks' thing is post-scarcity [1]. It's not too difficult to imagine PS as one of two possible steady state conditions for us.

[1] [https://en.m.wikipedia.org/wiki/Post- scarcity_economy](https://en.m.wikipedia.org/wiki/Post-scarcity_economy)

<comment><USER3> Technically, we are in a position of resource abundance, and ready for a PS economy. Unfortunately, greed is keeping us down.

<comment><USER4> If we ever want to develop into a Culture-style society, now is exactly the time when we need to start planning the path to get there.

</comment></comment></comment><comment><USER5> This is a really interesting article with a lot of insight into the way that human societies function and spread, and how the Culture works with those ideas.

The thing I find doubtful though - the novels are essentially an exploration of a type of society that may happen if technology progresses in a certain way. It's interesting and fun to read about, but we don't have any particular reason to believe that technology will work in such a way to make such a society possible, or whether it's actually practical if it is.

The nature of technology tends to direct a lot of the ways that societies work. The Culture essentially depends on the development of "perfect" AI. That we eventually figure out how to build General AIs that are smart and effective enough to accomplish useful things, yet never develop a desire to do things other than what their proscribed role is. Why do slap-drones become slap- drones? Do they have to volunteer for the duty, or do they just get manufactured with a desire to be a slap-drone that never changes? Can they actually be effective if they aren't capable of thinking about what they're doing and why? If they do change their minds, can they retire or switch to another job and be replaced? Who gives them their assignments and operating parameters? And the same question applies to all of the other drones and Minds and other machine intelligences that operate all of their factories and orbitals and GVSs and such.

The Culture also seems to be built around everybody in it cooperating with the general goals. In our actual society, there are lots and lots of people who disagree with the general goals, or work towards pursuing different goals that not everyone agrees with. We try to form parameters for how far you can go in various directions and how to deal with people who go past the bounds. We're also generally open to the idea of that - some of those ideas and goals might be better than what we have, so it's good to let them be tried at a small scale so they can be observed and adopted more widely if they work well. We don't see much about how the Culture would deal with such things, mostly just some people and sub-societies that leave or declare themselves separate for various reasons.

<comment><USER4> I think you're misrepresenting the way AI acts in the Culture. They're really just depicted as hyper-competent humans, essentially, with their own goals, quirks, and personalities. The reason The Culture works is because they're self-policing: Banks depicts the AIs as having their own moral compasses, which do occasionally come into conflict, and in stories like Excession you see how the AIs form coalitions and deal with these conflicts. You also see AIs who don't conform to their roles and are refitted, etc. Banks seems to believe in essentially universal moral principles that any sufficiently intelligent entity will have the capacity to reason out, and that there is a semi-inevitable progression towards a better understanding of and adherence to these principles as cultures develop (because cultures that don't will destroy themselves). Or, at least, he believes that humans will successfully pass their own arbitrary moral principles on to AI; they are raised in our culture with our values, after all.

The one thing that Banks does that I really like is that he highlights the fact that sufficiently advanced AI is not just a tool or machine the way we traditionally think of them in relation to ourselves; as thinking beings they are every bit as deserving of dignity as ourselves. Actually, the truth regarding the Culture is that the humans are more or less pets, in the sense that it's the AI's benevolence that allows the Culture to function at all. What makes this notion uplifting rather than depressing is that Banks sees a society of super-intelligences as inevitably benevolent, but not trivially so.

</comment></comment><comment><USER6> > Feudalism with energy weapons makes no sense – a feudal society could not > produce energy weapons, and energy weapons would undermine feudal social > relations.

I do not follow at all. It is easy to imagine a powerful figure to maintain feudal order with help of energy weapons.

<comment><USER7> Feudalism is premised upon a ruling class providing their noble/warrior class with land holdings that they divide among a peasant class that labours, feeds and pays tithe to the noble/warrior and ruling classes in exchange for their land holdings.

If the peasant class can fight as effectively as the knights, then why would you distribute land to an intermediate group for redistribution?

<comment><USER8> You mean like the English at Agencourt? There is a reason the English Civil War happened in England (and not just the obvious). The world would have been a very different place if the Levellers had won [0].

0\. [https://en.m.wikipedia.org/wiki/Levellers](https://en.m.wikipedia.org/wiki/Levellers)

</comment></comment></comment><comment><USER9> I heard Banks talk once and one thing he said was that the key idea with AI is that it will be good.

Most of our stories about AI is that it will look at humanity and want to destroy it. He said that AI in the Culture will look at humanity and actually like it!

</comment><comment><USER10> One problem that all cultures face is that of pecking order. Respecting the pecking order as animals gave us a lesser chance of death by violence within the species. But we've inherited the desire to rise up the status hierarchy and one of the commonest means of doing so is by trying to emulate the most successful. Unfortunately a high proportion of the most successful people appear to be sociopaths who seem to feed off of finding new ways to subtly weaken the culture as a whole.

</comment><comment><USER11> They're kind of odd books because mostly their characters and plots are quite poor-- dull and badly written, and yet the _world_ of the Culture stays with you because it's SO unique and SO well-crafted.

 _Excession_ is the one book in the series which I found to be EXTREMELY enjoyable, good characters and all.

<comment><USER12> Player of games is the only one I loved.

<comment><USER13> I agree with aerovistae and I found the plot of Player of Games particularly weak. Watching the protagonist winning the game thing didn't feel satisfying. It's hard to get excited about a game you cannot understand because it isn't explained. And the big "surprise" regarding a certain character towards the end was lame. It surprised me, a little, only because it seemed too obvious so I expected something more clever. That said, I liked the book because of the Culture.

</comment></comment></comment><comment><USER14> Is there any story about the Culture from within? I mean, how do the billions, that are not involved in Contact, actually live? I suspect we would find that grossly disgusting, and it would destroy carefully constructed appeal of the whole civilization.

<comment><USER4> There are plenty of descriptions of life inside The Culture; they essentially spend their time partying and finding increasingly complex ways to entertain themselves in an environment free of danger or responsibilities. Banks does depict plenty of members of the Culture as really shallow individuals, but their lives are not totally without conflict (largely emotional and intellectual), and there are plenty of people who decide they want to actually take on some sort of responsibility (like those who join contact), and people in the Culture are given that opportunity if they ask for it. One of the great things about the way Banks depicts the Culture is that within the bounds of the Culture's morality people are free to live any way they choose, and even outside of those bounds they are free to form splinter groups that live differently (like the Zetetic Elench).

<comment><USER14> Right - no group "within the bunds of Culture morality" is described closely anywhere. Everything Banks writes about revolves around the Contact or the fringe groups or other civilizations. We are left to imagination as to "the homeland", and my imagination makes me wary.

<comment><USER4> Excession and Player of Games both have sections of the book that take place inside the Culture and show some of their day-to-day lives. They just don't spend much time there because there's relatively little interesting conflict to show.</comment></comment></comment></comment>


Ask HN: who is hiring for Clojure? - <AUTHOR>

Anyone? <comment><AUTHOR> Glad to see that this got upvoted, but sad that there were no comments.</comment>


The People of Postgres: Tom Lane - <AUTHOR> https://medium.com/i-m-h-o/b6f105889466 <comment><USER2> Tom Lane is a great guy. He saved my tail a few months back on the postgresql mailing list when I was up against a real head-scratcher of a bug. It turned out to be a simple misconfiguration on my part, but he patiently walked me through the debugging process, even parsing my ktrace dump to diagnose the issue. My respect and many thanks to this guy. Congratulations to him on the transition.

</comment><comment><USER3> theres quite a few postgres developers that i like. you pop on their channel as a perfectly unknown guy with a complex issue, expose it like a noob, and they actually help you and explain to you what and why.

I find such people amazing. they don't just give you a solution, but explain it well to you. they also seem to understand the problems easily. they're my little heros.

<comment><USER4> RhodiumToad.

</comment></comment><comment><USER5> I clearly remember that somewhere in 2001 I wrote to postgres (-bugs maybe) mailing list about a problem that we had with one of our server (it's a pity I didn't find the thread on the archives). pg_xlog run out of space due to some problem with WAL files recycling mechanism. I got a reply from Tom almost immediately asking for more details. After a few hours he saved my day sending me a freshly baked patch that solved the problem. amazing. I have a deep respect for him.

</comment><comment><USER6> with the wave of other db's (schema and non schema based) that have swept through the industry over the last 10 years, I feel like a small town pastor in my tendency for conservatism and my deeply ingrained faith in postgres.

<comment><USER7> Continuing the religious theme, having been raised in a Pentecostal area (SW Missouri), one of the things I like about PostgreSQL is that I haven't felt the desire to lay my hands upon any of my servers and intone, "Lord, HEAL this database. SHOWER your MERCY upon...."

It's comparable to the quality I've enjoyed with Oracle and DB2 on Solaris and Windows in the bad old days, and has all the features I need to declaratively enforce consistency, correctness and referential integrity.

</comment></comment><comment><USER8> I filed my first ever bug report on Postgres last week and had a good experience thanks to Tom Lane. I was miffed there was no real bug tracker, just "email us", and figured my report would go ignored. Nope: Tom confirmed the bug and patched it in less than a day. Dedication like that is why Postgres has slowly been succeeding. [http://www.postgresql.org/search/?m=1&l=8&q=%238167](http://www.postgresql.org/search/?m=1&l=8&q=%238167)

</comment><comment><USER9> Lane is also the former head maintainer of libjpeg, which means just about any web browser or image editor uses some derivative of his code. That's pretty impressive.

</comment><comment><USER10> So is salesforce funding Postgres development? Or just taking advantage of Postgres' permissive license?

<comment><USER11> I'm speaking for myself and what I think, rather than on behalf or Postgres or any company. I assume you are also referring to the possibility of a private fork.

What I've seen in the Postgres community is a group of developers that takes an aggressive stance against companies "taking advantage" of the developer community. Companies that invest both time and money in development get far more attention for their patches than companies that try to either throw code over the fence, or do "drive by" development projects.

Tom deciding to take this job indicates to me that it is because Salesforce is making a significant investment in open source Postgres.</comment></comment>


Show HN: A super useful code bookmarking chrome extension - <AUTHOR> https://chrome.google.com/webstore/search-extensions/codemarks

<comment><USER2> I would suggest linking directly to the extension [0] instead of the search page also I've never heard of Codemarks before and the extension doesn't have any details on what it actually does, you might want to add that in.

[0] [https://chrome.google.com/webstore/detail/codemarks/mhajcmhk...](https://chrome.google.com/webstore/detail/codemarks/mhajcmhkopfpjkfhbeigigkhdajemnfa)

</comment><comment><AUTHOR> A fellow Starter League alumni did this and I am super impressed. Here is the main site too. [http://codemarks.com/codemarks](http://codemarks.com/codemarks)</comment>


How the Internet Will Change Personal Finance - <AUTHOR> http://thenextweb.com/insider/2012/09/09/how-internet-will-change-personal-finance/ <comment><USER2> Here in the UK none of the banks seem to let you pull out your personal finance data, bank statements etc as a raw data format and use it that way.

For example, I do 99% of my grocery shopping at about 3 shops. So if the data was available it would be trivial to write a script which totals the amount I have spent that month on groceries and graph that against previous months.

The only way I can do this is to go into the banks website and manually add these figures together every month. This just seems like a terrible inefficient way to do it.

It would be great to be able to just go through my bank statement and say "oh , Steam. That's an entertainment expense, so in future put that into a bucket called 'entertainment' so I can see what that is as a percentage and track how it changes". I could also see blips in things like quarterly expenses (utility bills etc) and plan for them accordingly.

What would be truly awesome would be to apply some stats and ML to it and say "If I make large purchase X for £Y in August what is the increase in probability that I will fall into arrears come Christmas based on my historical spending patterns"

If I was feeling cynical I might think that banks don't really want me to plan my personal finances _too well_. They would rather I mess it up now and again and have to apply for credit.

<comment><USER3> I actually solved the "get the data out from the homebanking system" via 100 lines of javascript executed in phantomjs to screenscrape the site.

It isn't pretty, it might break, and I haven't looked into how to store the credentials "safely" (so it executes just when I call it and enter the u/p, and not in the background), but it beats the hell out of logging in and extracting the data by hand.

You might get some mileage of this, if you want to automate the data extraction and have an homebanking account.

<comment><USER2> I actually did consider this option and technically it's probably not too difficult.

On the other hand I'd hate to get used to using this and then have them do some obfuscation deliberately to stop me.

I'd also be somewhat paranoid that if my bank figured it out then they'd come down on me like a ton of bricks and use some convoluted law to freeze my account or send me to prison for "hacking the bank" or somesuch.

</comment></comment></comment><comment><USER4> I can't think of anything in software lagging as much as personal finance. I use GNUCash, which is a glorified balance sheet, and my online bank accounts. This sucks, but only slightly less than Mint which sucks slightly less than Quicken.

Making personal finance decisions should take seconds, not hours, and yet data collection by hand remains the only reliable method to make such decisions. This is unacceptable in 2012.

</comment><comment><USER5> Interesting, but seems to just look at the US landscape. I think the author would be pleasantly surprised if he broadened his field of vision not just to other developed nations but in particular some African countries to see signs of true consumer finance innovation.

<comment><USER6> Africa has plenty of plucky innovation to solve local infrastructure shortcomings, but I think the author is talking about a finance "killer app".

</comment></comment><comment><USER7> Just want to put a shout out to a company run by some good freinds of mine, Pocketsmith [1]. They've got a really good product that is by far the best that I've used (the ones I've been able to use - Heaps, Xero Personal and a couple of bank ones here in NZ all suck). They're linked up to yodeley as well, so they get bankfeeds from all over the world.

[1]<http://pocketsmith.com>

</comment><comment><USER8> Sure but whats the killer app? What can the internet improve significantly? What is really useful to an average consumer? Not someone who is just looking for ways to spend time on a computer.</comment>


Software and the bogeyman - <AUTHOR> http://blog.erratasec.com/2015/07/software-and-bogeyman.html <comment> vezzy-fnord Though I agree that legacy should not be pejorative, BIND is a rather difficult example to defend. This is considering its ideal antithesis (djbdns) exists, which is a point in favor of the original article being criticized here.

The author's presentation of FOSS being a system that selects for what brings the most user value isn't exactly true. What becomes popular tends to be stochastic, and more often than not it's something people are conceptually content with. That is to say, programmers don't know what they want, they want what they know.

Yes, complex software is a reality. The problem is we're generally quite bad at managing it, and end up _complicating_ things. Complexity is intrinsic up to a point, but complication is the gratuitous complexity caused by improper or leaky abstraction.</comment>


The most-cited papers - <AUTHOR> http://www.nature.com/news/the-top-100-papers-1.16224?WT.ec_id=NATURE-20141030 <comment><USER3> Really enjoyed the relationship between this paper's title and the graph of its citation frequency:

[http://i.imgur.com/RO6BVRx.png](http://i.imgur.com/RO6BVRx.png)

<comment><USER4> I think it would have been useful to have the graph reflect discipline's color.

<comment><USER5> The Perdew-Burke-Enzerhof functional, and the generalized gradient approximation, is a Big Honking Deal for accurate quantum-mechanical simulation of solids.

For more: [http://en.wikipedia.org/wiki/Density_functional_theory](http://en.wikipedia.org/wiki/Density_functional_theory)

(I used to be a happy customer, back when I was a scientist.)

</comment></comment></comment><comment><USER6> Flipped through to find whether genomics/bioinformatics would be well represented; there were at least 2 papers on BLAST in the top 15, and 4th paper is the Sanger method. But where is Watson & Crick?

My guess is that all of the subsequent papers that uses BLAST as a tool have to cite it; similarly all sequencing papers cite Sanger as a tool which is why its citation rate dropped when next-gen sequencing method replaced it - <USER2> goes to show citation is not an accurate measure of scientific impact because it is equivalent of citing "Git/compiler/IDE" for a software project.

<comment><USER7> >which goes to show citation is not an accurate measure of scientific impact because it is equivalent of citing "Git/compiler/IDE" for a software project.

The key difference is that BLAST is not a commodity, or at least was not at the time of its release. If there was only one compiler, or one best compiler, we would cite this in CS papers. In fact, "David A. Wheeler's sloccount" is commonly cited in CS papers, as well as Weka, LLVM's Klee, and Z3.

On the other hand, nobody cites Djikstra's algorithm, or optimizing compilers, because those are considered foundational and have not changed in a long time. If BLAST was never supplanted, it would eventually not be cited because that would simply be how sequencing was done. But since it represents a new practice, citing it is necessary to place your work in the context of other work.

I don't see how you can say that BLAST or Klee were not scientifically impactful -- being the force multiplier that enables other avenues of research is possibly the most valuable thing a scientist can do. For example, should a paper that proves P!=NP become instantly the most cited paper in computer science? Though the question is one of the most deep open problems in the field, it isn't relevant at a lot of levels of study, and so it probably wouldn't ever become more cited than Klee.

Further still, I'd guess that the top 1% of cited papers are mostly methods papers, as the article bears out. The next tier are the top-notch findings papers that you mention, because they spark other research. The C45 paper probably doesn't have as much cites as Weka, because Weka is relevant beyond C45, but it certainly has more cites than a less-meaningful machine learning paper.

To carry all of this back to a software project, software projects in their Readme.md always cite their language (like a key method), always cite their dependent libraries (like specific methods used in that work), but never cite things like binary search (a foundational finding). A project MIGHT cite a compiler if that compiler is the only compiler that compiles it! In that case the compiler is not a commodity, and thus worth citing. Version control and editing are totally orthogonal to the actual software, unless you're working in a visual language or something, in which case you would in fact cite your IDE.

<comment><USER6> Thanks for your detailed response. I see your point. Tangential, I used to use Weka and liked it a lot and good to know that it is still very popular despite all of the Big Data/Data Science touting Hadoop/Apache Spark.

</comment></comment></comment><comment><USER8> An interesting follow-up would be to run PageRank on the citation graph. That should lower the importance of the scientific methods papers since they are likely to be cited by a very large number of random papers of limited importance, which bumping up papers that have led to further work that is itself important.

<comment><USER9> Citation count and co-authorship apparently does a better job at pointing out good papers. Even though PageRank was in part inspired by citations studies. [http://link.springer.com/article/10.1007%2Fs11192-007-1908-4](http://link.springer.com/article/10.1007%2Fs11192-007-1908-4) (Paywalled)

</comment></comment><comment><USER10> Two of the top 10 are different editions of a laboratory manual: Sambrook, J., Fritsch, E. F. & Maniatis, T. Molecular Cloning (1989), and Maniatis, T., Fritsch, E. F. & Sambrook, J. Molecular Cloning: A Laboratory Manual (1982).

Citing Laemmli et al was _de rigueur_ for many years, while it was certainly an influential technique, it doesn't rank above the discovery of DNA.

</comment><comment><USER11> What I find interesting is that many papers use references from the 70's and some times even 100 years old, in where they used very crude tools to come to certain conclusions.

I'm not saying we should discard old science discoveries, but it would be interesting redoing the experiments with today's technology.

<comment><USER10> Nobody discards "old" science discoveries, any more than they discard the "old" inventions of steel, or indeed the wheel, they build upon them.

Re-visiting previous studies with newer approaches is a common theme in archaeology, where sites or artifacts are sometimes incompletely investigated on purpose, with the view to leaving undisturbed material that can be investigated in the future using technology not yet conceived of.

<comment><USER12> I hope this doesn't make me sound like a hipster, but I can't remember the last time I saw a wheel.

<comment><USER13> How did you scroll this page?

<comment><USER14> Touchpad/touchscreen swipe?

</comment></comment></comment></comment></comment><comment><USER15> This is purely anecdotal, but one of my favorites is "Medical researcher discovers integration, gets 75 citations."

[http://care.diabetesjournals.org/content/17/2/152.abstract](http://care.diabetesjournals.org/content/17/2/152.abstract)</comment>


Dear Ad Networks - <AUTHOR> http://joeyh.name/blog/entry/Dear_Ad_Networks/ <comment><USER2> Is there any context here? I know about the new Intel microcode license, but I still don't get it.

<comment><USER3> It's a joke.

Since you "can't benchmark" the new Intel patch, ad networks would need to block that IP range, therefore OP would never see ads. Ad-blocking at the legal-level.

<comment><USER4> It still doesn't make sense, how does this guy publishing benchmarks of Ad networks servers, violate their agreement of them not publishing benchmark results?

<comment><USER5> > You will not, and will not _allow any third party_ to ... (v) publish or > provide any Software benchmark or comparison test results.

[https://perens.com/2018/08/22/new-intel-microcode-license- re...](https://perens.com/2018/08/22/new-intel-microcode-license-restriction- is-not-acceptable/)

</comment></comment></comment></comment><comment><USER6> A nice hack but one which I doubt will affect anything. Like those "click wrap" license hacks that people put in their email to companies that say "by responding to this email you grant me a non-exclusive right to use any of your products in perpetuity." which they send to a customer support portal or some such.

</comment><comment><USER7> I love this. I think the proper way at this point to combat these ridiculous anti-benchmarking clauses is through civil disobedience.

</comment><comment><USER8> Current Intel license:

"Unless expressly permitted under the Agreement, You will not, and _will not allow any third party_ to ... (v) publish or provide any Software benchmark or comparison test results."

If the author uses the ad servers to conduct benchmark or comparison tests of Intel software, and the ad networks allow the author to provide or publish the results of those tests, then it could be argued the ad networks are violating their license agreement with Intel. As a preemptive measure to prevent such testing, perhaps the ad networks would block the author's IP address.

The language in the Intel license probably was inspired from similar language first used by Oracle. This language is commonly copied and pasted into many software license agreements.

[http://www.eweek.com/c/a/Application-Development/DB-Test- Pio...](http://www.eweek.com/c/a/Application-Development/DB-Test-Pioneer- Makes-History/)

Question: Is this type of restriction enforceable?

The only way to answer this is for end-users to challenge it. There was a case where a state attorney general challenged it because it was used in a deceptive way. The AG won. However the AG was not challenging this restriction as an end-user. The Court appeared to suggest the restriction would be unenforceable, but was not asked to decide that question. The question was whether the state's consumers were being mislead. Excerpts of that case below.

       Excerpts from  http://www.leagle.com/decision/2003579195Misc2d384_1519.xml

       195 Misc.2d 384 (2003)

       758 N.Y.S.2d 466

       Supreme Court, New York County.

       January 6, 2003.

         </comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>----

       Supreme Court, New York County.

        OPINION OF THE COURT

       MARILYN SHAFER, J.

Network Associates included on the face of many of its software diskettes and on its download page on the Internet the following restrictive clause:

"Installing this software constitutes acceptance of the terms and conditions of the license agreement in the box. Please read the license agreement before installation. Other rules and regulations of installing the software are: "a. The product can not be rented, loaned, or leased-you are the sole owner of this product. "b. _The customer shall not disclose the result of any benchmark test to any third party without Network Associates ' prior written approval. "c. The customer will not publish reviews of this product without prior consent from Network Associates, Inc._" (Affirmation of Kenneth M. Dreifach, exhibit 2.)

In July 1999, Network World Fusion, an online magazine, published a comparative review of six firewall software products, including Network Associates' Gauntlet. It appears that Network World Fusion sought permission to publish the review of Gauntlet and that Network Associates denied it. Network World Fusion performed the review despite Network Associates' refusal to allow the review of Gauntlet. In response to the unsatisfactory results of the review, Network Associates communicated its protest, quoting the language of the restrictive clause.

This conduct prompted an investigation by the office of the Attorney General of the State of New York.

"This language implies that limitations on the publication of reviews do not reflect the _policy_ of Network Associates, but result from some binding law or _other rules and regulations imposed by an entity other than Network Associates._ "

Assume for the sake of discussion, there is some such entity.

That is, assume some entity (e.g., Oracle, Microsoft, Intel, etc.) has a license restriction prohibiting publication of benchmark results.

Does the Court think that restriction would be enforceable?

"Thus, the Attorney General has made a showing that the language at issue may be deceptive, and as such, _the language is not merely unenforceable_ , but warrants an injunction and the imposition of civil sanctions according to Executive Law S: 63 (12) and General Business Law S: 349."

Is the Court here suggesting that even if the restriction was not deceptive, it is nevertheless unenforceable.

Is it possible to read that sentence as suggesting the restriction has the qualities of being both unenforceable and deceptive.

As to unenforceability, no users challenged the enforceability of the restriction. Until they do, we cannot answer the question of enforceability.

However, as to deceptiveness, this can be a violation of state business laws and give rise to grounds for injunction and civil sanctions. This is what allowed the NY AG to take action on behalf of NY state consumers.

AG won. NA lost.

The Court granted a permanent injunction prohibiting NA from ever including the following notice with its software:

"Installing this software constitutes acceptance of the terms and conditions of the license agreement in the box. Please read the license agreement before installation. Other rules and regulations of installing the software are: "a. The product can not be rented, loaned, or leased-you are the sole owner of this product. "b. The customer shall not disclose the result of any benchmark test to any third party without Network Associates' prior written approval. "c. The customer will not publish reviews of this product without prior consent from Network Associates, Inc.";

The injunction also prohibits NA from "including any language restricting the right to publish the results of testing and review without notifying the Attorney General at least 30 days prior to such inclusion". NA was directed "to provide a sworn certified statement indicating the number of instances in which software was sold on discs or through the Internet containing the above- mentioned language in order for the court to determine what, if any, penalties and costs should be ordered."

</comment><comment><USER9> How can I help?</comment>