Researchers Develop Safer Opioid Painkiller - <AUTHOR> https://www.ucsf.edu/news/2016/08/403836/researchers-develop-safer-opioid-painkiller-scratch <comment><USER2> This was the story with heroin, oxymorphone, codeine... Practically every generation was in theory safer and actually turned out to be much worse.

<comment><USER2> I should clarify I'm supposing that by opioid they mean morphine based opioids.

<comment><USER3> You should probably read the article. It's not morphine based.

<comment><USER4> It is targeting mu opioid receptors, mentioned in the article as "morphine receptor."

It may not contain morphine, chemically, but neither does oxymorphone, it's only similar.. the only drug mentioned that actually contains morphine, ie. is morphine based is heroin. But having side chains and other additions can totally change a drugs' effects. Morphine-based is a poor distinction.

</comment></comment></comment></comment><comment><USER5> This is interesting, but doesn't get me too excited. Respiratory depression, whilst a perpetual issue in the the medical use of opioids, isn't necessarily a problem at the lower doses that the evidence suggests actually works. It's a problem at high doses, but those high doses don't provide significant benefits over smaller doses and are associated with hyper-sensitivity to pain and opioid tolerance.

Don't get me wrong though, respiratory depression and arrest is a real thing. I deliberately induce it as part of an anaesthetic every day.

</comment><comment><USER6> Wow, it would be nice if this one pans out. A friend of mine had open heart surgery, and he never once complained about pain in his chest or his leg or the fact that he couldn't push down and couldn't walk far or any of the other stuff that goes along with it.

But the side effects from the opioids were driving him crazy. After a few days he stopped taking them because the pain was easier to deal with.

</comment><comment><USER7> This is what Purdue Pharmaceuticals said about Oxycontin

<comment><USER8> The headline, yes. But if you read the actual article, it seems it's working with the same endorphin receptors without the dangerous side-effects; further, the article hedges its bets by explaining that this is purely in mice so far, and this drug candidate has a long way to go, but as it's a novel opioid that isn't derived from morphine, there is reason for very cautious optimism.

Numerous other novel opioids that aren't morphine derived have shown that it is possible to come close to the pain relief offered, while mitigating the negative side-effects. This is not impossible, I believe, but it'll be interesting to see whether PZM21 is 'the one'.

</comment></comment><comment><USER9> Could we now please get a move on and use the same trick to develop new antibiotics?

<comment><USER10> Roger, it's not the opiates, it's the trick:

"An international team of researchers — . . . Their secret? Starting from scratch — with computational techniques that let them explore more than four trillion different chemical interactions."

This was exactly my idea back in the mid-'70's when I thought of one of the ways computer science could be applied to drug development. Of course all we had were mainframes, but so did the drug companies.

Then, my first employment was with a pharmaceutical producer, I got the impression they were actually further away from this compared to starting a whole new drug company from scratch itself.

Until new antibiotics come along, when necessary you might want to investigate older, more well-established germ-killers like colloidal silver. The real thing is hard to beat, but it is outnumbered by quack versions and get-rich- quick schemers.

</comment></comment><comment><USER11> I wonder if the cost of this research is greater than the cost for lobbyists to legalize medical marijuana.

<comment><USER3> The problem with medical marijuana is that it doesn't relieve severe pain and it gets you stoned which not everybody wants.

<comment><USER12> Not to mention that smoking a plant isn't an efficient or specific drug delivery vehicle.

<comment><USER13> Patients often use liquid tinctures.</comment></comment></comment></comment>


Rust – Coroutines support merged - <AUTHOR> https://github.com/rust-lang/rust/pull/43076 <comment><USER2> This is _experimental_ so far: [https://github.com/rust- lang/rfcs/pull/2033#issuecomment-309...](https://github.com/rust- lang/rfcs/pull/2033#issuecomment-309057591)

> The key point then is that the experimental RFC is not enough to stabilize > something. For that, we need a proper RFC that lays out the plan. This RFC > can be informed by the experience and working implementation, so we should > have a lot of data to use.

It's being merged into nightly (behind a flag) to gain understanding of how the feature would work. This isn't the day to celebrate Rust having fully- fledged co-routines coming soon to stable because there is no such guarantee yet.

Edit: removed extra word

<comment><USER3> This is the first "experimental RFC", so that's also kind of fun.

</comment></comment><comment><USER4> async/await is nice duct tape to integrate cooperative concurrency into a thread-based concurrency model but it has the drawback of creating an incompatible sub-language for functions within the parent language. Now every time you call a function, you have to check if it is a "coroutine"/ returns a promise or if it's a synchronous function.

This divides the language ecosystem and makes it hard for library writers to support both concurrency models.

Ask anyone who has done significant async/await coding in C#/Python. It gets messy fast.

Async/Await works a little better in JS because it always has had a cooperative concurrency model, standard library functions usually don't block (except sometimes in Node). Though it's still annoying to have to check whether a function returns a promise or not.

Languages that do this right are Haskell and Go and probably Elixer/Erlang. This usually requires eschewing LibC (which usually assumes a threaded concurrency model) and writing your own runtime.

<comment><USER3> Yes, this is all discussed in the eRFC (which admittedly isn't what is linked here, but is the first link in the description, [https://github.com/rust- lang/rfcs/pull/2033](https://github.com/rust-lang/rfcs/pull/2033)) and various threads on internals before it.

> writing your own runtime.

This is not a cost that Rust can bear, and so that option is just not possible. Unfortunately, there's no such thing as a free lunch.

<comment><USER4> I wonder if Rust could support pluggable concurrency models (with the accompanying runtime support). That's what Haskell does and it seems to work. At the outset that seems like a better approach than creating an incompatible sub-language. The rust-facing side of the standard library already abstracts away many platform details.

<comment><USER3> Trying this was in fact one of the reasons that Rust ditched its runtime years ago; it doesn't work. Well, it didn't work for Rust at least.

The RFC that made this change [https://github.com/rust- lang/rfcs/blob/master/text/0230-remo...](https://github.com/rust- lang/rfcs/blob/master/text/0230-remove-runtime.md)

<comment><USER4> Interesting. The problems section isn't really convincing to me though.

Especially, indirect function calls for IO functions seem fine since IO is usually much slower than a function call. Also binary size increase may not be a problem for people writing web-scale servers.

It would be great if the Rust compiler infrastructure provided the necessary hooks for re-implementing std::/using a different concurrency model and independent projects could make those "problems" trade-off decisions on a individual basis. Rust-core could just only support/ship the native threaded model to lower maintenance burden for themselves.

<comment><USER3> > The problems section isn't really convincing to me though.

It was a big enough problem that the language was almost forked over it.

> It would be great if

Rust is low-level enough that you can do this, and some people have. But then people have to use your package. std is only special in that it can use unstable code but be part of stable Rust; otherwise, it's just regular old Rust code. The community overall abandoned all of those other things and is coalescing around Tokio, which could also be considered this, in a sense. Or Rayon, if you want data parallelism rather than async IO.

<comment><USER5> Would it be possible to make functions generic over coroutines/async?

One of the worst parts of async is there's an infectious duplication of pretty much all library code where there are separate sync/async versions of everything. See C# DoXxx() & DoXxxAsync() _everywhere_ , where the only difference between the two implementations are an annotation and some keywords sprinkled throughout. If rust can do the same thing without doubling the code/api surface/documentation across all libraries that would allay the fears of many opponents.

<comment><USER6> You can do DoXxx() using DoXxxAsync() by calling wait() on the future it returns. No duplicated code necessary.

<comment><USER5> It's nice that that'll work but it's... kinda gross. Can it call wait() automatically via some auto type conversion? Ideally in a synchronous context it would call wait(), and in an async context it would automatically annotate await (and you could always override by calling wait() manually).

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> Relevant RFC: [https://github.com/rust- lang/rfcs/blob/master/text/2033-expe...](https://github.com/rust- lang/rfcs/blob/master/text/2033-experimental-coroutines.md)

This looks like a huge win for Rust. The syntax seems to follow the C# and ES async/await syntaxes. With stuff like this, I can see Rust-based webdev becoming popular.

<comment><USER8> [http://www.arewewebyet.org/](http://www.arewewebyet.org/)

On a serious note, one of my middling priorities (unfortunately :( It always feels like you can't choose most your high priorities) is to build a rust- based website and a couple of other rust-based projects t get a feel for the language. It really dose feel like all the parts are there already and that as things get added, it'll just get easier and better.

<comment><USER9> Maybe try to find ways to bring rust into the middle of your higher priorities?

<comment><USER8> High priorities being family, house repairs, house maintenance, and work. Nothing extraordinary, and I guess one could argue house upkeep is chosen.

I also have some personal programming projects I've been working on where I decided to value getting them out quickly vs learning a new language, and those are, all together, half done.

I've also been debating learning modern C++ to open potentially open some doors to a slightly different career than web programming (railway signalling, probably a pipedream at this point, but).

So, all in all, I love what rust represents, but it doesn't bring value right now. Once those personal projects wrap up my goal is to have a round of rust and C++ ones to do.

</comment></comment></comment></comment><comment><USER10> [http://journal.stuffwithstuff.com/2015/02/01/what-color- is-y...](http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your- function/)

Coroutines with "color" are problematic.

<comment><USER11> Implicit coroutines are also problematic: [https://glyph.twistedmatrix.com/2014/02/unyielding.html](https://glyph.twistedmatrix.com/2014/02/unyielding.html).

As applied to Go: [https://blog.mozilla.org/services/2014/03/12/sane- concurrenc...](https://blog.mozilla.org/services/2014/03/12/sane-concurrency- with-go/)

</comment></comment><comment><USER12> Trying to wrap my head around this...as someone coming from .NET land, maybe someone can explain it in terms I understand? In F#, the default method of asynchronous programming is the async computation expression. That is, there is a type Async<'T> which represents a computation that can be executed to return a result of type 'T. The computation itself is "lazy" in the sense that it isn't hot at the time of creation, and can be executed multiple times to produce a result. This is in contrast to the C# way of doing things, in which a function has a return type of Task<'T> which is (under normal circumstances) kicked off at time of creation and can only be run a single time. Is this proposal advocating for the more F#-y way of doing things, or the more C#-y way of doing things?

<comment><USER13> C# has what you've described as well in the form of yield return.

It's largely the same thing to the compiler in that it needs to transform a method into a stack-less state machine. Whether or not the state machine is evaluated lazily on demand or concurrently is mostly irrelevant and just depends on the specific problem in question.

</comment></comment><comment><USER14> Here's a rendered version of the documentation added in that commit: [https://github.com/Zoxc/rust/blob/a996d5eec70ba6733e23f2e56e...](https://github.com/Zoxc/rust/blob/a996d5eec70ba6733e23f2e56e762f58e60bb4ff/src/doc/unstable- book/src/language-features/generators.md)

<comment><USER3> This will be located at [https://doc.rust-lang.org/nightly/unstable- book/language-fea...](https://doc.rust-lang.org/nightly/unstable- book/language-features/generators.html) after the next nightly comes out in ~4 hours (I believe).

</comment></comment><comment><USER15> Looks exciting. I'm interested in how this might relate to the following use of generators from Python:

      def gen(*items):
          for x in items:
              if x > 5:
                  yield x

      for y in gen(1,2,3,4,5,6,7,8,9):
          print(y)

Should print:

      6
      7
      8
      9

Does this new RFC support that kind of usage? I guess I'm not clear enough on how Futures work to figure it out myself.

<comment><USER16> Part of the RFC is the intent to use these to create iterators, so yes I believe that kind of usage would be covered.

</comment></comment><comment><USER17> Am I the only one who prefer epoll style concurrency than coroutines?

Are coroutines more somehow more powerful?

<comment><USER18> They're orthogonal? Coroutines as built here are just sugar around building the state machines you necessarily build by hand if you're using something like epoll. They can also be used to model state machines in general, so long as there's a well defined progression of states in one direction.

You would use these coroutines with tokio or mio, which are event driven frameworks that use epoll at their core on Linux.</comment></comment>


Terror attack: Dark day - <AUTHOR> http://www.economist.com/news/china/21598717-brutal-knife-attack-shocks-china-and-raises-questions-about-its-policy-towards-ethnic <comment><USER2> Not to discount these attacks, but 29 deaths and 140 injuries really doesn't compare to 9/11 (2,996 deaths and 6,000+ injuries).

<comment><USER3> National tragedies aren't really a numbers game. I can vividly recall a great deal of how I felt on the morning of 9/11, and it was quite some time after that before I learned how many lives were lost.</comment></comment>


Show HN: Super Looper – Disco Sequencer App - <AUTHOR> http://superlooper.universlabs.co.uk/

<comment><USER2> A bit of quantization (automatically aligning notes to the beat) would really improve this. Not sure how computationally feasible that is though.

<comment><USER3> Very possible with Flash, Java, etc. in the past.

In JavaScript, in particular if you're not using Chrome? Apparently just about impossible (I haven't seen reliable timing yet).

There are a lot of downsides to embedded Flash, applets, etc., but it's going to be a long time until the things that were possible with plugins a decade ago (including decent timing, and even plugging your MIDI keyboard in as an input for a application like this) are possible without plugins.

I have some faith we'll get there eventually; but it's a frustrating time during the gap.

<comment><USER4> you have sample level control- why impossible?

i've implemented naive quantization and MIDI control in my own web apps, without plugins...

<comment><USER5> links or it didn't happen. ;)

<comment><USER4> unfortunately i don't have the code for the step sequencer that used both implementations. but i can share some of the libraries i've released that stemmed from it:

MIDI file parsing/writing/manipulation for js: [https://github.com/kn0ll/midi.js](https://github.com/kn0ll/midi.js)

MIDI control for js (requires a server component, wrote this prior to the web MIDI API): [https://github.com/kn0ll/midi.io](https://github.com/kn0ll/midi.io)

experimental 3d step sequencer (quantized, naturally): [https://github.com/kn0ll/phon2](https://github.com/kn0ll/phon2)

the issue i had with quantizing captured input is the latency between the device input and the time it's read into the program. this is why i said "naive" quantization. but it works fine for 1/8 steps, and 1/16 steps at most tempos.

<comment><USER3> Ha, I post a cynical complaint like this once every few months, hoping someone will prove me wrong one of these days.

MIDI IO where the keyboard is connected to the _server_ is nifty; not what I meant, though. :)

But I can do without that (or only offer MIDI-in support for people who can run a bridge Java applet); still, the thing I keep looking for is reliable timing for on-the-fly audio (constructed MIDI works for me, I can do that!), and responsive changes based on user input.

Put another way; if you had the spare time, could you fix Vexflow playback[1] so it would actually play on tempo, the first time, in more than just Chrome? (Even in Chrome, it's not reliably solid...) Next, what kind of response time can I get when someone clicks on the UI (a keyboard, or whatever)? Is there a way I can play a MIDI note fast enough that it reliably sounds immediate?

I implemented this stuff in Java 11 years ago, mixing audio samples on the fly, not using MIDI, but I'd be happy to use MIDI if it can mean playing multi-voice audio cleanly and w/o loading a library of audio samples; I've spent enough time working with MIDI as well (as part of handling MIDI input).

I didn't try your phon2 yet (should I?), though I notice the latest checkin comment is about battling with audio choppiness. ;)

[1] [http://my.vexflow.com/articles/53?source=enabled](http://my.vexflow.com/articles/53?source=enabled) find "playback"

<comment><USER4> this is my point though. i recognize there are latency and timing issues but they seem related to playback.

but at any given point, you know what sample is being output. you also know if MIDI is being captured. isn't quantization just an exercise of mapping the input to a quantized sample? which is just a matter of rounding assuming you know your samplerate.

ie. if there are 1000 samples in a measure (entertain me), and you quantized to 1/4, and you capture MIDI while sample 700 is being output, you just map the note to the 750th sample. what am I missing?

just want to be clear my curiosity is genuine, I don't have a deep understanding in dsp or signal analysis or anything. my only knowledge of this stuff is just barely making it work for me. so I'm wondering what I'm missing :)

phon2 is definitely worth a try but I just realized the demo link wasn't working. if you're familiar with grunt it should be a quick setup though :) I also realize a server component for midi is not what anyone had in mind; but the web midi api wasn't implemented at the time I needed a solution.

<comment><USER3> > i recognize there are latency and timing issues but they seem related to > playback.

No need to get into DSP issues; it's just that playback and input go hand-in- hand.

Think about playing a click track so that someone can enter notes (by clicking on the screen, for a simple example).

If you have varying (audible) latency in your click track, the user can't hear where the beat is (ugh); secondly, even if you capture exactly when they click the screen, your next task is to map that onto when the beat fell. And if you don't know close enough to even quantize reliably, you're stuck.

Next, the latency between "user clicks a note" and "audio plays selected note" needs to be pretty small -- think of the user above playing along with the beat... and so they click the screen, timed exactly when when the sense the next beat will fall. If the actual audio can't be triggered for another 200ms, it just sounds terrible.

The latency I notice in the OP app is particularly bad because it seems to vary a lot -- I can click a key in a very steady rhythm, and it captures a very unsteady beat.

Quantization can't really fix this problem; the delays seem to be uneven enough (and serious enough) that even if it barely managed to figure out "8th notes", mixing in some triplets or grace notes wouldn't go well.

phon2 -- cool, I'll give it a try when I have a bit of free time.

WRT WebMidi -- that's awesome, I hadn't seen the spec appear; I can't rely on it until there's better browser support (it looks like Firefox has only just started initial work on this, for example), but it's certainly also prove another motivator for fixing up audio timing issues all over.

<comment><USER4> thanks for such a thoughtful response. I totally grasp the complexity of the issue now.

I guess my confusion then lies in the history that I've never had such drastic playback inconsistency in my applications- so midi input synchronization seemed consistent.

I wonder if this has something to do with lack of complexity in my applications. like you suggested I should definitely read up on these playback issues and why and when they're occurring.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> [http://essej.net/sooperlooper/](http://essej.net/sooperlooper/)

Thought it was going to be this (very fun) JACK-based loop-sampler. The name not _exactly_ the same, but it seems close enough to be mildly confusing.

Maybe change the name slightly? (I'm no expert on naming issues; it might just be my lack of coffee)

</comment><comment> vhost- I'm going to turn you in for selling crack. I'm also going to blame you for me getting fired. Thanks for making the next flappy bird. At least for me.

</comment><comment> carl-j I really liked to play with this. Being able to create "music" without knowing anything about it.

Thought I'd share my tune with you guys and perhaps the person who made this sees it and finds it funny.

Rather catchy if I may say it myself.

[http://superlooper.universlabs.co.uk/33un](http://superlooper.universlabs.co.uk/33un)

<comment> carl-j Couldn't help but to make another one... [http://superlooper.universlabs.co.uk/tFMe](http://superlooper.universlabs.co.uk/tFMe)

<comment><USER7> I love how the app fades the notes out when you switch tabs. Very addictive little program.

</comment></comment></comment><comment><USER8> Is anyone else noticing latency? In particular, keypresses seem slow when another element immediately preceeds them.

</comment><comment><USER9> This is insanely fun.

</comment><comment><USER10> Reminds me of Figure, an app (for iPhone) by Propellerhead of Reason fame. See [https://www.propellerheads.se/products/figure/](https://www.propellerheads.se/products/figure/).

</comment><comment><USER4> clicking around the screen without intent has resulted with some more interesting melodies than i had in mind when i started. instruments like this are great for brainstorming.

</comment><comment><USER11> Excellent fun, but when playing with the keyboard, only one of simultaneous keystrokes is recorded.

</comment><comment><USER12> Awesome. If you lower the tempo a bit, you can create pretty neat patterns.

</comment><comment><USER13> Awesome stuff. Would love to be able to save and share à la jsfiddle.

<comment><USER3> See the "record" button to the left of drums -- I hit it w/o realizing what it was for (and managed to record 5 seconds of just one drum going...), but it seems to let you record a session to share.

Edit: here's what one of the save links give you: [http://superlooper.universlabs.co.uk/OHu1](http://superlooper.universlabs.co.uk/OHu1)

or another that's a bit more spare: [http://superlooper.universlabs.co.uk/4wCk](http://superlooper.universlabs.co.uk/4wCk)

The slightly-missed timings are a bit hard to listen to, but this is great fun.

</comment></comment><comment><USER14> Seems to be lacking any way of knowing where you are in the pattern.

<comment> jack-r-abbit Visible grid lines would be great. It seems there is a 5x2 grid since you can use the Q-P keys on a QWERTY keyboard to access them. Q-T is the top 5 and Y-P is the bottom 5. I found the keys to work much better for more accurate placement and timing.

</comment></comment><comment><USER15> Anyone else getting audio clipping at the end of samples?

(Windows 7, Firefox 31)

</comment><comment><USER16> No way to know what keys to hit? Pretty cool though!

</comment><comment><USER17> Add a beat lock and this thing would be perfectttt

</comment><comment><USER18> yep. ok. so. thanks. for making me waste my time at work. this was. just. too much fun. :)

</comment><comment><USER19> Awesome!

</comment><comment><USER20> hah, awesome! had a beat going in seconds :)

</comment><comment><USER21> Very Cool

</comment><comment><USER22> Unfortunatly,your app doesnt hold a straight tempo(in the browser),cool idea,and cool app,but useless if that precondition is not met.

The problem is that even if you trigger something precisely with javascript,the way some browsers handle audio make sequencing totally unreliable.

<comment><USER4> "useless" seems a little extreme. the tool is fun and fluid enough that even if it were to vary within 20bpm i would still use it to get stuff out of my head. prototyping.

for what it is, i think the usability issues are more severe (no quantization, inability to modify a sequence, no pre-roll or indicator of where in the loop you are recording, can't seem to play over another track without recording).</comment></comment>


How Quantum Theory Is Inspiring New Math - <AUTHOR> https://www.quantamagazine.org/20170330-how-quantum-theory-is-inspiring-new-math/ <comment><USER2> If anyone was wondering what that next number was in the sequence, the sequence for the problem given is [https://oeis.org/A076912](https://oeis.org/A076912) and it is now known to begin,

        5, 2875, 609250, 317206375, 242467530000, 229305888887625, 248249742118022000, 295091050570845659250, 375632160937476603550000,
        503840510416985243645106250, ...

If you want a few more details you might check out [https://www.encyclopediaofmath.org/index.php/Clemens'_conjec...](https://www.encyclopediaofmath.org/index.php/Clemens'_conjecture) but of course mirror symmetries have whole textbooks devoted to them nowadays, one of which the Clay Mathematics institute has released to the public [PDF warning: [http://www.claymath.org/library/monographs/cmim01c.pdf](http://www.claymath.org/library/monographs/cmim01c.pdf)].

<comment><USER3> Thanks for the link. On page three this book refers to "high school vector calculus." Who studied vector calculus in high school?

<comment> semi-extrinsic I understand "vector calculus" here as meaning "how to compute stuff with vectors", i.e. adding and subtracting vectors, computing the inner product etc.; not doing calculus with vectors. Obviously both interpretations are possible, but the latter is usually called multidimensional analysis.

<comment><USER4> Surely (multidimensional) analysis is the term used in mathematics for a more rigorous investigation of the basis of calculus. In physics, vector calculus is taught as a tool and the course is called vector calculus (or maybe multivariable calculus.)

</comment></comment></comment></comment><comment><USER5> Is there a simple explanation of what the GCI means?

Preferably without mathematical wording "for such a set that", "for a given function X where", "under a frobnobian measure if" etcetcetc

Bayes has simpler, word descriptions, for example.

<comment><USER6> what's "the GCI" here? I don't recognize that acronym, I couldn't find any term in the article it could be, and googling doesn't really turn anything up...

<comment><USER7> I think they must have commented on the wrong article by mistake. The "Gaussian Correlation Inequality" was recently proved, and discussed in this thread: [https://news.ycombinator.com/item?id=13977554](https://news.ycombinator.com/item?id=13977554)

<comment><USER5> Ah, you are right...

</comment></comment></comment></comment><comment><USER8> Worth noting that all the results discussed in this article are 20+ years old. Bored reporter?

<comment> T-A Bored... something: [https://www.ias.edu/scholars/dijkgraaf](https://www.ias.edu/scholars/dijkgraaf)</comment></comment>


Scoble: Is Windows out and Macintosh in among Silicon Valley's elite? - <AUTHOR> http://www.betanews.com/article/Is-Windows-out-and-Macintosh-in-among-Silicon-Valleys-elite/1302720695?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+bn+%28Betanews+Full+Content+Feed+-+BN%29

<comment><USER2> WTF? Scoble is smarter than this. OS X (and Linux to a lesser extent) replaced Windows among hackers and developers a long time ago. Where's he been this whole time?

<comment><USER3> Exactly, this just leads to another lame mac vs pc war in the comments.

</comment></comment><comment><USER4> When was Windows ever in among Silicon Valley's elite?

</comment><comment><USER5> Yes, as several people have pointed out here, the trend of hackers and other developers going Mac was noted a long time ago. Paul Graham wrote about it in his essay "Return of the Mac" in 2005.

Scoble is talking about something bigger--not just hackers and developers. He's talking the business guys and the marketing guys. He's talking about people that came from Microsoft.

He's also talking about seeing a lot of iPads, and how maybe Jobs is right about the post-PC world.

</comment><comment><USER6> link-bait. Seriously, slashdot was reporting about the OSX mindshare takeover of hackers back in 2001 and the TiBook.

Ever since then, it's been a pretty engrained stereotype, and Apple's offerings have only gotten better.</comment>


Netflix blocking intl streamers who use VPNs to access the US version - <AUTHOR> http://www.independent.co.uk/life-style/gadgets-and-tech/news/netflix-is-blocking-international-streamers-who-use-vpns-to-access-the-us-version-9956469.html?cmpid=facebook-post

<comment><USER2> Luckily US proxies are still working for moi.

I just watched this incredible Doc on the most famous Chinese Artist/Activist : [http://aiweiweineversorry.com/](http://aiweiweineversorry.com/)

fyi, their chat support flat out said "we dont offer different content regionally, you should be able to see everything." ha!</comment>


I'm not IT bitch  - <AUTHOR> http://max-michaels.blogspot.com/2012/04/im-not-it-bitch.html

<comment><USER2> Everything in that post seems valid but when it's coming from an entrepreneur, copping this attitude seems self-defeating. That person who just mistakenly or ignorantly referred to you as desktop support could be someone that could help you in the end.

Please take this as just frankly worded feedback: It's quite possible you think more highly of yourself (or your work) than you really should. You say you agree with Linus Torvalds quote, but chances are you are not at his level (yet) to have earned the right to get so snotty.

</comment><comment><USER3> Wait until your firm is overrun by adherents of the Information Technology Indoctrination Library. Managers will become smug, platitudinous mansplainers (self-styled experts on matters of common knowledge) who believe that everything is IT, even programming, which they think they don't need to know anything about--for them, soft skills rule. And their intellectual apparatus cannot resolve the difference between computer science and IT--brief exposure to ITIL depresses cognitive ability in every measurable level, out of proportion to exposure. If you're not part of the ITIL "service as a service" process, you're part of the problem in their view. Never mind that foisting the mind-numbingly boring ITIL corpus on employees forces the better ones out. You'll be required to sit in on Bolshevist change management meetings with everyone else in IT.</comment>


Operation Rosehub – patching thousands of open-source projects - <AUTHOR> https://opensource.googleblog.com/2017/03/operation-rosehub.html <comment><AUTHOR> This is one of the most impactful projects I've seen built using the GitHub source on BigQuery dataset (since we published it).

If you want to see other use cases - I've collected plenty of other stories from multiple parties at:

\- [https://medium.com/google-cloud/github-on-bigquery- analyze-a...](https://medium.com/google-cloud/github-on-bigquery-analyze-all- the-code-b3576fd2b150)

Disclosure: I'm Felipe Hoffa and I work for Google Cloud ([https://twitter.com/felipehoffa](https://twitter.com/felipehoffa))

<comment><AUTHOR> <meta> Title change by mods \--

I submitted this post as "Googlers used BigQuery and GitHub to patch thousands of vulnerable projects". After it got to #1 on the front page, mods silently changed the title to "Operation Rosehub – patching thousands of open-source projects"

I wish HN had a more transparent way to show that the mods changed a title and why. Since HN does not, the least I can do is add this info for transparency.

(related [https://news.ycombinator.com/item?id=6572466](https://news.ycombinator.com/item?id=6572466) [https://news.ycombinator.com/item?id=4102013](https://news.ycombinator.com/item?id=4102013))

</meta>

<comment><USER2> I got a bit confused by your post. I gather that you are simply recording the fact that the title was changed in accordance with the posting guidelines, not that you are complaining about it. Just in case other people were similarly confused...

<comment><USER3> I think he wishes that here was an "edited" annotation next to changed titles.

</comment></comment></comment></comment><comment><USER4> I've built something like this for Python projects.

You add your repo and a bot is constantly checking for insecure and/or outdated packages and sends you a pull request if you need to update.

It's free for open source projects at [https://pyup.io](https://pyup.io)

<comment><USER5> Why does your service ask for write access to my repos?

<comment><USER4> For the pull request. The bot creates a branch, commits the changes and sends you a pull request on your repo.

This it how it looks like: [https://github.com/pydanny/cookiecutter- django/pull/1065](https://github.com/pydanny/cookiecutter-django/pull/1065)

<comment><USER5> Thanks for the reply. I guess forking each repo wouldn't scale well?

</comment></comment></comment></comment><comment><USER6> So many questions... What does this say about Google's hiring, about its employee's values, about values across the tech community? I can remember a time when managements would have shut this down, when employees would have said, "not my problem", when entire industries would have buried their heads in the sand.

Is it the lack of liability and regulation that clears the way for this kind of corporate citizenship? Is it cultural?

<comment><USER7> It may be, in part, Google's giant cash machine. It makes it possible to be altruistic. In a world of fierce/commoditized competition it is much harder to expend resources on 'side' projects.

Google also thrives in a healthy internet.

<comment><USER8> "They were happy to see employees spontaneously self-organizing to put their 20% time to good use."

Reminded me that I heard that Google's 20% time had generally like 10% participation and has had a number of conditions including manager approval for the last 4 years. Is this article breathing new life into the myth of 20% time or does this reflect the revival of the process?

Either way, incredible accomplishment on the patch army!

<comment><USER9> 20% time has always meant different things depending on who you talk to. It's likely that you're just talking to different people.

I took copious amounts of 20% time in my time at Google (2009-2014). Usually I'd start a 20% project with neither my manager's knowledge nor his approval; if it looked like it had legs, I'd let him know about it and ask what he thought. I never had a manager outright forbid me from working on a 20% project; responses ranged from "You should consider this your main project now; it's critically important that we understand this area" (along with a spot bonus for delivering on it) to "Well, you can work on it, but you are unlikely to get credit for it come promo time." In general, as long as I got my work done, my managers didn't care what else I was working on.

<comment><USER10> Was your workload such that you could put in a reasonable number of hours on your 'work' and still have 20% of your time for other projects? Or was 20% time just putting in an extra 20% on top of what other developers (who didn't do 20% projects) would do?

<comment><USER9> Over my career there, yes, I could put a reasonable number of hours on my work and still do 20% projects. There were short periods of time when I was asked to "bank" the time and focus entirely on my main project, eg. I might work straight for 5 weeks on my main project and then take a week straight of 20% time once it launched.

This suited me fine - usually the way I took 20% time was to fit it into time periods when I didn't really have much else to do or I was bored with my main project. And splitting it up like this let me focus more intently on both of them, which helped in delivering.

<comment><USER10> Cool, sounds like it worked out for you about the way I would expect. I suppose it may depend some on your team, I have heard some people say it is just 20% extra work.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> I like the "bank teller" analogy used in the article.

 _> it would be like hiring a bank teller who was trained to hand over all the money in the vault if asked to do so politely, and then entrusting that teller with the key. The only thing that would keep a bank safe in such a circumstance is that most people wouldn't consider asking such a question._

This does not only work for deserialization issues.

It is a great analogy for a huge class of IT security issues!

Maybe we should use that one when communicating with the media. This this works much better than the usual burglary analogy. I like how it points out that this is about stupid and/or malicious behaviour (code), where the attacker (hacker) just needs curiosity, and may find this out even by accident. The attacker did not have to break something, and did not damage anything, to get into something. In particular, this makes clear that this is caused by irresponsibile behaviour of the organization and/or other entities to whom they delegate trust.

Even for more complicated scenarios, I like the bank teller analogy more than the classic burglary analogy. In that case, the attacker observes multuple bank tellers, and notices e.g. that if you ask the first teller for form A and put in certain words, another bank teller will accept it and give you a stamped form B, which you can show to a third teller in another branch office who will look a bit confused, but finally accept it and hand over all money to you.

We need to get over blaming the messengers[1], buying zerodays and declaring cyberwar. What we really need to do is to finally make our[2] computer systems secure and trustworthy, at least up to a certain minimum-level of sanity: no exec, no injection (i.e. typing/tagging), no overflows (i.e. static analysis), input validation, testing, fuzzing, you name it.

And this cannot work by just adding more and more complex security measures outside, but more importantly simplifying and cleaning up inside. Although rewriting software from scratch is very risky, radical refactoring is not! And every good software engineering course tells you how to do it correctly.

[1] security researchers, but also "amateur" hackers, or just someone running into it by accident because the security issue became so large it finally _had_ to be noticed by someone.

[2] in the sense of: everyones!

<comment><USER12> I know it is not correct to ad "me too" comments here, but I went here for the same quote as you. It is the best quote I have never seen about security because it does not depict a evil hacker that breaks a not secure enough wall. It depicts a clever client that goes through a stupid security hole. That's a better analogy for 99% of security hacks.

</comment></comment><comment><USER13> Is [https://libraries.io](https://libraries.io) not a more comprehensive and community-focused response to the same problem?

libraries.io did make it to the front page a few months ago, but I think its underlying vision might not have been driven home from just glancing at its home page. It supports 33 package managers (not just Java, though I'm sure Rosehub doesn't just do that either) and Github/Gitlab/Bitbucket, not just Github. And it provides both email notifications and auto PRs.

But that's just the overlap with Rosehub. On top of that it offers the means to discover libraries based on a Dependency Rank (think Page Rank but using dependencies instead of hyperlinks). Which in turn allows it to surface projects with a high "Bus Factor" \-- projects maintained by few committers, but depended on by many (so they'd be more affected by said committers getting run over by a bus). AND it mines the licenses for a project, notifying if any of the dependent licenses are incompatible with the parent license. What's more it's a non-profit organisation receiving enough funding to employ 2 full time devs.

I think libraries.io is Rosehub and more, to quote the about page;

      Our goal is to raise the quality of all software,
      by raising the quality and frequency of contributions
      to free and open source software; the services,
      frameworks, plugins and tools we collectively refer
      to as libraries.

To take the liberty of extrapolating from the libraries.io vision: open source security isn't just about fixing patches, but about supporting the environment, people, conditions and tools that contribute to open source software.

<comment><USER14> I see nothing on the libraries.io website that explains how it would be used to solve the problem described in the OP.

<comment><USER13> OP here, yeah I agree, like I said the home page could be more explicit.

Here's some links:

[https://libraries.io/about](https://libraries.io/about)

[https://libraries.io/bus-factor](https://libraries.io/bus-factor)

[https://github.com/librariesio/lib2issues](https://github.com/librariesio/lib2issues)

</comment></comment></comment><comment><USER15> I am extremely sad that this turns into an argument _for_ making certain that all source code in the world is at least indirectly accessible specifically via GitHub (at which point people will find it there and expect the developers to respond and generally track everything going on there, even projects which are much happier using more open tools); like: it isn't sufficient that your code is "open", it actively has to be part of the unified GitHub empire.

<comment><USER16> Your gitweb [0] has always worked perfectly fine for me! I agree, I don't see the need for Github.

[0]: [http://gitweb.saurik.com](http://gitweb.saurik.com)

</comment></comment><comment><USER17> In their query they do:

       FROM (SELECT id,content
          FROM (SELECT id,content
             FROM [bigquery-public-data:github_repos.contents]
             WHERE NOT binary)
         WHERE content CONTAINS 'commons-collections<')

Why the subquery? Why not WHERE NOT binary AND content CONTAINS...? is this a bigquery thing?

<comment><USER18> This appears to be "legacy SQL" in BigQuery, which did not have query optimization - entirely rule-based query planning. The query is a little inefficient indeed.

BigQuery has since released ANSI 2011 "standard SQL", which would does have an optimizer and would push predicates down).

(work on GCP and worked on BQ until recently)

</comment></comment><comment><USER19> Wow. I wonder how much a query that searches the content of all of Github costs (if you're not Google). This page says the dataset is 3TB+ [https://cloud.google.com/bigquery/public- data/github](https://cloud.google.com/bigquery/public-data/github) and presumably most of that is content.

<comment><USER20> The contents table [0] which they ran the query on is 1.8TB. Assuming you only need to do a single pass (seems reasonable given that it's a simple regex), the price should be about $9 [1]. Free quota covers 1TB so the remainder would be $4.

[0]: [https://bigquery.cloud.google.com/table/bigquery-public- data...](https://bigquery.cloud.google.com/table/bigquery-public- data:github_repos.contents?pli=1&tab=details)

[1]: [https://cloud.google.com/bigquery/pricing](https://cloud.google.com/bigquery/pricing)

</comment></comment><comment><USER21> Nice! That's some good citizenry.

Interesting fact: Justine was the founder of occupywallst.org, which was the highest-trafficked publisher/web hub for the Occupy Wall Street movement before she worked for Google.

</comment><comment><USER22> "Patches were sent to many projects, avoiding threats to public security for years to come."

Are these pull requests that the project would still need to approve/merge or were they just pushed in?

<comment><USER23> They were PRs that required approval and merge from the Github project maintainers. Here is a search to see some of their work:

[https://github.com/search?q=%22Upgrade+Apache+Commons+Collec...](https://github.com/search?q=%22Upgrade+Apache+Commons+Collections%22&type=Issues&utf8=%E2%9C%93)

<comment><USER24> It's actually incredibly interesting to read how the developers individually responded to each of these PRs.

It would have been great to see a count of how many PRs have been accepted.

<comment><USER25> edutechnion's link says 1108 open PRs and 999 closed.

Interesting that 2100 of the PRs are "Upgrade Apache Commons Collections to v3.2.2" and just 7 were "Upgrade Apache Commons Collections to v4.1".

<comment><USER26> Probably v3.2.2 was lower hanging fruit for most projects. Instead of having to make code changes.

</comment></comment></comment></comment></comment><comment><USER27> As scary as Google's massive size and power is, it's pretty awesome that they're incentivized to do things like this to help the internet because they _are_ the internet.

</comment><comment><USER28> I read so many of these kinds of articles out of curiosity and rarely understand them.

Thank you for adding in the part about the bank teller.

For reference: "it would be like hiring a bank teller who was trained to hand over all the money in the vault if asked to do so politely, and then entrusting that teller with the key."

</comment><comment><USER29> > But unlike big businesses, open source projects don't have people on staff

To read that from Google is frankly disappointing. While this is true of many open-source projects, it doesn't have to be that way. Red Hat (and Google!) are brilliant proofs of this.

<comment><USER11> More generally, if a company uses software X (open-source or not), they need to:

a) make a contract with a company that takes responsibility for X, or

b) hire somebody who takes responsibility for X, or

c) take responsibility for X on your own

It doesn't help to "buy" closed-source software X from another company if you can't count on them in case of emergency, i.e. if they vanish, go bankrupt or put their lawyers onto you.

Then, better take open-source software where you can take responsibility on your own, for which it may help to hire one or more of the lead developers.

</comment></comment><comment><USER30> Really cool, kudos to people helping with this. I wonder if this could have been done in a way that non-Googlers could have pitched in too, given that this is for a public good -- but it's tricky with security issues.

</comment><comment><USER31> How does it work for transitive depandancies ? If you use a package that use a vulnerable Apache common? Does a pr is sent to update the package when it is updated?

</comment><comment><USER32> If I understand it right, this bug involves code pulling in old buggy libraries, sometimes indirectly via other libraries. It seems that there is a reference to a specific bad version, not the actual inclusion of cut-and-paste code.

Eh, why not just get rid of the bad version? Alternately, release a bug-fixed copy with the same version number.

Any breakage is a case of "oh well, you're safe now". Leaving the security hole is probably worse breakage.

<comment><USER33> The bug is not including this library. This library is 100% secure [i mean if this is a vuln in this library then a large proportion of libraries are insecure because they could be leaked to untrusted code and used to break the JVM trust model]. It just so happens this library used to contain a really- really useful gadget for exploiting another security problem. However, removing this gadget doesn't mean the security problem is fixed. There are other _fun_ libraries. In fact classes similar to the Mad Gadget have been used in the JDK to escape the sandbox in the past. Yes, stuff like this exists or has existed in the JDK [[https://github.com/jenkinsci/jenkins/blob/96a9fba82b85026750...](https://github.com/jenkinsci/jenkins/blob/96a9fba82b850267506e50e11f56f05359fa5594/test/src/test/java/jenkins/security/security218/ysoserial/payloads/Jdk7u21.java)].

And this work is very useful in so far as I'm sure the benefits it provides is going to massively outweigh the cost. However, if you have a naked ObjectInputStream#readObject in your code then you probably still have an exploitable security issue. Have a look at how well Jenkins strategy was to fixing this issue which was basically the same strategy as Operation Roshub. ie: removing the ability to access classes that were known to be used in gadget chains. Surprise, surprise it didn't last very long and people just found new gadgets.

And if you read this blog post then you might be mistaken into thinking that removing commons-collections from your classpath or upgrading commons- collection to the 'safe' version would make object deserialization safe but this is not the case. if you have a naked ObjectInputStream#read in your code then you are vulnerable to remote code execution.

<comment><USER34> Author here. There are individuals in the infosec industry who agree with you. They've stated on many occasions that the problem isn't gadgets, but rather that programming practices in general need to change. This might have something to do with the fact that no one told Apache about this weakness until nearly a year after it was presented at an infosec conference.

While gadgets may not be the root weakness, the gadgets certainly help. We may never be able to have perfect security. Hopefully the systemic paradigm shift infosec professionals are advocating will come some day. But until that day arrives, we can make people so much safer, with minimal effort, by simply disabling these gadgets.

Almost no one uses them. Out of all the projects I found, I was only able to identify one or two that were legitimately using the gadgets in question.

<comment><USER33> Thanks for your reply. This is Thursday in the UK so I'm going to pre- emptively apologise for this rant. But we in the infosec community informed the wider community in 2008 of the problems of Java Serialiazation. That is 8 years ago. Sami Koivu, peace be upon him in December 2008 showed that arbitrary deserialisation in Java was a security risk ([http://slightlyrandombrokenthoughts.blogspot.co.uk/2008/12/c...](http://slightlyrandombrokenthoughts.blogspot.co.uk/2008/12/calendar- bug.html)). Not to mention that SERIAL-5 ([http://www.oracle.com/technetwork/java/seccodeguide-139067.h...](http://www.oracle.com/technetwork/java/seccodeguide-139067.html)) of the Java Security Guidelines for Java SE has this to say:

 _Guideline 8-5 / SERIAL-5: Understand the security permissions given to serialization and deserialization Permissions appropriate for deserialization should be carefully checked. Additionally, deserialization of untrusted data should generally be avoided whenever possible._

And do you want to have a guess as to how many times Serialization was used to bypass the Java Sandbox between when Sami Kouvi made his blogpost and someone made a con talk on about Apache. Hint: it is greater than 1. [[https://tyranidslair.blogspot.co.uk/2013/02/fun-with-java- se...](https://tyranidslair.blogspot.co.uk/2013/02/fun-with-java- serialization-and.html)]

We have also demonstrated numerous times to the programming community that deserialzation of user data is dangerous. For example Stefan Esser has shown numerous times that PHP deserialization is dangerous both because PHP deserialization is a source of bugs in itself and because it is a source of bugs because it interacts with application code in unexpected ways. We have seen the same thing in both python with unpickle and ruby with YAML.

I'm going to let you in to a secret within the infosec community. You can find bugs by just applying existing research in new and novel ways because developers do not follow security research.

I feel like I'm falling into some rationalism fallacy by ranting at you because you are doing something useful to improve security. But you could be doing much much more. You have a voice and people will actually read your blog as compared to Sami :( You could have mentioned that we people should stop doing ObjectStream#readObject() or you could have pushed for updating the JavaDoc to say: THIS IS A BAD THING DO NOT DO IT.

EDIT: apologies to anyone that realized that java serialization was bad before the Sami post. I wouldn't be surprised if this was part of the Java secure code guidelines before then or if someone had exploited the issue before then. It just so happens that Sami's post was my introduction to Java Serialization vulnerabilities.

<comment><USER35> As one of the people who did the talk at Appsec Cali that was building on all this work outlined by benmurphy... our goal was to reach security minded developers and talk about a repeated anti-pattern that put software at risk that impacts things written in many different languages. Both Chris and I have a development background, and have seen the same issue show up in ruby, python, php, basically anything that has an object serialization capability. We hoped to change the focus from a specific library or gadget to the idea that deserialization is inherently dangerous.

The core problem really stems from the idea that OO models encapsulate data and behaviors. Behaviors mean code execution - so, anything that will deserialize objects is giving the person who serialized them the ability to control the execution flow. If this is a listener on the network, than things are really bad :-)

So, it's great that a set of gadgets have been removed, it's neat to see the application of resources to make that happen. I have to agree with Ben, that any system that relies on object serialization from untrusted sources (in any language) is still vulnerable, it just might require a more specific gadget chain. Too many vendors have fixed their products by just updating the library and not removing the dependency on dangerous object deserailization.

<comment><USER34> Why did no one tell Apache?

<comment><USER35> [https://blogs.apache.org/foundation/entry/apache_commons_sta...](https://blogs.apache.org/foundation/entry/apache_commons_statement_to_widespread)

"So replacing your installations with a hardened version of Apache Commons Collections will not make your application resist this vulnerability."

<comment><USER34> Ok well you can tell your cohort that this narrative isn't going to fly anymore.

<comment><USER35> What evs

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER36> I wish you could do the same thing with mental illness.. massively send pull request to correct everyone's bad brain code.. <sorry>

<comment><USER37> This is a genuinely cool idea. Seriously.

It raises a lot of questions about what sort of transformative spectrum (excuse pun) would be applied here though. It's is incredibly abstract as presented.

But even at the abstract level, the one thing I know would absolutely happen for sure is that the fixes that made the biggest difference would be hand- waved out of existence by infecting them with viruses, creating scare- campaigns, etc.

Source: I've learned a lot about Big Pharma over the past 10 years as I've quietly found real solutions to my own mental health issues. I'm sadly too scared to share what I've found and I keep seeing products disappear off the market or suddenly attract customs/overseas shipping issues. Suffice it to say that the medical industry is opposed to anything they can't patent - and that, as an industry, it must ensure its own survival. Interpret that any way you see fit.

</comment></comment><comment><USER38> It's interesting that this type of initiative, which is admirable, will spike up some java "popularity" metrics on GitHub.

</comment><comment><USER39> I was thinking about doing something similar with bigquery and github data to search for uses of strncpy in C code. But I am not that good with the query language and also bigquery didn't support multiple users properly (this adds friction).

I still think it's a good idea. It would be even better to search for a few C pitfalls more, but strncpy is probably the easiest to search for.

</comment><comment><USER40> I think this is one good concrete example of why the npm style of private dependencies for each lib is not the greatest thing ever, while the non- recursive style in python (or C) is overall more manageable (if you are actually managing your dependencies instead of ignoring them).

</comment><comment><USER41> we have been doing thus for a while now : [https://www.sourceclear.com/blog/millions-of-program- builds-...](https://www.sourceclear.com/blog/millions-of-program-builds- vulnerable-to-man-in-the-middle-attacks/)

<comment><USER42> Got a 404 on the above, looks like it should be: [https://www.sourceclear.com/blog/millions-of-program- builds-...](https://www.sourceclear.com/blog/millions-of-program-builds- vulnerable/)

But I don't see where it discusses sending PRs to affected repos, only detecting them.

<comment><USER41> Ah yeah I had the old link, thanks for fixing. Actually we privately disclosure the problem to the developers and get it fixed following responsible disclosure and not post PRs directly.

</comment></comment></comment><comment><USER43> So, Google does ... something and is showered with praise.

Thousands of volunteers work in the saltmines and get nothing.

Business as usual. Myths like "Google sponsored Python!!!" propagate when they do nothing at all.

Disgusting.

</comment><comment><USER44> That's outstanding news. Hats off to the volunteers doing the work on this.

</comment><comment><USER45> Mad thank-yous to Google for this!

</comment><comment><USER46> Awesome! Contgrats to the team!

</comment><comment><USER47> What is in it for google?

<comment><USER48> Employee satisfaction.

<comment><USER49> And a healthier internet

</comment></comment></comment><comment><USER50> Operation Rosebud

</comment><comment><USER51> Wouldn't a graph database be a more suitable tool for that kind of task?

<comment><USER52> Why would it be a more suitable tool? What can a graph database tool do that BigQuery lacks?

If you've already got the data conveniently preloaded into a SQL database for you, and all you need is a very simple SELECT statement with two WHERE clauses ... why would you use anything else? Spinning up an entire graph database unnecessarily seems like over-engineering.

<comment><USER34> Author here. There's some truth to what he's saying. One thing I've been meaning to do is get my hands on all the Maven pom.xml files that exist, so I can load them into a Guava Multimap (my graph database of choice) and figure out every single artifact that will transitively inherit vulnerable collections on the class path.

<comment><USER52> Try Neo4j?

<comment><USER34> Why use Neo4j for a program you're going to run once and not store the data? Multimap<T, T> works great.</comment></comment></comment></comment></comment>


Rare Are GC Talks - <AUTHOR> http://furious-waterfall-55.heroku.com/ruby-guide/internals/gc.html <comment><USER2> Wilson's highly readable "Uniprocessor Garbage Collection Techniques" is a far better overview: [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.138....](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.138.5038) You could write a decent GC with just the info in that paper.

It doesn't cover parallel or real-time garbage collection (which both get far more complicated); for those, you want Jones et. al's _The Garbage Collection Handbook_ ([http://www.amazon.com/The-Garbage-Collection-Handbook- Manage...](http://www.amazon.com/The-Garbage-Collection-Handbook- Management/dp/1420082795/)) and plenty of time to explore its bibliography. (His older book is also good, but doesn't cover those topics.)

<comment><USER3> That _is_ a good paper. Thanks for the pointer!

</comment></comment><comment><USER4> The reference counting metaphor leaves out a critical aspect of reference- counting: cyclical structures and how to collect them. Books never write their names in other books so this doesn't appear in his discussion, though it's one of the more important complications for implementations.

<comment><USER5> Honestly, this article is really bad and probably harmful to new programmers. I recommend against taking anything in it to be accurate, much less complete.

The author misses vitally key issues (like the inherent dangers of circular references in reference counting "GC"). I wouldn't even call myself slightly knowledgeable in GC, yet blatant holes in the clumsy analogies of this article are obvious to me. As someone who is not even slightly knowledgable in GC, it scares me that people are learning from someone who has massive information holes in the explanation -- holes in topics that every programmer on earth should know.

On top of all that, the only "Cons" that are mentioned of RC are absurd: "It's annoying if you forget to release your references." Wow. Profound stuff there - bugs and incorrect code is annoying. This isn't even a problem if your language supports RAII, in which it's impossible to "forget" in the first place.

<comment><USER6> I sincerely doubt people are going to read this, and then immediately write and ship a garbage collector. For a square-zero general-idea overview to someone who has zero concept of what a GC does, it's more than accurate enough.

It could probably be about 1/2 the size, as it shoots off in weirdly-complex directions for such an introductory text. And it could probably introduce e.g. reference cycles, because they're a real problem with reference counting. But it's a clearer and simpler text by far than I generally see on this topic.

</comment></comment></comment><comment><USER7> One of the most fun discussions of the complications of reference counting and how nuanced you have to be comes from the Python docs:

[http://docs.python.org/extending/extending.html#reference- co...](http://docs.python.org/extending/extending.html#reference-counts)

It relates "a true story. An older version of Python contained variants of this bug and someone spent a considerable amount of time in a C debugger to figure out why his __del__() methods would fail" in a considerable amount of detail.

</comment><comment><USER8> _At the December 2008 Kyushu Ruby 01 Conference, I asked the audience "How many of you here have some interest in garbage collection?" Out of 200 people, only 3 raised their hands_

Can you imagine how many people would raise their hands if you asked that question NOW at a ruby conference? Crazy how times change.

<comment><USER9> can you explain a bit why that has changed? I don't really know anything about ruby, but I would have thought that being able to ignore abstractions like garbage collection was the reason that people used high-level languages. Does programming in ruby involve a lot of interaction with the garbage collector, or are you saying that a lot of the ruby community is actually becoming involved with writing and improving low-level language features?

<comment><USER10> My experience in this regard comes from Java, not Ruby, but I think the basic concepts are the same: You are correct in that, for a lot of uses, ignoring abstractions like GC is just fine.

If you are building something that uses a lot of memory, requires high throughput or low latency, or are getting into situations where GC is taking up a decent chunk of your CPU time, then it can pay to have a better understanding of the GC concepts so you can make beneficial changes to your application or to the GC parameters.

Plus, stuff like GC is kind of interesting by itself for some people. Some of my favorite sessions at JavaOne were always the GC and JVM internals ones, despite the fact that their immediate utility to me wasn't always obvious.

</comment></comment></comment><comment><USER11> Another great GC discussion for LuaJit: <http://wiki.luajit.org/New-Garbage- Collector>

</comment><comment><USER12> Interesting that there's no mention of generational GC in this article. Anyone with a bit more expertise care to comment on why not?

<comment><USER13> Yeah, I noticed that. I first heard of generational GC in association with Smalltalk VMs and I think that's where they made their debut.

<comment><USER14> The first generational garbage collector was for Lisp:

[http://web.media.mit.edu/~lieber/Lieberary/GC/Realtime/Realt...](http://web.media.mit.edu/~lieber/Lieberary/GC/Realtime/Realtime.html)

</comment></comment></comment><comment><USER15> Articles like this are exactly why I come to HN.

<comment><AUTHOR> Agreed! Also, beyond the great tech content there's something charming about this one.

</comment></comment><comment><USER16> In the article he states "[for Copying] Conservative GC is unable to determine the difference between pointer and integer values. If an integer value was overwritten the result would be disastrous. It is likely that the result of 1+1 would change occasionally."

However reading through the RHG, it is shown that Ruby allocates memory for objects in 20 byte blocks and that this results in all pointers therefore being multiples of four. This is handy as it allows the direct usage of literals such as FixNum (an always odd 'pointer' that you shift right by one bit to access the value) and Symbol (a 'pointer' that always has 0xff as the trailing byte that you shift right 1 byte to access the unique ID) without requiring object creation.

With this in mind, can someone enlighten me as to why Copying could not be used inside Ruby? It seems as though it would be trivial for the GC to differentiate between literals and pointers as otherwise they would not be much use as literals.

</comment><comment><USER17> I don't see how a Copying GC can be faster than Mark and Sweep. Transferring a book may be easy, but copying an object can be quite costly. I can't imagine copying almost all objects everytime the GC runs.

<comment><USER18> Suppose that the program only needs about 1 MB of objects, but it will fill up 100 MB before performing a copying GC. In mark and sweep, you need to perform an operation for every garbage object (of which there are 99 MB), whereas with copying GC, you need only perform operations for every live object (of which there is 1 MB). Even if copying an object takes ten times as long as returning a chunk of memory to a free-list, the copying algorithm is faster. (Also, popping a free cell off a free-list is more expensive than incrementing an allocation pointer.) Of course, either way, the total work done for allocating objects and freeing them is O([number of objects allocated]).

Something like this is described in detail here: "Garbage Collection Can Be Faster Than Stack Allocation", Andrew Appel: <http://www.cs.princeton.edu/~appel/papers/45.ps>

<comment><USER2> Not necessarily -- lazy sweeping only needs to do work proportional to live data (like copying GC), rather than proportional to dead data.

I have a pretty straightforward implementation of a lazy mark-sweep collector here (in C): <https://github.com/silentbicycle/oscar>

Basically, instead of marking live data, then sweeping all unmarked data, you just mark live data, then the next time a slot is requested, you step the allocator over the heap until it finds the next unmarked cell and re-use that. If most cells are dead, this happens very quickly. If you have to sweep over too many live cells (left deliberately vague), you grow the heap.

<comment><USER18> Interesting... I still feel that this is for the most part merely deferring the free-ing work to when you next allocate an object. It may succeed in getting the "free" cost down to a single instruction--checking a mark bit--but it is still a cost per dead object that a copying GC wouldn't have to pay. (The difference may be, as some say, "academic".) It does have the slight advantage of reducing the maximum pause time due to sweeping down to being proportional to live objects rather than garbage.

Also, what if you want to allocate a large object? You might have to skip over some unmarked cells. Either you'd waste the memory, or you would probably want to put them on a free-list of some sort, in which case you are doing more than just bumping a pointer and checking for a mark. ... I see, it says "equally- sized chunks of memory".

Still, that's a cool idea. As mark-and-sweep goes, anyway. :-} I'm rather more fond of copying garbage collectors--I like the determinism, the fact that (normally) the memory address of each object after a full copying GC will depend only on the structure of the graph of objects, and not on the memory addresses of the objects prior to the GC. (In other words, it destroys the information "what the memory address of each object was before this GC". This implies a somewhat higher inherent cost, to eliminate this entropy. See [1] for musings about the relation of this to thermodynamics.) In particular, whether you have fragmentation issues won't depend on the past history of your program. (You could say that the space lost to fragmentation is fixed at a factor of 2.)

[1] <http://www.pipeline.com/~hbaker1/ThermoGC.html>

<comment><USER2> Checking the mark bit can be quite cheap, though, particularly if you store the mark bits in their own array - the next several bits will be in-cache. Still, the overall performance is likely to vary quite a bit for either scheme depending on the project's other quirks. Mark-sweep has the advantage that it doesn't need to move objects, but copying is easier with variable-size allocations, etc.

Another disadvantage with lazy sweeping is I don't see a clear way to make it generational. (Right?)

<comment><USER19> Mark each arena with the last time something was collected from it and just skip sweeping old pages until X cycles have passed? Just thinking out loud... that check wouldn't save time for half-pages of old objects.

</comment></comment></comment></comment></comment></comment><comment><USER20> I recommend Elise Huard's recent talk on Ruby GC: <http://skillsmatter.com/podcast/home/ruby-bin-men>

(Also, I'd love to see the Hungarian Folk Dance interpretation of different GC approaches <http://t.co/l8ADbEQR>)

</comment><comment><USER21> nice introduction, but it seems to be missing some really important concepts

\- generational GC

\- concurrent GC, the parts of GC which can be concurrent (which depends on your algorithm), and the tradeoffs needed to let your program run concurrently with the GC without stopping the world.

</comment><comment><USER22> very nice article. given its likely target audience, though, he really needs to explain explicitly why mark-and-sweep can't compact as it goes, rather than leaving a fragmented heap.</comment>


The Road Less Traveled to Fusion Energy? - <AUTHOR> http://nautil.us/issue/86/energy/the-road-less-traveled-to-fusion-energy <comment><USER2> I was hoping the article would go into the current state of boron reactors out there such as [0], but it kept reiterating the disadvantages of D-T reactiors.

Also interesting is using AI for plasma control. It didn't exactly say how they're using AI, which is nice to know when talking about real-time code that handles temperatures in the millions of degrees.

[0] - [https://www.hb11.energy/our-story](https://www.hb11.energy/our-story)

<comment><USER3> Calling it "AI" is a stretch, it's more like using the TensorFlow infrastructure to do Bayesian inference. Here's a presentation on the process: [http://hyperion.usc.edu/UQ- SummerSchool/pres/Langmore.pdf](http://hyperion.usc.edu/UQ- SummerSchool/pres/Langmore.pdf)

<comment><USER2> So their only paragraph on it,

> _Artificial intelligence software, recently developed by TAE with help from > Google, should keep the fusion fuel stable even at the 3-billion-degree > temperatures needed in a commercial-scale power plant. "A year ago I was > concerned about the ability to control the machine," Binderbauer says, "but > no longer."Courtesy of TAE Technologies_

which implies that it's being used for plasma control, is a lie.

<comment><USER3> No, it's used for control. But you should be thinking more like "process control" than "intelligent" control. Or to be more specific, deriving an optimal process control model via pretty hard-core plasma density reconstructions; my understanding is that they track plasma density fluctuations and tweak parameters in real-time, but I would be surprised if the TF model is in the control path.

It seems a bit overly-aggressive to accuse them of lying because some pop-sci journalist heard tensorflow and went straight to "artificial intelligence." And even there, it's an understandable and common hype.

It doesn't take away from the point that they're doing some really interesting and novel computational physics to make this thing work.

</comment></comment></comment></comment><comment><USER4> The thing I never understand about these fusion reactors is how they plan on generating energy from them, assuming they exceed break-even. In a conventional nuclear reactor the coolant is piped to a steam generator which then drives steam turbines to generate power. This is a relatively obvious design and it was pretty well understood from the start of development how the power generation would work. Of course, there were a lot of engineering challenges to overcome to make it work in practice. While fusion itself is a harder problem (as the lack of success for decades has demonstrated), it seems like the power generation aspect has many new challenges as well.

<comment><USER5> The people working on fusion generally think it is a solved problem. Namely, fusion is a heat source and we have numerous technologies which can convert a heat source to generate electricity a number of which can be connected to the fusion generator with "some engineering". That's why they talk about the parts which aren't yet solved.

I used to work with under a guy who worked on ITER, they were all aware of that part of the problem. It just wasn't part of the research they were engaged in.

<comment><USER6> Generally, a layperson can safely assume that whatever problem they come up with on the spot, it has already been thought of by specialists as well.

True for any field you are not an expert in.

</comment></comment></comment><comment><USER7> I have yet to read an article or paper which has convinced me that energy generating fusion is possible at low costs and with materials that we currently know of. I don't see why anyone (including governments) should invest in these types of projects. Not only is the science extremely complex, the implementation is a huge engineering challenge. We already have a huge fusion power plant at the heart of our solar system and we can harness its power with some pretty cheap solar panels.

If you want to read a better article explaining the science then try [http://fusionandthings.eu/2019/06/05/new-calculations- show-p...](http://fusionandthings.eu/2019/06/05/new-calculations-show-proton- boron-fusion-is-still-difficult/)

<comment><USER8> Sadly, that huge fusion plant is offline half the time for any given location on the Earth, and is severely degraded by cloudy weather. Also using it to power the world will require large areas of land to be dedicated to collecting its output.

<comment><USER9> So what?

I give you _Space Grid ©_.

[1] [https://en.wikipedia.org/wiki/Space- based_solar_power](https://en.wikipedia.org/wiki/Space-based_solar_power)

edit: I don't know why that symbol doesn't show as copyleft as intended.

<comment><USER10> Space advocates seem to have been going off the idea of that over the past few years. I'm not sure how much of that is due to a better appreciation of the downsides, and how much is due to the _very_ rapid improvements we've been seeing in ground-based solar.

A couple of recent (late 2019) articles from Casey Handmer:

[https://caseyhandmer.wordpress.com/2019/08/20/space-based- so...](https://caseyhandmer.wordpress.com/2019/08/20/space-based-solar-power- is-not-a-thing/)

[https://caseyhandmer.wordpress.com/2019/09/20/no-really- spac...](https://caseyhandmer.wordpress.com/2019/09/20/no-really-space-based- solar-power-is-not-a-useful-idea-literature-review-edition/)

It's worth reading the comments too; there are some interesting points there, although I think the game-changing ones rely on new tech and/or well-developed lunar industry.

<comment><USER11> Space solar is a way to get the most out of expensive PV cells. But PV is no longer expensive. It's cheaper to just overbuild on Earth and transmit energy through time (storage) rather than through space (microwaves).

</comment></comment></comment></comment></comment><comment><USER12> For just the bare facts on the fusion reactor design:

[https://en.wikipedia.org/wiki/TAE_Technologies#Design](https://en.wikipedia.org/wiki/TAE_Technologies#Design)

<comment><USER11> These comments would also be relevant.

[https://www.researchgate.net/publication/235032059_Comments_...](https://www.researchgate.net/publication/235032059_Comments_on_the_Colliding_Beam_Fusion_Reactor_Proposed_by_Rostoker_Binderbauer_and_Monkhorst_for_Use_with_the_p-11B_Fusion_Reaction)

</comment></comment><comment><USER13> > Even if renewables come to generate the majority of our electricity, they > need to be combined with so-called dispatchable power that can be switched > on at any time.

There is plenty of lithium in the oceans — among other places — and no shortage of iron or phosphorus (ignoring agricultural use of P which is much larger), so that it is realistic to store weeks or months of energy in LiFePO4 batteries, even at a global scale. What we lack here is investment.

</comment><comment><USER14> "Relying solely on solar and wind is a fool's hope, I think," Binderbauer says.

We've had this discussion many times on HN.

<comment><USER15> I agree that solar and wind have disadvantages, and they can't be used to cover 100% of the energy demand, but they are available right now, whereas controlled fusion has been "10-30 years away" for the last 60 (?) years.

<comment><USER16> _whereas controlled fusion has been "10-30 years away" for the last 60 (?) years_

One standard internet response to that is this graph[1]. No funding, no results.

The international community decided to pool ressources with ITER. The founding members? The US, EU, Japan and the Soviet Union. The Soviet Union collapsed, the US pulled out for a couple of years, other countries (eg Canada) joined and left gain, Brexit has put the future of JET (where some relevant preparatory experimentation is done) in doubt, etc.

[1] [https://i.imgur.com/3vYLQmm.png](https://i.imgur.com/3vYLQmm.png)

</comment></comment></comment><comment><USER17> This is the worst of science writing. I have zero interest in the man's childhood or what he or his desk looks like; just tell me about the science.

Articles like this are pandering to the lowest intelligence that purports to be able to read.

<comment><USER18> How do you go from "this article has too much human interest and not enough details for my taste" to "this article is for stupid people"?

<comment><USER19> I replied above but it was meant for you. I'll leave it, perhaps it might help the parent understand why there is much value in knowing about the people behind the science.</comment></comment></comment>


A Graphical Web Browser for Linux Command Line - <AUTHOR> http://soosck.wordpress.com/2010/11/16/netsurf-graphical-web-browser-command-line-css/ <comment><USER2> "command line" is a bit too ambiguous a term to adequately describe where this browser can be used.

Technically, it's made to use Linux's framebuffer, which means it can be used from virtual consoles outside of X, but not (for example) from within an xterm or other terminal inside X.

<comment><AUTHOR> You CAN also run NetSurf inside a X terminal like xterm or xfce terminal but anyway, what's the point? Everybody knows this is for Framebuffer...

<comment><USER3> _You CAN also run NetSurf inside a X terminal like xterm or xfce_

Really, you can render the framebuffer into an X terminal session? That is not possible, is it? Or is there some other port besides the GTK port, that you're referring to?

 _Everybody knows this is for Framebuffer..._

Well, I for one didn't. I clicked through thinking it might be a console web browser like lynx, and wondering how it would be graphical. "Command line" is not a good descriptive term.

Looks like a neat lightweight browser, though. I've been stuck with lynx on a linux virtual console enough times to want this. :)

<comment><AUTHOR> You'll forget lynx, elink or w3m the moment you start using NetSurf. It's much more sophisticated. As to the former part of your comment, I should state that as far as I'm concerned there are only two NetSurf frontends for Unix-like systems (GTK and Framebuffer) and the article is just about Framebuffer version.

<comment><USER4> _You'll forget lynx, elink or w3m the moment you start using NetSurf._

So the situation is, I don't have X, but I do have a mouse attached? That happens a lot...

<comment><USER5> My netbook almost never runs X, but it still has a trackpad.

<comment><USER4> How's that working out for you?

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> I think uzbl is much more promising. It is built on WebKit and with little tweaks you can install it without X: <http://lug.rose- hulman.edu/wiki/DirectFB>

<comment><USER7> Another WebKit-based browser, from the suckless.org guys, is Surf: <http://surf.suckless.org/>

</comment></comment><comment><USER8> What about performance? Despite all the criticism of X server, Linux framebuffer (out of the box) usually tends to be at least ten times worse than anything X server can offer on the same machine.

<comment><USER9> Well it sometimes has a lower memory footprint on lowend machines.

</comment></comment><comment><USER10> I'm sorry I don't understand how this is a Command Line application at all!

<comment><USER11> it's not, it's just that it doesn't run under X like most graphical applications. it's more like running a video game under DOS.

</comment></comment><comment><USER12> Note that this isn't entirely new, Links has had framebuffer console rendering for just under a decade I think <http://links.twibright.com/>

</comment><comment><USER13> I am thinking cheap arm processor+lcd screen+embedded linux+some keys = (cheap?) DIY tablet.

This could be a project which can make me learn embedded linux after all.

<comment><USER3> For cheapie gizmos, you might want to look into the ARM9 VT8500 & WM8505 based netbooks and tablets. These are <$100, there is an Android port and a Debian port, vendor kernel source is available and there is also OSS kernel support being developed from scratch.

Lots of fun to play with, on a budget. :)

<comment><USER13> To hell with the budget for now. I would love to learn all this, money can be arranged if I present a proper research proposal (with BOM) to proper authorities (Read my parents).

I would have to start from scratch though, all of my experience has been in AVR and avr-gcc. Lets see how all of this works out.</comment></comment></comment>


Amazon Is Aggressively Pursuing Big Oil as It Stalls Out on Clean Energy - <AUTHOR> https://gizmodo.com/amazon-is-aggressively-pursuing-big-oil-as-it-stalls-ou-1833875828 <comment><USER2> From today "Amazon Announces Three New Renewable Energy Projects to Support AWS Global Infrastructure"

[https://press.aboutamazon.com/news-releases/news-release- det...](https://press.aboutamazon.com/news-releases/news-release- details/amazon-announces-three-new-renewable-energy-projects-support-aws)

</comment><comment><USER3> AWS is by far the market leader in cloud ... this is a good reason to switch to its main competitors: Azure & GCP. They only got to be No! by network effects, not by being better.

<comment><USER4> And there are a bunch of smaller options that are totally reasonable, depending on your use case. I feel far too many people select AWS as a knee- jerk reaction, when something like Digital Ocean or Zeit. There are plenty of smaller players that are completely reasonable, and you'll probably get better support since they need your business.</comment></comment>


The Library of Congress wants to attract more visitors - <AUTHOR> https://www.washingtonpost.com/entertainment/museums/the-library-of-congress-wants-to-attract-more-visitors-will-that-compromise-its-scholarly-mission/2019/03/07/582d590e-1a90-11e9-8813-cb9dec761e73_story.html <comment><USER2> [https://outline.com/A5ArSK](https://outline.com/A5ArSK)

</comment><comment><USER3> If you've never been there, it is an _amazing_ building: the architecture wouldn't be out of place in a palace or something.

They also have a lot of great digital stuff if you're looking for something to browse.</comment>


'My 7-Step Plan to Destroy America' - <AUTHOR> http://www.americanthinker.com/blog/2014/04/my_7step_plan_to_destroy_america.html <comment><USER2> _> Then I would place all these subjects off-limits... taboo to talk about._

And yet here you are, talking about them! And there are no secret police dragging you away, or fines being imposed, or really any punishment at all.

It's a miracle!</comment>