Official Node.js support - <AUTHOR> https://stripe.com/blog/official-nodejs-support <comment><USER2> I'm happy to hear this. It was nice of Bjørn to build the node API in the first place. It's great that this became the official API, congrats all around.

<comment><USER3> Just curious, would the original author be compensated for this? How does that work?

<comment><USER4> I wrote the package because I needed it. My compensation came when my package worked and I could use it. Since then it was just a "cost" because I chose to maintain it for the community. Like most of us I write software for money, too, but in this case the stripe package was just "collateral benefit" from something else I was doing.

I didn't have time to do the cleanups and rewrite that the package deserved to get support for the new Stripe features with a consistent API.

I haven't been needing the new features myself, so honestly I was delighted when they asked months ago if they could take it over. For them and for the community it was better this way than if they'd just forked it or just published another package.

<comment><USER2> Thanks for your open source contribution, it's been a big help.

</comment></comment></comment></comment><comment><USER5> I'm glad to see more JavaScript libraries embracing promises!</comment>


Statement on LambdaConf 2016 - <AUTHOR> https://statement-on-lambdaconf.github.io/ <comment><USER2> There's one sentence that stands out:

      LambdaConf cannot live up to its goal of being a "friendly community of like-minded souls" when it does not protect current and potential members of that community who are vulnerable to those who would deny  their humanity.

I would love to know how those members of that community are supposedly vulnerable. In the end, it is one voice against many. That voice will, in a big group, be drowned out. And the speaker this statement is aimed against is at the conference for his technological/theoretical knowledge - not to spread his non-technical opinions. Therefore, any likely violation of the community members' personality(?!?) is simply an assumption.

Furthermore, the speaker was previously banned from another conference. I have yet to find accounts that such violations have actually taken place at a conference where he gave a talk. Shouldn't the basic assumption in a system based on the rule of law, that you're innocent until proven guilty be the case here as well?

<comment><USER3> They're 'vulnerable' because they choose to believe that all opposite makes them unsafe.

Call me when they don't let "Death to capitalists" radical communists speak, and then we'll talk about excluding people who are pro-slavery.

</comment></comment><comment><USER3> They're trying so hard to be the victims here.

I guess 'Debate ideas not people' isn't as important as making sure a man who holds an unpopular opinion isn't welcome at a conference.

<comment><USER4> I appreciate the sentiment you are trying to espouse, but the world is a very messy place that doesn't reduce neatly to a single logical dictum. Most especially, it contains people who will look at a "rule" like 'Debate ideas not people' and ask themselves how they can exploit or hack it to "win," regardless of whether their exploit or hack conforms to the spirit of the rule.

For example, we did debate ideas around race and opportunity and justice. We debated ideas around the freedom of women to choose what would happen with their bodies. We debated the idea of one citizen, one vote. We debated the idea that people should be free to love a person of whatever gender they pleased.

But such debates never end, because the people who don't like how the debate turns out just keep debating. They are conducing a century-long filibuster, designed to disrupt any attempt by the people they don't like to have ordinary, peaceful lives.

You can see this whenever they re-open old debates. They start off with the same arguments that were conclusively refuted and try to force you to refute them all over again. And again. And again. And again. In this regard, racists behave just like anti-vaxxers, young earthers, and others who have zero interest in "debate," because they believe in some divine truth that transcends evidence, science, or the rights of other people.

They are waging a war of attrition, trying to wear society down by sheer force of stubbornness. And naturally, moderate thinkers avoid having the same toxic discussion over again for the tenth or twentieth time, so gradually bit by bit they receive fewer challenges to their ideas, creating the impression that there multiple, equally valid sides to every issue.

But there aren't multiple, equally valid sides to every issue. We have limits on which ideas are valid in society. We have, for example, constitutional democracies, with judiciaries attempting to operate as a check and balance against certain "popular" ideas being considered valid.

For example, we have this idea that every adult citizen gets to vote. It's full-on wrong to pass a law that says that black people can't vote. We don't have to 'debate' this idea, we did debate it, there was blood shed over it, and the matter has been settled.

The idea that black people should not vote is not "just another point of view," or, "something upon which reasonable people may disagree." It's wrong, and to believe it or to espouse it is to be unreasonable _by definition_ in a constitutional democracy.

LambdaConf can obviously invite whomever it wants, for whatever reason it wants, and you are free to think whatever you want of the choices they make. All I am saying is that for those who disagree with LambdaConf's decisions, it is not a matter of "a man who holds an unpopular opinion." Nor is it a matter of not wanting to "debate an idea."

Certain opinions are so wrong as to not be trivializable as unpopular. Certain ideas have already been debated, and the people still pretending to debate them have not demonstrated any good faith interest in debate.

Whether the person involved in the fracas is such a person, and whether his ideas are or aren't in these categories is, of course, something your could debate. I am not going to, as I was not planning on speaking at LambdaConf, so I had no decision to make on the matter.

<comment><USER3> While I believe that I see where you are coming from, I cannot reconcile the concept that some ideas are not up for debate with liberalism.

Also this is not a talk about his taboo views on race, they are hoping to prevent him from speaking on functional programming. In a way, they are hoping to exclude him from the marketplace of all ideas because of a single idea.

<comment><USER4> I can't speak for other people, and I'm sure there are a wide variety of opinions people have about that situation. But I very much doubt this is about speaking-as-in-speaking about functional programming. After all, he can write about functional programming as much as he likes, and I haven't heard anybody suggest that his blog be blocked.

But speaking as in giving a conference talk is a different matter.

Conference talks aren't really about disseminating technical material. They're terrible for that! Conference talks are showmanship, entertainment, motivational speaking, and most crucially, they are cultural signals about what--and whom--is important.

That's why there is a vastly different dynamic of attending a talk in person vs. watching one remotely or on video. You are absorbing the social cues everyone else in the room is transmitting.

So when a conference announces that so-and-so is keynoting, they are announcing that the topic of the keynote is important, and they are also announcing that so-and-so is important, and traditionally, that they are the _most important_.

One rung down is speaking. Announcing that so-and-so is speaking is also announcing that so-and-so's topic is important, and that so-and-so is important in their own right.

My guess is that what this debate is really about is signaling that this individual is important in the community. Which is not, and never will be, completely divorced from all of this individuals choices.

From my reading of LambdaConf's statements, they fully realize this. They are making a political statement about what and who is important, and how the community should judge someone's importance. Likewise, the people who disagree are making a political statement about who should be judged important in the community.

tl;dr: This is not about whether somebody can speak as in communicate, it is about whether they should be judged important. And those people who disagree with LambdaConf's choice in this matter are refusing to implicitly support LambdaConf's chocie with their attendance, not to mention communicating their own opinions, as this individual has done.

Given that the individual, LambdaConf, these signatories, and even you and I have been able to write about this at length without censorship, I'd say that nobody's right to "speak" has been curtailed in any way.

<comment><USER3> Thats way more reasonable than how I had been think about it. I think about conference talks the way I think about guest lecturers at universities.

I might be really bad at noticing showmanship.</comment></comment></comment></comment></comment>


Giant 'soap bubble' found floating in space - <AUTHOR> http://www.newscientist.com/article/mg20327185.100

<comment><USER2> "...spotted the "Cygnus Bubble" while recording images of the region on 6 July 2008."

Why are they running the story today?</comment>


Ask HN: How are HN's comments rendered at scale (server-side)? - <AUTHOR> HN, as we all know, is a super popular website. I heard, randomly, that it's hosted on rather low-end hardware but somehow it is capable of handling all the load we rabid HN checkers give it. So without something like disqus how are the comments powered? <comment><USER2> Everything is served out of RAM and then when RAM fills up, the oldest threads are written out as static files. (Note that when you visit a super-old thread [1], there's no "reply" link and voting redirects you to a voteclosed page.)

Computers are stupidly fast at serving out of RAM - it's not unusual to be able to serve 100K requests/ _sec_ if a request never needs to touch disk (see eg. the JSON serialization test on the TechEmpower benchmarks, on physical hardware [2] - top frameworks can get 500K request/sec on a hello-world serialization task). They're also stupidly fast at serving static files (which, if the server is implemented right, can be served straight out of the filesystem cache in the kernel without any memory copies through user space). 100K requests/sec is a budget of 20,000 cycles/request, which is eminently doable if most of the page contents is in RAM (or better yet, cache...on a slow day, the entire front page of HN can fit in L3 cache).

The same approach has been used to good effect on other websites like Google, Facebook, PlentyOfFish, Mailinator, etc. Eschew databases, serve out of RAM (and now SSDs) entirely.

[1] [https://news.ycombinator.com/item?id=8863](https://news.ycombinator.com/item?id=8863)

[2] [https://www.techempower.com/benchmarks/#section=data-r14&hw=...](https://www.techempower.com/benchmarks/#section=data-r14&hw=ph&test=json)

<comment><AUTHOR> Thank you so much for the info. Makes sense and helped to scratch the itch I was having for a while now.</comment></comment>


A dive into spatial search algorithms - <AUTHOR> https://medium.com/@agafonkin/a-dive-into-spatial-search-algorithms-ebd0c5e39d2a <comment><USER2> This discussion is incomplete without mentioning space-filling curves. These curves map the data into a 1-d space and then you can use _any_ conventional search structure -- sorted array with binary search, balanced tree, skiplist, b-tree, etc. This allows you great flexibility, including the use of 3rd party libraries of data structures.

(A quadtree is basically the z-order space-filling curve combined with a trie of degree 4. Octtree -- degree 8.)

This approach also handles non-point data very cleanly. A non-point object gives rise to multiple index entries. (You can tune the number of index entries quite easily.)

Space-filling curves also leads to a very nice kNN algorithm: To find the k nearest neighbors of p, first search the index for p. Find k neighbors along the space-filling curve, (these are adjacent in the index), and of these, pick the point q maximizing distance(p, q). All k nearest neighbors of p must be in a circle centered at p with radius distance(p, q). Now search your index again using this circle and pick the k points n1, ..., nk minimizing distance(p, ni).

<comment><USER3> Yeah, the effectiveness of that idea mad me a little sad when I worked on google maps. I spent all this time studying computational geometry in college and it went out the window. Use space filling curves and then standard text indexing techniques and you don't need to deal with any fancy geometric algorithms.

<comment><USER4> Space filling curves, although elegant, are not a solution for every problem in computational geometry of course.

</comment></comment></comment><comment><USER5> Along similar lines Real-Time Collision Detection[1] is a tour-de-force in spatial partitioning algorithms. It's basically data-structures for 2D/3D along with some really good coverage of common floating point numerical issues and cache friendly algorithms.

Easily the in the top 3 technical books I've purchased. Each page is solid gold and very little space is wasted.

[1] - [https://www.amazon.com/Real-Time-Collision-Detection- Interac...](https://www.amazon.com/Real-Time-Collision-Detection-Interactive- Technology/dp/1558607323)

</comment><comment><USER6> Coming from a games background, I've not heard of an R tree before. Quadtrees (or octtrees for 3D) are probably the most common spatial query data structure used in game engines. I'm surprised they weren't mentioned in the article. Gonna have to do some research on what the pros and cons of an R tree are vs a quadtree now.

<comment><USER7> R-trees are relatively slow, particularly if you have a very high mutation rate. Their sole redeeming feature is that they can handle non-point geometries with constant space complexity whereas quad-trees have pathological space complexity for geometries like polygons.

If you are just dealing with points, an R-tree is a poor algorithm choice. Quad-trees are pretty close to ideal in terms of performance for that use case.

<comment><USER8> > quad-trees have pathological space complexity for geometries like polygons

Any idea what is used in the wild for closest point queries against non-point data (triangle meshes or otherwise)?

<comment><USER2> Quadtrees are fine. Just don't approximate the polygon to the limit of resolution of the space. I wrote a paper many years ago on a related problem. The conclusion was that a very small number of quads would be optimal. Too many is obviously a problem; too few and you have to filter out a lot of false negatives.

The paper was in ACM SIGMOD 89, Redundancy in Spatial Databases.

</comment></comment></comment></comment><comment><USER9> You should check out the Churchill Challenge ([http://churchillnavigation.com/challenge/](http://churchillnavigation.com/challenge/)) and the 54 solutions (source code available at [https://github.com/churchillnavigation/challenge1](https://github.com/churchillnavigation/challenge1)). I came in at #7.

Beyond the classic spatial search data structures, the fastest solutions all used SSE instructions (see [https://github.com/sDessens/churchill- challange](https://github.com/sDessens/churchill-challange) for nice writeup). People did try pretty hard to win the grand prize of $5000, and the fastest solution was 7 times faster than the best optimized code that Churchill's team had previously.

</comment><comment><USER10> In my experience most of the work in this area has been done in that you will rarely find yourself having to think seriously about spatial indexing, and even when the task forces you to, most of the tools one would commonly choose between have different indexing options implemented as standard.

</comment><comment><USER11> Nice algorithm. I am curious what happens when the dimension becomes really large (more than 100). I have heard about kd tree but as far as I know the performance becomes really bad for d > 16\. What should be the strategy of finding nearest neighbours for such high dimensions?

<comment><USER12> It often depends on what the actual dimensionality of the problem is. If it can be embedded in a lower dimensional space, as many problems are, then a well tuned kD-Tree can still do a good job up to about 16 dimensions of used space, as you say. Cover trees also work well in this case, and are better at extracting just the salient dimensions than kD-Trees are, especially as the discrepancy between the subspace and data space grows.

If the dimensionality gets larger then a better approach is to instead remove overhead and improve parallelism, for example by pushing the problem to the GPU or discretizing the measurement space down to 8-bits or even 1-bit per dimension. I know of products that do both, and solve O(N^2) problems 20x larger than a naïve approach could in the same time. POPCNT runs really fast on GPUs.</comment></comment>


SATCON1 Report on Impact of Satellite Constellations on Astronomy - <AUTHOR> https://noirlab.edu/public/products/techdocs/techdoc003/ <comment><USER3> We've been covering this issue frequently in Orbital Index ([https://orbitalindex.com](https://orbitalindex.com)). With around 10,000 stars visible with the naked eye, there will be significantly more satellites than stars, even from a single constellation like Starlink or Project Kuiper. Granted, most of those satellites won't be visible at any given time, but I do worry about the ability of LSST and other wide-field observatories to do their (very expensive & important) jobs—one of those jobs is spotting near Earth asteroids with a collision potential.

Here's a recording of the recent 'Impacts of Satellite Constellations on Optical Astronomy' webinar that discussed this and had someone from SpaceX: [https://www.youtube.com/watch?v=LaR6v0p6pB4](https://www.youtube.com/watch?v=LaR6v0p6pB4)

<comment><USER4> As a side note, let me say that Orbital Index is a fantastic service. It's the only newsletter that I read every week without fail.

<comment><USER3> I'm so glad you're enjoying it!

</comment></comment></comment><comment><USER5> What are going to be the cultural effects of having a constant stream of satellites wizzing around the night sky?

I've sat here for a few minutes thinking selfishly about never being able to experience Alaska wilderness on a clear night. Or late night stroll on a beach looking up, etc, etc forever about all kinds of personal moments that will never be the same.

However there's billions of people who have no idea what Starlink or Kuiper are that are going to find out the hard way when they no longer have that same night sky either.

I hadn't really thought about it much before because it was always framed as an astronomy problem, but it's really a whole Earth's peoples problem.

<comment><USER5> Follow on comment, what are indigenous tribes who aren't part of our connected world going to interpret this change as? I'm thinking about groups of people like the Sentinelese.

<comment><USER6> People appear to be downvoting my other comment so much that I can't even edit it so reposting again. Indigenous tribes won't have an issue with it as they don't have superhuman eyes to see 7th magnitude objects in the night sky. The satellites are not generally visible to the naked eye after changes SpaceX has made to the operations of the satellites and physical modifications to the satellites themselves.

[https://www.spacex.com/updates/starlink- update-04-28-2020/](https://www.spacex.com/updates/starlink- update-04-28-2020/)

</comment></comment></comment><comment><USER7> Dense satellite groups are a real game-changer in how we observe the sky from the surface of Earth.

We're at the choosing point right now, what's better for us - a good uniform global communications or the pristine sky. The sky was always the norm; but it ceases to be so.

In order to have some compensation to astronomical society - that is, the whole population of Earth, as even looking to bight sky is getting be different - it would be good for SpaceX to create a program of high-orbit astronomical observatories. Working together with Earth astronomers, the company can participate in changes to observation possibilities in a way which has some upside as well, in addition to anticipated degradation. Hubble space telescope was launched some 30 years ago; an observatory of similar class or better can be significantly more affordable today, there can be several - <USER2> many - of them and that could be a worthy trade-off to the degradation of Earth-visible sky. I'm sure astronomical society would be very interested in participating and helping with building such a group of space observatories.

<comment><USER7> Two more ideas here. First, it would help if each satellite can have an astronomical payload; this is likely hard to achieve due to quite conflicting requirements between these two different goals.

Here is the second one. Earth observatories demonstrate streaks of satellites across astronomical images taken over comparatively long time. We know that at each moment satellite is in just one place in the sky; more, those constellations tend to have rather predictable orbits. Image sensing devices can use knowledge of satellite orbits to suppress reading a particular pixel at particular time if a satellite is expected to shine towards that pixel.

It's quite delicate task, I understand, but I also know that practical astronomy is known for inventing solutions to very diverse problems. This approach won't completely restore the abilities to observe the sky, but can significantly diminish the impact, discussed in the article. Especially if Earth-based devices would be constructed with such "shielding" (remember star shielding invention when working on exoplanetary discoveries?) in mind.

<comment><AUTHOR> Satellites are not new, so many techniques have been used and tested. [https://aas.org/sites/default/files/2020-07/MitigationWG_Pre...](https://aas.org/sites/default/files/2020-07/MitigationWG_Present5_bySaunders.pdf) is a relatively nontechnical survey of some of the techniques that Rubin plans to employ.

Suppressing pixel readings is not plausible with the current CCD designs, as far as I'm aware. These devices are so sensitive that any wiring needs to be done _extremely_ carefully to minimize crosstalk and keep temperatures extremely low (think under -100C).

Masking in data processing is the more likely technique.

<comment><USER7> > Suppressing pixel readings is not plausible with the current CCD designs

Better than suppressing pixel reading would be suppressing pixel changes. The first process lets pixel to change as a result of incoming photons, the second one maintains the "unread" state.

Yes, current CCD devices can't do that. Advantage of this change is also in reducing crosstalk - if we need, say, to remove accumulated charge and for that we need to draw the current, the current may influence our reading of neighboring pixels. I'm tempted to look towards "smart mirrors" \- MEMS systems, even for mirrors of 1 sq. micron - but purely electrical solutions might be possible.

</comment></comment></comment></comment><comment><USER8> I'm generally a technologist however I am completely appalled at a private company getting the right to significantly impact humanity's view of the night sky, the original source of wonder.

<comment><USER7> Yes, this is a kind of Wild West situation. We should figure out the best way out, ideally with all interested parties getting something good.

<comment><USER9> except you as an individual could opt out of the original Wild West; this one, not so much.

perhaps it's exactly like the Wild West from the viewpoint of Native Americans: we're taking your skys.

<comment><USER7> Yes, that's very unfortunate.

I hesitate, however, concluding that we should ban satellite constellations on these grounds. Do you think that would be a good approach?

<comment><USER8> Yes, unless they are invisible in the sky. Put a barrier in place to force innovation.

<comment><USER7> Anything in the sky is something which at least obstructs the view. If we have too many even black body satellites - which have zero reflection or emission in interesting spectral parts - we're still can't see through them. If we have too many of them, we still have a problem. So - theoretically - it's a choice between space flights and astronomy.

Maybe there can be a more optimal solution than what we have today.

</comment></comment></comment></comment></comment></comment><comment><USER10> The document has an executive summary and clear images denoting the impact of these satellites on humanity's view of the stars.

It does a great job of highlighting the collision between high-sensitivity wide-field imaging of both deep fields and transients with the nascent satellite constellations.

Direct link to the PDF: [https://noirlab.edu/public/media/archives/techdocs/pdf/techd...](https://noirlab.edu/public/media/archives/techdocs/pdf/techdoc003.pdf)

Astronomers have been working for decades toward the Vera Rudin Observatory (nee LSST). It has been the big thing on the horizon for my entire career as a professional scientist. It sees first light in 2021. Images like that seen on page 7 of the PDF will be what they see.

[https://en.wikipedia.org/wiki/Vera_C._Rubin_Observatory](https://en.wikipedia.org/wiki/Vera_C._Rubin_Observatory)

<comment><USER11> LSST/Vera is probably the project most impacted by these satellites. Per my contacts, smaller scopes like LBT/Keck/MMT aren't as drastically affected yet. But if we start approaching 100k+ satellites, even they will be having some major issues to deal with.

<comment><USER10> The real tragedy, from a scientific standpoint in the optical, is that we're entering a new world of wide-field and sensitive time-domain astronomy due to huge improvements in computing power and sensors at the same time that humanity is about to generate thousands of bright transients.

The only practical approach I can think of for combating the constellations is distributed arrays of wide-field transient-hunters. As long as the transient is common-mode across telescopes, then it is real. If there are three LSSTs, and not too many constellation-satellites, that might work.

I just noted, in passing, that the term for the satellite arrays appears to be "constellation"; that is a bummer in this context.

</comment></comment></comment><comment><USER12> This NIMOP ("Not In My Orbital Plane") movement is very concerning and I feel the need to speak out against the dangers of this "luddite movement" type thinking.

First, of course its good for companies to reduce the negative externalities of things in space (paint, shades, etc). I think there is near universal agreement here, including from SpaceX.

However, there is no way to become a space faring civilization without putting lots of things in space, period. I want to see 1,000,000x more objects in space, including much larger ones.

If we truly become space faring, we have the matter and resources to have a quadrillion humans living in comfort (likely in large rotating space habitats like"O Neil Cylinders"). This is the kind of dream we could have, imagine over 100,000x the humans, artists, friendship, science, and love. We can simultaneously make earth an ecological paradise, with most of the planet reserved for nature.

This "anti-satellite" movement seems to generally have 4 kinds of attacks against the satellites: 1) The satellites "impede my rights to view the sky". The implication is that anyone who attempts to build in their "view to infinity" violates their rights. This seems like an extremely presumptuous statement, like they own trillions of miles. This also doesn't match the current legal framework of any country (nobody needs your permission to fly over your house in aircraft or space). 2) The usecase of these satellites is bad because they are "like cell towers". Getting low cost internet access for the planet is IMO an extremely good thing, one of the best things for the poor. 3) The satellites will kill astronomy. This is absurd, we will have much better astronomy if we are a space fairing species with cheap space access and many things/expertise in space and giant telescopes outside our atmosphere. It may kill amateur astronomy from earth eventually, but I just don't think that as society a relatively fringe hobby should stop the progress of humanity. Many hobbies have been killed over time. I personally think this will inspire more people to look to space, and have the opposite effect. 4) We should not put things into space, and we should look to reduce our impact on the universe. This also manifests itself in those that want to see population reduction on Earth. I believe consciousness is better than lifeless rock. I say this as a human and a parent who has brought consciousness into the world.

Lets upgrade our dreams, take to the stars, and make the universe a more interesting place.

<comment><USER13> Right. The problem is not that it is 'in my orbital plane', the problem is that it is in _everybody 's_ orbital plane whether or not they - or their government - has a say in the matter. You can't escape it. There is no backyard where this isn't an issue.

This is now presented as a fact we will have to live with and if you are against 10's of thousands of orbital satellites you are now - according to you - immediately against progress. This obviously isn't the case.

So, let's stop the clock for a bit until we have a much better idea of what the impact of all this littering will be and whether or not there is something that can be done about it before we end up with a situation that is out of control. SpaceX is _not_ the world, LEO is not US property and whether or not low cost satellite access is such a good thing that we should give up easy access to visible light astronomy is not a given.

This has nothing to do with 'taking to the stars', 'making the universe a more interesting place' or 'upgrading our dreams'. It is a simple commercial proposition.

<comment><USER14> > This has nothing to do with 'taking to the stars'

Actually it does, because Starlink is intended to fund Starship development and Mars colonization.

<comment><USER13> Sorry, but Mars colonization is nonsense. It isn't going to happen in the foreseeable future and the people that believe this stuff are no less stupid than the ones following the next messiah. I wrote about it here:

[https://news.ycombinator.com/item?id=24058911](https://news.ycombinator.com/item?id=24058911)

Just to put this into perspective a bit: every little bit of civilization in _Iceland_ , every nail, every staple, every fruit, every tool has to be flown in or brought in by boat. Everything. And Iceland is a place that compared to Mars is paradise. The only things they have plenty of are energy, fish and sheep. And on Mars you wouldn't even have that.

It's just too silly to take serious.

Musk and Hughes have a lot in common. They are both genius, wealthy, capable and at times a bit batty. I'm fine with Elon being a bit batty, if that give us electric cars, reusable rockets and more. But I'm not buying into the cult. Starlink is borderline for me. I can see the utility. But the idea that space can be commercialized with such total disregard for what effectively is our largest commons is simply wrong to me.

</comment></comment></comment></comment><comment><USER15> The technical report is quite large (105MB). Here is a compressed version (6MB): [https://www.dvaun.com/files/satcon1-compressed.pdf](https://www.dvaun.com/files/satcon1-compressed.pdf)

I'm a bit confused by why they included such high-res images in the PDF in the first place.

Edit: After taking a look at the original I see that they embedded some of the files they created with Photoshop and After Effects directly into the document. They forgot to resize the images, hence the huge object streams in the file.

<comment><USER10> Agreed that they could have downsampled it for expedience, though the point is that systems spanning the gamut from the human eyeball to cryogenically-cooled 3.2 gigapixel imaging sensors are impacted by these constellations.

Those sensors are 3.2 gigapixels at 16-bit depth, so each image alone is 6.4 gigabytes. 108 MB seems small by comparison; that is only one telescope out of all the sky-imaging (optical and radio) systems on the planet.

<comment><USER15> I appreciate the sense of scale that your explanation brings. I understand that.

The size of the PDF is not a symbol for the work being done here, though. Reducing it makes it cheaper to distribute and easier for viewers to read their findings. That is my reasoning for putting up another version.

</comment></comment></comment><comment><USER16> Whether you agree or disagree with huge sat constrellations, I think that a matter that clearly impacts the entire world should not be decided by a handful of people from a single country.

<comment><USER17> Nor a single billionaire.

If China or Russia was the one building starlink the reception in the US would be a lot different.

</comment></comment><comment><USER13> It's littering on the largest scale imaginable.

<comment><USER6> So solar panels and cell phone towers are littering too?

<comment><USER18> Unless you live in a country with 20000ft tall people where do you see solar panels? Cell towers can also be hidden fairly easily when needed.

</comment></comment></comment><comment><USER19> People, let's get real for a minute.

Why are we allowing a private company to ruin the night sky for everybody, and by everybody, I mean literally everybody on this planet?

This should be discussed in a democratic and international way. It's nor going to be easy, but it is necessary.

I can't believe future generations won't see a pristine night sky anymore...

<comment><USER6> > Why are we allowing a private company to ruin the night sky for everybody, > and by everybody, I mean literally everybody on this planet?

They're not being ruined for everybody. There's so many misinformed comments in this thread. These satellites aren't visible to the human eye generally.

<comment><USER19> "generally" is the key: not long ago in northern Italy everybody was freaking out because of this:

[https://cdn.dmove.it/images/18547/StarMain.jpg](https://cdn.dmove.it/images/18547/StarMain.jpg)

Those are the ones by Starlinks. That's not acceptable.

They are so invisible to the naked eye, that the weather website gives you the exact time and date of their passage in the sky: [http://www.meteoweb.eu/2020/05/passaggio-satelliti- starlink-...](http://www.meteoweb.eu/2020/05/passaggio-satelliti- starlink-14/1433303/) (in Italian)

<comment><USER6> 1\. That is only in the roughly week following launch, which doesn't harm anyone.

2\. They changed operations immediately after launch and so you generally can't see this anymore.

They are indeed invisible after launch. Your article is from March, before changes were made. Try to find anyone taking pictures of them in that chain of pearls form in the last few months.

</comment></comment></comment></comment><comment><USER20> Cheap rockets will make space telescopes way more accessible. Ground survey telescopes have serious limitations. So for science this could be a positive thing.

I am more worried about monopolies and accessibility of this new internet. If it ruins sky for everyone, but only a few can afford it, that would be a terrible deal.

<comment><USER21> >Cheap rockets will make space telescopes way more accessible.

I'm not convinced that's true. If you look at past space telescopes, launch costs are not a large part of the budget. Even if SpaceX gave everyone _free_ launches as a goodwill measure, you'd only make space telescopes maybe 2-5% cheaper. There's very real challenges that space based telescopes have that Earth based ones don't (how do you make a cryocooler that doesn't vibrate your optics? How do you make a machine that can run for years without maintenance? How do you get precise enough positioning to interferometry?). My understanding is that due to adaptive optics, being ground based isn't much of a problem for visual telescopes. And for large radio telescope arrays, I don't see how it's possible at all to move to space

<comment><USER6> > If you look at past space telescopes, launch costs are not a large part of > the budget.

Satellites are built to narrow margin to reduce weight because of the weight restrictions on satellites. Right now we have reusable rockets but the rockets aren't any bigger so while the cost of launch has come down the volume/mass that can be launched in one go has not gone up. If laser interferometry gets better then large telescopes can be launched as constellations. Larger reusable rockets are also being worked on. When that happens they can care less about mass margin and make cheaper lower margin telescopes.

Telescopes are also generally built as a one-off which also increases cost. I hope the technology for telescope constellations will improve.

<comment><USER21> Even if there's room for additional weight, that doesn't help with the other aspects of SWAP.

Size: There's still a finite fairing size. Even the biggest upcoming rockets can't fit a 30m mirror in one piece.

Weight: say you want to put something like ALMA in space, which weighs over ten million pounds. Even with platforms like starship,you're not going to be able to do that in one go. _Maybe_ one launch for each of the 66 antennas,but even that is iffy.

Power: Hubble's power consumption is around 2Kw. Using ALMA as an example again, each individual antenna uses about 10x more electricity than that on average. More electricity means you need RTGs or nuclear reactors (not politically popular, complicates thermal regulation, potential for spreading radioactive waste across orbit and earth a la RORSAT), or massive arrays of solar panels. Massive solar panels means even more weight and space required

For interferometry, knowing your location to the nanometer is just the first of many challenges. You have to station keep across the constellation (tricky to do continuously while you're in orbit!), it becomes even more important to minimize vibrations, etc

<comment><USER6> Is ALMA even affected by satellites? I would imagine parallax would make satellites disappear and satellites would just slightly increase the noise floor. So I'm not sure why you're using that as an example.

> For interferometry, knowing your location to the nanometer is just the first > of many challenges. You have to station keep across the constellation > (tricky to do continuously while you're in orbit!), it becomes even more > important to minimize vibrations, etc

I didn't say it wasn't an engineering problem, but the major reason it hasn't been used yet is there was no real benefit to doing that over launching a single large telescope in terms of launch cost. I hope to see these problems solved soon in the future.

<comment><USER21> At least from what other radio telescopes, the issue isn't actually the physical satellites. It's the spectrum they're using

>no real benefit to doing that over launching a single large telescope in terms of launch cost

It would be hugely beneficial to do it in space. When you do interferometry on the ground, your maximum aperture size is the diameter of the Earth (if you had two observatories at opposite points on the Earth). When you do it in space, the size of your synthetic aperture is practically limitless.

</comment></comment></comment></comment></comment></comment><comment><USER22> I thought this was going to be about collisions between satellites from two constellations.

As someone loudly lamented a long time ago: please stop using the word "impact" as a substitute for figuring out effect/affect.

<comment><USER23> _As someone loudly lamented a long time ago: please stop using the word "impact" as a substitute for figuring out effect /affect._

"A long time ago" might be the 1700's, because it's been in use that way for several hundred years. So don't hold your breath.

<comment><USER22> I'm this case it was within this century, which is quickly becoming a time before some participants here had learned to read. The older you get, the more the line between "long enough" and "a long time ago" begins to blur.</comment></comment></comment>


Ask HN: What to Buy with $1,500? - <AUTHOR> My company has a $1,500 education stipend. What would you spend it on to kickstart learning to program. <comment><USER2> The question itself isn't of much value without knowing your current skills, the domain you work in, nor the domain(s) you desire to work in.

</comment><comment><USER3> Nothing. I'd put it in the bank because we're about to tumble into a global recession-if-not-depression and take some courses online.</comment>


When Whales and Humans Talk - <AUTHOR> https://www.hakaimagazine.com/features/when-whales-and-humans-talk/ <comment><USER2> > Even the Yupik belief that beluga whales were once land-dwelling creatures > is rooted in reality: some 50 million years ago, the ancestor of modern-day > whales walked on land. As if recalling this, whale fetuses briefly develop > legs before losing them again.

Whales are mammals, who was doubting they had a land-walking ancestor?

In general this article is taking stuff that's obvious, scientifically proven, or at least possible, and mixes it with animals spirits and other obvious bullshit, and concludes "so maybe there's something there?".

It's weird.

</comment><comment><USER3> Its called channeling.

Please pretend I didn't say anything.</comment>


Interview About Web 2.0 - <AUTHOR> http://www.paulgraham.com/web20interview.html

<comment><USER2> Very insightful interview, Paul Graham has really thought out what went wrong during the last bubble and has drawn some good lessons from that period. I agree 100% with the "web 3.0" part, classifying the movement of the web as a version number is really playing down the advancements of the current web environment. Then again, as the web matures people get excited about the new offerings and subsequently buzzwords follow. Its good to see people that see past the hype and are making viable things that people actually want.

</comment><comment><USER3> I also agree that "web 3.0" has already lost it's credibility and is reserved for random bloggers' stupid predictions about the future of the web.

<comment><USER4> Nova Spivack is a serious counter-example to your claim:

[http://novaspivack.typepad.com/nova_spivacks_weblog/web_30/i...](http://novaspivack.typepad.com/nova_spivacks_weblog/web_30/index.html)

Themes like 'semantic web' or 'recommendation engines' are still very pertinent.

<comment><USER3> This is a pretty pointless debate, but Nova's blog actually supports PG's outlook that the web is gradually evolving. Nova acknowledges that Google and Powerset are pretty semantic already. Recommendation 'engines' have also been evolving slowly for a long time. Although Nova might indeed have interesting things to say about the future of the web, I was addressing the majority of bloggers, who merely label things they don't understand as 'web 3.0', in my original comment.

Google results for 'PhotoSynth web 3.0':

"Providing a glimpse of what we might expect to see from Web 3.0"

"I am sure that after the launch of Photosynth, we will have a new web all the way and may be we'll call it Web 3.0."

"Web 3.0 Explainded"

"Related Tags: facebook web digg social networks web 3.0 technology"

"This is incredible. Amazing. The conglomeration of social ... This is web 3.0, the semantic web in the making."</comment></comment></comment>


Ask HN: How can I come out from analysis paralysis? - <AUTHOR>

Recently, I am facing the problem of analysis-paralysis. I could not take the sticky decision to do anything. Interms of learning any technology like Whether to learn android or Golang or Machine learning. I want to learn these things well. But I am not able to be stick continuously. Switching over from one technology to another technology happens frequently.<p>How can i come out of this issue? Any suggestions or feedback? <comment><USER2> Quick and probably useless suggestion: Build a product. It's hard to learn a language/framework/technology "just because" \-- it's much easier when you're working on a specific project. (You mentioned Go and Android; combine those two; with Go 1.5 just now being released with solid Android support, that's a green field, i.e., there's very little that's been done in that area and people would be keenly interested in anything you might do.) Pick something and get started, and resign yourself to the fact it means over the next year you're going to be acquiring expertise in _that_ and not any of those _other_ enticing technologies. You can't learn everything. Sad but true.</comment>


Apple Reports Third Quarter Results - <AUTHOR> http://www.apple.com/pr/library/2011/07/19Apple-Reports-Third-Quarter-Results.html <comment><USER2> Ridiculous, Apple has nearly as much _profit_ per quarter as Google has _revenue_.

<comment><USER3> Indeed. Though most iPhone users are Google users, so Apple doing well should also help Google grow.

</comment></comment><comment><USER4> I bought my first Mac on Friday at the Mall of America. There's a Microsoft store right across from the Apple store. It was amazing to me how much more crowded the Apple store was. There were a lot of people at the Microsoft store but it was ridiculous how many were at the Apple store.

<comment><USER5> I'm mostly a Windows user, and I don't know why I'd go to a MS store. For example, if I want an iPad or MBP I'd go to an Apple Store. I'm not sure what other stores carry them (does Best Buy?). But I know that the Apple store does.

In contrast, I can buy a PC anywhere. But I don't actually know what specific PCs are available at the Microsoft store. I do know they have all of the available phones at the MS store, but given that the only phones I'm interested in are those by my carrier, I'm more inclined to go to my carrier store (to see all carrier phones) than the MS store.

<comment><USER6> I think one of the main benefits to the Windows stores (besides the fact that they sell Windows Phones, and by all accounts don't actively try to talk their customers out of buying them) is the fact that they sell systems there which are part of Microsoft's Signature PC line.‡

For those unfamiliar, it's basically a system from a vendor without all the crap that they install on a fresh system. The fact that Microsoft actually has to have a service like this is certainly an interesting topic for discussion (I'm sure Microsoft would prefer that vendors not load up new pc's with garbage software that slows them down)

‡ <http://signature.microsoft.com/Default.aspx>

<comment><USER7> I agree it's great you can get a PC without the crap installed but I think MS have to tackle off the root problem and have it so that's how they all come.

Seriously, you'd get a noticable performance increase on the average low end laptop right off the bat if they could do that.

<comment><USER6> My understanding of how that works (which I admit is an outsider's understanding) is that PC vendors subsidize the price of new PC's by installing crapware on them.

The race to the bottom in PC pricing basically left them scrambling to find ways to keep their prices low without gutting their already-falling margins.

I don't think Microsoft is in a good position to require PC's to stop doing this (unless they were willing to eat the loss of vendor revenue by lowering their Windows license cost, which would effect their own margin).

I'm left optimistic by companies like Lenovo, who are now marketing crapware- less machines as a competitive advantage (as a byproduct of their Rapidboot initiative), and hope maybe other vendors will try and compete.

I don't remember the last time I bought a Windows PC where the first thing I did wasn't to immediately wipe it and reinstall the OS from an OEM version, although it'd be nice not to have to do that.

EDIT: realized I started two comments in a row with "Yeah,". That makes me almost as angry as if I'd littered the comment with "like, you know"'s...sigh

<comment><USER7> I agree with every word, I just think for moderate users using mid-level or above machines, crapware is a big enough problem that MS should be being worried about the impact it's having on the public perception of Windows.

</comment></comment></comment></comment></comment></comment><comment><USER8> Absurd. It's also awesome that they continue to mock the Street by giving their ridiculously low estimates for the succeeding quarter.

<comment><USER9> I saw that - the revenue projection for next quarter is $3.5b _less_ than revenue for this quarter.

I don't understand this. Are there any rules for making accurate revenue projections?

<comment><USER10> I'm sure not going to buy an iPhone or iPad right now, given the heavily- rumored launch of the next-gen models in late Q3. That could be why they're sandbagging the revenue projections.

<comment><USER11> There's never a bad time to buy an Apple product since the resale value is so high.

</comment></comment></comment></comment><comment><USER12> Have to hand it to Apple leadership. How much has he accomplished in 12 years? Taking leadership, turning the org around, the iPod lineup, building a super- strong brand, then iPhone, iPad and now the Mac sees a healthy resurgence. I am sure that if he has to start afresh, Mr. Jobs will do it and build another apple.... what a fantastic leader.

</comment><comment><USER13> Apple is firing on all cylinders. What is hard to appreciate is just how fast they are growing even though they are already such a big company. Can you imagine Microsoft/IBM/Oracle growing profit 125% in a year company wide when they are already as big as they are now?

Most companies that get this big can't keep growing at this kind of rate. Yet, they still have a lot of room to grow in different markets - TV, PC, Mobile, Advertising, (other?)

Incredibly impressive.

<comment><USER14> Well, IBM isn't on markets that grow. Smartphones are obviously really hot right now, even the 'losing' players are growing. By leading the growing markets they can cut the biggest slice of the pie.

<comment><USER15> Yeah, but you could say the same thing about Apple if it had stuck to making only Macs. Also, Apple is to a great degree responsible for making some of the markets in hot. Thus the comparison is still valid

</comment></comment></comment><comment><USER16> Oppenheimer mentioned that Apple is transitioning products and that it will impact their projected revenues. Is this Lion/MBA/MB/iPhone? If so, how would that lead to a softening of revenue?

<comment><USER17> _If so, how would that lead to a softening of revenue?_

I would guess that the transition involves the iPhone 5 (or 4S or whatever you want to call it) - announcing the iPhone 5 will kill iPhone 4 sales, and Apple always seems to have trouble meeting demand at product introductions so product constraints will probably mean lower sales/revenues.

<comment><USER16> That makes sense. My initial thought was that iPhone "5" would lead to a spike in demand and therefore a spike in revenue. I forgot to take into account the ability (or lack thereof) to MEET demand. Thank you.

</comment></comment></comment><comment><USER18> Absolutely absurd and amazing at the same time. Tip your hat to Apple, nothing else to do in this case.

<comment><USER19> Or you can marvel in amazement at their greed in suing their competitors.

<comment><USER20> Please...take your trolling elsewhere.

<comment><USER19> Please explain how I'm trolling. aforty said that there was nothing you could do but admire Apple, I have a different yet valid reaction.

<comment><USER11> You do realize that the actions Apple takes are what is responsible for its success?

</comment></comment></comment></comment></comment><comment><USER21> $11.1 billion cash flow from operations for the quarter, $76 billion in cash and marketable securities.

</comment><comment><USER17> 20.34 million iPhones in the quarter comes out to about 225,000 iPhones sold per day.

Google recently announced 500,000 Android "activations' Android per day.

So Android still has the market share advantage?

<comment><USER22> Tim Cook just said that they sold about 33 million iOS devices in the last quarter. (20.34 million iPhones, 9.25 million iPads and half of the 7.54 million iPods.) That's the number you have to compare. That's about 363,000 iOS devices per day.

It's quite astonishing that Apple is doing so well compared to Android. Apple sells two phone models, two tablet models and one iPod model – compared to dozens of Android devices, some vastly cheaper than any iPhone.

<comment><USER17> ugh & tjogin, good points.

</comment></comment></comment><comment><USER11> Apple is in a position now where it could shave off some margin on the Mac side and try to bump that share number up to 10% or 20% or even 30%.

The world would be a better place.

</comment><comment><USER23> The nice thing about this is that they sold 20 million iphone 4, which is actually over an year old now ...

<comment><USER24> Exactly! I think a flood of new iPhones owners will be hitting this fall. I know plenty of iPhone 3GS owners waiting to pick up the rumored iPhone 5. By that point, I may even be ready to replace my iPhone 4 with the 5...

<comment><USER25> Chalk me up as one of those 3gs owners. I was originally waiting for the white iphone 4, but, well by the time April rolled around there didn't seem like much reason to bother with it anymore.

</comment></comment></comment><comment><USER26> iOS is about to displace Windows/DOS as the top selling OS in the U.S. A Microsoft OS has probably been the top seller for the last 30 years.

Just a rough back of the envelope calculation is 12-13 million iOS devices sold in the U.S. this quarter. (7 million iPhone, 3.5 iPad, 1.6 iPod Touch at 35% US/Global).

In comparison, 2Q11 non-Mac PC sales in the U.S. were about 16 million (and shrinking).

iPhone 5 on sprint/T-mobile and iPad 2 Christmas sales will give iOS a good chance to take the lead.

<comment><USER22> What about Android? It seems that worldwide quite a few more Android devices than iOS devices are activated per day. Are less than 35% of all Android devices sold in the US?

<comment><USER27> Where do you get that number?

<comment><USER22> From Apple's and Google's Q2 quarterly earnings call, respectively.

Google just announced that they activate 550,000 Android devices per day. That's in all likelihood (and considering the language) not an average value for the whole quarter but only the most recent value (that must have been rapidly growing during the last quarter). Still, the average for the last quarter is probably over or around 500,000.

Apple doesn't publish any such numbers but it's easy to estimate them. Apple sold 20m iPhones, 9m iPads and 8m iPods. Tim Cook said during the earnings call that about half of those iPods run iOS. That's about 33m iOS devices altogether for the whole quarter, for an average of 360,000 sold devices per day.

</comment></comment></comment></comment><comment><USER28> So many billions.

<comment><USER29> billions and billions...

<comment><USER30> their $75 billion in cash and negotiable securities compares with eBay's July 19th enterprise value of $37.5 billion, Microsoft's $188 billion, Sony's $28.5 billion and Google's $160 billion. They could buy eBay or Sony for cash.

Enterprise value is market capitalization (value of all the shares) less debt. Apple's was $316 billion on the same date.

Note - Apple is great at making awesome products, and most likely lousy, like everyone else, at taking over lousy companies.

<comment><USER31> I thought enterprise value was market cap _plus_ debt?

<comment><USER32> Yes, enterprise value (EV) is equity market capitalization (stock price times # of shares) plus the market value of debt (this can be lower than the book value of debt). Financial analysts often prefer to calculate something called "Net Enterprise Value", which is EV minus "excess cash" (total cash minus "required cash", where required cash is usually calculated as a certain % of sales). Net Enterprise Value gives you the present value of all future cash flows to the firm (you are ignoring the value of today's excess cash, which doesn't come from future cash flows). This is particularly useful when looking at companies like AAPL with tons of cash.

</comment></comment></comment></comment></comment><comment><USER33> and Lion comes Wed. Don't think they've ever announced a release date on a financial call.

</comment><comment><USER34> I think we need a windfall tax on Big Apple.

<comment><USER35> Will you offset the windfall tax with payments when Apple makes losses?

</comment></comment><comment><USER36> Bombastic numbers as usual. So, when can I get an MB Pro 13" with an up-to-par screen resolution? :)

<comment><USER37> A matte screen without having to choose an upgrade on the 15" MBP please.

<comment><USER38> Would you like an ExpressCard slot with that?

</comment></comment></comment><comment><USER39> While Apple is doing great, I see some flatlining starting next year once the market for high-price computers start to saturate. Just an opinion.

<comment><USER40> why is this comment being down-voted ?

<comment><USER41> Probably because people have been predicting this kind of thing for years. It is also worth pointing out that only 18% of Apple's ridiculous revenue comes from Mac sales.

<comment><USER42> To be fair, people have been predicting both sides of the coin for a while. People predicting success on hacker news tend to get upvoted, and people predicting failure tend to be downvoted, regardless of the reality of the situation.

Right now Apple is doing great. I would still question the sustainability of this business as an investor. What's next after ipad?

Long term, 5 years say, ipad/iphone margins will evaporate and Apple will need a series of "big things" to fill in the gaps to even have a flat growth curve to justify their market cap - that's my prediction.

<comment><USER43> And if any company has a track record over the past decade of delivering "big things" it's Apple. No company in recent history comes even close. Can they? Who knows, but if anyone can, it's Apple.

<comment><USER42> One decade is great. Plenty of companies have had their one decade in the sun - Microsoft, Cisco, IBM... Standard Oil.

I think your comment is a bit of hyperbole, the success of a company like Apple is not unprecedented in history. Apple could continue doing "big things" over the next decade, I just seriously doubt their ability to continue their growth and justify their market cap over the next 5-10 years. That's all. Possible, yes, in my opinion extremely, extremely unlikely.

And also like clockwork, I had a comment explaining why I thought they would fail, and have downvotes.

<comment><USER44> Take Microsoft: they are making most of the money off the same two products for _decades_ —OS and Office. Now take apple: 47% of they revenue is from product which was introduced four years ago (iPhone). 21%(!) comes from product which was released merely _a year ago_ —iPad.

Apple is not afraid to kill its cash cow if they think they have a better product in line—see how they dealt with iPod line before the iPhone—they were replacing the most popular model with something new.

I have no doubt Jobs has a vision many years into the future. It was amazing to watch his 1997 WWDC keynote and realize that now we see the fruits of his thoughts then.</comment></comment></comment></comment></comment></comment></comment>


Java on iPhone and iPad - <AUTHOR> http://chriskohlhepp.wordpress.com/java-on-iphone-and-ipad/

<comment><AUTHOR> I am happy to announce that a full Java "eco system" has been ported to IOS, the operating system powering the popular Iphone and IPad platform. This includes JIT based compilation, a graphical Swing subsystem and a development toolchain. The aim of this Java "eco system" is to deploy essentially unmodified Java applications on the IPhone and IPad - along with software written in the many other languages emerging on the Java virtual machine platform(JVM), such as Scala & Clojure. The significance is that there are many more existing, free Java applications than exist on the IPhone and IPad today. After all, the mantra of Java has always been "compile once, run anywhere."

</comment><comment><USER2> Very cool, but the sad part is that unlike most modern desktop systems, on mobile OS's all languages and runtimes other that the ones officially endorsed by the OS maker are probably a pain to use and deploy.

Its sad that you have to jailbreak to use the full capabilities of your own device.</comment>


Ask HN: How to get started in data visualization? - <AUTHOR>

Seeing that a few posts recommend having data visualization experience I was wondering how someone could get started in the field assuming that they already have a data-mining/statistics background.
I personally know that a top recommendation would be reading the classic by Tufte "The Visual Display of Quantitative Information.
What else is there on bookshelves and on the web? <comment><USER2> [http://www.amazon.com/Visualizing-Data-Explaining- Processing...](http://www.amazon.com/Visualizing-Data-Explaining-Processing- Environment/dp/0596514557)

beautiful book by <http://benfry.com/>

<http://infosthetics.com/>

<http://www.visualcomplexity.com/vc/>

<http://processing.org>

</comment><comment><USER3> <http://flowingdata.com/>

<http://junkcharts.typepad.com/>

for examples.

</comment><comment><USER4> Tufte's other books :)</comment>


Mitigations landing for new class of timing attack - <AUTHOR> https://blog.mozilla.org/security/2018/01/03/mitigations-landing-new-class-timing-attack/ <comment><USER2> > we are disabling or reducing the precision of several time sources

I am really happy to see this happening at long last. One can argue that almost every x86 SW side channel attack (Spectre and many prior ones that, for instance, leak parts of cryptographic keys from variable-time SW implementations) are being aided by userspace processes having access to high- frequency/high-res counters for which, in almost every case, they have no legitimate need. Perhaps future silicon should block direct access to the TSC & friends, except for processes that have a privileged flag set (hardly a new idea; IIRC some IBM PowerPC chips from the late 90s support that).

OS-level timing calls such as gettimeofday() could be degraded to a few microseconds resolution by default - poor enough to obscure delays caused by the state of various HW caches, branch predictors, etc.

Ultimately there is little reason to give every website js program access to a nanosecond-level counter by default, and many defense-in-depth reasons for _not_ doing that. So kudos to the firefox team here and hopefully chrom[e|ium] quickly copies this.

<comment><USER3> Timer coarsening is a bad idea. Turns out [1], a simple timer thread is a good enough poor man's cycle counter. What are you going to do: ban variables? Even if you could coarsen timers, adding random noise just makes attacks take longer. It doesn't make them impossible; attackers can just collect more measurements.

You need to address the specific vulnerabilities, not shoot the clock_gettime messenger.

[1] [https://www.usenix.org/system/files/conference/usenixsecurit...](https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_lipp.pdf)

<comment><USER2> > adding random noise just makes attacks take longer

No. It's true that with a low-res clock source in some cases an attacker can still get the precise measurement they're after by repeating an operation 1000x+ more times. In some cases though that is _not_ possible, because at that timescale the signal they're after is degraded by noise that they cannot predict or control: operating system interrupts, memory traffic from other threads, etc.

Anyway, even if a lower-res clock source helps only 10% of the time, on defense you should always prefer to make an attack complicated, slow, partially-reliable rather than trivial, fast, highly-reliable.

Developers who need a high-res clock source for profiling, etc, should of course still be able to enable one selectively.

> You need to address the specific vulnerabilities, not shoot the > clock_gettime messenger.

You can and should do both, to gain some defense-in-depth against future attacks.

<comment><USER4> > In some cases though that is not possible, because at that timescale the > signal they're after is degraded by noise that they cannot predict or > control: operating system interrupts, memory traffic from other threads, > etc.

Wrong, those signals will average out over a long enough data collection time.

<comment><USER5> I'm not sure that's true in this case, if your timer doesn't have the resolution to ever detect a cache hit, how can you measure impact?

<comment><USER6> It's one of those things that feel utterly counter-intuitive.

But if you have a signal overlaid with random noise [1], and you know what your signal's looking like and when it's happening, you can correlate. For example, a small delay that occurs at certain known points (or not), will introduce a bias into a timer measuring it, no matter how noisy or coarsely quantized that timer is.

Similar techniques have been used in other fields for decades to pull useful signals from far below the noise floor (e.g. a lock in amplifier can go dozens of dB below the noise floor, because it, essentially, correlates frequency and phase and thereby eliminates all but a tiny sliver of noise. E.g. GPS signals are typically 20 dB below the noise floor.

[1] It doesn't have to be random.

——

So these mitigations just make the attacks harder, hopefully hard enough that they become not feasible to be exploited widely.

<comment><USER7> Then why can't we make sharp high resolution photos of distant planets? Shouldn't we be able to average out all the noise for every pixel if we just collect light long enough?

<comment><USER4> No, because there's correlated noise: all the stuff in between us and the planet.

<comment><USER7> What type of stuff is between us and the planet and stays on the same pixel all the time? I would assume everything in the universe moves all the time. We move. The other planet moves. How can something block the same pixel of our view of the planet all the time?

<comment><USER8> I don't think other planets usually move enough to cross one pixel. They've mostly been detected by changes in brightness of their host star. [https://en.wikipedia.org/wiki/Methods_of_detecting_exoplanet...](https://en.wikipedia.org/wiki/Methods_of_detecting_exoplanets#Established_detection_methods)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I warned about high accuracy SharedArrayBuffer-based timers on Hacker News[1] when Firefox and Chrome were adding SharedArrayBuffers.

Glad to see people finally waking up to the fact that high accuracy timers are security vulnerabilities on most synchronous systems.

1\. [https://news.ycombinator.com/item?id=14057091](https://news.ycombinator.com/item?id=14057091)

<comment><USER10> Why is SAB different from other improvised clocks? Are other operations too slow?

<comment><USER11> The other operations, like posting messages, have much higher latency noise. SAB allows to observe counter changes effectively within few CPU cycles with stable latency.

</comment></comment></comment><comment><USER12> Mozilla previously reduced the precision of performance.now() to 5µs. This was trivially defeated by just running performance.now() in a tight loop, as described here: [http://www.cs.vu.nl//~herbertb/download/papers/anc_ndss17.pd...](http://www.cs.vu.nl//~herbertb/download/papers/anc_ndss17.pdf) see the technique in section IV called time to tick (TTT):

>The idea behind the TTT measurement, as shown in Figure 4.4, is quite simple. Instead of measuring how long a memory reference takes with the timer (which is no longer possible), we count how long it takes for the timer to tick after the memory reference takes place. More precisely, we first wait for performance.now() to tick, we then execute the memory reference, and then count by executing performance.now() in a loop until it ticks. If memory reference is a fast cache access, we have time to count more until the next tick in comparison to a memory reference that needs to be satisfied through main memory.

>TTT performs well in situations where performance.now() does not have jitter and ticks at regular intervals such as in Firefox. We, however, believe that TTT can also be used in performance.now() with jitter as long as it does not drift, but it will require a higher number of measurements to combat jitter.

So, what stops this method from working, even with 20µs resolution performance.now()?

<comment><USER13> Maybe the clock can be made to drift? You can make the clock a random walk with the restriction that it never decreases, and doesn't deviate more than 20us? That is, each tick of the clock actually just adds a non-negative interval to the clock. Since the deviation from real time at _tick_ depends on the deviation at _tick-1_ it would require more time to get good accuracy by averaging measurements.

<comment><USER4> Run it for long enough and you can average out the random drift.

The issue cannot be fixed without disabling speculation entirely on current hardware.

</comment></comment></comment><comment><USER14> > The resolution of performance.now() will be reduced to 20µs.

That will reduce the data rate of this particular covert channel, not prevent the attack altogether. Even adding random noise would not rule out the attack.

<comment><USER15> Presumably they are aware of this. If they reduce the data rate enough, the attack becomes less useful in practice. Presumably their engagements with security researchers and other browser vendors, along with internal experimentation, led them to believe that this resolution decrease offered significant mitigation.

</comment></comment><comment><USER16> Reducing the resolution coming from system timing sources is not going to stop attacks which have potentially infinite execution timelines.

Limiting JS execution resources, and in particular CPU cycles, will actually stop instead a whole swath of timing and resource-dependent attacks.

Please put your thought and chime into this thread:

[https://bugzilla.mozilla.org/show_bug.cgi?id=1414675](https://bugzilla.mozilla.org/show_bug.cgi?id=1414675)

Allowing infinite resources for remote programs is something we don't even do for _local_ programs. Giving a ceiling to the JS runtime is a _sound_ reasoning.

Please comment on the bug tracker!

<comment><USER16> Commenting on the upstream bug tracker would do so much more than lazily clicking upvote.

</comment></comment><comment><USER17> So a quick question. What is the alternative to SharedArrayBuffer? One of the web applications i work on use it a fair amount. Is our app now just going to break?

<comment><USER18> The feature was previously hidden behind a about:config option[0]. Not a ff user but might check if it is back and for now let your users now?

Is chrome or edge affected?

Should the SharedArrayBuffer mdn page be updated; or maybe moz://a will fix it quickly enough to allow it on by default again.

By the looks of it, this will affect a lot of webgl code.

[0]: [https://developer.mozilla.org/en- US/docs/Web/JavaScript/Refe...](https://developer.mozilla.org/en- US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer#compatNote_2)

<comment><USER19> Chrome will be taking the same steps-

> In line with other browsers, Chrome 64 will disable SharedArrayBuffer and > modify the behaviour of other APIs such as performance.now, to help reduce > the efficacy of speculative side-channel attacks. This is a temporary > measure until other mitigations are in place.

[https://sites.google.com/a/chromium.org/dev/Home/chromium- se...](https://sites.google.com/a/chromium.org/dev/Home/chromium- security/ssca)

</comment></comment></comment><comment><USER20> Perhaps high-resolution timing needs to be a permission and not a "right", just like access to location or webcam data.

<comment><USER21> Can the user be expected to make the right choice there? I have no problem deciding if Uber, a local restaurant or a news site need my location or camera data. I have no idea why they would need high precision timing data or even what "high precision" is. Maybe it's reasonable, maybe it isn't but it's not as easy to decide from the user's shoes as location/camera.

<comment><USER15> Most of the time, they won't need it. When they do, they'll need to make the case to you explicitly. Some phone apps currently do this when they need a non-obvious permission.

<comment><USER22> I think that the point is that most users can't be expected to understand the risks present in allowing a high resolution timer. It's a very nuanced decision, unlike camera and storage and location.

<comment><USER23> As a web developer, I am not entirely sure why a web app necessarily needs high precision timers.

<comment><USER24> I'm developing a website that will require somewhat high frequency timing to account for network latency but I think standard getTime is accurate enough with millisecond accuracy which I think will satisfy my requirements (I still need to test this though).

I can't think of something that needs microsecond level timing though.

<comment><USER25> A browser based DAW/soft-synth?

<comment><USER23> That's one of those things that seems better suited to being a native App.

I realize there's a whole group of people who want to make the web browser as powerful as native Apps but I for one do not.

<comment><USER25> I agree; I would like to see a move away from trying to cram everything into a browser app, and towards adding better connectivity and community features to native apps.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER4> This is basically an arms race. As long as the fundamental issue is not fixed and all silicon replaced, there will always be people coming up with new ways of exploiting the Spectre vulnerability in unexpected ways.

</comment><comment><USER26> Shall we now have a new website permission for timers or at least high- precision timers? (It will be _quite_ fun to get end-users to understand the ramifications of such a thing...)

<comment><USER27> Has there ever been discussion about a permission for "running as an app" that would make a distinction between web pages and apps? I like that the web gets more features that allow apps to run cross-platform, but I don't fully understand why we make no distinction between pages and apps.

</comment></comment><comment><USER28> I wonder if this part of the reason why Google added Site Isolation as documented here: [https://securityboulevard.com/2017/12/google-chrome- update-f...](https://securityboulevard.com/2017/12/google-chrome-update- focuses-enterprise-security/)

<comment><USER29> Yes:

[https://support.google.com/faqs/answer/7622138#chrome](https://support.google.com/faqs/answer/7622138#chrome)

<comment><USER30> I read on Twitter earlier today that they were working on site isolation for six months before they found out about this.

Certainly shows why it's useful though.

</comment></comment></comment><comment><USER29> If instead of measuring time directly attacker would try to increment a counter in a loop in another thread, would they get enough accuracy and precision for the attack to work?

<comment><USER31> That's exactly what the shared buffer array thing is about. You can use a webworker to do exactly that and still get accurate enough timing to do accomplish this.

</comment></comment><comment><USER32> Not being familiar with the usage of SharedArrayBuffer, is disabling it the nuclear option (eg bad for everyone involved and clearly not the best option if others existed)?

I see over 100 pages of commits on github referencing it, which tells me it's not exactly rare?

<comment><USER33> High performance applications need to share memory between threads. That feature let you do it in JS, and is the basis of the future shared memory multithreading feature in Web Assembly.

So if you want the next Photoshop or Final Cut Pro to be browser based we'll need to make it work securely.

<comment><USER28> Seems like virtualization is the way to go for JavaScript. Hypervisors such as XEN in HVM mode with modern processors don't result in a large performance penalty, so I imagine Chrome will gravitate towards it for running untrusted code.

<comment><USER34> If running JS (that can access the DOM) in a separate process from the renderer were feasible, browsers would have done it long ago.

<comment><USER35> You just need to run it in a process without any secret data from other origins or the browser/system.

That seems feasible without significant performance loss in most cases.

There is still a fingerprinting issue though that might not be possible to fully remove without huge performance cost.

<comment><USER4> Process isolation does not prevent Spectre. In theory, you can carry out Spectre over the network.

</comment></comment></comment></comment></comment></comment><comment><USER36> I am very curious about Yehuda Katz's question on CSRF attacks and GET requests: [https://twitter.com/wycats/status/948812652997885952](https://twitter.com/wycats/status/948812652997885952)

</comment><comment><USER24> Does anyone know how trustworthy client time accuracies are accounting for clock skews?

After a bunch of google searches, I have not found anything suggesting how accurate clocks are for clients besides Charlies algorithm [0] (I want to know UTC time/epoch... whatever from each client so i can compare them) which I am concerned about adopting due to the wiki page assuming a high quality network.

[0]: [https://en.wikipedia.org/wiki/Cristian%27s_algorithm](https://en.wikipedia.org/wiki/Cristian%27s_algorithm)

<comment><USER37> These timing attacks are all about local time deltas, not absolute times, because it's measuring the CPU caches, not anything to do with the network. Things like "this operation took 5µs, which suggests that the data it was accessing was/was not cached".

</comment></comment><comment><USER38> This says

> Specifically, in all release channels, starting with 57:

Firefox 57 was released nov 14, did it already include those changes?

<comment><USER39> I think that means it's been pushed to the release channel.

</comment></comment><comment><USER40> Simple explanation: one of the attack is based on speculative out of order execution to access memory you don't own.

Though cpu cleans registry, it doesn't clean up cache. So later through timing you can gain knowledge about something you shouldn't know.

That is why, reducing timers precision can help

<comment><USER4> If it cleans cache, you can detect the evicted address tag.

</comment></comment><comment><USER41> So what are the odds the NSA was aware of this?

<comment><USER42> No one can blame them for knowing… The real question is if Intel knew and were they in any way incentivised not to fix it sooner. It would be very "easy" to justify not fixing a bug that went undetected for let's say 10 years…

If there ever was NSA hardware bug this is how it would look like. I doubt we'll ever know.

</comment></comment><comment><USER43> I wonder if and how Waterfox or Palemoon can keep up with these security issues? (Disclosure: I am a Firefox Nightly user)

<comment><USER44> The performance.now() patch changed one line of code, so backporting that shouldn't be hard. The SharedArrayBuffer change is just flipping a pref, so that should be easy as well (assuming it's even supported - isn't Palemoon a pretty ancient fork of Firefox? Even Mozilla's most recent ESR release doesn't have SAB enabled by default).

</comment></comment><comment><USER45> And Chrome? Do they do similar things NOW?

<comment><USER46> Yes

[https://sites.google.com/a/chromium.org/dev/Home/chromium- se...](https://sites.google.com/a/chromium.org/dev/Home/chromium- security/ssca)

</comment></comment><comment><USER47> If you stack kernels like

Hypervisor Guest OS

Is there double the performance hit of all these attack fixes?

</comment><comment><USER48> And this is why running arbitrary software from people you've never heard of will _always_ suck more.

</comment><comment><USER49> Is latest ESR version (52.5.3) of Firefox patched?

I can't invoke SharedArrayBuffer for sure but what about the resolution of performance.now()?

<comment><USER44> The performance.now() change is due to ship to ESR users with the 52.6 release on January 23.

</comment></comment><comment><USER50> For the interim, may it be a wise idea to include an inlined script at the very top of any page which includes untrusted 3rd party code, in order to overwrite the Performance tools by a wrapper? By adding some random to the real numbers this may help mitigating effects ...

E.g.,

      (function() {
          var global = (function() { return this; }).call(),
              _Performance = global.Performance,
              _performance = global.performance;
          global.Performance = /* ... snip ... */;
          global.performance = /* ... snip ... */;
      })();

[Edit]: This could be injected by a browser-plugin on the user's side, as well.

<comment><USER50> Note: "May it be a wise idea ... ?" is an open question, meant to start a discussion. No need for down-voting, just answer or ignore it. :-)</comment></comment>


Hollywood Lobby Group's Creepy "Open Love Letter" to the TPP - <AUTHOR> https://www.eff.org/deeplinks/2015/02/hollywood-lobby-groups-creepy-open-love-letter-tpp

<comment><USER2> Lamest attempt yet, in the history of humankind, at a Trojan Horse.

<comment><USER3> Explain?</comment></comment>


Thoughts on an open Twitter replacement: Concentrate on what's done poorly - <AUTHOR> http://www.arpitonline.com/blog/2011/03/14/thoughts-on-an-open-twitter-replacement-concentrate-on-whats-done-poorly/

<comment><USER2> <https://identi.ca/></comment>


Show HN: Pknoa – A simple way to keep track of everything - <AUTHOR> https://pknoa.com <comment><AUTHOR> Hi, the developer here.. pknoa! is a very simple way to keep track of everything. It uses natural language processing, to help you store, quantify and organise information.

All you have to do is remember the pknoa! syntax:

unit action time

So, when you type something like:

30min meeting with Peter Smith 15 June 10:30pm

pknoa! breaks it down and quantifies it for you. All your entries, with the same name (in this case, 'meeting with Peter Smith'), are grouped together, enabling pknoa! to generate amazing statistics. pknoa! also has other cool features, such as tags (allowing the grouping of two or more activities, to create new views) and sharing (allowing invited users to see, in real time, what you are up to).

We hope you enjoy it as much as we do

<comment><USER2> How your landing page looks with javascript turned off: [http://picpaste.com/pics/pknoa- nojs.1405592840.png](http://picpaste.com/pics/pknoa-nojs.1405592840.png)

Note, no scroll-bar, content sitting on top of itself, no way to navigate around the page without javascript.

Therefore - no thanks.

<comment><AUTHOR> We'll definitely take your suggestion under consideration, but don't you think it's a bit harsh to dismiss the whole application because of it? Give it a try, you might like it :-)

<comment><USER2> It shows a lack of attention to detail, a lack of viewing the full picture of all your potential users, and a lack of consideration for those with accessibility issues.

Therefore, if the simple landing page has all of those lacks, it stands to reason (since the app. is written by the same folks that created the landing page) that the app. may likely have all the same lacks, and possibly even more.

</comment></comment></comment></comment><comment><USER3> Cool! Will look into it!</comment>


MacKeeper Is Malware - <AUTHOR> http://hypertexthero.com/logbook/2014/05/mackeeper-malware/

<comment><USER2> Crap article, crap title, written by someone who can't distinguish between "of questionable value and utility" and "malware". Nowhere in the article does it state _why_ MacKeeper is malware. In fact, the last linked article at the bottom ("Please beware") specifically says that MacKeeper is not malware. MacKeeper might be a waste of money, it might use questionable advertising (pop-ups, and typically on less savory sites), but malware it is not.

Additionally, the information on AV and how it works is sloppy. This is already covered well by other comments here, so I won't belabor it.

<comment><USER3> > _Nowhere in the article does it state why MacKeeper is malware._

I don't care about any such fine distinction. It's an annoying crapware that pops out of adds that pretend it's a system message for my Mac and that I'll be in trouble if I don't use it.

That's malware enough for me.

</comment></comment><comment><USER4> Who would have known that software agressively advertised through pop-ups, pop-unders and pop-everything-else on torrent websites would be malware?

</comment><comment><USER5> > Then they must find a copy of the virus, examine it and add it to the list > of virus definitions used by their software.

Strictly speaking, this isn't true. The major AV companies use file behaviours to try to stop viruses as a first line of defense, something like "if a file is an executable and it tries to run some manner of encryption on files in the My Documents folder, its likelihood of being a virus is high". AV companies use file checksums as a last-ditch effort because most viruses out there are polymorphic (they have some unused data section at the end that randomly changes, which busts checksum detection).

A lot of AV companies also offer virus cleanup services, so if you do happen to get hit by a virus that snuck by, they can help reduce the damage.

Nobody's saying "use AV and you don't have to ever worry again". AV is just another piece of the defense puzzle, along with user vigilance (don't click that link in that email from "eBay"), operating system restrictions, network restrictions (to help prevent data egress), and so on.

</comment><comment><USER6> "Another important thing to know is that no AV software is capable of intercepting a brand-new virus."

This is simply not true. Here's an test where AV products are frozen before the brand new malware samples are collected: [http://www.av- comparatives.org/retrospective-test/](http://www.av- comparatives.org/retrospective-test/)

</comment><comment><USER7> I read through this quickly and completely missed the part where MacKeeper is Malware. Malware is a strong word for something that just has questionable utility.

</comment><comment><USER8> Can't comment on MacKeeper itself, but i feel this is a really stupid advice...

You don't need an antivirus because virus definitions are slow to update? Fine, but what about old virus? Am I protected by magic?

Also, I feel like the increase in OSX popularity will result in an increase in malware for it, so I'm not sure for how long "OSX is virus-free" will be valid

<comment><USER9> You protect yourself by not clicking on executables you don't trust (or running them in a VM). Its not magic, and its better than a false sense of security from anti-virus which is easily bypassed / poisoned (for example, even a clean bitcoin client will set off AV alarms).

</comment></comment><comment><USER10> It feels like this article is confusing MacDefender with MacKeeper. MacKeeper is crapware, sure, but not malware.</comment>


Your Robot Butler—Or Factory Worker—Just Got A Lot More Affordable - <AUTHOR> http://readwrite.com/2013/10/21/robot-ubr-1-willow-garage#feed=/hack&awesm=~oldwfNbOVM6DSQ

<comment><USER2> It's still expensive at $35,000. If the price is brought down to say 3,500 then I'm guessing a lot of early adopters will try to get their own butler robot.</comment>


A parody blog of the tech industry - <AUTHOR> http://cloudegory.com/

<comment><USER2> Presentation layer: Cool site, and good stuff, but it might serve the writing better if this content was presented in a template that is more akin to a true news site. There's something about that three-column layout that is cool to look at, but the presentation tips the hand that this is parody pretty quickly.

</comment><comment><USER3> The thing about Airtime was so true! I remember their launch being such a big deal! After that, no one gave a crap about them!</comment>


Curiosity Killed the Mario - <AUTHOR> http://www.michaelburge.us/2019/05/21/marai-agent.html <comment><USER2> I find the video super interesting to watch. It's not like previous Mario bots I've seen that were programmed with path-finding and a go-right goal which race through the level perfectly. Instead, this makes me think of a very young kid playing the game who isn't entirely sure what they should do, and is more interested in figuring out what the controls do and usually goes right because that's where more stuff is, not because they know they're "supposed" to do that.

The idea of curiosity (seeking out states that lead to unpredictable stuff) being a good scoring function for an AI seems really compelling to me. I'm really curious about what other places this idea can be applied to.

<comment><USER3> At first glance it reminds me of an old paper I read on causal entropic forces, [1] which is kind of a thermodynamic approach to understanding the emergence of complex behaviors. I've always had that in the back of my mind as an idea I'd never have the time or resources to investigate further.

[1] [https://www.google.com/amp/s/phys.org/news/2013-04-emergence...](https://www.google.com/amp/s/phys.org/news/2013-04-emergence- complex-behaviors-causal-entropic.amp)

</comment></comment><comment><USER4> See also playfun/learn fun, a game-agnostic AI that uses raw uninterpreted memory bytes as it's signal: [http://tom7.org/mario/](http://tom7.org/mario/)

</comment><comment><USER5> I remember Schmidhuber showing off "artificial curiosity" stuff a while back (e.g. [http://people.idsia.ch/~juergen/interest.html](http://people.idsia.ch/~juergen/interest.html) ). In particular, ideas like "compression progress" have been influential on my own research about how to measure what's "interesting", and I've implemented a rudimentary version of PowerPlay (and a slight alternative) at [http://chriswarbo.net/projects/powerplay](http://chriswarbo.net/projects/powerplay)

(He's since applied these ideas to art, humour, etc. which I think is a nice thought, but not worth taking particularly seriously)

</comment><comment><USER6> Pity it actually fails at the truly complex levels. Gets stuck in decision minima due to lack of memory?

Or is the reward too sparse?

Cannot find the symbolic notion of progress and dies of boredom?

<comment><USER7> For longer levels, I think training on later parts of the level tend to be change the policy to not do as well in the earlier parts. I suspect it would do fine on the linear levels if the number of agents and batch size was increased.

</comment></comment><comment><USER8> The author mentions self-driving cars and, indeed, it works well for Mario Kart: [https://www.youtube.com/playlist?list=PLTWFMbPFsvz122oi3aEWZ...](https://www.youtube.com/playlist?list=PLTWFMbPFsvz122oi3aEWZ9QTJwwF78UPr)

</comment><comment><USER9> This is an interesting idea for concolic execution. I assume it's been done before, but a quick search doesn't turn it up. Maybe it's just assumed?

</comment><comment><USER10> There have been labs at my uni in non-deep RL or evolutionary robotics researching curiosity rewards a few years back, but I've not seen something like this until this year.

Is it just scale and more powerful NNs that is making curiosity work well now? The Montezuma and this Mario video look like a serious breakthrough to me (non-expert).

</comment><comment><USER11> The video is set to play back at a funky speed, playing at 1.5x speed is much more natural.

</comment><comment><USER12> Is there a paper published (or preprinted) on this?

<comment><USER13> There is a link to the original work by OpenAI: [https://openai.com/blog/reinforcement-learning-with- predicti...](https://openai.com/blog/reinforcement-learning-with-prediction- based-rewards/)

</comment></comment><comment><USER14> How is this different from brute forcing and then recording a success path? Not sure this qualifies as AI

<comment><USER7> A naive brute force algorithm would take a lot longer to complete some of these levels.

Let's say it takes 45 seconds to complete a level. That's 45 * (60 / 12) = 225 moves. The size of the action space is 14, so you'd be looking at 14 ^ 255 (give or take a few orders of magnitude) trajectories before finding the solution.

To brute force in a reasonable a time, you would have to look at the environment and have the algorithm iterate through all trajectories in a clever way. For instance, you may choose to only try trajectories that are constantly moving right. This strategy may find a solution to 1-1 fairly quick, but this does not generalize to other levels, especially ones that requires backtracking or waiting.

You'd have to design a pretty gnarly algorithm for it to beat 1-1, 1-4 and 2-2. This gets even more complicated if you bring in other environments: the original paper also trained on Montezuma's Revenge, Private Eye, Venture, Freeway and Gravitar

<comment><USER14> They have 350 hrs of playing per level. They could brute force on these parameters.

1\. Detect to_avoid moving obstacles

2\. Detect to_avoid stationary obstacles (gaps)

3\. Player moves (jump (left, right)(up, far, farthest), walk)

4\. Success (hitting the flag, reaching princess)

Once above information is available on the screen, it becomes easily bruteforcible. I was hoping use of genetic algorithms i.e. they could have taken success from one part (path) of level and crossed it with another and tried that with across levels. But there won't be a generic strategy or learning anyways as there's fair bit of randomness. So this does seem like brute forcing for success path

<comment><USER15> The whole point is that the algorithm doesn't know about obstacles or success as a concept baked into the algorithm. Likewise, this is pretty initial research, meant to inform and promote

In other words, this isn't meant to be super useful by itself. It seems tailor made (as many of these things do) to play super-simple 80's video games and literally nothing else, but it's an interesting proof of concept. I'd also be interested in different iterations on this general pattern - for instance, something that didn't translate directly from screen + button -> prediction, and instead had some interstitial systems - translating from screen -> entities, then predicting entity state of entities given button presses. It'd also be interesting to see how this performs with ML algorithms designed to learn on the fly instead of through training from a static set of data (at least, this looked like it learned through back propagation - I skimmed).

But I can see broader practical applications for this in, for instance, recommender systems trying to break users out of the closed feedback loop that people tend to end up in when going down certain rabbit holes (e.g. watch one Flat Earther conspiracy video and suddenly that's all you see for a week because the recommender system _knows_ that people who look at one will look at more). The point being: the real test comes when this strategy is exposed to more diverse problem spaces, it's just that those are harder to model and we need to weed out the pointless stuff first.

</comment></comment></comment></comment><comment><USER16> In reinforcement learning this can be attributed as exploration be exploitation

</comment><comment><USER17> A couple years ago there was a paper reported somewhere (it may have been here) that dealt with unsupervised learning using entropy as the only fitness function. Regardless of the task or any other factors, the researchers used maximizing entropy as the only goal. And this immediately led to the development of complex, interesting, and desirable behavior. When used for a system balancing a pole, it would learn to balance the pole upright. When given a ball where a hoop was present, it would automatically navigate the ball through the hoop. I tried to reach out to the author of the paper to get a full copy of the paper (could only find a paywalled abstract online) but never got a response. It seemed like a very interesting approach, and this sounds like doing basically the same thing. Favor moving to any state which increases the maximum likely future states. Increase entropy, 'intelligence' emerges.</comment>