Why is RDF so old, complicated, unpopular and still not discarded? - <AUTHOR> http://www.semanticoverflow.com/questions/2918/why-is-rdf-so-old-complicated-unpopular-and-still-not-discarded <comment><USER2> As a preamble, when RDF was conceived, databases drove many sites on the web, but their data tended to only be exposed as HTML, instead of a more machine- friendly format.

Now, there are two perspectives on what RDF is.

To an idealist, RDF is the universal data format. There are no semantics baked-in, and you can write arbitrary subject -> predicate -> object triplets to express any possible relationship. To an idealist, it's the perfect format for exposing all the structured data on the web in a machine readable form. The dream has always been for automatic agents to crawl the semantic web for you, understanding the meanings of the RDF triplets, and using them to reason out the solution to your query.

To a pragmatist, that dream has always sounded like a bunch of bull. Absent the presence of strong AI, it's a complete pipe dream that a piece of software will ever be able to infer the "semantic meaning" of interlinked RDF, just because it happens to be defined by triples. At the end of the day, you're going to have a programmer writing rules against specific terms in RDF, and if that's the case, than RDF is nothing more than an _extremely_ awkward API.

Fortunately for the web, the pragmatists won. APIs are everywhere, and RDF is nowhere.

Unless strong AI happens to be right around the corner, the web dodged a real bullet there. Personally, I'm of the opinion that any web agent that could possibly puzzle through RDF triplets should have no problem understanding our APIs, in any case.

<comment><USER3> Not quite. While RDF certainly isn't all that the idealists claim, you can still get some benefit of it without strong AI.

Mainly, it provides a consistent model for handling the notion of a "field". Non-RDF apis typically return fielded JSON or XML, the structure of which is only specified within the documentation. In order to integrate two services not originally designed to inter-operate, you have to write lots of custom glue code.

RDF is at least amenable to writing generic "rules" to govern field mapping and inference, rather than one-off glue code (which usually ends up being a hacky script). So sure, if you're integrating one service, a hacky script is probably easier. But if you want a coherent _system_ for integrating large numbers of services not originally designed to inter-operate, RDF makes things a lot easier.

So there's some benefit, even if it isn't as dramatic as its proponents claim.

Plus, there's the fact that while strong AI isn't yet on the horizon, RDF is a lot easier for weak AI (inference engines, data mining, etc) to ingest, and weak AI is getting better all the time.

Actually, one of the biggest problems with RDF, to my mind, is that it's structure makes it very difficult to get good performance with truly large numbers of subjects and attributes - and unfortunately that's just the area where it'd be most useful.

<comment><USER2> My claim is that your "generic rules" to govern mapping fields are _actually_ equivalent to hardcoding the names of JSON fields. Instead of seeing "title", and deciding what to do with the data, you see:

        <!DOCTYPE rdf:RDF PUBLIC "-//DUBLIN CORE//DCMES DTD 2002/07/31//EN"
          "http://dublincore.org/documents/2002/07/31/dcmes-xml/dcmes-xml-dtd.dtd">

        <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
          xmlns:dc="http://purl.org/dc/elements/1.1/">

        <dc:title>

... and decide what to do with the data.

In both cases, instead of having a machine understand the (semantic) structure of the data, you have a programmer writing a rule.

Can you provide a concrete example of where this isn't the case, and your RDF- reader is simpler than reading the equivalent (well-designed) JSON of the same data?

<comment><USER4> It's not just consuming the data though... it's when you go beyond that and start doing inference and combining multiple databases, that the RDF approach really shows it's value.

If you established a standard for doing that kind of field name exposure, using JSON, and then sure, you could achieve the same effect. But, in the end, you'd probably just wind up with a JSON encoding of RDF anyway. Define things as subject/predicate/object is all RDF really is... the RDF/XML encoding is just one way of expressing RDF.

<comment><USER2> Hey, I really wish that "subject/predicate/object" was all that RDF was, but I'm afraid it's a good deal more:

RDF Syntax: <http://www.w3.org/TR/2004/REC-rdf-syntax-grammar-20040210/>

RDF Schema: <http://www.w3.org/TR/2004/REC-rdf-schema-20040210/>

RDF Semantics: <http://www.w3.org/TR/2004/REC-rdf-mt-20040210/>

(Those are all current W3C standards)

<comment><USER5> I agree that a lot of the standards surrounding RDF are ugly. I've always particularly disliked the RDF-as-XML serialization, which took two fairly simple ideas (triples and XML) and combined them into a complex mess. This is why I always hate parsing RSS 1.0. Also, the full generality of OWL just confuses me: It seems to be Prolog done badly.

But just as with XML, it's possible to ignore the cruft (XQuery, XLink, XML Schema, the current SOAP flavor-of-the-month), and just use the useful bits. A similar argument could be made about HTML: For every HTML 5, there's an XHTML 2.0.

<comment><USER3> Agreed. You can get all the benefits of RDF while eschewing the stupid parts. Just because something has a spec doesn't mean you have to use it.

The full XML spec, for example, is _insanely_ complicated. But people still derive value from it by utilizing a more or less sane subset.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> [I have a client who sells RDF tools. Here's the latest version of what I've been saying to them.]

Let's look at RDF like a startup: The old RDF marketing from, say, 2003 was hopelessly out-of-touch with reality. Users were never going publish their metadata as RDF, and even if they did, you'd need strong AI to use it. Here are two classic articles spelling out why classic RDF wouldn't work:

<http://www.well.com/~doctorow/metacrap.htm> <http://www.shirky.com/writings/semantic_syllogism.html>

But things have been looking up in the RDF market lately. The complicated RDF XML serialization is mostly ignored in favor of simple n-triples. Google is making heavy use of RDFa metadata when searching for products, and something like 3.5% of web pages now contain RDFa. The RDF conferences are booming. There are cool projects like dbpedia that are organizing publicly-available information as RDF.

So if the RDF tool vendors are going to succeed, they need to pivot (and many of them are). They need to drop the AI hype, and focus on what their early users are telling them. Some possible sales pitches:

1) RDF is useful as a distributed, schema-free graph database. Competition: Neo4J, other NoSQL databases. There's a couple of very good sales pitches here, including the fact that RDF databases are available from multiple vendors, and that RDF inference can be used to normalize schemas between different data sources.

2) RDF is useful for embedding small amounts of data in web pages. Competition: Microformats.

</comment><comment><USER6> RDF/XML serialization of RDF graphs can be painful. I completely agree that other serializations like Turtle should be used in recommendations (e.g. R2RML).

But the RDF model is a wonderful thing. The use of URIs as identifiers for objects and properties makes possible for the first time to reuse knowledge and share data, linking APIs in the same way we are alreay linking web pages.

RDF semantics are maybe harder, but most people can start using RDF without caring about things like entailment.

I really think RDF has a future, specially since the steady growth of the LOD initiative. The revision of the standard is also a good opportunity to polish some aspects of RDF, for example, the use of named graphs.

</comment><comment><USER7> I see a lot of RDF bashing. Particularly from the "Web 2.0" crowd. It has warts, no doubt (is anything borne of the human mind without warts?), but it also has its strengths. A lot of people say it is a failed technology but it's less the failure of the technology and more a failure of the people saying so to properly understand what it can/does do.

Is that a failure of the technology? Because most people don't understand _what_ they would use it for or _how_ they would apply it? I don't think so. I think the claims that it would change the web were high-flown. I also think its creators did a bad job of explaining it. However, you'll find the people that do understand it and have a domain in which it is _clearly_ applicable - love it.

One of my friends works for the library at UCSD and they use RDF, RDFS, and OWL-DL _extensively_ \- I couldn't even imagine doing what she does with the library's book ontology using JSON (as some have proposed replace XML and it's vocabularies, even with a JSON "schema" language). I have another friend working for a biotech company - and he uses it there, extensively. These are only two examples and it excludes the other web projects and companies out there that also use it and it's higher level vocabularies/ontologies (UMBEL, etc...).

Is it a failure for the web? (a topic in another thread a few days ago) I don't think it is, I think it is a failure on the part of developers to understand it and apply it (Drupal has applied it).

</comment><comment><USER8> It was the right idea and almost the right implementation, but almost is a big word -- like a rocket that _almost_ has escape velocity.

There's work being done in this space that is really exciting and I think we'll soon see how much potential the semantic web really has.

</comment><comment><USER9> A major problem with RDF is that it is almost impossible to build applications on top of it. You'd need another layer of abstraction to handle the complexity.

I really like the approach Freebase takes. It's a proprietary format, basically. They use JSON rather than XML and they have their own query language MQL (also expressed using JSON). However their graph of entities maps to RDF as well, so they use RDF (along with common ontologies) as an export format. I think while their system is still complex under the hood, it's less verbose, less scientific, and more user-friendly w.r.t. the public interface. And most important, thanks to MQL it's super easy to build applications on top of it.

\---

For me modeling data as a graph (as RDF proposes) is a really great idea! What I always wanted to do is building client applications (single-page web apps) that can operate on a graph of data directly (instead of talking to a REST service). That's why I'm putting efforts in the creation of Data.js, which features a Data.Graph that can be manipulated in JavaScript environments (like the browser or Node.js). Such Data.Graphs can be persisted (synced) at any time. There's support for CouchDB as a backend.

Well, in the README I also pointed out why I decided not to use RDF and instead took inspiration from the Metaweb Object Model (that Freebase uses).

<https://github.com/michael/data>

I'd enjoy some feedback btw. The lib is actually working, but the examples are out of date. Have a look at the source or ping me if you want to try it out.

</comment><comment><USER10> Hands up anyone who uses RDF (when they could choose to do otherwise)? Anyone? _Bueller?_

<comment><USER11> Plenty of people (including myself) in biomedical fields (especially in bioinformatics, although I know of numerous medically-oriented users) use RDF for a variety of purposes- it's a fabulously flexible way to express and encode complex data models. With a little bit of coordination regarding semantics (OWL, etc.), it's also a great way for people in related but distinct fields to share their data with one another.

That said, RDF has historically had three main problems, IMHO. The first is that its proponents have, historically speaking, done a crappy job of explaining what it actually is, and a worse job of demonstrating what it can do.

The second problem is that it's not really the most human-readable format around, especially in its XML serialization format. This is largely because it wasn't designed for human-readability, but in the age of the Web (where human- readable formats have shown to have major advantages over non-human-readable ones), this is a liability. They've added some more readable serialization formats over the years, but a lot of the documentation and tutorials assume that you'll be working in XML.

When I first tried to learn about RDF, years ago, all I found were tutorials full of really gnarly-looking XML with minimal explanation of what was going on. I spent so much time getting bogged down by the syntax that I missed the point entirely.

The third, and bigger, problem with RDF, is that it's a little hard to "grok" if you don't have a good grounding in predicate logic, description logics, and some of its other theoretical underpinnings. Actually, wait, maybe I said that wrong- I should say instead that most of the old-school RDF people _do_ have solid theoretical backgrounds in description and predicate logic, and have a hard time talking about RDF and other semweb technologies with people who people who _don't_ have that background. So they use lots of jargon that, while useful, isn't very helpful to somebody just trying to get their feet wet.

<comment><USER12> Have you found any good introductions?

<comment><USER11> Personally, I found Antoniou & van Harmelen's "A Semantic Web Primer" to be very useful. I felt like it covered a lot of ground at just the right level- enough to explain what was going on and why, but not so much that it got bogged down in pointless detail. However, YMMV- I know some people who didn't care for it.

It's a little bit dated at this point, but Shelley Powers' "Practical RDF" was also helpful- but, again, I got a lot more out of it once I'd read Antoniou and had internalized the whole RDF thing at a "30,000 foot" level.

BTW, I just Googled for the Antoniou book to make sure I had remembered its authors correctly, and it turns out that one of the first results is a PDF version of the entire text. I don't know what it's doing up online, but grab it while it's hot. If it's helpful, Powell's has a couple of used copies:

<http://www.powells.com/biblio/1-9780262012423-2>

</comment></comment></comment></comment><comment><USER13> Many people here are answering the question of why RDF sucks, but that was not the question asked. The question is why this suckage has still not managed to bury the technology.

There is a very frequent problem people suffer from, which is mistaking _goals_ for results. When you start looking for it, you'll see it a lot. A new open source NoSQL database will pop up, post a long list of goals ("Fastest performance, maintain some integrity, transactions, trivial sharding, consistent available _and_ partition tolerant, and able to run on a TI-83 at web scale!"), put up a benchmark that shows that if you have no features and have no code written to ensure you don't fall down under real load you can put up _way_ bigger numbers than the products with features, and suddenly you have some set of very excited people. Why are they excited? It's not the code; the code is worthless, the _only_ thing it can do is run that benchmark. It's the promises.

You can see it in graphical programming. Graphical programming has a few modest successes, but the _promises_ are about changing how everybody programs and how even Granny will be able to program. The fact that it has never happened despite immense effort for tons of smart people doesn't stop a certain segment of people from still being True Believers.

And, today, we talk about RDF. It promises to organize the web, it promises glorious wonderful search engines, it promises the world. It can't deliver, because merely sticking URIs on some graph nodes is only the beginning of the solution, not even remotely the end, you still have issues of agreement and accuracy and all kinds of other things. But the promise is _so beguiling_ that some people just can't give it up, if we just try harder it'll happen, it's just that nobody has done it right yet, I'm smart enough to see what the previous hundreds of smart people haven't and I'll get it right, oh, it'll be _glorious_ when everybody gets their heads out of their ass and listen to me and just start doing it _right_.

But RDF can't get us there. It's so general it's nothing at all.

There are all kinds of places where people become excessively bedazzled by promises and never notice the concrete reality before them. Another interesting example is Object Orientation. This has proved useful, IMHO, if not the be-all end-all of development methodologies, but it is interesting to contrast the _promises_ made by OO back in, say, the late 1980s, with OO reality today. The promises were about how objects can represent things in the real world and you can model the real world with them. This turned out to be bunk. The real world has some place in OO but only carefully layered and wrapped and mixed in with a lot of other not-real-world things, iterators and factoryfactories and facades and data structs and ORMs and so on. The old promises were interesting and wrong, but also so beguiling that even today you will still hear this nonsense spouted about how this is the purpose of OO, even though it is now well understood that writing your programs with an excessively-strong tie to physical reality is asking for problems. Even as the reality is actually useful the old beguiling promises are still around screwing young developers up to this day.

(It is a tricky balance maintaining the proper level of skepticism because conditions change and sometimes wild promises become practical, and sometimes someone really does manage to pull off one of these things. The latest example would be the commercial success of the iPad, because for a long time smart money was on there being no market for tablets after numerous and repeated failures in creating the market. But in general, "show me the code" or appropriate manifestation is still the best way to avoid being trapped in one of these marketing traps, you will miss out on a few hits but pass on dozens of losers.)

(Also, I am aware there are still some True Believers using RDF. My point here is not disproved by a couple people using it, even using it in a big way. My point will only be disproved if someone brings about RDF Utopia, the actual promises. Of course you can bash RDF into submission, but that doesn't prove it was the _best solution_ for your problem.)

<comment><USER14> Great point. I think another example is XML. It was going to free us from proprietary, binary data formats by making everything nice and understandable pointy brackets. Everyone understands things written in pointy brackets, right?

The reality is that we got OOXML and ODF, which are so enormously complex that no piece of software except Microsoft Word and OpenOffice could hope to _fully_ implement them. If there was an ACID3 test for either it would be fully clear how bad and non-interoperable the situation is. We got XHTML, which turned out to be a horrible idea ( [http://diveintomark.org/archives/2004/01/14/thought_experime...](http://diveintomark.org/archives/2004/01/14/thought_experiment) ) and is now abandoned. We got XML schema, because everyone realized that when you have structured data you want data types, instead of everything being just text.

And because XML seduces you into thinking it's much simpler than it actually is, people write their own parsers and generators all the time that have no hope of knowing what to do with a CDATA section, and you get parsers that will fail if the whitespace isn't just exactly right. The whole thing is a giant farce.

Just like you say, people got excited about the _promise_ of XML. This is the perennial problem when people try to create standards in an area that doesn't actually have any compelling implementations yet. Good standards refine and codify existing practice. Bad standards try to invent something and standardize it at the same time.

<comment><USER13> I would agree. XML is like OO, it's actually still useful in some cases, but if you don't have marked-up textual content you're probably doing it wrong. I wonder what fraction of XML in the world is actually marked-up textual content, rather than data masquerading as marked up text?</comment></comment></comment>


Why Ethiopia is building a space programme - <AUTHOR> http://www.economist.com/news/middle-east-and-africa/21720129-and-why-critics-think-it-odd-use-scarce-resources-why-ethiopia-building?cid1=cust/ddnew/n/n/n/2017044n/owned/n/n/nwl/n/n/na/Daily_Dispatch/email&etear=dailydispatch <comment><USER2> > For Ethiopia, where few scientists have the expertise to make use of the > flood of cheap data, perhaps the best argument for a modest space programme > is that it might help the country develop its human capital. But at a time > when 5.6m Ethiopians need emergency food aid because of a drought, it seems > an odd priority.

What an odd argument. These are the same folks who complain about poverty in India anytime ISRO launches a new rocket. Like every country that has a space program, you cannot wait until you solve all your social ills before you develop your human capital. Investing in science and technology is something that every country should proudly do even as they try to resolve poverty, infrastructure and social issues. They need not and should not be mutually exclusive.

<comment><USER3> Well, they need to be mutually exclusive when you can't even provide the minimum necessary for your citizens to live by.

I can understand a poor - but not totally destitute - country to invest in advanced technology, but no, I can't understand why in a country where thousands every year die or are severely affected by hunger, is investing in space technology instead of helping the immediate needs of its citizens.

It would be like a family in a household were you are about to die of hunger or of some disease by lack of treatment, made their priority to pay your university tuition fees instead.

<comment><USER4> If you focus all the tooling and construction around your space program locally, it doesn't seem any different than the Public Works Administration in the New Deal. I don't know anything about Ethiopia, but if it's just a roundabout way to make work, then it may be more effective at stopping hunger than giving out food.

<comment><USER5> It's different in a sense that public works don't just create jobs, they also create public infrastructure that is then usable by, and materially beneficial to, everyone.

A space program would obviously create jobs, but 1) does it create as many jobs as infrastructure work, per unit of currency spent, and 2) does the output that it produces materially benefit the public?

Basically, it's not a question whether they should spend money on space program, or on handing out free food. It's a question of whether a space program is really the best bang for the buck in terms of creating jobs and other benefits.

And you could argue that it might be, because the space program will produce important _long-term_ benefits. But short-term benefits will produce "interest", so to speak (if you can get more people out of poverty today, you'll get more people who can become scientists tomorrow - and it might not even be the same people, but their kids). So such an argument cannot be a mere hand-wave - it must present the numbers.

</comment></comment></comment></comment><comment><USER6> > But at a time when 5.6m Ethiopians need emergency food aid because of a > drought, it seems an odd priority.

Not that it's the same scale, but U.S. has ~.5M homeless [0], and I'm very happy that NASA still exists.

Saying that a government shouldn't have a space program because people are starving, without first mentioning every other piece of inefficient spending, seems misleading.

And yeah, while it does seem like they could maybe get similar benefits from the private cubesat's mentioned, there are good reasons for developing countries to work towards domestic technology instead of relying on imports.

[0] [http://www.endhomelessness.org/pages/snapshot_of_homelessnes...](http://www.endhomelessness.org/pages/snapshot_of_homelessness)

<comment><USER7> That's a pretty big difference... 6% of the population versus 0.1%. For reference at the peak of the Great Depression homelessness in the US was around 1.5%

<comment><USER6> Yeah, that's why I opened with "Not that it's the same scale".

My point is just that arguing "don't spend money on X, Y is more important" only makes sense if X is literally the least important thing.

But as far as scale is concerned, it's worth noting that the US spends about .5% of its budget on NASA (again, IMO totally justified). The economist article doesn't mention how much Ethiopia plans to spend, but to get a super rough estimate, their current $3m investment [0] is only about .025% of national budget [1].

[0] [http://www.sciencealert.com/ethiopia-has-launched-the- first-...](http://www.sciencealert.com/ethiopia-has-launched-the-first-space- program-in-east-africa) [1] [http://www.ena.gov.et/en/index.php/politics/item/1435-counci...](http://www.ena.gov.et/en/index.php/politics/item/1435-council- of-ministers-approves-over-274-billion-birr-budget-for-upcoming-ethiopian- fiscal-year)

</comment></comment></comment><comment><USER8> > THE ancient holy town of Lalibela, perched some 2,500 metres above sea-level > in Ethiopia's northern highlands, boasts some of the clearest night skies > imaginable.

When I lived in Ethiopia (Haile Selassie was still emperor so it's been a while) there were many nights of the year where the starlight was bright enough you could walk around without a flashlight. Nights with a full moon almost felt like daylight. You could see stars at the horizon.

</comment><comment><USER9> The Ethiopian government has been engaging in pretty brutal repression for the last year or so. Much like the famine in the eighties, this one is also tied up in systematic deprivation of groups outside of the ruling elite.

[https://www.theguardian.com/global- development/2016/oct/20/s...](https://www.theguardian.com/global- development/2016/oct/20/state-of-emergency-likely-ramp-up-repression- fractured-ethiopia)

</comment><comment><USER6> The article fails to mention that U.S./Global Fund/Gates Foundation are willing to spend exorbitant amounts of their own money on Ethiopia's humanitarian crises [0][1][2], so there's a pretty strong incentive for Ethiopia to spend their own money on other things.

To be clear, I'm not at all against foreign aid.

[0][https://ethiopia.usembassy.gov/u.s.-response-to-the- ethiopia...](https://ethiopia.usembassy.gov/u.s.-response-to-the-ethiopian- drought.html) [1] [http://www.gatesfoundation.org/How-We-Work/Quick- Links/Grant...](http://www.gatesfoundation.org/How-We-Work/Quick-Links/Grants- Database#q/k=Ethiopia) [2] [https://www.theglobalfund.org/en/portfolio/](https://www.theglobalfund.org/en/portfolio/)

</comment><comment><USER10> Not too long ago China helped Nigeria build and launch a satellite. For a while, the thing was working until it fell out of the sky. That was the end of that one. Not sure if they managed to relaunch another one or got a refund from the Chinese.

I hope the Ethiopians fare better than the Nigerians.

<comment><USER11> According to this website [1], "China Aerospace Science and Technology Corporation (CASC) and its relevant organizations reached a decision to build and launch NIGCOMSAT-1R with no extra cost to Nigerian customer".

[1] [https://www.nasaspaceflight.com/2011/12/chinese-long- march-3...](https://www.nasaspaceflight.com/2011/12/chinese-long-march-3be- launches-nigcomsat-1r/)

</comment></comment><comment><USER12> > ...but at a time when 5.6m Ethiopians need emergency food aid because of a > drought, it seems an odd priority

As other commenters mentioned it's an odd argument - putting it into perspective I understand the writers point

It's not just drought that's crippling the country; huge youth unemployment, wealth [little of it] distribution & the government itself are some of the major issues

With the state of emergency being extended for another 4 months & Internet censorship at all time high - space program is an unlikely prioritized endeavor imo

PS: I am an Ethiopian

</comment><comment><USER13> Alternate Synopsis: For their military program. Huge country, you need satellite info, still at war, parts receded, dangerous borders.</comment>


Which is the fastest web framework? - <AUTHOR> https://github.com/tbrand/which_is_the_fastest <comment><USER2> What is crystal? And where are Java & C++? Also, who the hell writes web frameworks in Objective-C?

Also... does anyone notice that this test shows that the web framework hosted under the same github account is the fastest.

edit: re: "What is crystal?"

[https://github.com/crystal-lang/crystal](https://github.com/crystal- lang/crystal) ... I will say, I'm intrigued. I do love ruby, but unfortunately also hate dynamic typing. Crystal seems interesting.

<comment><USER3> Crystal is a relatively newer language aimed to combine Ruby's syntax with the speed of C per it's project page [1].

[1]: [https://crystal-lang.org/](https://crystal-lang.org/)

</comment></comment><comment><USER4> Doesn't look like the test is warming up the web server. It sleeps 10 seconds (I assume) after starting the server but it might just as well fire some warmup requests.

[https://github.com/tbrand/which_is_the_fastest/blob/master/t...](https://github.com/tbrand/which_is_the_fastest/blob/master/tools/src/benchmarker.cr#L137)

</comment><comment><USER5> This is measuring startup time, not actual request time. This seems nearly worthless unless you are looking to host something on AWS Lambda? Does the author honestly believe GET requests to rails sites take 2 and a half minutes on average?

<comment><USER4> The client is sending 5000 requests * 16 threads from what I can see. Still seems like a useless microbenchmark.

</comment></comment><comment><USER6> I think PHP is still the most popular web app language, a shame it's missing out having a couple of it's frameworks included in the test.

<comment> forgot-my-pw Can already guess they'll be at the bottom. Laravel, Symfony are pretty dang slow as frameworks.

<comment><USER7> Also without testing them to get the stats to prove/disprove my statement I'm going to say the exact opposite.

Can already guess they'll be at the top. Laravel, Symfony are pretty dang fast as frameworks!

<comment> forgot-my-pw They're already among the slowest in PHP:

\- [https://github.com/kenjis/php-framework-benchmark#hello- worl...](https://github.com/kenjis/php-framework-benchmark#hello-world- benchmark)

\- [https://medium.com/@asked_io/php-mvc-framework- showdown-7-1-...](https://medium.com/@asked_io/php-mvc-framework- showdown-7-1-performance-2da52ac9fcba)

</comment></comment></comment></comment><comment><USER8> Is the term "framework" really the best descriptor for this type of chunks of code?

</comment><comment><USER9> I understand wanting to show off your latest project, but comparing a large framework like Rails to your couple hundred lines of crystal is not an honest comparison, not even taking into account the concerns of startup time as others mentioned.

</comment><comment><USER10> Is this essentially this again?

[https://www.techempower.com/benchmarks/](https://www.techempower.com/benchmarks/)

They appear to have slowed down in the frequency of test rounds though.

</comment><comment><USER11> For Rust, this benchmark tests Nickel, Iron, and Rocket; all of which are synchronous. Async really helps here!

<comment><USER12> Why? I looked at the code, it doesn't really do anything with the request, shouldn't be blocking any threads or anything.

<comment><USER11> It's not in the program itself, it's in the frameworks. That is, the way the frameworks handle the HTTP requests uses blocking IO.

There are other frameworks that are non-blocking; they just didn't include them. To be fair, Iron and Nickel are the old guard of frameworks, and Rocket is one of the new, hyped ones, but they don't use any async stuff. This isn't so much blaming the benchmark as it is adding context!

</comment></comment></comment><comment><USER13> How important is speed really? It's not like it's the only parameter you can measure a framework by, just happens to be the only quantifiable one.

<comment><USER14> It's not even really that quantifiable, since a framework can be faster at some tests and slower at others.</comment></comment>


A New Mathematics for Computing - <AUTHOR> http://h30565.www3.hp.com/t5/Feature-Articles/A-New-Mathematics-for-Computing/ba-p/1390

<comment><USER2> This sounds like an interesting topic, but the article itself was more confusing than enlightening to me.

It seems that the actual explanation (using Hamming distances) could be used instead of the bubble-wrap analogy (in the same amount of space, without making more assumptions about the reader). I felt it didn't represent the trade-off (or rather the strict improvement in this case) very well. In fact, it seems to suggest something that is false (that the two methods are fundamentally different).

They also start using graphs without an (informal) definition.

I didn't know about tree codes before so this could have been interesting but I still don't know much about them. The article alludes to some kind of uniqueness theorem

 _but remarkably Leonard showed there is actually one out there that's useful_

but the end suggests that we do not actually know the optimal strategy (so I guess its just an existence proof?).

 _a set of structured binary strings, in which the metric space looks like a tree,_

doesn't tell me much either. How should I interpret "look like"? Do I approximately embed the space in R^n? Do they mean that its close to a tree metric?

Finally, the article also didn't mention how little we actually know about, say, the Shannon capacity of many (small) fixed graphs. The impression I got is that we already know all there is to know about "classical" Shannon capacity (which I believe is false).

</comment><comment><USER3> This article is light on the details. Here is the paper it's based on (I think):

[http://users.cms.caltech.edu/~schulman/Papers/intercoding.pd...](http://users.cms.caltech.edu/~schulman/Papers/intercoding.pdf)

<comment><USER4> I see the author is from the Imposing Wall of Text school of typesetting.

<comment><USER5> It's a paper about an abstract mathematical topic. Just be glad that the whole thing doesn't consist mostly of equations.

</comment></comment></comment><comment><USER6> Thank you all for your thoughtful comments. You've hit the heart of the challenge in any kind of science writing: Trying not to frighten off those unfamiliar with a field and therefore using metaphors, such as bubble wrap, while at least touching on some common ground for those who are (Hamming distances).

My respectful suggestion for those truly interested in following up is to contact either Leonard or Amit directly. Of necessity, any math that is translated into prose is going to be imprecise, and thus, unsatisfying.

To answer a few of the specific questions that were brought up:

Yes, it is essentially an existence proof. At least for now. Yes, that paper is a good starting point. Also take a look at Amit's work. Yes, buses inside chips is something they think it could be useful for. (Sigh) I wish there were a better introduction to tree codes at a lay level. Believe me, I tried very hard to find one. FWIW, both scientists vetted the explanations.

<comment><USER6> I got some further clarifications from Leonard Schulman:

He writes, "Yes, a tree code is a near-isomorphic image of a tree metric. No, this has nothing to do with the Shannon capacity of graphs."

Thank you again for your comments.

</comment></comment><comment><USER7> There has to be a better introductory to tree codes than this. One that doesn't assume you work in error correcting codes but can actually explain things in terms of basic math and information entropy.

</comment><comment><USER8> So, the advantage of this scheme seems to be that you don't have to wait for the end of a transmission block before doing error correction and extracting your data. But, we have error correction schemes which work well with fairly small blocks of data. It seems to me that this isn't a saving in network comms, where the delay for a few hundred bit-times is outweighed by other delays. Maybe it will be useful for buses inside chips.</comment>


Freeciv-web is back online - <AUTHOR> http://play.freecivweb.org <comment><AUTHOR> Freeciv-web is back online, after a peroid without a server.

[https://github.com/freeciv/freeciv-web](https://github.com/freeciv/freeciv- web)</comment>


Open Ventilator Registry - <AUTHOR> https://github.com/openventilatorregistry/OpenVentilatorRegistry <comment><USER2> If you are building vents, please consider submitting your plan to the DoD:

[https://vulcan- sof.com/login/ng2/submission?collectionUuid=2...](https://vulcan- sof.com/login/ng2/submission?collectionUuid=24028d51-267f-4f10-a455-230a13055c12)

<comment><USER3> What is Vulcan and how is their elaborate signup/info requirement justified?

<comment><USER2> It's a marketplace for innovators and special operating forces (SOF) who have requirements, a place where the parties can resolve their mismatches. The signup process has to strongly credential the various types of users with legally binding access controls.

</comment></comment></comment><comment><USER4> In practice it's much easier to move the patient to the ventilator than vice versa, and that's exactly what happens, at least in my country

<comment><USER5> So for most of the rest of the world, moving sick patients across political boundaries is going to be a hot potato nobody will touch. Autoclaving some medical equipment and dropping it off somewhere else is probably much easier.

And then there are the problems of the ridiculously huge countries, like China, the US, or Brazil, that most of the rest of the world simply fail to fathom the enormity of distance-based logistical problems. Two of the largest cities in just the contiguous United States are 4 and a half hours apart _at the speed of sound_ , and there's a bigass mountain range in the middle that you can't fly around. And it looks like a medical helicopter would take about 3 times as long, so moving sick people is a multiple of moving equipment. Even New York City to Chicago would handily kill a patient drowning in their own fluids before they arrived, and most first world countries are wider across than that.

</comment></comment><comment><USER6> I know that "hosting and much more is still TBD," but I can contribute a pretty snappy VM on a 1G symmetrical Internet/Internet2 connection. How do I do that?

<comment><AUTHOR> You can request to join by opening an issue here: [https://github.com/openventilatorregistry/OpenVentilatorRegi...](https://github.com/openventilatorregistry/OpenVentilatorRegistry)

</comment></comment><comment><USER7> Not sure this will work in practise. What hospital administrator will want to send away their limited supply of ventilators in the current environment?

<comment><AUTHOR> Original idea: [https://www.nytimes.com/2020/03/22/opinion/health/ventilator...](https://www.nytimes.com/2020/03/22/opinion/health/ventilator- shortage-coronavirus-solution.html)

During a crisis like this, the idea is to pool resources and create a nimble distribution network. Hospitals that don't need their ventilators can share them, and if they need them they'll know where to request them.

<comment><USER8> The problem is every hospital is forecasting a dire need for ventilators in the next few weeks. COVID patients can take up a vent for nearly 3 weeks. Any vent that's sent to another hospital right now will likely be gone for the entire duration of this outbreak.

Anecdotally, my wife's hospital, in a rural county with 1 case, is already at ICU capacity and pushing the limit on vents.

<comment><AUTHOR> Agreed this seems problematic and a significant challenge. Hopefully as the shock has worn off and a clearer picture emerges in the coming months there will be greater willingness to cooperate.

</comment></comment></comment></comment><comment><USER9> I feel like this is just going to end up being a website with no usage because no one has spare ventilators. But I really hope they prove me wrong.

<comment><USER10> First of all, big shout out to the Open Ventilation Registry crew! The registry will be quite useful.

There's an a small global army of biomedical engineers, ICU doctors, programmers etc. working very hard to have rapidly manufactured ventilators designed, approved, and manufactured. I've seen some of the designs and they're elegant, simple, just work. Yeah, they need fine tuning but that's part of the process.

Humans are insanely awesome when they collaborate.

Hang in there humanity.

</comment></comment><comment><USER11> Do you plan on this being a registry for US only, or you are planning to have this as an international registry?

</comment><comment><USER12> Do you have any experience in supply chain logistics, the medical industry, incident response?

There are already established systems. How will yours interface with the existing / international norms?

</comment><comment><USER13> I was trying something but found it difficult to pull through. If this can help someone. Had some specs in place [https://github.com/covid19-india/supplychain/wiki/Software-R...](https://github.com/covid19-india/supplychain/wiki/Software- Requirement-Specifications)

</comment><comment><USER14> So what's the administrative process of transferring a ventilator from one hospital to another like? Is there any precedent for this? I could see this kind of thing working in a socialized healthcare system but in the US wouldn't it create all kinds of liability and insurance problems?

</comment><comment><USER15> According to the Society of Critical Care Medicine, the United States has more than 200,000 available ventilators. However we have less than 50,000 staff trained in mechanical ventilation that can support a maximum of 125,000 patients. Furthermore, we only have 90,000 ICU beds to put patients in.

Yes the media is telling you there is a shortage, they don't understand that the problem isn't a lack of machines.

<comment><USER16> 50k staff and that can only support 125k patients?

So you are saying a person trained on the use of a ventilator can only support a little over 2 patients on a ventilator? Are they sitting between the beds and pedaling the machine?

These are $30k machines. I would think, especially in a crisis situation, a single medical professional could probably handle a few dozen people on ventilators if they had too.

<comment><USER15> > So you are saying a person trained on the use of a ventilator can only > support a little over 2 patients on a ventilator?

As I mentioned in the comment you replied to, the numbers are from the SCCM. You aren't going to find a more authoritative source for accurate information.

> I would think, especially in a crisis situation, a single medical > professional could probably handle a few dozen people on ventilators if they > had too

If you don't believe the SCCM, California law and the California Respiratory Care Board set the requirement as "a ratio of 1:4 or fewer each shift."

Short of a Cardiopulmonary Bypass Machine (basically a mechanical heart), invasive ventilation is one of the most intensive "intensive care" treatments, requiring constant monitoring and adjustment.

<comment><USER16> Im sure 1 to 4 is best case and best practice. In an emergency, I guarantee someone can and will handle more.

Additionally, the 1 to 4 ratio doesn't match with the 50k number.</comment></comment></comment></comment>


3 books every development manager should read - <AUTHOR> http://www.codovation.com/2011/09/3-books-for-every-development-manager/ Often, especially when small companies grow, good developers get promoted into leadership or management positions: Scrum Master, Development Team Lead, Development Director, or even CTO. I lately read three different books: Peopleware, Management 3.0, and Growing Software. I want to share this great and powerful introductions to development management with you. <comment><USER2> "Slack" is also a great read: [http://www.amazon.com/Slack-Getting-Burnout- Busywork-Efficie...](http://www.amazon.com/Slack-Getting-Burnout-Busywork- Efficiency/dp/0767907698)

</comment><comment><USER3> I'd add "Managing Humans" by Rands to the list.</comment>


Why You Should Disassemble Your Cheap Charger Before Use - <AUTHOR> http://www.pocketables.com/2013/01/this-is-why-you-should-always-disassemble-cheap-chinese-charging-accessories-before-use.html <comment><USER2> But author didn't test it :( It's entirely possible that blob of solder is for mechanical, not electrical purposes.

The author's complaint about the pins being too close together on the cable end seems silly to me. If you have a good feeling for how solder works (and how it sticks to pads and avoids solder mask), you can tell by eye that those pins will be OK.

<comment><USER3> Lol this is silly. >and when I opened it up to see inside, proof that I'm not paranoid was glaring back at me.

Or its proof that he's both paranoid AND lazy :)

</comment></comment><comment><USER4> It's lame to write up a post accusing seller of selling device w/ shorted-out pins, but not even take the time to confirm the accusations with an ohmmeter.

</comment><comment><USER5> I find very unlikely it doesn't work.

Shonky electronics are possible out of China but 100% not working electronics are rare. It's evolution.

The business would quickly go out of sale, so assuming you're buying from a reputable seller it's unlikely you get a total scam.

So to make a big call like that you really need more proof.

<comment><AUTHOR> I think you misunderstand the point. The shoddy wiring would work, while at the same time causing a short circuit, potentially destroying your device.</comment></comment>


Narratives - <AUTHOR> http://words.steveklabnik.com/narratives

<comment><USER4> First, let me claim that I am unwillfully ignorant of how to begin to treat these balancing issues - so take it easy in my response. I think that articles like this (the tone is a bit flame bait) actually are part of this problem.

We don't need to turn this issue into what race has become in America - <USER2> white people are scared to ask each other - let alone a black friend - <USER3> the issue in a rational way. It's fucked up a lot of good intentions, and replaced them with empty PC platitudes and non/empty-actions that help few.

I say I'm unwillingly ignorant because I've sought out everyone's opinion about this I can find: from guys/girls in and out of tech. I still don't know what the stance is with regards to such issues. All I know is that of course the ratio is bad, culture needs to change, etc. - we all know that.

I'm not looking for answers - I'm even before-that, I want the problem further defined. I bet these guys were a lot like me: not thinking actively about anything but trying to do something cool. They didn't cover their bases with regards to this problem - this is actually part of the problem. We need to not demonize these people, we need to find better ways of helping guys like this not miss these kinds of things.

Am I unreasonable?</comment>


What's the easiest toolchain to build a self-hosted "Netflix"? - <AUTHOR> There are times when I need to keep my daughter entertained with a screen but I don't want her to watch garbage. I have about 10TB of documentaries and educational videos and want to create a Netflix-like experience so she can watch what she wants, but only from the catalog I've created for her. What are the best tools here? <comment><USER2> Great idea, posting to remember it (for use on younger sibling who spends too much time on the web)!

</comment><comment><USER3> Maybe Kodi, you can customize skin so it shows only library you created, has lots of skins, can look pretty,.. It can scrape all the info from imdb or similar sites, so every movie can have cover art, description, rating,.. Once created library can be browsed by looking on cover art pictures listed by name, year, rating, director, genre,.. supports continuing where you previously left, bookmarks, can automatically hide already watched content,.. etc it has extensive plugin support, lots of official and unofficial addons for streaming stuff, trailers, audio, youtube, twitch, vimeo, program addons even some rom games addons,.. can be complicated or simple as you want,.. but for this use case you won't need any plugins, maybe a different skin, something your daughter would like. lots of tutorials on kodi wiki, forums and youtube works on all platforms, free and open source

</comment><comment><USER4> [https://mediagoblin.org/](https://mediagoblin.org/) might be of interest to you.

There's a bigger list here: [https://github.com/Kickball/awesome- selfhosted#media-streami...](https://github.com/Kickball/awesome- selfhosted#media-streaming)</comment>


Ask HN: How do you choose your web hosting? - <AUTHOR> Given the fact that most of the top web hosting companies have almost the same pros & cons, how do you now decide to choose your web hosting?<p>If you can share what you are using right now and why you've decided to use it. That would be helpful. Thanks. <comment><USER2> Day job - Rackspace. They were using it before I got here. They hired me because of my experience with it. We've considered moving to AWS but the time (developers time and mine) and expense to move didn't work out.

Side project - Rackspace. I've used Rackspace on various projects over the years and have grown to like it. More robust then Digital Ocean and not as complex to manage as AWS. It fits right in the middle IMO. For me, it's just enough. Spin up new servers relatively quickly; a CDN, a load balancer, cloud DB, Mailgun proximity - it has everything a growing site needs again, without all the complexity of AWS.

Freelance projects that find me on their own - Digital Ocean. Everything a mom and pop shop needs without the costs of Rackspace. Or 1and1.com - can be slightly cheaper then DO.

Freelance projects given to me by freelance companies - whatever they use. A Small Orange is one, one has used often. Most everyone runs Plesk or some variant of it so usually never a big deal to get up and running quickly.

<comment><AUTHOR> Thanks. :)

</comment></comment><comment><USER3> When I need raw performance: ovh.com .

When the client doesn't care about $$: aws/heroku.

When I don't need raw performance: [https://www.webfaction.com/?aid=9281](https://www.webfaction.com/?aid=9281) (the best/unique vps/shared hosting available)

<comment><AUTHOR> Thanks :)</comment></comment>


Facebook faces another privacy issues - <AUTHOR> https://saidit.net/s/SocialMedia/comments/5fc9/another_privacy_issue_of_facebook_is_there_any/ <comment><AUTHOR> Is it possible that FB will ever fix its bugs and will stop stealing user's data?</comment>


Reference Sheet of Excel Shortcuts (PDF/XLSX) - <AUTHOR> <comment><USER2> Wheres the find and replace shortcut?

<comment><USER3> Its on there. Control + H

</comment></comment><comment><USER4> This is a good one thanks!

<comment><AUTHOR> You're welcome.</comment></comment>


Israeli firm can steal phone data in seconds - <AUTHOR> http://phys.org/news/2016-11-israeli-firm-seconds.html <comment><USER2> "But privacy and rights activists worry such powerful technology can wind up in the wrong hands, leading to abuses."

Am I to believe that this firm is the right hands? Or government? Please...all hands are the wrong hands. These vulnerabilities need to be closed. I wouldn't be surprised if the NSA or some other government tentacle was paying them not to make whatever they found known.

Gee, I sound paranoid. What am I thinking, our government would never do that. Oh wait... [http://www.reuters.com/article/us-usa-security-rsa- idUSBRE9B...](http://www.reuters.com/article/us-usa-security-rsa- idUSBRE9BJ1C220131220)

<comment><USER3> Actually Cellebrite themselves have been victim of some kind of hack last month and had a load of their internal documents leaked online. So no, if any, this firm is _not_ the right hands.

<comment><USER4> Any place which hoards 0-days is a prime target. Even if they are considered to be the "right hands", the "wrong hands" could grab those exploits eventually.

<comment><USER5> When the target holds a lot of $1 million dollar 0-day exploits, the target is worth hacking into.

</comment></comment></comment></comment><comment><USER6> "Could you do anything to deprive them from throwing a stone at someone or from driving a car and running over people?

"You can't blame the car manufacturer at that point for delivering a car that was utilised to commit that kind of crime," he said.

This is specious reasoning. The point of a car is not to run over people; it's to go from point A to point B. This technology, on the other hand, has only one purpose: to break into cellphones.

<comment><USER7> I think it's an important distinction. The crime is in how it's used, not in the technology itself. Banning the tech outright only affects actors who are willing to follow the law. So banning this tool would mostly limit white hats.

The same principle applies to gun rights for example. The gun itself isn't the problem, it's how someone chooses to use it. (self defense vs crime). Nobody is an absolutist here, everyone draws the line somewhere slightly different.

Outlawing the tech ensures that good guys lose the arms race every time.

When I say good guys I don't necessarily mean the government. I mean anyone out there who is not using the tech for malicious intent. People using guns for self defense, breaking into an encrypted device to solve a crime, or retrieve lost work etc.

This keeps the focus on improving the actual technology (encryption). Rather than just banning law abiding citizens from taking part.

</comment></comment><comment><USER8> Regarding the iPhone 5c, the attack from Sergei Skorobogatov is very interesting.

"The bumpy road towards iPhone 5c NAND mirroring" \- [https://arxiv.org/abs/1609.04327](https://arxiv.org/abs/1609.04327)

And the video:

[https://www.youtube.com/watch?v=tM66GWrwbsY](https://www.youtube.com/watch?v=tM66GWrwbsY)

</comment><comment><USER9> It goes without saying - don't make this easy for them(or anyone). Use a _strong_ alphanumeric password on your mobile devices. It's annoying and inconvenient until it saves your ass - there is still no "fast" way to crack a password like "My 42nd spaceship had 4 hearts of gold.", but it's not that difficult for your brain to remember.

Fingerprint unlock can save you some of the PITA of typing it - just be sure you _power off_ your device when you have even the slightest chance of encountering an actor that could seize your mobile device - that way the passphrase will be required.

<comment><USER10> Another interesting point they mention is "recovering years long deleted texts." Consider the filesystem your phone uses for volumes it's writing data to and how it (probably just) unlinks files when deleted... [edit] and I should add a 'factory reset' will probably just write a new filesystem table over the old on disk without wiping anything on most devices

<comment><USER11> What about doing a factory reset, then use the 'encrypt device' option, then doing another factory reset. Would that provide an extra measure?

Unless, of course, the data in it's final state before the final factory reset is un-encrypted.

<comment><USER10> Of course it depends on the phone. Many qualcomm based Android devices seem to have had their Full Disk Encryption scheme broken at the moment [1], for example. But either way, if your disk encryption scheme doesn't fully wipe the disk before use, then any data that _ever_ hit the disk unencrypted could still possibly be sitting there until its physical blocks are consumed (or could be in a cache somewhere). Again this also depends on the hardware, in this case the storage hardware itself. SSDs using TRIM can wipe unused, unencrypted blocks for example in some scenarios. Who knows about the particular functioning of some SD type controller in a phone or even the card's own embedded OS [2].

I would say if you have data on your smartphone you don't want recoverable at rest, take module0000's advice, then also use encryption, and then also use a multi-pass wipe tool on particluar files. Of course all of this could still not work.

For example, I'm not sure what the forensic ramifications of a seemingly more complex filesystem like APFS will be in the near future when it hits iOS.

[1] [https://bits-please.blogspot.com/2016/06/extracting- qualcomm...](https://bits-please.blogspot.com/2016/06/extracting-qualcomms- keymaster-keys.html)

[2] [https://www.bunniestudios.com/blog/?p=3554](https://www.bunniestudios.com/blog/?p=3554)

</comment></comment></comment></comment><comment><USER12> This is completely off-topic, but can we take a moment and recognize how fantastic it is that the article has a picture of a hacker's desk with an assortment of mobile devices like: a calculator, 3 bluetooth mice, and a stapler?!

</comment><comment><USER13> It will be interesting if Apple went after Cellebrite under the DMCA anti- circumvention clauses. I would laugh if their product became illegal in the United States.

<comment><USER14> Probably not, it's an Israeli firm.

<comment><USER15> Any sales to US firms could be curtailed or punished.

Just like security itself - the goal is to provide enough barriers so predators go looking elsewhere for easier prey.

<comment><USER16> > Any sales to US firms could be curtailed or punished.

That doesn't really restrict the NSA, CIA, FBI or any other agency/PD from buying their services. After all, that's what "black budgets" are for.

<comment><USER15> Most of these firms use private contractors to do the dirty work. Unless contractors can bypass legal oversight, they'll be constrained as well.

<comment><USER17> Overseas contractor hired by a shell corporation gets the phone sent outside of the US. Boom.

All this assuming they even _care_ about legal compliance, which I am sure that under the correct circumstances it won't matter one bit.

</comment></comment></comment></comment></comment></comment><comment><USER13> So we have learned that some phone vendors give Cellebrite their phones before they reach market in order for them to discover and exploit vulnerabilities. Apple refuses to do business with these 'forensic' criminals.

Do not purchase a phone from a vendor that engages in this unethical practise.

<comment><USER18> Cellebrite got its start with the UME, a phone memory transfer tool for carriers' (POS and support). Carrier-oriented tools are a major part of its operations, though there have been some talks about spinning this off to a separate company.

Cellebrite gets early access to phones NOT due to its forensics operations, but for UME, since carriers (and that's lots(!) of carriers worldwide) are very much interested in good consumer experience on the devices' launch day.

I actually doubt it's been particularly significant to its forensics operations.

</comment></comment><comment><USER15> You'd think if they could crack the latest iPhone/iOS they'd crow about it.

The article seems to paint it as a "we're confident we could" \- which seems bizarrely vague. Why would they do that when they claim they can crack an LG G4 wide open?

<comment><USER19> From the article:

> _Ben-Peretz remains confident his company can crack even the newest > iPhones._

<comment><USER20> The next line in the article:

> _iOS devices have strong security mechanisms that give us a challenge, but > if anyone can address this challenge and provide a solution to law > enforcement, it is Cellebrite, " he said, referring to Apple's operating > system._

This makes it sounds like Cellebrite actually cannot currently crack the latest phones running iOS 10, but the CEO is merely expressing his belief that they'll figure out how to do it. See how he's not saying "we can do it", but instead he's saying "if anyone can do it, it's going to be us".

</comment></comment></comment><comment><USER21> I wonder, if Ben-Peretz and his _checks_ 250 researcher team can crack _checks_ 150 phones a month what is stopping the <scary US government agency>/<Chinese equivalent>/<Russian equivalent> from forming a 2,500 researcher team and doing the same?

It's not like there's a shortage of relevant skills in the US (supposedly responsible for stuxnet) or Russia.

Or is it just that <scary government agency> doesn't want to share its toys with <local police>?

<comment><USER16> Ah well, with Mediatek based phones it's pretty easy - you can readback the whole storage once you have its partition map from a rooted device, or you know the size of its flash chip and figure out the partition bounds later.

Dunno about the situation with other phones but given that many cheap Androids run Mediatek, it's not very difficult to claim a huge number of "crackable phones".

The only thing that should protect you from any kind of government snoops is encrypting your phone with a strong passphrase and shutting it off once you leave a room taking the cops less than 30 seconds to enter.

</comment></comment><comment><USER22> Does encryption defeat this? If not how are they getting the key from memory?

Does encryption defeat this when the device is off? If not what flaws exist in the encryption schemes?

<comment><USER23> Encryption could help but eventually exploits can beat anything.

<comment><USER22> Exploits are just ways to leverage those flaws. We have systems with no flaws for some older tech. Eventually we will fix this one too.

</comment></comment></comment><comment><USER24> On a similar note, back when cell phones were just getting started, I started parsing the ownership of the cell towers. I noticed an unusually high correlation of Israeli companies owning them (back before the phone companies themselves really started investing in them). Now think about the purpose of an imsi catcher/stingray. The Israelis seem to be on the edge of cybersecurity across the board. I know while in Iraq I got lots of training that was decidely sourced from Israel too.

All that being said, Israel is also known as being just as active if not moreso than Russia and China in their espionage against the US. I think that's also worth considering.

<comment><USER25> Any specific resources/links re: that final comment? Not something I've ever read about, although I could imagine.. Not discrediting your statement, genuinely curious.

<comment><USER24> Just a quick duck duck go search returns these:

[http://foreignpolicy.com/2015/03/24/spy_vs_spy_america_and_i...](http://foreignpolicy.com/2015/03/24/spy_vs_spy_america_and_israel_edition/)

[http://www.newsweek.com/2014/05/16/israel-wont-stop- spying-u...](http://www.newsweek.com/2014/05/16/israel-wont-stop-spying- us-249757.html)

[http://www.counterpunch.org/2009/03/12/israeli-spying-in- the...](http://www.counterpunch.org/2009/03/12/israeli-spying-in-the-united- states/)

[http://www.timesofisrael.com/new-nsa-document-highlights- isr...](http://www.timesofisrael.com/new-nsa-document-highlights-israeli- espionage-in-us/)

<comment><USER25> Cheers!

</comment></comment></comment></comment><comment> SG- Confused why they didn't demo it breaking a modern iPhone instead of a random Android device.

<comment><USER26> Probably because they couldn't? Far easier to show of a random android handset.

<comment><USER27> They probably chose the device they have the best success cracking. I wonder what phone they used ... checks ... LG G4 ... damn it, that's my phone.

</comment></comment></comment><comment><USER28> > Among the data the firm claims to be able to access are text messages > deleted years previously.

Among all the claims this one seems like it might be one that holds up with very recent iOS/Android releases. It would be interesting to find out whether they rely solely on the encryption to protect the deleted messages and whether overwriting the data would be thwarted by flash device wear-leveling indirection.

</comment><comment><USER29> Sounds like an Apple advertisment.

<comment><USER30> And the good kind of advertising, because it's accurate. Apple did the hard engineering work across their entire software and hardware stack because they knew it was important.

They didn't settle for _good enough_.

They didn't weasel out of it by blaming "ease of use" concerns.

They didn't argue that maximum security wasn't a high priority for their customers.

They didn't deploy marketing slogans to pretend like they had done the hard engineering work. ( _cough_ Knox _cough_ )

</comment></comment><comment><USER19> Does anyone track the quantity of U.S. tax dollars which go to such firms?

</comment><comment><USER31> PR based self-marketing article is just that. "Be Very Afraid" hyperbole AFAICS.

</comment><comment><USER32> They [the government] ask people to login to their email accounts and unlock their phones at the border. They really don't even need any technology to steal data. Intimidation works for them already.

<comment><USER33> Then we just need a phone inside a phone right? Provide them with a fake password that shows no data, has 1 kids game app, and gives you plausible deniability. If coded correctly would be no way to prove if there is or is not a second deeper encrypted device.

<comment><USER34> Backup your phone. Wipe your phone. Cross the border. Restore your phone.

<comment><USER32> Or don't take your phone. On the other hand they've asked people to login to their email accounts using laptops they've provided.

<comment><USER35> Same thing for your laptop. Back everything up to the cloud, then load a clean copy of the OS before crossing the border.

</comment></comment></comment></comment></comment><comment><USER36> And somehow I can't connect my android phone to a mac computer with a USB cable to copy photos out, without it crashing 50% of the time. I must be missing something

</comment><comment><USER37> Sounds like a risky business to be in. If Apple decides to change their encryption technology, you could be out of business some time soon after a new release.

<comment><USER38> Your mistaken, looking at the forensic side of things burner phones are not your regular iphone or android.

E.g random article from bestbuy.

[http://www.bestbuy.com/site/at-t-gophone-lg-b470-prepaid- cel...](http://www.bestbuy.com/site/at-t-gophone-lg-b470-prepaid-cell-phone- black/5444000.p?skuId=5444000)

</comment></comment><comment><USER39> I wonder if they can crack devices that haven't been booted. Many of the newer smartphones encrypt data and require a password on boot.

<comment><USER40> Too bad Google made the boot password the same as the screen unlock password. Since virtually everyone wants to be able to quickly unlock their phone, this makes security a Hobson's choice.

<comment><USER39> Yeah I can see that being the case for the vast majority of users. Also it's a damned shame that Google enforces a limit of 16 characters for the password.

My own password is a random 16 character string.

<comment><USER41> Fortunately the community has addressed both these claims, although you need root to set it up (you can remove after).

An app on F-droid known as "Cryptfs Password" can change the encryption password separately from your screen unlock password. It also bypasses the 16 character limit, as the encryption key I used on my last phone was 27 characters. At the end of the day Android encryption runs using dm-crypt, so the same sort of rules apply. The 16 character limit is a UI limitation, and there's no technical reason for it.

* Note: I fully acknowledge that Google needs to do better here, as I would never assume a normal user could root + install Cryptfs password + unroot after, but at least for those of us who can, we can do something in the meantime.

</comment></comment></comment></comment><comment><USER42> I mean, with Trump's administration banning cybersecurity and encryption, this company should go out of business soon, right?

<comment><USER43> it would be somewhat hard to ban math, right?

Take Trump's mouth blast with a grain of salt, as its not what he will end up doing once he learns what he is talking about.

</comment></comment><comment><USER44> Aren't they owned by a Japanese pachinko company?</comment>


DCHQ – Cloud Automation. Ridiculously Flexible. Reduce Costs. Access Controls - <AUTHOR> https://www.dchq.io/ <comment><USER2> I can make a web app, but know nothing about deployment. I need to use AWS (Elastic Beanstalk I guess) or another major service because of my location. No Digital Ocean, Linode, Heroku.

Will this take me from zero to hero?

(Could do a MEAN stack but want to use Clojure/JVM.)</comment>


Garbage Collection vs. malloc/free - <AUTHOR> https://groups.google.com/forum/#!topic/mechanical-sympathy/1TMjVjyyMmA <comment><USER2> Java's garbage collection is far from the state of the art in GC technology. While there are some non-standard JVMs that have advanced GC (such as IBM's real-time garbage collector), the standard JVM does not.

Current-generation garbage collectors have far better performance, and many research GCs support concurrent, pause-free collection.

(And that leaves aside the non-performance benefits of a GC, in terms of engineering simplicity and bug elimination.)

<comment><USER3> Any pointers on the state of the art?

<comment><USER4> [http://www.azulsystems.com/technology/c4-garbage- collector](http://www.azulsystems.com/technology/c4-garbage-collector)

<comment><USER5> Which I would add is a production garbage collector, an updated version of the Pauseless one that's got its own paper and is covered in the 2nd edition of _the_ book on garbage collection, [http://www.amazon.com/The-Garbage- Collection-Handbook-Manage...](http://www.amazon.com/The-Garbage-Collection- Handbook-Management/dp/1420082795/)

Does require custom hardware or kernel cooperation for speed (e.g. it needs to do batched MMU operations without clearing the TBL on each 2MB page). Looks like it's got a better read barrier than the Pauseless one; that does of course cost extra on stock hardware.

</comment></comment></comment></comment><comment><USER6> It's amusing to me that on the same thread GC proponents say that high- performance GC is easy ([https://news.ycombinator.com/item?id=6485114](https://news.ycombinator.com/item?id=6485114)) _and_ that Java's GC is "far from state of the art" ([https://news.ycombinator.com/item?id=6483704](https://news.ycombinator.com/item?id=6483704)).

And so the perpetual argument goes: GC is superior to manual memory management (the latter of which is "adding needless effort to writing software"), and the performance is just as good, and the cases where performance is _not_ just as good will be fixed Real Soon Now.

<comment><USER7> I'd like to point out that I did not say that a _high-performance_ GC is _easy_. What I actually said is that a _simple_ GC is _easier_ than simple automatic optimizations. A high-performance GC is the same level of difficulty as _good_ automatic optimization.

</comment></comment><comment><USER4> Actually the discussion at the newsgroup is more about Java's GC vs malloc/free.

In GC enabled systems programming languages like the Oberon family, Modula-3, D, Sing#, one has finer control over the types of available memory.

\- The usual GC allocated memory with the usual set of possible GC implementations

\- Stack objects

\- Global objects statically allocated

\- Manual memory management inside unsafe blocks via untraced GC references

Not all GCs are made alike.

<comment><AUTHOR> Real-time Java (RTSJ) actually has some very cool mechanisms for fine-tuned control over allocations.

</comment></comment><comment><USER8> In language garbage collection just makes implementing a compiler that much harder to do and frankly bloats up the runtime. Basic c++ shared pointers take up pretty much 90+% of what people need in GC (non stupid software architecture decisions should make up the remaining 10%).

I _have_ seen problems with stalling caused by kernel memory heap defragging on linux resulting in some serious thrashing. What that means is that the issue of heap fragmentation can be a problem regardless of which language used.

<comment><USER9> Reference counting, even ignoring the additional expense C++ places on std::shared_ptr by requiring it be thread safe, is far too slow for allocation heavy languages. Only "we don't give a damn" implementations (like CPython) use it, because it is simply not competitive.

<comment><USER10> iOS uses Automatic Reference Counting (ARC, [http://en.wikipedia.org/wiki/Automatic_Reference_Counting](http://en.wikipedia.org/wiki/Automatic_Reference_Counting)) by having the compiler add reference counting instructions, while refraining from having the developer add tedious reference counting instructions themselves. I'm not sure if there are significant performance advantages over proper GC.

It's worth noting that Apple has implemented GC in Objective-C on OS X before, but deprecated that in favour of its ARC implementation -- probably because its garbage collection implementation was no good at all (performance-wise).

<comment><USER4> Apple dropped GC in Objective-C, because they never managed to make it work properly.

By the time ARC was announced, there were still lots of corner cases in the ways code compiled with GC was interacting with code that wasn't compiled with GC enabled.

Plus, if you check Apple's developers documentation from those days, great care was required to make sure the GC could play nice with the C part of Objective-C.

So, in reality the ARC approach was taken, because it was the only way to have a working automated memory management, compatible with Objective-C code overall. Of course, being Apple, they sold it as a better solution for memory management and not as a result from their failure to implement a sound GC for Objective-C.

</comment></comment></comment></comment><comment><USER11> People tend to conflate garbage collection with Java's object model, but the two things are distinct. Most of the time when you're talking about performance limitations in Java, you're really concerned about Java's object model, not the GC.

Garbage collection limits your object model, but Java's is much more limited than required for garbage collection. For example, in C++, you might pass small structs on the stack or in registers, while in Java you pass a pointer to a heap allocated object. But there is no reason a garbage collector can't handle value objects allocated on the stack, and indeed the CLR supports that feature.

All a garbage collector prevents you from doing is hiding pointers in integers and the like. But that's not as significant a limitation as not being able to say flat arrays of structs, something that is totally compatible with garbage collection.

<comment><AUTHOR> Java value-types (+ arrays) are in the works. The JVM already does what is known as "scalar replacement" ([http://www.stefankrause.net/wp/?p=64](http://www.stefankrause.net/wp/?p=64)) automatically allocating some non-escaping objects on the stack.

</comment></comment><comment><USER12> I loathe the fact that all intermediate options get left off the table - reference-counting, or single-ownership-with-weak-references, or anything else short of memory-gobbling full GC that breaks your real-time guarantees.

And C++'s hideous shared_ptr system isn't a good thing to point to.

<comment><USER13> If you have real-time requirements reference counting is worse than GC; it has the same problem of some operations sometimes taking an unexpectedly long amount of time, and the worst-case guarantees are weaker.

Single ownership with weak references is effectively what you have in C, though I guess compiler support for distinguishing which references are weak or strong is nice.

<comment><USER12> Weak reference implies a safe error result if you access a weak reference after a "strong" reference has been released, not undefined behavior. Not just a pointer.

For refcounting and real-time, just let me leak the cycles. The non-realtime part of reference counting is the cycle detection, let me control the cycle sweeps and everybody's happy.

<comment><USER14> _> let me control the cycle sweeps and everybody's happy._

Maybe you are happy, but 99.9% of people would probably be very unhappy with that solution. How would you even go about accomplishing that in a way that wouldn't make you lose your mind? I can only imagine the complexity of having to decide how many cycle sweeps need to be done every time I reduce the reference count on an object/struct. I also think the runtime performance of enforcing that kind of limit would out way any possible benefit.

</comment></comment></comment></comment><comment><USER15> The benefit of malloc/free (or new/delete) is that you have control over your own memory layout to handle optimizations like cache alignment, keeping relevant data contiguous, deferring pauses that would introduce latency and such.

<comment><AUTHOR> Memory layout is a separate discussion. The kind of layout optimizations you're describing require work, and if you're willing to put in this work, you can do it in GCed environments as well (like the JVM, with finer-grained control coming in future versions).

Otherwise, several/most of the JVM's GCs do a very good job of handling this for you. Objects that were allocated consecutively by the same thread will tend to be kept close together in memory.

<comment><USER15> That's contiguity but alignment is a different thing. At a certain point though, the two things become one and the same. On the one hand, you could put in layout optimizations in a GC system. On the other hand, you can use things like shared_ptr and effectively write your own GC even.

I guess the debate really isn't an A vs B thing. It's just a spectrum and deciding which side of the spectrum you want to start on.

<comment><AUTHOR> > you can use things like shared_ptr and effectively write your own GC even.

This is hardly the case, especially on modern multi-core hardware. Writing a GC that works well in a concurrent environment is a huge undertaking. And unless your system is very simple, you'll often be better off with a good general-purpose GC.

A shared_ptr introduces _a lot_ of contention.

<comment><USER16> The thing is that most garbage generated is thread local. In fact, depending on the program, it's likely that most of it follows a stack discipline, albeit one that's not delimited by function boundaries.

A simple GC (or resettable arena allocator) can make use of this knowledge, and be quite a bit faster in those specific cases.

It's hard to beat the man years of work in generic GC that works in all cases, but it's not that hard to beat it in specific situations when the application author knows the details of how the application will do the allocations.

99% of the time, writing your own custom allocation strategies is a waste of time. Sometimes, though, it can be a win, allowing you to allocate _and_ free in a matter of tens of CPU cycles with no GC pauses at all.

<comment><USER13> >99% of the time, writing your own custom allocation strategies is a waste of time. Sometimes, though, it can be a win, allowing you to allocate and free in a matter of tens of CPU cycles with no GC pauses at all.

Agreed, but again, if you're willing to put that effort in you can do it in a GCed language.

</comment></comment></comment></comment></comment></comment><comment><USER17> > What makes a difference is gc pauses. Imho if you can live without managed > memory. You can do this in java as well and eliminate gc pauses.

Wait, how can you turn off GC'd managed memory in Java, and do it yourself? I didn't know you could do that in Java?

<comment><USER18> Now everyone will tell you that you can use ByteBuffer or sun.misc.Unsafe to keep memory off-heap and hence out of sight of the GC.

BUT what that means is that you have create your own custom implementations of the data structures you need and you'll have to write your own memory allocator.

Unless that memory area holds just a simple data BLOB (like a video file) you're always going to be much more productive implementing the entire thing in C++.

<comment><USER4> > Unless that memory area holds just a simple data BLOB (like a video file) > you're always going to be much more productive implementing the entire thing > in C++.

Or use a GC enabled systems programming language, that provides control over what is allocated via GC or via other means.

</comment></comment></comment><comment><USER19> > In my experience the biggest difference is stack allocation

This is the only voice of reason in this thread full of bullshit by monkeys who have obviously never used C to any extent.

<comment><USER4> Yes we did.

Some of us are even older than C, and are thus aware that there used to be a time and age where C only mattered for UNIX developers.</comment></comment>


MineRL 2020 Competition: Sample Efficient Reinforcement Learning in Minecraft - <AUTHOR> https://minerl.io/competition/ <comment><USER2> Took part last year, this competition was a lot of fun!

Bleeding edge research, very fun to see your agents training, and lot of interesting ideas to explore.</comment>


"We are more possible than you can powerfully imagine" - <AUTHOR> http://www.badscience.net/2009/07/we-are-more-possible-than-you-can-powerfully-imagine/

<comment><USER2> See also: <http://news.ycombinator.com/item?id=598385>

In addition to the book on cryptography, Singh also directed the film about Fermat's Last Theorem ...

<http://en.wikipedia.org/wiki/Fermats_last_theorem>

... first transmitted by the BBC as part of the Horizon series, subsequently aired in the USA as part of the NOVA series. He also wrote the book that followed. Both have been mentioned here before, both are most highly recommended. The Wikipedia article references Singh's book copiously.

The Code Book, mentioned elsewhere, was his second book.

You can read more here: <http://en.wikipedia.org/wiki/Simon_Singh>

I know Simon quite well, and am following the libel case with interest. (The wikipedia article also references John Stillwell, whom I also know. This is slightly worrying ...)

</comment><comment><USER3> Wow, you've got to love the internets. This reads just like a highbrow version of one of 4chan's "for great justice" campaigns. Surely the BCA must now be regretting the day they picked a fight with Singh. As Simon Perry noted, chiropractors were largely left alone to the back pain business by the sceptics, who were far more interested in exposing other alternative therapies. Suing Singh for the "bogus" remark was like posting a video on YouTube of a bunch of chiropractors abusing a cat. By the time they win their battle - still a possibility - they'll have lost the war.

</comment><comment><USER4> My favorite part of the article: "Professional complaints followed in May, mostly about individual chiropractors' claims. Then, in June, blogger Simon Perry found the BCA database of 1,029 members online, containing 400 website URLs. He wrote a quick computer program to automatically identify all the chiropractors in the UK claiming to treat colic, locate their local Trading Standards office, and report them (more than 500 in total) automatically, followed up with printed letters."

<comment><USER2> I like this bit:

        So Perry reported over 500 chiropractors to them,
        alleging they had made claims without adequate
        evidence. The GCC rejected his letter, saying it
        only takes individual complaints. A pile of
        individual complaint letters were instantly
        generated and delivered to their door.

</comment></comment><comment><USER5> chiropractors make their money off of ignorance. if you feel better after going to a chiropractor it's because the muscles that support and stabilize the spine (especially the posture muscles of the lower back) are weak. do some basic exercise like stretching, hyper extensions, and yoga and your back problems will go away.

there's a reason chiropractors don't grab a chunk of the lucrative physical therapy market, their techniques are a stop gap, they don't improve your condition.

<comment><USER6> Have you ever had a spine/neck injury? There is no amount of streching or yoga that will fix it. I fell rock climbing last summer and was in serious pain for weeks. I went to a physical therapist not a chiropractor, but "cracking" your back is necessary for some injuries and it is the only way to fix it. It took a hell of a lot of force to get my spine back in line but it feels better almost immediately. After that comes the therapy to rebuild the muscles, but that takes a LONG time, and resetting the spine is done multiple times in there.

<comment><USER7> Sure, strengthening back muscles is not enough for real injuries, but neither is going to a chiropractor. At best, the chiropractor is a physical therapist with a focus on the back. But mixed in with those people are a good number of others who believe in treatments with no basis in evidence. And at times, as gcheong pointed out, the treatments can result in making the condition worse.

</comment></comment></comment><comment><USER8> tangental - Simon Singh (the guy being sued) is the author of this excellent book on cryptography [http://www.amazon.com/Code-Book-Science-Secrecy- Cryptography...](http://www.amazon.com/Code-Book-Science-Secrecy- Cryptography/dp/0385495323)

</comment><comment><USER9> The article is interesting, but this is a really terrible title. The title contains zero relevant info, not even any irrelevant info, and it barely even sense as a sentence. The sentence doesn't appear in the article. I am at a loss as to what the titler was thinking.

</comment><comment><USER10> Talk about a crowdsourcing smackdown.</comment>