Stage0 – A set of minimal C compiler bootstrap binaries - <AUTHOR> https://github.com/oriansj/stage0 https://github.com/oriansj/stage0/blob/master/Linux%20Bootstrap/x86/cc_x86.M1 It also appears to be bootstrapped from hex <comment><USER2> _This is a set of manually created hex programs in a Cthulhu Path to madness fashion. Which only have the goal of creating a bootstrapping path to a C compiler capable of compiling GCC, with only the explicit requirement of a single 1 KByte binary or less._

What a wonderfully mad, cool goal.

<comment><USER3> I like to think of exercises like this as a "bootstrap pilgrimage", and I hope the phrase catches on.

</comment></comment><comment><USER4> Does it support all of C?

Is this actually smaller than TCC? It's kind of hard to tell since TCC is split into files and I don't know what's actually necessary. And that question includes earlier, less featureful versions of TCC. Note that the more-or- less-C compiler it's based off of is absolutely miniscule: [https://bellard.org/otcc/](https://bellard.org/otcc/)

<comment><USER5> Is it really fair to do a size comparision between a statically linked C compiler written in Assembly and designed for readability with an a dynamically linked binary which has none of those things and was designed to win the International Obfuscated C Code Contest??

Also if you add up the binaries used via dynamic linking you are looking at which in the case of /lib/x86_64-linux-gnu/libc.so.6 is 16MB in size?

Did I mention it works on all x86+amd64 POSIX systems too?

<comment><USER4> The un-obfuscated version is commented and readable and still only ~15KB of source code.

It uses calloc, fopen, and fgetc, which can trivially forwarded to the same syscalls as stage0 with only a few bytes. It uses isalnum, isspace, isdigit, strcpy, and strstr, which are all trivial. Add all that up and maybe you need to penalize it by a kilobyte. I'm not sure exactly how it's using dlsym but it seems to also be worth few bytes. It's not using the other 16MB of libc.

</comment></comment></comment><comment><USER6> >Additionally, all code must be able to be understood by 70% of the population of programmers. If the code can not be understood by that volume, it needs to be altered until it satifies the above requirement.

How is this measured/qualified? These days, I would doubt that 70% of people whose primary job is to write code have any knowledge of assembly whatsoever, so a naive reading of the above paragraph seems unlikely to succeed.

<comment><USER7> _These days, I would doubt that 70% of people whose primary job is to write code have any knowledge of assembly whatsoever_

If pressed, I suspect most of the programmers in the world could read assembly. They might hate it, but they could do it, if given sufficient motivation. Simplified assembly used to be written as a game.

[https://www.corewars.org/index.html](https://www.corewars.org/index.html)

 _so a naive reading of the above paragraph seems unlikely to succeed._

How naive are you going here? Turn it into a contest, where the versions of the code contain backdoors, and contestants are ranked by how quickly and accurately they can identify them. Arrange for cash prizes, and you'd have your determination.

<comment><USER8> I wouldn't be surprised if none of the people who learn to code through bootcamps or self-taught modules can read assembly. Most of that kind of stuff teaches web development and not much else, so any code lower on the stack is mostly unfamiliar territory.

<comment><USER9> None is a rather final quantity.

I'm self taught, and pull out IDA pretty often in my job. My wife went to galvanize, and I taught her 6502 asm before she started there. So that's two for you.

</comment></comment></comment></comment><comment><USER10> Do the instructions in the README count as part of the program?

If not, then the smallest binary to bootstrap a C compiler is actually a single jump to a C compiler in memory with a README containing the memory dump in to be typed in :)

Seriously though, it reminds me of the Toaster Project [0] where an RCA student attempted to build a modern toaster without using the modern supply chain.

[0] [http://www.thetoasterproject.org/](http://www.thetoasterproject.org/)

</comment><comment><USER11> What oriansj (as well as rain1 and others) are doing is both very impressive and important. The objective is to get us to bootstrappability and, for example, escape Trusting-Trust attacks; one reason it's profoundly important is the long-term archival problem. _Media longevity_ is one crucial part of archival, but as we were discussing today in [https://news.ycombinator.com/item?id=20272557](https://news.ycombinator.com/item?id=20272557), there are plausible solutions to that problem.

 _Interpretability_ is another part of the problem: even if we recovered an executable copy of Ivan Sutherland's historically groundbreaking program SKETCHPAD, for example, we wouldn't be able to run it because we don't know the instruction set for the computer it was built for. Remember that the entire body of knowledge about Ancient Egyptian culture was lost in the 5th Century, when the Christian Dark Age closed the temples, and not regained for almost 1400 years — and then only due to the great good fortune of the Rosetta Stone.

A bootstrappable computing stack is a crucial part of the "Rosetta Stone" that will be needed to preserve 21st-century knowledge. One of the few papers tackling the interpretability problem in this form is [http://www.vpri.org/pdf/tr2015004_cuneiform.pdf](http://www.vpri.org/pdf/tr2015004_cuneiform.pdf), "The Cuneiform Tablets of 2015", by Long Tien Nguyen and Alan Kay.

There's a more immediate necessity, though. As recent events make clear — Chrome's extension API kneecapping ad-blockers, the increasing effectiveness of Chinese censorship, and the shocking US$12 million award to Nintendo last November against ROM site operators for preserving classic video games, for example — the current political and economic system cannot be trusted to preserve our access to our cultural heritage, even during our lifetimes. That means that we need an autonomously-bootstrappable trustworthy free-software infrastructure that is viable without the massive economies of scale that fund mainstream platforms like Linux, Android, MacOS, Chrome, and even Firefox. If your personal archive of the Tank Man photo, the Arab Spring tweets, or the video of the murder of Philando Castile runs afoul of future malicious-content filters integrated into your operating system, there is no guarantee that it, or you, will survive.

So we're doing our best to get some green shoots established before the situation has any opportunity to get worse.

<comment><USER12> If you asked me what the 4 best documents regarding bootstrapping are i'd say:

* Egg of the Phoenix (Blog post) - [http://canonical.org/~kragen/eotf/](http://canonical.org/~kragen/eotf/)

* The Cuniform Tablets of 2015 (Blue-sky academic research) - [http://www.vpri.org/pdf/tr2015004_cuneiform.pdf](http://www.vpri.org/pdf/tr2015004_cuneiform.pdf)

* Preventing The Collapse of Civilization (Video) - [https://www.youtube.com/watch?v=pW-SOdj4Kkk](https://www.youtube.com/watch?v=pW-SOdj4Kkk)

* Coding Machines (scifi story about trusting-trust attack) - [https://www.teamten.com/lawrence/writings/coding-machines/](https://www.teamten.com/lawrence/writings/coding-machines/)

<comment><USER11> This is awesome! Thank you! And thank you for the flattering reference to my own thought experiment there.

</comment></comment></comment><comment><USER13> The trouble with bootstrapping GCC is that it requires flex/lex. Have you tried bootstrapping flex/lex?

It sounds stupid, I know, but when I investigated possible paths for getting from zero to GCC, flex looked like the biggest potential obstacle.

There are also C files in GCC/binutils that are generated by complex shell scripts. That was perhaps the second biggest obstacle.

(If I recall correctly, bison's not a problem: old versions of bison don't use bison.)

<comment><USER14> My project [https://gitlab.com/giomasce/nbs](https://gitlab.com/giomasce/nbs) is currently able to more or less bootstrap flex, starting from only tcc and musl (within a running Linux kernel). Terms and conditions may apply: I haven't tested much the generated binary yet, and given the tricks I have to do to get there miscompilations and introduced bugs are everything but impossible. But at least ideally it should be possible to iron them out without changing the whole path.

Bison is my next target, but I haven't been able to work much on it lately. It's true that it should be possible to bootstrap it by history, although I really hope that one does not need to many steps to get to the latest release.

<comment><USER13> That's very interesting! Could you summarise the relevant steps? It appears that you're using "The Heirloom Project", which I hadn't heard of. (Is it related to Illumos?) So the Heirloom lex can be built without lex/yacc, and then you can build which version of flex using that?

<comment><USER14> The Heirloom project is a collection of utilities coming from open-sourced versions of Caldera and Sun code ([https://en.wikipedia.org/wiki/Heirloom_Project](https://en.wikipedia.org/wiki/Heirloom_Project)). It's nice because it is very easy to bootstrap, even if nowadays it is unmaintained. It also features a lex and yacc that are written in plain C, without depending of themselves (well, lex depends on yacc, but at most one dependency is ok). It is not easy to use it to bootstrap flex, but it can be done. Basically you have to use an old enough version of flex (I am using 2.5.11) and modify its lexer so that it can be parsed by a simple lex that is not flex (plus some additional light patching of the C code). It is a bit tedious, but a rather mechanical task.

Flex 2.5.11 is rather old now. When I have time I will also add another stage to compile a more recent version. But I believe that flex 2.5.11 can generate the lexer of the latest flex release. Also, it seems that all flex versions can be compiled with a yacc that is not bison. This is what I recall from my experiments of a few weeks ago, that I still have not consolidated in a working script. Surprises might happen!

The other steps of the project are described in the README (where flex is not mentioned yet, but all the rest is). Later steps will be bison, as I said, and then who know. I should aim at Perl, which then would unlock autotools, but Perl itself has a lot of dependencies and I haven't yet begun unravelling them. Also bash and coreutils should be bootstrapped at some point, but for the moment dash and Heirloom can do their job without problems, so it is not a priority (although it might become at some point). Later targets will be binutils, gcc and glibc, but do not hold your breath. Eventually I would like to bootstrap Debian packaging tools and start building Debian packages. That might require that I train my offspring into programming and convince them to continue my mission.

<comment><USER13> That all sounds good!

It is also possible (though tedious) to build GCC and binutils without autotools.

Getting from epsilon to Debian seems a bit like following stepping stones across a swamp. It's unclear which route is most likely to work, but whichever way you pick you end up ankle-deep, or waist-deep, in mud.

Also, it's not clear what "epsilon" should be. My feeling is that the ideal minimum prerequisites would be something like a C89 native compiler on an operating system that is much simpler than Unix. So you could read and write files and execute a subcommand, perhaps, but no pipes and no fork. So the first major goal would be to build GCC and binutils as a cross-compiler, and the second major goal would be to cross-compile those things plus the Linux kernel and some kind of shell so that you could then move to Linux.

The whole thing is a big project, but if you can describe a way of getting from stepping stone A to stepping stone B, where B is a bit closer to Debian than A was, then that may help future explorers, and that use of Heirloom Project's lex to build flex sounds like a good example of that.

<comment><USER14> Currently epsilon is a Linux kernel with musl and tcc. Since tcc can be used for scripting, I believe that with a non-trivial amount of work (including patching musl), musl could be built by tcc. Therefore epsilon would become just Linux and a static build of tcc.

To further reduce epsilon (or, in somewhat standard terminology, the "binary seed") I am also writing asmc ([https://gitlab.com/giomasce/asmc](https://gitlab.com/giomasce/asmc)), which is a minimal operating system with a minimal compiler, totalling less than 6 KiB binary seed. This compiler is not for C, but for G, which is a C-inspired language that I invented for this purpose (see the project README and links for more details). The target for asmc is to be able to build and boot Linux and a static copy of tcc (the latter is more or less already there; Linux is, of course, the hard part), at which point it can bootstrap nbs. I'm not there yet, though.

Also, to be fair, although asmc's seed is 6 KiB, you still need an x86 CPU (I'm not aware of free implementations), a loader and a BIOS to have a functional environment. This is not little thing, but I guess it is a reasonable target so far.

</comment></comment></comment></comment></comment></comment><comment><USER15> What special limits can this limbo under? 15 kb is so small.

It's even smaller than l1 cache. (In case you wanted to bruteforce every possible text file you could feed this binary, and wanted to do it all on cache in the CPU).

Maybe it's the size of payload you can put into a usb C cable or something.

I mean it's just so freaking small. Any ideas what limits this is "small enough" to fit under?

<comment><USER4> > Maybe it's the size of payload you can put into a usb C cable or something.

The part you grab on a typical usb C cable is just about the size of a microsd card. You might have to use a slightly narrower chip, but you can also go many times thicker. So if you're sneaking storage into a usb C cable think "terabyte".

(Unless you mean hijacking an existing chip, which might have 0 writable storage or might have a megabyte, who knows.)

<comment><USER15> yes I meant the latter. maybe that's all a malicious agent has for usable payload for whatever reason. I really had to think hard to come up with that, it's not meant to be realistic.

</comment></comment></comment><comment><USER16> See also Edmund Grimley Evans' bcompiler, mirrored at [https://github.com/certik/bcompiler](https://github.com/certik/bcompiler)

On a related note, one can write arbitrary bytes to a file (on a Unix-like system) using GNU 'echo' or the 'printf' utilities. Chris Wellons' post "A Magnetized Needle and a Steady Hand" describes how to write a basic utility in this way.

</comment><comment><USER17> Read this code if you think you don't understand assembly. A wonderful project, very clearly written.

</comment><comment><USER18> Oh so this is similar to kragen 'basement experiment' .. kudos

<comment><USER19> [https://github.com/kragen/stoneknifeforth](https://github.com/kragen/stoneknifeforth) is 114 non-comment lines of code, compiles to 4063 bytes. (he mentions it's about half the size of the otccelf tinyc)

<comment><USER11> Yeah, SKF could definitely be improved. I'm excited about stage0!

</comment></comment></comment><comment><USER20> So cool to see Org mode READMEs in projects unrelated to Emacs!

I hope Github improves Org mode support at some point.

<comment><USER21> I like Org mode, but the syntax highlighting support is not great:

[https://github.com/wallyqs/org- ruby/issues/64](https://github.com/wallyqs/org-ruby/issues/64)

I have been looking at AsciiDoc:

[https://asciidoctor.org/docs/asciidoc-syntax-quick- reference](https://asciidoctor.org/docs/asciidoc-syntax-quick-reference)

<comment><USER20> > I like Org mode, but the syntax highlighting support is not great

I know. That's why I said "I hope Github improves Org mode support at some point." :)

> I have been looking at AsciiDoc

If you like Org mode, you can overcome the Github/etc. Org mode rendering limitations by hosting small static sites for your projects. You may choose to do so by simply exporting Org to HTML using ox-html[1], or even exporting Org to markdown for Hugo (disclaimer: my package -> ox-hugo)[2].

I mean, if you like using Org mode, and Github et al don't see the value in improving support for that, don't use them to render Org mode docs :)

[1]: [https://eless.scripter.co/](https://eless.scripter.co/)

[2]: [https://ox-hugo.scripter.co/](https://ox-hugo.scripter.co/)

<comment><USER21> The problem isnt the quality of GitHub with Org-Mode.

The problem is that the GitHub syntax highlighter, PrettyLights, is closed source:

[https://github.com/github/pages- gem/issues/160](https://github.com/github/pages-gem/issues/160)

So you cant get consistent results between github.com and github.io

</comment></comment></comment></comment><comment><USER22> Does this have similar goes as [https://www.gnu.org/software/mes/](https://www.gnu.org/software/mes/)?

<comment><USER23> > Mes is inspired by The Maxwell Equations of Software: LISP-1.5 – John > McCarthy page 13, GNU Guix's source/binary packaging transparency and > Jeremiah Orians's stage0 ~500 byte self-hosting hex assembler.

stage0 is one of mes' inspirations, so I'd say there's a level of connection there.

Apparently mes is also working towards being able to be compiled by M2-Planet by the same author [0], at which point it might be possible to eventually build mes from stage0 as it can now bootstrap M2-Planet.

[0] [https://github.com/oriansj/mes-m2](https://github.com/oriansj/mes-m2)

</comment></comment><comment><USER24> Does this have any implications for trusting trust?

<comment><USER25> It just means that your compiler is _theoretically safe_ if you build a assembler manually and verify it. This can be done by printing the assembly of the assembler (an assembler is just a glorified find and replace if you leave out optimizations) and then translate it into machine zeros and ones by hand (See Intel Reference Manual for more details on the translation table). To speed this up, distribute the typing input, use e.g. Mechanical Turk, minimum wage clerks etc. and compare the result from multiple sources to ensure accuracy. Once your confident that your machine code translation is an accurate representation of your assembler, run the assembler on Stage0 and the bootstrap process should take care of itself.

<comment><USER26> How do you get those zeroes and ones into a machine-readable format, without a trusted text editor or OS?

<comment><USER11> If you're willing to trust your hardware, you can theoretically program an SPI Flash chip one bit at a time using two debounced switches. I haven't tried running SPI at sub-kilohertz speeds, much less sub-hertz speeds, and it wouldn't be surprising if that failed to work on some hardware. But you can probably find hardware where it will work. A punched paper tape reader or barcode reader would be a more practical way of programming the Flash than manual toggle switches.

If you have a machine that can boot from floppy, you can use an untrusted system to write the floppy, and then verify the contents of the floppy with a magnetic force microscope or magnetic developer fluid, before booting it.

It isn't obvious why you would trust the hardware, though, if you didn't fabricate it.

</comment></comment></comment></comment><comment><USER12> of course this isn't the smallest C compiler, it's 5000 lines of code. There is a 500 line C compiler written in C.

But perhaps it's the smallest if we count transitive dependencies! (by this I mean count the lines of code of the program, and every program you need to build the program and everything you need to build them and so on)

<comment><USER12> Why are people downvoting this comment?

<comment><USER11> Because Hacker News is largely populated by "finance-obsessed man-children and brogrammers", in JWZ's memorable phrase, whose votes it unfortunately weighs as highly as yours, mine, or oriansj's.

</comment></comment></comment><comment><USER27> why link to github when main repository is in savannah.gnu.org?

<comment><USER23> > pull requests can be made at > [https://github.com/oriansj/stage0](https://github.com/oriansj/stage0) and > [https://gitlab.com/janneke/stage0](https://gitlab.com/janneke/stage0) or > patches/diffs can be sent via email to Jeremiah (at) pdp10 [dot] guru or > join us on freenode's #bootstrappable

The main contributing points seem to be github and gitlab, rather than savannah.

</comment></comment><comment><USER28> Clickable link: [https://github.com/oriansj/stage0/blob/master/Linux%20Bootst...](https://github.com/oriansj/stage0/blob/master/Linux%20Bootstrap/x86/cc_x86.M1)</comment>


Why Multitasking May Make You Less Productive - <AUTHOR> http://mashable.com/2011/08/24/multitasking-productivity/

<comment><USER2> This is nothing new.

"... Brain activation for listening is cut in half if the person is trying to process visual input at the same time. A recent study at The British Institute of Psychiatry showed that checking your email while performing another creative task decreases your IQ in the moment 10 points. That is the equivalent of not sleeping for 36 hours—more than twice the impact of smoking marijuana..."

Taken from [http://www.fourhourworkweek.com/blog/2008/05/25/the- multitas...](http://www.fourhourworkweek.com/blog/2008/05/25/the-multitasking- virus-and-the-end-of-learning-part-1/)

</comment><comment><USER3> Multitasking is highly inefficient, but multiplexing can be great. Kevin Kelly wrote a good post about it.

[http://www.kk.org/thetechnium/archives/2011/03/multiplexing_...](http://www.kk.org/thetechnium/archives/2011/03/multiplexing_vs.php)</comment>


Why is max slower than sort in Python? - <AUTHOR> http://stackoverflow.com/questions/35014951/why-is-max-slower-than-sort-in-python <comment><USER2> "How not to benchmark" would be a more accurate title.</comment>


Get ready for tens of millions of climate refugees - <AUTHOR> https://www.technologyreview.com/s/613342/get-ready-for-tens-of-millions-of-climate-refugees <comment><USER2> Most important part to me: "Last year a report for the World Bank, the first to model migration due to climate change on a large scale, estimated that as many as 143 million people in sub-Saharan Africa, South Asia, and Latin America could have to relocate within their countries by 2050."

As with most models predicting widespread human problems due to climate change, this one is predicting the largest negative impacts on equatorial, mostly poor countries. As usual, rich countries will mostly do alright for some time.</comment>


Cracking Passwords In The Cloud: Amazon's New EC2 GPU Instances - <AUTHOR> http://stacksmashing.net/2010/11/15/cracking-in-the-cloud-amazons-new-ec2-gpu-instances/ <comment><USER2> This is insane for password cracking. I've been trying to get Pyrit working on the Centos image but have been having trouble compiling it. An 8 node cluster with Teslas is going to bring WPA-PSK cracking down to where WEP was a few years back for those who can afford it.

This is an incredibly disruptive thing for Amazon to do. They've just brought near-government grade crypto-breaking capabilities to the mass market.

<comment><USER3> _They've just brought near-government grade crypto-breaking capabilities to the mass market._

No, they really haven't. Near-government grade KDF-cracking capabilities will be when Amazon announces FPGA Compute instances.

<comment><USER4> Last time I checked, GPU processing has a better-bang-for-the-buck than FPGA processing, and the gap continues to widen.

<comment><USER5> I suspect the NSA doesn't care too much about bang-for-buck. 22nm FPGA's [1] seem like they would work pretty well.

 _In an interesting twist, The Register claims that Achronix's decision to use Intel was driven in part by national security considerations. We've reported extensively on the idea that chips fabbed overseas in insecure facilities could contain hidden kill switches or backdoors that would let an opponent cripple the US military, and Achronix allegedly wants to be able to sweeten its pitch to military customers by offering a home-grown solution._

[1] [http://arstechnica.com/business/news/2010/11/intel-shifts- st...](http://arstechnica.com/business/news/2010/11/intel-shifts-strategy- sells-22nm-fab-capacity.ars)

<comment><USER6> Those are particularly interesting because they're asynchronous FPGAs -- they use local handshaking rather than a global clock to keep everything synchronized. That should make them easier to port to new, smaller process nodes, and they say it's responsible for their unusually high throughput.

Cool stuff, and all the more intriguing considering that Intel's getting involved.

</comment></comment></comment></comment></comment><comment><USER7> The sha1 hashes he provides are super weak. I can crack half of them in less than 30 seconds on my CPU with my software (16crack). Hardly material for a GPU:

EF8420D70DD7676E04BEA55F405FA39B022A90C8 "Password!"

5BAA61E4C9B93F3F0682250B6CF8331B7EE68FD8 "password"

A9993E364706816ABA3E25717850C26C9CD0D89D "abc"

1902E3D6FC4E78A0BCC50BA12B882769AFBF4A8C "bad"

8F2005004F8BAA7A1090A9BF3B03C48D38E78157 "P4s$"

CD3724AC40034097A3D27865D710E4F791B6AEDB "Bwah"

7110EDA4D09E062AA5E4A390B0A572AC0D2C0220 "1234"

[http://stacksmashing.net/blogfiles/2010_11_15/sha1_hashes.tx...](http://stacksmashing.net/blogfiles/2010_11_15/sha1_hashes.txt)

<comment><USER8> How long does it take your software to crack the other half?

<comment><USER7> That depends on the number and type of CPUs available. I know better than to race my CPU against a GPU. I won't win. I'm just pointing out the fact that if you have a fast GPU (or a cluster of them as the article suggests), then the hashes should be _much_ better than what is provided. With those hashes, any CPU based cracker can do them easily.

The article says 1 to 6 character passwords. With 12 cores, I can enumerate the full printable ASCII character set in about 30 hours, but I would not try to do that. I would use patterns and word lists first. After 6 to 7 characters, brains begin beating the hell out of brute speed.

<comment> nezza-_- In the article it's stated that the hashes are brute forced using the provided charset. I think you used a wordlist, which is a whole other thing.

<comment><USER7> I use brute force on 0 to 5 character attempts. I get smart at 6 or more characters. No need to use brains or GPU speed to crack 5 char passwords. Here are more weak hashes from his list:

DA39A3EE5E6B4B0D3255BFEF95601890AFD80709 "" no_pass

0D824508182A1AA0EEF9A0B6EE52F8A32AF06F0A "GoOd!" brute_5

A94B95A7A4D432DE056B0030DA879AF841376069 "GPGPU" brute_5

BFE06C47BE2390ACA934AB6A128C141DCEB4072F "G0o|)" brute_5

My point still stands. These should have been better/stronger hashes. I did all of that in less than an hour on a CPU using full enumeration (aka brute force).

</comment></comment></comment></comment></comment><comment><USER9> > _This just shows one more time that SHA1 is deprecated_

This just shows ignorance about hashing functions, especially fast ones. If they had used SHA-512, or say SHA-65536, it wouldn't be any more secure against brute-forcing / dictionary attacks. Barring SHA1 being _cracked_ \- ie, finding an efficient way to find SHA1 collisions - ie, "reversing" SHA1, it's no more deprecated than any other non-cracked hashing function.

About your only option is s/bcrypt, or something similar, which are _intentionally_ slow / hard, to defeat brute-force attacks like this.

</comment><comment><USER10> Allow me to be the first one to argue that we should require that users use longer passwords. This is not much of a burden. Your password will last for a very long time, and it takes only a few minutes to memorize a long password.

If computing power doubles every 2 years, then every two years we should bump up the required password length to match. If each character is randomly chosen from amongst 2^6=64 possibilities, then every 6 years we add 1 to the number of characters required.

For instance, if today we require 12 character passwords, which are clearly uncrackable even with Amazon's fancy new GPU instances, then in 2016 we require everyone update their password to a minimum of 13 characters. Every six years we bump it up again.

I think people vastly underestimate their ability to memorize passwords (just think how many phone numbers you probably had memorized back in the day). You could probably memorize passwords thousands of characters long if you bothered, and the only problem is that it would take too long to type them in.

You only need one long password, and all others can be stored in an encrypted file. This is not much to ask.

Bcrypt and scrypt look great, but not fundamentally any better than longer passwords.

<comment><USER11> We can't even get people to dump IE6. How the hell are we going to get them to use longer passwords, and change them every year?

Scrypt is better than longer passwords because it's actually possible to implement in practice.

<comment><USER10> > How the hell are we going to get them to use longer passwords, and change > them every year?

You merely require them to use longer passwords, and require them to change them every year. My university does this. They require a minimum of 8 characters, and they require that we change our password every 3 months or we can't log in.

<comment><USER12> If I had to come up with a new long password every three months I'd do what undoubtedly countless other people would do in the same situation: I'd write down my password somewhere nearby the computer so I could look it up when I needed it.

Overly onerous password requirements reach a point where they no longer increase security, they just shift vulnerability to a new area. They also piss off users.

<comment><USER13> If I had to change my password every three months, I'd do something even simpler: I'd stop using the service.

</comment></comment></comment></comment></comment><comment><USER14> OK, I'll bite (and show my ignorance regarding security). What should we use in place of SHA1?

<comment><USER15> At the very least, add a random salt added to each plaintext before you run it through the SHA1. This will at least defeat rainbow attacks fairly well.

That said, you need to do more and move beyond SHA1 since you can now reverse a SHA1 into plaintext with the computing power EC2 gives you.

<comment><USER16> It should be pointed out that it's impossible to "reverse a SHA1 into plaintext". The reason being that, since SHA1 produces a fixed size output and takes a variable (unbounded) input, there are an infinite number of input values for every unique output value. You may find a string that happens to come to the same SHA1, but there's no way to know whether or not it's actually what went into the SHA1 algorithm in the first place.

<comment><USER17> But that's ok, the server doesn't know what originally went into the algorithm either.

<comment><USER15> Bingo.

</comment></comment></comment></comment></comment><comment><USER18> First of all, only the initial release day, and the EC2 GPU killer app has been discovered.

Second, store passwords as a salted hmac, not just a shaXsum. It is still usually a singel command, and WAY more secure than simple shaX or mdX, as it eliminates the risk of prefix/postfix attacks. Adding the salt makes dictionaries pretty irrelevant (as long as each install has a unique salt).

I mean GEEZ; even php does it now: <http://us2.php.net/manual/en/function.hash-hmac.php>

<comment><USER19> An HMAC is a message authentication code, not a salted hash. It allows you to send a message over an insecure link and verify, by providing the secret key, that the message (and hash) was not tampered with.

Also, a salted hash is not a particularly good storage mechanism, ranking only above cleartext and naively-hashed storage. Once someone discovers the salt, then it's fairly easy to attack the password database in the same way that you'd attack an unsalted hash -- make a list of all possible passwords, hash them, and see what matches. Machines like the one mentioned in the article can make trillions of these in a second. The problem with hashes for protecting passwords is that they can be calculated very quickly; good for ensuring that every packet on your VPN arrived without errors, bad for ensuring that a bad guy can't make a list of passwords, hash them, and check the hashes against your database.

The best password storage mechanism is via the use of a "slow" hash function like bcrypt or scrypt. You set these up so that it takes a full second of CPU time to generate the hash from the message. Then instead of trying trillions of passwords a second, the attacker can only try one!

If you're using Perl, use Authen::Passphrase. It's a module that lets you easily use bcrypt for new passwords but your old method for older passwords. With an API like that, there's no excuse for endangering your users by using salted hashes!

</comment></comment><comment><USER7> With the right hash type (unsalted md4... yes, I'm looking at you Microsoft Windows Active Directory) and a top notch Nvidia or AMD graphics card, one can attempt roughly 600 million hashes per second at home in the living room.

Also, one of the teams used Amazon servers during the 2010 Defcon Crack Me If You Can contest. Here is their write-up: <http://contest.korelogic.com/team_CrackHeads.html>

<comment><USER20> 600 million? Try 10x that. A top notch HD 5970 does _6 billion_ MD5 hashes/sec. The MD4 rate is roughly 25% faster.

<http://golubev.com/gpuest.htm>

</comment></comment><comment><USER21> While the author does not seem to know much about computer security, he identified a legitimate threat. Brute force isn't so brute anymore.

</comment><comment><USER22> We should stop calling them pass _words_ to users, seeing as the best passwords are the concatenation of several non-words.

<comment><USER23> I believe we made a misstep with the password. I learned a long time ago that how you frame something for the user changes how they will interact with it. A common example I refer to is a small textarea on a website, and some clients wanting to see that textarea larger to encourage their users to write more in the box, or witnessing a user stopping and editing their post to frame it inside of the available area without going over.

The same with passwords, they imply a word. A much better solution is a pass phrase. As far as the system is concerned, functionally identical. But to the human mind, a completely different animal. A word is a word, but a phrase is limitless. With proper punctuation and capitalization, it has everything that makes a good password good: A-z, symbols, length. Except, being a phrase, it has an edge to a long complex password: you can remember it. A phrase has a beginning, middle, and an end.

<comment><USER24> Passphrase still implies using actual words, passcode on the other hand does not.

That said, for the most part people who chose weak passwords do so for ease of memory, not because they're so stupid that they think they are only allowed words.

<comment><USER25> Using actual words isn't such a bad thing if what you want to optimize is password entropy at a given level of memorability for naive users.

</comment></comment></comment></comment><comment><USER26> What about passwords 8 or more characters long?

<comment> nezza-_- You can easily calculate that using the charset and the time it took to crack a 6 character password. 6 character passwords with 95 different characters per digit take 49 minutes (Utilizing one machine and using CUDA-Multiforcer). 7 digits would mean 49*95=4655 (77 hours). But remember: The common password does not use special chars, so the actual number can be much lower.

</comment></comment><comment><USER27> If anyone is looking for a super simple bcrypt solution for PHP, check out Phpass:

<http://dev.myunv.com/articles/secure-passwords-with-phpass/> <http://www.openwall.com/phpass/>

</comment><comment><USER28> Anybody know how tarsnap's scrypt lines up against HMAC-based key derivation[1]? In particular, let's say I used the HMAC of the password as the bases for the HKDF?

[1] <http://tools.ietf.org/html/rfc5869>

<comment><USER29> HKDF isn't a password hash; it's a key derivation function, used to transform a passphrase into key material suitable for something like AES. It doesn't address the key security problem that scrypt addresses (iterative brute force cracking). It isn't an acceptable substitute.

You _can_ use PBKDF2 as a password hash (even though it too is designed mostly to turn passphrases into keys), because PBKDF2 is iterated to slow down brute force attacks.

</comment></comment><comment><USER30> isn't it illegal for a German to post on how to crack passwords ?

<comment><USER31> Why ?

<comment><USER32> In Germany, _linking to_ illegal material _can be_ considered a crime itself, for example linking to pirated software/media.

[http://www.webtvwire.com/linking-law-expert-dr-stephan- ott-t...](http://www.webtvwire.com/linking-law-expert-dr-stephan-ott-talks- about-linking-to-pirated-video/)

IANAL but I suspect the OP feels the same laws could be applied here.

<comment><USER30> No. Germany (and the UK) applied the American drafted European legislation that says it is illegal to distribute software that could be used to break into computers.

<http://www.securityfocus.com/brief/567> [http://www.schneier.com/blog/archives/2007/08/new_german_hac...](http://www.schneier.com/blog/archives/2007/08/new_german_hack.html)

This has had a chilling effect already

for instance

<http://en.wikipedia.org/wiki/Hydra_%28software%29>

On September 2007, to comply with new German laws regarding distribution of hacking tools to the public, THC stopped making the program available.

<comment><USER31> I see. I was not aware of this, thanks for the details.

</comment></comment></comment></comment></comment><comment><USER33> If this becomes a serious issue, my bet is that the government will be forced to start monitoring or background-checking GPU cluster use. This is one case where the cloud will hurt privacy, because Amazon will not risk their empire to protect criminals. Distributed computers cannot be subpoenaed so easily.

<comment><USER34> Not sure about this, but I could definitely see "export restrictions" on the service... though what's to stop a foreign company from building the same type of service.

<comment><USER35> export restrictions easily circumvented by getting a US credit card, just like you need one for Amazon Mechanical Turk</comment></comment></comment>


Microsoft sides with Oracle against Google in Java appeal - <AUTHOR> http://newsandinsight.thomsonreuters.com/Legal/News/2013/02_-_February/Microsoft_sides_with_Oracle_against_Google_in_Java_appeal/

<comment><USER2> This article is trying to make hay out of various tech wars because it makes a nice narrative, but the key here is

 _A copy of Microsoft's legal brief was not immediately available on the Federal Circuit's docket._

All we know is that they filed a friend of the court brief. Oracle's appeal has a fairly diverse[1] spread of approaches to copyright and APIs in an attempt to make their case. Microsoft's brief could be equally sprawling or limited to a single aspect of Oracle's claims. We won't know until we can actually _read it_.

[1] [http://arstechnica.com/tech-policy/2013/02/on-appeal- oracle-...](http://arstechnica.com/tech-policy/2013/02/on-appeal-oracle- insists-googles-android-copied-key-java-code/)

</comment><comment><USER3> Does Microsoft actually have a vested interest specifically in the use of Java by Google, such as being upset that Google had free use of the Java APIs while itself was scolded for implementing their own JVM?

Or is it basically "the enemy of my enemy is my friend"?

<comment><USER4> Microsoft's interest lately is to conduct a kind psyops campaign against Google. They seem to have identified that a huge problem for them is that people really like Google _a lot_. So basically their strategy is to throw mud at Google until people don't trust them or like them any more. So I would say this fits into their campaign very neatly - if they can portray Google as stealing technology from other companies then they can keep piecing together this picture of "bad Google" that they want everyone to see.

<comment><USER5> I doubt that their amicus briefs are driven by PR considerations as opposed to actual business concerns. How does a court filing that most of the public will never see help them shift public perceptions of Google?

<comment><USER4> Obviously the public are not going to read the court filing, the judges will. And if the court accepts an appeal there will be years of bad PR for Google as the world discusses how and whether Google "stole" the APIs for Android. Even if Google is cleared there will still be smear on their name from people perceiving that they are profiting from something they didn't really make themselves.

</comment></comment></comment></comment><comment><USER6> kind of ridiculous that this even makes to court in the first place....

edit: Does anyone know much about the precedence (in court rulings) of ownership of programming languages?

<comment><USER7> I think you would have to separate out the different aspects of implementation, syntax and publicity into things that could be patented, copyrighted or trademarked.

You could look at Loglan vs Logban for strongly related issues on what you can and can not copyright in language in general.

</comment></comment><comment><USER8> If anytime I have to use a 'managed' language, I go for python and haskell. Though, most of the time, I work on my video game renderer engine in evergreen C/C++ , I am quite satisfied and productive :)

There is no use of C#, Java or VB for my work, and for good reasons ;)

<comment><USER9> So you as the sole developer have decided that python and haskell are good for you.

Well I am convinced.

Sure, you aren't dealing with 30 year old databases. Or 100s of developers with millions of code. But that shouldn't matter.

<comment><USER10> The original post was off-topic and useless. There's no need to draw the entire conversation off-topic.

</comment></comment></comment><comment><USER11> As a good in my opinion (but unfortunatelly improbable) move from Google, will be to switch from Java to Haxe in next versions, because it compiles both to native Android and to Java and looks similar to Java, on the first view.

</comment><comment><USER12> I guess they dont understand why they lost in court with Sun and why Google did not with Oracle. And an android legal "takedown" would be very beneficial to MSFT ,since they try to sell phoneOS too.

<comment><USER13> Microsoft had licensed Java code under a trade agreement and was found to have violated the terms of that trade agreement.

Google had not licensed Java code, and was sued under patent law. Totally different kind of case.

<comment><USER14> Oracle made both patent and copyright claims against Google. The patent claims were pretty bog-standard, but the copyright claim (the loss of which is what Oracle is appealing) was quite... novel.

</comment></comment></comment><comment><USER15> Considering where .Net and C# came from (as in heavily, heavily, heavily influenced by) and we have to forget those pictures of Bill Gates chumming around with then President Clinton riding around in golf carts; it seems clear why Microsoft would take the stance they did..."Android is the best-selling smartphone operating system around the world.".</comment>


Tools we use for running our startup - <AUTHOR> http://www.balsamiq.com/blog/2009/10/30/tools/

<comment><USER3> Peldi keeps impressing us with his carefully written and useful articles. I have followed his blog since he released his product and I am convinced he is a great role model for bootstrapped entrepreneurs. The thing I am most grateful for is that Peldi actually took the time to document the whole process all the way back when he still was working at Adobe and secretly planning to start his own business. Just go back to his blog and check all the postings from 2008.. a great story. Good work and thanks for sharing how do you do run a great company!

</comment><comment><USER4> I'd love to know the monthly cost for using all of those hosted tools. Sounds very expensive.

<comment><USER5> Good question, let me try to answer. I'm doing this out of memory so some might be wrong: \- DropBox is $99 a year for 50Gig \- we get Confluence hosted for free because we're a successful Atlassian plugin developer \- I forget how much Last.fm is but I think it's very cheap, plus the free version is plenty good enough \- I think we're paying about $250/month for Slicehost right now \- a SkypeIn number is $60/year I think \- we're spending $34/month for Freshbooks \- we're spending $49/month for GetSatisfaction because we signed up at the very beginning (they had an offer) \- E-Junkie is $10/month \- Paypal pro is $30/month (I think) \- Spreedly is $19/month

That's pretty much it! What does it add up to? In general, I can tell you that it's a very small percentage of our expenses (salaries is where our $$ goes)...in a way we can't afford NOT to use these tools.

<comment><USER6> Depending on what you're doing, you can definitely save money and get similar service with Linode:

<http://journal.dedasys.com/2008/11/24/slicehost-vs-linode>

Basically, with Slicehost, you are paying to have 64 bit pointers - <USER2> that, the price difference isn't that big.

<comment><USER7> Never change a running system!

I would never change my hosting company if I would be fully satisfied and I would only be saving 100$ a month.

</comment></comment></comment></comment><comment><USER8> +1 for WriteRoom. I write all of my blog posts there and then migrate them over to Wordpress. I wrote most of our initial business plan in WriteRoom as well. The biggest lesson: formatting is a distraction. Worry about it only after you've got content.

Generally I'm also a big fan of full-screen things -- I live and die by my full-screen, light-colored-on-black Emacs.app.

<comment><USER9> I haven't upgraded my mac to snow leopard yet. Main reason is I'd like to know - do they have a way of getting a smooth, full-screen console on it yet? On leopard iTerm works. (You used to be able to get a plugin to make normal terminal do it, but this broke in leopard)

<comment><USER8> I use Megazoomer: <http://ianhenderson.org/megazoomer.html>

</comment></comment></comment><comment><USER10> "Enough said."

No, not really.

I don't get the use of this phrase. It feels smug, as if there's some special understanding that everyone is just supposed to "get", or else you're just not with it.

Either say something, or don't, but please don't play coy in telling me that you're not going to tell me something.

<comment><USER5> Hi there Peldi here from Balsamiq. You know, you're totally right. I wasn't happy about how it came out either. Using it was a lazy move. Let me go back and expand it.

<comment><USER11> You're really good at customer service.

<comment><USER12> Maybe that's how he started with $10,000 and built a $800,000 company in only 10 months. Dude's like a customer service (and marketing) machine.

I argued with a friend the other day that his success was just a perfect storm of such things as "great idea", "great implementation", "lucky timing", and a few other things.

But on reflection, and talking it out, we came to the conclusion that his obsessive attention to customer service and business transparency where the true multiplier that basically got him to the freedom line in less than a year instead of the 5 years it should have taken him.

</comment></comment></comment></comment><comment><USER13> He mentions using Typinator. Does anyone know how it compares to TextExpander or TypeIt4Me?

<comment><USER14> I've used TextExpander and Typinator. I'm currently using Typinator because it seems to perform better. I remember I was getting some freeze ups with TE a couple years back (don't know if that's still the case).

</comment></comment><comment><USER11> Nice to see him recommend and praise GetSatisfaction and showing that although they've received quite a beating recently, 37signals/etc, there are customers out there that actually find them useful.

</comment><comment><USER15> Thanks for the post. I don't have a job so this is how I find out what people in the industry use, and what's tried and true.

Also pickledonion's articles was how I moved from hosting on an xp home box to Linux.

</comment><comment><USER16> Interesting sets of tools. I have one recommendation though...check out Grooveshark instead of Last.fm and let me know what you think :)

<comment><USER17> Lala.com is pretty good too.

</comment></comment><comment><USER18> That's quite a comprehensive list. Cool to see Screenr on there in addition to Screenflow.

Have you tried Google Quick Search Box? (<http://code.google.com/p/qsb-mac/>) It does all the same things as Quicksilver, but has always seemed snappier to me.

<comment><USER19> I would love to know why there seems to be such variation in QSB performance. It has been nothing but a dog-slow pile of fail for me and everyone else I know who has tried it, but apparently it works just fine for some people. Quicksilver was always fast for me and now that it is no longer being maintained I have found LaunchBar to be a nice alternative if you don't mind paying a couple of bucks.

<comment><USER20> Agreed. On Leopard QSB was much faster than spotlight. However, when I upgraded to Snow Leopard, it got 10x slower. Funny thing though... Spotlight got significantly faster. So I'm not using just Spotlight.

</comment></comment></comment><comment><USER21> Great write up and list, thanks for sharing, very interested Typinator. I might have to check that out soon.

We use a lot of the same tools. We ended up going with Presently over Yammer, but I bet we would be happy with either. It has been a great way to keep a remote team on the same page.

<comment><USER22> "very interested Typinator. I might have to check that out soon."

This is the program I've always wanted but didn't realize I want. I already bought it.

</comment></comment><comment><USER23> Interesting tools, Typinator looks very nice.

For me, a very simple tool: iTunes

good music = less stress, relaxed mind, hit those milestones.

laugh if you wish.....

</comment><comment><USER24> What tools do you use?

 _IBM_

or

 _Microsoft_

or

 _Java_

Can you see the attraction to one-size-fits-all solutions as opposed to the bewildering panoply witnessed above?

<comment><USER25> Sure, if someone was able to make all those tools as well as the start ups. Half of those things IBM or Microsoft doesn't even have a product for.</comment></comment>


Google+ users are 88% male - <AUTHOR> http://www.breakingcopy.com/google-plus-men-women

<comment><USER2> As far as I know, I'm 100% male.

</comment><comment><USER3> Most of the invited users are from the overwhelmingly male tech community so these stats have little to do with the product.

</comment><comment><USER4> This article is incredibly biased. It's hard to believe that he actually lists the fact that advertisers are not allowed in yet. It's an early test version of the product, and no one other than the most persistent can get in at this point. It's as if a nervous fb fan boy wrote this out of envy at all the positive buzz around g+.

</comment><comment><USER5> 88%+11%+2% = 101%

That's the real shock. Double check your graphs before posting.

<comment><USER6> I am not defending the conclusion of the article... but this is easily explained by rounding... not necessarily a fault of the author's. Although the author probably should have gone one further sig fig</comment></comment>


Trails Made by Starlink Satellites - <AUTHOR> https://www.iau.org/public/images/detail/ann19035a/ <comment><USER2> Here's a statement [1] by one of the largest upcoming surveys, LSST. At least for them, this isn't going to be too much of a problem as,

1) Exposures are fairly short and frequent and so pixels contaminated by these satellites can be easily rejected. 2) There is a huge amount of resources (relative to other projects) going into the image pipeline for this project.

I don't actually know in details how much this will affect other types of observations though. [2] is the IAU's statement (that goes with that image), which points out some types of observations that might be more affected.

[1] [https://www.lsst.org/content/lsst-statement-regarding- increa...](https://www.lsst.org/content/lsst-statement-regarding-increased- deployment-satellite-constellations) [2] [https://www.iau.org/news/announcements/detail/ann19035/](https://www.iau.org/news/announcements/detail/ann19035/)

<comment><USER3> It's great that the LSST can survive this, but there's not much point to building it if all the spectroscopy follow up is killed by efforts to put facebook in front of every face in the world.

Spectroscopy is the _fundament_ on which modern astronomy is built, and it inherently takes long times even with big mirrors. Getting a satellite in the field of view as you take your spectrum most likely means having to start over, and you run out of usable darkness faster.

<comment><USER4> If it can be cheap enough to send 10s of thousands of internet satellites into orbit every 5 years perhaps the same innovation will help drive cheaper space based observance where light pollution, radio pollution, satellite/airplane streaks, and usable darkness aren't a problem.

<comment><USER3> No, you either do spectra slowly or with large mirrors, no amount of ad spend will change the physical reality.

<comment><USER4> What about space implies fast? Sitting equipment at L2 is all the rage these days and SpaceX has launched there before.

Not that I think this particular case wouldn't work in LEO, just for things that wouldn't there are much better options - we just have historically had trouble getting there affordably/reliably enough to make it worth while.

<comment><USER3> > What about space implies fast?

Oh, sorry, I though you were going to argue that we don't have to put science on hold just to let all the world stream gagnam style via Elon owned infrastructure.

In any case, a single SpaceX heavy costs about the same as 9 2.5 m telescopes, so good luck making the argument that science won't suffer.

If you put a science instrument in space, you do it because it's the _only_ option, not because Elon says so.

<comment><USER4> Do you actually want to talk about the real world impacts of this or have you already made up your mind? What your saying is going against what the captions on the image from the IAU site point to and seems to have more to do with your personal opinion of Elon than the actual merits of space based astronomy or the impacts of satellites on ground based astronomy.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> "Although this image serves as an illustration of the impact of reflections from satellite constellations, please note that the density of these satellites is significantly higher in the days after launch (as seen here) and also that the satellites will diminish in brightness as they reach their final orbital altitude."

<comment><USER6> Also, not mentioned, satellite trails are only an issue just around twilight and only near the horizons, both of which are bad seeing conditions for telescopes.

<comment><USER3> "Just" being several hours, cutting shorter the finite amount of dark time available.

<comment><USER6> Several hours at increasingly lower inclinations as satellites in LEO reflecting sunlight are low in the sky. Telescopes generally don't point in that direction because of the amount of atmosphere makes for bad seeing.

</comment></comment></comment></comment><comment><USER7> It's worth pointing out that airplane and satellite trails have long been a problem in astrophotography, and there's many techniques that exist to remove them in the stacking process.

</comment><comment><USER8> Damn it, can't the entire country be wired with fiber? It was done 100 years ago with heavy conductors for electrification, then again with twisted copper pair for phone, but now in the 21st century no one can do shit.

Save wireless for what it's actually useful for, mobile and extremely remote use.

<comment><USER9> The answer is No because your local friendly NIMBY will kill such a project for some asinine reason.

<comment><USER10> I tried laying fiber to an island in Canada. The locals were ecstatic about it, the telco did what they could to block it, then, when it was clear we were going to succeed they finally put in minimal broadband for the one village within the area we were going to cover killing all economies of scale that we might have found.

NIMBY had nothing to do with it.

<comment><USER11> You could've signed up the people in advance into a yearly contract before even starting the build (with fair clauses that they can opt out at no charge if the service turns out different than expected, etc).

<comment><USER12> It doesn't make sense for the average individual homeowner to sign something like this. It's also very difficult to do a fair clause for service quality that works at scale for a contended residential service.

Also a one-year contract still doesn't protect you from them setting predatory pricing for expiring contracts the next year. A large telco can play many games of attrition that are very hard for a small (non-VC) startup to win.

</comment></comment></comment></comment></comment><comment><USER13> Read an article saying that we might not see stars anymore...'gazillions' of satellites will be launched in space over time.

<comment><USER14> I live in a big city and never see the stars now.</comment></comment>


Model of COVID19 outbreak hospital resource use in US - <AUTHOR> https://covid19.healthdata.org/projections <comment><AUTHOR> Note the caveat: ' The projections assume the continuation of strong social distancing measures and other protective measures.'</comment>


Show HN: Stash Belts, for hiding USB keys and SD cards - <AUTHOR>

Stashbelts is a physical product startup that aims to make the best moneybelt you can buy.
The product is a regular looking belt with a slot for a USB key, and a zippered compartment in the back that can hold SD cards, documents or cash. I use one as my main belt, no complaints.
A friend of mine started this company after living in Kenya and finding cheap leather and good craftsmanship.
I haven't told my friend yet that I'm submitting this, but I'm sure he'd appreciate feedback on the site and how to market the belts.
http://www.stashbelt.ca/ <comment><AUTHOR> Also, a question: does anyone know of a really thin brands of USB keys, that isn't any wider than a normal USB stick?

Most USB sticks are fairly fat compared to the connector. This makes for a slight bulge in the stash belt.

<comment><USER2> I use one of these: <http://salestores.com/stores/images/images_747/FCRMRG2.jpg> with a 32Gb Class 10 Samsung micro sd card.

<comment><USER3> Does the "Kingston" part on the left slide over to cover the contacts?

<comment><USER2> Nope, it's just like that no moving parts.

</comment></comment></comment></comment><comment><USER4> You may want to reduce the speed of the slider on the homepage. It moves too fast to be able to read the text on the slide comfortably.

Alternately, if you want to keep the current speed, you could reduce the text to just a heading.

</comment><comment><USER5> If you brand it in a visible way it will self depreciate itself very quickly if it is ever a success in the first place.

</comment><comment><USER6> Any chance of more than 4GB?</comment>


The secret cost of pivoting to video - <AUTHOR> https://www.cjr.org/business_of_news/pivot-to-video.php <comment><USER2> > But advertisers don't need publishers: Advertisers long ago realized they > can pay Facebook directly for traffic.

That's a very good point.

<comment><USER3> or "advertisers don't need publishers, they can pay the top publisher of the day for traffic"

that's the same argument since yahoo was the top publisher. but the world keep spinning, and you can't reach anybody under 35 via Facebook anymore.

</comment></comment><comment><USER4> Video playback is also trash on most sites besides youtube when on mobile.

I can be on excellent wifi connected to excellent internet where a desktop browser gets to play the video almost instantly, but mobile chrome will barely show the video. It'll show me empty black boxes, maybe a disabled video position control. It'll sit there like that for a good 30 seconds, then maybe play a 30 second ad, then finally the 1 min clip I was linked for. Often it will again stutter hard when it gets to the clip.

By contrast, loading a mobile youtube page will play the video almost immediately, if not within 15 seconds, even on LTE.

<comment><USER5> I agree with you with one exception: when a youtube video is preceded by an advertisement. Youtube videos load very quickly, but I have had to give up watching some videos because the 30-second advertisement gets stuck in loading mode. I assume this is because there is a contractual obligation to show the ad at a minimum resolution, while the video itself can be delivered at a low quality and improve depending on the connection.

</comment></comment><comment><USER6> The company I contract for uses video. But it wasn't a pivot, CNN directly subsidized the genesis of Great Big Story. We have an army of video-makers and are just blowing up, having received a $40MM infusion from CNN earlier this year.

It's the exception that proves the rule. If you want to make money with video, bring your checkbook.

</comment><comment><USER7> Everyone who pivots to video to try to drive Facebook traffic seems to come to regret it.

</comment><comment><USER8> A counter example, i.e. a pivot to video that was done correctly, would be the motor trend network of publications.

</comment><comment><USER9> Excellent article, but it will not be read by people being in charge of publishing houses, sadly.

<comment><USER10> They lost all credibility when they said Facebook and Google collect 99% of all revenue growth. Which is designed to be as deceptive as possible.

It suggests but does not mean "They keep all that money", "Nobody else gets any money", or even "They are the only thing that's growing".</comment></comment>


Ask HN: Have there been any layoffs yet due to the virus impact? - <AUTHOR> <comment><USER2> [https://www.google.com/search?q=covid+layoffs&ie=UTF-8&oe=UT...](https://www.google.com/search?q=covid+layoffs&ie=UTF-8&oe=UTF-8&hl=en- us)</comment>


NYT: Trump Endorses Turkish Military Operation in Syria, Shifting U.S. Policy - <AUTHOR> https://www.nytimes.com/2019/10/07/us/politics/trump-turkey-syria.html <comment><AUTHOR> Screwing over (and killing many of) our Kurdish allies who fought so long and hard against authoritarian states, and were actually winning.

I can't wait to see this fucker impeached...</comment>


Death and Resurrection of an SSD - <AUTHOR> http://rentzsch.tumblr.com/post/12955002148/death-and-resurrection-of-an-ssd <comment><USER2> Ugh. Rentzsch is a braver person than I.

I wonder how many days of backups he keeps. If his drive were to start silently corrupting data again, how long would it take him to figure it out? Even if he did have an extensive archive, hunting through it to find good versions of all of his files would be a royal pain.

I'm also curious how stressdrive handles the various levels of caching between it and the disk. I guess the expectation is that the cache is not nearly as large as the entire drive?

</comment><comment><USER3> Btw, a quick way to write random data to your disk is with "dd":

        dd if=/dev/random of=/dev/your_disk_dev bs=1m

Use "bs=1M" on linux. It read back the data like his stresstest program does, but it's really about as "close to the kernel" as you can get without writing your own C code (and you probably aren't going to do anything faster or better than dd already does).

<comment><USER2> EDIT: I see now that you say "It read back the data like his stresstest program does" and probably mean "It _won't_ read back data..."

While your command is a good way to quickly write random data to a drive, note that it doesn't match the functionality of his stressdrive program, which keeps a checksum of the random data it wrote, and then reads back the data from the drive to verify the checksum.

Not that that would be hard to do with a few pipes. Note that the docs for stressdrive[1] mention this:

        "Pshaw! I could do this with dd, /dev/random & shasum!"
         Indeed you could. I prefer a minimal focused tool whose
         operation is fixed, its source simple+readable and
         offers good built-in progress reporting.

[1]: <https://github.com/rentzsch/stressdrive>

<comment><USER4> How is this different from writing each block's own address to it, and reading them back? That's the test I use to detect errors in block addressing.</comment></comment></comment>


Bug Prediction at Google (2011) - <AUTHOR> http://google-engtools.blogspot.com/2011/12/bug-prediction-at-google.html

<comment><USER2> This was abandoned by Google shortly after launch.

Several previous discussions: [https://hn.algolia.com/?query=bug-prediction- at-google.html&...](https://hn.algolia.com/?query=bug-prediction-at- google.html&dateRange=all)

<comment><USER3> An unmentioned reason why this didn't work overly well is some people and teams use a "bug-fixing commit" to resolve a feature request, which unfairly marks actively developed code as dangerous. Disorganized teams who didn't track feature developments with bugs didn't get that penalty.

</comment></comment><comment><USER4> > In fact, as previously noted, 50% of the Google code base changes every > month.

Is this because 50% of the code base is new projects? Is Google growing exponentially that fast?

<comment><AUTHOR> [https://news.ycombinator.com/item?id=3411134](https://news.ycombinator.com/item?id=3411134)

I think it is per file. 50 % of source files (not sure).</comment></comment>


The Gomboc: The Self-Righting Object - <AUTHOR> http://blog.wired.com/gadgets/2008/02/the-gomboc-the.html

<comment><USER2> Self righting objects have been commonly found in Children's toy cupboards for several generations - cost a lot less as well.

<comment><USER3> This topic appear on hacker news a couple months ago.

<http://news.ycombinator.com/item?id=87471>

As the other guy says, there's no uneven distribution of material, but it self-rights due to the properties of its surface.</comment></comment>


JetBrains releases new code review tool, Upsource - <AUTHOR>

<comment><USER2> This might help.

[https://www.jetbrains.com/upsource/](https://www.jetbrains.com/upsource/)</comment>


Math ∩ Programming - <AUTHOR> http://jeremykun.com/ <comment><USER2> I'd like to find a way to reorganize Math ∩ Programming eventually. As alfonsodev points out, it's a bit hard to find stuff for newer and less mathy readers, though there is a handful of articles aimed at them. [1,2,3,4] As other users point out, they don't like the layout. Suggestions?

[1]: [http://jeremykun.com/2011/06/26/teaching-mathematics- graph-t...](http://jeremykun.com/2011/06/26/teaching-mathematics-graph- theory/)

[2]: [http://jeremykun.com/2014/05/26/learning-to-love-complex- num...](http://jeremykun.com/2014/05/26/learning-to-love-complex-numbers/)

[3]: [http://jeremykun.com/2013/05/11/bezier-curves-and- picasso/](http://jeremykun.com/2013/05/11/bezier-curves-and-picasso/)

[4]: [http://jeremykun.com/program-gallery/](http://jeremykun.com/program- gallery/)

edit: updated font size & article width. I'm considering changing the theme completely.

<comment><USER3> It would be interesting to have a kind of "course" (emailed to subscribers) to discover your blog.

Something like "Every day, a chunk of Math ∩ Programming in your inbox", I would personally be interested at least!

<comment><USER4> Same here! I've had the Math ∩ Programming blog open in a tab on my computer for probably 3 weeks now, trying to get around to reading it. Having it delivered once or twice a week in a logical order would be amazing, and worth paying for.

</comment></comment></comment><comment><USER5> I found this blog time ago in Google, searching with terms "mathematics for programmers" I arrived to the article : Why there is no Hitchhiker's Guide to Mathematics for Programmers [http://jeremykun.com/2013/02/08/why-there-is-no- hitchhikers-...](http://jeremykun.com/2013/02/08/why-there-is-no-hitchhikers- guide-to-mathematics-for-programmers/). Which I'd recommend to read as entry point to this website if you feel intimidated by maths. Nice to find it again :)

<comment><USER6> Previous discussion: [https://news.ycombinator.com/item?id=11129375](https://news.ycombinator.com/item?id=11129375)

</comment></comment><comment><USER7> Jeremy is actually a HN user, soon enough he will be summoned. Kudos to you sir if you are reading this!

</comment><comment><USER8> I'm looking forward to the persistent homology posts. It's a really interesting subject with a lot of potential, but I don't feel that the existing textbooks do a good job explaining the computations.

<comment><USER2> This has been on my todo list for a long time :(

Also, sadly, my rate of progress toward this goal has averaged at around one blog post per year. But hopefully in the coming months (after defending my thesis) I'll have more time to dedicate to blogging.

</comment></comment><comment><USER9> i love this guy's blog posts - not only is he clearly a very smart talented mathematician - he explains concepts in a way that is very easy to understand. I wish i would have found this website when i was getting my master in math. some of the signal processing stuff would have helped a lot.

</comment><comment><USER10> @j2kun: Just curious, any chance you'd do an explainer on the Curry-Howard equivalence[1] sometime? Forgive me if you already have, but a search didn't turn anything up. Seems pretty relevant to math ∩ programming and also of interest to a lot of generalists.

[1] [https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...](https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspondence)

<comment><USER2> I think I have a mildly different opinion about the correspondence than most other people on the internet. That's mostly why I have decided to write about other things. But then again maybe that's a good reason to write about it.

</comment></comment><comment><USER11> This website is doubtlessly one of the best introductions to higher math available online. I've gained a lot by both finding and reading the website posts. A weekly email with some sort of order would be great and I would sign up immediately. Keep it up!!!!

</comment><comment><USER12> QUOTE: << Is it possible to condense high-dimensional data into smaller dimensions and retain the important geometric properties of the data? >>

Like in a non-dimensional number?. For example the Reynolds number in fluid flow.

<comment><USER13> No, the meaning dimension there is different. In the Reynolds number it's a ratio of measures where the units cancel.

Here we're talking about mapping data into a high dimensional space, then trying to project that down to the minimum dimension space that can preserve the relevant information.

In the example we don't have any reason to even think all our dimensions are orthogonal, we're assigning a new dimension for each word, but we know there is a lot of overlap in word meaning.

<comment><USER14> > Here we're talking about mapping data into a high dimensional space, then > trying to project that down to the minimum dimension space that can preserve > the relevant information.

Isn't this what M/PCA is all about? [https://en.wikipedia.org/wiki/Principal_component_analysis](https://en.wikipedia.org/wiki/Principal_component_analysis) and [https://en.wikipedia.org/wiki/Multilinear_principal_componen...](https://en.wikipedia.org/wiki/Multilinear_principal_component_analysis)

</comment></comment></comment><comment><USER15> > A consequence is that, if you're trying to cluster data points by looking at > points within a fixed distance r of one point, you'll have to make r > exponentially large in the dimension.

That does not follow ...

<comment><USER16> His point is the volume of the unit sphere is tiny. The "volume" of the unit 20-cube is 1. The volume of a 20-sphere is just 0.0258. 100-cube, volume is 1. 100-sphere, 2.36e-40

If your algorithm works well for "nearby" meaning 1, i can just keep adding dimensions till you find nothing. If "nearby" on the other hand is related to the number of dimensions, you're going to have to grow the "nearby" value exponentially.

<comment><USER15> You've made the same error as the OP. The volume of a unit n-dimensional sphere decreases exponentially in n. But that doesn't mean you need to increase r exponentially to compensate - in the volume formula, the r also has an exponent of n. The distance between opposite corners of a unit cube in n-dimensional space is root(n). That's hardly exponential.

<comment><USER16> Huh. I always thought n! grew faster than c^n which would be even worse than exponential. Maybe enough cancels out to make it simpler than it appears.

 _edit_

Actually, for even dimensions it's pretty clear. n = dimension/2

        pi^n / n!

factorial wins. The problem is worse than exponentiation.

<comment><USER17> At some point, I wondered if n! is proportional to n^n.

Turns out, it is:

n! ~= (2 * pi * n)^1/2 * (n/e)^n

([https://en.wikipedia.org/wiki/Stirling%27s_approximation](https://en.wikipedia.org/wiki/Stirling%27s_approximation))

<comment><USER18> Not exactly "proportional", because you have that pesky e in the denominator. But that's the right idea. Basically, to get n! you're multiplying together n things that are sort of n-ish, so you'd expect n! ~ n^n. (When the numbers get really big, like in statistical mechanics, I've seen the approximation log n! ~ n log n.) The next step is to figure that you're multiplying together n things that are on average n/2, so n! ~ (n/2)^n. But then it really turns out that you should have been using a geometric average (since you're multiplying), not an arithmetic one, so n! must be smaller yet. (I don't know a way to get (n/e)^n without doing an integral, though.)

<comment><USER17> Right, proportional in the sense that for both n! and n^n, the fastest-growing component is ~n^n. (I was curious in the context of which grew faster for larger values of n in the big O notation).

</comment></comment></comment></comment></comment></comment></comment><comment><USER19> This is really, really awesome. I have a long train ride coming up, so I just put together a quick wget script to get all the pages for when I have zero internet. :)

</comment><comment><USER20> I know it's a HN cliche to complain about form over the message, but I find this impossible to read without first removing the entire left-hand panels (and ::before p-elements) which re-grab my eye every line-break.

<comment><USER5> If you use safari or firefox, reader view is available for the articles.

[https://support.mozilla.org/en-US/kb/firefox-reader-view- clu...](https://support.mozilla.org/en-US/kb/firefox-reader-view-clutter-free- web-pages)

<comment><USER21> The reading view in Edge (and metro IE fwiw) works as well

</comment></comment></comment><comment><USER22> Math ∩ Programming = Math.

Just sayin'. :P

<comment><USER23> That would imply Math is a subset of Programming, when in fact it's pretty clearly a strict superset of Programming.

<comment><USER24> Unless you are a Russian / Markovian constructivist!</comment></comment></comment>


Defund Facial Recognition Before It's Too Late - sneeze-slayer https://www.theatlantic.com/technology/archive/2020/07/defund-facial-recognition/613771/ <comment><AUTHOR> I am not sure what "defunding" would accomplish at this point. At this point, the research has been done, and the products are available for astonishingly low prices. We don't stop oil spills by defunding offshore oil wells that have already been dug, and attempting to do so is likely to result in more oil spills as the money for maintenance is cut. Similarly, I'd expect that if budget for facial recognition is slashed, the parts that will be cut are oversight and training, and so now instead of just dealing with questionable technology, you're dealing with questionable technology used by people who have no idea what they're doing.

Also, as long as public spaces are under constant video surveillance, stopping facial recognition now only solves the problem temporarily. I think at a bare minimum, we need standards for when this evidence should be admissible in court (at current tech levels, probably approximately never) and when it is acceptable to use it in searches. The technical ship has sailed, so any fix is going to have to be legislative at this point.

<comment><USER2> "> Amazon, Microsoft, and Google have continued efforts to ensure federal regulation that offers a stable and profitable market in which facial- recognition technology is, in fact, used by law enforcement, in direct opposition to the movement the companies claim to support."

if these Tech companies were serious about fighting inequality it would be more effective to start banning racist AI/facial recognition instead of scrubbing Technology language from words like "blacklist", and "master/slave".

</comment></comment><comment><USER3> ...I will have to hide my elongated skull now...

</comment><comment> core-questions > Rooted in discredited pseudoscience and racist eugenics theories that claim > to use facial structure and head shape to assess mental capacity and > character, automated facial-recognition software uses artificial > intelligence, machine learning, and other forms of modern computing to > capture the details of people's faces and compare that information to > existing photo databases with the goal of identifying, verifying, > categorizing, and locating people.

I'm sorry, what? Does this person think that phrenology / physiognomy, two old pseudosciences that have been discredited for a hundred years or more, are actually at play within ML systems?

I'm totally willing to believe that ML facial recognition systems insufficiently trained on a wide enough set of faces will mistake one person for another. Sure. But to pretend that the system is based on eugenics belies a critical lack of understanding of what these things actually do, and ascribes agency and racial animus to a computer program. Pretty clear to me that this shows the author doesn't really know anything about how these things work.

The reason to not want facial recognition in public spaces is the same as to not want mass surveillance: a reasonable expectation of privacy by citizens. Of course, if also want no police in these areas at the same time, one should not be surprised if eventually they go from peaceable public squares, to a haven for petty crime, to eventually a fearful place people avoid.

<comment><USER4> > I'm sorry, what? Does this person think that phrenology / physiognomy, two > old pseudosciences that have been discredited for a hundred years or more, > are actually at play within ML systems?

[https://callingbullshit.org/case_studies/case_study_criminal...](https://callingbullshit.org/case_studies/case_study_criminal_machine_learning.html)

This _shouldn 't_ be a problem, because as you note phrenology is pseudoscience that's been discredited for over a century. And yet.

To the broader point, I'd argue that in generally any attempt to predict criminality from facial structure/face picture is phrenological in nature. And people who do know what's at play with ML systems _do_ agree with this take: [https://twitter.com/ylecun/status/1276147230295166984](https://twitter.com/ylecun/status/1276147230295166984)

<comment> core-questions > And yet.

Well, what's happening there is not a study of phrenology at all (which posited specific regions of facial/skull structure being indicators). It's actually a very interesting thing to look at. There was a previous one that was reporting some degree of success in determining whether someone was homosexual via ML.

Here's the thing: if this turns out to have actual predictive power, then it's a subject worthy of scientific study, whether you like the outcomes and conclusions or not. Plenty of other worthy areas of endeavour (e.g. psychometric IQ research) have also revealed uncomfortable truths. If instead these things turn out to not have any legitimate research value (i.e. can't make predictions that can be experimentally verified), then we can stop looking at them, but as long as they continue to maintain a relatively consistent relationship to observable reality, they're as worthy a form of science as any other anthropological research is.

We have a choice either face this head on, include it in policies, and build our sociology around the truth, or we can put our heads in the sand to make people feel better. I for one believe that truth is far more important than feelings, and that if we had continually given higher credence to feelings the Enlightenment and most scientific progress we've had would have been far slower if it had happened at all.

<comment><USER4> > one that was reporting some degree of success in determining whether someone > was homosexual via ML

Yes, which was also phrenological in nature.

> one that was reporting some degree of success in determining whether someone > was homosexual via ML

I've yet to see any "uncomfortable truth" from psychometric research that wasn't relatively easily explained as culturally tied.

> but as long as they continue to maintain a relatively consistent > relationship to observable reality

The point is that neither this study nor the homosexuality study have a relatively consistent relationship to observable reality. Your priors on us being able to predict, independent of social conditioning, some arbitrary social attribute based on someone's face should be very, very low.

And "predicting" some arbitrary social attribute based on social conditioning is just encoding social bias into the model, which is bad.

> and that if we had continually given higher credence to feelings the > Enlightenment and most scientific progress we've had would have been far > slower if it had happened at all.

You mean like all the scientific progress that came out of phrenological research?

<comment><AUTHOR> > Your priors on us being able to predict, independent of social conditioning, > some arbitrary social attribute based on someone's face should be very, very > low.

What does "independent of social conditioning" mean here? Can you give some examples of social attributes that arise independent of social conditioning?

<comment><USER4> > What does "independent of social conditioning" mean here?

So we know, for example, that if you train a model to predict "criminality" based on face here in the US, it will find that race is a strong predictor of criminality.

The first problem with this specific example is that the data is biased: certain communities are overpoliced. We know, for example, that black and white people use marijuana at around the same rate, but that black people are more likely to be arrested for use. So they'll be more likely to be represented in a dataset of "criminals" even if they aren't actually more criminal. So that's one social factor. But let's pretend that we can construct a socially untainted dataset that represents the true underlying crime rate, we correlate it with face images, and the racial disparity still exists. I want to reiterate that we're well off into the world of fantasy here, but for demonstration purposes.

There are generally 3 conclusions you can draw from a correlation like this: A directly causes B, B directly causes A, or something else more complex is at play. It's unlikely for facial structure changes to directly cause criminality, and unless you're Pinocchio, criminal behavior isn't going to directly cause changes in your face.

So what more complex thing is at play? Well one answer is genes. It could be that the genes that make someone darker also make them more naturally predisposed to violence. That is, some factor C directly causes both A and B. Or it could be even more complex, for example that socially, people who exhibit black skin are more likely to be placed in conditions that breed criminal behavior[0]. Since economic and social status are heritable, and so is skin color, this seems reasonable to conclude, and there's lots of other evidence that this is the case.

But if that's the case, then looking at a picture of a face doesn't actually have predictive power. At best it just recognizes that your average black person is likely to have been raised in a situation where they were more likely to commit a crime. It doesn't have any predictive power about a black person who wasn't raised in those conditions.

So by independent of social condition, what I mean is that such a model isn't useful unless you ascribe to the belief that the genes that cause facial structures are correlated with the genes that cause criminal behavior (which presupposes that those genes exist).

Otherwise you aren't actually looking at even a direct correlation and in fact it's very likely that if you divide your subpopulation up in smart ways, you'll find that there are groups against which you are unfairly biased.

Just because the social signifiers in the study we're looking at aren't as obvious to you or I as skin tone doesn't mean they aren't there, and again that presumes the data is good, which we know it isn't.

> Can you give some examples of social attributes that arise independent of > social conditioning?

I don't know that I fully explained this above, so let's create another fantasy world to explain this more concretely. Let's agree that murder is bad. This is solely a social agreement, but we decide on it based on ethical beliefs.

Imagine that in this fantasy world there are genes that cause one to occasionally enter a bloodlust that forces one to go on a relatively uncontrollable killing rampage. Or for a more direct fantasy example, turn into a werewolf that then goes into a relatively uncontrollable killing rampage. Or be an Orc which is "naturally" evil (this is actually a relatively common fantasy trope, huh).

This genetic marker would imply a genetic predisposition to criminal behavior, despite any social conditioning. Compare this to a relatively normal human child who is trained from a young age that they are "no one", a part of a greater movement that requires assassinating the wrong people, and that this assassination is sometimes necessary for the greater good.

Both are more likely than your average person to commit a crime, but one was conditioned to this socially while one was genetically predisposed. If this cult holds onto family lines, there will be similarities between cult members, but people who escape the cult or who were never a part of it might be unjustly thought to be criminal, solely because they resembled the cult.

To jump back to the original statement, this means that if you set up a confusion matrix that includes "family of cult members" as a category, your model will perform badly and discriminate negatively agains them. You can see why this might cause huge issues like, for example, causing the justice system to chase or harass people who are related to criminals.

Hopefully that explains. It's essentially a correlation vs. causation issue.

[0]: I should note here that this is true whether you ascribe to the belief that "black culture" encourages/celebrates criminality, or the belief that "white supremacist and social structures" push black people into situations where they can't avoid crime.

<comment><USER5> > There are generally 3 conclusions you can draw from a correlation like this: > A directly causes B, B directly causes A, or something else more complex is > at play. ... Hopefully that explains. It's essentially a correlation vs. > causation issue.

Note that for predictor it does not really matter correlation vs. causation. That matters for intervention. If you have feature A that in the population always causes features B and C, and no other causes them, then presence of B is perfect predictor of C, but intervention that changes B (and not A) does not affect C.

On the other hand if feature D causes feature E in 90% cases, and no other causes it, then D is predictor of E with 10% false positives, but intervention on D affects E.

It is true that 'perfect causality' predictors, where the set of accounted factors are only causes of a predicted feature, have advantage that they work the same for any subset (or any change) of population. While predictors that ignore some causal factors (like the predictor from the first example that ignores common causal factor) may have vasly different probabilities for subset (or change) of population (when distribution of ignored factors change). But in practice most real-world causal networks are super complex and many real-world tests ignore many causal factors. So it is kind of isolated demand for rigor [1].

[1] [https://www.lesswrong.com/posts/fzeoYhKoYPR3tDYFT/beware- iso...](https://www.lesswrong.com/posts/fzeoYhKoYPR3tDYFT/beware-isolated- demands-for-rigor)

<comment><USER4> > Note that for predictor it does not really matter correlation vs. causation. > That matters for intervention.

Yes, but you don't build a model without intending some intervention, so this distinction is irrelevant in the context of applied ML, although it is a true statistical fact.

> So it is kind of isolated demand for rigor [1].

I don't see how. The isolated demand for rigor "fallacy" that Scott Alexander talks about is when you ask for rigor to be applied in ways that are bad faith. Let me rephrase my concern:

We have lots of evidence that real-world causal networks are super complex and many real-world tests ignore many causal factors. Similarly, we have no a-priori reason to believe that facial structure is correlated with {sexual orientation, innate level of intelligence, innate criminality}. And in fact we have strong reasons to believe that most of the way that those things present is due to cultural influence.

So if someone shows up with a groundbreaking study that shows that they can predict some innate attribute based on facial structure, it's likely that they're actually seeing cultural biases (or cultural correlations) and not innate factors (or correlations with genotypic things). In other words, our priors should be that any model in this space is simply discriminating based on stereotypes, not revealing some innate way to "predict" these attributes.

The model isn't a _predictor_ but a _recognizer_ , and while semantic, that's a very important distinction.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> It's too late.

<comment> core-questions It was too late even before facial recognition technology existed, because it's not like you can't apply modern tech to older video footage.

</comment></comment><comment><USER7> Sorry, I'd much rather defund The Atlantic. Facial recognition at least has valid uses beyond surveillance. The Atlantic has no known good uses that I can discern.</comment>


Britain and America dominate list of best universities. - <AUTHOR> http://www.timesonline.co.uk/tol/life_and_style/education/student/news/article660845.ece I was surprised to see the amount of British universities at the top. <a href="http://www.topuniversities.com/worlduniversityrankings/top20..." rel="nofollow">http://www.topuniversities.com/worlduniversityrankings/top20..., Caltech guys will be happy to see they got ahead of M.I.T.<p>And we Brazilians are just happy to see that USP and Unicamp got into the list (175 and 177). I do believe, however, that Unicamp got really impressive scores for quality of faculty.

<comment><USER2> This will merely reflect reputation. How many people in academia have been at Harvard long enough to have an actual opinion?

Question: if China built the best university in the world today, how long would it take to reach number 1 on the list? I think decades. So much for having a yearly list.

I looked for details, from <http://www.thes.co.uk/worldrankings/story.aspx> . It requires registration, but here's a part of the method section:

 _"Our rankings contain two strands of peer review. The more important is academic opinion, worth 40 per cent of the total score available in the rankings. The opinions are gathered, like the rest of the rankings data, by our partners QS Quacquarelli Symonds (www.topuniversities.com) which has built up a database of e-mail addresses of active academics across the world. They are invited to tell QS what area of academic life they come from, choosing from science, biomedicine, technology, social science or the arts and humanities. They are then asked to list up to 30 universities that they regard as the leaders in the academic field they know about, and in 2007 we have strengthened our measures to prevent anyone voting for his or her own institution.

This year we have the opinions of 5,101 experts, of whom 41 per cent are in Europe, the Middle East and Africa, 30 per cent in the Americas, and 29 per cent in the Asia-Pacific region."_

The rest of the score is split:

-10% which university do recruiters favor

-20% publication count

-20% student stuff, like teacher-to-student ratio (the only example they give)

-10% I couldn't find.

So there's definitely two selection effects: how did they collect their addresses, and who decided to answer a Times Online survey. I'm pretty sure my advisor wouldn't do it. Too busy.

Furthermore, they don't give the detailed methodology, and they admit to changing it since the first published results.

Overall, it seems like an honest effort, but what's the point? Would you advise a young person to make a decision based on this ranking?

</comment><comment><USER3> In a study done by English speaking people, is this result really a surprise?

<comment><USER4> "The rankings were based on a survey for the THES of 3,703 academics worldwide..."

<comment><USER5> Not sure how this quote disproves the previous statement (English speaking bias).

<comment><AUTHOR> How many people in the academia _do not_ speak English?

<comment><USER6> Not sure how many.

But I know a significant amount of scientific papers are published in other languages than English, e.g. in Chinese language on Wanfang Data (www.wanfangdata.com - affiliate of Chinese Ministry of Science and Technology). This is probably the case in many other non-English languages.

</comment></comment></comment></comment></comment><comment><USER7> _In some countries this is the result of a deliberate policy. The German and Dutch governments, perhaps from fear of elitism, try to ensure that all universities are roughly equal in quality._

<http://www.paulgraham.com/america.html>

</comment><comment><USER8> "Harvard, whose endowment of $26 billion (GBP13.8 billion) exceeds total annual funding for all British universities"

that gives a powerful indication of the sheer scale of the imbalance between the resources at the disposal of UK and US education institutions.

<comment><USER4> Except that it's comparing a capital sum with an annual expenditure, which is rather misleading. A capital sum will ordinarily be around 20x as big as the annual income it generates (though no doubt Harvard gets better returns than the average nonprofit).

</comment></comment><comment><AUTHOR> I was surprised to see the amount of British universities at the top. [http://www.topuniversities.com/worlduniversityrankings/top20...](http://www.topuniversities.com/worlduniversityrankings/top200) . Also, Caltech guys will be happy to see they got ahead of M.I.T.

And we Brazilians are just happy to see that USP and Unicamp got into the list (175 and 177). I do believe, however, that Unicamp got really impressive scores for quality of faculty.

<comment><USER4> Can you post the list? It's so lame that they make you create an account to see it.

<comment><AUTHOR> <http://top200.jottit.com/>

<comment><USER9> Wow... not one mention of the Indian Institutes of Technology which may just be the toughest engineering schools on the planet. Just have a look at their entrance exams...

Bias anyone?

<comment><USER10> High entrance requirements are only one of the factors which define a good university. I'd say that the quality of teaching and the quality of research performed are more important than the quality of incoming students.

<comment><USER11> The IITs have a good _undergraduate_ program, one of the best in the world. The graduate and research output of the IITs is nowhere as good. In fact , the research in particular is fairly mediocre, compared to say Stanford or MIT.

</comment></comment></comment></comment></comment></comment><comment><USER12> No Indian university?

<comment><USER9> Apparently not...

</comment></comment><comment><USER13> woot - no 2 and no 5!</comment>