PostgreSQL Performance on Raspberry Pi - <AUTHOR> https://blog.rustprooflabs.com/2019/04/postgrseql-pgbench-raspberry-pi <comment><USER2> I use postgres on my raspberry Pi 3B as my development server (along with PostgREST and Nginx) and it has served me faithfully for more than a year. Many people are surprised to see the setup and how fast it is to spin up a backend server. The response time is also pretty good and it's amazing to see such a tiny machine pull off such a feat.

<comment><USER3> I run Postgres for Miniflux, both on Raspberry Pi3. The Postgres images on Docker Hub officially support armv7. It could not be any easier. Of course, make sure to back up often. As others have said, SD cards can fail.

<comment><USER4> I really hope the RPi4 has eMMC or M.2

<comment><USER5> YES! And also move entirely to USB-C for all connectors.

Even if they made this another $10-15 more expensive, and treated it like a hi-end version, I'd bet they could never make enough of them.

</comment></comment></comment></comment><comment><USER6> I love postrgres, and obviously this is just a test, but for anything where I would use a pi sqlite is insanely good.

I use sqlite for full fledged websites with millions of rows with no issue, fast, easy to deploy, easy to move.

<comment><USER7> Sqlite is amazing performant and scalable, as long as you don't need high availability or horizontal scaling. While I don't recommend using sqlite in production, I have done it before and had a similar experience. Millions of rows with excellent performance. Just hope you never have to scale beyond what a single machine can handle tho :-)

<comment><USER6> Agreed, I use SQL Server in my day job and always reached for PG when doing OSS stuff, was just blown away how well SQLite managed to hold up.

Generally sharding/HA/DR are difficult problems with any db tech, so yeah, you are definitely right there.

</comment></comment></comment><comment><USER8> Hopefully nextgen raspberry pi will take sata or some other storage interface. I think SD cards are the main problem with the platform right now.

<comment><USER9> ODroids are a great alternative with much better performance and SATA for not much more money (~$55).

<comment><USER10> And non-unique MAC addresses and lower stability, at least when we tried to use them a few years ago. Bought 3 and they all had the same MAC.

<comment><USER11> I've run into the MAC problem, but found it easy enough to deal with via my standard home Chef configs. I can't echo the stability issues, though; my two have been ticking away happily for a couple years.

</comment></comment></comment></comment><comment><USER12> For comparison, I ran this on a $80/mo OVH dedicated server (SP-32 Server - E3-1270v6 - 32GB - SoftRaid 2x450GB SSD NVMe)

      $ sudo su postgres
      $ createdb bench_test
      $ pgbench -i -s 10 bench_test
      $ pgbench -c 10 -j 2 -T 3600 -P 60 bench_test
      starting vacuum...end.
      progress: 60.0 s, 9192.5 tps, lat 1.088 ms stddev 0.924

<comment><USER13> So the performance was lower on the server than the pi?

<comment><USER14> Not sure how you're seeing that.

From the article it says the incremental reporting (every 60 seconds) were:

> progress: 540.0 s, 171.8 tps, lat 55.105 ms stddev 946.851

> progress: 600.0 s, 24.6 tps, lat 435.693 ms stddev 2945.727

> progress: 660.0 s, 405.8 tps, lat 24.108 ms stddev 134.692

So they were getting between 24.6 and 405.8 transactions per second.

On the server, he's seeing:

> progress: 60.0 s, 9192.5 tps, lat 1.088 ms stddev 0.924

So the server is doing 9192.5 transactions per second. The test wasn't run as long, but the server is showing much lower latency and standard deviation, as well.

</comment></comment></comment><comment><USER15> Redis is quite fast on the Pi3, completely another story compared to the original Pi. With AOF enabled and fsync policy every second, without using pipelining it does 28k ops/sec. With pipelining it reaches 80k easily. Not bad for a hardware that is quite cheap and limited. About the problems with the SD card, in the specific case of Redis there are mitigations _if_ it is possible to give up on durability. One could use snapshotting and configure a delay that will not trash the SD too much. I think PostgreSQL does a lot of random access in the on-disk btree, but other systems using some log structured storage in append only should create less problems in theory.

</comment><comment><USER16> Is the limit he found CPU bound? IO bound?

I suspect he basically tested some random SD card's performance.

<comment><USER17> That was my first thought. Especially when the article says

> Starting with Postgres 10, the default is to give 2 cores to parallel > processing. Switching max_parallel_workers_per_gather between 2 and 0 had > nearly zero impact on that Pi, either for or against.

With that extra info I'm 99% sure it's I/O limited.

<comment><USER4> It's almost certainly IO. IIRC, Sdcard write speed on the RPi is known to be slow. You can improve it with superior cards, but the USB interface generally remains the path to fastest disk reads and writes.

</comment></comment></comment><comment><USER18> Please for the love of god, do not use Postgres on a SD card, you'll corrupt it quickly, even spinning rust over USB2.0 is faster and more reliable. That's my experience with it at least.

<comment><USER19> How long have we had SD Cards for them to still be unreliable like this? This is unacceptable.

<comment><USER20> You can have high capacity, low cost, or reliability. Pick 2.

Consumer cards choose capacity and cost, because that's what sells. Industrial SD cards pick capacity and reliability, but cost at least 4x more than similar consumer cards due to inherent tradeoffs in the hardware/firmware design.

<comment><USER21> Are they though? I looked on Amazon; the price difference between a 'regular' 32GB microsd and a 'high endurance' card of the same capacity is $3 (7.99 vs 10.99).

<comment><USER20> Consumer high endurance cards are still MLC or TLC. They're better than regular consumer SD cards, but still not what you'd consider ideal reliability. Industrial cards are better QC'd and use more reliable (but less dense) SLC, as well as various smaller changes to operate reliably in extreme conditions. I'm sure some of it is industrial product markup as well, but the technology is fundamentally more expensive.

</comment></comment></comment></comment></comment><comment><USER22> You better have some good redundancy on those SD cards because I've run Pi clusters before and most of those cards go corrupt pretty quickly with any frequent number of writes.

<comment><USER23> after boot, the sd cards for my pis go into readonly :) checkout cattlepi.com on how this is done. i've had good luck with the lifetime of the cards, but it's also true I treat the Pis as disposable.

<comment><USER16> I started building "real" firmware for my raspi appliances using buildroot. That makes it super easy to build images that completely run from initramfs without a single write on the sd card (and are fully booted in a few seconds including userland).

Only works if I don't need any local persistence but makes the Pi just work super reliably. Upgrading is as easy as swapping the Linux Kernel image on the fat32 and rebooting.

<comment><USER23> cattlepi has the advantage of being able to essentially build and distribute images over the network + full raspbian.

on boot it self-updates if needed, builds on overlay filesystem w/ the squashed image as the bottom ro layer + tmpfs as top rw layer.

<comment><USER16> Thanks, cattlepi looks super interesting!

</comment></comment></comment></comment></comment><comment><USER24> I'm wary of running anything with data I care about on a device without ECC RAM.

From the docs:

> PostgreSQL does not protect against correctable memory errors and it is > assumed you will operate using RAM that uses industry standard Error > Correcting Codes (ECC) or better protection.

[https://www.postgresql.org/docs/11/wal- reliability.html](https://www.postgresql.org/docs/11/wal-reliability.html)

Re: IS ECC RAM really needed?

[https://www.postgresql.org/message- id/20070526145214.GA21290...](https://www.postgresql.org/message- email@example.com)

<comment><USER25> You'd be surprised to read about bitrot then.

Well, storage is the one that causes maximum damage there, especially really old data.

<comment><USER24> That may also be a concern, but there are software workarounds. I would also not run Postgres on a non-checksummed filesystem or without data checksums enabled in the postgres cluster.

<comment><USER25> Well, one also needs periodic scrubbing. Without that, it rots!

</comment></comment></comment></comment><comment><USER26> If the storage is handled properly, RPi can do wonders.

I've used RPi1 as a git sever in our company for 5 years with encrypted USB flash drive as storage.

We must have committed at-least a million lines of code into it & we were using it every day.

Except for changing some memory related flags in git, I didn't change much w.r.t to git.

I had another encrypted USB flash drive to which the files were backed up, interestingly this flash drive failed but the primary drive never died till date.

The git server is running 24*7 on RPi1 for past 5 years.

</comment><comment><USER27> cool article!

It would be great to know the SDCard speed and maybe see some comparisons based on the SDCard type since the speeds vary quite a bit

see also: [https://en.wikipedia.org/wiki/SD_card#Speed](https://en.wikipedia.org/wiki/SD_card#Speed)

<comment><USER28> I believe Raspberry Pi's SDCard port is limited to a bit below 25 MB/s, so that's gonna be the actual limit very often.

<comment><USER29> There's also IOPS for random 4k block access, which is usually way below 25MB/s. Typically around 2000/500 r/w IOPS for regular A1 cards. With expensive cards you can get to 2500-3000 range.

Read queries will be cached in RAM and fast (for <1GB database), and write queries will be limited to 500 IOPS, and heavily dependent on how you use transactions and write queries in your database.

Also disabling statistics collecting is the first thing I do when putting a postgresql db on an sd card. Because that causes a lot! of writes.

</comment></comment></comment><comment><USER30> The amount of problems I have with the SD card on the raspberry pi is crazy. The thing is never stable.

<comment><USER31> Change out your SD cards with high performance ones (U 10 or whatever). Mine runs just fine for weeks.

<comment><USER32> Not just high performance ones, high _endurance_ ones.

</comment></comment></comment><comment><USER33> They need to re-test on the grid using different PI for each version. And then re-test using same pi, different SD card.

Beecause it is clear from the bad SD card comment that the SD card and PI can influence the speed of the tests.

</comment><comment><USER34> Would be interesting to see a pgsql vs mariadb Rpi benchmark.

<comment><USER35> Definitely! The author of the article mentions how lower power hardware doesn't let you ignore database anti-patterns quite so much, so I'd like to see comparisons of tuning the 2 different databases.

It's all too easy to dismiss it as an old school engineer being curmudgeonly with a desire to make everyone develop on low-powered hardware. I think there's definite value to telling people to work out the kinks and bottlenecks of their code on a raspberry pi. Especially since those lessons might translate to highly-performing k8s/containers/clusters on lower-power hardware.

</comment></comment><comment><USER36> I've always wanted to make an arm based cluster for kicks and giggles. Just can't find a fun project to warrant it.

</comment><comment><USER37> I wonder how the Asus Tinkerboard and the Pine64 RockPro 64 (RK3399) compare

</comment><comment><USER38> workloads > RAM ≈ 0 ∴ benchmark = premature optimization # ;)

</comment><comment><USER39> I only read the title; but I'm expecting performance generally to be abysmal.

I/O is quite poor on rPIs and that's what PostgreSQL needs more than anything. More than memory or CPU for sure.

The best thing you can ever do for your database is (in this order):

1) Have enough ram to fit your entire dataset (mostly due to joins)

2) Have the best raid card money can buy.

\--

An SD card is shocking enough but on the rPI it's going over a USB bridge which takes CPU cycles too!

Still probably performs better than postgresql on WSL; since the I/O is the major bottleneck there too.

<comment><USER40> For comparison's sake, I ran exactly the test described in the blog post on my house server, which is an AMD FX4130 with 32 GB RAM. The database was stored on SATA SSDs managed by ZFS. Lots of other things were going on at the same time.

pgbench -c 10 -j 2 -T 600 -P 60 bench_test

starting vacuum...end. progress: 60.0 s, 5373.1 tps, lat 1.860 ms stddev 0.709 progress: 120.0 s, 5277.3 tps, lat 1.894 ms stddev 1.383 progress: 180.0 s, 5323.4 tps, lat 1.878 ms stddev 0.733 progress: 240.0 s, 4606.6 tps, lat 2.170 ms stddev 1.508 progress: 300.0 s, 4122.0 tps, lat 2.425 ms stddev 1.517 progress: 360.0 s, 4062.0 tps, lat 2.461 ms stddev 1.532 progress: 420.0 s, 3957.3 tps, lat 2.526 ms stddev 1.598 progress: 480.0 s, 3939.6 tps, lat 2.537 ms stddev 1.599 progress: 540.0 s, 3906.5 tps, lat 2.559 ms stddev 1.627 progress: 600.0 s, 3808.6 tps, lat 2.624 ms stddev 1.747

transaction type: <builtin: TPC-B (sort of)>

scaling factor: 10

query mode: simple

number of clients: 10

number of threads: 2

duration: 600 s

number of transactions actually processed: 2662601

latency average = 2.252 ms

latency stddev = 1.435 ms

tps = 4437.570722 (including connections establishing)

tps = 4437.599312 (excluding connections establishing)

So you get roughly an 11x performance improvement by running on a 7 year old desktop processor and rather cheap SSDs. On the other hand, if we say that the Pi setup cost $50 and we estimate the current value of my server at $500, performance/price is basically linear.

<comment><USER35> Do you know roughly how much power your server draws? Like you said, the cost/performance may be linear but I wonder what happens when you add in power to that equation. Idle speculation, to be sure, but it would be fascinating to see what amount of solar panels plus wind power plus batteries would be needed to power each system.

<comment><USER40> Oh, loads of power. The CPU is spec'd for 125W and would be completely outclassed by a 65W Ryzen 5 1600 (150% cores, 300% threads, each thread probably 2x performance) and the rest of the system likely draws another 100W or so. It's irritatingly reliable and my house server needs really don't encompass a lot of CPU power, so replacing it hasn't been a priority.

<comment><USER35> > irritatingly reliable

This made me laugh. I have an AMD Sempron 3850 cpu+gpu sitting in a small case under the TV that has been chugging along for 5 years now. The somewhat recent addition of a cheap SSD has increased its usability even more. As much as I'd like to replace it with something more up to date, I can't justify it while it's still working fine.

It's funny to think of what is "good enough" for a lot of needs.</comment></comment></comment></comment></comment>


Schneier's take on the alleged backdoor in OpenBSD - <AUTHOR> http://www.schneier.com/blog/archives/2010/12/did_the_fbi_pla.html

<comment><USER2> I'm where he is. Interesting discussion, though -- and it really highlights the limits of "with many eyes all bugs are shallow".

</comment><comment><USER3> he didn't add anything new to the discussion but his opinion. Crypto scholars are excellent at cryptography and security theory, but when it comes to actually implementing secure systems (exception being crypto algorithms), and securing systems, Crypto scholars are horrible. For example he mentions that it would be better to just find an existing vulnerability instead of planting an FBI backdoor in the OpenBSD code: good luck Schneier, obviously you don't know that much about OpenBSD security culture and history. Plus the NSA has a history of putting backdoors into solutions. This is just my opinion from experience.

<comment><USER4> Bruce Schneier isn't some random academic. He's extremely highly respected, and is the Chief Security Technology Officer of BT Communications. He has _tons_ of experience with securing systems in the real world, and to say he "obviously [doesn't] know that much about OpenBSD security culture and history" is crazy.

<comment><USER3> sorry I didn't mean to appear to disrespect Bruce Schneier, I've met him, gone to his book signings, own all his books, I even buy his books for gifts to my friends. I'm a huge fan of his work. We need people like him who have done highly advance studies in the security field; he is the best and an amazingly lucid writer. I never said he was a random academic or that his overall research should be disregarded. I really don't think you understand my original comment. Theory is not practice; they are two separate things. I guess I should have just said that. do you see my point now?

<comment><USER5> I am not so much a Schneier fan, so if you feel like you need cover for leveling any kind of criticism against anything he says, don't worry too much. Are you _sure_ you believe Schneier would know anything about the code quality of a specific IPSEC implementation --- or really, about the _code quality_ of any IPSEC implementation?

<comment><USER6> Alright, I will bite. What are your reasons for not liking Bruce?

(I can't wait for this)

<comment><USER5> I don't so much "not like him". But, compare cite records:

[http://scholar.google.com/scholar?q=Serge+Vaudenay&hl=en...](http://scholar.google.com/scholar?q=Serge+Vaudenay&hl=en&btnG=Search&as_sdt=400001&as_sdtp=on)

[http://scholar.google.com/scholar?q=hans+dobberton&hl=en...](http://scholar.google.com/scholar?q=hans+dobberton&hl=en&btnG=Search&as_sdt=400001&as_sdtp=on)

[http://scholar.google.com/scholar?hl=en&q=eli+biham&...](http://scholar.google.com/scholar?hl=en&q=eli+biham&btnG=Search&as_sdt=400000&as_ylo=&as_vis=0)

[http://scholar.google.com/scholar?hl=en&q=bruce+schneier...](http://scholar.google.com/scholar?hl=en&q=bruce+schneier&btnG=Search&as_sdt=400000&as_ylo=&as_vis=0)

<comment><USER6> oh is that all. An entire generation were taught cryptography on the back of AC, so he definitely the widest read crypto dev.

<comment><USER5> He is definitely the widest-read crypto dev. There can be no question of that.

<comment><USER7> Out of curiosity, who else is out there writing essay-length on crypto and security in general?

<comment><USER6> One of the better recent sites has been RSnake's:

<http://ha.ckers.org/>

But he just retired from blogging about netsec and is done with the industry, I think (a lot of ppl get sick of it, I left the sec industry 10+ years ago and never looked back)

<comment><USER5> rsnake doesn't do any crypto work. Like, at all. A fine guy to go to for XSS or SQLI.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Given the past feats from Theo de Raadt, my guess is on a nice stunt to get a free thorough code check :)

<comment><USER9> Too many stupid people are saying too many bad things about the OpenBSD project for this to be a net positive for him.

After all, he doesn't really profit from a free audit, and all the auditing I've seen so far has been done by the OpenBSD team itself.

</comment></comment><comment><USER6> Easy way to prove it isn't true:

Has there ever been a criminal case prosecuted in the USA where the FBI entered or revealed intercepted VPN data as evidence?

<comment><USER10> This is false logic. This way you can only prove that the backdoor exists, not that it doesn't.

<comment><USER6> What I meant to say was that a reason why it may not be true. I started typing the response with one thing in mind and ended with another.

Point still applies though. No cases where prosecution has cited intercepted VPN traffic.</comment></comment></comment>


Zed Shaw, as of yesterday, is no longer the coolest kid in the Rails community. - <AUTHOR> http://www.workingwithrails.com/browse/popular/people <comment><USER2> The mere existence of that website makes me wonder if being part of the Rails community is really something to strive for...

<comment><AUTHOR> I like the idea of the site. In principle, it should be a useful way to pick a Rails contractor.

Unfortunately, it ends up looking like a popularity contest to a lot of people. I'm not sure how that can be avoided.

<comment><USER2> Yeah, I only looked at the linked page with the rankings, no idea about the context.

</comment></comment></comment><comment><USER3> Being ranked second, and only behind the creator of the framwork cannot be all that bad.

</comment><comment><USER4> Is my code hot or not?

Thats just sad

<comment><USER5> Funny idea though.

</comment></comment><comment><USER6> ???

Didn't DHH invent it? Zed Shaw always seemed like an unfunny Maddox. How could he be more popular? Why are they tracking popularity? Did Struts have an ongoing popularity contest?

</comment><comment><USER7> yeah that site is the to the shame of the creators. Def a popularity contest. .rb</comment>


The average American home now have 3 TVs - <AUTHOR> http://www.businessweek.com/the_thread/hotproperty/archives/2009/07/the_average_ame.html

<comment><USER2> _I'm part of the 2% of Americans that doesn't own a TV. I haven't had one for 7 months now and I don't miss it._

I just can't help but think of: [http://stuffwhitepeoplelike.com/2008/01/26/28-not-having- a-t...](http://stuffwhitepeoplelike.com/2008/01/26/28-not-having-a-tv/) every time I read that. I gotta admit, I'm 100% guilty of the same, but doesn't mean I can't smile every time I come across it.

<comment><USER3> I was actually immediately reminded of the Onion article : <http://www.theonion.com/content/node/28694>

</comment></comment><comment><USER4> Perhaps 2 of those tvs are analog and it might be just a little more than most people want to do to mess with that silly converter box and wires and all that jazz? TV's are _disposable_ now. How many tvs thoroughly != tv watched.

Its like saying "Americans have _3_ boxes of kleenex for just one nose!" They're cheap, they're ubiquitous, get over it.

<comment><USER5> Cheap and ubiquitous, but also those previous "displosable" TVs just aren't getting disposed. It's not so hard to imagine that an "average" home went from a perfectly fine tube TV to an upgraded version to a more recent flat screen. The previous two televisions end up in the garage/kitchen/guest room.

</comment></comment><comment><USER6> I don't have a DVR and have never been on Hulu. Yet I too watch what I want when I want.

The difference is that if what I want to watch isn't on when I want to watch it, I just turn the TV off. What's so hard about that?

</comment><comment><USER7> We should be proud to live in a country where the average household can _afford_ three TVs. They're big, complicated, expensive luxury items.

Miracles of modern commerce, overlooked.

<comment><USER8> I would be more proud of living in a country where people didn't buy things they don't really need.

Not to say that TV is one of those things -- I'm just saying that having the ability to show restraint is a admirable trait.

</comment></comment><comment><USER9> What's a TV?

</comment><comment><USER10> No TV here, either. Haven't had one in nine years.

I actually do miss it, for one reason: vegging out is a fundamental human need, and nothing helps you vege out better than a TV.</comment>


Show HN: MDAnki – convert Markdown to Anki cards - <AUTHOR> https://github.com/ashlinchak/mdanki <comment><USER2> I basically ruined myself for Anki. I started like gangbusters, and also created some _really_ cool jazz music theory decks for myself that have lilypond notation snippets and sound samples of various chord qualities and advanced jazz modes. And this is on top of other subjects like ML, scala, logarithms, etc.

And then... I hit this mathematical limit where I just couldn't keep up. You can't add 20 new cards a day for any length of time without the review cycle getting overwhelming. I'm not sure it's even physically possible to Anki full recall of everything learned in a college course - each class would need to be much longer than a term to keep up with the review.

One solution is to just keep plugging away, reviewing what you can and hoping the things you answer correctly will relieve the pressure as they come up less frequently. But since you're late on many card, recall sucks and I never really made much headway.

I also have a really hard time retiring cards. It's like deciding to cross things off your TODO list that you'll never do, you always want to keep it on thinking you might do it someday. Same with a not very relevant piece of info, it's right in front of you, it still feels like it might be useful to know...

<comment><USER3> Why were you adding 20 new cards a day? IMO its better to adjust your new card rate based on your retention rate. Your ideal retention rate probably depends on how you react to the "stress" of missing cards. Personally it would majorly stress me out if my retention rate was below 80%, I ideally liked to keep it around 90-95% because for me this was where it went from feeling like work to actually being fun and exciting.

I might go for a couple of days without adding anything, do a couple days adding 5, add 10, 15, 20, or even 30 depending on the circumstances.

<comment><USER4> 20 new cards per day to review is the default. You can add as many or as few as you want but if you didn't change the default it will put 20 in your review queue daily.

<comment><USER5> It's only going to put 20 new cards in your review queue daily if you have a huge backlog of unseen cards, which will only happen if you mass import a whole bunch of pre-made cards, which is not recommended. Actually creating 20 new cards a day would imply a pretty heavy study schedule.

<comment><USER6> > It's only going to put 20 new cards in your review queue daily if you have a > huge backlog of unseen cards

Correct. However, using a huge premade deck is not a bad idea in the sense that it's a huge time saver.

For me, creating an anki card takes between 1-2 minutes, depending on how slow the dictionaries (plural) are. So, 30-60 words per minute (45 avg) means it would take me almost 4.5 hours to input a measly 200 words.

I can write down 200 new words from one episode of a tv show. You can see where this is going. For Korean, about 5-6k words are recommended to begin having non-basic discussions. 5k words at 45 words/hour is ~110 hours.

Instead, I suggest using a parent deck that has a premade deck + your personal deck as children. This lets you have the best of both systems. You can use the Hoochie Mama! plugin [0] to properly randomize your review queue for child decks. Anki by default pulls cards in order from child decks.

Note: All of this assumes that your pre-made deck is of good quality. I can't speak about other languages, but Evita's Korean deck (5.8k cards) is sorted by frequency. It's fairly good, at least until you get to more of a intermediate level.

[0]: [https://ankiweb.net/shared/info/1460733408](https://ankiweb.net/shared/info/1460733408)

<comment><USER5> It also assumes that you are trying to learn thousands of new words in a foreign language, which requires less effort per card than almost any other exercise. Humans are natural language learners, after all. The OP was struggling to deal with 20 new cards a day on music theory and computer science.

</comment></comment></comment></comment></comment></comment><comment><USER7> If this interests you I built an entire SRS app around this idea: [https://mochi.cards/](https://mochi.cards/)

<comment><USER8> How long has this been around? Surprised I haven't seen it before! Fantastic work. I would love if I could use this with Roam Research somehow

<comment><USER9> Seconded. This looks really nicely done.

</comment></comment></comment><comment><USER10> maybe i'm just a jerk, but it bothers me when someone posts a project and doesn't link to an example of give a full description. i have no idea what anki cards are so i googled it and it's a pretty cool little open source flash card program. most people aren't going to do the research so it's best to have an explanation what all the components are. you don't have to be overly verbose, just a one liner explanation is fine most of the time.

<comment><USER11> Technical writing is a skill. Little things like linking related topics requires foresight into the range of knowledge your audience could have. It is easy to assume everyone shares the same knowledge you do.

[https://en.wikipedia.org/wiki/Curse_of_knowledge](https://en.wikipedia.org/wiki/Curse_of_knowledge)

<comment><USER12> This is absolutely true, but you can seldom go wrong assuming that proper nouns (like Anki) should be accompanied by links.

</comment></comment></comment><comment><USER13> I've spent many years using Anki and I have thousands of cards. I love it but I notice a few very common failure modes:

1\. Putting far too many cards in a deck.

2\. Putting in information that you want to learn rather than information you have _already learned_.

3\. Using large pre-made decks (which is really just 1 and 2 together).

If you're planning on using Anki longterm (such as for language learning) then keep the number of new cards low. Something in the range of 3-5 a day is totally fine.

You should put things into Anki after you have learnt them. In general I suggest putting in information at the end of a study session when you feel like you know it so well you couldn't possibly forget it. Cards should feel a bit too easy _when you create them_. By the time you come to review you will have forgotten a little and the difficulty will be perfect.

</comment><comment><USER14> Oooh, neat! Reminds me of this addon:

LaTeX Note Importer: [https://ankiweb.net/shared/info/1199027445](https://ankiweb.net/shared/info/1199027445)

I'm almost sold on downloading this, but I have a question.

Let's say I create notes.md with 2 cards:

## hey

hi

## hello

howdy

But, then I modify card #2:

## hello modified

howdy modified

and recreate the deck.

Does the 2nd card that has now been modified get treated as a whole new card? (In other words, if I've been using card 2 to study, do I have to start all over?)

If this is NOT the case, I am absolutely sold.

<comment><AUTHOR> MDAnki produces a file with a deck for importing to Anki. Every new import will generate new cards. I don't know anything if it's possible to make updating cards during the import. I'll try to investigate it. Thanks for this note!

<comment><USER15> This would change a lot of the current design but one option is to use AnkiConnect[0], it provides a network interface for interacting with Anki and creating/updating cards.

There's a somewhat similar project[1] that lets you use Org mode to make cards. It uses Emacs so there may not be too much you can apply but could be useful to take a look at.

[0] [https://github.com/FooSoft/anki-connect](https://github.com/FooSoft/anki- connect) [1] [https://github.com/louietan/anki- editor](https://github.com/louietan/anki-editor)

</comment></comment></comment><comment><USER16> Shameless self-plug: I created [anki-slides- import]([https://github.com/musically-ut/anki-slides- import](https://github.com/musically-ut/anki-slides-import)) to convert slides into Anki decks.

The program requires notes (written in Markdown, with one section for each slide) along with the pdf slides and it produces a deck which can be practised in Anki. It may prove to be useful for someone here.

</comment><comment><USER17> This is neat!

I did a similar thing to manage my Japanese vocabulary. I encoded it in TOML and compiled it to Anki decks. [1]

There is a lot of room for utilities like this. Managing decks in the Anki interface and persisting them as zipped sqlite is a headache.

[1] [http://github.com/echelon/nihongo](http://github.com/echelon/nihongo) (It's a bit of a mess, sorry.)

<comment><USER18> I did something similar, but to handle orgmode files:

[https://github.com/eindiran/orgmode-to- anki](https://github.com/eindiran/orgmode-to-anki)

<comment><USER12> Ooh, neat. How is image support?

</comment></comment></comment><comment><USER19> Thanks for this! Going to be useful - I am still getting used to Anki and the process of adding Cards sometimes seems like drudgery.

</comment><comment> r-zip This is very nice. I recently tried creating a bunch of Anki flash cards for theorems in real analysis, but found it too tedious. Is there any support (current or planned) for converting latex fragments?

<comment><AUTHOR> Yes, LaTeX on my Redmine board.

</comment></comment><comment><USER20> great stuff, was looking for something similar for a long while, time to try it again.</comment>


Online v. print reading: which one makes us smarter? - <AUTHOR> http://www.sciam.com/blog/60-second-science/post.cfm?id=online-v-print-reading-which-one-ma-2008-12-23

<comment><USER2> _The reasons: The process involves so much physical manipulation of the computer that it interferes with our ability to focus on and appreciate what we're reading; online text moves up and down the screen and lacks physical dimension, robbing us of a feeling of completeness; and multimedia features, such as links to videos and animations, leave little room for imagination, limiting our ability to form our own mental pictures to illustrate what we're reading._

They acknowledge later in the article that the findings about PC manipulation might be different for people who were raised in a different time, and I definitely agree -- a blank browser tab with focus on the URL bar is about as calming as possible for me. I definitely get annoyed when I'm reading a paper news article and I want to research a reference, doubly so when online at a paper news site when they don't link to things they obviously would if they were true online sources.

While I agree that the second half holds merit, I think it's wrong to hang that on the internet itself. I definitely "watch" reddit sometimes, just clicking pics links and skimming programming articles without really soaking anything in.

I think that being online doesn't change the need for discretion when choosing your information sources when you're attempting to do more than be passively entertained.

</comment><comment><USER3> Having received the results of this article, I have ordered a printing of the internet.

</comment><comment><USER4> I would rather read more books online than reading paper ones.

Instead of paying $30 for printed book, I'd pay $5 for online book with no ads.

Paper books are so one-way communication. Online books are interactive with 2-way communication abilities.

Picture is worth of 1000 words. So paper books now add images. Now video is worth of 100,000 words. What's wrong in adding Videos in the book?

And most importantly, I cannot search something in paper books. That's the single most reason all paper books should be converted into online books. That advantage surpasses all these negligent disadvantages by huge margin.

My $0.02s of course!

</comment><comment><USER5> I always read things which interest me by printing them. With HN there, it becomes hard to read interesting stuff with concentration because you always go back to HN pretty soon

</comment><comment><USER6> For me, it would seem the Internet increases my knowledge while reading a good book puts my mind to work. I guess it depends what you mean by smarter.</comment>


OpenAI unveils multitalented AI that writes, translates, and slanders - <AUTHOR> https://www.theverge.com/2019/2/14/18224704/ai-machine-learning-language-models-read-write-openai-gpt2 <comment><USER2> OpenAI's original article: [https://blog.openai.com/better-language- models/](https://blog.openai.com/better-language-models/)

"Slanders" comes from the ability of the software to generate text. It was trained on Reddit, so can talk like any Redditor you want:

> OpenAI's researchers admit that they're unable to fully answer this. They're > still exploring exactly what the algorithm can and can't do. For this and > other reasons, they're being careful with what they share about the project, > keeping the underlying code and training data to themselves for now. Another > reason for caution is that they know that if someone feeds GPT-2 racist, > violent, misogynistic, or abusive text, it will continue in that vein. After > all, it was trained on the internet.

> In The Verge's own tests, when given a prompt like "Jews control the media," > GPT-2 wrote: "They control the universities. They control the world economy. > How is this done? Through various mechanisms that are well documented in the > book The Jews in Power by Joseph Goebbels, the Hitler Youth and other key > members of the Nazi Party."

Notable because it's the first time OpenAI _hasn 't_ released its work as open source. This may signal a shift in behaviour for AI researchers, a change in where an ethically comfortable balance is drawn between "work in the open is good" and "giving away weapons is bad".</comment>


Goldmoney Inc. Adds Bitcoin and Ethereum to the Goldmoney Holding - <AUTHOR> https://www.goldmoney.com/corporate/news/goldmoney-inc-adds-bitcoin-and-ethereum-to-the-goldmoney-holding <comment><USER2> Rather than taking refuge in a flying palace or hiding under a mountain with 100 women for every man, the Prez and the members of congress ought to be chained to their desks at ground zero in the event they get us into a shootin' war -- douba douba III.</comment>


The Problem With The Big Bang Theory - <AUTHOR> http://butmyopinionisright.tumblr.com/post/31079561065/the-problem-with-the-big-bang-theory <comment><USER2> This again? TBBT is a comedy show. People laugh because it's funny. The culture that has spawned around TBBT has made nerd/geek culture cool. People don't buy and wear "Bazinga!" shirts because they like Penny and finds Sheldon to be a some kind of freak. The general audience love these characters. And through building that love and care of these guys, Chuck Lorre if anything has made Science and geek culture more populare than ever. Look at how HUGE geek and science-related things has become in social media. Look at the following astronauts are getting. Youtube channels and facebook groups about science.

Yes people find them quirky, just as the characters of FRIENDS were quirky. People still love them and even identify with specific things they do or issues they face.

My guess is people who find TBBT insulting are people who project self loathing. Clue #1: Feeling ANGER towards a TV show.

<comment><USER3> I guess part of "geek culture" is having been the uncool guy in school, having been pointed at and laughed at, getting bullied, being an outsider, andsoforth.

A comedy show where outsiders are being pointed and laughed at will often bring those experiences back, and really isn't helping.

<comment><USER4> I wonder if there's an age breakdown for people who hate/like the show. I'm well past the age where early high school is something I flash back to, so I'm perfectly OK with the show.

But if I were younger I could see myself hating it. I can say that the "Revenge Of The Nerds" movies did me no favors at that stage of my life.

</comment></comment></comment><comment><USER5> I discovered TBBT through a friend who nicknamed me 'Sheldon' due to my tendency to be somewhat socially awkward, geeky, and sharing some of Sheldon's OCDs.

I fell in love with the show, and I don't really agree with the conclusions of the article.

I don't feel like the writers are making fun of me, and I certainly don't feel that Penny is the 'normal' and cool character in the show. She's an underachiever, has a borderline drinking problem and poor choice in boyfriends before she met Leonard.

Yes, there is a self-deprecating component to all this. The characters are caricatures of various nerdy personalities, but I find them lovable, and I can relate to some of what happens to them.

I can laugh at myself, at my fondness for Star Trek, at my awkwardness in public, at my OCDs, at my love for everything technical and my interest in sciences. Being able to laugh at yourself is a strength, not a weakness.

What if others are watching the show and believe that it is to be taken at face value? Why should I care?

These nerds are successful scientists who work on the Mars Rover program, fly into space and go on adventures to the North Pole while Penny -the supposedly normal and cool character- is waiting tables in a dead-end job, shows no real passion for anything apart from shoes and it going nowhere.

Anyway, it's just a show. Some like it, some hate it. I personally don't think it fuels such a negative view of what nerds are. Maybe it's a caricature that some may take at face value but I'm not worried about that, the joke's on them really and they don't even know it.

</comment><comment><USER6> I've read theories like this before, and I gotta say, I don't agree with them. A big part of his argument hinges on "The humour in The Big Bang theory relies on the audience siding with and relating to Penny, the character coded as "normal" in comparison to the main four guys."... and maybe we're watching different shows, but the one I watch has more humor from the guys' standpoint (and often at Penny's expense). Yes, there is some humor from her standpoint, for people that identify more with her—but there should be, she's a member of the cast, and there are people who relate to her. But there's far more humor for people who relate to the geek characters (since there's more of them).

<comment><USER7> There's also significantly more deprecating humor aimed at the geeks, which for me offsets any humor I derive from relating to them.

Plus, the geeks really come across to me as too overly stereotypical to be relatable.

<comment><USER8> And even that would be relatable if it was self-deprecating. The article is right, the audience is positioned as an outsider; "It's funny because he mentioned that thing which is nerdy" rather than any realization of the character that Penny might find that reference impossible to relate to. You're clearly intended to identify with Penny.

<comment><USER9> I guess just making it sound nerdy/technical should be enough for most of the audience to think it's nerdy as hell and thereby funny, even if it is nonsense? Like "I'll make a GUI interface in Visual Basic to track their IP". :) I haven't watched the show myself.

<comment><USER7> That might help. Instead, it's something like "... I would do this like Spock <pause for laughtrack>"

</comment></comment></comment></comment></comment><comment><USER10> I thought the problem with the big bang theory was it's inability to explain the low entropy starting condition?

EDIT: btw if you want a comedy show that's rooted firmly in the geek world, dig up the TV version of the dilbert cartoons. superb.

<comment><USER11> I was going to upvote you but it's "its" instead of "it's"

</comment></comment><comment><USER12> > "We aren't laughing with Leonard, Sheldon, Raj and Howard. We're laughing at > them"

Well, here is my confession... I do like the show, and I guess it is a guilty pleasure.

I like it because I feel identified by the characters (my wife even call me Sheldon when I am being pedantic about something). But the author is right, when I watch the show I laugh at them and by transitivity I laugh at myself, which I have learnt is the best way to keep a good humor.

<comment><USER13> Sure, you laugh 'at' them, but they're still portrayed as good people who are worth hanging out with, as Penny finds out with time over the first few seasons.

</comment></comment><comment><USER14> Big Bang Theory is a depiction of what idiots think intelligent people are like. Those characters do not exist in real life.

IMHO UK's IT crowd is a much better and funnier depiction of geeks (albeit Moss is a bit exaggerated too).

<comment><USER15> > Big Bang Theory is a depiction of what idiots think intelligent people are > like. Those characters do not exist in real life.

I've known people in real life similar to all the main characters of BBT. The BBT characters are more concentrated, so I'd have to use maybe 7 or 8 people to span the characteristics of Sheldon, Leonard, Howard, and Raj, but I don't think that invalidates the point.

In fact, I knew such people at Caltech, which is where BBT is set.

As a Caltech graduate, I see no problem with their portrayal of Caltech people. It is the portrayal of the facilities that bugs me. They have geologists in the same building as theoretical physicists!

The "likes BBT" (x axis) vs "viewer intelligence" (y axis) curve seems to be, as far as I can see, convex.

<comment><USER15> Oops. Axislexia.

<comment><USER16> Wait, now I have absolutely no idea what you think the correlation is between BBT appreciation and intelligence.

<comment><USER15> The intent was to say that ordinary people and extremely smart people seem to like BBT [1], with the dislike coming from those in between. Among BBT fans are George Smoot, Neil deGrasse Tyson, and Stephen Hawking--they aren't doing speaking cameos on it to boost their resume for a career switch to acting. :-)

I probably should have just went with the original quip I had, which was "People you consider idiots like it, and people who consider you an idiot like it", but that seemed a little insulting so I dropped it.

[1] or at least ordinary people and extremely smart people who do not like it do not seem to complain about the show online.

<comment><USER14> Sorry, your attempt at condescending me does not work. I am a smart person with a PhD in Artificial Intelligence from University of Edinburgh's School of Informatics.

While I am no Hawkins, I do have first hand of experience of these levels of genius. I have met three nobel prize winners. I have never met anyone as dysfunctional as Sheldon. From what I have read, even Erdos had a sense of humour.

</comment></comment></comment></comment></comment></comment><comment><USER17> I think Apatow's Freaks and Geeks was the best representation of what it's like to be a nerd/geek/freak/outsider.

<comment><USER18> Somewhere in the DVD commentary he calls out a scene as what killed the show because it was just so realistic no one liked it. Everyone was making fun of a kid and he fell and broke his arm or something. They said after that scene aired the ratings dropped off and never recovered. So yeah, best representation, but it would seem people don't like things to be "real" they'd rather watch something like Big Bang and it not being so realistic.

</comment></comment><comment><USER19> Note that this is about the tv series, not the physical theory.

<comment><USER20> Please could someone change the title. I was disappointed, and wouldn't have clicked through if it had been clear that the article was about a TV show.

<comment><USER17> It would just get changed back. The powers that be at HN usually like to make sure the original title is preserved.

</comment></comment></comment><comment><USER21> It's a damn TV show, not a documentary. If you don't put your brain into low power mode and stop pondering about stuff, you won't enjoy it. It's a sequence of light jokes and a very loose story-line.

Another example that comes to mind: I like SciFi. If I'd analyse every movie for scientific credibility, I'd have a horrible time and would hate the genre. Just put your brain in error correction mode and switch on your creativity and you'll have a great time.

And if you don't like it, watch documentaries.

</comment><comment><USER22> As someone who considers themself (and many of their friends) a nerd, I have to say I thoroughly enjoy the show. I study physics, I love Sci fi, I read comic books etc. I don't feel genuinely insulted by the show. Yes jokes are made at nerds' expense. But if you can't laugh at yourself and your own "culture", there's something wrong.

If you don't like the show, that's fine. Don't watch it. But don't try and turn me against it just because you don't like it.

</comment><comment><USER23> IT Crowd - Non-nerd characters are the actual clowns.

The Big Bang Theory - Nerds are the clown characters.

In IT Crowd, the nerds are put into hilarious situations because of an insane normal world. In Big Bang Theory, the nerds are put into hilarious situations because they are insane.

<comment><USER24> If you don't think that Moss and Roy are both also clowns then you haven't been paying much attention.

<comment><USER23> Of course they also are. This thought has to be seen in the context of the subject of the thread - The Big Bang Theory, and the actual article which also deals with the outsider (non-nerd), the audience and the nerds.

In other words, Penny or the people at the university on the whole are not clowns, rather the nerds are the primary clown characters. Jennifer, her boyfriends, Mr. Renham or most other non-nerd characters in the IT Crowd are the primary clown characters. We (the general audience) sympathise with the nerds more in the IT Crowd in the same way that we sympathise more with the non-nerds.

</comment></comment></comment><comment><USER25> I watched an episode of this show once. It wasn't very funny. I don't think it's part of some grand conspiracy to belittle nerds. It's just a crappy show that anyone with an ounce of sense won't waste more than 20 minutes of their life on.

<comment><USER22> Just because you don't enjoy something, doesn't mean you should belittle those that do. I like to think I have plenty of sense, and I quite like the show. Just because you didn't enjoy that particular episode, doesn't mean you wouldn't find another hilarious.

</comment></comment><comment><USER26> The problem with it is that you're watching it as if it's a show for nerds when it's a show about nerds as seen by people who are not nerds. You're not the target market for this show.

As a side note on your page layout - My screen is massive. Please don't be afraid to use the space available!

</comment><comment><USER27> A doubt most people idenity with to hot, farm girl in big city, little slutty, stuck working at cheese cake factory because acting isn't going so well stereotypes. Comedy is funny because they take quirks people see in themselves and others and magnify them. The jocks aren't cool and relatable in TBBT either.

</comment><comment><USER28> Disagree with this authors sentiment. I too am a nerd and relate to the caricatures of nerds on the show.

I collect comic books, was obsessive about magic cards, star wars vs star trek, dressed in nerd costumes rather than sexy costumes, and have a mild case of OCD.

I like those things but I still see the humor in pointing out their absurdity on some levels.

Many shows do this -- even the mainstream comedies. They make fun of binge drinking, hookup culture, being stupid, etc.

Penny is made fun of constantly for her poor life choices, lack of degree, poor job, loser bfs, etc. AND the audience laughs at her constantly for it. Quite frankly I am proud that I relate with Sheldon more than Penny.

</comment><comment><USER29> I saw a reference recently that summed up why I dislike The Big Gang Theory: "Nerd Black Face." It is treating the nerds and geeks on the show as unself- aware fools for "real" society to laugh at.

<comment><USER30> Agreed. It is literally a minstrel show where comical, exaggerated "nerds" parade their comical, exaggerated neuroses and yet it is somehow one of the most popular shows in the nation.

No other social group or category would stand for this - it's basically blatant racism, but because it's not applied to a specific racial group but instead a social category, it's "funny" rather than "bigoted" or "mocking".

Can you imagine the outcry if there were such a show about blacks or Chinese people? The NAACP or Chinese community groups would sue the network so fast that their heads would spin!

</comment></comment><comment><USER31> Interesting perspective that I had never thought of. . .I totally get it. . .but I still enjoy the show, only now I may feel guilty about it. Maybe I was so nerdy/naive I never perceived that the audience could be laughing more _at_ them than _with_ them. And honestly, I've been wishing we could see less of Penny. I get why Leonard is _supposed_ to pine after Penny, but I keep thinking "get over her and move on to someone better". . .And I'm speaking as a female nerd.

</comment><comment><USER32> That entire (mostly valid) point could have been made in 1 paragraph.

<comment><USER33> Then you wouldn't have been able to spend as much time reading text against that background.

</comment></comment><comment><USER34> Every mass-media depiction of an activity or group will be simultaneously exaggerated and incomplete. In particular, TV comedies will emphasize anything that can make you laugh. They'll portray as idiots and make fun of: nerds, geeks, musicians, politicians, assistants, policemen, scientists, managers, presidents, models, spies, secretaries, firemen, astronauts, unemployed, lawyers, extraterrestrials, students, depressed, celebrities, immigrants, writers, and even the dead ones if need be.

</comment><comment><USER35> I'm not sure which side I'm on really. There are both good jokes there and some really mean episodes that are laughing about really nasty behaviour.

But to see how depressing some of the scenes actually are, search for "big bang theory without laugh track" on youtube. Once you listen to it like to a real situation... it's just terrible.

</comment><comment><USER36> I knew I hated that show for some reason I just could never put it into words like he did.

To be honest i never even got into it it a lot of people wanted me to watch it but after a few episodes it was clear this is not something worth my time.

I couldn't help it cringe when the main characters put themselves in a socially awkward position unlike regular geeks I values social interaction as much as technology so my social skills are quite high for a geek.

Plus i didn't find any of the jokes in the show funny frankly i found them quite insulting to geek culture.

There's no one you can identify with the guys are just too awkward and the girl seems more like an extra that happens to be sexy and the guys are drooling on her.

I know this is supposed to be a exaggeration of the typical geek but it just doesn't work for me.

</comment><comment><USER8> I quite like this writeup. I've seen a few episodes of The Big Bang Theory, and could never quite put my finger on why I didn't enjoy it, when I knew people who claimed it was the cleverest thing on TV and absolutely hilarious.

Clearly, though, I need to check out Community.

<comment><USER37> 'Cheap' is the best adjective that describes TBBT. The jokes, the characters, the situations, all of it feels cheap. And that is because it is less about the character of the four guys and more about this one specific aspect of them: being nerdy. And it gets old really fast.

Community has more interesting characters. They're still neurotic (the old guy gets a bit tiring), but they're much easier to care about.

</comment></comment><comment><USER38> He doesn't produce it, but "Two Broke Girls" being picked up from a pilot, nevertheless going into its third season, pretty much solidifies the point about Lorrie thinking.

</comment><comment><USER39> I can't quite put my finger on why I find this article annoying.

</comment><comment><USER40> I find black on brown extremely hard to read.

</comment><comment><USER41> It's TV, what did you expect?

</comment><comment><USER9> I had the impression that the whole embracing of being a geek/nerd was self- deprecating. But it seems that some people are being so ironic these days that it's hard to see where the meta-irony ends and the genuine feelings start.

It seems that people who aren't nerds respect (that is, dis-respects) nerds just as much, just that now they can do it with more general social approval, and the nerds can't really talk back if these are relatively small slights because of all the seeming self-deprecation from nerds themselves; you don't want to be seen as humour-less and overly sensitive. Journalists can refer to working adults as _nerds_ because, you know, self-irony and all that. Never mind the subtle put-downs; don't be sensitive.

I think that people who have nerdy interests while also being a well-rounded person (a trait nerds aren't known for) are the biggest winners in this case. They already have social status, and then get even more respect for having interests that are associated with intelligence. Plus, since they already "have a life" in the mainstream sense, you know that they are being nerdy out of genuine interest and not because they were socially awkward and had to "find comfort" or whatever in some nerdy pursuit.

I don't really consider myself a nerd, nor really a "normal" person. I think I have a lot of the negative traits of a nerd, while lacking some crucial traits/skill of a "normal" person. I don't really fit in anywhere in this dichotomy.

</comment><comment><USER42> Very interesting points of view. I share the analyse.

</comment><comment><USER43> _sigh_ not the shitty TVshow again.</comment>


Zenith - A film by anonymous released on BitTorrent - <AUTHOR> http://vodo.net/zenith

<comment><USER2> I watched it. Not bad. Unnecessary sexual content and language would be my main complaint. Acting seems strained, but just like Anakin seems like a whining twerp in Attack of the Clones, these characters seem a little stiff and emotionless. One could argue (in both cases) that the character is supposed to be a twerp or stiff and emotionless and the actor did a decent job.</comment>


Google's Chrome will soon crack down on sites that trick you into a subscription - <AUTHOR> https://www.cnet.com/news/googles-chrome-will-soon-crack-down-on-sites-that-trick-you-into-a-subscription/ <comment><USER2> It's interesting to consider what, if any, impact there would be on these measures if Google were regulated more aggressively as a monopoly.

Not that I disagree with restricting clickjacking etc, but at some point you have to curtail the power that any one company has. That was the motivation behind the Microsoft antitrust cases in the 90s.</comment>


You are no longer free to move around the country - <AUTHOR> http://reason.com/blog/2010/12/01/you-are-no-longer-free-to-move

<comment><USER2> The Fourth Amendment to the Constitution explicitly protects the people from unreasonable searches.

"The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."

One might say that you submit to this as part of the deal to get on the plane. But if that were true, it would be between you and the airline, not the government. That means it really should be the airlines that decide how intrusive searches should be. And you would be free to choose an airline whose policies you agree with. And the airline would be free to reject you as a passenger.

The government has hijacked the relationship between you and the people you've hired to get you from point A to point B. Just because of previous lapses in their intelligence efforts enabled some people to do something terrible doesn't mean you lose your rights.

<comment><USER3> The Fourth Amendment only protects you if you aren't inside the "constitution free zone". And as it happens, about two thirds of the US population live within this zone.

See: [http://www.aclu.org/national-security_technology-and- liberty...](http://www.aclu.org/national-security_technology-and-liberty/are- you-living-constitution-free-zone) and <http://yro.slashdot.org/article.pl?sid=08/10/24/197215>

<comment><USER4> Not particularly relevant, since that's not the justification they are using for the TSA.

</comment></comment></comment><comment><USER5> Let's say that we had machine vision software good enough to not need a human to look at the scanner images unless it discovered undesirable objects on the person (I don't think that's so far fetched technologically). What percent of people who currently oppose these machines would do so if they were completely automated?

Personally, I still would, as I am more concerned about the government scanning me and finding other things that it finds undesirable. I have no intention of blowing stuff up, but if these things become widespread, I think they might well be used to search for things other than weapons. Constantly being searched for illegal objects when I go out in public would disturb me. That's why I don't like these things. But I bet most of the people are just opposed because of the naked pictures.

<comment><USER6> I don't have a problem with the scanners. I have a problem with:

        • Inconclusive reporting about the health effects.
        • Conflict of interest with Deepak Chopra.
        • Inconsistent attitude and treatment by TSA officials
        • Firm statements that the machines do not store images,
          and then stories about images being saved.
        • Policies such as banning TSA employees from wearing
          radiation measurement badges.
        • Threats of $10k "civil lawsuit"
        • Pat-downs for children.
        • etc, etc.

The scanners are not the problem. The problem is the TSA.

</comment></comment><comment><USER7> I'm quickly becoming disturbed with what is happening in the US.

I was reading the last few chapters of _1984_ last night, for fun, and with what has been happening in the past few months, particularly with the TSA, I am starting to feel as though we are headed toward an Orwellian future. I really do not want to be unnecessarily inciting, but these incremental changes that are occurring are becoming more unsettling as time passes.

<comment><USER8> Absolutely. The state isn't expending all this energy (the development of the DHS, warrantless wiretapping infrastructure, etc.) for no purpose. The purpose is likely counter-insurgency, not terrorism.

</comment></comment><comment><USER9> I've said it before- take a train, bus, or don't visit (or live in) the USA. Well, looks like the last option will soon be the only one. I'll say it again- Americans, fix your government. The rest of the world is laughing at you. (and again, yes it's a nervous laugh. You scare us a bit too).

<comment><USER10> Never underestimate American zeal for non-public transportation.

edit: some biographical information might clarify this comment. I'm a 21 year old male from the midwest and budding car enthusiast. Additionally, I have a thing from making tongue-in-cheek comments that are absolutely non-obvious to everyone besides myself :)

<comment><USER11> I'm sorry, but even though I'm an American (or USAian, if you prefer) and support public transportation, wish it was better and more widespread, &c.-- it's simply shortsighted and simplistic to ascribe it to zeal for private ownership of automobiles. Although enjoyment of the private auto is a distinctly American and capitalist thing, the fact is that it's drastically less expensive and challenging to leverage public transportation in Europe, which is a much smaller land mass.

<comment><USER10> That statement was _very_ tongue in cheek. I'm from Ohio originally and love driving. I brought a car with me to college, and will be living in Northern California with a car (I actually plan to buy a new one soon). Having lived with a "convenient public transit system" in NYC for 3 months, I can safely say I far prefer driving, if only for the purely selfish and capitalistic reasons :)

You're hitting on what I was getting at though, which is that it's just not that big of a deal to the average American, because we drive. Next time I will leave being clever to the professionals :P

<comment><USER12> Be as clever as you wish and do not worry about downvotes.

</comment></comment></comment></comment></comment><comment><USER13> I honestly don't think I could live in this country if this sort of thing became the norm for bus and rail travel. I take the train every day to work, each way and back, and given the crowding and delays, it would be so infeasible to have this sort of security theatre when no such threat to these transit systems exist.

<comment><USER14> I heard the same thing said about the atrocious security theatre at airports, especially the really busy ones like O'Hare. It was going to be temporary, I heard, because there was no practical way it could be permanent. It's been 9 years. Feels pretty permanent to me, as much as I wish that it didn't. I'm afraid at this point the question is whether we're too late with our outrage (or perhaps if the present outrage is sufficient, which it probably won't be.)

</comment></comment><comment><USER15> If you think the naked picture machines are going to be limited to airports, you're just not paying attention.

</comment><comment><USER16> I've already been opting out of flying. If these checkpoints were everywhere, i'd opt out of citizenship.

<comment><USER17> > If these checkpoints were everywhere, i'd opt out of citizenship.

Bullshit.

<comment><USER18> That might be abrasive, but it's realistic. These changes (as always) will come gradually so that each increment is not quite so bad as to spark riots, but when viewed as a batch change over 30-50 years will be something amazing. Scanners will be the norm for all public transport, possibly private transport and potentially many public spaces over the coming decades.

<comment><USER4> The idea that this will be limited to the US is rather charmingly naive too.

<comment><USER18> For sure. I think it will feature in the US, then Europe, maybe Northern Africa?

Australia would love to follow the lead of the US but not sure if we'd be able to justify the cost too early.

</comment></comment></comment></comment></comment><comment><USER19> In the Netherlands your only option to move around 'untracked' is to use a bike or to walk, any other mode of transportation is monitored. And better leave your cell phone at home because telcos are required by law to keep location information for quite a long period.

<comment><USER20> What do you mean by untracked? You can buy a single ticket to any place without showing your ID. I don't think I even needed one for an Amsterdam -> Brussels train.

<comment><USER19> Since a few months we have the "OV Chipcard" which requires regular users of public transport to check in / out when entering / leaving the public transport system.

On a large number of stations the tickets are only sold using electronic vending machines that do not accept cash.

<comment><USER21> In the US, one can purchase pre-paid credit cards with cash. Is that option easily available to people in the Netherlands?

</comment></comment></comment></comment><comment><USER22> This is not hacker news. Leave it on reddit.

<comment><USER2> When only the state is free to do what it wants, only the state will hack.

<comment><USER23> Since when has legality had to matter for hackers when hacking?

<comment><USER2> i speak of freedom, not legality. If the gulag awaits you if you're caught, that's a completely different story than it would be if you hack in a society governed by rule of law. But don't take my word for it. Ask the North Korean hackers.</comment></comment></comment></comment>


Things We Believe After Building Maptia - <AUTHOR> https://medium.com/founder-stories/ten-things-we-believe-2c85c462d508

<comment><USER2> "And secondly, that we weren't motivated by making a lot of money "

You've learnt a lot, that's worth fair bit. Going forward, you will need to make money. How are you making _any_ money from this?

<comment><USER3> Jonny (a Maptia co-founder) here. Thanks and yes we know that it is incredibly important to get this aspect of Maptia right (having a sustainable way to grow will enable us to have more of an impact) and we also believe in being 100% transparent.

We have felt comfortable supporting Maptia ourselves for now. However, it is our long-term aim to grow Maptia into a sustainable organisation whose primary aim is to re-invest in its community to directly fund and support talented photographers and writers to tell impact-driven stories.

We are currently in the process of deciding whether to convert to a non-profit foundation or to another similar structure of organisation that is better suited to our long-term mission and our values. At the very least, over the next two years, we are committed to launching a non-profit sister organisation for Maptia, and are already working on a number of storytelling partnerships with non-profit organisations (like charity: water) from all over the world.

As a first step towards our goal of becoming sustainable, we are investigating a small number of partnerships with inspiring brands and organisations who share our values, who believe in our vision, and who will add value and engage with the storytelling process.

Our primary aim is to be able to start compensating some of the talented storytellers on our site for their work, and to be able to cover the running costs of the Maptia platform. We pledge never to stick logos or other branding on (or anywhere near) Maptia stories unless the authors are being well compensated and are fully on-board with the idea, and furthermore, we will only partner with authentic organisations who believe in the power of storytelling as much as we do.

If our persistence over the last few years is anything to go by, we will figure it out... and even if all else fails, then Dean (our CTO and also a talented musician) has promised to busk part time to cover our costs. In other words, we will do everything humanely possible to make sure that Maptia will be around for many years to come!

<comment><USER2> You're as mad as a sack of ferrets. Can't knock your enthusiasm.

</comment></comment></comment><comment><USER4> Sounds like of those 1000 days you also gained an education in business and coding. It may be 1/30th of your life, but so is college. Even if everything goes south, you guys have an entirely new and highly useful skill set, and have shown a willingness to go to an extreme to get things done.

If you keep it up, you are definitely going somewhere!

</comment><comment><USER5> Moroccan here. Hope you enjoyed the country.

Did you try to hire any local coders or even interns at your start-up?. There a are a lot of aspiring engineers in morocco who would have benefited from an experience of working for a start-up like yours.

<comment><USER3> We were still building v1.0 of our product and so weren't looking to hire during that early stage. However we did connect with a group of Moroccan entrepreneurs who travelled all the way from Tiznit, Marrakech and Agadir and have kept in touch ever since. It definitely feels like there is a passionate startup community growing over there.

</comment></comment><comment><USER6> You have an idea, or vision, that you're passionate enough to move, learn, chase, fall, get back up, and continue to chase. This is so much more important that the Technical or Marketing knowledge that so many people have. Its something to be intimidated by, even if you're the worlds greatest Marketer or Full Stack Developer. Finding an idea you're passionate about is infinitely harder than learning skills required to make that happen. Most people's reaction to seeing that is a sense of inadequacy in their own life, which leads to jealousy & some "advice" that is meant solely to make you question your cause....bring you down to their level, and impart a feeling of superiority in themselves again.

How hard was/is it to decipher advice from jealousy?

</comment><comment><USER7> Indeed Kudos! Doing the same thing, but doing it myself while I make a game. Need to put together a blog like you guys. Got some writings and paintings down...just need to build the site. Thanks for sharing!

</comment><comment><USER8> What an awesome place in Morocco. Thanks for touching on life style balance. I think it's a struggle most of us deal with.

</comment><comment><USER9> Hey guys,

Fellow Chilean here, just glad to hear Startup Chile was in some way useful to you guys. Would have loved to meet you when you where around ;)

Cheers!

</comment><comment><USER10> Nice article - always fascinating to see your insider perspective. Kudos on the transparency.

</comment><comment><USER11> I enjoyed reading that very much. The quote at the end is fucking amazing.

</comment><comment><USER12> Great story guys

</comment><comment><USER13> Love this post.

<comment><USER3> Thanks Jared – it has been a crazy journey since Startup Chile back in 2012!

</comment></comment><comment><USER14> that's an awesome story :)</comment>


Show HN: A curated list of the best vacuum cleaners for hackers [ES] - <AUTHOR> http://www.lasaspiradoras.com/ <comment><USER2> I have a good hack, a 4 year old who loves vacuum cleaning.</comment>


Brainwash is a dataset of webcam images taken from the Brainwash Cafe in SF - <AUTHOR> https://megapixels.cc/datasets/brainwash/ <comment><USER2> Funny how every time someone from a university asks you 5 questions for research or lets you click around in a prototype it needs forms and ethics review, but photographing random people and publishing it as a "dataset" is apparently fine?

<comment><USER3> I thought the same thing. Also from the article:

> _Who used Brainwash Dataset?_

China.

</comment></comment><comment><AUTHOR> Here's the dataset itself.

[https://exhibits.stanford.edu/data/catalog/sx925dc9385](https://exhibits.stanford.edu/data/catalog/sx925dc9385)

I wonder if you could detect which images are from Tony Spark's Open Mic comedy nights, by searching for people laughing and smiling?

<comment><AUTHOR> There's Tony, introducing the first comedian!

brainwash_11_24_2014_images/00083500_640x480.png</comment></comment>


Ask HN: Best book on making decision and overcoming fear. - <AUTHOR>

What is the best book on making decision, choosing between choices, and overcoming fear and laziness to make decision. <comment><USER2> "The Fountainhead" by Ayn Rand. If you're in Melbourne (Victoria) you can borrow my copy.

<comment><USER3> How did fountainhead helped you?

<comment><USER2> By providing a role model, and inspiration during looooong evenings when working on a startup. Some quotes ...

=====

"Do you mean to tell me that you're thinking seriously of building that way, when and if you are an architect?" "Yes." "My dear fellow, who will let you?" "That's not the point. The point is, who will stop me?"

=====

"The creators were not selfless. It is the whole secret of their power--that it was self-sufficient, self-motivated, self-generated. A first cause, a fount of energy, a life force, a Prime Mover. The creater served nothing and no one. He had lived for himself. And only by living for himself was he able to achieve the things which are the glory of mankind. Such is the nature of achievement."

=====

</comment></comment></comment><comment><USER4> The Art of non conformity.

But really, you can't "learn" this concept. You either live in fear for the rest of your life or u don't. U decide.</comment>


Faster Unsigned Division by Constants (2011) - <AUTHOR> http://ridiculousfish.com/blog/posts/labor-of-division-episode-iii.html <comment><USER2> I just tested GCC, ICC, and Clang to see how well they perform these optimizations on their own. Code for my test is here: [https://gist.github.com/nkurz/6844adbc58a03eb3420f](https://gist.github.com/nkurz/6844adbc58a03eb3420f)

The short answer is that they all do it well. For a loop like this compiled with -O3:

        for (int i = 0; i < COUNT; i++) {
            compilerSum += randomInt[i]/CONST;
        }

ICC 14.0.3 takes 4.1 cycles per iteration, Clang 3.4.1 takes 3.8 cycles, and GCC 4.8.1 manages to spend only 1.3 cycles for each division!

Apparently GCC realized it was able to not only use a shortcut for the division, but also managed to vectorize the loop. Here's the inner loop it produced:

      4019f0:       66 0f 6f 00             movdqa (%rax),%xmm0
      4019f4:       48 83 c0 10             add    $0x10,%rax
      4019f8:       48 39 c5                cmp    %rax,%rbp
      4019fb:       66 0f 6f c8             movdqa %xmm0,%xmm1
      4019ff:       66 0f 6f e0             movdqa %xmm0,%xmm4
      401a03:       66 0f 62 c8             punpckldq %xmm0,%xmm1
      401a07:       66 0f 6a e0             punpckhdq %xmm0,%xmm4
      401a0b:       66 0f f4 cb             pmuludq %xmm3,%xmm1
      401a0f:       66 0f f4 e3             pmuludq %xmm3,%xmm4
      401a13:       0f c6 cc dd             shufps $0xdd,%xmm4,%xmm1
      401a17:       66 0f fa c1             psubd  %xmm1,%xmm0
      401a1b:       66 0f 72 d0 01          psrld  $0x1,%xmm0
      401a20:       66 0f fe c1             paddd  %xmm1,%xmm0
      401a24:       66 0f 72 d0 02          psrld  $0x2,%xmm0
      401a29:       66 0f fe d0             paddd  %xmm0,%xmm2
      401a2d:       75 c1                   jne    4019f0 <main+0xb0>

I put the other two here: [https://gist.github.com/nkurz/b1c3e01498dd11fda42f](https://gist.github.com/nkurz/b1c3e01498dd11fda42f)

I didn't look closely enough to identify which methods they were using. Can someone else tell?

<comment><USER3> GCC's vectorized loop is using the usual "round-up" method, but we can easily implement the distributive method there by mixing 32 and 64-bit arithmetic:

        ; xmm7 = 0 magic 0 magic
        ; xmm6 = 0
        ; xmm5 = 0 (accumulator)
        division_loop:
          vmovdqu  xmm4, [rax]         ; xmm4 = w3 w2 w1 w0

          vpunpckldq xmm0, xmm4, xmm6  ; xmm0 = 0 w1 | 0 w0
          vpunpckhdq xmm2, xmm4, xmm6  ; xmm2 = 0 w3 | 0 w2
          vpmuludq xmm1, xmm0, xmm7    ; xmm1 = w1 * magic | w0 * magic
          vpmuludq xmm3, xmm2, xmm7    ; xmm3 = w3 * magic | w2 * magic
          vpaddq   xmm1, xmm1, xmm0    ; xmm1 = w1 * magic + w1 | w0 * magic + w0
          vpaddq   xmm3, xmm3, xmm2    ; xmm3 = w3 * magic + w3 | w2 * magic + w2
          vpsrlq   xmm1, xmm1, XX      ; XX = shift constant
          vpsrlq   xmm3, xmm3, XX      ; XX = shift constant

          vshufps    xmm1, xmm1, xmm3, 10001000b ; back to w3 w2 w1 w0
          vpaddd     xmm5, xmm5, xmm1  ; add to accumulator

          add rax, 16
          cmp rax, rbp
        jnz division_loop

The shifts can be further tweaked to allow to use a blend instead of a shuffle, in case there's a penalty for changing from integer to floating-point domains. Saturating increment also works well, despite there not being a PADDUSD instruction; it can be done in two cycles:

          ; xmm6 = -1 -1 -1 -1
          vpcmpeqd xmm3, xmm4, xmm6 ; xmm3[i] = (xmm4[i] == -1 ? -1 : 0)
          vpsubd   xmm4, xmm4, xmm6 ; xmm4[i] += 1
          vpaddd   xmm4, xmm4, xmm3 ; xmm4[i] += xmm3[i]

<comment><USER2> Nice work! Seems like it extends fine to AVX2 for doing 8 divides at a time in less than 1 cycle per division if for some reason you need to divide a whole array by the same constant. Though rare, this is impressive considering the tradition of expensive division.

I don't know if you will have a domain penalty for the vshufps, but I think the answer is yes, and that the blend would be better. The blend can also run on p015, versus only p5 for the shufps, so it might pipeline better. I hadn't realized until now that vshufps had different semantics than vpshufd and is able to combine from two different registers. If there is no penalty, this could be quite a useful instruction.

Any reason for using 'w' for your 32-bit elements rather than 'd' for double- word as used in the instruction names?

<comment><USER3> According to Agner Fog, data domain penalties with SHUFPS are nonexistent, or at least pretty hard to trigger, on Haswell. Previous chips might complain, though. Problem with blends is that they require SSE4.1+, unlike the current version that is fully SSE2 (modulo AVX encoding niceties).

I chose 'w' for no real reason other than being thinking of 'word' at the time.

<comment><USER2> I just tested, and I can now confirm that there is no domain switching penalty for using vshufps versus vpshufd on integer data for XMM and YMM on Haswell, nor for XMM on Sandy Bridge (no YMM comparison possible there because vpshufd requires AVX2).

</comment></comment></comment></comment></comment><comment><USER4> Nice analysis! There is a similar problem in base-255 pixel math, with a similar solution. You want 255 * 255 = 255 when you multiply pixels, so you are doing divide-by-255 (not-256), which in 8 bits is somewhat "uncooperative".

I wrote up a few solutions many years ago, one of which is Sree Kotay's idea: adding one to the numerator (what this article says but without the awesome proof): [http://stereopsis.com/doubleblend.html](http://stereopsis.com/doubleblend.html)

It would be great if compilers could do all this for you, with a proof like this to go with it.

<comment><USER5> I would think the better (symmetric and slightly more accurate) solution would be to add the MSB of the result to the LSB and then round to the nearest MSB (by adding 0x80 before shifting)? e.g:

        ((0xFF*0xFF = 0xFE01) + 0xFE + 0x80 = 0xFF7F) >> 8 = 0xFF
        ((0xC0*0xC0 = 0x9000) + 0x90 + 0x80 = 0x9110) >> 8 = 0x91
        ((0xB4*0xB5 = 0x7F44) + 0x7F + 0x80 = 0x8043) >> 8 = 0x80
        ((0xB4*0xB4 = 0x7E90) + 0x7E + 0x80 = 0x7F8E) >> 8 = 0x7F
        ((0x80*0x80 = 0x4000) + 0x40 + 0x80 = 0x40C0) >> 8 = 0x40
        ((0x00*0x00 = 0x0000) + 0x00 + 0x80 = 0x0080) >> 8 = 0x00

(To be precise, you should also add the MSB to the right of the LSB, etc., since 1/.FF = 1.01010101..., but now you're getting lost in the noise.)

EDIT: screwed up some math; thanks gjm11.

<comment><USER6> But 1/.FF = 1.01010101..., not 1.02030405... .

(Multiplying the RHS by FF gives FF.FFFFFFFFFF etc. = 100. 1.020304... is 1/(.FF)^2.)

<comment><USER5> Whoops, good catch! Edited.

</comment></comment></comment></comment><comment><USER7> I know many compilers use this optimization. However I recently used this for the case where the divisor was a variable but restricted to a few possible values. I expanded it to a switch case statement where each possible divisor was expanded to a constant value allowing for this optimization. For this processor, there is no hardware divide so this made for significant performance gains.

</comment><comment><USER8> I think all modern compilers do this now. I know the D compiler does:

[https://github.com/D-Programming- Language/dmd/blob/master/sr...](https://github.com/D-Programming- Language/dmd/blob/master/src/backend/divcoeff.c)

<comment><AUTHOR> Modern compilers do some variant of fast integer division by constants, but I think most use the "round-up" algorithm described in an earlier post in this series ([http://ridiculousfish.com/blog/posts/labor-of-division- episo...](http://ridiculousfish.com/blog/posts/labor-of-division- episode-i.html)) and a variety of other places and not the "round-down" algorithm described in this post when the multiplier is a bit too big. You'd know better than I, but it looks like this is the round-up algorithm: [https://github.com/D-Programming- Language/dmd/blob/master/sr...](https://github.com/D-Programming- Language/dmd/blob/master/src/backend/divcoeff.c#L206-L208)

</comment></comment><comment><USER9> My first question was: how slow is hardware divide? The article didn't answer that, but there are many resources on the net that do (more or less, it's tough to precisely benchmark these very sophisticated CPUs). One paper is at [1].

The general conclusion is that simple operations like add can be done in 1 CPU cycle. They can be easily pipelined for a throughput of 3 different add operations in 1 cycle. Modern CPUs also pipeline integer multiply so, while the latency is perhaps 4 cycles, the throughput is 1 multiply in 1 CPU cycle.

But hardware division can be slow, really really slow. A Pentium 4 might have a latency of 80 cycles for a 32 bit operation, 160 cycles for a 64 bit operation. And there's no pipelining, multiple divides don't appear to be done in parallel. Even newer CPUs like the Sandy Bridge architecture have a latency of 26/92 cycles for 32/64 bit operations.

So it's not surprising that compilers try to avoid doing hardware divide.

[1] [https://gmplib.org/~tege/x86-timing.pdf](https://gmplib.org/~tege/x86-timing.pdf)

<comment><USER10> I hadn't realized hardware division was that slow, yow.

I suppose that part of this is that we now have 64-bit operations everywhere - the length of the critical path for a 64-bit divide is absurd.

</comment></comment><comment><USER5> Note that while most compilers already replace fixed-divisor divisions with multiplications, this algorithm presents a more optimized solution for certain "problem" divisors, notably 7.

</comment><comment><USER11> A compiler I use, based on lcc, did this automatically without even enabling optimizations.

The division is transformed into multiplication and a right shift.

I don't think your usual programmer must care about that with the modern compilers, but it is nice to know if you are implementing one or you are coding in asm.

<comment><USER12> Or if you're dealing with data that isn't compile-time constant, but can be expected to remain unchanged for the duration of something that needs to be optimized all the way.

</comment></comment><comment><USER13> Compilers have done this sort of thing for a very long time. Here is a well- known paper from 1994 that already tries to do it for more arbitrary constants.

[https://gmplib.org/~tege/divcnst- pldi94.pdf](https://gmplib.org/~tege/divcnst-pldi94.pdf)

<comment><USER5> No, they haven't. This extends the algorithm you reference to cover the special case of certain "magic numbers" which don't fit nicely in a machine register. From the article:

 _It does not seem to appear in the literature, nor is it implemented in gcc, llvm, or icc, so fish is optimistic that it is original._

<comment><USER13> Actually, looking more carefully at the Fish webpage I think they cite 7 as an example because 7 does not divide 2^32+/-1 (or indeed 2^64+/-1). If it did, the quotient would be a "magic number".

However, 7 does divide 2^33 - 1. This "magic number" _does_ fit in a 32 bit word. It's just that division by 2^33 - 1 isn't as trivial as division by 2^32 +/\- 1 (it's not much worse though).

Multiplication by/storage of a 33 bit number also isn't a problem anyway, because you can use the "IEEE trick" of storing the leading 1 bit implicitly. Multiplication by such a number can be done with one multiply and one addition. Not that you actually need to do this for the divisor 7.

The original Montgomery-Granlund work (apparently the basis of GCC division by invariant integers for quite a while) was to always find a quasi-"magic number" which fits in a word. So that was definitely not the innovation of Fish.

The followup paper of Moller-Granlund does even better by using a mullow instead of a full mul or mulhigh, which is usually available sooner on modern processors.

Fish seem to have found some highly optimised sequences. I'm not entirely clear on how new the ideas themselves are. It's clearly not the same as Montgomery-Granlund, but Montgomery-Granlund use (B^2 - 1)/d (where admittedly d has to be first "normalised"), Fish uses (2^p*B)/d where they fiddle the dividend and do some post-shifting.

I'm not even sure if Fish is better than Moller-Granlund, though the latter was 2009, not 1994.

I don't want to take away from the contribution of Fish. But it just needs to be viewed in the context of rather a lot of previous work.

<comment><USER5> Thanks, that's a very detailed analysis.

</comment></comment></comment></comment><comment><USER14> From 2011, has this made it into gcc? Is it part of LLVM now since they tested with that?

<comment><AUTHOR> It looks like LLVM trunk still uses a subtraction and a shift for "uncooperative" divisors: [https://github.com/llvm- mirror/llvm/blob/1bdc6c4519d9ad254b2...](https://github.com/llvm- mirror/llvm/blob/1bdc6c4519d9ad254b20534a1664f9d2809a95f1/lib/CodeGen/SelectionDAG/TargetLowering.cpp#L2751-L2766)

<comment><USER15> I found two other implementations of software division in LLVM:

[https://github.com/llvm- mirror/llvm/blob/master/lib/Transfor...](https://github.com/llvm- mirror/llvm/blob/master/lib/Transforms/Utils/IntegerDivision.cpp)

[https://github.com/llvm- mirror/llvm/blob/master/lib/Transfor...](https://github.com/llvm- mirror/llvm/blob/master/lib/Transforms/Utils/BypassSlowDivision.cpp)</comment></comment></comment>


Hacking Perl in Nightclubs - <AUTHOR> http://www.perl.com/pub/a/2004/08/31/livecode.html

<comment><USER2> Are duplicate posts only impossible within a period of _n_ days?

<http://news.ycombinator.com/item?id=263571>

<comment><USER3> In this case, I'm glad for it. An unusually enjoyable read.</comment></comment>


Berkeley Lab Building Own Open Architecture Quantum Chips - <AUTHOR> https://www.nextplatform.com/2018/09/24/berkeley-lab-building-own-open-architecture-quantum-chips/ <comment><USER2> Irfan Siddiqi is one of the leading researchers in the field of quantum feedback and has done some really awesome experiments. I think developing new architectures for superconducting quantum chips is very important and there are many things that are still unresolved. John Martinis' group had (and maybe still has) a very open-source approach to this as well, they published most of the source code and custom-built electronics they use in their labs for controlling qubits (the architectures themselves are documented more in papers and theses). It seems their wiki is no longer reachable from the public web, shame.

The quantum computing startups try to convey the impression that they solved all problems for building real-world quantum computers already, I think that many things are still open though (and so far none of the startups has really produced convincing evidence that they really have a working quantum computer, or even an interesting qubit chip).

What I'd like to see from these startups are at least some basic measurements that characterize the quality of single-qubit manipulations, two-qubit gates and qubit readout. With that one could at least estimate what one could do with their quantum devices.

</comment><comment><USER3> Does anyone know how things are leaning as to whether useful quantum computing is even possible/scalable? I recall reading somewhere that it's an unanswered question, and that people are optimistic but it may be that there's some undiscovered law of information that forbids it.

<comment><USER4> Gil kalai's argument is essentially that quantum error correction is impossible because he believes that the error will affect too many qubits at once.

<comment><USER4> To clarify, since I can't edit anymore: I don't agree with him, I'm just adding the viewpoint.</comment></comment></comment>