The world's cities with the most bookstores and libraries per capita - <AUTHOR> http://qz.com/741099/the-worlds-cities-with-the-most-bookstores-and-libraries-per-capita/ <comment><USER2> Not all bookstores are equivalent; one Powell's should not be counted as equal to a small specialty shop. A better metric might be local retail book purchases per capita (in units, not currency).

<comment><USER3> >A better metric

Metric of what? As far as I can tell "bookstores per capita" is a pretty good metric to measure bookstores per capita. If this is a good proxy for something else, that I don't know, but it's certainly interesting.

<comment><USER4> I think what was meant is that a giant bookstore like Powell's sells many times the number of books compared to small "hole in the wall" shops.

I happen to live in Portland not far from Powell's main store, which is _huge_ and _busy_ all the time. Portland also has numerous smaller shops which serve different niches and out of the way locations. I think the best metric would be number of city-wide annual book sales per capita. (r = total_annual_book_sales/population)

</comment></comment></comment><comment><USER5> Does this include university libraries? If so, I would love to see a college city like Boston or Cambridge, MA, USA report their numbers.

The title is very misleading. It's "The most bookstores and libraries in a self-selected small sample of cities"

Seems this could be done automatically and globally with other resources.

Edit: quick very non-scientific google maps search... looks like Boston has about 9.5 libraries per 100,000 people if you include universities.

<comment><USER6> Regular citizens may not have borrowing ability at a university library, it varies greatly.

<comment><USER7> In the US, at least, don't most states mandate library access at public universities for taxpayers of that state?

<comment><USER8> At least here in NC, there's no statewide mandate, to the best of my knowledge, regarding public access (in terms of checking books out) to university libraries. But, at the same time, most - if not all - of them do allow members of the public to purchase a library card and to checkout books. Even some private universities do so. Duke[1] for example, has a program where unaffiliated visitors can get borrowing privileges. It does cost a minimum of $100.00 though. :-(

OTOH, to the best of my knowledge, most of the public university libraries (and perhaps even some of the private ones) will let anyone walk in and view/read books in the library itself, for no fee. I know at least last time I was at the big new library at NC State, anybody could walk in and browse with no problem.

[1]: [https://library.duke.edu/services/visitors](https://library.duke.edu/services/visitors)

</comment></comment></comment></comment><comment><USER9> It's sad in a way, but I drove past a Barnes & Noble while traveling through Lincoln, Nebraska and thought, "oh, I haven't seen one of those in a while. I thought they were out of business."

Then thinking a little further, "Yeah, just like vinyl records. Oh. And CDs. And VHS tapes. And DVDs. Shit. The hazards of living in the future."

<comment><USER10> I went in a Barnes & Noble recently and found they had a surprising amount of vinyl. Decent stuff for a good price, too. Also DVDs. I have a bluray player and do get them when they are in $5 bins, but otherwise DVDs are fine for me and super cheap now. Can't complain.

I also recently ordered 12 books in a series from the Barnes and Noble website because they were about 5% cheaper overall than on Amazon.

<comment><USER11> > books, vinyl, DVDs, bluray

Wow, we covered a _lot_ of technological development rather quickly there.

</comment></comment></comment><comment><USER12> Funny how 1 data point can be extrapolated to make such a bold claim (best cities for X). Hong Kong is a terrible place for book lovers.

\- I think almost all the bookstores are not profitable, many of them closing down (especially the independent "second-floor" ones", ie not street-level stores). The book choices in mainstream bookstores are rubbish. (PageOne and Eslite slightly better.)

\- 99% of the HK publishing houses are controlled by Chinese government by proxy. This imposes editorial censorship and determines which books will get promoted in the stores.

\- If you publish or sell books that offend the CPC, you may be arrested in China with some false charges

\- Generally speaking, most people in HK are not book lovers. If you visit Eslite (a Taiwanese chain), you'll see a large area of their store is devoted to lifestyle goods, not books.

Go to Taipei, go to Charing Cross in London, but don't go to HK for books.

<comment><USER9> Yeah, it's pretty funny because Hong Kong bookstore owners find themselves kidnapped, locked up, interrogated and brain washed for months for selling books not to the liking of the CCCP:

[https://www.washingtonpost.com/news/worldviews/wp/2016/06/17...](https://www.washingtonpost.com/news/worldviews/wp/2016/06/17/complete- lawlessness-booksellers-dramatic-tale-spooks-hong-kong-underlines-concern-for- swede/)

<comment><USER13> I very much doubt the CCCP was involved.

<comment><USER9> CCCP is a correct acronym for the Chinese Communist Party:

[https://en.wikipedia.org/wiki/Central_Committee_of_the_Commu...](https://en.wikipedia.org/wiki/Central_Committee_of_the_Communist_Party_of_China)

The abductee claims, and Britain states, that the Chinese government, née communist party, was responsible. The books that the communist party found objectionable, and which led to the booksellers' abductions, contained gossip about the personal lives of party members.

<comment><USER13> Interesting. I did not know that 'CCCP' was overloaded. Thanks! I cannot find any source other than you who claims that, however - can you?

<comment><USER9> I know a lot of people who live in Hong Kong. I visit there periodically. In vernacular parlance, they frequently say 'CCCP' when referring to the Chinese government. Or they just say 'Beijing'. So, granted, it's hearsay, so I'm happy to start using a different and more acceptable epithet for the Chinese government. Any suggestions?

<comment><USER13> Well, the party is CPC. The Central Committee is abbreviated CC in Wikipedia, though that seems unofficial.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> Shame they only include cities; the town of Hay-on-Wye in Wales has 24 bookshops for 1,598 people.

[https://en.wikipedia.org/wiki/Hay-on- Wye#Book_town](https://en.wikipedia.org/wiki/Hay-on-Wye#Book_town)

</comment><comment><USER15> I think it's missing a lot of cities. The concept is good, but I don't trust the metrics. (Bookstore in an airport versus something else?)

I'd be interested in books read per person (though hard to measure) or at least books sold per capita.

<comment><USER16> They said that they didn't actually do a proper survey, 18 cities reported the amount of bookstores, and 20 reported the amount of libraries and i think those figures are way to low to be accurate to begin with. I can name at least 15 book stores within <20 min walk from where I live in London, Waterstones alone has about 50 stores in London.

I think using Google Business Directory, Yelp, Yellow Pages and a few other repositories would give a much better picture, I've checked a few random cities (both on and off their list) and the numbers are considerably higher than what the article describes.

</comment></comment><comment><USER17> Portland has Powell's, so I think they win by default. If you ever go to Portland, visit Powell's... it's fantastic!

<comment><USER18> I'm a huge fan of Powell's.

There is also a simply massive Half-Price Books in Dallas on Northwest HWY. It's like an entire city block (single level). Powell's is a sight to behold, but if you ever make it to Big D, their HPB is pretty great.

</comment></comment><comment><USER19> Any city in China shouldn't even be counted in this study. The books are filtered through the government and you are only allowed to read what is 'approved'.

I lived there for 5+ years and as an avid reader, it was infuriating.

<comment><USER20> cough...pirating ebooks...cough

<comment><USER19> Of course this works. So does a VPN back to the US. However, I shouldn't have to jump through all of these hoops for the freedom of information.

Anyone who claims China is 'free' should try living there for at least a year. It will open your eyes to the reality of the situation. It also makes you appreciate the freedoms we have in the US.

<comment><USER20> I've done 5 in total. I live in Beijing, so it's right underneath the watchful eye of the Communist Party. It's not 'free' free but outside of the Great Firewall (nuisance), and various visa and registration related shenanigans (fairly lax), I haven't gotten into situations where I really felt the iron grip of the government.

</comment></comment></comment></comment><comment><USER21> i think its a bit misleading. I grew up in munich, germany and there are not many public libraries, but they are massive. The münchner stadtbibliothek has stores (?) in every district. Then a few extra stores near major places and a big headquarter. It is roughly half the size of the new york public library for a city of only 1.5 million inhabitants. its not the only one, the state has one, the universities etc. The bavarian state library has nearly twice as many books as the new york public library.

Maybe books per capita would be a better metric to rank cities (albeit a bit more complicated to research)

</comment><comment><USER22> I do appreciate a good bookstore. How much of my money i impulsively spent on animal covered volumes thinking they'd make me smarter I cannot be sure but it must be over a thousand. However they don't sell those anymore in my city.

<comment><USER23> My city is the same. Borders used to stock a fantastic selection of technical books. Since they closed down, our only other large (multi-story) bookshop is Waterstones. Their computing selection gradually dwindled to a couple of bookcases, dedicated primarily to introductory books (getting started with Microsoft Excel, make the most of my new Macbook - that sort of thing). They had one O'Reilly book (a Docker one).

I've got so many great memories of spending time in the bookshop, looking at all the newly published tech books and the venerable classics, deciding what I was going to learn next. Books that had been edited and reviewed, giving such in-depth overviews that you could come away from them saying "I understand this technology".

Recently I've realise how reliant I've become on a mishmash of blog posts, API docs, incomplete/outdated product documentation...

I know that books haven't gone anywhere, but there's something to be said for the discoverability and tangibility of books at the bookstore vs catalog pages on a website.

And ebooks are just not a great experience (currently) when it comes to technical books with diagrams, tables and code examples. They do make for a fantastic complimentary product, however (CTRL-F when I want to quickly look up something I've already read).

</comment></comment><comment><USER24> I'm not sure about the data; Buenos Aires reported 25 libraries in a year and now it disappears from the ranking. I don't think they closed down their libraries in a couple of years?

<comment><USER25> I'm pretty sure we haven't lost that many libraries, and I have even seen new ones in recent years.

</comment></comment><comment><USER26> I am a bit surprised to see Singapore so low on the list. I wonder if they count all the small bookstores in shopping plazas? The last time I lived there (early 90's), there were a ton of micro-bookstores like this: [http://friendlybooking.com/wp- content/uploads/2014/10/Orchar...](http://friendlybooking.com/wp- content/uploads/2014/10/Orchard-Road-A-great-street-for-shopping-in- Singapore-4.jpg)

<comment><USER27> I was surprised by that too, and also the disparity with Hong Kong makes me suspect something is up with their data.

There are lots of little neighborhood bookshops in the heartlands, and also plenty of big ones like Kinokuniya, MPH, Page One in the big malls, and there was previously a huge Borders too.

The quality is a whole other issue, but for mass market stuff it's as good as anywhere.

</comment></comment><comment><USER28> This is just making me remember the days when there were dozens of bookstores here in Cambridge, most of which are gone now.

<comment><USER29> There was a time when I had a bimonthly or so ritual of driving into Harvard Square on a weekend and sending half a day poking through new and used book and music stores, in addition to various other errands. They're mostly all gone with just a few exceptions.

</comment></comment><comment><USER30> It would be interesting to see a historical plot of: # of bookstores vs cost of land vs Amazon stock price.

I see bookstores closing every year in NYC, Houston and other major cities because land prices go up, and it gets too expensive to have what is essentially a warehouse you can browse in the middle of the city when you can have it delivered.

</comment><comment><USER31> For an amazing sample size of 18 self-reports. The results will surprise you, especially #5.

</comment><comment><USER32> There might be a quality missing with ebooks, or even stories on websites (my typical method of reading something for fun today), that printed books have.

I think ordering online, and automated small batch runs are how a more luxury item (printed books) are going to be delivered in the future. Just like today you special order prints of legacy audio containers.

Cities with few bookstores only has a marginal correlation with how people obtain reading material versus the cost of having a store in that city.

</comment><comment><USER33> The headline is completely wrong.

From the article: "Over the past two years, 18 cities have reported how many bookstores they have, and 20 have reported on their public libraries."

It's not "the most book stores and libraries per capita", it's just the numbers each city sent them.

With Amazon and eBooks, I don't know if counting bookstores really matters.

</comment><comment><USER34> Aren't online bookstores real bookstores?

<comment><USER2> I'm not going to argue 'real', but the experiences are sure distinct.

</comment></comment><comment><USER35> Tehran is not even on this list. A glaring omission of a nation historically hooked on books: [http://www.irantravelingcenter.com/amazing-intellectual- trea...](http://www.irantravelingcenter.com/amazing-intellectual-treasure-of- tehran/)

</comment><comment><USER36> I've never seen a city with book vendor streets like Yangon, Myanmar.

[https://www.google.ru/search?q=yangon+myanmar+book+vendor&pr...](https://www.google.ru/search?q=yangon+myanmar+book+vendor&prmd=imvn&tbm=isch)

<comment><USER37> Chatuchak ('JJ') markets outside of Bangkok have a section with a very similar street. Many places in India also, particularly Calcutta. As these are both neighboring countries, it's not surprising to find Myanmar with a similar model of retail.

</comment></comment><comment><USER38> Interesting to read this in an office that is about 100m from the Book Festival being set up in Edinburgh's Charlotte Square:

[https://www.edbookfest.co.uk/](https://www.edbookfest.co.uk/)

</comment><comment><USER39> I cannot reply directly to wahsd, but: on the contrary, the 'large and inviting Barnes & Noble' offers a shallow selection that can be found online cheaper, while the 'hole in the wall hoarder bookstore' has a diverse stock.

</comment><comment><USER40> One reason for Hong Kong's top ranking might be because Amazon doesn't serve Hong Kong well, particularly for local titles.

<comment><USER41> Also, HK has a lot of teeny, teeny, bookstores, squeezed into the fourth floor of a building.

</comment></comment><comment><USER42> Bittorrent trackers are the best libraries.

<comment><USER43> They tend overwhelmingly toward technical manuals and sci-fi. Great if you like those things, I guess.

</comment></comment><comment><USER44> Don't worry, China has been doing its utmost to relieve Hong Kong of its #1 spot.

</comment><comment><USER45> Seattle isn't on the list. So yea, not a serious list :P

</comment><comment><USER46> If it were a survey of most temples/religious places per capita, I'll bet anything top 10 cities would be in India.</comment>


Bill Thurston's answer to "What's a mathematician to do?" (2010) - <AUTHOR> https://mathoverflow.net/questions/43690/whats-a-mathematician-to-do/44213#44213 <comment><USER2> I don't have any post graduate education. I'm ok at math, but not exceptional. I managed to publish a paper a few years ago that is mostly a math paper that I'm really proud of. [https://arxiv.org/abs/1809.04052](https://arxiv.org/abs/1809.04052) (It's a useful algorithm for big-data work, but it's also a bit of math about how to align probability distributions so they collide as much as possible. I tell my kids that daddy discovered something new about how triangles fit together.)

There are lots of useful corners of math out there, lots of things that are worth thinking about that no one has thought about just because there are so many things to think. There are plenty of things worth poking at that aren't The Big Problems.

<comment><USER3> You like math? You do math? Guess what? You're a mathematician, regardless of post-graduate education or self-assessment of your ability!

(I love to see 'amateur' mathematics, not in the derisive sense of the word but in the formal sense of "not done by a professional mathematician". Good on you!)

</comment></comment><comment><USER4> Here's a longer article by Thurston on the same ideas [https://arxiv.org/abs/math/9404236](https://arxiv.org/abs/math/9404236). Highly recommend irrespective of how involved you are with mathematics.

</comment><comment><USER5> Thurston's answer is obviously well meaning and trying to breath inspiration into people who are pursuing maths as a discipline, but it should be observed anyone taking him at his word without context will be led astray. It isn't useful guidance for a young aspiring mathematician because it is generic, contains nothing actionable and not even a clue on how to judge a successful/failed state as they move through life.

Even "your name will go down in history if you consistently find patterns in daily life that other people can't see" would probably be more useful advice to the next Euler or Gauss.

> The product of mathematics is clarity and understanding.

Mathematics is clarity and understanding of _mathematical objects_ which is a very small subset of the things that most people seek when they go looking for clarity and understanding. Anyone looking for clarity and understanding in the abstract is better off starting their search in the Psychology or possibly the Philosophy departments.

I don't think he was being arrogant with that quote, but I do think that it is the perspective of someone who has spent so much time looking at maths they might have lost track of all the social manoeuvring that is what satisfies most humans. In my case I'd rather have a deep understanding of what someone is saying to me than of Fermat's Last Theorem - communication abilities tends to be more of a bottleneck to satisfaction than abstraction abilities. Even in Thurston's answer, he is alluding to the fact that communicating with other mathematicians is as important to him as understanding abstract concepts.

> follow your heart and your passion. Bare reason is likely to lead you astray

This is lousy advice. Following your passions only works for people lucky enough to have productive passions. A lot of people are passionate about eating good food - if they want to be productive they will need a plan other than following their passions.

<comment><USER6> I think Thurston's answer is absolutely to the point.

If you are passionate enough about good food, you probably have a great shot at becoming a famous chef. I agree though that it is very dangerous advice: most people are just not passionate enough about something, but mistake fondness for passion. I'd say that applies to your "good food" example. But on the other hand, for truly passionate people it is very dangerous NOT to follow this advice.

> The product of mathematics is clarity and understanding

Somewhere else Thurston qualifies this in a recursive definition of mathematics that is bootstrapped with numbers and geometrical objects. I'd say in the age of the computer this qualification becomes less and less necessary: there are other things than numbers and geometrical objects that are of interest (for example distributed file systems). So more and more things are becoming amenable to clarity and understanding, if we try hard enough. I think a lot of things in computing could use a good helping of clarity and understanding.

<comment><USER7> Love the phrase, "mistake fondness for passion." Thanks!

</comment></comment></comment><comment><USER8> RIP Bill Thurston! Thank you for sharing!

His mini-essay made a lot of sense. I think he nailed it in the beginning by saying that the world collectively benefits from Mathematics as a whole. Or rather, benefits are a "side effect" of people's Mathematical achievements.

</comment><comment> P-NP I like the Feynman quote in the comments: "You keep on learning and learning, and pretty soon you learn something no one has learned before."

<comment><USER9> I like that quote and I like Feynman in general, but I think it's worth keeping in mind that Feynman was much more gifted than most people, even other physicists. Some colleague of Feynman once said he thought Feynman harmed some students' development because Feynman didn't realize how intelligent he (Feynman) was and that he would suggest students approach problems in ways that they just didn't have the ability for.

</comment></comment><comment><USER10> That's beautiful -- thank you for sharing the post. I don't get to think about pure math very often any more, but this was a wonderful reminder of clarity in a world that's a little more disordered than usual.

One of the little joys of math, well known, but always makes me smile: Though there are infinitely many rational numbers, they are countable.

<comment><USER3> > One of the little joys of math, well known, but always makes me smile: > Though there are infinitely many rational numbers, they are countable.

I think 'infinite but countable' isn't so surprising—even without a formal definition, I think most people would expect the counting numbers to be countable—but maybe the fact that there are infinitely many _more_ rational numbers than counting numbers, and yet there are exactly as _many_ rational numbers as counting numbers?

(This is one of many ways of phrasing it, but it perhaps understates how much bigger the rationals seem to be than the counting numbers; the description I've given would apply as well to the set of all integers, whose countability is still perhaps surprising, but not _as_ surprising.)

<comment><USER11> There's one for each of 'em, to use an idiom. A bijection exists between the Z and Q. Both sets are exactly the same size. Cantor's argument is certainly quite clever, but "there are infinitely many more rational numbers than counting numbers, and yet there are exactly as many rational numbers as counting numbers" is nonsensical to me. Am I missing something?

<comment><USER12> |Q-Z| is countably infinite, but |Q| = |Z|. There are countably infinite rational numbers that are not counting numbers, yet there are exactly as many rational numbers as counting numbers.

</comment></comment></comment></comment><comment><USER13> Russell once wrote "Work is of two kinds: first, altering the position of matter at or near the earth's surface relatively to other such matter; second, telling other people to do so." Maths is a constructive proof that his case analysis was not exhaustive.

</comment><comment><USER14> "This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help "

Yeah, that's StackOverflow for ya (or MathOverflow I guess).

</comment><comment><USER15> closed as interesting</comment>


No, Google's AI Program Can't Build Your Genome Sequence - <AUTHOR> https://www.forbes.com/sites/stevensalzberg/2017/12/11/no-googles-new-ai-cant-build-your-genome-sequence/#61163ec25774 <comment><USER2> Salzberg's articles are really just ads for his own software. Read carefully, you'll see what he's doing is touting his own software while throwing shade on other people. He did the same thing when we launched Google Cloud Genomics.

That said, Wired is about the worst scientific coverage around.</comment>


Split Stacks in GCC - <AUTHOR> http://gcc.gnu.org/wiki/SplitStacks <comment><USER2> If you twist your perspective a little, you can change how you view the stack; instead of looking at it as a literally linear blob of data, consider it the 'data' or 'scope' pointer of a closure (frequently implemented as a pair of data and code pointers). A key element of this closure is the continuation closure, which is implicitly passed in to closures when they get called; the 'RET' type of CPU opcode is the normal way of calling this closure.

If one carefully adds stack splitting and duplication capabilities to an ordinary procedural language, it amounts to a conversion to continuation- passing style. Not only does that cut down on some of the drawbacks of massive multithreading on 32-bit (address space exhaustion, though not the expense of context switches), but it also permits interesting implementation strategies (e.g. making call/cc work in C or languages implemented in terms of C, which can in turn be applied to implementation of stateless servers).

</comment><comment><AUTHOR> Stack splitting is also central to efficiently implementing parallel logic programming languages. This was the missing insight required for efficient or- parallelism that the 5th generation computing project was hoping for.

Nice summary and much more about that here: [http://bizrules.info/conference/ORF2008DFW/GopalGupta_Buildi...](http://bizrules.info/conference/ORF2008DFW/GopalGupta_BuildingUltimateRuleEngine_ORF2008.pdf)

<comment><USER3> Wow, that IS a nice summary and bibliography. Thanks for posting it!

There are some other good presentations on C/LP in the same directory, too.

</comment></comment><comment><USER4> Curious idea, clearly borrowing from managed runtimes with garbage collected stack frames.

Random idea (feasible only where address space is plentiful): why not let the operating system's virtual memory subsystem worry about allocating stack space (reserve a large stack with mmap(), let the page faults be responsible for actually allocating the memory and periodically munmap() and re-mmap() the upper part of the stack.

<comment><USER2> This is exactly how Windows allocates the stack. The address space is reserved, but only the first page is comitted; the next page after that is called the guard page. Touching the guard page causes an exception, and the first-chance handler for that exception commits the page, sets up a new guard page at the next position, and resumes execution.

Compilers targeting Windows need to generate stack touching prologues for procedures that allocate more than 4K of local variable data, or else they risk code first touching the page beyond the guard page, which will cause a much more unpleasant access violation.

<comment><USER4> Ah, I vaguely remember encountering that years ago now that you mention it. I assume it doesn't actually ever purge pages that go unused for a while unless the thread exits?

</comment></comment></comment><comment><USER5> I've quickly skimmed through it, but when reading it thought about he's gonna handle alloca() (and later I've read dynamic arrays).

So the solution is to dynamically allocate them, and at the end of stack routine free them. That's all good, but what if you longjmp() and have alloca() function still allocated. That would leak... Unless unwinding is done, and RAII kind of type for alloca is made. (Or maybe wrap alloca in C++ RAII, or maybe that's what the compiler would be doing).

</comment><comment><USER6> Here's a related paper from 2003 called 'Capriccio: Scalable Threads for Internet Services'. It uses call graph analysis to figure out where to place checkpoints that allocate and deallocate stack frame chunks.

[http://ece.ut.ac.ir/Classpages/S86/ECE404/Papers/capriccio.p...](http://ece.ut.ac.ir/Classpages/S86/ECE404/Papers/capriccio.pdf)

<comment><USER7> The only scalable thing for "Internet Services" I know is non-blocking I/O, and just one thread for every CPU physical thread (i.e. avoiding thread context switching).

Edit: After RTFM, is the purpose of the paper, simulating threads handling blocking I/O with a epoll and/or select. Great paper.

</comment></comment><comment><USER8> > It becomes possible to run millions of threads (either full NPTL threads or > co-routines) in a 32-bit address space

If the minimal "reserved stack space" per thread is 4K, than a million threads would consume about 4GB of memory. That's not quite feasible in a 32-bit address-space. Would most threads run with even less than 4K of reserved stack space?

<comment><USER9> Using a 4k minimum was just one of the possible implementation strategies. It looks like there are several ways to get it much, much lower.

</comment></comment><comment><USER10> On a 64-bit architecture, this is completely pointless as there is no problem with the address space. Memory is also not a problem since all modern operating system allocates stack memory as needed.

So, my question is why spend time on implementing this?

<comment><USER8> > Memory is also not a problem since all modern operating system allocates > stack memory as needed.

A modern OS can only allocate memory as needed with a page-level granularity.

For many millions of threads, this is not good enough and will waste GB's of memory completely unnecessarily. Maybe in a decade wasting GB's of memory will be negligible (but then we might want trillions of threads!). And pages seem to be growing too, so the OS granularity is too coarse for micro-threads.

</comment></comment><comment><USER7> What about side effects such as heap memory fragmentation?

</comment><comment><USER11> While we're at it, could someone fix Java so that I don't have to guess the f*!$ing heap size all the time?</comment>


Chrome-style OmniBar for Safari - <AUTHOR> https://github.com/rs/SafariOmnibar

<comment><AUTHOR> As much as Chrome has been my favorite browser for Mac for a while now, with the release of Lion I've really been longing after some of the features of Safari - namely the well integrated full screen mode, and the iOS style zooming.

However, I can't stand having a separate URL bar and search box. This plugin seems to be working really well at emulating the Chrome omnibox!

(I have no association whatsoever with the development of this plugin, just found it by googling for safari omnibus)</comment>


Ask HN: A real-life use case for Lisp macros that beginners can understand? - <AUTHOR> I'm a novice in Lisp (specifically Clojure) and I got interested in the language because of the macro system.
However, all the examples I've seen so far seem pretty convoluted and just made me think "Why would I want to do that?". I think I just didn't "get" it yet.
Is there a simple real-life example that shows the usefulness of the Lisp macro system to a beginner? <comment><USER2> > _Is there a simple real-life example that shows the usefulness of the Lisp > macro system to a beginner?_

No; all the simple macro examples are in fact in the "why would I do that" category. Their value is the _how_ , rather than the _why_.

A very simple macro can often be eliminated by writing whatever it writes by hand without a lot of additional effort, and without the baggage of carrying the macro definition. It doesn't seem useful, especially if you fail to imagine it being used thousands of times, rather than just the two or three times that occur in the example.

It's the convoluted macros that answer the "why": because if you don't have the convoluted macro to generate code in a complicated way, and you need to do that dozens of times, then your alternative is to do it all by hand, which is tedious and error-prone. THe macro does the job consistently, and if there is a bug, you just fix it in the macro and re-expand all the uses of it to spread the bugfix.

</comment><comment><USER3> You are thinking about this wrong ... and it's not your fault, because everyone (a large subset of lispers included) thinks about this wrong. You can approach lis programming as normal programming, if you like, but that's not the right way to approach lisp. The way to approach lisp is to answer the following question: If there was a way I could wave a magic wand, and a special purpose programming language would be created just for my problem, what would it look like? Lisp macros are that magic wand. This seems abstract, but it's not. Really, take your favorite problem, imagine a programming language (aka. DSL) that perfect for that problem. Write down what it would look like. (The one constraint is that the top level has to be in sexpr format, but even that's not deeply required!) Now use macros to translate your fantasy into whatever implements it algorithmically. Poof! You're a lisp hacker!

<comment><USER2> I'm sorry, but that's only a good way to approach Lisp programming when you know everything that your language comes with, and are sure that none of it provides a good representation for the problem.

Otherwise, the way to approach Lisp programming is the same as with any other language: how can the existing syntax and library features (such as macros!) be orchestrated to solve this problem?

Which kinds of macros would be useful will become apparent during development. You often don't know what language features may be nice to have until you have worked in that problem domain.

Lisp programming sometimes happens without knowing what the ideal language of the problem domain would look like; you don't know that upfront to just it down and make macros for it.

Secondly, there may be third-party libraries for the problem domain. For instance, if you're sure that the problem domain requires solving goals stated as queries over logical predicates, that doesn't mean you have to implement your own Prolog in macros.

So the priority is roughly: built-in macros > stable third-party macros > experimental/own macros.

<comment><USER3> QED

<comment><USER2> I suspect you've never written Lisp code that was peer-reviewed or maintained by anyone other than yourself.

<comment><USER3> You'd be really really wrong about that.

</comment></comment></comment></comment></comment><comment><USER4> It might sound cliche by now, but I think you can glean a whole lot from the book Practical Common Lisp, by Peter Seibel.

It's one of those books I learned from online, but I like it so much and kept on returning to it. So I bought a copy, and I _still_ read it and write notes in the margins. It's that good. I don't call many things "bibles" but PCL is pretty much the Common Lisp bible. And it is as practical as the title suggests (as per your request!)

Chapters 7 and 8 are both entirely about Lisp macros. But even then you aren't done with learning about macros yet. The natural progression of the book means you keep building on the prior concepts until the last chapter.

Of course, the CD database you create while working through the book is a tad naive, but the idea isn't to make a commercially viable application - it is to demonstrate Lisp _thoroughly_ by creating apps right from the get-go. And you would do exactly the same things in a commercial setting.

PS: It's Common Lisp and not Clojure. But you were asking about macros specifically, and the Lisp flavour doesn't matter much at all when it comes to how macros work. This book will lead you in the right direction.</comment>


Django 1.6 alpha 1 released - <AUTHOR> https://www.djangoproject.com/weblog/2013/may/26/django-16-alpha-1/

<comment><USER2> This is the most exciting and interesting thing in the release:

Persistent database connections

Django now supports reusing the same database connection for several requests. This avoids the overhead of re-establishing a connection at the beginning of each request. For backwards compatibility, this feature is disabled by default. See Persistent connections for details.

<comment><USER3> > This is the most exciting and interesting thing in the release:

Absolutely. My expectation is that this alone should dramatically increase app response times for any reasonably DB-dependant application.

<comment><USER4> It does. You can currently do this with an external Django DB Pool module, for some numbers see: [http://mixedbit.org/blog/2013/02/06/django_db_pool_on_heroku...](http://mixedbit.org/blog/2013/02/06/django_db_pool_on_heroku.html)

I also can't wait to have such mode fully supported by the core product.

</comment></comment></comment><comment> deepak-kumar Good. Keep it coming. For those who are interested further, here are the release notes to see "What's new in Django 1.6" <http://j.mp/13XRnoD>

<comment><USER5> Unshortened URL for the release notes: <https://docs.djangoproject.com/en/dev/releases/1.6/>

</comment></comment><comment><USER6> Have they moved to a faster release schedule? I feel like 1.5 just came out and we can already expect 1.6 in August (not that I'm complaining about it).

<comment><USER7> We're trying to, yeah.

For this release specifically we wanted to have it out before the end of summer so that we can be ready to merge Andrew's migration work and any GSoC code and turn around another release quickly.

</comment></comment><comment><USER8> I'm amazed by the rate of the Django releases lately; the core devs must be very busy.

</comment><comment><USER9> Pardon my ignorance but, `Persistent database connections`

Is this the same as pgBouncer connection pooling? It does sound like the same - but unsure. I just spent a week learning and configuring pgBouncer! :(

<comment><USER7> Not exactly. It's not proper connection pooling -- for that, you still should be using an external pool like pgbouncer. Instead this is a single persistent connection per thread. The speed improvement should be about the same, but you something like pgbouncer gives you more control over the number of connections and offers a ton of extra features.

<comment><USER9> Thank you very much for the clarification `jacobian` - I am assuming that django's connection pooling on 1.6 would still play nice with pgBouncer.

<comment><USER7> Mmmm.... I mean, it would, I guess, but I wouldn't recommend using both unless you can show a marked speed improvement over just using pgbouncer. A real connection is (probably) better than persistent connections, so you probably only need the one.

Now, I've not benchmarked this, so it's just a guess. Further, the specific performance characteristics will depend on your specific setup -- particularly upon where you have pgbouncer running -- so the only real answer is to benchmark and see.

New Relic does a good job showing time spent connecting to the db, fwiw, so you may want to give that a try.

<comment><USER9> Thanks again for your answer - I have noted this down now. I will benchmark both cases (CONN_MAX_AGE and PgBouncer) and see which one is faster. Would like to thank you (!!) for the awesome python framework - Django.

</comment></comment></comment></comment></comment><comment><USER10> Are there usually this many backwards-incompatible changes? In the past I've usually been able to upgrade Django without having to do any significant changes in the code.

<comment><USER11> So, it's a big list. But a lot of them are either:

* Fallout from transaction changes. Whether this affects your code will vary, and when and how it can affect your code is documented, or

* Bugfixes: lots of the stuff in there consists of "we had a bug, we fixed the bug, but you should know this in case you relied on the buggy behavior".

Split those out and it's actually quite a bit shorter.</comment></comment>


A Visual Introduction to Machine Learning - <AUTHOR> http://r2d3.us/visual-intro-to-machine-learning-part-1/ <comment><USER2> Previous discussions:

[https://news.ycombinator.com/item?id=10189259](https://news.ycombinator.com/item?id=10189259)

[https://news.ycombinator.com/item?id=9955553](https://news.ycombinator.com/item?id=9955553)

Other submissions, but without discussion:

[https://news.ycombinator.com/item?id=10046596](https://news.ycombinator.com/item?id=10046596)

[https://news.ycombinator.com/item?id=10008432](https://news.ycombinator.com/item?id=10008432)

[https://news.ycombinator.com/item?id=9991141](https://news.ycombinator.com/item?id=9991141)

[https://news.ycombinator.com/item?id=9983828](https://news.ycombinator.com/item?id=9983828)</comment>


Google maps 8bit quest mode.  A setup for 4/1 perhaps? - <AUTHOR> http://maps.google.com/maps?q=queens,+ny&hl=en&sll=37.0625,-95.677068&sspn=40.188298,71.630859&t=8&hnear=Queens,+New+York&z=11

<comment><USER2> Try going to 0.00,0.00 and zoom in.

(spoilers warning)

This is what you get: <http://i.imgur.com/JJxGn.png>

There's also another one in a deeper zoom level.

<comment><USER3> Madison, Wisconsin (<http://g.co/maps/pnsev>) has an imagined Monona Terrace as well as some kids with silly hats.

<comment><USER4> The region around MIT in Cambridge, MA shows similar detail: [https://maps.google.com/maps?q=cambridge+ma&hl=en&ll...](https://maps.google.com/maps?q=cambridge+ma&hl=en&ll=42.361556,-71.091027&spn=0.012652,0.01929&sll=43.076913,-89.380302&sspn=0.012507,0.01929&t=8&hnear=Mid- Cambridge,+Middlesex,+Massachusetts&z=16)

</comment></comment></comment><comment><USER5> Youtube announcement:

<http://www.youtube.com/watch?v=rznYifPHxDg>

</comment><comment><USER6> Interesting, if you look at Pyongyang (North Korea), it has a bunch of labels that don't show if you use their regular maps. They're quite detailed too, it shows "Statue 2 of 'The Newly Freed Proletariat Thanks To The Grat Leader Kim Il-Sung'", schools and such. They must have hidden map data they don't show normally that leaked into this map. <http://g.co/maps/tg9jy>

</comment><comment><USER7> A nice little detail for Pittsburghers: <http://i.imgur.com/fQR4u.png>

I wonder how many cities they made landmarks for?

<comment><USER8> Here's a tumblelog of easter eggs, open for submissions: <http://gmaps8biteastereggs.tumblr.com/>

<comment><USER9> Tokyo, Japan:

[http://maps.google.com/maps?q=Tokyo,+Japan&hl=en&ll=...](http://maps.google.com/maps?q=Tokyo,+Japan&hl=en&ll=35.68808,139.694595&spn=0.024225,0.052357&sll=35.689488,139.691706&sspn=0.775175,1.675415&oq=tokyo&t=8&hnear=Tokyo,+Japan&z=15)

<comment><USER10> Sky Tree (recently huge Tower built in Tokyo) here: [http://maps.google.com/maps?q=Tokyo,+Japan&hl=en&ll=...](http://maps.google.com/maps?q=Tokyo,+Japan&hl=en&ll=35.707667,139.809952&spn=0.031502,0.066047&sll=35.689488,139.691706&sspn=0.775175,1.675415&oq=tokyo&t=8&hnear=Tokyo,+Japan&z=15)

</comment></comment></comment></comment><comment><USER11> It's a pretty impressive map for an April Fool's joke. Particularly like the 2d rendered buildings for landmarks like the Eiffel Tower, Sutro Tower in SF, etc. Presumably somewhat inspired by Brett Camper's 2010 project 8bitNYC. <http://8bitcity.com/map?New%20York>

The YouTube video is hilarious.

</comment><comment><USER12> Google is finally going to support the NES: <http://www.youtube.com/watch?v=rznYifPHxDg>

<comment><USER13> Even though I think this is great, it still kind of annoys me that they keep calling it a NES when they're clearly using a Famicom. I suspect that's actually going to confuse a lot of people.

<comment><USER14> It looks like the video was made by Japanese, but they probably used the name "NES" because it's more familiar to English-speaking audiences who're likely to make up the majority of video views.

<comment><USER10> This was probably made by their Google Japan branch, that's why they only had regular Famicom available.

</comment></comment></comment></comment><comment><AUTHOR> Drag the hero on to the map somewhere and he will land somewhere with streetview, which gets turned into 8bit, hilarious.

<comment><USER15> That's pretty great. I've wanted for a while to find a way to layer image effects onto Google Street View, to be able to move around the world in a sort of alternate processed reality. But afaik, there's no straightfoward way to add a filter like that unless you're Google. There's an image API, but the resolution is only 640x480. There's an undocumented image API with higher-res images, but you'll get banned if you make more than occasional usage of it. Perhaps it'd be possible to do it via a browser extension that renders the effects locally?

[edit: Just thought of one possible route. There's a WebGL beta for Street View; perhaps you could add effects by injecting some WebGL shader code, given appropriate local security permissions?]

So far I've been just experimenting with grabbing a few hundred images from the undocumented API and applying effects locally. This is the result of one experiment (hit [reload] at the top/right for new images): <http://www.kmjn.org/projects/cyanocph/>

<comment><AUTHOR> nice! I really like that. Thanks for posting this.

</comment></comment></comment><comment><USER16> Dragon Warrior! Man, I loved that game, despite its flaws. The gameplay was really slow-paced, and the random battles were kind of annoying. But it was unabashedly itself, and I guess I fell in love with the world and the good ol' swords-and-sorcery style. It helps that the NES is just an excellent system—mine's still in perfect working order.

<comment><USER17> Yeah the music in the video really brought me back for a second. I was just thinking about my old NES wondering where in my parents house it could be and if I could play earthbound.

</comment></comment><comment><USER18> 8bit city with map tiles have been for awhile now using Openstreetmap as the source. An example that has been arou d for years is this <http://8bitcity.com/map?New%20York>

</comment><comment><USER19> Seriously, more casual maps can work well in presenting information. <http://sh.edushi.com/>

<comment><USER20> That's gorgeous! I found more info here if anyone's interested (I can't read Chinese): [http://www.popsci.com/scitech/article/2009-06/edushis-3-d-pi...](http://www.popsci.com/scitech/article/2009-06/edushis-3-d-pixel- art-maps-chinese-cities-put-google-maps-shame)

</comment></comment><comment><USER21> You can use these 8bit tiles in your own project, its quite easy to switch the tile layer of any google maps site: [http://blog.bn.ee/2012/03/31/how-to-use- google-maps-8-bit-ti...](http://blog.bn.ee/2012/03/31/how-to-use-google- maps-8-bit-tiles-in-your-own-project/)

I updated walksy.com to use these tiles: <http://walksy.com/?8bit>

<comment><USER22> You show Nob Hill, San Francisco as a part of the Presidio:

"Nob Hill California and Mason, SF Tags: neighborhood, architecture, history, parks 34 reviews on [http://www.yelp.com/biz/felicia-gelardi-skin-care-san- franci...](http://www.yelp.com/biz/felicia-gelardi-skin-care-san-francisco) StreetView"

That's very much the wrong neighborhood.

<comment><USER21> This is fixed - thanks for pointing that issue out!

</comment></comment></comment><comment><USER23> Google offices are apparently marked with surprisingly-unpixelated propellerheads. There's also some custom art for SF landmarks.

<comment><USER24> I was so hoping I could see the propellerheads in closer detail with the Street View, but alas...

</comment></comment><comment><USER25> Funny, I met Brett of <http://8bitcity.com/map> this week ;)

</comment><comment><USER26> Google Streetview is also available in 8 bit mode:

[http://maps.google.com/maps?q=googleplex&hl=en&ll=37...](http://maps.google.com/maps?q=googleplex&hl=en&ll=37.421725,-122.083747&spn=0.012611,0.023217&sll=52.531887,13.434992&sspn=0.038428,0.092869&t=8&hq=googleplex&radius=15000&z=16&layer=c&cbll=37.420836,-122.084885&panoid=tLjwO4QEuljVhPL1ERQ8lw&cbp=12,0,,0,0)

<comment><USER27> ...only in the Flash viewer though: not on WebGL (to save anyone clicking the link who's opted into the WebGL beta).

<comment><USER28> a possible word of warning: I disabled MapsGL to see it. In the past there has always been a button to get back into GL mode, but I can't find it anymore so I am now stuck in Flash.

<comment><USER27> Yes, me too. I _assume_ it will be coming back post April 1st...

</comment></comment></comment></comment><comment><USER29> As of an hour and a half ago, it's April 1 in Japan.

</comment><comment><USER30> My first instinct was to look under the hood for some sprites I could use for a map based game. No luck, all the cool building/characters seem embedded in the the actual tiles.

Is there any way Google would try to make an actual game out of this? I know there are other successful online 8-bit-ish RPGs that use the Google Maps as a base...

</comment><comment><USER31> Can anyone identify this strange object?

[http://maps.google.com/maps?q=queens,+ny&hl=en&ll=39...](http://maps.google.com/maps?q=queens,+ny&hl=en&ll=39.741102,-75.322115&spn=0.005956,0.010697&sll=37.0625,-95.677068&sspn=40.188298,71.630859&t=8&hnear=Queens,+New+York&z=17)

<comment><USER32> Looks like a druinlord.

</comment></comment><comment><USER33> Cool they did the montreal Biosphere and the Habitat 67 building but no Olympic stadium :(

[http://maps.google.com/maps?q=Biosph%C3%A8re,+Montreal,+QC,+...](http://maps.google.com/maps?q=Biosph%C3%A8re,+Montreal,+QC,+Canada&hl=en&ie=UTF8&ll=45.507974,-73.532529&spn=0.030406,0.066047&sll=46.104145,-74.299193&sspn=3.850534,8.453979&oq=biosphere+&t=9&hq=Biosph%C3%A8re,+Montreal,+QC,+Canada&z=15)

</comment><comment><USER34> A troll or something near Mount Rainier: [http://maps.google.com/maps?q=47.04299,-122.051239&hl=en...](http://maps.google.com/maps?q=47.04299,-122.051239&hl=en&ll=47.040211,-122.036251&spn=0.005111,0.011362&sll=47.042054,-121.784821&sspn=0.658835,0.553436&t=8&z=173436&t=8&z=10)

</comment><comment><USER35> Just look on the copyright message in bottom right: SQUARE ENIX. Final fantasy.

<comment><USER36> Actually Dragon Warrior not FF, which was an Enix property pre-merger.

</comment></comment><comment><USER37> Monster: <http://goo.gl/ER97T>

<comment><USER26> I created a tumblr to collect all the monsters!

<http://map-monster.tumblr.com/>

Found a monster? Leave a link into the comments.

<comment><USER38> Try:

Volgograd

Pogradec

Randazzo

Paceco

Balfour, Mpumalanga, South Africa (zoom out)

Makasa, Northern, Zambia (zoom out)

Goch

Mallow, Ireland

Just found them by searching around. Not sure if there's any logic to where they're placed, besides on land.

</comment></comment></comment><comment><USER39> Discussion on reddit:

[http://www.reddit.com/r/technology/comments/rmfr1/browse_goo...](http://www.reddit.com/r/technology/comments/rmfr1/browse_google_maps_in_8bit/)

</comment><comment><USER40> If you try to replicate the route from Los Angeles to New York on Google Maps 8-bit Quest mode, it's not nearly the same route as the one in the video. Curious they missed that.

</comment><comment><USER41> It's been April Fools Day in New Zealand for 14 hours, so I guess rather than it being setup in advance, they just launched it in time for the first country to hit 1/4.

<comment><USER42> It was created by the Tokyo office, so was launched on time for them. :)

</comment></comment><comment><USER43> There's a :P monster lurking at 35.371205,138.639039. He alludes you despite zooming in. There are some interesting images all over this area of the map.

</comment><comment><USER44> Try searching for something, then specify the time range. Check the last option (changes on reload) :)

</comment><comment><USER45> If it is possible post links to images not to google map links because they do not work for everyone

</comment><comment><USER46> Somehow I got the map to split-screen with street view, and all the street view's in 8-bit, too.

</comment><comment><USER47> Damn google, I don't want to look at it, I want to play it!!!

</comment><comment><USER48> Haha, they're insane.. I love this. Great April fools'

</comment><comment><USER49> Reminds me of Ultima II. Or was that Ultima ][ ?

<comment><USER50> Specifically it is the original Dragon Quest (Dragon Warrior in the US).

<comment><USER51> Yes: <http://en.wikipedia.org/wiki/File:Dragon_quest_battle_2.png>

</comment></comment></comment><comment> why-el This looks like an early April's fool to me.

<comment><USER52> Why early? It's already April 1st in eastern part of the world.

<comment><USER22> Everyone knows the Internet only work in California.

</comment></comment></comment><comment><USER53> I'm curious how this was found?

<comment><USER54> There's a quest button in the top right when using Maps, as of a hour or so ago I guess.

<comment><USER55> Matt Cutts tweeting is how I found out.

</comment></comment></comment><comment><USER56> Try Street View! Awesome.

</comment><comment><USER57> Good lord, nothing's made me miss Zelda and SimCity as much as this. Kudos Google.

</comment><comment><USER58> Oh god the world is SimCity Classic...

</comment><comment><USER59> Launching these changes couldn't be quick, easy, or cheap, especially when accompanied by the polished intro video, and while incredibly cute, I wonder how many here would have preferred for Code Search to continue life in exchange for not expending resources on feel-good projects like this, or e.g. the frequent doodles, the novelty of which has long since worn off.

<comment><USER60> True, but—a happy employee is a productive employee. If you get to build something a bit silly, a bit cheeky, a bit different every now and again—wouldn't you like your job just that little bit more?

</comment></comment><comment><USER61> Does anyone else see these projects, the always changing Google logo, and G+ as proof that Google just has way too much talent and not enough useful ideas?

<comment><USER62> I think Google likes to have fun.

<comment> steve-howard Further, if your job is somewhere that you have fun and are free to occasionally make interesting-but-useless things, you're less likely to quit.

</comment></comment></comment><comment><USER63> This looks... erm ... neat. A "sophomore OpenGL term project" kind of neat. Baidu on the other hand does the maps right [0], which hardly surprising given their 20 years of head start. On a more general note, this reminds me of old Microsoft's tactics. Google should really stick to the search, but instead they throw together something that mimics competitor's feature. Something that looks more featureful and which is free, but upon closer inspection is effectively a half-ass effort, because it's an entirely different domain that's not their specialty.

[0] [http://map.baidu.com/?newmap=1&l=15&tn=B_DIMENSIONAL...](http://map.baidu.com/?newmap=1&l=15&tn=B_DIMENSIONAL_MAP&c=1070242,8946925&cc=sh&s=cur%26curtp%3D0%26wd%3D%E4%B8%8A%E6%B5%B7&sc=0)

<comment><USER64> And now I'm left wondering how this comment relates to <http://news.ycombinator.com/item?id=3777709> from yesterday...

<comment><USER65> The same comment, from two different accounts. Both related to Google products. Is this the 2012 version of "Netcraft confirms that BSD is dying"?

solsenNet 32 minutes ago | link This looks... erm ... neat. A "sophomore OpenGL term project" kind of neat. Baidu on the other hand does the maps right [0], which hardly surprising given their 20 years of head start. On a more general note, this reminds me of old Microsoft's tactics. Google should really stick to the search, but instead they throw together something that mimics competitor's feature. Something that looks more featureful and which is free, but upon closer inspection is effectively a half-ass effort, because it's an entirely different domain that's not their specialty. [0] [http://map.baidu.com/?newmap=1&l=15&tn=B_DIMENSIONAL...](http://map.baidu.com/?newmap=1&l=15&tn=B_DIMENSIONAL..).

huhtenberg 23 hours ago | link | parent | flag This looks... erm ... neat. A "sophomore OpenGL term project" kind of neat. Wolfram on the other hand does the graphs right [0], which hardly surprising given their 20 years of head start. On a more general note, this reminds me of old Microsoft's tactics. Google should really stick to the search, but instead they throw together something that mimics competitor's feature. Something that looks more featureful and which is free, but upon closer inspection is effectively a half-ass effort, because it's an entirely different domain that's not their specialty. [0] [http://www.wolframalpha.com/input/?i=sqrt%28x*x%2By*y%29%2B3...](http://www.wolframalpha.com/input/?i=sqrt%28x*x%2By*y%29%2B3..).

<comment><USER66> Haha what are these some weird PR-bots? (or in this case bad PR-bots)

<comment><USER65> Or astroturfing...</comment></comment></comment></comment></comment>


Pandas on Ray – Early Lessons from Parallelizing Pandas - <AUTHOR> https://rise.cs.berkeley.edu/blog/pandas-on-ray-early-lessons/ <comment><USER2> How does this compare to dask.distributed? Dask dataframes are also a wrapper on pandas API.

edit- They explain differences in a section of this blog post: [https://rise.cs.berkeley.edu/blog/pandas-on- ray/](https://rise.cs.berkeley.edu/blog/pandas-on-ray/)

</comment><comment><USER3> Does anyone here know if Ray is some sort of Yarn competitor? If not what problem space is it in?

<comment><USER4> I don't know what Yarn is, but Ray is a distributed heterogeneous computing framework. It's meant to make it easy to take fairly arbitrary computational programs (with machine learning being an important test/use case) and run/debug them in parallel across lots of machines with high performance in a natural fashion and without drastic changes. [1] The advantage compared to (say) Hadoop is that it allows for heterogeneous programming models and isn't limited to or designed for (say) MapReduce; you can invoke functions in parallel in a dynamic fashion pretty easily.

[1] You can get an idea of that here: [https://ray.readthedocs.io/en/latest/](https://ray.readthedocs.io/en/latest/)

</comment></comment><comment><USER5> Code Repo: [https://github.com/modin-project/modin](https://github.com/modin- project/modin)

<comment><USER6> For those of you confused like me, "Pandas on Ray has moved into the Modin project"

[http://ray.readthedocs.io/en/latest/pandas_on_ray.html](http://ray.readthedocs.io/en/latest/pandas_on_ray.html)

</comment></comment><comment><USER7> This could be really helpful for implementations that were written with relatively smaller datasets in mind but now need to be scaled up. However, for someone starting from scratch, it is not clear what advantages do they plan to offer against Spark used with the Dataframe API.

</comment><comment><USER8> Unclear what this is good for.

<comment><USER9> It is a WIP distributed Pandas implementation. Allows you to spread massive dataframes, their data and computations, over multiple machines vs. Pandas which is only local to your machine. It's not quite there yet [1]

[1] [http://modin.readthedocs.io/en/latest/pandas_on_ray.html#usi...](http://modin.readthedocs.io/en/latest/pandas_on_ray.html#using- pandas-on-ray-on-a-cluster)

<comment><USER10> Pandas replacing PySpark? Sign me up.

<comment><USER11> Dask is already this! They have a dataframe replacement, a numpy array replacement, and some lower level primitives like dask.delayed too. Plus, the nice thing is that it's already being used with large amounts of data, and the warts (which were plentiful two years ago) are rapidly reducing.

[http://matthewrocklin.com/blog/work/2018/06/26/dask- scaling-...](http://matthewrocklin.com/blog/work/2018/06/26/dask-scaling- limits)

<comment><USER10> Nice, I've heard of it before but never used. How is deployment compared to Spark?

<comment><USER11> The deployment is /way/ more simple, especially if you already have a python environment (even more so with conda), which is one of the main attractions for me. The other is that the API is way richer - spark dataframes tie your hands down in so many ways and require you to write tons of custom code for routine stuff, while pandas (and dask) has builtins for almost everything imaginable these days.

The tradeoff is that spark and hadoop in general have invested /serious/ efforts into resilience, while dask only really provides protection against worker failure, but not really scheduler failure. In practice is this really an issue? _shrug_ it depends. "Works for me". How many tasks do you run in parallel? If you're doing ad-hoc analysis on a very large dataset, dask might be a great fit. If you have a data warehouse use case and have tons of people running analytics queries, then you have uptime requirements, and spark might be a better fit. That's where I ended up.

</comment></comment></comment></comment></comment></comment><comment><USER12> This looks interesting. Thanks for sharing and will have my DS team try it out.</comment>


Julian Assange: Haskeller - <AUTHOR> http://www.haskell.org/pipermail/haskell/2001-January/006410.html

<comment><AUTHOR> The title of this submission overstates his involvement somewhat. It looks like he posted to the Haskell mailing list a handful of times in early 2001, and his first apparent post does not prime any assumption he was a devoted enthusiast:

      This is why all non S-exp like lanaguage are doomed to progressive
      syntactic cancer as the useful parts of operator name space and syntax
      space become progressively polluted and mutated by one fad after
      another.

In any case, it's pretty cool to see a major public figure of significant worldwide importance making an insightful point about syntax space pollution.

<http://www.google.ca/search?q=assange+site%3Ahaskell.org></comment>


Fixing Google Map Transit Feed Mistakes in Taiwan - <AUTHOR> https://taipeiurbanism.com/fixing-google-map-transit-feed-gtfs-mistakes-in-taiwan/ <comment><USER3> Somethings up with transit data feeds in general...

Creating a transit data feed typically requires local knowledge, technical knowledge of data feed formats, and access to make changes to the data feed.

Actual transit providers typically lack the technical knowledge, especially for making day-to-day service updates. Transit enthusiasts lack the access to make changes to the feed. Google and mapping companies lack the local knowledge.

End result: Probably half the worlds population's local bus/train isn't accurate/present on Google/Apple/OSM maps.

How can this be solved? Perhaps some kind of feed-wiki so the three interested parties can do human collaboration?

<comment><USER4> The Transit app crowd-sources this from users who are using the "Go" feature in select markets.

There's also the Tripshot model -- an enterprise product where drivers install an app on a cell phone to track ridership, and it also sends the gps coordinates along.

That said, the other arcane part of transit directions is that if you update or break your feed, you have to wait hours or days before the result shows up on Google. (Sigh, batch processing...)

<comment><USER5> A major issue with relying on drivers is that if the real-time position of a driver is known, you can see both what they're doing and what they're _not_ doing. Which is bad if you want to get away with, say, taking longer breaks than you're supposed to.

Anecdotally, at a local system I once used there was an issue where drivers were intentionally not turning on the locating tracking, or configuring it incorrectly, because of this.

</comment></comment></comment><comment><USER6> This will come out as cynical, I know, but I have to say that when I see people so excited about freely contributing their time and efforts to the product of a global behemoth, something feels off. I know we all have a better free service as a result, but still this is basically free work. If anything, Google has been the most successful company ever in convincing people to work for free (and hand over their data). No judgement intended, just an observation.

<comment><USER7> GTFS feeds are open. The 'G' stands for general. This person is excited about freely fixing this issue for _everyone_, they've just chosen to frame their blogpost around the most popular surface for that feed.

huge disclaimer: I work at google, and on transit at that

<comment><USER8> well, the "G" originally stood for Google.

<comment><USER9> Somebody has to write the standards. Transit agencies don't care about making a data format that works for other transit agencies, as it doesn't affect them. Only companies that need data from multiple transit agencies will do this kind of work.

There are two approaches to aggregating data. You can either treat each agency as a unique snowflake, or you can convince all agencies to use the same format. The former approach means that all work you do is only for you, and you've built up an interesting base of intellectual property in an industry that doesn't matter. (There is no money in providing transit directions.) The latter means that anyone can show up and do neat things with transit data. (I have my own little webpage, jrock.us/mta.html that uses this data to provide me with exactly the interface I want. Do you think the MTA wanted to hire programmers to make this for people like me? Absolutely not. But they did make it for Google, and now I benefit.)

</comment></comment></comment></comment><comment><USER10> While it's true that there will be a lot of visitors using Google Maps, I still think that it's a shame that we've ended up reliant on a private company to direct us around. In the ideal longer term, it'd be better to use this effort to improve crowd-sourced or collaborative mapping solutions.

<comment><USER11> I'm only aware of one open-source public transport lookup app, which is Öffi [https://oeffi.schildbach.de/index.html](https://oeffi.schildbach.de/index.html)

It doesn't have data for Taiwan, unfortunately. Are there any that do?

Aside: Audrey Tang is also on HN, though not active much lately: [https://news.ycombinator.com/user?id=audreyt](https://news.ycombinator.com/user?id=audreyt) It sometimes strikes me as surreal that powerful people actually hang out on the same websites as people like you and me.

<comment><USER12> There's also Transportr, they use the same library for accessing transit data though. Transportr has some really good guides here, how it works under the hood and how to help with contributing other networks:

[https://transportr.app/contribute/](https://transportr.app/contribute/)

</comment></comment></comment><comment><USER13> > _It's unclear how long the feed was incorrect, but the first time I noticed > the error was in 2018._

My first visit after the new Taoyuan line was completed was in July 2017, and I'm pretty sure it was incorrect then, too. A local told me about the new line and said it was so much better, but when I tried to use GMaps to plot the trip, it showed as much longer than the older, more complicated options (and yes, I did end up taking a taxi).

On my more recent trips I've ignored GMaps and taken the Taoyuan line to/from the airport, and it's so much nicer. If you're going somewhere near Taipei Main Station, the total trip time is roughly the same as that of a taxi; even if you're not, the cost savings is nice.

</comment><comment><USER14> Love it, someone who cares and is willing to drive the issue all the way home.

I'm currently visiting San Francisco (SF) and experienced an opposite issue where GMaps reported cheaper and it ended up being more expensive. The BART train is a relatively fast, very cost effective way to get from SFO to downtown SF, Powell station in particular.

I was debating Uber at $30-$40, but looked at GMaps for public and it told me $2.50 to make that trip! Sure it takes 40 minutes with a bit more sketch, but I'm visiting, I'm willing and it's cheap.

Loaded up my Clipper card (BART's transit payment card) and went - <USER2> surprised when exited the turn style that my charge was $9.80.

The to/from the airport is significantly more expensive, which is understandable, but $9.80 vs $30 is much different than $2.50 vs $30.

Mildly frustrating, but in the end, not the end of the world.

<comment><USER15> Looks like this is a known issue due to a misconfiguration of BART's transit feed.

(Source: I work at Google and located the relevant bug, but I don't work on maps.)

</comment></comment><comment><USER16> When I was in Taiwan this past summer we took Uber to the airport from Taipei. I'm pretty sure our decision was based on in part on this misinformation. Well, now I'll know for our next visit!

<comment><USER17> My preferred method is a bus that gives you a great view of the mountains on your way into Taipei. Plus they're like five bucks or something and take an hour just like all the other ways.

<comment> _-___________-_ Except that the train actually takes 36 minutes, which is most of the point of TFA.

<comment><USER17> Add on getting to and then waiting for the train to leave... The bus depot is right outside the terminal exit.

I mean yes trains good, just saying.

</comment></comment></comment></comment><comment><USER18> Exactly too late for me.

I've been there in October and the data was not updated yet. I took the bus (which was shown as being faster!) and got stuck in a traffic for 2h. I arrived at my hotel exhausted and frustrated.

<comment> _-___________-_ I've got a strategy that's worked well almost everywhere in the world: never take road-based transport if there's a rail option, even if the rail option supposedly takes a bit longer.

Buses are less reliable, prone to traffic, less comfortable (especially if crowded)...

</comment></comment><comment><USER19> What I don't understand is that Google sees exactly how long a trip with the train takes and could have automatically fixed the estimate to the correct one...

<comment><USER15> Google Maps entirely relies on the data given by transit agencies through their GTFS feeds, whether they be static or real-time. It doesn't do the same type of calculation that the driving navigation does where they average travel times based on other users' movement.

There are some other apps like Transit App that attempt to crowdsource the same data to reflect ETAs.

</comment></comment><comment> Misteur-Z I only use CityMapper now in Paris

1) I don't want to feed the Google behemoth

2) It's very very accurate, even tells if you should get in the back/center/front to be closer to next commute/exit

3) They joke about strikes to brighten a bit those shitty days

[https://citymapper.com](https://citymapper.com)</comment>


WannaCry Ransomware decrypts Taiwanese netizen's computer due to his low income - <AUTHOR> http://www.taiwannews.com.tw/en/news/3161826 <comment><USER2> > "if you indeed liked something about ThunderCrypt and would like to donate" > ~ in their reply

What? The owner of the ransomware is asking for donations?

This is hard to parse.</comment>


Kinesis Rant - <AUTHOR> https://voltdb.com/blog/kinesis-rant <comment><USER2> 100% agree with the post. AWS and Amazon in general prefers to release bare and unpolished code that just works. To integrate that in your env is awful.</comment>