Android Tries Harder - <AUTHOR> http://pogue.blogs.nytimes.com/2010/04/19/android-tries-harder/

<comment><USER2> Most of the hard work of promoting the Android platform to developers has already been done by Apple.

Originally, I wanted to spend my tax return on a Mac so I could partake in iPhone development. Given Apple's recent actions, I'm just going to get an Android phone, and use my current laptop loaded with Ubuntu to develop for it.

<comment><USER3> I could not agree more. Sent from my Nexus One that literally just arrived. My semi-new iPhone is on Craigslist, and I'm already working on an Android app.

That's quite some free advertising.

<comment><USER4> I just wish there was a "main" android phone, or one that was a clear leader. So far, it's felt like "the" android phone to have is whatever is next in the pipeline.

I understand that the Nexus One is probably the nicest piece of hardware out there right now; but it's hard to decide to jump to it when it constantly sounds like there's going to be something even better just around the corner.

I guess my complaint is that I feel like the Android ecosystem is still too volatile to want to spend money on a phone.

<comment><USER5> I was frustrated by the same aspect of the Android ecosystem, until I had this minor revelation: it's not really about "the" phone, it's about the phone that works best for you. The nice thing about Android is that you can pick the carrier, form factor, and price point that work for you.

Want a keyboard and like Verizon? Get a Droid. Like Verizon but don't need a keyboard? Get an HTC Incredible later this month. Like the Nexus One but want the freedom of a SIM card? Use it on T-Mobile or AT&T. Want a cheaper phone with a keyboard? Get one of the new Motorola models that are coming out for just about every carrier.

You can think of it as confusing, or you can think of it as having options. Both are probably true :)

<comment><USER6> Right. Hardly anybody asks themselves "which is the Official Best Windows Computer?"

Android is not a single, tightly-controlled product line. It's an OS. Buy whatever hardware you think you'd like to run it on.

It's just a different approach than Apple's, and one that has worked well in PCs.

<comment><USER7> Has it really worked well in PCs? The average PC manufacturer is hardly rolling in money, unlike Apple. Nor is the average PC user likely to be too happy with their computer. Meanwhile in the Apple ecosystem both the normal user and the supplier seem to be rather happy. There is something to be said for tight integration of hardware and software in providing a good user experience.

(That said, personally I hate my MacBook Pro from work with the fire of a thousand suns, and have only used an iPhone for playing music on long plane trips since getting a Nexus 1.)

<comment><USER8> > _Has it really worked well in PCs? The average PC manufacturer is hardly > rolling in money, unlike Apple._

Assuming we are segmenting the Mac market from the PC market, it is much better to be in the PC market whether you are consumer or a producer. For manufacturers, it sucks to be in Apple's market if you are not Apple (PsyStar?). For consumers, they get a wider selection of products at lower price points, though they do not get the design cachet.

To make a belabored car analogy, would you rather be in the market for Italian supercars or that of the rest of the world? Both producers and consumers tend to choose the latter.

<comment><USER9> Your argument doesn't make sense here.

Most consumers would _choose_ to be at the luxury end of almost every product. It's just not a choice for almost everyone.

As to producers, it isn't inherently better to be a luxury market or a general market -- it's highly dependent on your product, market, and competitors.

 _For consumers, they get a wider selection of products at lower price points, though they do not get the design cachet._

That's a kind way of saying that consumers have a very diverse pile of crap to choose from.

<comment><USER10> Some people are just rich and can buy whatever they want.

But other people buy fancy houses, fancy cars, fancy computers, fancy holidays, fancy clothes etc. by virtue of choosing what is important to them and spending less on everything else. You can't spend more than average on everything unless you have more than average money. On average, people don't.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> > "And even now (after we peaked at the No. 7 paid app), we still have no > relationship with anyone there."

This, for me, is a real problem with the Apple model (the "hide the engineers" model, also followed by Oracle), we can not simply write an e-mail about a bug (Apple do have a closed bug tracker) or suggest a feature or even get motives about why some features do not exist in the iPhone (such as OpenVPN support), compare this with the extensive network of blogs that Google and Microsoft engineers have.

This is sad, you're always dealing with a cold company and not with people just like you that probably have some good motives for including or not a feature in a product, I wish to be alive in the day Apple becomes more open.

</comment><comment><USER12> This is very smart on Google's part. What Android seems to lack is polished apps (at least on the scale of the iPhone), and wooing specific developers could go a long way to remedying that.

<comment><USER13> Exactly.

Apple managed to get quantity, and with that the quality that you can often expect from the top 5% given enough quantity.

Google's much more surgical approach makes sense in that they can't realistically compete in quantity of apps any time soon, and thus have to specifically seek out quality to compete with Apple.

<comment><USER14> Reminds me of the "first-to-market advantage" that entrepreneurs talk about all the time. Apple took a huge, huge chunk of market and has had it since the iPhone came out. I think we'll see over the next year or two how well that works for them, and if developer sentiment is an early indicator, there could be some big changes in market share.

<comment><USER15>
      and if developer sentiment is an early indicator

Be careful: what you see on HN can be a very bad indicator of the real situation. All that was kind of blown out of proportion. There was kind of poll on HN asking iPhone developers how they feel about 3.3.1 and most of those who _do_ develop for iPhone said more or less along the lines "I don't care much". The biggest outcry was probably from those who never did nor were going to develop for iPhone. At least they found some justification for that now.

It's easy to get scared by rejection stores, but: App Store has close to 200 000 apps now. What are the real chances to get your app rejected?

<comment><USER12> I've been kind of curious how much 3.3.1 outrage was from people who were already outraged. I really don't know how to get a good measure of that, though.

<comment><USER16> Agreed.

</comment></comment></comment></comment></comment></comment><comment><USER17> Apple's AppStore reminds me of Google AdSense policy of isolation and almost inexistent customer service. If they ban you: They ban you. There's no possibility to discuss things.

<comment><USER18> Uh, I beg to differ, and hopefully help some people out, Google's Adwords customer service number is 1.866.2.GOOGLE.

I've had 2 great experiences with them, both involving multiple calls over a week's time solving Adwords account problems. (I'm only managing ~$15,000 ad spend per month, so it wasn't because I'm a big fish.) I do believe that's US support only, but I could be wrong. It's been the better part of a year since I've had to call in.

<comment><USER19> AdSense is much different than AdWords. If you're rejected by AdSense there's no recourse.

Anecdotally, my past experience with support for both AdWords and AdSense has been terrible. I'm glad they've got an 800 number now.

</comment></comment></comment><comment><USER20> Smart move by Google but at the same time I feel like they have some bigger issues to address. We know Android devices are selling pretty well. Why aren't good apps appearing organically? I'd look at 5 major problems:

1) Descriptions and other meta data for apps in the Android Market is atrocious. Lots of apps don't even have an informative description of what they do. Most apps don't have screenshots. This is the developer's responsibility however it hurts the platform as a whole when the user gets frustrated.

2) The top charts are heavily biased towards geeky applications. I have no doubt they register the most downloads but they may discourage people from exploring the Market deeper. A "top grossing" list would be valuable here.

3) The Market displays apps in other currencies. This is confusing. Easily fixable.

4) Some apps in the Market simply aren't compatible with some devices or are otherwise very buggy. This also may discourage people from digging deeper

5) Google Checkout is a total flop. Few use it outside of the Android market. Not sure what the solution is here but it impedes impulse buying for lots of people.

</comment><comment><USER21> Classic 80/20. 80% of the best apps in the App Store are probably created by 20% of the developers.

Therefore Google is smartly attempting to come in and skim the cream from the top.

</comment><comment><USER22> Competition is good, but there's something in that article that made me think:

>"Contrast with Apple's approach: it took us about three months of resubmitting our app to Apple before they stopped rejecting it for inappropriate content. And even now (after we peaked at the No. 7 paid app)"

It's the "after we peaked at the No. 7 paid app", part. While I think it's pretty amazing that Apple managed to create it's own microindustrial economy within the App store, it seems that a leading motivation for writing Apps is just to be in the App store and get ranked.

Now I know full well this isn't the _only_ reason people write apps for the iPhone, and maybe it's because of the walled garden effect that once you write an App you're dying to get in and hope people use it. But personally, I like the Android ecosystem where writing and distributing software is as native as it is on any computer.

And based on that conjecture alone, I don't think it's an issue of 'quality control'. Taking one giant step back, when you have a format that's easily distributed across whatever medium you choose, and don't put quality apps beside apps of less quality in a closed-off 'store', people don't even notice. They go for the apps they want to use and they can search whatever marketplace has it, to purchase and install it.

Competition in the App Store is good. Consumer choice in the market place is even better. That's why I don't own an iPhone and never will purchase an iPad.

<comment> brown9-2 I think that comment was just intended to highlight that even the developer of a pretty successful iPhone application (by the metric of popularity) hasn't received any outreach from Apple.

</comment></comment><comment><USER23> Developers will develop where they think they will make the most money. Currently, that is Apple's App Store. A $500 phone isn't going to push someone to spending the time and resources to release to Android. At least not someone with very much business sense. I'm not trying to be a hero; I'm trying to make a dollar.

<comment><USER19> I know an excellent developer who just took a significant paycut to work on something he's passionate about.

I largely agree with you that its still the bottom line that matters, but mindshare is often under-appreciated.

</comment></comment><comment><USER24> The reason they try harder is because there is far less money to be made on Android. I have the 6th top paid app in the store, Newsroom,(app, not game) and the revenue is nowhere near enough to live on. If I had the 6th top paid app in the iPhone catalog, I'd be writing this on my solid gold Macbook Pro.

There's a reason they are giving away handsets like mad (we've gotten two droids free in the past month).. Money is not enough of a draw yet.

</comment><comment><USER25> Gotta love the competitive spirit... Now if only their other services would offer such "human" support, as opposed to the maddening, hair-pulling, teeth- wrenching -- yes -- god-questioning expereience it is trying to contact Checkout or AdWords support. (Don't get me wrong -- these are great products when working proper -- it's just when they aren't working, and I'm giving them $500/mo., it seems reasonable that I would be able to talk candidly with someone.)

<comment><USER26> Adwords has excellent human support. Of course, you have to reach the level where they take enough interest in you. I've found that to be about the $2000/month level.

If you get to the $100K+/yr level then you get a rep. with a direct line that you can appeal to.

Then once you have your adwords rep on board, they are good at helping you get support from other groups... Checkout, Google products, etc.

</comment></comment><comment><USER27> As soon as Google stops caring so much about Android marketshare -- and they _will_ get bored, or will lose against Apple, or start dominating -- as soon as that happens, developers' helpful "personal contacts" will disappear.

Ask any Adwords advertiser or Google Checkout implementor.

</comment><comment><USER28> So if your app does well in the iPhone app store, you may get an email from Google evangelists. OTH if your app does well in any other app store, you don't get emails from any evangelists.

This seems to speak to the strength of the iPhone app store and this strength also explains why most developers seem to prefer the iPhone app store :)

Note: I wrote a prototype Android app (with their beta SDK) long before the first Android phone was in the market (and before the iPhone app store was launched). I have successful apps in the iPhone app store, no apps in the Android app store/market.

<comment><USER29> _OTH if your app does well in any other app store, you don't get emails from any evangelists._

A few months ago Google gave out free Droids and N1s to developers whose apps had over 5000 downloads and at least a 3.5 rating. (Of course at the time my app had 4200).

</comment></comment><comment><USER30> Anyone want to try guessing a date when the Android Market passes the Apple App Store in raw application count? (I know ultimately quality is more important, but it could be a symbolically important day... and might prompt a lighter touch towards developers from Apple.)

My wild guess: November 15.

<comment><USER31> More important guess: Will Android pass Apple in terms of market share? They went from nothing to 1/3 of the way there in 1 quarter, pretty much.

Depends on how quickly Apple can make it to other carriers. If I were the carriers, I'd have a few backroom meetings to NOT offer the iPhone (lest they become dumb pipes!).

If I were Android, I'd drop the rev-share for apps to 15% from 30% so developers make more money with less sales. And I'd make sure the carriers got a bit of the rev share.

If I were Apple, I'd take that pile of cash and build out a cellular network ASAP. ;-)

<comment><USER32> The advantage of Android is that it has an army of hardware manufacturers that can churn phones of different price points. Android can be released in much more countries quickly than the IPhone since these HW companies will be the ones making deals with carriers in different countries.

This is why I think it's a matter of time for Android to overtake the IPhone in terms of unit sales. If the 30+ Android models are actually released later this year, Android will likely surpass the IPhone early next year.

The profits crown will remain with Apple though for a very long time.

<comment><USER33> Android isn't so much competing on price point (the cheapest android phone is only slightly cheaper than the cheapest iPhone), but on feature sets. Want a large phone with a large high res screen, there's a phone for that. Want to sacrifice a bit of screen size and resolution for longer battery life, there's a phone for that. Want a phone with a keyboard, no problem. Want a slightly smaller phone with a keyboard, you can get that too. And so on and so forth.

<comment><USER34> But that's exactly the problem for developers. How many different devices are you willing to test on? Will your app's interface easily scale to all these different form factors?

A huge advantage of the iphone is that there's a single screen resolution to optimize for.

<comment><USER33> If the rumours are to be believed then that advantage will disappear this summer. The iPhone already comes with two different CPU speeds and soon it will have two different resolutions, so any advantage on that front is rapidly disappearing.

<comment><USER16> Daring Fireball points out that iPhones with double the resolution will be out soon.

</comment></comment></comment></comment></comment></comment></comment><comment><USER35> Now android has to try harder with it's marketplace software and infrastructure, where it will really make a large difference in getting developers over.

</comment><comment><USER36> I still get the message "Sorry, the Nexus One phone is not available in your country or region" on google.com/phone. I can buy a Nexus1 for 500€ on eBay, compared to 139€ for an iPhone 3G with a data contract. I'm still holding out for a decent Android option though.

<comment><USER37> that's kind of comparing apples to oranges. an unlocked iphone is about $1000, but an unlocked nexus one is $529. a t-mobile locked nexus one is only $179, and an at&t locked iphone is $199.

<comment><USER36> Sure, but right now my only two options are a bad apple and an expensive orange.

<comment><USER38> Where are you? Plans without a phone to subsidize are typically cheaper.

</comment></comment></comment></comment><comment><USER16> To develop for android does one use a large subset of Java? ...There must be many more developers with a medium-to-heavy Java background than there are Objective-C developers. Any thoughts?

<comment><USER39> Actually, with Android Scripting (<http://code.google.com/p/android- scripting/>) you can use other languages that run on jvm, such as ruby, python, javascript and others. Charles Nutter even supports a ruby version on Android directly: [http://blog.headius.com/2010/01/busy-week-jruby-with- android...](http://blog.headius.com/2010/01/busy-week-jruby-with-android- maven-rake.html).

If you like static typed languages, you can even use Scala on Android (<http://www.scala-lang.org/node/160>).

Many possibilities beyond plain old java.

<comment><USER40> It should be noted though that Android Scripting isn't usable for producing a standalone .apk. ie, you can't use it to build an app to put in the market.

Scala on the other hand, is supposedly production ready. I haven't tried it yet myself, but it's on my to-do list. :-)

</comment></comment></comment><comment><USER16> An interesting chapter in what is likely to be a long and interesting story. As a 17+ year Microsoft developer I have always been impressed with Microsoft's developer support & tools relative to those of Apple and, earlier (for me) Sun's Java. I am often SO VERY underwhelmed with the support offered to non-Micorsoft developers. Google seems to be using this (the Microsoft) approach and I imagine they will have a lot of success with it!

<comment><USER41> The person driving the Google's developer relations is Vic Gundotra who was responsible for Microsoft developer relations and development tools for close to 10+ years(I think from 96-06). He is using all the strategies that worked for Microsoft plus new ideas which would never have been possible at Microsoft due its closed nature.

</comment></comment><comment><USER42> Still not hard enough. Google needs to focus on what they do best: Search.</comment>


Freezing out Fukushima's radioactive water - <AUTHOR> http://www.aljazeera.com/indepth/features/2014/10/freezing-out-fukushima-radioactive-water-20141079111376195.html

<comment><USER2> It's an interesting approach, freezing the ground has been used for quite a long time in large projects(Léon Chagnaud did in the early 1900's to build tunnels through the water logged banks of the Seine (if you want an interesting story about large scale civil engineering that one has it all, the stations where built on the surface then "sunk" to position and effectively where designed to be water tight while the tunnels where built by sinking caissons in and then connecting them, incredible work))

</comment><comment><USER3> So what would the consequences be of letting the core just melt and eat its way downwards? It's not as if the environment around the plant and seawaters hadn't had a hefty portion of radioactive waste already and the structural damage hadn't happened. A quick look at Wikipedia tells me that in most nuclear accidents the worst had already happened before the actual meltdown.

<comment><USER4> [http://en.wikipedia.org/wiki/Nuclear_meltdown#China_Syndrome](http://en.wikipedia.org/wiki/Nuclear_meltdown#China_Syndrome)

not that it really answers your question. I don't know what the real consequences would be.

</comment></comment><comment><USER5> I thought they were dumping Radio active contaminated water every day into the pacific ocean.

And yet no other country actually care about it.</comment>


We Owe Food Regulation to a 19th-Century Chemist Who Poisoned His Colleagues - <AUTHOR> https://www.eater.com/2020/1/28/21112258/pbs-the-poison-squad-documentary-food-regulation-history-deborah-blum-interview <comment><USER2> Enter the lobbyists... Because of the recent creation of the political class in modern democracies, where people are able to now become career politicians rather than common people who make the sacrifice of leaving their jobs temporarily to serve the people.

It has empowered corporations with the ability to corrupt politicians who are more concerned with getting re-elected rather than protecting the people they represent.

Just look at how difficult it is to get a lable put on food containing GMO products and you will see how bad things still are.

Yes there is no longer chalk and formaldehyde in milk, or milk from cows eating spent mash, but what about steroids, antibiotics, peticide laden grains and gmo crops?

There is still a very long way to go, and until holding political office becomes a noble sacrifice rather than something to aspire to, change will be very difficult if not impossible.

<comment><USER3> I don't think labeling GMOs is going to help anything. There's strong evidence that GMOs are safe: [https://onlinelibrary.wiley.com/doi/full/10.1111/pbi.12798](https://onlinelibrary.wiley.com/doi/full/10.1111/pbi.12798)

I think there's a much stronger case to be made for getting rid of high fructose corn syrup subsidies. Added sugar is the real enemy.

<comment><USER2> Sure GMOs might be safe? Golden Rice? Yes that one is probably good for humanity, increased protein levels means you can feed more people. But lets get real, the majority of GMOs being used in agriculture are produced by Monsanto or Bayer, and what is the improvement that modifying their genes has made? Oh thats right now you can spray pesticides on them and they won't die!!

Sorry but Roundup ready GMOs might be safe if grown without drenching them in poison but if the only reason they are "GMO'd" is to allow this, I would prefer products made with GMO to be labeled.

<comment><USER3> There's a study about that too: [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5394230/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5394230/)

"Although GE crops have been previously implicated in increasing herbicide use, herbicide increases were more rapid in non-GE crops."

<comment><USER2> Isn't it sad that herbicide use was increased in both, regardless of the rapidity?</comment></comment></comment></comment></comment>


Ask HN: Tiling Window Manager for Windows - <AUTHOR>

Does anyone know of or use a decent tiling window manager in windows? I generally use awesome with linux, but I've had more and more things that require windows lately.
Unfortunately I find myself far less productive without the workflow and layout I'm used to, and find that organizing my work on multiple big monitors is tedious and gets in the way without something to handle it for me.
I've used Mosaico from SoulidStudio, but it's quite buggy, and even when it actually works, the automatic window management is pretty much useless so it's still a matter of manual management. <comment><USER2> Anything that makes Windows act differently than it wants to is shaky, in my experience. Mostly I try not to want that.

My compromise:

\- cygwin, plus the cygwin X server, plus urxvt (for the 256 color support, makes a big difference for old eyes with syntax coloring), plus tmux. I generally do as much as possible with a urxvt full screen running tmux (which is now a first class citizen of cygwin, yay).

\- For things that must be windowed, like Excel etc, try Winsplit revolution. It's not FOSS, but you're already on Windows.

Unfortunately Winsplit can be had from all sorts of downloader sites, and it's hard to know which ones are safe. This article has a link to the author's site, but it goes to the wayback machine (which still seems to have a download). I don't remember where I got it. [http://alternativeto.net/software/winsplit- revolution/](http://alternativeto.net/software/winsplit-revolution/)

That link also has some alternatives to winsplit, maybe something there will like you.</comment>


Strong passwords fine, but not necessarily necessary - <AUTHOR> http://blogs.msdn.com/b/tzink/archive/2010/09/13/strong-passwords-fine-but-not-necessarily-necessary.aspx

<comment><USER2> One solution to have both different and strong passwords for all your sites is to utilize Password Gorilla (<http://github.com/zdia/gorilla/wiki>). That way you get the benefits of strong and different passwords, without the mental effort of remembering all of those strong, different passwords.</comment>


That Time I Left $60MM on the Table - <AUTHOR> http://johnnystartup.com/2016/02/14/that-time-i-left-60-million-on-the-table/ <comment><USER2> that was a remarkable lack of foresight</comment>


Ask HN: Where can I find test data? - <AUTHOR>

Hello Friends,
I'm doing some work comparing various data storage engines.  I'd love to get a large dataset or two to use for the comparison.  What do folks here use for large amounts of test data?  The domain doesn't matter so much.
I've previously looked at the US Census data and that qualifies for the "large" part, but it's a little unstructured.  It'd be cool to get some CSV files that are easily loaded into a variety of tools. <comment><USER2> <http://aws.amazon.com/publicdatasets/>

<comment><AUTHOR> Yes, totally forgot about this. Thanks!</comment></comment>


Future of the *Social Internet*? - <AUTHOR>

Social media seems to be a hot topic but with a lot of tension from two aspects:
(1) There is a lot of talk about developing and delivering more in social media.  The more might be from things new in each of data sources, data manipulation techniques, Web sites, or companies.  There might be more in just the content or in social search to find such content.
(2) There is not much clarity about just how to have more in social media or just why to have it.  For the "why", what users want it, and what would they do with it?
Here are two examples of some of the recent talk about social media:
First, here on HN is the thread:
"Sergey Brin:  We've Touched 1 Percent Of What Social Search Can Be (techcrunch.com)"
at
http://techcrunch.com/2011/01/20/sergey-brin-weve-touched-1-percent-of-what-social-search-can-be/
Second, is the thread "Building Better Social Graphs" at Fred Wilsons blog A VC* at:
http://www.avc.com/a_vc/2011/01/building-better-social-graphs.html#disqus_thread
with a lot of relatively good relevant comments.
In Fred Wilson's thread "Building Better Social Graphs", there were two strikingly different themes:
First, Wilson started the thread with a post where he wanted to be able to download each of his social graphs and then curate them himself.
Second, in the comments, the theme was strong that given the data on the social graphs, we should have computer-based means to process the data for curation, etc.  Curiously, the goal of this processing was Wilson's "Building Better Social Graphs" by stronger means than just Wilson's manual curation!  That is, Wilson's title was stronger than Wilson's post, and the comments were closer to the title than the post was!
So, from 40,000 feet up, it appears that many people have some vague, ill-defined, intuitive, poorly identified and articulated visions of making progress with social graphs.  Each of (A) the broad subject of social, (B) Facebook, (C) Twitter, and (D) the Internet is so big that we should take the visions, as crude as they are, seriously.
Three issues:
(1) Meaning.
An arc in a graph from the definition in applied math has essentially no meaning in any sense social or even practical.  So, if we are to make use of data from social graphs, etc., then we should make some progress, if only rough, on what the arcs, or other data, mean.
(2) Purpose.
We should identify the purpose of the software.  That is, what will be the output of the software, and why will users like that output?  Or, what do users want, or at least would like if they saw it, that such software might provide?  What the heck is the darned purpose?
(3) Techniques.
Given the data, what data manipulation techniques will we have the software use to get the results good for the purposes?
I raise one more point:
The US has something over 300 million people.  In some important respects, this number is not very large.  E.g., it is easy enough for current computing and data base techniques to have, say, 1 million bytes on each person and still be able to store and process that data.
So, it can appear that there is a chance that we could have a single, grand solution in the space of social graphs and social search.  If so, then we will guess that the present efforts in social media are only tangential or indirect solutions for a central problem not yet identified, articulated, or solved and that a single, grand solution might be possible.
Net, we have potentially a grand answer to the issue what data.
Then we can move on to what purposes?  What will people what to do with this data?
What more is there to be said?
Where can we be more clear on the data, purposes, processing, and future of social whatever via the Internet? <comment><USER2> I say start with the purpose and work backwards. As the good Dr. Lecter said, "First principles, Clarice. Simplicity. Read Marcus Aurelius. Of each particular thing ask: what is it in itself? What is its nature?"

People really only use data for a few reasons - to be or take action towards pleasure, to reduce risk, save time or make money. Combine any of the following for extra points.

Why then do we love data that is socially generated or a derivative thereof? It turns out to be really good at leading to those outcomes as it communicates experiences others have actually had.

Take Amazon's 5 star rating system. A very nice graphical device that helps me lower the risk I will buy a bad product. You know what is even better? The graph showing the counts of ratings in each star. It tells so much more! I make better decisions. I don't buy anything without first checking if Amazon has a rating.

If you are going to show me a graph - it better lead to one of those outcomes and rather quickly. Once you nail down the purpose then you can decide everything else.

<comment><AUTHOR> Okay, in general, good.

Now, for _the social Internet_ , what are the _purposes_? E.g., why do 500+ million people use Facebook? Why do so many people use Twitter?

Okay, once we have thought about those old cases, assuming the huge amounts of data on _social graphs_ in principle available, to what _purpose_ might that data be put?

E.g., Fred Wilson wants to get copies of his _social graphs_ and then manipulate, _curate_ , them himself. Okay, but for what _purposes_?

Then as in the posts in his thread, how to process the _social graph_ data automatically for Fred's or other _purposes_ , and, again, what _purposes_?

E.g., maybe the _purposes_ are _introductions_ of one person to another or, say, one person to a group. Again, for what _purposes_? "May I have the envelope please?" Here is a list -- hobbies, careers, looking for a supplier, customer, employer, employee, date, spouse, cello in a string quartet.

Then if the purposes are mostly just introductions, how are the problem and solution much different than the _introductions_ via the present romantic matchmaking sites?

Uh, Brin indicated that 99% of what can be done has yet to be done, or some such. So, what are the purposes of the other 99%? So, is Brin's 99% just an expansion of the idea of romantic matchmaking sites?

In simplest terms, what are the important candidate purposes for which people would want the results of software that manipulates data on social graphs?

I'm asking because the question seems big, _hot_ , and important, and I don't have any very good answers. Here I may not be alone: Facebook, Twitter, etc. seem important, but finding clear and seemingly accurate statements of the important _purposes_ they serve is not so easy.

But there is potentially a LOT of data so that if we could be clear on purposes, then maybe we could achieve some of them.

Let's do some guessing:

 _Social search_ might necessarily be quite close to _personal interest search_. Then the main _purpose_ of _social_ whatever on the Internet would be some personal _interest_ so that doing a _personal interest search_ would yield _arcs_ in the user's _social graph_ , where each such arc leads to a _node_ good for that user's _personal interest_ and, thus, leads to Wilson's "Building Better Social Graphs".

In particular, we can have bars, art galleries, concerts, adult education classes, country clubs, yacht clubs, high school and college alumni meetings, and, quite particular to the Internet, topical blogs.

So, each of these social activities can further a person's personal interests and build arcs in the person's corresponding social graph.

Then, perhaps, finding such a topical blog, the user could post some comments. Eventually maybe someone would post "For all blog users in and near Seattle, this Friday let's meet for pizza and beer at Joe's place".

So, the purpose would be to meet some people who share some interests and a geographical area.

The Internet has been used for such things. So, the _purpose_ is to meet people with shared interests and geography. So, the purpose seems to be one that could be accomplished better by a slight generalization of Internet romantic matchmaking services.

If so, then how come Facebook and Twitter are big things far from the matchmaking services that are much smaller things?

<comment><USER2> The social internet garners its masses by feeding, often simultaneously, into the most basic human needs. Our needs for social information exchange are fulfilled, we can find and cultivate our own community. Humans are hardwired to be connected. Socially we evolved to be as such because at its base it is life sustaining. We survive because somebody bit the dust eating a mushroom and somebody else learned from it and then taught us.

The other side of the success of Facebook is that it serves to entertain us. Americans watch on average 34 hours of TV a week. Most of it is nearly worthless and not a value add other than to be amused.

But, people in the course of fulfilling these tasks on FB are leaving behind information trails. We know know what they are talking about, who they are talking to, who they just friended. Now we can compile this information and do something with it - back to the purpose.

Let's take a look at a site like hunch.com - a very good use of collecting data about an individual by answering questions (a form of amusement to many) written by other members. They then leverage it to solve some problems that Amazon's rating system fails at - letting me know what user reviews are more like me. Am I more similar to people who give it 1 star reviews or 5 star reviews. I really enjoy the site and I willingly give it information about me so it can help me more.

This data is usually used as you stated to find similarities . In people, for matchmaking sites, here are things purchased by people who...read the same stuff, watch the same movies, etc. It can turn out well, but for what was known by the Netflix challenge teams as the Napoleon Dynamite problem. A movie that based on similarity clusters was almost impossible to predict how a user would rate it.

Now as you ask what are the candidate purposes for which people want the results of software that leverages social graphs.

Companies are much easier than people because unlike our list of uses and goals earlier, companies just want to make money. Facebook for example.. How do they make money? By selling advertising (other sources comprise only about 10%). How do they make more money? By entertaining more people, for a longer time, and charging more for their ads. If you can serve a targeted ad you can charge more, etc. If you know what feed information to show a user, they will stay on longer, you then serve them more ads. And as such, their purpose for using the social data and derivatives thereof will be for those purposes. Similar to amazon, the purpose of the data are to sell you more goods by recommending things you might buy based on predictions made.

So why does Facebook lack dating as part of their core offering? Because while they could be good at it they up to now have just been a platform company. This has changed a bit with the launch of FB messaging and @facebook email. But look deep into the reason why? Gmail makes Google money because they serve you contextual ads based on your content. Why does Facebook hope you use their mail? Ah yes, you get it now...so you spend more time on their site and they have more data about you so they can serve you more ads. With one small feature change you might say the showed a glimpse of who they are becoming.

Before they were a true social media company – a platform where users generated content to share. Now they want you to spend more time on the site, use it as a common login for other sites, send and receive email through it? Why..again the money. So they can do what they do best. Serve you better ads and have a longer time to do it. Every move you see FB make you will see it is now to broaden the platform and increase your dependence on it so you spend more time on it.

Facebook is becoming the new money printing machine by taking exactly the same path Google originally did – by getting people to the information they want faster. How did Google do it? Google page rank – they figured out that rather than word density – the social element of a link to the content was more important. Google is scared now because the content that is being generated inside of Facebook's closed walls is massive and it is a powerful predictor of what is relevant.

In both Facebook and Google's case they make money by helping us perform the transactions we desire by using social data. Social search on the other hand, at least how I see it going, is leveraging all this data to wade through the murky garbage everybody is producing with social media and actually help me find what is relevant. Relevancy is key because what is relevant to me I look at, I take action with, I buy. This makes other people money usually (if it does not then they usually fail to build or go under).

Again the purpose is user specific and the social data helps you determine relevancy. A matchmaking site would use it to find you a date you liked faster, a consumer retail site would use it to help you find the garlic press you are going to like faster.

Finally - back to the graph. What are your thoughts on using them? Personally I would find them useful for discovering trends in myself, my research etc. They help us explore, sort through data. But they are the end product and for most companies just a form of entertaining you with all the data they have about you.

<comment><AUTHOR> Maybe we're making some progress:

So, we started with social graphs, social media, and social search (below I refer to these three as 'social X') and asked the questions about purposes, the utility of social X, and the means to the purposes.

For entertainment, there is a huge industry from movies and TV. One lesson is, high quality content with broad appeal is expensive to produce. In particular I don't see Internet social X doing much better at generating good entertainment content.

I will insert another point: The more such good entertainment is produced, the more expensive it is to produce even better entertainment. So, we should be moving to about enough high quality digital recordings of such entertainment to satisfy anyone for life and, thus, have less pressing reasons to produce more.

You mentioned:

"This data is usually used as you stated to find similarities . In people, for matchmaking sites, here are things purchased by people who...read the same stuff, watch the same movies, etc. It can turn out well, but for what was known by the Netflix challenge teams as the Napoleon Dynamite problem. A movie that based on similarity clusters was almost impossible to predict how a user would rate it."

I can believe that! Apparently the idea was that persons A and B are 'similar' and person A likes movie X so that then maybe person B will also like movie X. That is not promising because the 'connections' are too crude: E.g., two sisters can be very close yet be very different in the movies, music, clothes, boys, etc. they like.

It appears that the Netflix people fell into a trap: Apparently commonly in the literature of 'collaborative filtering', 'recommendation engines', and 'data mining' has been the suggestion that we should start by putting people into 'clusters' and for each cluster observe what movies, music, etc. were popular in that cluster. Then, given another person, say, a user, find what cluster they were in and then recommend to them what was popular in that cluster. Not a good idea. Interesting to hear that the idea failed!

So, we might conclude that some of the data for such clustering, recommending, etc. can be represented as social graphs, but that step appears to be just a tangential afterthought and not a good example of the utility of social graphs.

Yes, people like to communicate and form communities: Significant examples from the past include back fences, kitchen tables for coffee, lunches, dates, parties, various meetings (e.g., PTA, alumni, school board, political parties, church), school, and work. Such conversations and communities have been significant for romance, building a business, getting a job, education, skills, arts, crafts, politics, and more. But I'm failing to see that the communities that result from Internet social X are very significant.

Net, I don't see current Internet social X doing very well on entertainment, conversations, or communities.

I see the Internet as terrific for a lot, e.g., news, short video clips, technical information, the content on Wikipedia, general information on organizations, information from topical blogs and long tail Web sites, shopping, and more. But so far I don't see social media as leading to very significant utility.

Then in particular I don't see much value in the data on social graphs.

Still, Facebook has 500+ million users and is worth maybe over $50 billion; Twitter, Foursquare, etc. are significant; and Page believes that 99% of social search has yet to be done.

So, here we can look at the weakness in social X, and that might mean that there is an opportunity to do better and have some astounding success. Or, a guess is that current social media efforts are not really addressing the fundamental purposes and opportunities very directly or well so that a better effort could be quite successful.</comment></comment></comment></comment>


Google Pixelbook with Linux - <AUTHOR> Google Pixelbook is nice, can I install other Linux and run Chrome OS together, means two Systems? I wanna ask some Google Staffs here. <comment> cat-turner I am not google staff member but I am a dev who uses a chromebook pixel at home. This Nov will be 2 years since I first received my chromebook pixel. Here is my set up: 1) I unlocked machine, put it in into dev mode 2) crouton set up to use xfce 3) ssh to remote servers to use things like docker

Here's the repo for crouton. I don't know if they have hardware support for pixelbook yet. [https://github.com/dnschneid/crouton](https://github.com/dnschneid/crouton)

I had some hardware/software issues from time to time but they were resolved with OS updates or a powerwash. My set up was ok but not ideal - it would fluctuate between bliss and frustration. Opening your laptop to find half of your led screen dark is not fun. Sometimes the laptop would simply poweroff - though this may have been an issue with regards to overheating. This is the price to pay to go into dev mode.

Other than the points above I think using a chromebook pixel for development has been a ok experience. For simple tasks my experience is very good. My battery life is still very good after 2 years. The screen is perfect. Switching between xfce and my other tabs is a breeze, I would have my xfce instance open in one tab, and have a few chroot instances open in another tab (think of the feel like python's venv).

If google staff is here, please answer my only question: is there any plans to make this set up more friendly for dev set ups?</comment>


FBI: Businesses Lost $215M to Email Scams - <AUTHOR> http://krebsonsecurity.com/2015/01/fbi-businesses-lost-215m-to-email-scams/

<comment><USER2> Title is slightly misleading in that that's only the REPORTED losses. I'd venture to guess that the actual numbers are significantly higher. A lot of these go unreported due to embarassment, etc.</comment>


Android 9 Pie:Powered by AI for a smarter, simpler experience that adapts to you - <AUTHOR> https://blog.google/products/android/introducing-android-9-pie/ <comment><USER2> Automatic-automated features (as opposed to user-set automated featutes) bother me, because of the small, nuanced ways that the arbiter can push the user into different patterns they may not have been in on their own.

This was, however, the natural progression of Google Now.</comment>


Critical 'Backdoor Attack' Warning Issued for 60M WordPress Users - <AUTHOR> https://www.forbes.com/sites/daveywinder/2019/08/31/critical-backdoor-attack-warning-issued-for-60-million-wordpress-users/#48a2dd78732e <comment> ga-vu forbes and its clickbait titles.... jfc</comment>


Book Idea: Great Teams At Work - <AUTHOR>

<comment><AUTHOR> Interviews with teams that created: Flash, iPhone, Google Talk (heard that they won the Google Founders award), Akamai

(ok maybe these are trade secrets that companies won't release -- but it'll be really cool...)

Their strenghts, how they work, what "process" they follow etc

what do you guys think?

</comment><comment><USER2> I think it'd be interesting to hear how teams of founders work. What kinds of team dynamics they have - things like:

democratic vs. dictatorship 1 technical & 1 business ...etc

Woz and Jobs Hewlett and Packard Allen and Gates Sergey and Larry ...</comment>


AMD Unleashes First-Ever 5 GHz Processor - <AUTHOR> http://www.amd.com/us/press-releases/Pages/amd-unleashes-2013jun11.aspx <comment><USER4> This is pretty important for AMD because they've been having a terrible time matching Intel at per cycle efficiency. Basically, ever since the core i series started, AMD has been behind at single threaded performance, especially when going clock for clock. They've been trying to make up for it by both offering more cores/threads than comparably priced Intel components, as well as trying to scale their clock rates. Unfortunately for AMD, their last few generations have actually been falling short on their attempts to boost clock rates, as a result, when comparing comparably priced AMD and Intel chips, while AMD would typically have a clock speed advantage, it would often not be enough to overcome Intel's efficiency.

This product release is kind of an attempt to show that AMD can actually deliver on their planned strategy.

As for why AMD is going for this route, rather than trying to beat Intel in per clock efficiency? Probably because AMD's resources are severely limited compared to Intel, and this approach offered lower risk at lower cost.

<comment><USER5> > As for why AMD is going for this route, rather than trying to beat Intel in > per clock efficiency?

Well, because clock speeds are something they can improve _now_ , and not in $n years time when their next major microarchitecture is ready. Intel had the exact same problem with the Pentium 4, and they were similarly stuck with minor tweaks and desperately increasing clock rates for years before Core was ready.

<comment><USER6> I wish I had a cite for this.

It was my understanding that P-IV was originally conceived as an architecture that could be clocked up and up for years to come. Developing a new architecture is expensive and risky (see P-IV, Itanium), so the hope was to design something that would scale up so well as manufacturing improved that a few generations of architectures could be skipped, so to speak.

They had hoped the P-IV would eventually reach 10GHz or so. Which made it OK that P-IV retried fewer instructions per clock than the P-III that came before. Scaling up like that isn't such a radical idea; the P6/i686 architecture behind the Pentium Pro, Pentium II and Pentium III had spanned a spectrum from 150MHz to 1.4GHz, after all, nearly an order of magnitude.

But it turned out that somewhere between 3 and 4 GHz, things got really difficult.

"Minor tweaks and desperately increasing clock rates" was more or less the P-IV plan from the get go. It just turned out not to work.

<comment><USER7> Anandtech discussed this in their Bulldozer review:

        AMD's architects called this pursuit a low gate count
        per pipeline stage design. By reducing the number of
        gates per pipeline stage, you reduce the time spent in
        each stage and can increase the overall frequency of
        the processor. If this sounds familiar, it's because
        Intel used similar logic in the creation of the Pentium 4.

[http://www.anandtech.com/show/4955/the-bulldozer-review- amd-...](http://www.anandtech.com/show/4955/the-bulldozer-review-amd- fx8150-tested/3)

And since I read this, I really wondered what trick AMD has up its sleeves.

<comment><USER8> there is no trick, just desperate attempts to survive.

10 years ago AMD had more efficient architecture, Intel - more GHz (remember 2.2 Athlons having 3700 "PR-rating"?). Intel's approach was commercially more successful - customers were still buying GHz, and AMD was trying to educate market about real performance while radiating impression of a looser who just can't get good process and fabs. AMD gave up and decided to pursue P4-like approach for their new architecture, while Intel hit GHz "sound barrier" and went efficiency way by resurrecting PIII style architecture which resulted in Core CPUs. AMD made a huge, strategic mistake 10 years ago. How the execs in charge of Bulldozer have been BS-ing their way inside AMD last 5 years - <USER2> is a typical everyday miracle of a big company internal life.

<comment><USER9> There is also the billions intel spent making sure dell and hp and co. would never buy AMD

<comment><USER10> I actually take notice of how many consumer laptops I see in a walmart / target / best buy are actually running AMD APUs. The big brands might not, but they see bigger numbers on the A6 than a Pentium and feel better about the buy even if the Pentium dominates it.

<comment><USER11> The trouble is that most computers aren't bought at Best Buy or Walmart, and the ones that are tend to be the low end garbage with no margins for the hardware vendor.

Keeping AMD out of Dell and HP is what kept them out of corporate America. Corporations literally buy PCs by the pallet, and then they pass on the volume discount to employees who want to buy one for home.

<comment><USER12> > they pass on the volume discount to employees who want to buy one for home

Really? I don't see the use case for ordering a PC this way for the home. When I'm buying a home PC, or recommending one for others, it's either:

(a) A bottom-of-the-barrel PC. As long as it has 1GB of memory and more than one core, you can use it for web browsing, Youtube, email, and word processing. This is what non-techies usually want (but they don't know they want it and may get upsold by good marketing). This is what _I_ want unless I'm planning on running a specific application that requires more.

(b) A powerful PC for gaming. It needs a decent discrete GPU if it's going to play current games. Most office PC's don't have one, unless you work for Pixar.

AFAIK the machines purchased by corporations for general office use are usually middle-of-the-road beasts that cost more than category (a) but don't have the discrete GPU of category (b). I'd be guessing they'd be a waste of money for home use, even after the volume discount.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> Based on how bulldozer performed, it'll end up that sure it's 5ghz, but did we mention all our instructions take two times the number of clocks now?

Virtualiztion host performance on our 8 core bulldozer (esx 5.1, private kbs from vmware to try to help, 32gb ram, rad10 zfs san) was so bad (think p4 era) that we finally tracked down how to force the cpu into only using 4 cores, one per real fp core.

The reality is that there is no mainstream scheduler out that that can efficienty use cores set up like that, especially with the long pipelines. I'm not sure it can't be done, but what improvements have been made have been minimal, or just in an academic/not a real os situation.

That's why Intel ships a compiler, duh.

It is true that the # one thing holding that part back was the raw clock speed (as long as you view it more like a 4 core, 8 thread part ala Intel), but i've gone back to speccing intel - it's just not worth being that much of a ginae pig for a firm thats basically trying to scrape by until the arm64 parts start getting stamped.

<comment><USER14> _we finally tracked down how to force the cpu into only using 4 cores_

Was the issue the shared fp units, or the turbo-core? I wonder if you can disable the turbo-core?

<comment><USER15> If it was anything like the Niagra processors the shared fp units normally are a bottleneck for fp. But the larger problem was the register remapping/pipelines. They were fast, if you were running certain workloads. God help you if you had to compress anything on those systems. Without pbzip2 or pigz it took forever. Really bad example but bulldozer seemed way too niagraish to me based on its goals.

<comment><USER11> Running threaded floating point workloads on bulldozer-derived architectures is just folly. If you have parallel floating point code you should in general be running it on a GPU.

<comment><USER13> These weren't fp intensive workloads at all - mostly your typical IT IO workloads. I don't know the internals to say exactly what or why, but something seems to go really wrong on bulldozer when you try to schedule two different vms on the same coupled pair of cores.

<comment><USER11> It's because they're not independent cores. You're pretty much never going to get the same single-thread performance with two threads running on a module as with one, the idea is that you ought to get better than 0.5X the single thread performance, such that if you have two threads then 2*0.75X is better than X, while still allowing you to get X (or better with turbo) on strictly single threaded workloads.

Where this can fall apart is if you're trying to use eight homogenous threads at once and the threads have large working set sizes, such that the second thread causes spill out from the per-module caches. Then you have eight threads contending for L3 bandwidth, or if you're really screwed you fill up the L3 and start to hit main memory.

Out of curiosity, have you tried any of the Abu Dhabi Opterons? They doubled the L3 from 8MB to 2x8MB, which I would expect to help by both keeping you out of main memory and reducing contention by splitting each L3 between half as many cores (assuming you don't get the new twice-as-many-cores models).

</comment></comment></comment></comment></comment></comment><comment><USER16> Some comments from Anandtech:

[http://www.anandtech.com/show/7066/amd-announces- fx9590-and-...](http://www.anandtech.com/show/7066/amd-announces-fx9590-and- fx9370-return-of-the-ghz-race)

Based on an old review:

[http://www.anandtech.com/show/6396/the-vishera-review-amd- fx...](http://www.anandtech.com/show/6396/the-vishera-review-amd- fx8350-fx8320-fx6300-and-fx4300-tested/)

and single thread performance:

[http://www.anandtech.com/show/6396/the-vishera-review-amd- fx...](http://www.anandtech.com/show/6396/the-vishera-review-amd- fx8350-fx8320-fx6300-and-fx4300-tested/4)

If single threaded scales linearly with turbo frequency (and it looks like it might):

The FX8320 (turbo boost 4.0Ghz) scores 240.7, while the FX8350 (turbo boost 4.2Ghz) scores 252.1:

The difference aligns quite nicely: (240.7/4) _4.2~252.74

And for 5Ghz should give about: (240.7/4)_5~300.88

This is still lower than intel's i5 3570k (302.2 - Turbo 3.8Ghz) and i7 3770k (312.4 - Turbo 3.9Ghz)

And Haswell has even higher performance:

[http://www.anandtech.com/show/7003/the-haswell-review- intel-...](http://www.anandtech.com/show/7003/the-haswell-review-intel- core-i74770k-i54560k-tested)

<comment><USER11> You're comparing the performance of the respective top of the line models. That only matters for bragging rights. Most people don't buy that one, which leaves AMD open to sell chips to people who would have bought midrange Intel chips -- or people who are willing to sacrifice 312.4/300.88 -> ~3.8% performance (which is almost certainly within the margin of error) in order to keep competition alive or because AMD offers a lower price.

<comment><USER16> Well, I just wanted to guess at what that 5Ghz announcement actually would mean in terms of performance.

I'm not convinced ~11 pixels per second is within the margin of error (the numbers were from the single threaded povray test) -- but 3.8% difference certainly mean very little in the real world. I'd guess it falls within the bracket that is measurable but ignorable ;-)

Also, the "5Ghz chip" will most certainly be AMDs top of the line model?

</comment></comment></comment><comment><USER17> IBM shipped a 5.2 GHz chip for it's mainframes over 2 years ago. This is not the first.

<comment><USER18> Exactly, this is the first ever 5GHz x86 chip. The title is factually inaccurate.

<comment><USER19> That's not surprising, given the title comes from and links to a _press release_.

</comment></comment></comment><comment><USER20> Does it really qualify as a 5 GHz processor if it only runs at that speed in Turbo mode? (which I assume can only kick in for a few milliseconds...) What is the "normal" speed that it can maintain for more reasonable time periods? How come this isn't mentioned in the press release?

<comment><USER21> Turbo mode is limited by heat so with cooling you can stay in turbo mode.

PS: Intel and Nvidea do the same thing. [http://www.techarp.com/showarticle.aspx?artno=745&pgno=1](http://www.techarp.com/showarticle.aspx?artno=745&pgno=1)

<comment><USER22> If older chips from Intel and AMD could be overclocked to twice their baseline rates with high-end cooling, I wonder what you could clock this beast up to?

And yes there is a market for this. There are certain workloads that are simply not parallelizable -- they're linear chains of dependencies where the output of the first process goes into the second and so on and each step depends on all N-1 steps.

<comment><USER23> Here's a nice example of a challenge that is very hard to parallelize ([http://people.csail.mit.edu/rivest/lcs35-puzzle- description....](http://people.csail.mit.edu/rivest/lcs35-puzzle- description.txt)) ([http://crypto.stackexchange.com/questions/5831/what-is- the-p...](http://crypto.stackexchange.com/questions/5831/what-is-the-progress- on-the-mit-lcs35-time-capsule-crypto-puzzle))

([https://people.csail.mit.edu/rivest/pubs/RSW96.pdf](https://people.csail.mit.edu/rivest/pubs/RSW96.pdf))

</comment></comment></comment></comment><comment><USER24> Wasn't IBM's POWER in this range already?

<comment><USER25> Yes, according to Wikipedia, IBM was shipping 5.0GHz POWER6 processors by 2008: [https://en.wikipedia.org/wiki/POWER6](https://en.wikipedia.org/wiki/POWER6)

The article cited in the Wikipedia entry is dated 08-Apr-2008: [http://www.theregister.co.uk/2008/04/08/ibm_595_water/](http://www.theregister.co.uk/2008/04/08/ibm_595_water/)

<comment><USER26> Don't let the numbers fool you, though. POWER6 achieved those speeds due to a change in chip architecture that actually wound up making them worse processors than the much lower clocked POWER5s in a lot of situations. IBM reversed course and changed the chip architecture back for the POWER7s, which are clocked lower and outperform the POWER6s.

<comment><USER27> I'm not an expert here but it looks like IBM is back up to shipping 5.5Ghz chips in the EC12 as of December 2012. Are these still POWER6 chips as an option or fixed and faster POWER7?

PDF Redbook reference: [http://www.redbooks.ibm.com/redbooks/pdfs/sg248050.pdf](http://www.redbooks.ibm.com/redbooks/pdfs/sg248050.pdf)

<comment><USER26> As sibling mentions, they're not POWER. The max Ghz for each rev is POWER5 2.3Ghz, POWER6 5Ghz, POWER7 4Ghz.

<comment><USER28> apaprocki, you've been absolutely spot on.

Highest clocking POWER processor offered is the 4.42 Ghz P7+ System p 780: [http://www-03.ibm.com/systems/power/hardware/780/specs.html](http://www-03.ibm.com/systems/power/hardware/780/specs.html)

And yea, POWER 6's design was a bad decision.

</comment></comment></comment></comment></comment></comment><comment><USER29> Q: What will a more powerful CPU core do in a comparatively slow environment that is the rest of the computer?

A: Wait faster!

<comment><USER30> A lot of that comparative slowness was caused by mechanical storage, replacing it with parallel and lower-latency SSDs makes it a lot easier to get full use out of more and faster cores. It doesn't cost much to set up a system that's completely CPU-limited on OLAP database-like workloads these days.

I suspect that as more software stops being optimized for ~10ms serial disk I/O with huge caches this will become more common and more and faster cores will be a big(er) deal.

</comment></comment><comment><USER31> Can you hear it - the shrieks of all those D-14 and Phantex-s screaming.

I would like to see review though. And pricing. If it has decent single thread performance and that number of cores with all next gen games being multhithreaded by default it could be a compelling processor if it is in the 4770 price range.

<comment><USER10> Not really, because the AMD 8 physical core layout is still 4 packages of 2 register sets sharing ALU and FPU. It is exactly like a hyperthreaded Intel part, but Intel has much better per clock performance.

That, and I see next gen games more favoring openCL / GL 4.3 compute shaders to offload all their parallel workloads than to aggressively optimize for greater than 4 core processors. Your returns on moving traditionally CPU bound workloads (per agent logic, path finding, collision detection) to compute class GPUs (when available, with the cpu fallback for now) gives you significantly more returns than optimizing for the CPU.

Also, you can take a 4770k to near 5ghz on air. This part is already pushing the thermal limits of the Bulldozer architecture, AMD is just fabbing them out this high speed because they are floundering in this low-per clock performance rut the entire architecture put them in.

Now, I would point any budget oriented gamer to the 4 or 6 core AMD models around $120 - 130, because since they are all unlocked, you can get real performance gains (but terrible power efficiency) over Intel parts below the 4 core unlocked part they put out each generation. Since they are effectively 2 / 3 module parts, they are well suited for the next gen of GPGPU everything in the engine and let the CPU do control flow.

If you even approach $200, the performance gains from jumping from any non-K part to a 4.8ghz 4570k are huge, and that alone outclasses every AMD cpu for gaming, but does trade blows on some titles with the 8 core parts.

<comment><USER16> It'll be interesting to see if AMD cornering both the new xbox and ps4 has an effect on game engines for the pc as well -- specifically if the tuning that probably will go into console versions will translate to pc -- and whether or not Intel (and nvidia) will end up being penalized as a result.

</comment></comment></comment><comment><USER32> The problem is that clock doesn't really mean anything concrete in terms of real world performance. It's strictly a marketing thing.

For an example, what if a chip used a 10 GHz clock for distribution, and divided it down to 5 GHz everywhere it was actually used (not that I know of any reason to do such a thing besides marketing). Would it be marketable as a 10 GHz chip? The manufacturer would certainly be in hot water if enthusiasts ever found out...

Even without such contrived scenarios, CPUs get different amounts of stuff done per clock.

Something I keep seeing, even on Slashdot and Hacker News, is the idea that a CPU that has to clock higher for a given performance will use more power. It seems to me that if you've got double the clock, the likely explanation is that half the transistors are switching per clock, and power consumption should be orthogonal to clock/IPC ratio.

If anyone's got any contrary ideas on that, I'd love to hear them. All I can think of is that higher clocks would correlate with longer pipelines, but bulldozer's pipeline isn't even that long.

<comment><USER33> "is the idea that a CPU that has to clock higher for a given performance will use more power."

This is like a dog whistle to the EEs, they're going to get all riled up by programmers with screwdrivers. You can model a stereotypical FET gate as a capacitor, all you're really doing is charging and discharging capacitors either in FET gates or the transmission line theoretical capacitance. Right out of the C=Q/V definition of what capacitance is, mushed up against some ohms law and some algebra, and you end up with P=C times V squared times F. So you can see the intense excitement in lowering core voltages, making gates and lines smaller (lowering C) all in a tradeoff to improve the P/F or F/P (whatever) ratio.

The important part is its pretty easy, right outta ohms law and the def of what capacitance is, power is directly proportional to frequency.

<comment><USER34> _The important part is its pretty easy, right outta ohms law and the def of what capacitance is, power is directly proportional to frequency._

There's also the fact that your transistors have a particular voltage that they switch state at, which means that they switch faster if you drive the gate/line capacitance with a higher voltage.

Which means that chips _designed_ for lower frequencies can be designed to use lower voltages, which can save far more power than what would be directly proportional to the lower frequency.

<comment><USER33> "which can save far more power"

yes, right out of the equation provided.

In "CS" terms that may be better understood on HN than "EE" terms, electrical power scales O(n squared) with voltage and O(n) with frequency.

If you really wanna get people riled up and talking you can roll out the old power "EE" stuff about maximum power transfer happening when source and sink impedance are the same, and you want to get the most bang for your buck so you'd like that, right, and a transistor gate being near infinite resistance would imply ... Or if you like to think about interconnects being signal to noise level limited, then an RF analysis about noise voltage across a resistor vs preamp noise figure vs current bias from a communications standpoint would imply... But it turns out in practice most of the time, the first mental model is by far the most effective way to look at it compared to these.

</comment></comment></comment></comment><comment><USER35> I remember like ~10 years ago on Slashdot some people overclocking to 7-8ghz. Of course this was on single core chips, but we've really pretty much completely stalled on the mhz progression haven't we?

<comment><USER36> Also, I think there are some physical limitations that keep chips below a certain clock speed. Besides, the bet has been on "smarter instead of faster", i.e. producing chips that suit our computing needs, which are more adequately supported by parallel processing.

High performance cores are useful for problems that are hard to paralellize, but so far it seems that the breakthrough only occurs when a new approach to the problem makes it feasible on multiprocessing platforms (e.g. graph processing is hard to paralellize due to dependencies among graph nodes, Pregel and similar offer a different approach)...a 50GHz CPU won't save you if you need to process a huge graph (i.e. billions of nodes) on a single thread, it'll always take a lot of time.

As to the "record", I think IBM already had a Series Z that is over 5GHz.

<comment><USER14> _I think there are some physical limitations that keep chips below a certain clock speed._

Not _hard_ limits, but yes, to my knowledge it is primarily physics that keep chips where they are. The requirements for power and heat dissipation start to balloon.

</comment></comment></comment><comment><USER37> > "FX-9590: Eight "Piledriver" cores, 5 GHz Max Turbo "

Ridiculous name. Maybe if they put a 'Go faster stripe' on the top of the chip people will believe it goes even faster!

<comment><USER38> "FX", "PileDriver", "Max", "Turbo" ??? Whats next? obviously, "Super", "Mega", "Ultra", "Extra"?

</comment></comment><comment><USER39> In other news: PCs now even more versatile. Now also replace radiators.

<comment><USER31> It has been like that for a while ... in 2000-ish we were calling Athlon Kotlon (Kotlon is stove in bulgarian)

<comment><USER40> In the mid 2000s I was running a Prescott oven in my room.

<comment><USER14> Just wrap that computer case in tin foil, and you got a makeshift oven!

</comment></comment></comment></comment><comment><USER41> I wonder how two of these (16 cores) compare against the new mac pro with the best CPU in software that benefits from many cores like 3D rendering and such or virtualization. Id bet they are close while the AMD only costs a fraction of the Xeon. I know its not a fair comparision since the FX-9000 is not a workstation cpu, but still...

<comment><USER42> The multi-threaded Pov-Ray and Cinebench tests are just about the only two benchmarks where the AMD 8-cores beat the i7 2600k, and just barely beats it.

The Intel chips soundly win in anything else (encoding, Photoshop...), and by almost 2X in some of the single-threaded tests.

</comment></comment><comment><USER43> Yes AMD, we all know you like your big numbers like core counts and clock speed. It'd, however, be just excellent if you could put out a product whose single threaded performance isn't garbage. I mean, thubans are beating your newest and greatest!

But at least you can say you got the bigger cache, clockspeed, core count, and debt than intel.

</comment><comment><USER44> In the end it's not the frequencies nor number of cores but performance per watt that matters.

Most computers run on batteries these days, and those that don't drain ever more expensive electricity from the wall socket and at the same time waste a lot of it producing huge amounts of heat.

The more you get out of a watt the better. You can either trade in speed for lower power or trade in power for better performance, but in either case you want the performance/watt ratio to be the highest.

I would guess the power consumption of running the chip at 5GHz is pretty high. And running temperatures as well. And yet there are fewer and fewer of those huge tasks that you can only do with one core.

<comment><USER14> _In the end it 's not the frequencies nor number of cores but performance per watt that matters._

It depends on the workload, really. It should already be obvious that this part is not meant to be a Joe Everyman processor.

</comment></comment><comment><USER45> Since Pentium 133 I never had Intel processor in desktop computer. I wanted few times but AMD was always cheeper for the same speed. Sure fastest were almost always the Intel ones but the additional bit of speed never justified the price.

<comment><USER12> My hyperthreaded [1] Core i7 makes kernel compile jobs fly.

[1] [http://en.wikipedia.org/wiki/Hyperthreading](http://en.wikipedia.org/wiki/Hyperthreading)

<comment><USER45> It just splits core into sort of two virtual cores. I'm not sure how that could help. Have you check how disabling it influences compiling speed?

<comment><USER12> > I'm not sure how that could help

Hyperthreading keeps two threads "hot" in each physical core. When one thread is waiting on memory access, the core can do work on the other thread rather than sitting idle. (Memory access isn't _that_ slow, so switches need to be fast to capture those otherwise-wasted cycles, which is why this is a CPU hardware feature rather than an OS-level software feature.)

Purely CPU-bound tasks [1] don't get any performance gains from HT. But almost all real-world applications spend a lot of time reading and writing memory, and memory access is pretty slow compared to CPU speeds, so in practice HT helps (otherwise Intel wouldn't have bothered to develop it and put it on their chips, which probably cost a lot of money).

> Have you check how disabling it influences compiling speed?

No. But I'd guess it would be substantially less than 100% speedup since they aren't actual, physical cores; but substantially more than 0% speedup since the compiler uses dozens or even low hundreds of megabytes of memory.

[1] By "CPU-bound" I mean register-to-register arithmetic. You might also be able to get away with hitting the L1 cache, which is a few KB, without triggering an HT context switch.

</comment></comment></comment></comment><comment><USER46> How ironic that AMD are now suffering from the same thing that once gave them the edge (that is, Pentium 4's overly aggressive clock speed roadmap and lacklustre per-clock efficiency).

</comment><comment><USER47> and here i was hoping never to see a high-clock-speed headline again...(supercomputers excluded)

<comment><USER48> I guess their marketing was out of other ideas and just went back to the well one more time. It has probably been 10 years now since I really considered CPU clock speed as a factor when buying a computer.

</comment></comment><comment><USER49> I seem to recall reading that these processors would have a 220W TDP - <USER3> makes the whole 5ghz thing much much less impressive...

<comment><USER50> I'm yet to find an actual source for that figure whenever it comes up. Is it just some tech site comment section spitballing or did they actually disclose the TDP?

<comment><USER33> If there's a dimensioned pix of the heatsink, a bored enough engineer could calculate the theoretical degC/W rating of the heatsink, and given a presumably constant deltaT, there's your wattage.

Doesn't have to be dimensioned that accurately. To a first approximation a 1% error in surface area would be about a 1% error in TDP.

I'd like to see very high temp CPU technology. That would be an interesting, challenging direction for hardware tech to move. A tiny lightweight 5 deg C/W heatsink is plenty if you're allowed to run at, say, vacuum tube redhot glow temperatures. I'm well aware of the solid state physics challenges of this, that's why I think it would be very interesting to see if anyone could pull it off.

<comment><USER51> I doubt that a heatsink on an engineering-sample test board would be sized within a 1% margin. Seems more likely they'd be error generously on the side of big.

</comment></comment></comment></comment><comment><USER52> I've been waiting since my Athlon64 died for AMD to make a chip worth getting. Them taking the brute force (P4) approach is not all that encouraging.

</comment><comment><USER53> I think maybe AMD should move to niche markets and stop trying to compete with Intel, giving how vast the gap is in technology and resources.

<comment><USER40> If there's no one trying to compete with Intel, I would guess that gap would close fairly quickly.

</comment></comment><comment><USER54> I think there should be a "their" in there, somewhere.

</comment><comment><USER55> yaay, I am eagerly awaiting for intel to catch up :) (so I could buy quality stuff that does not hang or overheat)

</comment><comment><USER56> YAYYY!!! Okaynowwhydothebenchmarkssuck.

</comment><comment><USER57> I wonder how far they will OC on air.

</comment><comment><USER58> What's the MIPS for this CPU?

</comment><comment><USER59> finally, angry birds will be totally responsive!

</comment><comment><USER60> (for consumers)</comment>


Ask HN: What kind of index is Google using here? - <AUTHOR> http://google.com/codesearch?hl=en&lr=&q=%5Cd%5Cw%7B2%7D%5CW%7B3%7D%5C%28%5Cw%7B2%7D%5C.&sbtn=Search <comment><USER2> AFAIK, Google Code Search was written by Russ Cox, who also wrote this: <http://swtch.com/~rsc/regexp/regexp1.html>

My guess, back when I was doing code search at Apple, was that he was actually scanning all the docs with his fast regexp, which would be do-able with the number of machines at his disposal + the expected traffic.

If anyone knows any differently, I'm all ears. :)

<comment><USER3> If Russell Cox wrote it, it must be good. I programmed competitively with him at USACO and in college (we got to 8th best college team in the world), and he was a consistent ass-kicker. Hardcore and disciplined and fast.

<comment><USER4> Other (open source) projects by Russ people might find interesting:

* Plan 9 from User Space - <http://plan9.us>

* vx32/9vx - <http://pdos.csail.mit.edu/~baford/vm/>

* libtask - <http://swtch.com/libtask/>

(Plus quite a few bits of Plan 9 itself, obviously.)

Edit: forgot this really cool storage system: <http://doc.cat-v.org/plan_9/misc/foundation/>

</comment></comment></comment><comment><USER5> Reverse Aho-Corasick matching.

Basically, you store all of your searchable text in a suffix tree. You then do a depth-first search of the tree for matching strings.

<http://www.cs.brandeis.edu/~mfc/cs120/papers/ptrees.pdf>

<http://en.wikipedia.org/wiki/Aho-Corasick_algorithm>

<http://en.wikipedia.org/wiki/Suffix_tree>

<comment><AUTHOR> How does that help here? My query doesn't contain any directly-searchable strings, unless you start abstracting things by treating all alpha chars as the same, etc.

<comment><USER5> A nice thing about regular expressions is that you always know where you are in the expression (no backtracking, at least in theory).

Say your query is \d+(th|nd|rd), and your searchable text is "The 3rd planet from the sun is 20000000 miles away." Your tree search will hit 2, then fail on 20, then 3, then 3r, then 3rd, and then terminate successfully. So we followed 5 pointers, and tested another 20 or so for nullity.

This was independent of the length of the searchable text (but not the number of found matches). It would have performed equally well on "The 3rd planet from the sun is 20000000 miles away. Lorem ipsum (insert 10mb of random Latin here)".

<comment><USER6> How much RAM do you think it'd take to store all the world's source code in a suffix tree, and how would you partition it across multiple machines?

<comment><USER5> Well, there's always suffix arrays, so two 64-bit pointers, (plus the original char) per char?

Assuming their claim of 116,000,000 matches for /./, and 5k mean document size, 70TB, uncompressed.

Papers like "Compressed Suffix Arrays and Suffix Trees with Applications to Text Indexing and String Matching" [1] lead me to believe that the state of the art could be as low as 4 bits/char, in which case the answer would be a much more tractable 270GB.

As far as partitioning goes, now I'm getting really out of my league. At a first pass, I imagine that you could keep the root and neighbors on a primary server, and as big of subtrees as you can fit in RAM on a bunch of slaves. The first couple chars get processed in the master, and then however many subtrees are alive get delegated in parallel to the secondary servers. The secondary servers return the answers to the primary, which returns to the client.

I feel like I'm answering hypothetical questions at a job interview :p

[1] <http://www.di.unipi.it/~grossi/PAPERS/sicomp05.pdf>

<comment><USER7> The math doesn't work. 116,000,000 matches x 5k is 580 GB. It fits on one big disk, and with 5800 machines holding 100 Mb of plain text each you could search all of it naively in real time. (Grep searches 100 Mb in 50 mSec.)

5800 machines is practical for a real product. Hardware probably costs Google $40/month/CPU, so $230k / month, about equal to 10 engineers. So if a more clever solution needs a whole team to maintain it then it's probably better to just use grep.

<comment><USER5> 10 engineers to program one data structure? I'd be greatly surprised if 1 PhD and 2 engineers couldn't easily do the additional work beyond distributed grep in less than six months. Maybe one engineer to maintain it? That saves ~$200k/mo--more if you go over 20 queries/second it takes to max out the 5800cpu limit.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> Is your question "How do you create an index that allows regexes?"

In that case, I'd like to know the answer to the question, as well.

<comment><USER9> perl has a "study" function that examines a string in anticipation of doing many pattern matches on it. I'm not sure what it does exactly, but the kind of structures it creates could be serialized and stored along with an input file (or perhaps replace it), and used as kind of regular expression optimized index. This isn't necessarily an "index" in the normal sense, but then, neither is the likes of a full-text index either. But really, any data structures that are optimized for a specific kind of processing are indexes.

<comment><USER10> study does not do anything really interesting. It looks up substrings in a table of substring -> average frequency. It then scans for the rarest character first, so that it can fail quickly for most input.

From the docs:

(The way "study" works is this: a linked list of every character in the string to be searched is made, so we know, for example, where all the 'k' characters are. From each search string, the rarest character is selected, based on some static frequency tables constructed from some C programs and English text. Only those places that contain this "rarest" character are examined.)

Basically, this is not what Google is doing for code search.

<comment><USER9> Study doesn't do anything really interesting, but neither is creating a keyword index for a keyword based search all that interesting either.

Considering that Google code search is not intended to search English (for which their regular search technology would most likely suffice for that) using regular expressions, then I agree that the naive study implementation you've quoted is, basically, not what Google is using for code search. However, there's no reason that the frequency tables need to be static nor need to be constructed solely from from C programs and English text. Frequency tables can be continuously updated based on the input corpus and the input corpus should be the code that is being searched. Part of search is being able to rule out significant portions of the search space in the name of getting positive results as fast as possible, so being able to fail quickly for most inputs (considering the example regular expression only returns three hits, it seems a wide majority don't match), a study-like implementation would appear to be a possibility.

</comment></comment></comment></comment><comment><USER11> This is off-topic, but I just realized Google Code Search is great for seeing where code you open source ends up. Came across a few things I had no idea were using my code. Kinda cool.

<comment><USER12> Tom, did you know there are extra filters you can use to search Github.com?

Try "tlrobinson package:github.com lang:javascript"

</comment></comment><comment><USER13> IIRC, this was a rather high-compute-cost project inside of Google, far more so than other search projects. Obviously it was a favorite though of the eng decision makers, so was well cared for resource-wise.

</comment><comment><USER14> This presentation talks about using N-grams to make an regex searchable index

[http://www.informatik.hu- berlin.de/forschung/gebiete/wbi/tea...](http://www.informatik.hu- berlin.de/forschung/gebiete/wbi/teaching/archive/sose08/pr_text/vortraege/ngram_indexing.pdf)

<comment><USER5> No complexity analysis. Random student results.

It's viable to build an index with n-grams, but I don't think you'd get any benefit over suffix-tree search. There might be a way to select n-grams cleverly some of the time, but you can also easily run into pathological cases.

</comment></comment><comment><USER12> I'd (probably) consider building and searching an index of ASTs (<http://en.wikipedia.org/wiki/Abstract_syntax_tree>).

Not only would it allow searching with regexp, but it could allow more sophisticated search queries like "Search all my Ruby code for blocks that rescue StandardError and are not followed by an ensure clause"

<comment><USER15> But you'd have to do that separately for every language.

Edit: the OP is such a great question. I've been saving up the joke of commenting "How is this Hacker News?" on the _most literally appropriate_ HN post I run across... and almost used it on this one. Sadly, I probably never will.

<comment><USER12> > But you'd have to do that separately for every language.

That's correct. It's one of the challenging parts.

<comment><USER15> It's one of those ideas that would be best tried out on one instance first. It's not obvious that the ASTs of programs in different languages would be "A" enough for a generalized querying system to be useful. Still, I don't mean to shoot the idea down - it would make for an interesting experiment.

</comment></comment></comment></comment><comment><USER16> Something like this?

\- Regular expression searching on compressed text, <http://portal.acm.org/citation.cfm?id=967612>

<comment><USER17> The cost of that algorithm still grows linearly with respect to the length of the compressed text. It's possible Google is doing something like this (searching over the entire corpus until they find a match), but is there any work on precomputing an actual "index" to speed up this type of search?

<comment><USER16> I'm certain there are many approaches to be found via citation-following.

But consider that you can use the linked algorithm, and a bit of trickiness, to create an index already:

\- Step (a): prepare another version of the document where every digit is replaced with 0, every word char with 'a', etc.

\- Step (b): note how much better it compresses. :)

\- Step (c): Perform a similar 'weakening' on the search regex.

\- Step (d): Done! Not an exact index, but it will allow you to quickly eliminate a large number of documents.

You can also combine this approach with text-based ones for the cases where there are parts of the search regex that contain literal text. (This is almost certainly the most common use case).

Or you could make it hierarchical if you need to by creating several watered- down texts for each input with different levels of substitution.

\</comment></comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>---

Since this is the sort of thing that's fun to think about, here's a whole different approach:

\- Break each document apart into code lines.

\- Create a hierarchical tree: leaves are the code snippets (with a pointer back to source doc), each node represents a clustering-by-text-similarity of all its children. (Appropriate distance measure left as an exercise for the reader. :) )

\- Let's say that for two regular expressions, _RA_ & _RB_ , _RA_ < _RB_ iff any text matched by _RA_ will also be matched by _RB_.

\- At each node, create a regular expression _R_ so that for each child regex _C_ , _C_ < _R_. (The leaves become a regex that simply matches the source line literally.) Along the way, make a best effort to restrict the length of _R_ by introducing generalizations ( _R_ can't simply be _Ca_ | _Cb_ | _Cc_...) without making it so loose that it matches everything.

\- Then, searching for a regex _S_ can be done recursively: For each child regex _R_ of the current node, if you can construct a new regex _T_ such that _T_ < _S_ and _T_ < _R_ , descend into that node.

n.b. I typed that out in a hurry and without being rigorous... so it's probably not quite right, but an interesting path to think about exploring. Also: testing if one regex is less than another (as defined above) is actually fairly straightforward, even though it seems hard.

(And a quick edit to add: the last step is definitely wrong. Unfortunately I need to hop onto a conference call, so it'll have to wait until the indefinite future.)

(Edit to that: I think I fixed it for now. Shorter version: if the intersection of _S_ and _R_ is nonempty, descend.)

<comment><USER16> Replying to myself, since editing was getting out of control.

A neat property of the 2nd approach is that if you ever reach a node _R_ where _R_ < _S_ you're done: all children will be matched.

A less neat property is that it won't allow multi-line matches, but I'm not sure those are allowed anyway.

</comment></comment><comment><USER18> It might be possible to adapt a regex state machine to use an AVL TDAWG ( [http://www.ssp.isee.kyushu-u.ac.jp/~inenaga/papers/tcs- terna...](http://www.ssp.isee.kyushu-u.ac.jp/~inenaga/papers/tcs-ternary.pdf) )

</comment><comment><USER19> I am 100% postulating here, but this is what seems most likely to me:

Google is known for MapReduce. It seems like it would make regex searches manageable when put in such an algorithm. All Google would have to do would be to take whatever key system they use for their big-tables and use MapReduce to expand the regex over the keys. Then they just have to return the data associated with those keys.

<comment><USER20> ..what?

I would love to see you try to explain how you think this works :)

<comment><USER19> to quote <http://infolab.stanford.edu/~backrub/google.html>

Indexing Documents into Barrels -- After each document is parsed, it is encoded into a number of barrels. Every word is converted into a wordID by using an in-memory hash table -- the lexicon. New additions to the lexicon hash table are logged to a file. Once the words are converted into wordID's, their occurrences in the current document are translated into hit lists and are written into the forward barrels. The main difficulty with parallelization of the indexing phase is that the lexicon needs to be shared. Instead of sharing the lexicon, we took the approach of writing a log of all the extra words that were not in a base lexicon, which we fixed at 14 million words. That way multiple indexers can run in parallel and then the small log file of extra words can be processed by one final indexer.

So we have a hash table of words/searches -> results

Because we have this, we can regex search the keys, and end up with the result pages.

we do have other options though. It could be:

a) magic b) a trade secret (think PidgeonRank :-) )

<comment><USER20> Quote is unrelated to what you are saying!

 _use MapReduce to expand the regex over the keys_ _So we have a hash table of words/searches - > results Because we have this, we can regex search the keys, and end up with the result pages._ ...

Looking forward to further explanations!

<comment><USER21> Imagine the following code:

money = 5;

Could be tokenized and mapped to a few regex keywords, with one or another as key-value in a datastore. Since Google is big on MapReduce, which could be seen as computation on sets:

[a-z] = (DOCID_14566, DOCID_15999, DOCID_888)

\d{5} = (DOCID_15, DOCID_15999, DOCID_552)

Then when someone searchs for a "number with five digits", the value for the \d{5} key is retrived, and it has the documents where the search appears, and this set/list is combined with the other set/list that match other search keywords, and you have your results.

On a normal web search, the document is split into words and you can make these words into keys, so know where they appear, right? (the key's values).

Well, if you think that the user will type a regex expression "\d{5}", then you can map it to documents too.

It's like pre-processing the possible regex that would match a document token, and searching _these regexes_ instead of words like a normal web search.

It's "just" an extra step, splitting words (tokens) from a document, and assigning possible regexes for these tokens.

Not saying Google does it :p But I guess that's what the OP meant...

Edited: Before anyone notices how inefficient my example is, I wouldn't implement every possible \d as a key, or [a-b], [a-c], [a-d] etc as another key, this is just an example of how sets of precomputed regexs (from any depth) can be (ab)used.

<comment><USER20> This makes sense and it could work :) I imagine you would have to spend time tuning the tokens you map out. I think this would break down pretty quickly with moderate or complicated regexes, but those might not be the majority of searches.

Is there any trickery you could do with the inner nodes on the b+trees? I spent a few minutes thinking about tokens mapped out and intermediate reductions but didn't come up with anything.

</comment></comment></comment></comment></comment></comment><comment><USER22> Easynews also has a global search where you can search through all the newsgroup posts with a regular expression (> 150 days).

You might ask them? They are certainly not as secrective as google is ;)

<comment><USER10> Is the global search really a regex? I thought it was just substrings.

<comment><USER23> The labels are links to regularexpressions.info so I'd guess yes.

I'm pretty sure EN just greps subject + output of something like 'strings' though, it chugs pretty hard on broad searches.

</comment></comment></comment><comment><USER24> I've always wanted to know that.</comment>


Is Facebook down? - <AUTHOR> https://www.facebook.com

<comment><USER2> The input

<http://www.downforeveryoneorjustme.com/facebook.com>

says Facebook is down, but I was just on Facebook (and still have the tab open in my browswer), and Facebook was operating normally for a user in the United States Midwest. These test websites don't report results that are reproducible for me, and I recall on a previous occasion such a test website told me Hacker News was down, while I was using Hacker News. Anyhow, to the point of this thread, I'm not noticing any problems at all on Facebook.

</comment><comment><USER3> They are trying to roll out IPv6, but it's not going well.

</comment><comment><USER4> <http://www.downforeveryoneorjustme.com/>

<comment><USER5> This website says it's down but it does work for me (UK)

<comment><USER6> Mine says it's up and it works fine from the UK.

<comment><USER7> Works for me in the UK too.

</comment></comment></comment></comment><comment><AUTHOR> Doesn't load for me from Amsterdam. Maybe I can get some work done for a change?

</comment><comment><USER8> East Coast US here - it's really slow and timing out occasionally.

</comment><comment><USER9> It's unbelievably slow but still up here in Japan..</comment>