Show HN: Cloud-based GIF editor to augment any video - <AUTHOR> https://gifs.com/?source=https://www.youtube.com/watch?v=Q5JnHDPyEBk <comment><AUTHOR> Hey HN, I'm one of the back-end engineers at Gifs.com and we just launched a brand new editor that anyone can use to create and augment videos online.

We started with the premise of democratizing video editing by making an iMovie-lite in the cloud. Our goal is to empower people around the world to express themselves. We still have a lot of ideas in the works to make finding, creating and augmenting short-form media frictionless but launching our new editor is the back-bone of that dream.

You may remember us from gifyoutube.com where anyone could add gif in front of youtube and we would let you trim the video and add captions. Now you can simply replace youtube.com with gifs.com or put gifs.com/edit/ in front of any video or GIF to open it up in the new editor with even more editing capabilities.

Everyone knows a picture is worth a thousand words, so does that mean a GIF must be worth n pictures where n is frames in the GIF? After checking out /r/reactiongifs/ and /r/wastedgifs I now believe so.

<comment><USER2> The player looks super nice and was very easy to use. Hope y'all do well.

<comment><USER3> Thanks for the kind words Ian!

</comment></comment></comment><comment><USER4> Slick interface and arguably the top domain name possible. Nice!

The youtube integration and cropping is really cool as well. It took less than a minute to try it out. I can imagine this getting some heavy usage which begs the question, how do you support the costs behind this?

The video processing cpu time isn't free and the end result is an image hosting site which traditionally are impossible to monetize. What's the secret sauce for sustainability?

<comment><USER3> Koolba, thanks for the kind comment! For supporting the costs: we ended up raising $ from some awesome individuals. We've built out a team (we're hiring! firstname.lastname@example.com), and the team has done a lot of work to ensure that gifs/webms/mp4s are transcoded insanely fast. We've had over 4,5000,000 gifs made, and over billions of .gifs delivered, which has started to run up some cost...which leads us to sustainability: we look at the gifs player almost like the youtube embedded player, but for, well, short media. Maybe one day ever gif/webm will be in the gifs player. As we continue to build tools for empowering people to augment videos, we hope to see more people make cool short clips within our ecosystem.

<comment><USER4> > For supporting the costs: we ended up raising $ from some awesome > individuals.

Okay but that's capital, not revenue.

> We've built out a team (we're hiring! email@example.com), and the team has done > a lot of work to ensure that gifs/webms/mp4s are transcoded insanely fast.

A great dev team can arguably keep costs down with efficient systems but that's still not revenue. Also, dev teams cost money too so that's more expense.

> We've had over 4,5000,000 gifs made, and over billions of .gifs delivered, > which has started to run up some cost

Okay sounds like some traction but I read that as about $11.5K/month of bandwidth spending[1] (just bandwidth, not counting transcoding costs).

> ...which leads us to sustainability: we look at the gifs player almost like > the youtube embedded player, but for, well, short media. Maybe one day ever > gif/webm will be in the gifs player. As we continue to build tools for > empowering people to augment videos, we hope to see more people make cool > short clips within our ecosystem.

So where's the revenue? Embedded player licensing? Intermingling ads? Changing the overlay icons for Brazzers to link straight there?

I love VC fueled fun as much as the next guy, but I'm genuinely curious what the end game is. Ride it out and hope the user base becomes worthy of an acquisition before the money burns?

[1]: _1 Billion Images x 100 KB /Image x $.12/GB[2]_

[2]: _Based on the IP I 'm guessing they're running on GCE with rack rates for bandwidth at $.12/GB. Probably over estimating though._

<comment><AUTHOR> I'm not Rory, our CEO who you're replying to, but he couldn't respond due to a HN "posting too fast" error so I thought I would chime in.

What we are truly passionate about is exposing the general public to a new means of expression through short form media. What we are seeing is publishers writing articles that are just lists of looping GIFs, sports fans browsing /r/nba and watching the 3 dunks and a block instead of the 3 hour game, etc... so we built our embeddable GIFs player that allows viewers to share, turn on sound, view source, and much more. It also has the benefit of using mp4 or webm format over the actual gif file which is not a good choice when compared to those modern formats. This has led to a lot of publishers using our product and the tremendous growth of people viewing media in our embedded player. Now we want to expand another segment of our user-base, the creator (who we see as the next-generation of publishers) by making a easy-to-use video editor in the cloud.

To finally answer your question, how could we generate revenue, we could show highly targeted ads during prime-time slots. Imagine the NBA finals game, people making GIFS and sharing them with captions. Nike wants to promote some new product and they use our ad platform to expose a viral campaign / advertisement on all of our media. That's just one idea but there are many other potential avenues for monetization that we are considering as well.

tl;dr we're not aiming to be an image hosting company

</comment></comment></comment></comment><comment><USER5> Really swift interface - this is an incredible tool, nice work!

Q: How come when I visit gifs.com I see a brazzers logo in the bottom right corner...?

<comment><AUTHOR> So that gif logo in the middle of the homepage has a background of a recent and random GIF created. Looks like someone made one with the brazzers logo as a sticker.

Thanks for pointing this out though, the team had a good chuckle and it reinforced the need for a manual or better system to elect gifs that show up on the homepage.

</comment></comment><comment><USER6> I just leave this here as a form of sanitizing contrarianism to counter all the empty buzzword abounding in this thread.

[https://www.theguardian.com/technology/2015/dec/29/irans- blo...](https://www.theguardian.com/technology/2015/dec/29/irans-blogfather- facebook-instagram-and-twitter-are-killing-the-web)

Democratization /= fueling the televisization of the Web

<comment><AUTHOR> I can't understand the argument of social media making the world a worse place. Television is made by publishers who get to decide what to deliver to their audience which is in stark contrast to facebook, twitter, etc... where you are in control of who you follow or friend. Additionally we don't care where you share our media, you can feel free to use it in a blog post.

The only thing we're trying to fuel is the ability of the common person to express themselves through short video. If you think that is a bad then we'll have to agree to disagree.

</comment></comment><comment><USER7> Wow, that's a pretty nice domain name you got there.

</comment><comment><USER8> This is really cool! Is it possible to select a specific time range of a video via the API?

E.g. I only want the first 3 seconds of a video to be turned into a gif.

<comment><AUTHOR> Yes this is possible, we havn't updated the docs[1] yet but the editor uses our public API for all the effects.

        {
          "source": "https://www.youtube.com/watch?v=jhyANGHDDH8",
          "trim": {
            "start": 145,
            "end": 148
          }
        }

1\. [http://docs.gifs.com/v1.0/docs](http://docs.gifs.com/v1.0/docs)

<comment><USER8> Cool but please clamp the values to be within 0 and currentDuration.

I'm getting an error that the duration that youtube sends me back is higher than your read out duration:

{ errors: { message: 'end(8s) > currentDuration(7.383s)' } }

<comment><AUTHOR> Good idea of just trimming to the end if the end value is greater than the video duration, I'll add that next week.

As a quick hack since the youtube video duration is 7.383s, and you want to trim to the end just omit the end property.

</comment></comment></comment></comment><comment><USER9> The interface is very easy to learn.

Found one usability issue: when starting a crop and without clicking 'check' or [x] also place a sticker or caption, the sticker or caption shows an interactivity outline but only the crop is active.

<comment><AUTHOR> I'll forward the compliment and feedback to our UI / UX designer, thanks!

</comment></comment><comment><USER10> 2 Issues I faced 1) The timeline moves too fast when cursor reaches right end . 2) Impossible to click on footer links ( infinite scroll )

</comment><comment><USER11> How come the quality is different when looking at the GIF file and when looking at the default share link?

<comment><AUTHOR> The gif is lower quality than the mp4 or webm formats which we serve on our website and the embeddable GIFS player[1] if your browser supports it.

We've chosen to prioritize speed over quality for the initial launch so at this time but are working on transcoding to various sizes and qualities to return based on the client's device size and internet speed in the player or when someone wants to share a super high quality (and file size) GIF.

1\. [https://gifs.com/docs/player](https://gifs.com/docs/player)

<comment><USER12> Hey, the site's not working for me. Possibly because the source video is long?

I'm trying to create a 10-second gif starting at 71:16 of [https://gifs.com/watch?v=LW6hPl6zx5Q](https://gifs.com/watch?v=LW6hPl6zx5Q).

I get "Your Gif could not be created at this time.". I hope you're able to fix it, gifs.com looks fantastic. If/when it works, I'll use it a lot!

<comment><AUTHOR> That indeed looks to be the problem, we have some ideas in store to fix this but for now source media is limited to one hour.

I filed an issue to report the correct error message and log this as an event so we could see how many times it's happening.

Sorry about that and I hope you find some other AoE videos to gif!

<comment><USER12> Cool, thanks for the quick response!

</comment></comment></comment></comment></comment><comment><USER13> love the simplicity, replacing "youtube" in a url for "gifs". very clever.</comment>


Fed Bailout of AIG Was "Illegal", Government "Violated Federal Reserve Act" [pdf] - <AUTHOR> http://online.wsj.com/public/resources/documents/StarrvUS06152015.pdf

<comment><USER2> The meat of this fascinating ruling is in pg's 8 - 10. Quoting:

      Moreover, there is nothing in the Federal Reserve Act or in any other federal statute that would permit a Federal Reserve Bank to take over a private corporation and run its business as if the Government were the owner. Yet, that is precisely what FRBNY did. It is one thing for FRBNY to have made an $85 billion loan to AIG at exorbitant interest rates under Section 13(3), but it is quite another to direct the replacement of AIG's Chief Executive Officer, and to take control of AIG's business operations. A Federal Reserve Bank has no right to control and run a company to whom it has made a sizable loan.

      ...

      The analysis here leads to the conclusion that, if the Government had done nothing to rescue AIG, the company would have gone bankrupt, and the shareholders' equity interest would have been worthless. Accordingly, the Court finds that the first plaintiff class prevails on liability because of the Government's illegal exaction, but recovers zero damages.

      ...

      As the Court noted during closing arguments, a troubling feature of this outcome is that the Government is able to avoid any damages notwithstanding its plain violations of the Federal Reserve Act. Closing Arg., Tr. 69-70. Any time the Government saves a private enterprise from bankruptcy through an emergency loan, as here, it can essentially impose whatever terms it wishes without fear of reprisal. Simply put, the Government often may ignore the conditions and restrictions of Section 13(3) knowing that it will never be ordered to pay damages. With some reluctance, the Court must leave that question for another day.

<comment><AUTHOR> So AIG head Greenberg was indeed correct in claiming the government overstepped its legal boundaries in its "unduly harsh treatment of AIG in comparison to other institutions" which was "misguided and had no legitimate purpose."

Further interesting quotes:

"The weight of the evidence demonstrates that the Government treated AIG much more harshly than other institutions in need of financial assistance. In September 2008, AIG's international insurance subsidiaries were thriving and profitable, but its Financial Products Division experienced a severe liquidity shortage due to the collapse of the housing market. Other major institutions, such as Morgan Stanley, Goldman Sachs, and Bank of America, encountered similar liquidity shortages. Thus, while the Government publicly singled out AIG as the poster child for causing the September 2008 economic crisis (Paulson, Tr. 1254-55), the evidence supports a conclusion that AIG actually was less responsible for the crisis than other major institution."

The illegality of this decision may also have dramatic repercussions for all future government/Fed bailouts of further banks.

<comment><USER3> My favorite part... The court noted "a troubling feature of this outcome is that the Government is able to avoid any damages notwithstanding its plain violations of the Federal Reserve Act…Simply put, the Government often may ignore the conditions and restrictions of [the Act] knowing that it will never be ordered to pay damages."

</comment></comment></comment><comment><AUTHOR> The Fed's answer to all this:

[http://www.federalreserve.gov/newsevents/press/other/2015061...](http://www.federalreserve.gov/newsevents/press/other/20150615a.htm)

"The Federal Reserve strongly believes that its actions in the AIG rescue during the height of the financial crisis in 2008 were legal, proper and effective."

</comment><comment><USER4> Oh god, this will probably be used by dummies that don't understand either the Fed nor macroeconomics to flip out once again about the Fed like they're the illuminati blahblahblah..</comment>


Show HN: Kriegspiel - <AUTHOR> http://krgspl.com/ <comment><AUTHOR> Hi All. Server is getting absolutely pummelled right now. If you get stuck in a game, you can hit 'refresh' and it should come back. Thanks for playing!

Technicals: It is a medium EC2 running a single node process with socket.io. Database is an EC2 micro running rethinkdb.

\--EDIT-- Replay functionality seems to be down entirely due to load. Games are still running though.

\--EDIT2-- Replays are back up! Just need to wait a while. For those who are discouraged by the signup - I understand, but due to the nature of the game it is necessary to prevent cheating. Email is _not_ required. Feel free to create a random string as a username and password if you want to give it a try.

<comment><USER2> Any insight on what's killing your server? I've been recently looking into socket.io's issues with scaling.

<comment><AUTHOR> Trying to figure that out now. DB seems spiffy, and express is logging requests just fine. Thats the trivial stuff - now I need to start getting my hands dirty.

EDIT - OK serious issues now. getting a range error on redis and the call stack is blowing out, recycling node every 15 seconds or so. Will let you know when its back up.

EDIT2 - Turned out to be an exception due to an invalid date string. Fixed!

</comment></comment></comment><comment><AUTHOR> Hello! I've been sitting on this for awhile. I mostly built it so my cousin and I could play against each other (both long time fans of the classic variant). The source is here: [https://github.com/binarymax/kriegspiel](https://github.com/binarymax/kriegspiel)

<comment><USER3> Hey, I just want to say: good for you to develop your thing and show it off on HN and make the source available. It takes a little bravery.

</comment></comment><comment><USER4> Awesome game, but you could make two imporvments:

1\. Add a turn indicator or a "waiting for opponent" message of some sort.

2\. Make it clear in-site that you can refresh and still get the same game.

<comment><USER5> 3\. Clarify the timeout. I have 3 games running in which the opponent is not responding. Do I win? Also, is there a score?

</comment></comment><comment><USER6> The up-front requirement to create an account put me right off.

<comment><USER7> Well, at least it doesn't require an email so it's just the username and password.

</comment></comment><comment><USER8> German here:

You're technically missing an second s: [http://en.wikipedia.org/wiki/Kriegsspiel_(wargame)](http://en.wikipedia.org/wiki/Kriegsspiel_\(wargame\))

<comment><AUTHOR> Yes you are right. You know us crazy Americans. My apologies for the misspelling! The chess variant is typically spelled with one 's':

[http://en.wikipedia.org/wiki/Kriegspiel_(chess)](http://en.wikipedia.org/wiki/Kriegspiel_\(chess\))

<comment><USER9> Does anybody know the etymology of "Kriegspiel"? All I find are references to "Kriegsspiel" but nobody explains where the 's' went. Was it just a misspelling by those who popularized it?

<comment><USER10> It's a lot harder (for me) to say Kriegsspiel than (the incorrect) Kriegspiel, because the first 's' is voiced (IPA /z/) and the 'sp' is unvoiced[0].

I'm not saying that's _why_ this alternate spelling has arisen, but it could be a transcription of a mispronunciation.

[0] IPA /ʃp/ rather than /sp/ because initial 'sp' in german is pronounced that way. I don't know if this rule has a name... "post-alveolarization"?

<comment><USER9> No no, actually the first s is /s/ i.e. voiceless as in "sea" or "pass", so the complete transcription should be /kʀiːksʃpiːl/. The transition between the /s/ and /ʃ/ is then just making your lips round for the /ʃ/ after the wide and flat /s/. It is still not as easy as just leaving it out, but it is whole lot more doable.

<comment><USER10> I must be extremely out of practice with German :\ I definitely thought that the 'g' in 'Krieg' was /k/ but the 'g' in 'Kriegs' was /g/.

</comment></comment></comment></comment></comment></comment><comment><USER11> I started working on a similar thing last year [0]. In my version, your pieces can see any spot on the board they are allowed to move to.

[0] [https://github.com/valtron/fog](https://github.com/valtron/fog)

<comment><USER12> Too bad you didn't finish it :)

</comment></comment><comment><USER13> See also: [http://r-s-g.org/kriegspiel/](http://r-s-g.org/kriegspiel/)

</comment><comment><USER14> This needs some easy way to try it out. Or at least some screenshots right on the landing page.

I don't really know what kind of game this is just by going to the url, and I am surly not registering at some site without having some solid idea that I will use it more than once.

<comment><USER15> It explains it decently well on the landing page... registration doesn't require email validation just use random atrings.

</comment></comment><comment><USER16> Nicely done.

"AI a Modern Approach" has a nice section on the game if anyone wants to read up on AI a bit.

</comment><comment><USER12> That's really cool. I wish I could see the game unblinded when it's over though.

<comment><AUTHOR> Thanks! Replays will be back up eventually - DB server is tiny and having memory trouble. Can't mess with it now since it will interrupt the games.

</comment></comment><comment><USER17> I used to play Kriegspiel, Bughouse, Crazyhouse, Loser's Chess and many other variants at the ICC. Anyone remember that?

My favorite, however, was Kung Fu Chess, at kungfuchess.com . Whatever happened to that?

</comment><comment><USER18> really really nice! thanks for creating it. the problem with the "hanging" games must be fixed though. you definately need clocks. so the enemy canmot think forever.

</comment><comment><USER19> Is it related to the old Prussian game, or Guy Debords game?

</comment><comment><USER20> I was having fun, and then I timed out trying to make a move.

Clean interface, fun concept.

</comment><comment><USER21> @asdfasdf, get back there and finishbour game ;)

</comment><comment><USER22> I am hooked.

</comment><comment><USER23> very fun, thanks for making this.</comment>


Innovators are shaping Africa's 'informal economy' - <AUTHOR> https://afridigest.substack.com/p/innovators-are-shaping-africas-informal <comment><USER2> The OECD[1] has been covering the topic of the "informal" or "shadow" economy for some time. At one point the term "System D"[2] was circulated by various media outlets[3][4][5] on the back of a 2009 OECD paper (which I haven't been able to source yet).

As mentioned in the article, the informal economy was often regarded as a liability due the difficulties of collecting tax revenues, but policymakers have started realizing that a shadow economy can forge successful entrepreneurs (even though the process is often due to brutal necessity).

[1] [https://www.oecd.org/about/](https://www.oecd.org/about/)

[2] [https://en.wikipedia.org/wiki/System_D](https://en.wikipedia.org/wiki/System_D)

[3] [https://foreignpolicy.com/2011/10/28/the-shadow- superpower/](https://foreignpolicy.com/2011/10/28/the-shadow-superpower/)

[4] [https://freakonomics.com/2011/11/01/the-black-market-is- the-...](https://freakonomics.com/2011/11/01/the-black-market-is-the-second- largest-economy-in-the-world/)

[5] [https://www.wired.com/2011/12/mf-neuwirth- qa/](https://www.wired.com/2011/12/mf-neuwirth-qa/)</comment>


Justice Markandey Katju claimed, "90% of Indians are 'idiots'." - <AUTHOR> http://timesofindia.indiatimes.com/india/90-of-Indians-are-idiots-Markandey-Katju-says/articleshow/17536647.cms

<comment><USER2> Having experienced both the Indian (CBSE) education system and the Western (IB) education system, I can say that the CBSE system definitely lacks in encouraging children to think critically.

<comment><USER3> IB isn't a typical example of western educational system, so you are wrong there. For example I would never say that the Italian Maturità or German Abitur is better than the Indian one.

I have done both british A levels and CBSE 12, and I can say that both have their own positives. But encouraging critical thought isn't one of them for either.

</comment></comment><comment><USER4> It's essentially true for the entire human race.</comment>


Hiring Experiment at Shyft – No Resume Required - <AUTHOR> https://medium.com/shyfting-mobility/clone-wars-the-no-resume-experiment-7eec52c082be#.aycdswn9z <comment><USER2> Based in Paris so remote work or remote interviews?

<comment><USER3> Hi, I'm the head of comm. No remote work for us at the moment but remote interviews :)

<comment><USER2> Good luck. I can't move to Paris at this time as much as I'd want to.

</comment></comment></comment><comment><USER4> Love it!</comment>


IronPython vs cPython benchmarks - <AUTHOR> http://ironpython.codeplex.com/wikipage?title=IP26FinalVsCPy26Perf&referringTitle=IronPython%20Performance <comment><USER2> Well, it was pretty lopsided, but there were some things that IPy was pretty good at. Exceptions are much more efficient, there were some bright spots in dictionary and string manipulation.

Given that CPython is about Python, whereas IronPython is about Python and the MS environment (libraries, IIS, MSSQL, etc.), it looks like a decent alternative if you like Python, or are just tired of C#.

<comment><USER3> I think you've got that exactly backwards. IronPython is much _slower_ than CPython at exception handling and somewhat slower with strings, but very good at, e.g., simple arithmetic operations. (Presumably the JIT turns them into reasonably nice native code.) It's somewhat faster than CPython on PyStone and quite a lot slower on the Richards benchmark. It takes substantially longer to run the whole PyBench suite than CPython, but that's entirely accounted for by its rotten performance with exceptions.

<comment><USER4> Interesting that IronPython is faster than CPython at processing exception handlers when no exception is thrown, but much slower at throwing.

<comment><USER5> I remember when FePy was first coming out, and Jim Hugunin was encountering considerable slowness in regard to exceptions, and he conferred with Guido about it because the fail-first idiom is a preferred python strategy. Anyways, Guido was like, you are functionally equivalent so it doesn't matter. (I'm just regurgitating my rough memory of Jim's blog about it.) But that was also the first time I learned how disruptive an exception was to .Net program flow. I've also seen it in general with other .Net languages -- exceptions kill performance, and defensive programming is very worth the extra effort.

</comment></comment></comment></comment><comment><USER6> Can someone please remind me why interpreter startup time matters? Maybe it's because I originally came from Java-land. I really don't get it.

<comment><USER7> It matters in some contexts.

For quick scripts that you invoke from the console, it's painful to wait a couple of seconds just to see if it does something.

For web servers, if you're using something like Google's App Engine which starts VM instances as needed, or stops them when there's no traffic, users have experienced what is called a "loading requests" ... which is basically a very slow request or requests throwing DeadlineExceededExceptions.

That's because the startup time of the runtime is very slow, especially when you have big dependencies like JRuby ... see this blog post for instance ... [http://googleappengine.blogspot.com/2009/12/request- performa...](http://googleappengine.blogspot.com/2009/12/request-performance- in-java.html)

</comment></comment><comment><USER8> At the day job, they run an all-MS-windows environment. So i use IronPython to write administration scripts for the database and to do some calculation or migrate datasets or things like that.

It is definitely slow, but I am willing to deal with that because Python as a language shines with such tasks.(or any dynamic language I guess)

<comment><USER9> IP is great for that. you might want to check out PowerShell as well: <http://en.wikipedia.org/wiki/Windows_PowerShell>

</comment></comment><comment><USER7> Last time I tried IronPython (~ 2 months ago) it was painfully slow ... slower than Jython which is really discouraging since they've put some effort into this for quite some time now (also hiring Jim Hugunin, the lead Jython dev).

</comment><comment><USER10> What about memory consumption? What's the maximum memory CPython or IronPython would use to run through one of the benchmarks?</comment>


Facebook's Top Open Data Problems - <AUTHOR> https://research.facebook.com/blog/1522692927972019/facebook-s-top-open-data-problems/ <comment><USER2> I strongly dislike Facebook the product, and to lesser extent Facebook the company, but I'm continually impressed with Facebook's approach to engineering in the open. I find this an interesting dichotomy. Would I want to work there? I still don't think so, but my opinion on that front is getting less strong over time.

<comment><USER3> Former Facebook intern here. Facebook the company is a lot more 'hack-y' in the right way of the word than what it looks like on the outside. The company and its products are extremely open, and the projects you do working there have extremely little management and corporate bs.

My experience was really similar to my Google internship, and probably even closer to a "cool startup". I know several people who worked on Google, Facebook, and X (with X another major Silicon Valley company) and say that the first two were a lot closer to each other than to X.

<comment><USER4> Where do you work now?

<comment><USER3> I'm currently finishing my degree while working in a smallish start-up in Buenos Aires.

</comment></comment></comment></comment><comment><USER5> I had a really great time talking to the Facebook engineers during my interviews there. The main pattern I noticed was Harvard (i was applying in management). Even more so, the guys interviewing me were extremely talented and smart. What always weirded me out was... the problems they work on are not that difficult. Once you grasp sharding and operations, you are pretty much set. These guys are _not_ the Manhattan project. The true hard problems in their space: developing their own mobile hardware, keeping teens engaged, pushing the boundaries of design, losing tracking systems in mobile, etc; they don't face head on. Moving petabytes around or caching lots of things in memcache - my roomate and I could do with an aws account and a few beers. Memcache for god sakes is what 300 lines of C?

<comment><USER6> No idea how much you're trolling, but the single file:

[https://github.com/memcached/memcached/blob/master/memcached...](https://github.com/memcached/memcached/blob/master/memcached.c)

is more like 4000 lines of C (by guesstimating the amount of comments etc).

<comment><USER7> I liked his comment (don't shoot me for that), made me smile.

4000 isn't that much either & whats memchached? A hashmap?

He maybe means 300 lines of _relevant_ code (+ ~3700 sugar)

<comment><USER6> Hehe. Well, this is just _one_ file. I might agree that memcached is 300 lines of relevant pseudocode -- but then again, it's implemented in C, not pseudocode...

I agree that 4k lines isn't that much, but it's an order of magnitude off from 300. And again, that's just for that one single file.

</comment></comment></comment></comment><comment><USER8> Can any body throw some light on how facebook's database is designed? I am sure it will be an interesting read.

I was reading somewhere sometime back that each user at fb has its own database. I think that is not possible.

edit: I am googling now again on this topic. First link found is [http://www.quora.com/What-is-Facebooks-database- schema](http://www.quora.com/What-is-Facebooks-database-schema)

<comment><USER9> There isn't one database, although there are a few major types.

The majority of core information (attributes of people and places and pages and so forth, as well as posts and comments) is stored in MySQL and queried through TAO.

Some data is primary stored in things like HBase, such as messages.

Non-primary-storage data (indexes and so forth) exist in various forms optimised for different workloads - so data in either MySQL or HBase might also exist in Hive for data warehouse queries, or in Unicorn for really fast search-style queries.

Other data (such as logs) might reside in one or more of the various data stores, such as Scuba, Hive, HBase, and accessible via Presto, for example.

TAO: [https://www.facebook.com/publications/507347362668177/](https://www.facebook.com/publications/507347362668177/)

Unicorn: [https://www.facebook.com/publications/219621248185635](https://www.facebook.com/publications/219621248185635)

Hive: [https://www.facebook.com/publications/374595109278618/](https://www.facebook.com/publications/374595109278618/)

Scuba: [https://www.facebook.com/publications/148418812023978/](https://www.facebook.com/publications/148418812023978/)

Presto: [http://facebook.github.io/presto/](http://facebook.github.io/presto/)

</comment></comment><comment><USER10> This is slightly off topic, but has any experienced an increase in "fake" toasts from facebook mobile? It seems if I haven't used facebook mobile in a few days or I don't respond to their toasts about very minor people in my life uploading a photo, I tend to start getting toasts that say "You have 5 notifications, 3 pokes and 2 messages.", then I open the app and it takes me to an unknown error page.

Am I being too cynical in thinking that Facebook is intentionally misleading its users in an attempt to bump up their metrics? It interests me that they are seeing jumps in their mobile users (and consequently, ad sales) at the same time that I have been receiving more notifications than ever. Interestingly, the slowdown in fake toast notifications coincided with their quarterly earnings report that show mobile ads accounting for an increasingly large portion of revenue and also mentions an increase in mobile user usage.

Comparing Q1 with Q2 with Q3, Q2-Q3 showed double the increase in ad revenue percent from mobile (59% to 62% to 66%). Maybe this is just all anecdotal evidence, but it seems like these sort of fake notifications should either not be sent out (failure of the system that keeps track of what user receives what toasts) or there was a conscious effort to send these notifications....

<comment><USER11> It's been doing this for me via email lately and it's really annoying.

<comment><USER12> I use gmail's filter to dump them and 99% of my emails to various labels. Once every 2-4 weeks, when I am in the mood, I check them out then delete them all. :-)

</comment></comment></comment><comment><USER13> Something does not add up about hive: They say it has 300 PB, and it generates 4PB per day - which means, at this rate, all data was generated within the last 75 days.

<comment><USER14> Most likely that the 300PB are distilled/normalized/compacted data whereas the 4PB per day are raw logs.

<comment><USER15> This is the correct answer. Source: I am a FB employee.

<comment><USER13> Thanks.

So, how much does the 4PB of data go down to when it goes into hive? My guess would be something like 200TB.

Is it zipped (or lz4/lzo/zopfli/lzma whatever)? Or is it just "distilled"?

<comment><USER16> 99% of raw data is useless. Just as a rule.

Most of it likely gets tossed into a program to determine if somebody actually needs to do anything, or if something is actually breaking.

For example raw user interaction doesn't really grow. While the event is likely ~1kb or so of raw data, at the end your just incrementing a 64bit counter.

This is a baseless exaggerated post but should shed some light on.

<comment><USER13> > 99% of raw data is useless. Just as a rule.

Well, as a non-facebook user, I think 99.9999% of facebook data is useless :) But facebook is in the habit of tracking everyone's surfing habits across the web (through "like" links), and I assume they do more than just "increment a counter with it", even if they don't keep every single detail.

</comment></comment></comment></comment></comment></comment><comment><USER17> I'm really curious how they handle paging if they're only using memcached. E.G. If a a photo node has 10,000 comment nodes (and thus 10,000 edges linking the photo to the comments), chances are you only want to display the most recent 50 comments. Are all of the 10,000 edges stored in memcached under one key and then paged on the application servers? Are they stored in chunks under multiple keys? How is cache consistency maintained if somebody makes a new comment (maintaining the time ordering seems tricky and expensive)?

This is a problem I'm actively trying to solve for a project, so if somebody knows the answer, please get in touch!

<comment><USER18> That's what TAO (mentioned in the article) is for

<comment><USER17> Yes, I've read the TAO paper and referenced it heavily for my project. I'm curious about the implementation specifics of the caching layer.

EDIT: More specifically, how do you efficiently cache your data such that an assoc_range query can be answered from cached without O(n) operations on your application servers. Memcached can't answer a query like "give me 50 items starting from position 0 in the list" as far as i'm aware, so you'd need to pull the whole list to the application server and slice it up there. When you consider that you want the 50 most recent items too, maintaining sorted lists adds extra complexity.

<comment><USER19> TAO is used in place of memcached. You don't ask TAO for a whole list and place it into memcached. You ask TAO for "give me 50 items starting from position 0 in the list", and the TAO cache keeps this list in sorted order.

<comment><USER17> Yes, that's the mechanism I'm wondering about. Let's say I have:

        Application -> TAO -> Cache -> Database

I have a photo node (P1) and 125 comments nodes [C1, C2, ..., Cn] attached to P1 by the edges [(P1,C1), (P1,C2), ..., (P1,Cn)]. I'll ignore the fact that there can be different edge types for simplicity.

Lets say my page size is 50 and I want to view 3 pages of comments for the photo from my application. My application makes the following TAO queries:

        assoc_range(P1, 0, 50)
        assoc_range(P1, 50, 50)
        assoc_range(P1, 100, 50)

My question is, assuming all the necessary data cached such that all of those queries will be a cache hit, how are those edges stored and retrieved from memcached? How are the keys named in memcached?

A naive implementation might be to store the list of all edges for P1 with a key of "P1". To answer te above 3 queries, TAO then needs to pull "P1" (all 125 edges) from memcached 3 times to answer each of those 3 queries and slice the edge list up on the TAO application server... Not great, but probably an improvement over hitting the DB for it (up to a certain list length at least).

A less naive implementation might be to store the edges in buckets of 50, such that the 125 edges are stored with keys of "P1_0_50", "P1_51_100", "P1_101_150", but then time ordering comes in to play...

If my application now wants the 50 most recent items, we could store the edge lists by created date descending and I can retrieve "P1_0_50" from the cache and guarantee I have the 50 most recent items. However, lets say 10 new comments are posted... Now I need to update all my cache pages to ensure the ordering is correct, which is horrendously ineffecient!

To fix this issue, edge lists could be stored in created date ascending order instead, but then how do I know which cache page to fetch to retrieve the 50 most recent comments (seeing as "P1_0_50" is now the oldest 50 items)?

I hope that makes sense!

<comment><USER9> There is no memcache - there is just TAO, and it talks to the database. TAO is a read/write-through cache, so the only way the data changes in the database is through TAO. TAO contains the indices necessary to answer queries like that efficiently (although there may be other systems for doing similar but more specialized or slightly different queries), as well as a cache of the data.

So, when you add a comment, TAO updates its internal structures with the new comment in the right place (after the DB is updated), and there are no "keys" that need to be updated beyond that.

<comment><USER20> Are the TAO APIs asynchronous? Are there any document on the parameters of those APIs? :-)

<comment><USER17> There is a whitepaper about the details of TAO at [https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers...](https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers/11730-atc13-bronson.pdf)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> I'd like to use this opportunity to ask: is it a technical limitation that users still can't search their timeline?

</comment><comment><USER22> So ~650M daily active users..4PB of data warehouse created each day, that means ~7MB of new data on each active user per day. Given that its data warehouse, I'm going to guess its not images, seems like a lot to me. I guess it shouldn't surprise anyone that every interaction on and off the site, is heavily tracked.

<comment><USER9> A lot of that data is duplicated to allow for efficient querying or transformation. It often is too slow to process the data as it comes in, so an initial process will write the data in a raw form, and some other process might select a subset of the data to process, and then submit it in an "annotated" form (filling in, say, the AS number of the client IP). Another process will run later in a batched fashion and perhaps annotate the full set of information and summarize it into a bunch of easily-queried tables.

A lot of that data is also not tied to individuals either - for example the access logs for the CDN (which, being on a different domain by design, does not share cookies so is not attached to an account) even reasonably heavily sampled is probably tens of gigabytes a day, and is rolled up into efficient forms for queries in various ways. A lot of it isn't even about requests coming through the web site/API - it may just be internal inter-service request information, or inter-datacenter flow analysis, or per-machine service metrics ("Oh, look, process A on machines B through E went from 2GB resident to 24GB in 30 seconds a few seconds before the problem manifested").

(Not that it makes too much of a difference at this scale, but it is closer to 860M daily actives.)

</comment></comment><comment><USER23> _3\. Hive is Facebook 's data warehouse, with 300 petabytes of data in 800,000 tables. Facebook generates 4 new petabyes of data and runs 600,000 queries and 1 million map-reduce jobs per day._

So 4 PB per day, but only 300 PB total?

<comment><USER24> Was wondering the same thing. My guess is that some also gets removed each day as well, but it seems unlikely.

<comment><USER25> Think of it like monitoring data. You may collect one-second data on 500 counters per system over 1000 systems, but then you will do a weekly or monthly roll-up where you dump some of the granularity to save space, and after a year you have aggregates that are basically daily trend lines. The more you collect smaller percentage you actually keep.

</comment></comment></comment><comment><USER26> Still using Memcache wow.

<comment><USER27> DevOps here using their mcrouter tool [1] in production. It's a phenomenal swiss army knife for using memcached.

[1] [https://github.com/facebook/mcrouter](https://github.com/facebook/mcrouter)

<comment><USER18> Aw shucks.

Mcrouter was the first piece of software I worked on at facebook. It's nice to see that you like it! (Though the only thing that lives on from me is the umbrella protocol)</comment></comment></comment>


A Blogger Mocks the Denizens of Silicon Valley - <AUTHOR> http://bits.blogs.nytimes.com/2013/08/25/disruptions-a-blogger-mocks-the-denizens-of-silicon-valley/

<comment><USER2> Oh look, a news website reporting on a blogger for a news website and his feud with another news website. And they say Silicon Valley is self-absorbed. :)

Talking about mocking Silicon Valley though... [http://jesuschristsiliconvalley.tumblr.com](http://jesuschristsiliconvalley.tumblr.com)

</comment><comment><USER3> I see the value in providing an alternative to press sycophancy. But Valleywag doesn't solve this problem. An alternative would include substantive long form reportage, not capsmashing screeds.

They're right sometimes in identifying embedded BS in the world of startup and tech businesses.

But a lot of the time all I hear is "HAH! Rich guy! What a jerk!" Like when they published an article about Larry Page's sex life. It sounded puritan and prudish and it was awful.

<comment><USER4> ...

Okay, what's this about Larry Page's sex life?

<comment><USER3> He sleeps with a bunch of women in a fancy apartment. Some of them are important.

That's the whole story.

Dragged out into this shit: [http://valleywag.gawker.com/google-boss- enjoys-15-mil-manhat...](http://valleywag.gawker.com/google-boss- enjoys-15-mil-manhattan-sex-penthouse-909299764)

<comment><USER4> That's about Eric Schmidt.

<comment><USER3> Whelp, you're right. My point stands but that was stupid.</comment></comment></comment></comment></comment>


Google Fiber is coming to Atlanta, Charlotte, Nashville and Raleigh-Durham - <AUTHOR> http://googleblog.blogspot.com/2015/01/google-fiber-new-metro-areas.html <comment><USER2> For anyone interested... the whole motivation of Google Fiber to motivate ISPs to stop gouging us does exactly what it is designed to do.

Cox, in Phoenix, miraculously doubled everyone's speeds > 100mbps last year because they were concerned with our general happiness levels and wanted to show their affection.

Nope... that's not it. Ran into two leads at Smash Burger in June of 2014 and asked them on the spot "When are we getting gigabit speed?" and their reply was "End of year, we gotta move before Google Fiber gets here."

None of that would have happened if G Fiber wasn't coming - CenturyLink and Cox were perfectly fine dabbling in 50mbps service for the rest of eternity without true competition.

Now in less than 12 months my 65mbps service was _freely_ upgraded to 150mbps and we are suppose to have a 1Gbps option any month now (there are test neighborhoods live since last year around Scottsdale)

I have no illusions about how fortunate we are being in a well- wired/competitive environment, but damn am I happy and appreciative.

It actually has more impact on me and where I would want to live than I thought... the idea of going back to < 50mbps makes me sad.

<comment><USER3> Given that reality one wonder why the incumbents don't raise all their speeds to dissuade google from entering there various markets. Perhaps the shear magnitude of their collective greed obscures their reasoning?

<comment><USER4> There's no money in building "dumb pipes" to the internet. Verizon's wireline margin over the last couple of quarters has been under 5% ([http://arstechnica.com/business/2015/01/verizon-nears-the- en...](http://arstechnica.com/business/2015/01/verizon-nears-the-end-of-fios- builds)). Note Google has never revealed how much profit Fiber actually makes.

The existing networks are fast enough to service what makes the money: video services. The cable companies will only upgrade the pipes to the extent that it threatens usage of their video service.

<comment><USER5> Walmart has 3% net income margins; Costco has 2% net income margins; Target has 3% net income margins; BestBuy in its best days had 4% net income margins. By that premise, nobody should have ever built grocery stores or most retail outlets.

<comment><USER4> Retail is far less capital intensive. WalMart generates almost 6x the revenue per dollar of invested capital as Verizon.

</comment></comment></comment></comment></comment><comment><USER6> Advice to all the folks in the new cities: Google deploys per-neighborhood, once signups reach a critical level. So get your apartment manager, your HOA, your neighbors to sign up, even if they don't have any immediate plans to switch. Post tear-off-tag signs in your community with the url: google.com/fiber

<comment><USER7> Yep, I had to do exactly this here in Austin. I literally went door to door, met many of my neighbors for the first time, and mentioned that we were only a few signatures away from fiber in our neighborhood. Even after getting all of the signatures we have to wait until June for our install, which kind of sucks, but it's better than continuing to deal with Time Warner.

<comment><USER8> You went around collecting literal signatures, or got your neighbors to enter addresses on fiber.google.com?

I'm in Nashville and willing to do a fair bit of legwork if it means it'll bump my neighborhood up on the install list.

</comment></comment></comment><comment><USER9> >> It's been nearly five years since we offered to build a fiber-optic network in one U.S. city as an experiment — and were met with overwhelming enthusiasm.

I suspect the enthusiasm had to do with Google not acting like a phone/cable company. My own hope is that this remains true, and that they come to the Boston area.

<comment><USER10> Excavation projects are famously successful in Boston, so the Google Fiber people are probably chomping at the bit to get their shovels in Beantown dirt. <\s>

But seriously, your hypothesis about GF's non-evil fundamental DNA is a good one. It makes for a much better user experience.

<comment><USER11> Lol, I hope that was a joke (i think it was), for those who aren't aware, this: [http://en.wikipedia.org/wiki/Big_Dig](http://en.wikipedia.org/wiki/Big_Dig)

<comment><USER12> <\s> is short for "end sarcasm".

<comment><USER13> </s>

<comment><USER12> Eh, I copied avn2109

</comment></comment></comment></comment></comment></comment><comment><USER14> I am now surrounded by Google Fiber to the east and west, and Chattanooga's public fiber to the south, out here in Knoxville, TN. I really wish we could manage something here. This area has the biggest national lab, and a host of tech industry around it, I was kind of surprised more options didn't exist when I moved down a year ago.

It makes sense to me that Google went after these southeastern markets, because we are the area where Comcast is already enforcing (I've had to pay a couple times) data caps[1]. It's called a "trial" on their site, but the charges are real.

[1] [http://customer.comcast.com/help-and- support/internet/data-u...](http://customer.comcast.com/help-and- support/internet/data-usage-trials/)

<comment><USER15> Same situation for myself currently. Doing some grad work at ORNL also.

I'm really frustrated by comcast, my fiance and I have a hard time doing bandwidth using things simultaneously (netflix/twitch). I constantly watch our data meter to make sure we don't approach the cap.

Although it's kind of baffling because I recently lived in the bay area. AT&T's UVerse I had wasn't much better in terms of the bandwidth and price, I don't recall a cap though.

I'm here for a few years before fleeing back to CA. I've been researching how to start any sort of movement for reasonable service in town but it seems the town council is pretty alright with comcast, even though some tech business have actually moved from knoxville to Chattanooga for the internet. So really not sure what to do other than give up.

EDIT: Wow, I actually didn't realize how bad the situation is here in Knoxville, read more here [http://www.metropulse.com/stories/features/downtown- knoxvill...](http://www.metropulse.com/stories/features/downtown-knoxvilles- broadband-internet-access-kind)

<comment><USER14> Unrelated to this, but do you guys like board games or really almost anything else? I made the mistake of living in Oak Ridge and I've had a hell of a time meeting people ( I mean, clearly, I'm propositioning a random person in an unrelated HN thread). When my mentor isn't around I can go days without talking to people, and it's starting to drive me a bit mad. I keep buying board games and local multiplayer PC games, but they all languish unplayed.

Back onto the topic, I seriously feel Comcast is "trialing" their data caps her because the southeast is perceived as less tech-savvy, and they're probably aware they can get away with it more.

EDIT: Thanks y'all, I'll be sending out some emails when I get home.

<comment><USER15> Hah, I feel the same way! We have carcassonne and pandemic, but playing with just my fiance and I got boring quickly. I usually play single player pc games for the same reason. I'm in a bit better situation socially since I take classes at UTK and commute to ORNL twice a week so I talk and meet other students, but I'm always up for meeting new people who are similarly bored :) I'd probably be available on either the weekends or Friday, feel free to email. I'm also trying to get into hiking and exploring the area if you are into that stuff also.

I agree with your data cap comment. Although it's not just the southeast. Any cornered internet market seems to have them. They're pretty widespread in Canada.

</comment></comment></comment></comment><comment><USER16> I'm fully aware of the extreme cost of rolling out a fiber infrastructure in new locations, but I still wonder why Google isn't more aggressive with their expansion. While the upfront costs are great, they more or less have guaranteed success in most major cities. Nobody likes the incumbent telcos-- they're hated all over the US. I just dream of a world in which we aren't getting raped by Comcast, TWC, and Cox. They are absolutely abysmal and have no incentive to make their customers happy given their market monopoly/oligopoly.

<comment><USER17> Google does not want to be in the telco business at all, it just has to be to protect the source of its revenue which is people connecting to them via some ISP. They are purposely building out in cheap to build areas. If they started hooking up rural areas and areas with high land values and labor it would be a very expensive exercise. What Google is trying to do is get into enough homes as cheaply as possible to light a fire under other telco companies without actually having to build out the whole country.

</comment></comment><comment><USER18> It's kinda sad that the current ISP climate in America is such that faster speeds being rolled out is at the top of Hacker News.

That said, I'm excited as hell for the roll out. I'm not in one of the target cities, but Google Fiber spreading gives me hope that they'll eventually make it to me. I've lived with Bright House and their terribleness for far, far too long.

<comment><USER5> There's nothing trivial about 1gbps consumer broadband (maybe if we were talking about 100mbps). It's very worthy of attention.

Very few countries have competitive access to 1gbps consumer broadband. You can name them on one hand.

</comment></comment><comment><USER19> Whatever Google's motive, this does help the internet to fully self-realize: every endpoint should be able to both consume and provide services. I don't care if it's a phone or a tablet, or a smart toaster.

</comment><comment><USER20> I wonder if, for customers who sign up for the basic (free after install fee) package, Google will ramp up their speed if the FCC announces that "broadband" is officially 25/3\. That'd be a nice bonus.

</comment><comment><USER21> Can any Google fiber user tell us how the TV offerings differ from Comcast/Uverse/TWC/Direct TV? Do they include local channels and other channels I've come to expect?

What is missing from Google TV?

<comment><USER22> Here is the TV lineup for Kansas city Google fiber customers. Looks like they have all the local and traditional channels. Then add-ons like HBO for $20/month.

[https://fiber.google.com/cities/kansascity/channels/](https://fiber.google.com/cities/kansascity/channels/)

<comment><USER21> Seems slightly more expensive than the Comcast/Uverse packages I've had but with much much more bandwidth (not that I am capping out my 45mbs right now).

</comment></comment></comment><comment><USER23> Although I'd love to see them come to Houston, we're 640 square miles, so it's not really possible to do a "city" rollout. If Google ever rolls out fiber by Zip Code, that'll be for us.

I would love to see these guys bring us their MVNO Sprint product and/or build a technology park fed by Google Fiber instead, either of which would still have the desired effect of lowering consumer bandwidth costs.

</comment><comment><USER24> Competition isn't always the boon to free enterprise that we would like it to be. I live in Omaha, NE, where Cox and CenturyLink are the primary ISPs. CenturyLink has limited gigabit fiber in some neighborhoods (not mine, unfortunately), launched late last year.

Cox has raised internet rates about 7% twice in the last 14 months.

</comment><comment><USER25> I wish Google Fiber would come to SF. I understand they're doing something like that in the southbay though.

</comment><comment><USER26> Apple earnings, @ $18 billion, are a smidge higher than Sprint's market cap. T-Mobile US has at various times been for sale. It makes you wonder if the major Web ecosystem players won't just start buying SPs so they can break out of bandwidth scarcity as a business model.

<comment><USER27> Also LEO and MEO satellite internet, balloons, drones.

</comment></comment><comment><USER28> Hooray for me and all my Raleigh-Durham brethren! Besides just being a great development, I think this reflects the rapid growth of the tech sector in this area, and having cheap fiber ISP options will go a long way to ensuring we continue to attract top talent & companies.

<comment><USER29> Any idea what cities in the Triangle are actually getting this? Cary? Chapel Hill? Durham? Raleigh?

<comment><USER30> If you go here: [https://fiber.google.com/newcities/](https://fiber.google.com/newcities/) and hover over the main cities it gives you a breakdown.

Carrboro, Cary, Chapel Hill, Durham, Garner, Morrisville, Raleigh

It has it in the hero picture at the top here: [https://fiber.google.com/cities/raleighdurham/home/](https://fiber.google.com/cities/raleighdurham/home/) as well.

</comment></comment></comment><comment><USER31> I believe Cleveland will still have the fastest 100gig fiber.[http://www.cleveland.com/business/index.ssf/2014/11/clevelan...](http://www.cleveland.com/business/index.ssf/2014/11/cleveland_prepares_to_light_up.html)

</comment><comment><USER32> Sadly, though, the Atlanta rollout only goes as far north as Sandy Springs and Smyrna. No love for those of us up here in Cherokee county.

Hopefully it'll at least scare Comcast and AT&T into stepping up their game. I'd be really thrilled if it straight-up killed Windstream.

<comment><USER33> But also no love for places near Atlanta like Decatur, Avondale Estates, College Park. They're just gonna go for the core + rich northern suburbs. It's a business decision and I get it, but hopefully they'll expand out to those areas soon. To many of those communities in the southern unincorporated areas of fulton/dekalb and southern near-suburbs, free "high speed" internet will be a HUGE boon to the younger generations.

</comment></comment><comment><USER34> I wonder what put those cities ahead, while Portland remains "potential".

<comment><USER35> I had a chance to hear a discussion with the Durham City Manager about this last year. Basically the cities that are most willing to work with Google (favorable laws/willingness to guarantee removing construction red tape/etc) have a huge leg up. There are strategic reasons too, but Google wants to come in, lay down a bunch of cables and move on to the next.

They have the leverage to ask cities to actively pull down barriers that other ISPs would face, and the cities willing to do that are the ones who get consideration.

</comment></comment><comment><USER36> Only the wealthiest of the Atlanta suburbs, it seems. What good is "affordable" true broadband that challenges the cable and DSL companies if it never makes it to those who could most use the savings?

<comment><USER37> Which suburbs are they? I tried 30004 - part of Alpharetta and had no luck.

<comment><USER36> I'm going by the map graphic they have here[1] showing certain suburbs highlighted. Granted, College Park and Hapeville aren't necessarily wealthy areas but they are near the airport so that makes sense. I was thinking more of Sandy Springs and Avondale Estates, and to a lesser extent Decatur.

Of course, that graphic could have zero to do with their plans and I'm going on about nothing. I do wish you could check by ZIP code alone instead of needing a street address as well. It would be nice to know just how close the service is to me geographically.

[1] [https://fiber.google.com/cities/atlanta/home/](https://fiber.google.com/cities/atlanta/home/)

</comment></comment></comment><comment><USER38> MY SWEET REVENGE ON TIME WARNER CABLE AND COMCAST!! Take my money Google. I'd love to see the look on their faces when they realize that Google is coming to eat their lunch.

</comment><comment><USER39> I really wish they start the service in the D.C Metro/VA regions too. We have FIOS but they charge ridiculous prices (i.e. ridiculous compared to G Fiber prices :-))

</comment><comment><USER40> I'm surprised they didn't choose San Jose. There must be some things outside of their control considering they would have expanded into their own neighborhood.

<comment><USER41> In the middle of last year, Google announced that they will also look at other bay area cities like Santa Clara for Fiber. I don't see what happened to that and there is no more mention of Santa Clara in the latest post. Maybe it fell through? The last update I see is from the city claiming that the discussions have ended - [http://siliconvalleypower.com/index.aspx?recordid=3191&page=...](http://siliconvalleypower.com/index.aspx?recordid=3191&page=2065)

I wish I knew the reasons for not picking a city.

<comment><USER42> Santa Clara was always included (and still is) in the potential San Jose rollout. As is Sunnyvale, Mountain View and Palo Alto.

</comment></comment></comment><comment> mkr-hn I just hope it doesn't take too long to radiate out from Atlanta. Barrow County is the middle of nowhere, but it is right on the road between Atlanta and Athens.

<comment><USER32> I've got family in Winder. Y'all are kinda out there.

</comment></comment><comment><USER43> I bet it never comes to the DFW area. Home of Verizon FIOS. Oddly enough I live in Arlington and FIOS is not (nor ever will be) available.

<comment><USER44> I dunno, Nashville's got the AT&T building, heck Google put it squarely in the shot and images for this rollout, seems almost like a message, though it is pretty iconic in a city with little other skyline.

</comment></comment><comment><USER45> Man, that map is impossible to read. Even as a fairly mildly colorblind person, there's no way I can distinguish the colors.

<comment><USER46> Perhaps you're not color blind _enough_. :-) I'm much more than mildly color blind and I can distinguish the legend items just fine. Certainly more so than most graphs/maps I see.

</comment></comment><comment><USER47> I live in Austin. Can't wait until they make it available to me. Some day. Years from, probably.

</comment><comment><USER48> As someone living in Atlanta I can't friggin wait for my break-up call with Comcast. Soon!

</comment><comment><USER49> I'm perfectly happy with 15mbps. I'd rather see prices go down than speeds go up.

</comment><comment><USER3> Is there any rime or reason to the cities that Google chooses to put fiber into? Is it a matter of more friendly local government, less resistant local teleco's, a combination thereof or some other factors?

Just seems the locations aren't exactly tech hubs so I'm trying to get a better idea of how the expansion works.

<comment><USER50> Friendly local government is a big factor.

For example: Google looked at Seattle's NIMBY laws regarding utility boxes on sidewalks (each requiring a lengthy community approval process) and said, "Nope!"

Seattle's new mayor is now looking how to streamline the process to make the city more Google friendly, should the opportunity arise again.

<comment><USER51> > Google looked at Seattle's NIMBY laws regarding utility boxes on sidewalks > (each requiring a lengthy community approval process)

Which is good, as sidewalks are for walking, not utility installation pads.

In much of the UK outside conservation areas BT has powers to install cabinets as it sees fit. Combined with utility poles there are now some pavements on my town impassable to prams.

<comment><USER52> The parent was downvoted, but this is actually true in the UK. The are utility boxes in some really strange locations. Such as a bicycle lane that runs straight into a box, then just continues on the other side.

</comment></comment></comment></comment><comment><USER53> Could be really good for getting some tech love in the southeast.

</comment><comment><USER54> Google Fiber: Coming to a neighbourhood other than your own soon!

</comment><comment><USER55> I'm looking for an apartment now!

</comment><comment><USER56> Does anyone know if Google Fiber monitors your network traffic to serve targeted ads?

<comment><USER57> Take a look at the privacy policy: [https://fiber.google.com/legal/privacy.html](https://fiber.google.com/legal/privacy.html)

"Technical information collected from the use of Google Fiber Internet for network management, security or maintenance may be associated with the Google Account you use for Fiber, but such information associated with the Google Account you use for Fiber will not be used by other Google properties without your consent. Other information from the use of Google Fiber Internet (such as URLs of websites visited or content of communications) will not be associated with the Google Account you use for Fiber, except with your consent or to meet any applicable law, regulation, legal process or enforceable governmental request."

</comment></comment><comment><USER58> PHILLLLLLLLLLLLLLLLLYYYYYYYYYYY! We have Cheesesteaks...

<comment><USER59> Philadelphia is insanely corrupt, owned by Comcast, and will not allow Google to build a network like these other cities will.

Read about the building of Comcast Tower and how a plumbers union forced Comcast to install unnecessary pipes:

[http://forums.civfanatics.com/archive/index.php/t-166039.htm...](http://forums.civfanatics.com/archive/index.php/t-166039.html)

<comment><USER58> Wow! But do they know we have cheesesteaks?

</comment></comment></comment><comment><USER60> No Albuquerque?

</comment><comment><USER61> Great for competition now, sure, but once copper and coax become obsolete Google becomes the new incumbent monopoly. How is that good for anyone?

<comment><USER62> I believe that in amsterdam, even though you only have 1 fiber line running to your house, you can choose from many ISPs over that line. Since (I assume) google is not installing an entirely new fiber network, but merely buying fiber rights and doing last-mile construction, it seems like that would also be a reasonable framework to establish here.

I'm not saying it would be easy. It would take a lot of legal work. But if google's missions to not be evil, and disrupt the ISP market are to believed, they could make it easier.

<comment><USER63> Google said they considered the open access model but rejected it. I don't think they explained why, though.

<comment><USER64> They don't want to dig only to have someone else undercut them.

<comment><USER63> Usually in open access the wire owner sets prices so that they're profitable regardless.

</comment></comment></comment></comment></comment><comment><USER65> Listing cities is really disingenuous as the number of neighborhoods they will actually serve will be quite low, let alone the metro area of cities like Atlanta is quite large and the area most likely buried under cable.</comment>


Spot the job application form on this web page - <AUTHOR> http://digital.cabinetoffice.gov.uk/

<comment><USER2> Only thing I can spot is this twitter feed on the side bar <http://twitter.com/#!/Directgov/statuses/117169143756767232>

</comment><comment><USER3> Old wordpress.com gag ... X-hacker: If you're reading this, you should visit automattic.com/jobs and apply to join the fun, mention this header.

<comment><AUTHOR> Well done. Didn't know that was a WP thing.</comment></comment>


Changemakrs - dream big and do great things - <AUTHOR> http://blog.changemakrs.com/post/37344635029/dream-big-and-do-great-things

<comment><USER2> I love quotes as well. I've seen some people use pinterest as an effective medium for aggregating / sharing inspirational quotes... a quick search yielded this person's quote board:

<http://pinterest.com/ashtond/life-quotes/>

Sure, quotes are embedded in images, but I think that many people like a little bit of visual context for the quote. It might be worth checking out what works/doesn't work for quotes in that medium to incorporate into your site. Just a thought.

<comment><USER3> Hey arscan, designer for changemakrs here. We've also found that adding a visual element to something so textual has been really beneficial. Our big question is how to increase the effect of inspiration in the experience of the user. One answer has been images. We'd really like to use this alchemy as a jumping off point to an even more inspirational experience.

</comment></comment><comment><USER4> Solid service. I've been using it to educate my teammates about entrepreneurial thinking.

Thanks for the new slideshow feature!

<comment><USER3> I'm glad you like the slideshow as much as we do.

</comment></comment><comment><USER3> "We are what we think. All that we are, arises with our thoughts. With our thoughts, we make the world." Good read.

</comment><comment><USER5> Great post...thanks. Will check it out :)</comment>


Embedding Lisp in C++ – A Recipe - <AUTHOR> http://chriskohlhepp.wordpress.com/embedding-lisp-in-cplusplus-a-recipe/ <comment><USER2> This is amazing! Thanks for the tip. I'm trying this out on my next game. CL is much more powerful than Lua and something I'm more comfortable with as a language. Seems to be nearly as painless integrating too.

</comment><comment><USER3> A Lisp embedded in C++

[https://github.com/rongarret/ciel](https://github.com/rongarret/ciel)

It's a toy, but it works.

</comment><comment><USER4> Lisp is everywhere because compilers use ASTs?

<comment><USER5> Yeah, that bugged me, too.

Lisp code is written as an AST? True. C++ (and other) compilers convert code to an AST? Also true. Therefore "No matter if you are programming in Python or C++, Lisp is invariably what holds up the scaffolding behind the scenes. This is true because Lisp is the Lingua Franca of your compiler"? Non sequitur (literally, "it does not follow").

That logic boils down to "A is a C, and B is a C, therefore A is a B". No, it isn't. It would be equally (il)logical to claim that "B is an A", that is, that Lisp was a C++ compiler.

<comment><USER6> In my reading, it appears to be an assumption based on GCC, which apparently uses lisp-like code to represent its AST. (Is this true? I don't know, I haven't looked.)</comment></comment></comment>


Ask HN:  The Secret Sauce of Angel Investment? - <AUTHOR>

I am a single founder who has put a lot of work into the product that I have.
Joining up with a YC type system wasn't going to happen for me for various reasons so I instead sought after some small investment money from individuals and that has worked for me.  I am about to close another round and I wanted to share the document I used this time and the formula that I believe will work for people like myself:
The Formula:
1. Create a version that works good enough, 2. Add users, even if it is for free. 3. Continue to enhance the software, 4  Once you have some traction, find people who see the vision to invest in that vision
Once you have a strong product you can find people who believe in it and in you since you are the one who created the product.  Your product lends you credibility.
I found myself needing more resources to get to the next step, which is to finish the programming for a vertical market on the officezilla.com system.  I have been working too much to pay the bills and I needed some room to maneuver so I was stuck with selling off another 5% which will leave me with 90% of the company.
Here is the document I used to close the last amount I needed...  I had a hard time with OpenOffice on the formatting and my grammer isn't great... but people tend to focus on the strengths of the system and not your grammer if your dealing with the right person.
The document: http://www.officezilla.com/officezilla2008.pdf
For the single founders out there, I am not nearly as smart as most of the people on HN but regardless if you have passion and don't give up you will be successful.  I am not quite there yet, but I can see the light at the end of the tunnel and it is really pulling me forward.
Funding from a "VC" is next to impossible but there is money out there from angel investors, people who see your vision and like your product. <comment><USER2> You are extremely impressive. We need 1,000 more hacker founders like this. This is, incidentally, exactly what Paul Hawken says he did for Smith and Hawken in "Growing a Business".

<comment><USER3> This is so awesome. I read this thread, upmodded it, and left. But somehow I started feeling really happy, and had to come back to write! We do need 1000 more hackers like this! Persistence!

</comment></comment><comment><USER4> Great post. you followed some important start-up rules:

(1) get a prototype or "good enough" version out early and be prepared to iterate

(2) hit a milestone that impresses investors before raising funds. measurable customer traction is an impressive milestone

(3) only raise money when you need it, and make sure to raise enough to safely hit the next milestone that will impress investors

<comment><USER5> Amen to #2.

I'm stunned by the # of people who are in the "I have a brilliant idea and a couple of powerpoint slides!" phase who expect to get funding.

<comment><USER6> Yeah, and there continue to be lots of people in that phase. It's even more surprising because you'd expect the market not to give them funding, and eventually people would learn its not worth their time to seek funding at that stage.

So that means either some people _are_ getting money with just an idea, or the market is really slow and inefficient.

<comment><USER7> I think this kind of bullshit is the new conventional wisdom. It all depends the idea. Ideas are valubale in exactly those cases where the idea is an elegant solution to a big problem.

There was a point where the Polio vaccine was just an idea. Would you rather of funded the polio vaccine or some crap web app that has a missing vowel in it's name with half a million users?

The thinking that an undeveloped idea is not valuable is symptomatic of the lack of vision in the tech world. People interested in elegantly solving big problems know a solid idea is way more important than a little traction. This thinking is wrong for a class of ideas that is essential for solving the big problems in the world that represent the best business opportunities. Silicon Valley and it's associated culture has become the new tired conventional wisdom. This is why silicon valley no longer solves big problems.

<comment><AUTHOR> > solving the big problems

If you have a good presentation and can make a case that your solving a big problem will be financially successful I don't see why you couldn't get funding for it.

I am not solving the worlds problems and have nothing totally unique, so a little traction is more important in my case but maybe not yours.

<comment><USER7> I wasn't talking about your case in particular. These cases are different. I'm just saying that the popular (wrong) "wisdom" is that ideas are worthless, and that's an example of a pre-post fallacy.

I'm talking about the guy above you who said "I'm stunned by the # of people who are in the "I have a brilliant idea and a couple of powerpoint slides!" phase who expect to get funding." I may have replied to the wrong person! I think I was more with the guy I replied to. Sorry!

The very best business ideas will come from people with only an idea and a presentation because elegantly solving a big problem is both the best business proposition and the type of thing that requires only a well reasoned idea that simplifies complexity.

</comment></comment></comment></comment></comment></comment><comment><USER8> Can you tell us how you happened to find your first investor? Since he is in Italy it doesn't seem like this is someone you got through connections.

<comment><AUTHOR> Through HN. I posted a document that I wanted to send to angel investors, I didn't get too much excitement from those who replied, they were not so sure it would work but some gave excellent suggestions.

One of the people "lurking" but didn't reply was this guy in Italy. He emailed me, personally I wasn't sure if he was for real or not but he turned out to be totally real. So that really moved me forward a great deal and I am greatly indebted to him. I have never actually spoken to him on the phone, we did all of this through email. Pretty soon I am about to register his 5% of the shares with Nevada so he is going to be good to go, totally official owner of 5%. I am presonally excited about kicking this thing into high gear and returning his investment many times over.

</comment></comment><comment><USER9> What is a Pro Version?

"Finish "Pro" version. This is simply the current offering but with the menu moved to the top so space is freed on the width of the page for more user data. _New features would go to Pro version users and not to Free version users._ Eventually the pro version features would be very compelling."

Congrats on funding.

<comment><AUTHOR> One of the biggest problems is with the left hand menu taking up too much room. People want to fill up the system with more data but there just isn't enough room on the screen. Since I give everything away for free, I had this amazing idea that the "pro" version would use a different template and not taking away anything from the free version it wouldn't have any new features yet but would as I make them :) So for the moment, the grand benefit is a different more usable template, longer term there would be "Pro" only features, everything that I do that is new, wouldn't get passed onto the free users so that I could have something for the Pro version people to be happy about (new stuff!).

<comment><USER9> Perfectly understandable, just initially, I laughed.

Just imagine telling users that a pro version is the same thing but with a different template.

</comment></comment></comment><comment><USER10> Great work. This is a great advice for folks like us who are looking for some heads up. I know its hard, you have proven that with confidence and passion, nothing can stop you from acheiving what you want to

</comment><comment><USER11> _I am not nearly as smart as most of the people on HN_

All evidence to the contrary.

Excellent overview for a potential angel. May not tell them everything, but it's a good starting point for questions and discussion. It also shows that your "finger is on the pulse," perhaps one of the most important things.

I'm sure this will inspire other single founders. Thanks for sharing.

</comment><comment><USER12> thank you a lot for sharing this!

</comment><comment><USER7> Your story sounds inspiring. I'm just about to pull in some money from an angel kind of like you are. We are in a very similar boat. My start up in not software related though.

In the spirit of helping you to make your business better, I was much less inspired by your name. officezilla? Really? Maybe one problem of working by yourself is that there is no one to tell you when you are making a huge mistake you are blind too. Officezilla? You are pouring your life into something you are calling officezilla? You want that on your tombstone? The thing that was your life's work is officezilla? You can't find a better name for what will be your customers primary way of mentally representing you? This will have more of an impact on what people think of your product initially than almost anything else you do. I would suggest you use a little bit of the money to find a good person to help you pick a better identity for yourself. This is just an unacceptable name. I'm sorry, thats what I think. It's not the same skillset as the work you do for a website. But it's important enough you should change it, before your next round, in consultation with an expert, even if it's a really big pain in your ass to do.

Good luck with your funding and your business. You sound like the type who will be successful. I hope you change the name though. It is really bad. I'm sorry for being so harsh, but it really is an epic fail, and you really should recognize your limits and hire someone talented to help you change it. Good luck.

<comment><AUTHOR> I own the domain "sharecircle.com" as well. I have been buying domains that fit for specific verticals like RealtyGoLive.com, etc.

I am also not hoping to have a tombstone <http://en.wikipedia.org/wiki/Freejack>

<comment><USER7> That's a start. The next step is to not present officezilla to anyone you want to think positively of your project. You look like you are doing very good work and are going to get a good shot at making it. It'd be a shame to have it unfairly tainted by that bad name. This matters a lot to how people perceive you.

sharecircle is much better, BTW. Like a million or billion or million million or million billion or even billion billion times better.</comment></comment></comment>


Postgres 9.2 will feature linear read scalability up to 64 cores - <AUTHOR> http://rhaas.blogspot.com/2012/04/did-i-say-32-cores-how-about-64.html <comment><USER2> As much as I love PostgreSQL (and I do; it's put food on my table for the last decade), I have to stress that this linear-ish scalability needs both pg >= 9.2 and a Linux kernel >= 3.2. It seems to be a combination of the lseek(2) changes in the kernel, and the lock contention/handling changes in the db.

That is: if you're running on an older kernel, you probably won't see quite as much gain.

<comment><USER3> We upgraded from 2.6 to 3.2 recently and our Postgres has been flying ever since. The scheduler changes (and presumably the lseek changes) make a huge difference in load. We have not done any performance timings, though, so we don't know if the changes translate to better performance.

To be specific, the change reduced read I/O (<http://i.imgur.com/L8NWO.png>) and load average (<http://i.imgur.com/7793A.png>) both by an order of magnitude, and the variance is much tighter than before. (The system is a dual Xeon quad-core X5355/2.66GHz with 32GB RAM and RAID5.)

That improvement was fairly miraculous — a factor of 10x just by upgrading a kernel is not something that happens every day. Still, I would not be surprised if Postgres 9.2 pushes performance even higher.

<comment><USER4> what distros are offering a 3.2 kernel?

<comment><USER5> Ubuntu 12.04 will, when its released later this month.

</comment></comment></comment></comment><comment><USER6> Database vendors typically charge by the number of cores. 64-core can get really expensive with database licenses. The hardware cost has decreased drastically over time but database licensing are still in the dark age. Postgres 9.2 has real competitive advantage here. Hopefully it would force the other vendors' licensing cost down.

<comment><USER7> Evidence:

[http://www.oracle.com/us/corporate/pricing/technology- price-...](http://www.oracle.com/us/corporate/pricing/technology-price- list-070617.pdf)

"The number of required licenses shall be determined by multiplying the total number of cores of the processor by a core processor licensing factor specified on the Oracle Processor Core Factor Table"

[http://www.oracle.com/us/corporate/contracts/processor- core-...](http://www.oracle.com/us/corporate/contracts/processor-core-factor- table-070634.pdf)

They give you a big discount for buying Sun servers (.25 factor). Either way, it's a huge amount of money, the standard edition costs a cool $17,500 per processor so with 64 cores and the best .25 multiplier you're still looking at 16 x $17,500 or $280,000 for the DB processor license (that doesn't cover support or anything else). The Enterprise edition runs an astounding $47,500 per CPU, so you can easily run north of a million dollars per server if you're running a lot of cores.

<comment><USER8> Oracle tends to be pretty opaque in their pricing, and part of that is because with any sale this big, there's always going to be a lot of negotiation.

Sure, it's going to be expensive, but only schmucks pay full price for a 64-core license.

Still, it's good to see the best open source database out there delivering cutting edge performance. Great work!

<comment><USER9> Yes you can get a big discount. Then when your business booms, and you need to expand, upgrade or (for some licenses) renew, Oracle gets decide how much of your profit to take off the table in licensing fees, up to the full list.

But of course I'm speaking out of school here. I have no proof Larry Ellison or his minions will pursue such ruthless business tactics. Just a feeling :)

</comment></comment></comment></comment><comment><USER10> To me, Postgres is the most underestimated database. Not sure if this is a bad thing...

<comment><USER11> Agreed. I get that MySQL is incredibly simple to set up, so I can sort of understand why people use it for pet projects or whatever. But what I never understood was why doesn't PostgreSQL see better adoption from the big player (Google, Amazon, Facebook, etc.)?

<comment><USER12> > But what I never understood was why doesn't PostgreSQL see better adoption > from the big player (Google, Amazon, Facebook, etc.)?

An interesting selection of companies given that all three are known for their use of home-grown databases (BigTable, Dynamo, Cassandra) for their primary offerings that are not of the SQL variety at all.

Though I think it is still a good question. It may have something to do with the ease of setting up MySQL when you are a young startup trying to get something working as quickly as possible, leaving it often hard to justify a change after you've hit the big leagues.

<comment><USER13> They may be known for their home-grown databases, but they shouldn't be known for their use of them:

Amazon's primary database is Oracle. Dynamo is used for their shopping carts i.e. for storing sessions; Memcache would probably work just as well.

Facebook's database is MySQL, with sharding and Memcache. I was under the impression they stopped using Cassandra entirely?

Google's business (advertising) is built on MySQL, as are many of their sites e.g. YouTube. I think the newer Google-developed sites (e.g. GMail, Reader etc) are indeed built on BigTable.

<comment><USER14> > Amazon's primary database is Oracle. Dynamo is used for their shopping carts > i.e. for storing sessions; Memcache would probably work just as well.

What? Dynamo is a distributed, persistent, highly-available storage system with incremental scalability and advanced techniques for dealing with slow or unavailable servers. Memcache is a single-process daemon that vends an in- memory hash table via TCP. They are not even remotely comparable.

I'm not dissing memcached, it's cool and useful, but its scope is far, far more limited.

In particular, have you noticed that when you add something to your shopping cart on Amazon but don't buy it, it's still there months or years later? The data doesn't disappear just because some process that was holding the data in- memory crashes.

<comment><USER13> I can't say I've ever put something in my cart, not logged in to Amazon for months, gone back to see it still there, and thought "wow - they must use some sort of webscale storage solution - I'm now going to buy the thing I didn't want to buy six months ago."

From the point of view of solving the actual business problem, therefore, I think something based around Memcache would work just fine!

Dynamo is very useful for persuading you that working for Amazon would be interesting, though.

<comment><USER14> Have you ever worked in an environment at the scale of Amazon? Machines go down all the time. I used the "six months" example to illustrate that the shopping cart is persistent, but the machine crash could just as easily happen the moment before you push "checkout." Losing shopping cart data just because one machine crashed is totally unacceptable.

Programming distributed systems that must survive machine failure and network partitions is a completely different ball of wax compared to simple web programming. If you haven't done it before, you would not believe how much more complicated it is.

Here is an extremely simple example. Suppose you're a radio station that's taking phone calls from people and you want to give an award to the 5th caller. Implementing a program that does this for a single machine is easy, and could be accomplished with a program something like this:

      import SocketServer

      class MyHandler(SocketServer.BaseRequestHandler):
        def handle(self):
          self.server.caller += 1
          if self.server.caller == 5:
            self.request.sendall("Congratulations, you are the 5th caller!\n")
          else:
            self.request.sendall("Sorry, you're caller #%d\n" % (self.server.caller))

      server = SocketServer.TCPServer(("localhost", 9999), MyHandler)
      server.caller = 0
      server.serve_forever()

Using this program, you can "call" the program by doing "telnet localhost 9999" and the program will tell you what caller you are. This took me about 5 minutes to write, and I'd never used this Python API before.

Now imagine that you want to implement this same logic, but using a cluster of machines that could go down at any time. You want the group of machines to form "consensus" about which number each caller is; consensus in this context just means that the group of machines arrives at a single answer, and any machine you ask will give you the same answer.

Finding an algorithm that can do this robustly is so difficult that it was a major breakthrough when one was discovered in 1988. It's called Paxos and you can read about it here: <http://en.wikipedia.org/wiki/Paxos_(computer_science)> Even though it has been known for over 20 years, it is still a complex topic that very few people understand the details of.

The point of all of this is just to say; you can't compare a single-process in-memory cache to a distributed and fault-tolerant system. They are completely different beasts, and many business problems do indeed need the latter.

<comment><USER13> I know about Paxos. I know when to use it, and when not to use it.

Seeing as you came up with the example, using Paxos for an "Nth caller wins" is a really bad idea - the Nth caller to your switchboard likely won't be the one selected. You probably _need_ a single server system, like the example you wrote (but without the threading errors.)

<comment><USER14> Then why would you compare dynamo and memcached? Or suggest that Amazon could use an in-memory cache as the primary store for shopping cart data? It makes no sense.

If you want to be particular about ordering, then you can always use paxos to elect a master that handles everything serially and failsover.

<comment><USER13> I see you work at Google. How about advancing the conversation by telling us how Google stores its sessions?

<comment><USER14> Let's make a deal. I'll start "advancing the conversation" once you stop misleading people (ie. admit that your initial sweeping claims were incorrect).

<comment><USER13> Deal. My initial sweeping claims were incorrect.

So: How does Google store its sessions?

<comment><USER14> So I'm not an expert in Google's front-ends and I'm not sure that Google stores "sessions" in the way you'd generally think of them in web apps. But usually when we have requirements like what you'd need for sessions (highly available, low-latency, highly scalable) we use Megastore:

<http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf>

[http://www.readwriteweb.com/cloud/2011/02/megastore- googles-...](http://www.readwriteweb.com/cloud/2011/02/megastore-googles- answer-to-no.php)

Megastore is a layer on top of Bigtable that adds indexing, synchronous replication across data centers, and ACID semantics within small partitions called "entity groups."

<comment><USER13> Good deal! Agree that traditional sessions are best avoided, but good to know that megastore is suitable for session data.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> Is there such a thing as a suite of standardized performance tests for large- scale, multi-core databases? How are people comparing Cassandra, PostgreSQL, mySQL and other options against each other for raw performance?

<comment><USER16> > Is there such a thing as a suite of standardized performance tests for > large-scale, multi-core databases?

There is: the TPC family[1] and their opensource dopplegangers, the OSDL-DBT family[1].

I don't think they've been applied to non-relational databases as yet.

[1] <http://www.tpc.org/information/benchmarks.asp> [2] [http://sourceforge.net/apps/mediawiki/osdldbt/index.php?titl...](http://sourceforge.net/apps/mediawiki/osdldbt/index.php?title=Main_Page)

</comment></comment><comment><USER17> With this improvement, how much is Postgres hampered by a lack of a parallel query processor?

For OLAP work, it seems to be the primary bottleneck.

<comment><USER18> A good project to watch in this regard is Postgres-XC. Maybe not quite ready for production but it's close. (<http://sourceforge.net/projects/postgres- xc/>)

</comment></comment><comment><USER19> Yikes, that's a lot of cores. Glad to see the Postgres team keep pushing the scalability envelope.

<comment><USER20> Is it? I mean, bargain basement budget desktops have more cores than a typical server of 10 years ago. A 24 or 32 core server can't really be considered that exotic these days, can it?

<comment><USER21> The number of places that run e.g. 4U $10,000 HP servers with 4 socket, 16 core Opteron servers is reasonably low I think.

<comment><USER22> Change the branding to Supermicro and cut the cost to $5,000 and suddenly, many dedicated host providers out there have these servers. We're probably only 2-3 years away from this core count being a $2,000-$3,000 box. Throw ARM and Tilera architecture and whatever AMD is doing with the acquisition of SeaMicro in there and it might even accelerate that pace.

</comment></comment></comment></comment><comment><USER6> That's awesome. I have to admit I have always know Postgres is great and toyed with it but never used it in a real project, due to the availability of MySQL or client preference. I'll try to put it into the current project. Client wants Oracle since they already have Oracle license, but I will change the requirement to support Postgres as well.

<comment> j-kidd If possible, try not to go down the "support Postgres as well" route.

I was in your situation, where client wanted SQL Server since they already have the license. During development, I use PostgreSQL instead, to "support Postgres as well".

At the end, roughly one-third [1] of the _total_ development effort was spent on overcoming SQL Server's limitations, things that you would never have to think about in PostgreSQL.

So, try telling the client that they already have PostgreSQL license as well, with unlimited future upgrade.

[1] This figure was pulled from ass. The actual productivity loss could be more due to similar reasons outlined in <http://news.ycombinator.com/item?id=3784750>

<comment><USER23> Maybe the requirement is Oracle or SQL Server because the client has the resources (DBA, support contracts, etc) to support those platforms in house?

<comment> j-kidd Not in my case, the client doesn't have the resource, and we don't have the expertise. At the end, I became somewhat of an expert in SQL Server (i.e. I know how to start SQL Profiler and read query plan), at the cost of important functionality not delivered to the client on time.

</comment></comment></comment></comment><comment><USER24> I'm new to all of this, so does this mean that other databases don't have this kind of performance?

<comment><USER25> Performance is subjective, but let's dig into the performance optimization this patch includes. The update addresses an issue where to do a llseek (read) of the database, the linux kernel would lock the read, causing it to have only one output at a time. The patch removes the lock because it was unnecessary and thus the performance scales concurrently without a lock creating contention.

This kind of performance optimization isn't new, concurrency is the name of the game. Erlang is a language built around concurrency and it has some databases written in it (couchdb) that scale with more cores due to erlangs inherent capabilities. So has this kind of performance increase been seen before, yes.

<comment><USER26> The bulk of the fixes were in the locking in PostgreSQL though, and before those fixes were made the llseek problem did not appear to many since in almost all cases people hit a bottleneck in the PostgreSQL code before hitting the Linux Kernel one.

<comment><USER25> Thank you! Do you have any experience with CouchDB? I'm sure you've at least of heard of erlang and all the benefits it promises, I'm considering whether to use it for a future project, I might because the JSON REST api seems convenient to use without having to worry about scale. But I haven't done testing and could use advice.</comment></comment></comment></comment>