The 50th Anniversary of Noam Chomsky's Aspects of the Theory of Syntax - <AUTHOR> http://chronicle.com/blogs/linguafranca/2015/06/22/revolutionary-methodological-preliminaries/

<comment><USER2> "Chomsky's theories carried an atomistic, bottom-p approach to language to it's natural conclusion. It certainly did inspire a generation of research, most of it misguided. But its real flaw was one it inherited from its structuralist predecessors. It treated utterances as decomposable structures rather than as wholes, which is how the brain treats them. From the very first the Chomskyites had to admit that no research supported any connection whatsoever between the structures they posited and what went on in the brain. They assumed that further brain research would be needed to uncover those connections. But all the brain research since then has shown that transformational grammar (and even its rivals, which I studied) bears no resemblance to what the brain does with language. So what Chomsky gave us was the foundation for the ultimate mathematical elaboration of a structuralist theory that bears no relationship to anything in nature."

-My dad (Ph.D. Linguistics, Yale - way back in the day)

<comment><USER3> Thing is, Chomsky is (at least partly) responsible for the so-called Cognitive Revolution. Look up B.F Skinner and notice what happened to his career after Chomsky reviewed one of his papers. Skinner's brand of behaviorism was surely something we didn't need around anymore. He also changed the way everyone thinks about language. Before Chomsky, there really was no real theory of syntax that was even close to being scientific and formal.

Although probabilistic machine learning methods replaced the Chomskyan brand of formal theoretic methods in NLP, a lot is owed to is work. Obviously there is the Chomsky Hierarchy but even things like the Penn Treebank POS tags are taken straight out of work in the Generative tradition from circa 1970 (not sure why no one has tried the updated and more accurate tags currently being used in theoretical linguistics). Anyways I could go on and on but Im wondering what your Dad has to say about this ?

P.S There is an ongoing joke in the field of Linguistics about people who either think they understand Chomsky or just straight up don't understand him. Its difficult stuff.

<comment><USER2> "No question that a lot of the tools developed under generative grammar have proved useful, and one can certainly cite the Huxleyan principle that truth comes sooner from error than from a vacuum. I think I would say that the American structuralist tradition was headed in a wrong direction from at least 1933 (Bloomfield's Language) and that Chomsky (despite the claims of his followers that he made a revolutionary break from structuralism) simply gave structuralism a mathematical rigor that allowed it to be taken to its logical conclusion. The tools that were developed in the process have allowed the field (or the more brain-oriented parts of it) to move in a rather decisively different direction.

Full disclosure: I got my PhD at Yale under Syd Lamb, so naturally I would not be classed as a Chomskyan. It took me some years to realize that Lamb's StratGram itself suffered from the same reductionist flaws. The only theoretical school I know of that had a chance of moving in the right direction (until very recently) was Firth's. Unfortunately, neither he nor his followers developed the theoretical tools that were needed. Certainly Halliday's Systemics became as reductionist as any of the others. Hudson's word grammar made some good moves but never really developed well. So yeah, Chomsky set something in motion that produced some useful theoretical tools that could be used to help push off in new directions. But the TGG tradition, being a well developed extension of the Bloomfieldian tradition, is still basically a dead end."

-My dad

<comment><USER3> I mean, if you happen to have some data that Stratificational Grammar can explain better than Generative Grammar I would love to see it. There are a bunch of theories directly influenced by Chomsky which offer alternatives to generative grammar. I've yet to see someone show that one is superior. In many ways, they are all extensionally equivalent.

<comment><USER2> "I never said that StratGram can explain things better than TGG. I said it had the same reductionist flaws. It and all the theories influenced by Chomsky are really outgrowths of Bloomfieldian structuralism and share its underlying orientation. As you say, in many ways they are all extensionally equivalent. The field needs to move beyond all of them. Don't know what assumptions you are referring to, but a forum like this really isn't the place for detailed arguments (even though I used to require my students to provide data, analysis, and argument in two-page, double spaced papers). Just check out J. R. Firth, and read him not for the outdated details but to see what his underlying assumptions are."

-My dad

</comment></comment></comment></comment></comment><comment><USER4> Recommended: Is The Man Who is Tall Happy? on Netflix. A nice introduction to Chomsky and his linguistic ideas.

<comment><USER5> Does somebody know any alternative (legal) sources? Netflix is not available to me without proxying through another country.

</comment></comment><comment><USER6> > It is rather surprising that more has not been done this year (thus far, > anyway) to commemorate a significant semicentenary: the 50th anniversary of > what could reasonably be called the most influential linguistics book of the > 20th century.

Linguistics as a field is in a post-Chomsky hangover and none-too-happy to celebrate the man that put them there. Chomsky's many years as the dominant voice in the field were traumatic for linguistics. Once a decade or so he would promulgate The New Way of doing syntax, handed down from on high from this very influential man, which meant incredible disruption not only for syntacticians but also for those working in semantics and morphology/phonology.

Much of Chomsky's influence came not from his intellect (though that was surely prodigious) but from his forceful personality. His will to publicly ridicule not only opposing ideas but opposing personalities was a big part of his rise and continuing dominance. Linguistics was a nasty place to work for a long time. (See "The Linguistics Wars").

Now that he's not actively working to maintain it, Chomsky's academic reputation has suffered substantially and the field is largely trying to move on. Large-scale corpora and computational capabilities enable more empiricist rather than rationalist approaches to human language, which has made the argumentation less personal and more productive.

At least, that's my take.

</comment><comment><USER7> This was actually not Chomsky's first book on this topic. That was Syntactic Structures (1957).

<comment><USER8> Neither the title here nor the article itself claim it to be his first book, only an incredibly influential one.

The article even agrees with you:

> §3 sets out a radically revised structure for transformational grammars, > quietly abandoning the notion of "kernel sentence" from _Chomsky's first > book, Syntactic Structures_ , and introducing the term "deep structure," > which in a sense replaces it.</comment></comment>


Common ground and cultural prominence [pdf] - <AUTHOR> http://www.stanford.edu/~nfast/Common_ground_and_cultural_prominence.pdf

<comment><AUTHOR> Why do well-known ideas, practices, and people maintain their cultural prominence in the presence of equally-good or better alternatives? [...]</comment>


Universal adversarial perturbations - <AUTHOR> https://arxiv.org/abs/1610.08401 <comment><AUTHOR> Abstract: Given a state-of-the-art deep neural network classifier, we show the existence of a universal (image-agnostic) and very small perturbation vector that causes natural images to be misclassified with high probability. We propose a systematic algorithm for computing universal perturbations, and show that state-of-the-art deep neural networks are highly vulnerable to such perturbations, albeit being quasi-imperceptible to the human eye. We further empirically analyze these universal perturbations and show, in particular, that they generalize very well across neural networks. The surprising existence of universal perturbations reveals important geometric correlations among the high-dimensional decision boundary of classifiers. It further outlines potential security breaches with the existence of single directions in the input space that adversaries can possibly exploit to break a classifier on most natural images.

<comment><USER2> Super interesting. I'm on mobile and haven't had time to read the whole paper yet - would it be feasible to continuously compute these perturbation vectors during training and include them as part of a larger heuristic? For instance, to incorporate the objective of maximizing the size of the perturbation vector necessary for misclassification? The goal being to end up with a net that is more resistant to such perturbations.

<comment><USER3> Short answer: No. Computing these perturbations requires an expensive optimization with multiple passes through the dataset, and this would be prohibitively expensive to do in the inner-most loop of training.

There are other work in the literature describing faster algorithms to compute these perturbations, which makes it possible to use them while training. See, eg.: [https://arxiv.org/abs/1412.6572](https://arxiv.org/abs/1412.6572)

<comment><USER4> See also: [https://arxiv.org/abs/1511.04599](https://arxiv.org/abs/1511.04599)

IMO, (at least) two pieces of research on the subject means that the short answer really is "yes". Maybe not the exact technique used in the paper in the original post, but conceptually similar techniques.

<comment><USER3> It's easier to find fooling perturbations of one image, but not of the whole dataset. I assumed the question was can we use the universal perturbations for robust training? The answer to that is still "no", I think.

</comment></comment></comment></comment></comment><comment><USER5> This seems to imply the features lernt by neural networks are very different from the features humans use to distinguish the same objects because they are affected by distortions that do almost not interfere with features used by humans at all.

<comment><USER6> One thing is neural networks are much smaller than human brains and most likely have far few overlapping redundant systems. If you had three separate neural networks that called on a consensus you might find it much harder to find adversarial inputs.

<comment><USER7> This reminds me of signal attenuation/gain/feedback (by neurotransmitter release [I want to say dopamine...]) due to error in the visual cortex.. Hopefully someone who's studied that might have something to share.

<comment><USER8> I believe the parent was referring to having an ensemble of models with different trained networks can reduce variance and perhaps avoid issues like these.

Attention, localized gain, etc would not have this effect, but they tend to allow a smaller network to perform more sophisticated tasks.

<comment><USER7> Like so? [https://en.wikipedia.org/wiki/Stochastic_resonance_(sensory_...](https://en.wikipedia.org/wiki/Stochastic_resonance_\(sensory_neurobiology\)#Multi- unit_systems_of_model_neurons)

</comment></comment></comment></comment></comment><comment><USER9> Several of the universal perturbation vectors in Figure 4 remind me a lot of Deep Dream's textures.

I wonder what it is about these high-saturation, stripy-spiraly bits that these networks are responding to.

Is it something inherent in natural images? In the training algorithm? In our image compression algorithms? Presumably, the networks would work better if they weren't so hypersensitive to these patterns, so finding a way to dial that down seems like it could be pretty fruitful.

<comment><USER10> My _intuition_ is that these patterns "hijack" the ReLU activations in the lower levels, causing either important features to not fire or features that shouldn't fire to do so. Usually the lower layers learn very primitive shapes like lines and curves, and _I think_ (although I'd need to double check) that they usually pass through entire color channels rather than nuanced mixings of colors. (So one features would either pass through all of red or all of blue or all of both, rather than pass just 66% red, 47% blue, and 33% green -- if it did the latter it wouldn't be able to generalize well) This propagates the error through the network, where the later activations start firing in the wrong places, causing the mis-classification.

(This is totally unsubstantiated though)

<comment><USER11> No intuition necessary

> The surprising existence of universal perturbations reveals important > geometric correlations among the high-dimensional decision boundary of > classifiers. It further outlines potential security breaches with the > existence of single directions in the input space that adversaries can > possibly exploit to break a classifier on most natural images.

The paper unpacks that explanation pretty well along with actual pictures and how they are related to the classification boundary.

</comment></comment></comment><comment><USER12> This is really great research and interesting: (very roughly) how to compute a very small mask which, when applied to any image, makes the neural network misclassify it, whereas humans would notice no essential difference.

Quite remarkable.

<comment><USER13> It says these universal vectors are the same across different classifiers. Why would that be?

<comment><USER14> I'm not an expert but it seems these perturbations are fiddling with the fundamental approach used with a NN. Mainly, that a NN works in layers. So these perturbations must be messing up the lowest layers and then the higher layers end up generating the wrong features and ultimately the model misclassifys. See [http://i.stack.imgur.com/jpYdN.png](http://i.stack.imgur.com/jpYdN.png)

</comment></comment></comment><comment><USER11> This is why I'm never driving a car that is classifying stuff with neural networks. Some dust, some shitty weather conditions and that pigeon becomes a green light.

<comment><USER15> This wouldn't affect that because the perturbations were specially picked to mess up the network. It wouldn't just happen naturally.

Also self-driving cars have distance sensors and wouldn't just drive into upcoming traffic because of one sensor anomaly.

<comment><USER16> Ok, so some guy invents a device that tricks every car at an intersection into seeing a green light, and maybe blinds them to the presence of other cars.

<comment><USER17> I dont think you quite understand the problem. This can't happen.

<comment><USER10> I would imagine you could print the pattern onto a film that you stick on the lens of a camera to throw off its classifier.

<comment><USER18> It won't be able to focus on something so close. It'll just slightly darken the whole image uniformly.

</comment></comment></comment></comment></comment></comment><comment><USER19> In signal processing you often have to pass the data through some sort of low- pass filter before attempting your analysis. I would be surprised if that isn't one of the methods being tried to protect deep neural nets from some of these attacks. Obviously there are some issues (needing to train on similar data, and such blurring interfering with first-level features that emulate edge-detection and so on).

</comment><comment><USER20> So what happens when you stick this procedure in the training loop? Do you get networks which are robust against doubly-universal perturbations?

</comment><comment><USER21> What happens if you include the perturbations in your training data?

<comment><USER22> If my understanding is correct, the perturbations are inherent in the model, not the data. It's a vulnerability in the high dimensional decision boundary of n nets.

</comment></comment><comment><USER23> Reminds me a little bit of the short story BLIT [1], where scientists have accidentally created images that crash the human brain. Cool stuff!

[1]: [https://en.wikipedia.org/wiki/BLIT_(short_story)](https://en.wikipedia.org/wiki/BLIT_\(short_story\))

<comment><USER24> "Snowcrash" is the more realistic Neal Stephenson version where it gets at the eye-brain-embedded hardware. And of course the original, "the joke so funny that if read or heard would make you laugh yourself to death".

Humans seem really good at being imprevious to these, due to millions of years of ignoring things..

</comment></comment><comment><USER25> I'm guessing it won't be long until someone uses this technique to computer and apply perturbation masks to pornographic imagery and make NN-based porn detectors/filters (like the one Yahoo recently open-sourced) a lot less effective.

</comment><comment><USER26> Is there reason to think the human visual system is sufficiently well modeled by deep neural nets that our brains might exhibit this same behavior? My first thought was the perturbation images would need to be distinct per person, but photosensitive epilepsy like the Pokémon event [0] might suggest the possibility of shared perturbation vectors.

[0] [https://en.m.wikipedia.org/wiki/Photosensitive_epilepsy](https://en.m.wikipedia.org/wiki/Photosensitive_epilepsy)

<comment><USER27> What I find interesting is that the labels for the perturbed images aren't completely off in all cases, eg, wool for a shaggy dog.

<comment><USER14> that image also seems very black (the dog takes up most of the image) so the perturbations probably didn't have much to perturb. Also the perturbation is "universal" so it could have simply landed on the same classification.

</comment></comment></comment><comment><USER28> My science-fiction brain is, of course, interested in this as a method to defeat face-detection _in a way humans can 't see_. I'd like to think that the crew of the Firefly used this technology to avoid detection when they did jobs in the heart of Alliance territory.

</comment><comment><USER29> Could you just add noise to any image before passing it through a NN to defeat this kind of attack?

</comment><comment><USER26> Can someone help with a notation question? In section 4 of the paper, the norm of the perturbation is constrained to a maximum of 2'000 which presumably is "small" but I don't know how to parse an apostrophe like that

<comment><USER26> Update: later in the paper, the authors mention that 2x10^4 is an order of magnitude larger than 2'000 so perhaps this is just a way of introducing a thousands separator without introducing cultural ambiguity over whether it's a thousands separator or a decimal separator?

<comment><USER5> It is a thousands separator and for example used in Switzerland and on calculators with LCD display.

</comment></comment></comment><comment><USER30> My intuition is that the existence of adversarial images with barely perceptible differences but a high-confidence misclassification will lead to a new NN architecture for image classification.

</comment><comment><USER31> This is like Godel incompleteness for deep learning.

<comment><USER32> Why?</comment></comment>


Ask HN: Can someone tell me a good tutorial to learn Apache Spark? - <AUTHOR> Hello,

     I'm trying to learn and understand Big data tools like Storm, Spark etc and then evaluate which will fit my company's use case. Can someone point me to a good(preferably free) tutorial for Spark? Thanks a lot, HN folks.
 <comment><USER2> What is your company's use case? What is the data volume? velocity? variety? What is your company currently using?

Given the technical complexity of Storm and Spark, this might be a situation where paying a consultant for assistance makes sense.

Good luck.

<comment><AUTHOR> Nah man. No consultants. But a good tutorial will do. We're just starting up.</comment></comment>


Ask HN: First time giving a speech at a conference. How do I prepare best? - <AUTHOR> Dear HNlers
I recently submitted a proposal for a conference without expecting anything to happen. To my big surprise I got accepted and after the first excitement has settled I'm slowly getting the chills ;-)
The conference is in a significant niche for what I do so I'm going to meet and present to people who could be quite important for me.
To those who have done this before, how did you prepare yourself for the experience? I'm talking less about the content of the speech but everything else.
Any tips highly appreciated, many thanks in advance! <comment><USER2> I just gave a presentation at a local show, here is what helped me:

1) Finish the presentation long before your talk

2) Practice the presentation out loud and make final tweaks to the slides

3) Want to add something "funny"? Yep, plan it out

4) Once locked down, practice the presentation once a day (7 times?)

5) On the day of the presentation, don't review the slides

6) Show up early and stand on the stage and watch people enter

7) You are ready. Start on time.

</comment><comment><USER3> To prepare your speech, follow the advice given in "Death by PowerPoint":

[https://www.slideshare.net/mobile/thecroaker/death-by- powerp...](https://www.slideshare.net/mobile/thecroaker/death-by-powerpoint)

In the longer run, join a Toastmasters club - this is a nonprofit organisation aimed at improving its members' public speaking skills: [http://www.toastmasters.org](http://www.toastmasters.org)

</comment><comment><USER4> What I do:

1\. Know the shit out of the topic and your slides.

2\. Present to your $Town $Technology User Group first.

3\. Listen to something by Lonely Island before you get on stage - you want to feel energetic but happy.

</comment><comment><USER5> If at all possible (and it's not always), watch the earlier speakers and their presentations. At most conferences there's an overlap in themes, and few things are more annoying for an attendee than a speaker who blindly repeats what a previous speaker said.

If you've seen them, you can reference them. You may still be a little repetitive, but use this to reinforce the point - as X said earlier, etc etc; or Y said this earlier, and I disagree because etc etc. You look like you really care about the conference, not just yourself (and other speakers love being name-checked, so it will help you build relationships at that level too).

Lastly, don't be surprised if there's minimal feedback. I often find myself coming off stage on a high, only to have 1-2 people proactively speak to me for the rest of the event. What really amazes me is how many people remember you - I got introduced a few months ago to a business owner in another city, and his first comment to me was "I saw you speak at a conference in 2008 and you gave me some really practical tips on XYX". Blew me away. Good luck!

</comment><comment><USER6> Read this post:

[http://waitbutwhy.com/2016/03/doing-a-ted-talk-the-full- stor...](http://waitbutwhy.com/2016/03/doing-a-ted-talk-the-full-story.html)

It resonated with my experience a lot, especially the bit about finalizing and rehearsing the talk the night before a conference. :)

Overall, don't overthink it. It's not that hard and scary as it seems.

</comment><comment><USER7> Having done this many times myself I can tell you to check in-advance, what tech they have and then asked what backup they have. If they don't have a backup, think of what you can bring. As for your talk, whatever it is, a narrative say, make sure you have 2 copies, 2 devices ready to serve you. Handouts. I know it sounds old-school but it is nice. Also, bring business cards to hand out. Oh and about that handout. Get some really great feeling folders to put them in. I wish you the best.

</comment><comment><USER8> Know that everyone gets stressed when presenting - it's normal. Take deep breaths, practice, and visualize it as a conversation between you and your audience. They really want you to succeed and want to hear you speak.

</comment><comment><USER9> In addition to the other great comments here, I would add: _have some extra material_. The worst thing would be to get through all of your slides in 10 minutes when you planned for 30. On the other hand if you only get through half your slides, skipping over some in the middle, no problem! If there's a Q+A session, people will naturally ask you to elaborate on certain points, and you'll then be prepared.

</comment><comment><USER10> Don't worry buddy,

Follow this instructions -->[http://michaelhyatt.com/how-to-be-better- prepared-for-your-n...](http://michaelhyatt.com/how-to-be-better-prepared-for- your-next-major-presentation.html)

I hope it will be helpful

</comment><comment><USER11> Check out Carmine Gallo, Talk Like TED > [https://www.youtube.com/watch?v=AFTPyvO6kcY](https://www.youtube.com/watch?v=AFTPyvO6kcY)

*Ironically, not the best audio track. But the idea still comes through.

</comment><comment><USER12> Prepare early. Present often to someone else and get feedback. Be honest about what you know and what you don't. Know your audience and engage with them. Make your slides and code available online and tell people that so they pay attention.

Good luck!

</comment><comment><USER13> Be well prepared in Advanced, Also prepare a list of questions that you might think asked at conference, so this will help you to get relaxed.

Just be relaxed and dont get excited during your speech.

</comment><comment><USER14> I liked [http://perl.plover.com/yak/presentation/](http://perl.plover.com/yak/presentation/)

</comment><comment><USER15> Record yourself giving the talk a week beforehand, and watch the recording.

</comment><comment><USER16> Try giving your talk at a smaller, local meeting (if possible) for practice.

</comment><comment><USER17> Try to present your ideas/content in form of "storytelling".</comment>


Why There's Nothing Wrong with Being an Average Developer - <AUTHOR> http://readwrite.com/2015/05/08/average-developer-10x-programmer-myth?utm_content=buffer86304&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

<comment><USER2> This may be the case, but I've seen some pretty bad code in my time. We should all strive to create more maintainable and efficient code.

<comment><USER3> Indeed, and it doesn't take an exceptional amount of effort to learn how to do so, either. I've inherited a large plate of untested spaghetti at my current job, but we've managed to make some good inroads through testing and refactoring; developers of all experience levels are able to produce better work thanks to this emphasis.</comment></comment>


A Cat Parasite May Be Controlling Our Minds - <AUTHOR> http://www.pbs.org/wgbh/nova/insidenova/2010/06/t-gondii.html <comment><USER2> <http://news.ycombinator.com/item?id=789517> (the longest discussion on this so far, I think)</comment>


The best career decision I made - <AUTHOR> https://medium.com/p/the-best-career-decision-i-made-f5825683004 <comment><USER2> Nice post, thanks.

Just to counsel some additional advice on this - be aware that eating (and your diet) is much more important than exercise when it comes to maintaining a healthy lifestyle.

I point this out because I enjoyed exercising for many years before really getting to grips with this point. You can do lots of exercise but still eat badly and consequently not really make any progress.

However exercise is still beneficial.

There are lots of articles on this, but this NYT article[1] includes links to meta-studies for anyone interested in that level of detail.

[1] [https://www.nytimes.com/2015/06/16/upshot/to-lose-weight- eat...](https://www.nytimes.com/2015/06/16/upshot/to-lose-weight-eating-less- is-far-more-important-than-exercising-more.html)

</comment><comment><USER3> > It's one of those weird circular things. If you want to be good at your job, > you need to be good at not doing your job too. Neglecting your life to focus > on your job is neglecting your career too.

my goodness yes. Having trouble with this precise thing right now. Thanks for this.

<comment><USER4> I only take new job inquiries seriously if I think they can improve my life away from work.

</comment></comment><comment><USER5> Scott Adam's a big believer in maximizing personal Energy as the #1 goal [http://blog.dilbert.com/post/102892940091/the-right- priority](http://blog.dilbert.com/post/102892940091/the-right-priority)

</comment><comment><USER6> Thanks! I bought a MTB a couple of days ago and is basically in the same situation as you. My personal wellness always gets lower priority compared to work and nowadays I'm always feeling tired. Hoping some cycling will change that! :)</comment>


Peter Thiel – American Democracy March 14, 2019 Lecture Video & Transcript - <AUTHOR> https://activationenergy.io/thiel/ <comment><AUTHOR> Credit to Garry Tan! Thanks for posting it on Twitter, thought I should share it on HN</comment>


Ask HN: Where does old hardware from public cloud providers go? - <AUTHOR> They must be decommissioning a ton of machines every day. Are they recycled? Destroyed? Are they resold? <comment><USER2> Google has a whole thing on this ([https://sustainability.google/projects/circular- economy/](https://sustainability.google/projects/circular-economy/)) that I find super fascinating.

</comment><comment><USER3> Many companies have annual sales of out dated hardware and other items which are made available for purchase to employees and to their freinds and family.

My general experience with this is that employees receive an email list of the items for sale and family and friends can purchase these items through said employees.

I have obtained a few office chairs, servers and video game systems through friends.</comment>


PostgreSQL partitioning explained - <AUTHOR> https://github.com/fiksu/partitioned/blob/master/PARTITIONING_EXPLAINED.txt

<comment><USER2> This is quite a good informal high-level explanation.

A thing I can't understand is why this isn't made automatic.

If there are 100 companies, you don't want to create 100 tables, then add 100 FOREIGN KEY constrains, and then again each time a new company appears, do you? Wouldn't a "syntactic sugar" like "PARTITION BY company_id" that does all this automatically be possible?

Just asking if there is a reason something like this is not implemented.

<comment><USER3> Why can't they just improve the indexing to do this automatically behind the scenes? Isn't the problem of finding things quickly in exchange for more work during a write exactly what indexing is supposed to solve?

<comment><USER4> Well, be specific about what indexing is: it layers a B-tree on top of what programmers would think of as an array structure. Indexes are not just something which is always faster than a sequential scan in exchange for slightly slower writes - there are important cases where the index is slower than just doing a sequential scan. Obviously you hope the query planner will avoid using an index in these cases, but the important thing to bear in mind is that indexes are not magic go-faster stripes :)

The problem happens when this B-tree is too large to fit into your working memory: swapping parts of the B-tree in and out of memory repeatedly to answer a query is very slow. The solution that partitioning offers is to allow you to split your indexes (and your data) so that, depending on your access-patterns you could see less swapping of that index.

<comment><USER3> Why can't the index store the branches of the b-tree as separate objects on disk instead of loading all branches into memory piece by piece?

</comment></comment></comment></comment><comment><USER5> Reading this reminded me a little of old works of philosophy. I found it tremendously educational, not just about postgres partitioning, but about databases in general. That degree of patience and thoroughness on the part of the teacher isn't something you see every day.

I wish more such conversations A) took place, and B) were transcribed and shared in a similar fashion.

<comment><AUTHOR> Well, that is about the best compliment I've every heard. thank you!

</comment></comment><comment><USER6> Somehow [http://www.postgresql.org/docs/current/static/ddl- inherit.ht...](http://www.postgresql.org/docs/current/static/ddl- inherit.html#DDL-INHERIT-CAVEATS) has always stopped me from using table inheritance; you don't get unique constraints over the table and its child tables, so for example you can't have an autoincrement primary key that works safely across them all. (Disclaimer: I haven't run into large enough tables yet that I'd need partitioning).

If this constraint was lifted, table inheritance would allow some really cool things, but currently I'm too conservative to use it in production.

<comment><USER7> Sequences are guaranteed to produce unique values. Just manually configure each child table to use the same sequence for the primary key column. Yes, uniqueness won't be enforced on the parent table, but the values will still be unique (unless you manually update them in your queries)

<comment><USER8> Until the sequence rolls over. Why design a schema with built in failure (an integer PK that overflows)?

<comment><USER7> Sequences won't roll over by default. nextval() fails if you're out of values. You can create a sequence which is allowed to roll over (by specifying cycle), but it's not the default.

Yes. It's not optimal and you can shoot yourself in the foot, but it works in its default configuration. I just gave a workable workaround for a problem, I wasn't saying it's the perfect solution (a working unique constraint on the parent table would be)

</comment></comment></comment></comment><comment><USER9> Great introduction to partitions. I did not know how exactly they are used. And now I think that I do. Thank you!

What I noticed (some improvement suggestions):

1) "the planner could tell (using knowledge from the check constraint) that employees_1 was the only table it needed to look at."

Well, I think it still checks the parent table, right?

2) You could mention that it is possible to create a trigger function which inserts new data automatically into the correct child table (as described in your final link, here: [http://www.postgresql.org/docs/9.3/static/ddl- partitioning.h...](http://www.postgresql.org/docs/9.3/static/ddl- partitioning.html)).

I think it would be cool if PostgreSQL could create these trigger functions automatically based on the check constraints...

I'm also wondering why foreign key constraints are not inherited?

<comment><AUTHOR> yes, the parent is also a target table -- but for partitioning you never put anything in the parent. I could probably talk about the troubles of putting data in the parent table.

and yes, a trigger function is possible -- and I could have mentioned it (although this was written as an explanation of partitioning using the gem I created). I should mention the alternatives and why they weren't employed.

</comment></comment><comment><USER10> Great intro to Pg partitioning. I understand partitions can improve performance of a single query, scanning less rows, but any of you guys have idea about throughput implications? Is partitioning better or worse when it comes to 1000 small queries/updates per second? Thanks!

</comment><comment><USER11> So in large web applications would the partitioned tables be distributed among multiple DB servers?

<comment><AUTHOR> you could -- that isn't "partitioning" in the way I discuss it in the linked article

when it is split across machines it is generally referred to as sharding: [http://en.wikipedia.org/wiki/Shard_(database_architecture)](http://en.wikipedia.org/wiki/Shard_\(database_architecture\))

<comment><USER11> I see, thanks for clearing that up!

</comment></comment></comment><comment><USER12> Can this be used to enforce a temporal limit on data? That is, if the check is for time to be within 24 hours of now will it purge older than 24 hours data as time progresses? My gut tells me no, but my gut is often wrong when it comes to databases.

<comment><AUTHOR> if I understand you correctly, this type of thing is implemented by partitioning by hour in a day and then having a janitor process drop tables that are older than 24 hours.

<comment><USER13> ALTER TABLE .. TRUNCATE PARTITION should be much faster than deleting as well. At least it is in MySQL.

</comment></comment></comment><comment><USER14> Could you use a check constraint of FALSE on the parent table to prevent rows from being erroneously inserted there instead of the actual partition tables?</comment>


Ask HN: What can I do in 2-3 months to secure a developer job? - <AUTHOR> I graduated from my university 3 weeks ago with a Computer Science degree. Unfortunately I never really spent too much time working on side projects or getting industry experience; most time was just spent working on my coursework.
I've sent out a lot of job applications and either I've only gotten rejections back or it's been radio silence. I've applied to about 10 companies, ranging from multinational companies to smaller startups.
At this point it's getting sort of stressful not finding work. Can anyone shed some light on what I should do to maybe land a job in 2-3 months? I would say I have a strong background in Computer Science and I can definitely program. Should I work on a portfolio or something like that?
Thanks! <comment><USER5> Some other ideas, besides the ones here:

\- A lot of developers never think in "normal" business as possible employers as developers. Maybe is a USA thing, but in my country the #1 is always the NO-STARTUP/NO-TECH company. Like for example, shops, service companies, etc. The small/medium are not that into recruit developers, so is likely you could get something just telling them you are a developer.

A lot of small & medium size business want/need a developer to automated a lot of (surely boring!) stuff. If you relatives have one of that kind of company, look at there. Maybe doing light sysadmin, some users support, make some scripts here and there, make the company website/app, etc.

This is valuable learning. And you can't get this at a startup. The idea is relieve the burden of not have a job and have the leisure to pursuit without pressure.

Is not a bat career start, if you are smart about it. Plus: Is not crowded!

\- Think seriously in what kind of software you truly want to make, that could be profitable. A start doing that now. I regret badly not make my own software early, for fear of get crush by large companies.

You will need a lot of time to get the skills and true knowledge to have a real product, so that is why is smart to start early and iterate for a while. Don't rush to market too early. Your first attempts will suck (likely) but time help a lot.

\- Focus in learn how be a great developer. Don't stop at that. Is your plan to have your own startup? Learn how make business too. Maybe do games? Learn about that. Etc..

\- Make exercise. Now is better than tomorrow.

</comment><comment><USER6> Honestly, filling in your skill gaps, or creating side projects is definitely a fun way to spend a few months. If you can afford it, enjoy that time.

However, if you want to get a job, you already have the skill set you need. All you need to do is apply to a ton more than 10 places in 3 weeks. You should be getting in contact with 15-20 potential employers each day. Keep a text file or spreadsheet with notes on who you contacted each day. You should be sending out enough emails that when someone replies, you'll have to look them up in your notes to find out who they are.

If you have a CS degree, and are a reasonably competent developer, there is definitely a job out there for you right now. It's just a matter of getting in contact with the people who may want to hire you.

<comment><AUTHOR> Thanks for the reply. Both of what you said (filling the skills gaps and applying to more places) is great advice. I think what I'm planning on doing is a combination of both e.g. apply to 10 places per week and also build on the side and potentially join a team.

</comment></comment><comment><USER7> First, don't feel bad. I've seen a lot of smart people have a lot of trouble finding entry level jobs lately. Luckily you are in an industry with ample opportunity.

Next, be proactive. Apply to 10 companies a day. Just think about the numbers, your conversion rate, and sending out resumes. You won't know your conversion rate until you apply to a ton of jobs. With the background you said and lack of any real experience, 95% of companies are going to want to pass because they want someone who knows framework/language X and can hit the ground running. However, that leaves 5% that are willing to train interns or even full-time hires with 0 experience. The problem is that you can't really tell which are which type of company so you've got to send resumes to everyone. After sending out 100 resumes take a step back, evaluate your approach and success rate, then get back to it.

While you're sending out resumes, spend a small portion of your time reaching out to anyone you know in the industry (including your friends from school who just got jobs) and ask them to take a look at your resume.

There's really no shortage of tech companies you can apply to, especially if you're willing to relocate. Where are you located now?

One more tip: My friend who had next to no experience got a job at Google as a contractor through a 3rd party company. He gets very solid pay and access to all of Google's lux facilities/food and he didn't even have to do a technical interview. I think they do this because if you suck they can basically fire you anytime.

Good Luck!

<comment><USER8> > My friend who had next to no experience got a job at Google as a contractor > through a 3rd party company. He gets very solid pay and access to all of > Google's lux facilities/food and he didn't even have to do a technical > interview. I think they do this because if you suck they can basically fire > you anytime.

Yup. Not a lot of people know this, but a significant percentage of people at Google are contractors (red badges). One of my friends is currently working there as a contracted web developer. He said that while working there is awesome, the turnover in the contractor ranks is very high and there is not a lot of support for newbies. It is pretty much a sink-or-swim environment. Outside of "codelabs", there isn't much documentation of their internal tools so my friend has to constantly read the source code to figure out how to use something.

</comment></comment><comment><USER9> Something worthwhile is matching the types of jobs you want in the future as well as now and creating a list of the experience or skills they want.

After that, look at what you are lacking or need to learn to fill in those gaps. Create a list of projects and just jump in. Make a blog/journal to track your development for your own sake as well as a good resource to point at. Often grads have lots of good solid skills.

Also worthwhile getting a friend to look at your CV. It might not be highlighting the right skills. Similarly, have a base template but ensure you tailor it to each company and job post (I'm sure you do that but is an easy mistake).

A nice tool (UK job posting only but good none the less) is [http://www.itjobswatch.co.uk/](http://www.itjobswatch.co.uk/) which lets you see what is trending and important in IT job posting. Have you got what is the new hotness?

Make sure you create a compelling cover letter for your CV. Place I work they just hired 3 new people, a bad or boring cover letter killed the chances of many amongst the hundreds of applicants.

Job hunting can be hard. I remember finishing my Masters, having a newborn to look after and no other income coming in and pressure on me to get a job. Took longer than I wanted but I got there and I am sure you will too!

<comment><AUTHOR> Thanks for the advice. I really like the idea of figuring out where my skill gaps are and then working to fill those.

</comment></comment><comment><USER10> In my experience, companies tend to look for what you have done rather than what you simply know from a degree. Couple of suggestions maybe -

\- To start with, identify any specific individual or team projects from your courses and highlight them on your CV, instead of just listing the classes.

\- Look for small coding projects - talk to friends, family and neighbors, see if they know someone who might need help. It could be something as simple as helping a local club or mom & pop store to set up their first web page - <USER2> from there.

\- While searching, on the side pick an area of technology and get familiar with it. For example - if you are not hands-on familiar with various databases: download MongoDB, see if you can configure it, get it running, and build a sample project with it. Then maybe download MySQL, get it running as well and figure out the difference. You could look into web servers, CMS, AWS, mobile apps, and all kind of things.

All these little bits you learnt and did will come in handy on your CV and, more so, if you do land an interview.

</comment><comment><USER11> Building a portfolio of projects that you care about seems like the best way to go.

Here's an old thread talking about side project ideas you may find useful:

[https://news.ycombinator.com/item?id=5234692](https://news.ycombinator.com/item?id=5234692)

<comment><AUTHOR> Thanks for the link. Definitely looks interesting.

So what would be the process here? I build up a portfolio of 2-3 projects, and then I link them on my resume. Would HR and recruiters bother to look through them? I guess I'm wondering what a company like Google would do considering they probably get 20,000 applications a week. Would their recruiter even take the time to click through my apps even though I have no industry experience?

<comment><USER12> Side projects have three uses, HR and recruiters only have one

\- projects give you actual real world development experience, possibly with others, they provide a genuine validation of your skills, and (if built with others) give you a network of peers / mentors to work with

\- HR and recruiters just filter out the obvious chaff so the person who is really going to hire you is not drowning.

Really do not expect HR to hire you, do not expect anyone to beat a path to your door because you built a better mousetrap - you must sell your mousetrap. (Which is considerably easier if you build really good mousetraps alongside well connected mousetrap builders )

The way to sell your services is to a) talk to people already working in the companies you want to be, b) Identify the companies you want to be hired by c) politely remind a) of b).

</comment></comment></comment><comment><USER13> It would be hard to land a dev job if you have nothing to suggest you can be productive on day one.

I'll suggest you start looking at other IT jobs where training is available. Tech support type jobs are a great way to break into the industry while getting your dev skills to a marketable level.

A 2-3 months goal is not a healthy way to approach it; you wouldn't ask "what can I do to be a competent doctor or lawyer in the next 2-3 months", the same goes for any profession. Software development isn't a magic bag of tricks, it is a skill set that you build up over time.

</comment><comment><USER14> A lot of people are going to say "side porejcts," which are great after you get your foot in the door...but your problem is you can't even find the door.

1\. Network. 2\. Show initiative, be proactive. 3\. Be persistent.

Find a few companies you want to work for and see if you can figure out how to take one of the hiring managers out for a coffee or whatever. Tell them you are interested in the company and want to learn about their team and the projects they work on.

If you don't get hired for that company/team/whatever you will likely get to nted in the right direction.

<comment><USER15> How does he show initiative? A couple of reasons people work on side projects are to gain experience and show initiative. While I'd agree that 1, 2, and 3 are good things....I would also say that side projects are a way to achieve #2.

<comment><USER14> Totally agree.

</comment></comment></comment><comment><USER16> All the advice so far is great. The only thing I'll add is that applying to 10 companies is not enough. I work with one of these coding bootcamps and what we've found is finding a job has less to do with technical expertise and more with persistence. The students who applied to the most places got the most offers sooner.

That said, this finding a job process is hard, and tech companies are notoriously bad at knowing how to bring on new junior-level people. Keep at it, and good luck to you!

</comment><comment><USER17> I looked at it as a numbers game. I think you should invest the most time in the next week just applying to more than 10 places. It's not glamorous, but if you apply to every single startup out there, you'll at least end up with some options.

for more information: [http://web.livefyre.com/jobs](http://web.livefyre.com/jobs)

</comment><comment><USER18> What programming languages do you know and like?

1.Figure out what language or community you are interested in and go to a Meetup? Usually there are people who are hiring.

2\. Find a side project and put it on github. This might take a little longer, but when somebody is going to ask you your experience, you say that you've completed your degree and you can point them to your github account.

Good luck!

</comment><comment><USER19> 1\. You didn't say where you're living right now. If you're still in the city where you went to university, you might want to try talking to your professors, explain your situation, and ask them if they have any projects you could work on for free. Even if they don't, they might have suggestions for projects that they'd be willing to advise you while you worked on the project (e.g. meet semi-regularly with you and offer feedback.)

This will be harder to pull off if your self-assessment is inaccurate :). Start with classes where you got As or really clicked with the prof, then work your way down. Whenever you talk to someone, ask if he or she knows other people who would be a good fit for your interests and who you could also talk to. If they mention someone you don't know personally, ask to be introduced by email. The usual job search/networking stuff. (As long as you're not obnoxious about it, this isn't pushy --- most people will be happy to send an email that will make it easier for you to talk to someone else but might forget to offer unless you bring it up specifically.)

2\. At the same time, think about what sort of work you want to do. Then start learning more about that area and try to become part of that "community." Are there relevant conferences you could go to? Do you have friends who know people in the field? (This is a rhetorical question; you probably do, even if you don't know it.) Ask to be introduced, and then ask those people how to get started in the field. Don't be pushy about it, and try to make it clear that you're not asking anyone for a job. AND don't ask people for jobs! You'll get a lot more out of these conversations if everyone views them as "conversations" and not "job interviews."

2b. If you don't know what sort of work you want to do, maybe do a version of those steps, but aimed at learning about lots of different aspects of development.

3\. I would strongly discourage you from spending 2-3 months working on your own on side-projects. It's too easy to fool yourself, and it's likely that you'll be back in the same place after 2-3 months spent unproductively. But, if you have friends in a similar position, and if working with a former professors isn't possible, it could be useful to work on a project with those friends. You're likely to learn more about "work" that way than working on your own, and you're more likely to get something interesting done in a manageable time frame.

</comment><comment><USER20> Hi there, I get this line of questioning a lot from new programmers in my local Ruby community so I wrote a little book about it getting your first programmer job. You can get it for free here: happybearsoftware.com/kickstart- your-developer-career

<comment><AUTHOR> Thanks for the link. I just downloaded the book and subscribed. Seems really interesting.

<comment><USER20> Hope you find it at least partly valuable. Please feel free to get in touch if you have any specific questions.

</comment></comment></comment><comment><USER21> A lot depends on your situation:

\- Do you live in an area with many opportunities?

\- Are you willing to relocate?

\- How well-regarded is your school / department?

\- How have you applied to the jobs you've applied to?

\- What kind of work are you particularly interested in and prepared for?

\- Does your school provide any placement help?

<comment><AUTHOR> \- I'm about 500 miles away from SF in southern california \- Definitely willing to relocate \- School is a top 25-30 I believe \- Applied using their online systems \- Hard to say what kind of work I'm interested in because I haven't really done too much outside of academia. I took a databases and web applications class and those were definitely enjoyable. \- Unfortunately not. It's a big university and individuals tend to get lost in the crowd

</comment></comment><comment><USER22> You _need_ to do side projects, and put them on github. As a hiring manager, I'm telling you that people who have a github account with side projects automatically move to the top of the pile.

</comment><comment><USER23> great suggestions already.. just a possible addition i'd suggest:

put together a public repo of your coursework, you could even guise it as side projects by personalising the work

surely your coursework had some practical andor mathematical algo exercises that can show your understanding of and interest in programming

</comment><comment><USER24> check your local city/county governments - governmentjobs.com - they got good benefits and I always see these type of jobs sitting for months. I don't think they give a damn if you have any side projects - just that you have CS degree.

</comment><comment><USER25> Go to a code school, you'll meet people and learn the basics. And it's super fun.

</comment><comment><USER12> I had a similar question from a taxi driver (CS grad!) in Greece last month and have been putting together an ever-expanding advice column. I will publish it soon but most of it is as follows

1\. The world is changing in some fundamental and unavoidable ways.

2\. This includes hiring. Previously the broadcast mentality was the only economically sane approach. Now not so much.

3\. Build a specialism 4\. Build a peer and mentor network 5\. Build a pipeline of "sales" 6\. Build a corporation mindset, not an employee. 7\. Build a financial cushion 8\. Build a professional approach

Specialism. Specialisms can be domain (banking, energy), technology (Java, lisp, mobile) At your stage in your career this is less important than broad development experience at a good software house.

Peers and mentors. The oft cited approach is to find a largeish OSS project in an area you care about (do not do games / graphics unless you will sell your grandma to get into the industry) - and contribute slowly, focusing on getting high quality code, with tests and documentation in atomic commits. Listen to the older wiser heads, and keep active on the IM channels. Slowly you will find people you connect with and do not be afraid to ask "smart questions"

Pipeline of sales - this is the bit you are specifically asking about bu it is part and parcel of the whole. A pipeline is work that is lined up to commence she the current gig ends. Even if you are a full time employee, you should be thinking like this. Firstly identify the twenty (yes there will be twenty) best software houses in the industry / specialism you care about. Let's say you love the MooC idea. Look at the folks offering Moocs - coursera, udacity, khan, and the folks offering services to those Moocs (can't remember right now but include rice university, Oxbridge, open university) About a days googling and reading journalism on the area will give you a brief overview. I bet there is even a datamonitor report out there on this.

Now you have a list of twenty companies that do software in MOOC area. Hit their websites, LinkedIn and Twitter feeds - find the developers and Dev leads in these companies. Find those that look interesting - yes interesting. This is mostly a matter of taste and fit so go with your gut here.

If they are in town offer to take them for a coffee saying "hi, I am a recent CS grad and looking to work in the MooC industry, would love to get your views / insight into the industry and workin practises in exchange for a coffee."

If they are not in town - exactly the same, only offer to send them enough bitcoins for a latte and do the chat over Skype whilst supping

At this point you are a mile ahead of the competition, but not home and dry. Always ask "if there are any opportunities going at their place of work, then do let me know". Don't push, don't mention you are desperate, and never ever ever lie. If you don't like Moocs find something you do care about - <USER3> vision monitoring of traffic ? Medical devices ?

Now actually approach those companies that have jobs advertised - and approach them through HR as normal, But mention when calling HR (always always call to confirm "they got my email") you had coffee with xxx.

You will stick out in people's memory here. And that's 90% of the game.

Extend your runway. This is hard - work stacking shelves or coding for local small businesses. Doing this in and around a jobhunt is never easy - but at minimum make every lunchtime count.

Assume you will be doing this for six months - it took me two years of upgrading jobs to go from redundant to 200k - and lunchtimes is the key

Keep working this - maybe a conference if you can afford it. And even after you get a job, keep looking around, making social contacts and OSS contacts.

This is something I call developer contact management - it's an interesting area.

Anyway

6\. Finance. Don't be a schmuck. Save, invest, clear credit cards. You can't do much of this, but read the motley fool, put aside 10% if you ever can and come back to this in three years

7\. Think like a one person corporation. What value do you offer to a business - what's your USP? If you had to do one thing that would transform your perception in any company it is oddly - documentation . Make auto generated docs like pythons sphinx your friend - writing about what you did _and why_ will always end up with you writing sentences like "this approach will save the company 5% on xxx yearly" \- and matplotlib makes useful graphs

Be professional Never lie (except in salary negotiations and only then about why you aren't telling them a number first. Read patio11's excellent essay on this) Always be looking to improve your work, not cut others down Find a good software house (oddly international Banks are now mostly global software houses), and or good mentors Focus on the best practises in OSS world - <USER4> are mostly world class - things like mandatory tests, mandatory code reviews and good developer comms.

Tl;dr

\- buy coffee for devs in a dozen of more companies in an area you care about. (Can't find a dozen companies - try harder, they are there) \- apply to them direct through usual channels, but keep back channels open. This is not sneaky - this is developer contact management. Be open. \- improve your professionalism - clean code, good tests, peer reviewed and communicated respectfully. Read uncle bobs _Clean Coder_ (with R on end)

\- keep on doing it. At some point in five years you will find people respecting your opinion and track record. Then look up "imposter syndrome" and stop worrying

\- have fun.

</comment><comment><USER26> "Unfortunately I never really spent too much time working on side projects"

This tells me as someone who graduated cs 10+ years ago that you have no _real_ interest in computers and took it just for the job/to be trendy. Personally I would not hire you.

That said he's some actual advice. Now that your a recent graduate a company can hire you full time and get subsidized a large chunk of your pay. This will only last 6m-1yr, so make it count (do a good job, meet people which is usually how you get jobs at other companies, work overtime (boo!) and at least pretend to care about whatever time wasting project they put you on).

Theres a lot more advice I can give, but meh.

<comment><USER26> oh yeah and ignore all the stupid advice to "start a blog" and "journal your development" etc garbage from the social-hacker-marketing-idiots that infect HN like a plague spreading their opinionated FUD everywhere.

Unless your a skilled and seasoned dev this can only hurt you. Seriously, I speak from experience as someone who has looked through many resumes and portfolios.

For better or worse you chose the CS path, not a diploma and not a code.joke.org. You are on the right path, where as most jobs didn't ask for a CS degree before most do now. The landscape is LITTERED with hacks and posers who all seem to have amazing resumes full of amazing synergy, but it's all poop. People will tell you your dumb for taking 4-5 years to get a legit degree and make no mistake, the people saying that have none of their own...

<comment><USER27> > Unless your a skilled and seasoned dev this can only hurt you. Seriously, I > speak from experience as someone who has looked through many resumes and > portfolios.

Given your experience, could you explain the reasons for it only being damaging?</comment></comment></comment>


Moonpig vulnerability - <AUTHOR> http://ifc0nfig.com/moonpig-vulnerability/

<comment><USER2> They also make it very difficult to delete your account. Rather than just have a link on the site, you have to contact customer services and they say they'll respond in 24-48 hours.

Not to mention the ways they try to hide you removing your card details. If you want to remove your card details, do the following:

 _The easiest way to do this would be to go to the My Account page then click on the 'Add Moonpig Prepay Credit' link, click on the Buy link and your saved card details will be shown onscreen. Click on the 'Remove Card' option._</comment>


Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store - <AUTHOR> https://databeta.wordpress.com/2018/03/09/anna-a-crazy-fast-super-scalable-flexibly-consistent-kvs/ <comment><USER2> The link to bloom is broken: [http://bloom-lang.net](http://bloom-lang.net)

This is a result of what appears to be a chain of ideas, papers and prototypes started more than eight years ago (see [http://boom.cs.berkeley.edu](http://boom.cs.berkeley.edu) \- maybe even earlier, hard to tell since most older URLs are gone). I'm amazed that people are able to remain funded working on something with a very theoretical and long-term payoff, and incredibly thankful at the same time for the entities supporting this! Wish success to the team on releasing Bedrock.

<comment><USER3> I wonder what the performance would be like if instead of running on ruby they ran it on a scalable virtual machine with near-first-class actors (not truly first class, but the engine is optimized to handle them, and the constructs are extremely important in the standard library) like BEAM.

<comment><USER4> While Bloom was indeed coded in ruby, they use C++ for Anna:

> We get this rich consistency in Anna with a very clean codebase, by porting > design patterns of monotone lattice composition from Bloom to C++.

In the paper:

> The Anna actor and client proxy are implemented entirely in C++. The > codebase—including the lattice library, all the consistency levels, the > server code, and client proxy code— amounts to about 2000 lines of C++ on > top of commonly-used libraries including ZeroMQ and Google Protocol Buffers.

<comment><USER4> Curious to see such effective code, I looked for the source code, but found nothing.

The more approaching code base I found is [https://github.com/ucbrise/LatticeFlow](https://github.com/ucbrise/LatticeFlow).

Is the Anna source repository public ?

</comment></comment></comment></comment><comment><USER5> Being faster (in throughput) than Redis doesn't seem that difficult, Redis is mostly single core. But with that restriction comes the magical constraint that you can run Lua on that single core without having to think about race conditions or concurrency. For many complicated systems this is an incredible property and can provide incredible power. Yes, you may eventually outgrow Redis, but if you do then you are entering pretty crazy territory where you will likely need something custom anyways.

All that to say, that particular comparison feels a bit Apples and Oranges to me.

<comment><USER2> The comparison with Redis is on page 10, and (obviously) takes into account the number of threads. It is a bit presumptuous to post a comment like this without minimal effort, please don't.

Regarding embedded processing, it's very easy to embed whatever single- threaded language you'd like into C: Lua, Javascript or something new like Gravity. This is orthogonal to the storage / network architecture.

A better argument would be on the extra operations and data structures that Redis offers, not being a simple key-value store.

<comment><USER5> The comparison to Redis is in the 4th paragraph of the linked article. "The paper includes numbers showing it beating Redis by over 10x on a single AWS instance"

I'm sorry I didn't go read the original paper, but I thought reading the article qualified me to comment. Sorry dad.

I think you also misunderstood my point about Lua. Embedded Lua in Redis is so powerful BECAUSE it is single threaded, not because it is just snazzy to have an embedded language. That with the primitives Redis provides allows you to build your own domain specific data structures with their own custom semantics that just aren't possible in other systems without rolling your own. And you can do it simply.

That Anna is faster is great, but it comes with its own set of constraints, I don't think I would be wrong in saying that includes you not having exclusive access to the data while you are in an embedded script running on the store.

<comment><USER6> > "The paper includes numbers showing it beating Redis by over 10x on a single > AWS instance"

Right single core against single core,the best scenario for Redis, and Anna's an order of magnitude faster. I'm not sure what you don't find impressive about this.

<comment><USER5> Ahem. Single AWS instance does not equal single core.

See my earlier note, it is a bit apples to oranges. Redis is not optimized for throughput on a machine, it is optimized for throughput on a single core. And that property allows lots of interesting things due not having any parallelism.

They are different tools and they have different performance characteristics. My only point was that comparing against Redis is somewhat misguided due to those differing goals.

</comment></comment></comment></comment></comment><comment><USER7> This link claims to be 700x faster than something called Masstree, which itself claims to do about 3 million requests per second with 16 cores. I'm not sure I buy 131 Million request per second per core.

> it was up to 700x faster than Masstree, up to 800x Intel's "lock-free" TBB > hash table. In fairness, those systems provide linearizable consistency and > Anna does not. But Anna was still up to 126x faster than a "hogwild"-style > completely inconsistent C++ hashtable due to cache locality for private > state, while providing quite attractive coordination-free consistency.

</comment><comment><USER8> I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be.

What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases.

I don't need another key value store...

<comment><USER9> Cassandra is JVM GC issues

Redis is in-memory only

Cloud Storage - Not sure, how we can use it outside of cloud vendors

RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced?

There is also badger but most of these only offer low-level operation.

Sorry, most of my developers are unable to consume them just like Redis.

<comment><USER10> Agree w/ all these points (forgot "S3 has provider lock-in" though others replicate the API). I do use Cassandra for most use cases and I don't hit GC things, but I understand the concerns that come with the JVM (no, haven't subbed Scylla in yet). One that I haven't tried but want to hear the downsides of is [https://github.com/pingcap/tikv](https://github.com/pingcap/tikv) (not the DB built on top, but just that one for KV). A nice published list of all cons of all database systems would be ideal.

<comment><USER11> From what I could tell, its tikv is virtually useless without the rest of the tidb (the MySQL layer that sits on top written in go) to be useful.

TiKV has no replication / sharding built in, that is actually handled by a Placement Driver (PD)

From the docs: > TiKV is a component in the TiDB project, you must build and run it with TiDB and PD together.

</comment></comment></comment></comment><comment><USER12> So reading through this announcement, it seems the key to ANNA's speed is:

local cache (in the form of the actor's mailbox) + background gossip

The usual restrictions (and increases in latency) still apply when you want to make sure that something's actually written (quoruming) after you've written it, from what I can tell.

Can someone explain to me why this is a step forward for the field -- I haven't yet read all the papers they linked to (including their ANNA papers), but this doens't seem to be one of those times where a bunch of disparate papers are combined into creating something truly groundbreaking?

I feel like I must be missing the point

<comment><USER2> I don't think that's correct. The real key to its performance seems to be the usage of _distributed lattices_ as data structures, and the ability to perform compile-time checks that guarantee the data will be eventually consistent, both of which allow code to be completely lock-free. This comes from the CALM paper cited in the article - lots of reading to do!

<comment><USER13> I thought so before too, but after reading the paper again, I came to the same conclusion as the parent. The usage of distributed lattices is key, but it only works because it allows them to reduce messaging cost and gossip at background intervals. As far as I can tell, this means that you can receive a successfully written response, have that machine die , and all data within the last multicast period is lost. Therefore, it isn't suitable as a datastore, and the benchmarks are mostly worthless with the exception of Redis.

<comment><USER12> Super duper late, but I still haven't had time to read any of the papers (there are like 4 if I really want to get anywhere close to understanding their spin on gossip + the lattice thing) -- glad the discussion is still interesting though.

I'm starting to think that the quorum strategy is something like a theoretical lower bound -- at least until someone brilliant figures out a way past it (or technology shifts in some gigantic way or something).

</comment></comment></comment></comment><comment><USER14> Has there been a public release of the source for this?

</comment><comment><USER15> Looks interesting, especially about coordination free consistency and lock free structures !!

</comment><comment><USER16> Looks exciting but very light on details.

<comment><USER17> There's a lot more detail in the linked paper.

[http://db.cs.berkeley.edu/jmh/papers/anna_ieee18.pdf](http://db.cs.berkeley.edu/jmh/papers/anna_ieee18.pdf)

</comment></comment><comment><USER18> Claims in the blog post (orders of magnitude faster than the current state of the art systems, universal linear universal scalability from threads to many nodes, dimissing Dean's rule of redesign after x10 scale) seem overblown to me.

What have they really built: a purely in-memory KV store that doesn't support synchronous secondary writes for durability. So, any comparisons with ACID KV stores, either disk based (Cassandra, Mongo) or in-memory, are not apples-to- apples comparisons from the beginning. What could be production applications of such system, other than cache?

On their benchmarks: they don't really compare with state of the art.

Selection of competitors in the single-server, multi-core benchmark doesn't include systems like [https://github.com/fastio/pedis](https://github.com/fastio/pedis). Also, they still use 100 millisecond granularity of gossip (within a single server!), while for all other compared systems corresponding metric could be evaluated as nearly 0 by construction, that gives Anna a huge edge.

In multi-node benchmark, they claim 10x over Cassandra. ScyllaDB ([https://www.scylladb.com/](https://www.scylladb.com/)) claims the same, while being ACID and linearizable, unlike Anna. Also, Anna achieves stronger consistency levels by holding off reads, that kills latency, given 100 millisecond gossip granularity. If it applies only to their multi-key consistency (Read Committed/Uncommitted) it's probably OK, because I suppose that there is no magic bullet that allows to preserve super low latencies and providing similar consistency in Scylla either. But if Anna needs to hold off reads for any of their claimed single-key consistency levels (all of which are weaker than linearizable), that's worse than Scylla. The authors of the paper didn't detail the algorithm for each consistency level.

Seems like the authors don't benchmark multi-node scalability of Anna on any consistency levels except the weakest, simple eventual consistency. It would be interesting to see if Anna scales as well on stronger consistency levels.

To me, the main outcome of this paper is another confirmation that shared- nothing, thread-per-core, message passing designs are beneficial in the modern computing environment. This is not new, however, see H-Store, Scylla/Seastar, Tarantool ([https://github.com/tarantool/tarantool](https://github.com/tarantool/tarantool)), Aeron ([https://github.com/real-logic/aeron](https://github.com/real- logic/aeron)), Tempesta ([https://github.com/tempesta- tech/tempesta](https://github.com/tempesta-tech/tempesta)), etc.

Novelty is the framework that generalizes thread/node scalability, different consistency levels reusing the same codebase, and having just a single knob - gossip granularity. Practical applications are limited. Certain techniques are probably going to be cherry-picked by systems such as Redis Cluster and In- Memory Data Grids.

<comment><USER19> Can you point me to where Scylla claims to be ACID or linearizable? As far as I know there is no Paxos implementation yet, not that Cassandra's LWT implementation is anything to write home about.

<comment><USER18> Anna paper itself says that Cassandra and Scylla are Linearizable per-key. Yes, obviously Scylla is not ACID, sorry for my loose usage of this term. I was referring to durability, i. e. "D" from ACID.

</comment></comment></comment><comment><USER20> Can Anna's approach improve current solutions to the problem of managing secondary indexes in a partitioned KV store while preserving consistency?

<comment><USER21> I don't think this problem exists outside of the realms of linearizable or serializable consistency, which this system doesn't provide.

</comment></comment><comment> dis-sys > Totally ordered request processing requires waiting for global consensus at > each step, and thus fundamentally limits the throughput of each replica-set.

really? such global consensus increases latency (network round trip plus fsync write), with a fully batched and pipelined concurrent design, when CPU cycles are being saturated in those benchmarks, why throughput is fundamentally limited by such increased latency?

<comment><USER21> Let's say we have a primitive system with total ordering and single static coordinator per each replica-set. And we want to update a global counter from every node. Now since counter is global it lives on a single replica-set and every node has to communicate with this one coordinator. Be it 4 nodes or 4000, still one coordinator. If we drop total ordering and use something like gossip protocol to communicate between nodes, we can merge this counter everywhere in the system as it propagates, eliminating all unnecessary communications and distributing communications in the network. So, yeah, global consensus fundamentally limits both throughput and latency.

</comment></comment><comment><USER22> If anyone is interested in looking at the benchmarks or implementation: [https://github.com/cw75/tiered-storage](https://github.com/cw75/tiered- storage) is from the main developer and it looks similar to Anna.

</comment><comment><USER23> Rant:

Why so much focus on Key Value stores? That's the easy part of the problem.

I would like to know more about the interesting ones: secondary indexes, range scans, performance on mixed workloads, robustness, operational complexity.

<comment><USER24> > Why so much focus on Key Value stores? That's the easy part of the problem.

There aren't enough good/fast/reliable ones

<comment><USER25> Does Redis not count?

<comment><USER13> Redis is an impressive piece of engineering, but it performs best as an in- memory kv-store on a single core. Its distributed capabilities target a different problem than other distributed kv-stores attempt to solve. Redis Cluster focuses on reasonable functionality for an in-memory store. However, Redis Cluster is neither highly available, nor consistent. There are multiple modes that can cause catastrophic data loss, so Redis Cluster works best in situations where losing data isn't a big deal. For it's intended use case, nothing else comes close to offering the same functionality, performance, reliability, and ease of use.

</comment></comment></comment></comment><comment><USER26> There is already a database product called Bedrock, using SQLite.

[http://bedrockdb.com/](http://bedrockdb.com/)

</comment><comment><USER20> Can anyone with a good understanding of latest db research share their thoughts on the paper?

</comment><comment><USER27> Thats not a very lucky name IMHO, given that there already exists a data storage related software called Hanna ([https://www.sap.com/products/hana.html](https://www.sap.com/products/hana.html))

</comment><comment><USER28> no link ?

<comment><USER29> [http://db.cs.berkeley.edu/jmh/papers/anna_ieee18.pdf](http://db.cs.berkeley.edu/jmh/papers/anna_ieee18.pdf)

</comment></comment><comment><USER30> Why this fad with naming apps as persons? Found this: [http://fortune.com/2014/12/22/startup-names- human/](http://fortune.com/2014/12/22/startup-names-human/)

<comment><USER13> Anna is a hummingbird. Known for its fastest relative speed.

<comment><USER30> Why this fad with naming birds as persons?</comment></comment></comment>


What's happening to Valleywag? - <AUTHOR> http://www.kumar.vc/whats-happening-valleywag/

<comment> tyler-b I think the simple answer is that Valleywag's primary domain is now valleywag.gawker.com instead of valleywag.com. Here's a screenshot from the same tool that shows the new primary domain trending upwards: [http://i.imgur.com/dLpcEZP.png](http://i.imgur.com/dLpcEZP.png)

<comment><USER2> This is correct. In fact, this is the expected behavior for a website which has changed domains; it means that people who used to visit the old domain are now correctly visiting the new one.

The submission receiving 8 points is an instance of people on HN upvoting based solely on the headline itself, and that's unfortunate.

</comment></comment><comment><USER3> I wouldn't say this drop in unique linking domains says a whole lot about the fall of valleywag - what domains dropped off, and at what rate? Is it just because of a certain story thread acquiring huge waves of links across other gawker networks or other blogs responding to topics and linking in the course of discussion?

My guess would be that gawker pushed some of their partners/syndicates toward other more profitable Gawker properties once VW got rolling and it was clear how much traffic/revenue they could expect from the current valleywag style & audience.

Were other gawker properties launched at the time of this drop?

</comment><comment><USER4> The present Valleywag is a mere shadow of the original one. My gut says they have extremely strict editorial restrictions from their bosses at Gawker.com.

</comment><comment><USER5> Very insightful article.

No, wait: THE OPPOSITE. WTF is this? Have a take or don't bother.

<comment><AUTHOR> I was curious what caused the drop.</comment></comment>