The Verge Hack, Explained - <AUTHOR> https://blog.theabacus.io/the-verge-hack-explained-7942f63a3017 <comment><USER2> "Exactly how the largest protocol-level hack of a cryptocurrency in recent memory could preceed said cryptocurrency increasing in price and then announcing a partnership with the most trafficked porn-site on the internet is a question I'm forced to leave open-ended; my personal pet-theory is that it has something to do with the fact that the world makes no sense and human beings are all completely out of their goddamn minds."

<comment><USER3> Please don't quote portions of the article with no added commentary or insight.

We're all capable of reading and recognizing well-written portions of the posted content.

<comment><USER4> This was a particularly interesting bit though and I was hoping some people might explain/comment more about it.

It seems that getting your crypto-currency hacked gives publicity and boosts value! See also: the Experian hack.

<comment><USER5> Clearly any plans with pornhub.com would have been made prehack. As to the value of the cryptocommodity (I refuse to call them currencies)... it's like the author said.

</comment></comment></comment></comment><comment><USER6> This hack results from a basic design error: for difficulty adjustment to work properly, the window of recent blocks it considers must be relatively immune to time stamp manipulation. In other words, the allowed timestamp drift must be much smaller than the adjustment time window. Bitcoin allows a 2 hour drift, but has an adjustment window of 2 weeks.

In verge's case, both are identical, at 2 hours. This is an open invitation to difficulty adjustment abuse.

<comment><USER7> > In verge's case, both are identical, at 2 hours.

Naive question - is verge adjustment window defined in block time (as opposed to "human" time)? because

> Bitcoin allows a 2 hour drift, but has an adjustment window of 2 weeks.

The adjustment window is technically 2016 blocks, which is about two weeks in human time, but can in theory vary greatly if mining capacity fluctuates.

<comment><USER6> I went looking through the source code and found that the adjustment window is indeed defined in block time as PastBlocksMax = 12 while nProofOfWorkTargetSpacing = 150 (half a minute for each of 5 different pow algorithms), giving an expected adjustment window of 12 * 150 = 1800 seconds.

Apparently, the adjustment window, which needs to be way longer than the allowed timestamp drift, is in fact way shorter...

</comment></comment></comment><comment><USER8> The gate metaphors in the article are so messed up they distracted me from the actual story.

      >> Gates Only Work When They're Raised

Well normally gates swing open and shut, maybe you mean some kind of portcullis, like in a castle? You have to raise those to open them.

      >>  a gate that's far too strong to break
      >>  through and too high to climb over —
      >> this hack gets past it by finding a way
      >> to lower it so close to the ground that
      >> it can be stepped over

So now gates rise out of the ground? I don't think the author has seen an actual gate.

<comment><USER9> You're holding it wrong. Think: [https://en.m.wikipedia.org/wiki/Floodgate](https://en.m.wikipedia.org/wiki/Floodgate)

</comment></comment><comment><USER10> One can always invent blockchain buzzwords like "Dark Gravity Wave" but even copying code requires understanding of algorithms.

<comment><USER11> I snorted when I first heard the term. An old co-worker used to call 'universal differential', for weeks. Yeah that was IF statement.

</comment></comment><comment><USER12> This hack underscores the need for formal modeling systems like TLA+ or Coq. Essentially, you define _successor state axioms_ , where:

1\. start with a seed state.

2\. non-deterministically choose the next step (e.g. submit a tx)

3\. ask the system to prove the inverse of an invariant you want to check (e.g. hash power cannot fall below X% of total network power.)

Decentralized protocols have bugs; trust only what you can prove.

</comment><comment><USER13> Not sure who said it first, but cryptocurrency truly is the Internet-wide bug bounty.

<comment><USER14> What does that mean? How is it different than Google being an Internet-wide bug bounty?

<comment><USER15> it means, you wont get magical internet money when you hack google

</comment></comment></comment><comment><USER16> "Reality, however, often has a way of stubbornly refusing to conform to the axioms of free market economics."

Great article.

</comment><comment><USER17> > _Indeed, major cryptocurrencies like Bitcoin and Ethereum have maintained > their security quite well — better, arguably, than any other digital asset > /payment system in history_

Am I missing something? And here I thought having to hard fork your currency goes in the "not very secure" bin.

</comment><comment><USER5> Can someone who touts the "but it's decentralized!!" aspects of these systems- please to explain to me how an algorithm that adjusts mining difficulty is not a centralized and possibly destructive feature in control of a handful of people?

</comment><comment><USER18> I'm very curious about the price increase.

The only thing that comes to mind how something like that would be possible is if the news trends generated by this were somehow driving algorithmic trading .. either that or people investing in something they're just hearing of for the first time.. But even these possibilities don't seem rational.

It just doesn't seem to make much sense... unless the whole thing was being driven and pushed by some larger unknown actor with a lot of resources .. (state government, goldman sachs, etc.)

<comment><USER5> But what lunatic reads of the hack - then invests!?

<comment><USER19> I'm guessing an algorithmic one, which invests based on "buzz" and/or price trends leading to positive feedback cycles.

</comment></comment></comment><comment><USER20> >>>In both cases, this hack presents a strong argument for tending towards sticking to things proven to work and to be wary of overcomplicating things and thereby introducing unnecessary risks when people's financial assets are involved. Which, I suppose, means two points for team Bitcoin.

What? The same bitcoin that is betting everything on an untested and incredibly complex second-layer protocol called "lightning network" instead of just scaling with larger blocks?

<comment><USER21> > instead of just scaling with larger blocks

Blocks will need to be ~5TB to match Visa transaction volume. Some lightning network advocates are concerned about the effect that a size increase necessary to support even a fraction of this volume would have on mining centralization.

Other concerns are listed here [https://en.bitcoin.it/wiki/Block_size_limit_controversy](https://en.bitcoin.it/wiki/Block_size_limit_controversy) and here [https://en.bitcoin.it/wiki/Scalability_FAQ#General_Block_Siz...](https://en.bitcoin.it/wiki/Scalability_FAQ#General_Block_Size_Increase_Theory)

<comment><USER22> > Blocks will need to be ~5TB to match Visa transaction volume.

No, it's on the order of 3 gigabytes. Visa claims they can handle up to 25k Tx/s.

226 B / Tx. 600 sec (10 min) per block.

(25000 Tx / s)(226 B / Tx)(600 s / block) = 3.39 • 10^9 B / block

Visa normally handled around 2k Tx/s, which would only require a few hundred megabytes per block.

<comment><USER23> Re Visa TPS:

*Source: 50,000 transactions per second, as of July 15, 2017

[[https://ripple.com/xrp/](https://ripple.com/xrp/)]

</comment></comment></comment></comment><comment><USER11> So basically this time warp hack is allowing eval() on the chain history... What could go wrong?

</comment><comment><USER24> _This is because, in decentralized networks that obstinately refuse to grant any special authority to third parties, accurately enforcing time synchronization is no simple matter._

That depends on what you're going for. The concept of time is actually a local phenomenon, as relativity shows. To compare whether events A and B happened first, you must first trace some path from A and B to a third party C (maybe C = A or B if you really want) and then ask C which came first.

We had to deal with this when building our DLT. See this: [https://intercoin.org/technology.pdf](https://intercoin.org/technology.pdf)

<comment><USER25> Relativistic effects are irrelevant at the level of precision required for running a system distributed across the surface of Earth, what you are describing is just due to a lack of properly synchronized clocks. Place an atomic clock at each node and there is no need for any C deciding the ordering of A and B. Alternatively you may also be able to use logical clocks like a Lamport clock or a vector clock to establish an ordering in a distributed system in absence of properly synchronized clocks but that depends on the use case.

<comment><USER24> No. Relativity is actually a special case if this general principle.

For a large and complex network, events A and B which are very far apart cannot be compared in any canonical way. The larger the network the less they can be compared.

However, A can come before B _relative to_ some reference point C.

That's enough to achieve most things. The point is that you _shouldn't_ have a global and continuous sense of time in distributed systems. Leslie Lamport has developed many things including vector clock over the years to address this.

Our PTN is just the latest in a long line of examples.

When you say that you have "no need" for C once you have synchronized clocks, not only are you ignoring relativity but you are also ignoring the byzantine generals problem. If Han Solo can benefit from claiming he shot first, why would I trust his clock? If you're implementing a video game or cryptocurrency in a byzantine resilient way, how exactly would you have a continuous global sense of time? Go ahead the floor is yours.

<comment><USER25> You are mixing two things up - figuring out the time an event happened and trusting a timestamp that someone attached to an event. Within a single inertial reference frame there is a canonical way to order events, namely by the time coordinate in that reference frame which is the same everywhere. And for all practical purposes of building a distributed system on Earth, Earth is an internal reference frame. Ensuring that nobody cheats when assigning time stamps to events is an entirely different problem. As said, hand an atomic clock to everyone and you can know the order of any two events, relativity plays no role. Making sure that nobody can or will cheat will require some more effort, but again relativity plays no role.

<comment><USER24> You didn't explain how a distributed _byzantine_ system can synchronize a global time that's continuous.

I am saying that you won't be able to — local time is continuous and you should use that for comparisons.

It's nowhere near as straightforward as you think to make global time work. You will find that "making sure nobody can or will cheat" will require exactly the kind of effort that will make the concept of a global timestamp meaningless. Also, why do you need a global timestamp in the first place? Upon further reflection you will probably realize that all your concepts of time actually require exactly what I said: two paths to the same local reference point, where a comparison is made.

Your ideas about global time are an illusion based on simplifying assumptions that are not true.

<comment><USER25> I only object your claim that relativity - as in special or general theory of relativity - has any bearing on the design of distributed systems on Earth. In particular, you can order events occurring at different places if you have properly synchronized clocks. The fact that a common solution is to use logical clocks like Lamport clocks is because it is easier to do so than to maintain a network of precisely synchronized clocks. But that does not mean it can not be done with synchronized clocks, see for example Google's Spanner [1] database.

I never addressed what else is necessary to make this work in the presence of Byzantine faults. But I am also pretty sure that just sending information about events from A and B to C and asking C about the ordering is not sufficient, after all C may be faulty and give a wrong answer, change its mind after every query, or just not respond at all. Admittedly you did not provide many details, so I may very well totally misunderstand what you had in mind.

But, just to repeat, I am not really addressing the requirements of a Byzantine fault tolerant system, I am just saying that the special or general theory of relativity is not relevant for distributed systems on Earth.

[1] [https://en.wikipedia.org/wiki/Spanner_(database)](https://en.wikipedia.org/wiki/Spanner_\(database\))

<comment><USER24> I think you misunderstood my point.

I said that relativity is a special case of this principle that time is a local phenomenon.

<comment><USER25> Which is not really true, time is frame depended, not location depended. Just take the twin paradox, when they start and later meet again, they are in the same place but different amounts of time have passed for each of them. The closest thing I can think of is that the ordering of two events is not ambiguous if they are timelike separated while for spacelike separated events you can find different reference frames which disagree on the order of the events.

<comment><USER24> What you said doesn't disprove what I said :)

<comment><USER25> Given that we have never actually defined what »time is a local phenomenon« is supposed to mean, that may or may not be true. But given that relativity puts time and space on a more equal footing than Newtonian mechanics, where you now can turn time into space and space into time, I remain pretty doubtful that something that can be summarized as »time is a local phenomenon« is likely to be interesting and correct because it seems to point in the direction of not treating time and space in a unified manner.

<comment><USER24> Listen, space is a local phenomenon too. It is just about correlations.

A lot of the nonlocal effects in De Broglie Bohm theory become classical once you consider close-by objects and neighborhoods. They are just super correlated.

Why? We don't know. But with so many correlations, you can compare way more easily between them.

[https://www.wired.com/2016/01/quantum-links-in-time-and- spac...](https://www.wired.com/2016/01/quantum-links-in-time-and-space-may- form-the-universes-foundation)

<comment><USER25> This still does not answer what you mean with »time is a local phenomenon«. What feature of time is local and in which sense? And I am even more skeptical about »space is a local phenomenon«, I mean »local« means »close in space« and therefore it seems you are saying something recursive, i.e. space has some property within itself which doesn't really make sense, at least to me.

So unless you can define or at least explain in more detail what you mean with »time (or space) is a local phenomenon«, we won't really get anywhere.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER26> Is it possible to add (Cryptocurrency) after 'The Verge' in the title please? I thought it referred to the news outlet.

<comment><USER27> I was not aware that The Verge was still moving forward after Joshua Topolsky left, and their show "On The Verge" was cancelled.

</comment></comment><comment><USER28> PoW coins outside Bitcoin will never work. This will continue to happen over & over again to every smaller PoW chain

<comment><USER24> PoW is leader based consensus. It's slow and can't handle millions of simultaneous payments. Why do new distributed ledger technologies still prefer to use it, when there are other, far faster consensus algorithms out there?

<comment><USER29> Probably because they're not trying to actually solve a problem. They're just on the hype train.

More seriously, if the name of the game is efficiency then a distributed ledger isn't the answer.

<comment><USER30> > More seriously, if the name of the game is efficiency then a distributed > ledger isn't the answer.

And if you're only looking for immutability, then time-stamping is already good enough (and available since the late 90s or so). I suppose you can use a Merkle tree and then proceed to put "Blockchain" in your marketing materials without technically lying.</comment></comment></comment></comment>


Ask HN: Is there a car trunk rental service? - <AUTHOR> I'd like to use something like blablacar but I don't need to travel, I just need to transport some of my belongings that won't fit on a plane.<p>Maybe something like this would also need hubs to transfer luggage between cars and check them for anything suspicious. <comment><USER2> [https://www.luggageforward.com/](https://www.luggageforward.com/)</comment>


Reverse Engineering Might and Magic III Compression - <AUTHOR> http://blog.rewolf.pl/blog/?p=1202

<comment><USER2> Heh, this is a blast from the past. Back in the day Chris Lampton and I wrote a character editor for Might and Magic 3, and I was surprised to discover not long ago that it is still in use by MM3 fans. This is a program that was written in C++ in the early 90's if I recall correctly. [http://shrines.rpgclassics.com/pc/mm3/downloads.shtml](http://shrines.rpgclassics.com/pc/mm3/downloads.shtml)

</comment><comment><USER3> I did a similar thing to a Mischief decompression algorithm.

[http://m1el.github.io/mischief/part-2.html](http://m1el.github.io/mischief/part-2.html)

</comment><comment><USER4> Click on the link to the blog caused Super Anti-spyware to say a threat was detected. I recommend scanning if you already clicked it.

<comment><USER5> Avast warned be as well.

<comment><USER6> I can assure you, that those are false positives. I'm already trying to contact those AV companies to resolve this issue.

<comment><USER6> FYI, Avast and Emsisoft removed their detection.

</comment></comment></comment></comment><comment><USER7> Interesting that there is still an active mod scene. Good memories from the Might and Magic series, especially the later Windows versions 6-8 and the early Heros 2-3 off-spin series. I applied some mods and messed around with modding tool for MM6-8.</comment>


Koinify Raises $1M for Smart Corporation Crowdfunding Platform - <AUTHOR> http://www.coindesk.com/koinify-1-million-smart-corporation-crowdfunding/

<comment><USER2> I've been following the development and progress of you guys for the past month (after I saw Tom speak at the Google event) - glad to see the funding has come in. I became very intrigued after that Bitcoin conference, and thank you guys to helping me learn about the industry.

Wish you all best of luck on journey.

</comment><comment><USER3> Link to the homepage: [http://koinify.com/](http://koinify.com/)

</comment><comment><USER4> Fake upvotes and accounts are a really bad idea here, not to mention comments that look like sockpuppetry. Please don't do any of this. Just let your work speak for itself.

<comment><AUTHOR> Nothing fake, just sharing the link with our community networks and directing them to check it out here.

Hopefully our work does speak for itself.

<comment><USER4> > Nothing fake, just sharing the link with our community networks

Your "community networks" must be remarkably cohesive to think of such similar account names as these:

[https://news.ycombinator.com/user?id=lkjfdlkj](https://news.ycombinator.com/user?id=lkjfdlkj)

[https://news.ycombinator.com/user?id=kjlasdlkjaskldj](https://news.ycombinator.com/user?id=kjlasdlkjaskldj)

[https://news.ycombinator.com/user?id=jkhkjshkjhf](https://news.ycombinator.com/user?id=jkhkjshkjhf)

[https://news.ycombinator.com/user?id=lkjdlfkjslkdjf](https://news.ycombinator.com/user?id=lkjdlfkjslkdjf)

[https://news.ycombinator.com/user?id=dsklfjlksjflksj](https://news.ycombinator.com/user?id=dsklfjlksjflksj)

[https://news.ycombinator.com/user?id=lkjlfkj](https://news.ycombinator.com/user?id=lkjlfkj)

[https://news.ycombinator.com/user?id=kjdlkjslk](https://news.ycombinator.com/user?id=kjdlkjslk)

[https://news.ycombinator.com/user?id=ljdhhskj](https://news.ycombinator.com/user?id=ljdhhskj)

... and to register them all within an hour.

From our point of view, and that of our anti-gaming software, this is abusive. It's one of the surest ways to get accounts and sites banned on HN. Please don't.

</comment></comment></comment><comment><USER5> Awesome news!

</comment><comment><USER6> FINALLY, this is what crypto needs

</comment><comment><USER7> Awesome!

<comment><AUTHOR> Why thank you!</comment></comment>


A rare condition causes a doctor to experience other people's sensations - <AUTHOR> http://www.psmag.com/health-and-behavior/is-mirror-touch-synesthesia-a-superpower-or-a-curse <comment><USER2> Important excerpt (to de-sensetionalize the title):

 _[..] But to be clear, Salinas cannot read minds. He doesn't know whether Josh felt the impact of the reflex hammer, and the tingling in his kneecap says more about his own extraordinary nervous system than it does about that of his patient._

<comment><USER3> Even that's misleading - as so many of the examples depend on visual cues, the phenomenon appears to be a purely mental one and has nothing whatsoever to do with one's nervous system.

<comment><USER4> Mental phenomenon do have something to do with your nervous system...?

</comment></comment></comment><comment><USER5> One of my favorite podcasts (really any NPR podcast) did part of an episode on this: [http://www.npr.org/programs/invisibilia/382451600/entangleme...](http://www.npr.org/programs/invisibilia/382451600/entanglement)

The woman in the podcast is debilitated by it. It also includes her family, specifically her daughters, views on it (she doesn't believe it is real) but also hints she might be developing it as well.

</comment><comment><USER6> I mis-read the title as "A race condition ..." Too much multi-threaded coding.

<comment><USER7> I came here to say exactly the same, haha. Now I wonder, how does the brain handle concurrency.

<comment><USER8> It might actually be the reason why you (and I) misread it. One of the theories of lexical access (how you retrieve the semantic content in your memory through speech/text) is that it starts with the first few sounds/characters and then just picks the one that gets accessed first, which is based on how often you use that word, etc.

</comment></comment></comment><comment><USER9> In spite of the misleading, click-bait title, I think it is interesting:

 _Mirror-touch synesthetes struggle with the constant intrusion of others' feelings. At a symposium on mirror-touch synesthesia last year in London, a woman named Fiona Torrance of Liverpool described how she had once seen one man punch another. She promptly passed out in a car. Her boyfriend at the time found her unconscious and took her to the hospital. "I felt the punch," she explained. As a child, she once saw a man kill an otter with a spade on television. She was inconsolable for a month, feeling as if she'd killed the otter herself. To this day, she takes medication to control the sensory onslaught, and she does not own a television. A recent episode of the NPR program Invisibilia profiled another woman with the condition who has essentially become a shut-in._

</comment><comment><USER10> there are so many kinds of synesthesia. having diverse forms of synesthesia myself what I find interesting is that within some forms only direct confrontation or sudden discomfort results in the awareness of them

I m extremely accurate in detecting smells , while certain smells will causes extreme pain . It was only in adulthood that I was confronted to explaining why smelling cooked onions cause so much pain in me.

as a result I realized that I actually smell geographically within a about 1 meter cylindrical field around my head . where as onions are in an area inside my head that hurts though it is the same for, calendrical time and numbers.

We often just question our perception of the world when it is immediately compared to other perceptions, or results in discomfort.

I wonder how diverse human perception of the world can actually be without us knowing. If humanity was blind but one person perceived Eyesight but had never learned to communicate his ability to "foresee things", run without hitting objects , this human wouldn't be able to reflect about it because it wasn't integrated in his way of communication.

Maybe for this human seeing a beautiful sunset would remain a dumb feeling, rather than a realization of what it actually is that made them feel that way, since his communication and reflection of the world was not focused on this sense.

</comment><comment><USER11> > "heightened empathic ability." ... condition is often more debilitating than > it is empowering.

I experience this in terms of what other people are feeling emotionally. Its awful.

[https://news.ycombinator.com/item?id=9195553](https://news.ycombinator.com/item?id=9195553)

</comment><comment><USER12> I really want to hear what happens when a mirror-touch synesthete watches porn.

<comment><USER13> "Oh, she's faking it!"

</comment></comment><comment><USER14> Ironically, this seems like a very hard condition to empathize with.

<comment><USER15> I've had this experience from time to time. I don't have this "on" all the time. This has been useful for stuff like martial arts.

It's possible to empathize with, if you have sufficient skill in concentration and visualization. You can empathize with anything.

<comment><USER14> From what I understand, we all have mirror neurons.

[https://en.wikipedia.org/wiki/Mirror_neuron](https://en.wikipedia.org/wiki/Mirror_neuron)

So that's probably contributing to empathy in all of us.

</comment></comment></comment><comment><USER16> Related : Here is a news about boy who can tell what you are thinking. My friend has met him and told this is true. [http://www.thehindu.com/todays- paper/tp-national/tp-kerala/a...](http://www.thehindu.com/todays-paper/tp- national/tp-kerala/abled-truly-with-a-difference/article4308624.ece)

</comment><comment><USER17> Reminds me of this: [http://liveboldandbloom.com/08/self-improvement/empath- trait...](http://liveboldandbloom.com/08/self-improvement/empath-traits-of- highly-sensitive-person)

"Highly Sensitive Persons" and "Empaths".

</comment><comment><USER18> if the title were accurate this was exactly what Carl Pilkington said would be exciting technology on the Ricky Gervais podcast / audiobook, teeheehee.

</comment><comment><USER3> Sounds like a case of early-life Altruizine exposure.

</comment><comment><USER19> Click bait title, still interesting.</comment>


"Brown and Cornell Are Second Tier" - <AUTHOR> http://chronicle.com/blogs/percolator/brown-and-cornell-are-second-tier/27565 <comment><USER2> There are several untrue points in this article:

MIT is most certainly not a second tier school for finance/consulting recruiting. It is easily a top tier school that is considered along with Harvard/Princeton/Yale (notwithstanding the inanity of having such a thing as top tier schools in the first place). The raw proportions of grads that end up in those destinations is probably less than other top schools because entrepreneurship is glamorous now and engineers usually end up in tech.

Brown might not be up there with Harvard in terms of recruitment, but it is still a "target" school for top finance/consulting firms. Goldman Sachs is one of the 3 largest employers of recent Brown grads (though no doubt some of the reported positions were for so-called "back office" positions). The other two are Teach for America and Google.

The fact that interviewers like to hire people with similar interests (incl. sports and fraternities) is sadly true from what people have told me.

One important point to keep in mind is that at the large finance/consulting firms, each school's applicants are screened and interviewed by the alumni from that school. So an engineer applying from MIT may very well get screened by someone from MIT with an engineering background.

</comment><comment><USER3> This is overstated. It's true that top banks/consulting firms couldn't care less about a Rutgers grad, and that some elite hedge funds make distinctions between Harvard/Wharton and Yale, but it's overstating it to say that they aren't interested in MIT grads. Tons of MIT grads go into consulting/finance (I think they make up the top two target professions). Also, the consulting firms are very interested in people who went to non-target undergraduate schools, but went to a top graduate school (PhD, MBA, MD). Also, McKinsey/BCG/Bain are entirely willing to look at Dartmouth/Duke/etc grads.

And note we're talking about the top firms here, in the same sense that Harvard/Yale/Princeton are the top schools. Outside of Goldman Sachs/Morgan Stanley/JP Morgan, hedge funds and private equity funds, and McKinsey, BCG, and Bain in consulting, the name snobbery drops of precipitiously. Lots of people from top state schools end up working at Deloitte, Accenture, etc.

Also, the stuff about grades versus traipsing around in Costa Rica is outmoded. My brother is an analyst at one of GS/MS/JPM, and his summer class is, to be glib, has a huge representation of 3.9+ asian math/science nerds from Harvard/Yale/Princeton. It's not 2007 anymore, hiring at investment banks is way down, and the days of 3.7+ with a great personality skating past the 3.9+ math nerd are over. To be sure personality is still a requirement, but quantitative skills are increasingly valued.

Also, people arguing that this is an inefficient way to hire are ignoring a basic fact of the business model: these hires aren't keepers. These firms are built on up-or-out models and forced attrition. Most analysts are hired for a two year contract and not asked to stay on board. In essence, the initial hiring decision is just the first round--the two years on the job as an analyst is where the top people are screened out. Indeed, this process never stops. Even at the partner level, which a very select few ever make it to, there is substantial forced attrition. GS adds 100+ partners every other year, but maintains the partnership at a pretty constant 500-ish people. You do the math on that.

By hiring only from Harvard/Yale/Princeton/etc, they are just making the bet that statistically good people are more likely to be found in that pool than broader pools.

<comment><USER4> Is the analyst/consultant career path a solid way to get an upper-middle class life? I don't know anything about that sector but I imagine the average GS/MS/JPM finance-type peson makes 300-700k? Or am I way off?

Would you be limited to living and working the East Coast or getting huge pay cut elsewhere?

<comment><USER3> Yes, though how much you make depends heavily on the level at which you flame out. Investment banking generally is much more lucrative than consulting (and within investment banking you have to distinguish banking from trading, with the latter having higher headroom). Starting salary is similar, but progression is much steeper. Analysts make about $110k with bonus the first year. After 10-12 years total (remember, there is huge attrition at every step) you make MD (managing director). MD compensation at a top bank can vary wildly, especially in trading where it's based on performance. For a front- office position (people in HR/IT/etc often hold the title but don't make the money) I think $500k-$5m is a reasonable range, though very often the highest paid person is not a partner or the CEO, but some MD trader who brought in $1 billion or something.

Most people never make MD. Some people leave voluntarily to go to hedge funds or private equity funds. Compensation at these funds vary by size, but is broadly similar to banking at larger firms, less at middle market firms, and possibly more at mega firms. Most people don't end up in HF/PE. They go to a top business school (almost guaranteed admissions) then enter upper corporate management. These folks don't make banking money, but 100k starting out after MBA is probably the minimum outside of non-profits, and you'd be surprised how many "assistant controller" types make $300k+. CFO's can make a few million a year at a large company

Consulting pays a lot less, though I know less about it. Average partner compensation at McKinsey is rumored to be $500k to a couple of million. Most, of course, don't make partner. They end up in MBA programs, and either take a second bite at the apple, try to lateral to finance, or compete for the aforementioned management positions. All else being equal, financial experience is really valuable, though (the M&A department at a place like Google is very well compensated).

Wow that got long, haha. This was my pitch to my brother when I told him to go into banking.

Re: east versus west. Consultants can work anywhere, and indeed spend their lives traveling. For banking, you want to start in NY, but good funds exist in every major city for after. Corporate jobs are everywhere. Again, you'd be surprised how many assistant controller types make in Dallas or Atlanta.

</comment></comment></comment><comment><USER5> This is what happens in situations where you can't measure an employee's marginal productivity with any reasonable degree of accuracy. You hire based on pedigree and promote based on popularity. Excessive focus on these things implies that no particular measurable skill is actually required to do the job. It's not really "we want to hire people who can use their skills to create the most value for us." It's more of "we control a large source of income, and we're going to share access to it with the people we deem worthy." It's the old way.

</comment><comment><USER6> This hilariously demonstrates what I've long thought about consulting companies: they don't provide "smarts as a service" as they claim (otherwise they would just take the MIT engineering grad, or simply compare based on GRE scores, etc...), instead they provide "corporate ladder infiltration as a service". They _want_ the kind of people with an impressive diploma, with the right hand shake, who exude athletic acumen and _look_ smart and accomplished. Their job would be to impress middle management which would then permit the consultants to do what is likely a trivial job.

It's sad to see so many people with STEM degrees going down the consulting route. I can't obviously understand why anyone would want to go down that route. If someone is truly intelligent and are purely after the money itself, quantitative finance, or working at Google and sticking around for promotions and stock grants (I'm not even talking about, e.g., Founder's Awards -- which I'd argue aren't out of reach for someone truly "best of the best") would pay much better. If (like most people) they want to actually derive some kind of satisfaction from their work, it becomes even more puzzling.

Carl Sagan mentioned in one of his books that he felt the way science is taught in high schools is largely mechanical and robotic and that might draw the wrong kinds of people into science. Without extra-curricular help (youth science/math camps for some, or in my case dad being an maths professor and mom being a software engineer), many creative minds tend to shy away from those fields for that reason. They presume that these fields are _remain_ like that at university level: I've often tried my best to help-out younger hackers who decided not to pursue a CS education thinking they were "bad at math" (based on their experience with high school maths).

As a result by the time of high school seniors apply to college, STEM majors end up attracting many individuals that _like_ mechanical and robotic work ("plug and chug") in pursuit of external awards and social status. Once they graduate, however, they know they can't contribute meaningfully to their field with that attitude; management consulting seems like a good "refuge" for those types. The lesson isn't "these firms are being silly in their recruitment practice", the lesson is "buyer beware" when it comes to being these firms' customers.

<comment><USER2> I think some of your points are a little off the mark. A bit about my perspective: I used to be (and still am) interested in business (vaguely defined), and did lots of research and talking to people and I used to see consulting as the best option for my career after college. But now I'm an engineer at a startup.

1) They do take MIT engineering grads (see my own comment)

2) Consulting pay is actually not very good. We're talking $80k and not much bonus. Google pays more (STEM roles). Finance pays even more. STEM people don't go into consulting over finance for the money. The exception might be liberal arts majors for whom consulting does pay the most out of all their options. The reasons they go into consulting...

3) The reason people (like me in the past) want to do consulting is because it's a "career supercharger". If you get into McKinsey/Bain/Boston you have not only that pedigree on your resume and near guaranteed entrance into a top 10 business school, but you also get a very broad set of experience working with (and connecting with) CEOs/executives and working in multiple markets and with multiple business models and analytical models. You get to work in many projects and with many different teams. Consulting has a multiplicative effect on your career. 2 years in consulting has the equivalent effect on your career as, say, 4 years in a biz dev role at a Fortune 500. A lot of people who know they are interested in "business" but don't know what they specifically want to do in business also go to consulting to figure that out. Finally, for _any_ business position you apply to, nobody ever questions whether your consulting background is _relevant_ experience. Consulting basically keeps all business doors open. The same does not apply for, say, finance.

4)Whether consulting is value-adding is a mixed bag. Sometimes it can really help a company. Oftentimes consulting is just used politically to cover an executive's ass and is used to tell the other executives what that executive already knows.

<comment><USER7> >If you get into McKinsey/Bain/Boston...

Fun fact: If you are a college dropout and apply to these places, they will at best politely rebuff you. However, if they need you, and you run your _own_ consultancy, they have no problems being your customer in in order to get a job done. How you handle the awkward inevitable conversation at lunch about your lack of degree is up to you, though personally I always try to have fun with it.

<comment><AUTHOR> How do you get them to contract out to your consultancy?

<comment><USER8> Same as always: have something they need. I've worked for a top-three consulting firm (only because they bought the consulting firm I worked for at the time).

Surprise: their expertise is Excel and PowerPoint (mostly PowerPoint). These are not the ideal tools for all projects. Much better to just buy what you need when you need it (and mark it up 400%).

</comment></comment></comment></comment></comment><comment><USER9> I'm thankful this isn't true in Engineering (for the most part). I go to UPenn so I see snobbery like this all the time, but from my experience my friends from 'lesser' (not my assessment) schools are all given a fair shot at engineering companies of any size. I definitely see a ton of bullshit like that described in this article from large financial companies like Goldman Sachs, etc. I have friends who go to Penn and will get an interview, but then when the interviewer finds out they're not in Wharton in particular they lose all interest without even giving the kid a chance. It's crazy, so much emphasis is put on the words on your degree rather than your abilities. It's one thing to use it to filter resumes, but it's crazy when my friends go into an interview for which the interviewer has already allotted time and are not given a fair shot.

<comment><USER10> Engineering has always been a put up or shut up kind of field. I've seen people with very little education who are some of the most respected programmers in their field and I've seen people who barely graduated from State U making 175K leading huge departments. It's kind of hard to do that with say, investment banking. How is an undergrad supposed to prove they are an amazing investment banker? What kind of portfolio do you assemble for that?

Part of the issue here is not just the name on the degree, but that they want people that are homogenous and from that Harvard/Yale milieu so they can chat up other bankers who also graduated from Ivy League schools.

</comment></comment><comment><USER11> Linkbait article aside, in the actual paper[0] Lauren does not name the firms, and draws these conclusions based on 120 verbal interviews. Not exactly scientific, even for sociology. To be fair, her abstract isn't as hyperbolic as the above article (albeit obvious).

>> Employers sought candidates who were not only competent but also culturally similar to themselves in terms of leisure pursuits, experiences, and self- presentation styles. Concerns about shared culture were highly salient to employers and often outweighed concerns about absolute productivity.

No way! An interviewee from MIT (...heavily math/science/engineering bias) has a harder time getting an elite professional services gig? Next I'll find out NASA prefers MIT grads to Harvard. People hire/like people like themselves.

Deciding on Harvard over MIT (or vice versa) is both an education and cultural choice. That cultural matters is no surprise. And neither is that people hire with similar backgrounds to themselves.

And, not for nothing, guess where Lauren went? [1]

[0] : <http://www.asanet.org/journals/ASR/Dec12ASRFeature.pdf>

[1] : [http://www.kellogg.northwestern.edu/faculty/directory/rivera...](http://www.kellogg.northwestern.edu/faculty/directory/rivera_lauren.aspx#vita)

<comment><USER12> > And, not for nothing, guess where Lauren went?

Oh, let's not resort to ad hominem arguments. The paper stands on its own.

</comment></comment><comment><USER13> The fact that top firms have blunt, inefficient hiring algorithms is in theory a good thing for equality/competition/dynamism in the long run. Imagine they accurately assessed every graduate in the job market and always harvested the best ones.

<comment><USER14> I see what you're saying, but hasn't this been going on for decades? Wouldn't that indicate that it is actually efficient for the scale of their business?

I think one of the key points in the article is when the recruiter suggests that basically they are free-riding off of the admissions committees of the top schools. That makes a lot more sense than blanket elitism, which would be costly and inefficient in the long term.

<comment><USER13> And for decades established firms have been tumbling from their perches and replaced by new ones. I don't have stats to back that up though so I could be wrong.

But even just the idea of free-riding college admission committees - if that's their idea of capturing the best candidates, then they're turning a blind eye to students who underachieve until they actually reach higher education (or who can't afford to enter institutions like Harvard anyway). Clearly they leave a gap which allows the possibility of weaker firms grabbing the best talent.

</comment></comment></comment><comment><USER15> All college admissions and personal branding are determined by age 16 at the latest. Records considered prior to the seminal test given at that age include: early high school grades (ages 14 through 16), extracurriculars and sports (ages 3 through 16), and middle school placement tracking (ages 10 through 13). These are all factors either directly or one step removed in the process.

So the question has to be asked: Are these brands as childish as what formed them?

This "elite school" concept could probably be considered a "simulacrum" in the Baudrillard sense. Let us stipulate: working for Goldman isn't the only way to make money; _it is_ for the people who are branded by these schools. Would your parents or friends be happy to hear you're starting a startup that anybody without that branding could start? I think not. Your destiny then rests on increasing your liquidity by working the same 100 hours a week (doing nothing, mind you, because Wall Street _produces nothing_ ) for people who are also members of your brand. Then you get to live your ideal life in your Manhattan flat: meet your trophy wife: with a weekend house in East Hampton: re-live all the mistakes your parents made: propagate and create dysfunctional children...

The corollary then asks: Does the brand define one's life as their Geworfenheit? With such fixed determinism from such a young age, were you ever truly free? Is your quest for freedom what these employers are truly capitalizing on?

</comment><comment><USER16> The title is linkbait, but I'm not sure what about this is really news. Yes, degrees so-called "elite" schools like Harvard are essentially an expensive form of signalling.

Whether or not that signalling is accurate is a topic that can be (and has been!) debated ad nauseum, but the observation that people in the financial, etc. industries[1] _do_ treat those degrees as a signal is nothing new or surprising.

[1] Really, anything except tech, where this trend is absent (or arguably reversed - ie, respect for dropouts).

</comment><comment><USER17> No one care about Dartmouth? Puh-lease. Hank Paulson, long-time CEO of GS and Treasury Secretary, was a Dartmouth grad, and I met several Dartmouth grads during my short time there.

</comment><comment><USER18> Then down the line the smart people from those firms can disrupt the existing players at their own game or a new game when the current-elite get lazy or can't innovate. That's how the market's supposed to work.

<comment><USER3> You're mistaken about the game that is being played.

</comment></comment><comment><USER19> Sounds like Moneyball to me. If true, these firms are just taking the "No no every gets fired for using IBM" route and overspending, which isn't actually bad for the companies if they've got the money for it, as graduates from these schools with "really awesome" extracurriculars should be pretty decent filters.

</comment><comment><USER20> In computer science, this doesn't really apply. I mean, hiring decent people is hard enough without restricting ourselves to certain top tier schools. Sure, a company like Google might have a bias for MIT or CMU, but they'll take you if you are from UW, Cornell, or Brown, as long as you can pass the interview!

<comment><USER21> I went to the University of Michigan for my computer science degree. Google recruits pretty heavily there (Larry Page's alma mater after all). I had lots of friends that went to Michigan State for their degrees. The Google recruiter showed them a list of jobs they could apply for. When one of them asked about other jobs that were on their website, the recruiter said they didn't usually consider people from Michigan State for those jobs.

Now, I'm not saying Google doesn't hire anyone from MSU, I know several engineers at Google from there, but it can be a pretty huge disadvantage to go there.

After college, I worked with a bunch of people that went to Michigan State, and we went to a Michigan career fair to recruit for our company. They were shocked by the number of big software companies recruiting at Michigan vs. MSU. Microsoft and Google might be big enough to recruit anywhere, but places like Apple, Amazon, Hulu, Expedia, and dozens of others never made it to MSU.

<comment><USER20> There are high tier state schools and there are other lesser tier state schools (usually anything "XXX State", but not always!). But the number of high tier schools is around 30 in the US, so its not a very exclusive club, but once you come from outside that group, life can get very hard to get into a big company. But then, life doesn't completely suck, you can still get a job, especially if its at a local company that already has many employees from that school.

</comment></comment></comment><comment><USER22> I might be completely crazy or have had biased experiences, but the content and tone of this article are the opposite of what I've observed. Stanford, Dartmouth, MIT, Cornell, and Michigan (to name just a few of the schools cited) all enjoy robust recruiting efforts from top companies as far as I've been able to tell.

Additionally, based on conversations I've had with friends at some of the firms that were (presumably) polled, the desire to cut down on active recruiting at schools is usually due to lack of resources rather than deep prejudice on the part of people doing the hiring. If you only need to hire 3 people, it's simpler to just recruit at a few places where you've been successful in the past and call it a day -- it doesn't matter where you find good people, just that you got some in the end.

<comment><USER23> It depends what companies are recruiting for. Tech companies recruiting software engineers have a much different crucible than a top flight consulting company (deloitte, mckinsey etc.)

<comment><USER24> McKinsey was aggressively recruiting PhDs in physics and engineering (to do business consulting) from Cornell when I was there. They claimed to be cutting edge by going after such people instead of MBAs.

</comment></comment></comment><comment><USER25> It is true that Brown and Cornell are second tier. Cornell isn't even truly private. Part of Cornell is actually a public school (most people don't know this).

I disagree that nobody cares about MIT though. MIT is definitely top tier.

<comment><USER26> > _Part of Cornell is actually a public school (most people don't know this)._

It's not so relevant when it comes to technical subjects, though (which I guess is what this article is about), because the "public part" of Cornell is the agriculture/ecology/industrial-relations schools...

For CS/engineering, Cornell is very, very, good (and a big plus on a resume). Probably not as good overall or as useful on one's resume as MIT/Harvard/Yale, but those three are pretty much in their own category for U.S. universities.

</comment></comment><comment><USER27> Most people in a position of hiring want to hire someone "like me."

</comment><comment><USER28> It's not just Goldman Sachs. Hollywood and big media in general have elitist hiring. And the presidential race is always a farce.

      1992: Yale/S&B vs Yale
      1996: Yale vs Arizona (!)
      2000: Harvard vs Yale/Harvard/S&B
      2004: Yale/S&B vs Yale/Harvard/S&B
      2008: Harvard vs Annapolis (!)
      2012: Harvard vs Harvard

...and it goes back further with S&B even making more appearances.

<comment><USER9> What is S&B?

<comment><USER29> Skull and Bones. (<http://en.wikipedia.org/wiki/Skull_and_Bones>)

</comment></comment></comment><comment><USER30> I know a couple of twins named Winklevoss that would be perfect for their firm.

</comment><comment><USER24> This is what happens when the people doing the hiring don't know enough to really evaluate the candidates, so they rely on the "brand" (school) and punt on measuring the person's actual ability.

</comment><comment><USER31> If true (and there's probably more than a little fire behind that smoke), then don't work at those places. Supply and demand works both ways.

</comment><comment><USER32> From the article

 _the gatekeepers at our nation's most prestigious firms are pathetically shallow, outrageously parochial, and insufferably snobbish._

<comment><USER33> That reminds me of my university search. Looking for an engineering program, visited a campus (top 5 engineering school) and they asked where else I was considering. I mentioned GA Tech (had lived in GA for part of high school, a number of friends intended to go there), and Notre Dame (sister's intended school, had visited and liked it). The response, completely straight, "Well, I _guess_ those are good schools." It's incredibly easy to get people to dismiss an entire institution from consideration when their representatives are so crass.

</comment></comment><comment><USER34> Love this kind of elitism. Means easy money for those of us who have the patience or imagination to look past the name of a school.</comment>


Not Everyone Can Program: As Seen in Pixar's Ratatouille - <AUTHOR> https://medium.com/@Grayfox/not-everyone-can-program-as-seen-in-pixar-s-ratatouille-e74147ffbeb5

<comment><USER2> Anyone can cook. It may be the case that most people cannot cook to the standards we would expect of a proffessional chef, every non-disable adult (and most disabled adults) can, with little instruction, cook and edible and non-disgusting meal, and we generally consider this skill to be something that all independent adults should have.

In this same sense, everyone can, with training, program. It may be the case that not everyone can program at a professional quality, but they can still make simple programs to do the specific tasks.

<comment><USER3> Note that you can just learn how to make about a dozen dishes to a high standard and that will serve you for life, but learning how to write a dozen algorithms in one language won't get you through CS 101.

</comment></comment><comment><USER4> Given how many people are either trying to learn to program, or want to learn to program, I'm not sure what the point of this is yet. If we were inundated with people who were attempting to program, but can't—then this seems like a more meaningful concern. But at this point there's a lot of people on the outside trying to get in. Let's work on getting them in, and then we can set about weeding out the people who don't belong. Any weeding we do now will also keep out people who want to belong and should.

</comment><comment><USER5> Anyone can blog

<comment><USER6> Actually, I don't think anyone can blog well [You need to be able to write well, to express your ideas well and so on.]. The article says that everyone can start learning how to cook or to program by practicing but this doesn't mean everyone can do it well. This is why not everyone can program. The author emphasises on the fact that not everyone can become a good programmer. This is true for almost everything: some people are just more gifted than others. For instance, everyone can learn how to play music, play sports, market a product, learn about marine biology, sciences... but this doesn't mean they will be good at it.

<comment><USER5> yup, you completely missed my point. good job.</comment></comment></comment>


Databricks open-sources Delta Lake to make data lakes more reliable - <AUTHOR> https://techcrunch.com/2019/04/24/databricks-open-sources-delta-lake-to-make-data-lakes-more-reliable/ <comment><USER2> There's a lot of confusion around data lakes. One source of confusion is that "data lake" versus "data warehouse" is often presented as a choice, where you can have either:

1\. A data lake, where all data is stored in its native format (CSV, JSON, ...), in an object store (S3, GCS, ...), with the schema defined on read (Hive, Presto, ...).

2\. A data warehouse, where all the data is organized in a highly structured tables (star schema) in a commercial database (Snowflake, Redshift, ...).

This is a false choice! Modern data warehouses, particularly Snowflake and BigQuery, are fully capable of storing semi-structured data.

Furthermore, you do not need to curate your data into a star schema before loading it. The ideal way to set up a modern data warehouse is to establish a "staging" schema that matches the source, and then transform that data into a star schema or data marts using SQL. In this scenario, your "data lake" and "data warehouse" are just two different schemas within the same database.

There are still some scenarios where it makes sense to build a data lake in addition to a data warehouse, primarily future-proofing. I wrote a blog post where I tried to outline these scenarios: [https://fivetran.com/blog/when-to- adopt-a-data-lake](https://fivetran.com/blog/when-to-adopt-a-data-lake)

<comment><USER3> Has anyone written about privacy implications of data lakes and data warehouses? The Extract in ETL is usually supposed to filter out private data, but if instead all of the raw native data is dumped into a data lake, what ensures that data is handled with the same care as the individual systems that normally handle the data? What stops some random business analyst from running individual or aggregated queries that would be contractually or legally forbidden?

<comment><USER4> The solution is to divide your Data Lake into different zones with access control, so that user can only access what they're allowed to. That said, it's a lot of work to do this properly, so it's often neglected.

</comment></comment></comment><comment><USER5> We (data curation lab at Univ of Toronto) are doing research in data lake discovery problems. One of the problems we are looking at is how to efficiently discover joinable and unionable tables. For example, find all the rental listings from various sources to create a master list (union); or find tables such as rental listings and school districts that can be used to augment each other (join). The technical challenges in finding joinable and unionable tables in data lakes involve the following: (1) the data schema is often inconsistent and poorly managed, so we can't simply rely on that schema; and (2) the scale of data lakes can be in the order of hundreds of thousands of tables, making a content based search algorithm expensive. We came up with some solutions that are based on data sketches with several published papers [1,2,3]. The python library "datasketch" was a byproduct if these work.

Many challenges remain though, and we would like to explore some of the more pertinent ones. In fact, we are conducting a survey to understand the current state of data lakes in industry and the challenges experienced. If you're interested in learning more, see what we came up with here: [https://www.surveymonkey.com/r/R7MYXSJ](https://www.surveymonkey.com/r/R7MYXSJ) \- would love to see what the HN community thinks about the current state of data lakes.

[1] [http://www.vldb.org/pvldb/vol9/p1185-zhu.pdf](http://www.vldb.org/pvldb/vol9/p1185-zhu.pdf) [2] [http://www.vldb.org/pvldb/vol11/p813-nargesian.pdf](http://www.vldb.org/pvldb/vol11/p813-nargesian.pdf) [3] [http://www.cs.toronto.edu/~ekzhu/papers/josie.pdf](http://www.cs.toronto.edu/~ekzhu/papers/josie.pdf)

</comment><comment><USER6> I don't really understand the concept of Data Lake, and wikipedia isn't helping much... is it just a buzzword for a collection of data stores?

<comment><USER7> [https://martinfowler.com/bliki/DataLake.html](https://martinfowler.com/bliki/DataLake.html)

<comment><USER8> Awesome link!

The key line for me:

"The data lake stores _raw_ data, in whatever form the data source provides."

The emphasis on"raw" was his, not mine.

</comment></comment></comment><comment><USER9> Other efforts in improving the parquet datasets on cloud storage world:

[https://github.com/apache/incubator- iceberg](https://github.com/apache/incubator-iceberg)

[https://github.com/apache/incubator- hudi](https://github.com/apache/incubator-hudi)

Happy to see Delta go open source.

</comment><comment><USER10> They tried to keep it closed and sell it as a premium service, but looks like they need help from the open source community to make the product better. Great to see. Databricks has its roots in open source (the founder created Spark) and it's great that they're still making a lot of open source code rather than making everything private.

</comment><comment><USER11> What are the other alternatives for data lakes that can be used (both open source and close)?

<comment><USER12> Apache Iceberg is probably the closest product to what databricks is open sourcing, but none of these products are everything that's needed for datalake management.

What these products do is make it as easy to use decoupled storage and compute as your analytics system as it would be to use a fully managed analytics DBMS system.

<comment><USER13> Yes, when I heard about Delta I thought the same. Would love to see a comparison between Delta and Iceberg. I wonder if Ryan Blue is on HN.

</comment></comment></comment><comment><USER14> I appreciate the thought TechCrunch put into the image representing ACID- compliant data lakes.

</comment><comment><USER4> What I don't like about these ACID storage layers is they reduce compatibility between different query engine. For example, Spark cannot read Hive ACID tables natively and Hive cannot read Spark Delta tables either. Then there's other tools such as Presto or Drill which can read neither.

When you use an ACID storage layer, you're kinda locked into one solution for both ETL and query, which is not nice.

</comment><comment><USER15> Recently I was interested to learn more on Data Lakes, how to design and maintain them.

There is a lot of information in articles, blogs, but I prefer books as a solid source of structured and aggregated information.

Surprisingly, I found just a single proper book on the topic: [https://www.amazon.com/Enterprise-Big-Data-Lake- Delivering/d...](https://www.amazon.com/Enterprise-Big-Data-Lake- Delivering/dp/1491931558)

</comment><comment><USER16> What's the difference between a Delta Lake and Change Data Capture? Seems like in both cases you're creating a type 2 dimension against a source table.

<comment><USER17> It is a different technology entirely. CDC is just the log of changes on a relational table. Delta Lake appears to be providing more native administrative capabilities to a data lake implementation (schemas, transactions, cataloging).

</comment></comment><comment><USER18> Are there other ways of implementing ACID transactions on Spark tables?

<comment><USER19> What do you mean Spark tables? Generally speaking it is a bad idea to try to combine ACID with data warehouses.

</comment></comment><comment><USER20> Sounds cool, but then I'd have to use Spark...

</comment><comment> 5874-4b22-a4e0 Cloud -> Data lake -> Data stream -> Data Ocean -> Cloud</comment>


ISP Frontier is failing so badly it won't take questions from investors - <AUTHOR> https://arstechnica.com/information-technology/2019/08/isp-frontier-is-failing-so-badly-it-wont-take-questions-from-investors/ <comment><USER2> I've used them up in Northern California and they are terrible. My mom pays $29/month for 100Kbps. I've called them three times and they flaked both times sending somebody out to troubleshoot.</comment>


Ask HN: If you have access to an IDE, why would anyone use a text editor? - <AUTHOR>

What's the rationale behind text editors like emacs/vim vs IDEs like IntelliJ/Eclipse/Netbeans?
I mean if you have access to an IDE, why would anyone use a text editor?
If there's a really obvious answer, please excuse my ignorance. Just started reading up on programming.
P.S. Also, can one use IntelliJ for python and ruby in addition to JAVA? <comment><USER3> I can't speak for all emacs users, but I wouldn't be caught dead using an "IDE"

IDE ostensibly stands for "integrated development environment." AFAIK, not a single IDE is as integrated, programmable, hackable, flexible, or stable as emacs (or vim).

In Emacs, I can effortlessly open and navigate through an indefinitely large number of files. I can take advantage of myriads of language-specific modes for syntax highlighting, auto-completion, snippets, verification, prettification, truly integrated compiler/interpreter environments, you name it.

If I find I need to do some repetitious gruntwork, I can C-x ( gruntwork C-x ) it away. If this gruntwork seems excessive, I can program it into my editor and forget about it. If I find I really need some feature or functionality, I can pop open a _scratch_ buffer, hack it, and use it instantly. No boilerplate, no compiling, no restarting.

These are just the tip of the iceberg, though. The overwhelming sense an emacs or vim user experiences, is one of power. We have complete control over our programming environment. We dictate it, hack it, and tweak it until it is effortless. This sense of power and hackability so permeates our programming environment, that it becomes physically painful to have to navigate a swarm of inconsistent modal dialogues, accept some kludges and pray they get fixed, and accept that The Developers know how best to configure our environment.

<comment><USER4> I think it would be a worthwhile project to get some of the top emacs/vim users doing a set task (e.g. spit out a rails app, or build a small java app) and compare them to the gold standard ide in a guided screencast setting.

It would be enlightening for the wider programming community to see some stats about how fast each truly is. Obviously approaches could be streamlined, and best practice could be learned by beginners from the get-go.

<comment><USER5> This is a really interesting idea.

Top emacs/vi users : please list your favourite shortcuts.

</comment></comment></comment><comment><USER6> _I mean if you have access to an IDE, why would anyone use a text editor?_

But, why would I use an IDE if I have access to a proper text editor like vim or emacs (I use vim, but I've used emacs enough to know I could be productive in it if vim didn't exist)? Your question makes no sense to me.

I've known a (very) few people who are productive in IDEs, but generally speaking, the best developers I've known use either vim or emacs. I've never experienced an IDE that didn't make me angry, so I've never spent any significant time with one, so I don't know what the valid arguments for using them are, though I'm sure there are some.

I'm curious about the framing of your question. What is the rationale behind your question? When you use a big IDE do you feel more productive? Do you just want to avoid learning how to use vim or emacs? What are you trying to achieve here that brings up this question? Do you think an IDE will make learning to program easier?

One of the reasons I've never used an IDE heavily is because I find them very difficult to use and understand. They pile together a bunch of different concepts under one roof, and assume that it fits the work I'm doing. It pretty much never does, so I would just be using it as a big slow editor anyway. The other reason is that whenever I've wanted something IDE-like (like completion, or highlighting and indenting for various syntax, etc.), I've easily found a plugin for vim to do the same job without all the extra baggage.

The answer is generally that IDEs are big, slow, cumbersome and distract from the software you're building. If you're working in a language where the assistive tools are a core part of the experience (Java and C# seem to fall into this category), then maybe you should use those tools. I work in dynamic languages, and big slow editors just get in my way, slow me down, and fight against my own toolchain and workflow (which includes shell scripts, the shell itself, the ability to open and edit files from anywhere, including in-place on the server).

<comment><AUTHOR> _I'm curious about the framing of your question. What is the rationale behind your question? When you use a big IDE do you feel more productive? Do you just want to avoid learning how to use vim or emacs? What are you trying to achieve here that brings up this question? Do you think an IDE will make learning to program easier?_

I am not a programmer. I have recently started to learn programming. I was reading up on mediums to write programs in (text editors vs IDEs) and came across all the features provided in IDEs. Which was when I became curious as to why I see a lot more chatter about text editors like vim/emacs instead of the other way around. That's all.

Again, I'm a beginner, so just trying learn the ins and outs. Thanks for your help.

<comment><USER6> _came across all the features provided in IDEs_

Maybe merely having a lot of features is not necessarily the most valuable thing when writing software. I suspect most truly great developers (I'm not one of those, but I've worked in the field long enough to know them when I see them work) just want the tools to get the hell out of their way. Stability, speed, predictability, consistency, generality; those are probably the most important features for a great developer. I like those features, too, and it's why I use vim.

Also note that I use vim for everything. I don't have to context switch when I go from Perl, to Python, to C, to bash, to sh, to configuration files, to HTML, to CSS, to JavaScript, to RPM spec files, to deb control/rules files, etc. It's always the same in vim (and vim has syntax highlighting for all of them). This is where generality has value. I don't have to think about how to copy/paste, how to find the end of a block, or how to perform a search/replace across the whole file or just a segment of it. It's always the same.

The greatest Java IDE in the world, whatever that is, is definitely not the right tool for editing files on a remote server. So, I would have to switch mental modes to use it. That would slow me down. I can use vim or emacs for that, no problem.

And, of course, as I mentioned, there is probably no feature in any IDE for which there isn't a parallel in vim or emacs. It is probably optional, it is probably not pretty, and it probably requires you to learn some new key sequences. But, it almost certainly exists, whatever it is. But, after spending a few years with a good editor, you'll probably find that simpler but more general solutions are superior to special purpose tools.

 _Which was when I became curious as to why I see a lot more chatter about text editors like vim/emacs instead of the other way around._

Age has it privileges. emacs and vi have been around for a _long_ time. Assuming something is good and continues to get better, the longer it exists, the more users it will have. emacs and vi (in the form of vim) are both excellent and continue to get better. Thus they have a very large, very dedicated following. Which leads to faster improvement (though in directions that might not be immediately apparent if big IDE-style feature lists are what you think of as "improvement"), more third party tools and support, and more docs and examples.

You'll never hear any long-time practicioner of software development or systems management say that learning one of the two major editors is a waste of time. Knowing vim or emacs will very likely serve you well for the rest of your career. Knowing an IDE might make working in one language in one environment a little bit more awesome for year or two or three that you work in that language in that environment.

</comment></comment></comment><comment><USER7> My usual development environment for is a Macbook Pro with six terminal windows open. The first one (Command-1 shortcut key) is a connection to the database, the second (command-2) is Vim with program code, the third (command-3) is library code or test scripts or whatever and the next three (smaller, with blue backgrounds) might be showing tail -f of Apache error or access logs or something like that. If I need to edit an image I command-tab to the GIMP, to view a web page, command-tab to Firefox or whatever. I can connect those windows to one or more remote boxes with ssh and not even have to think about what machine I'm working on. It's easy to reuse those windows for other things, close one with command-w or open new ones with command-n. Vim does syntax highlighting and will even auto-complete if I wanted to turn it on.

What would an IDE give me that this setup won't? To me, the setup I use IS an integrated development environment.

I have used Visual Studio, XCode, Eclipse, etc. but whenever I do I feel slow and inhibited because of the necessity of mouse movements (unless I want to learn a completely different set of key shortcuts that will only work in the IDE and not the OS) and the annoying tendency of IDEs to reformat my code or complete blocks I'm not ready to complete.

Also, OS X and Ubuntu both allow great control over terminal window transparencies and color schemes. I'm not aware of any other IDE that does. I like to keep my main editing windows at 70-80% transparent so I can watch logfiles/test scripts run in the background while I'm editing.

</comment><comment><USER8> IDEs are big, slow, rigid. Text editors are lightweight and fast. You'd have to weigh what you get from the IDE vs. what you're giving up.

As for your other question, yes, there are ruby and python plugins for IntelliJ. The python one seems new and immature. The ruby one should be better. They (Jetbrains) actually just came out with a standalone Ruby IDE as well. I think it's meant for people not already using IntelliJ, because it wouldn't be worth paying for both products since the Ruby plugin works well enough.

</comment><comment><USER9> I use a text editor (like vim) because it's so much faster than an IDE (like eclipse). If you use keyboard shortcuts and work quickly, you'll quickly find that the IDE slows you down considerably.

Also, I find that IDEs hide a lot of the inner workings from you. That can be convenient, but when you run into troubles during a build or when debugging, I'd rather have more control, more details, and a better idea of what's going on.

</comment><comment><USER10> I use a text editor for all of my hacking needs (vim but emacs is good times too). That being said I'm amazed the HN community has so much trouble understanding the usability. Do you feel similarly when you load up into the latest Rate My Startup: "Wow this interface is really just holding me back, I'm completely stifled by all these buttons all over the place.This interface should just be a blinking cursor" Of course you don't, feature discoverability is an important feature that designers strive for. When you're competent at programming and you really know your editor then you boot up and you know exactly which feature you want to use and the shortcut to do it, but it's almost impossible in vim/emacs to learn these features on the fly. In my opinion IDE's can be a great way to learn to program particularly one's designed with a particular language in mind. The best example of this I've seen is DrScheme which is lightweight, intuitive and (believe it or not) does by and large stay out of your way.

</comment><comment><USER11> IDEs baby new programmers in such a way that they will become crutches. Text editors (in theory) only modify the source code. All compilation and backend work is done by the programmer with external tools. IDEs will usually hide that work away in some button or drop down menu, so the new programmer will be less inclined to learn how everything is run.

When I learned java, I broke away from the IDE I was given to use in class and switched to a text editor. I read about command line tools and what a classpath is. I doubt that I would have known about how programming works behind-the-scenes if I hadn't broken away. Many (not all) of my classmates haven't; they rely on the almighty 'Compile' button, don't question it, and won't bother to learn otherwise.

</comment><comment><USER12> Because Unix IS an IDE already, with text editors like emacs or vim fitting in nicely with scripting, make, etc, etc.

<comment><USER13> I consider the Unix command line to be my IDE too (I'm mostly a vim user).

Just last night my plane was delayed due to heavy storms in Europe and as other peoples heavy laptops were running out of juice I still had hours left on my netbook. The command line is a great environment on low power hardware, it doesn't need a mouse and you don't have to wait for IDE support to start working with a new or obscure technology.

That said, I still feel I need netbeans/jvi to do anything with Java -- personally I feel this is due to shortcomings with the language rather than a problem with the way I normally work. In fact, I recently started playing with Scala and, since netbeans didn't really add anything useful, I was able to go back to vim. I feel much more productive in Scala/vim than I ever was in Java/netbeans.

</comment></comment><comment><USER14> And here we have another battle:

Console or GUI?

The only really good developer I know who prefers Visual Studio to Emacs is the one who prefers TortoiseSVN to svn.exe and just dislikes command line.

However, there are two extremely good reasons to use command line (and emacs/vim):

1\. You're typing text when coding or debugging, and going back and forth with mouse is counterproductive and frankly, really irritating.

2\. You can automate text-based controller much more easier than a GUI-based one.

</comment><comment><USER15> I think the line between an IDE and a text editor is not that clear and instead there is a spectrum of editing. At one end you might find Leafpad (aimed at simple text editing) at the other end something like IntelliJ. IntelliJ provides more code tools than Leafpad, but is more heavyweight.

So the question comes down to how many code tools do you need. If an editor has too many then it might feel slow, too few and it might be less effective.

Many editing environments have a plugin style extension mechanism so that you can add new tools to it if you need them, or remove them if it feels too heavyweight.

A key difference between say emacs and IntelliJ is that IntelliJ has more specific tools loaded by default (e.g. Java focused). Perhaps not the ones you want, but you don't have to go to all the effort of finding them, loading them and they all fit together well - of course there is a cost that goes with those tools: increased CPU / memory.

When choosing one of these pieces of software it might be useful to look at the range and power of plugins that are available, however, that's not an inherent difference between IDE and editor.

</comment><comment><USER16> load time on a text editor is much faster.

if you know exactly what you want to do, but still are doing things in code, popping open a file in a good text editor is superior.

if you're looking to open up an IDE and sit down for a few hours' work, then that is probably the better choice.

<comment><USER17> Sometimes I just have to make a small link change or replace a word. Text editor opens up fast and just gets the job done.

<comment><USER16> exactly.

or, stuff like config file editing. why would i ever open up an IDE for that?

</comment></comment></comment><comment><USER18> When I'm on laptop battery I switch to "LispIDE", which despite the name is an ultra-lightweight Lisp text editor for Windows. It has the absolute bare minimum; runs a lisp process, highlights syntax, indents code, and is able to evaluate code in the editor buffer.

I get so much done in the 2-3 hours I'm at the coffee shop, I also take copious notes in notepad, by the time I come back home I can fire up Emacs and cut and paste tiny little independently developed routines and classes, and orchestrate them to a good chunky module.

I only wish I could get more mileage out of this little brain of mine for more than 4 hours a day. I have to stimulate myself with hackish sub-projects and chase "performance" red-herrings just to keep going.

</comment><comment><USER19> vi and emacs (I prefer emacs) are consistent and have been around for years. IDEs come and go with passing of languages, trends, etc. With vi/emacs you have consistency and a bug-free editor.

As a side note: I once saw UltraEdit eat a text file my co-worker was editing; once saved as UTF-8 it blew the file away and crashed. Another reason not to use closed-source junk.

</comment><comment><USER20> In general, I tend to prefer using a text editor over most IDEs, which are usually big and fat and slow, with poor usability. IntelliJ - for me - <USER2> always been an exception. I can't comment on the Ruby or Python plugins for IntelliJ; however, the Clojure and (my) Arc plugin work just fine. I find working with them preferable to SLIME or TextMate or whatever else.

</comment><comment><USER21> [Cue the "all _real_ programmers use vim/emacs/etc" comments.]

I've used both. I like both. If you're new, I'd start with an IDE and lots of online, context-sensitive help.

And an easy language.

Once you get your feet wet writing a few apps, keep challenging yourself by moving to more complex languages and less-visual environments.

</comment><comment><USER22> I use Acme and Plan 9 from Bell Labs, a text editing tool, I edit text. I don't need something to do the programming for me. Acme's ability to run commands by clicking on text is more productive than you're giving it credit for right now. Want another menu item? Just type it into the menu bar! Use a bunch of commands on one file regularly? keep them in the comments. Want to pattern match text snippets and execute commands, no problem it's built in to userland.

I won't call it an IDE because it knows nothing about the types of text you're editing, doesn't do syntax highlighting but it can run a shell in a window, and an HTML renderer and has an email client and a wiki editor and and and</comment>


I run a service that uses real people to review resumes, not robots - <AUTHOR> http://resumereviewers.com <comment><USER2> Theoretically, I'd like to like this, but paying to have 2 college students ( _i.e._ people who have likely not even had a professional job before) review my résumé doesn't sound like a wise investment. I know you stated elsewhere in the comments that you have "other people," but I'd want to make sure that if I'm paying for this service, at the very least the reviewer was someone who'd hired for a job similar to what I was after.

I guess what I'm saying is this: why should I pay for this when I can just have lunch with a tech recruiter I know and get free advice?

</comment><comment><USER3> In your marketing copy you call yourselves experts, but then in the team section you mention that you are students in college.

I would suggest making it more clear what your qualifications are that make you consider yourselves experts. As it stands now, I would be skeptical about your service.

<comment><AUTHOR> I was just waiting for someone to say it. It was more of an experiment to see how people would respond to that word. I changed it to my original idea.

Thank you for pointing it out.

</comment></comment><comment><USER4> It would be important to know your team. I see it's just a team of 2, which clearly won't scale given the 'demand' that's present for the work you are into.

Do you have a team? In that case, please put up their bios on the page as well for credibility's sake. This is different than a usual software company. :)

<comment><AUTHOR> You are correct, and I do have people, but I have to consult them first before putting them on the site. Haha :P

</comment></comment><comment><USER5> If I click "Get Feedback" without doing anything else on the page then I see "internal server error"

Are you just testing out the concept? I didn't see pricing anywhere on the page. Even if it's free you should probably mention the price.

<comment><AUTHOR> Oh thanks for mentioning that, I will fix that bug, and you are right I will mention the price. I am not just testing this out, this is a real thing.

</comment></comment><comment><USER6> i think there is a problem. Your site insists that "Your resume must be in the format PDF, DOC, DOCX, JSON, or ODT". I tried for pdf and docx.

</comment><comment><USER7> Make sure to move your site to HTTPS... As it is right now you're sending people's emails and resumes over an unencrypted connection.

</comment><comment><USER8> Does it cost? What do you do with my resume? What's in it for you? Why does your privacy page throw a 500?

<comment><AUTHOR> It does not cost, I will specify that. I will eventually charge for services such as editing and what not. I have fixed the privacy, thank you for pointing that out to me.

</comment></comment><comment><USER9> since this service is free, why does it connect to stripe.com when uploading resumes?

</comment><comment><USER10> There's no client-side validation on the inputs.</comment>


Analysis of a Kubernetes Hack - <AUTHOR> https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c <comment><USER2> kubeadm seems to configure the kubelet with `--authorization-mode=Webhook`, which prevents the use of the exec API by unauthenticated users:

      $ curl -vk -X POST https://...:10250/exec/test
      ...
      < HTTP/1.1 403 Forbidden
      ...
      Forbidden (user=system:anonymous, verb=create, resource=nodes, subresource=proxy)

OTOH some endpoints on the `--read-only-port=10255` API are completely open for unauthenticated requests... that leaks all the pod metadata/spec/status information:

      $ curl http://...:10255/pods
      {"kind":"PodList","apiVersion":"v1","metadata":{},"items":[{"metadata":{"name":"kube-proxy-knfqg","generateName":"kube-proxy-","namespace":"kube-system", ...}

Not what I expected, and the `--authorization-mode=AlwaysAllow` default seems like a very bad idea :/

</comment><comment><USER3> I'm trying to understand what the issue here was. Did they publicly expose the kubelet port on the internet?

<comment><AUTHOR> Yep. The kubelet ports were exposed. From the article:

As it turns out, our coworker's server was also publicly exposing the kubelet ports (tcp 10250, tcp 10255). Although the problem here was obvious, it should raise some questions about your own Kubernetes deployment, as it did for us.</comment></comment>


The Biology of Sloppy Code - <AUTHOR> http://journal.stuffwithstuff.com/2010/11/26/the-biology-of-sloppy-code/

<comment><USER2> Quite an interesting post and metaphor. I'm sure that someone is going to complain and say you've got it all wrong, but I think you've found a useful way of thinking about programming at the macro level. I started programming as a "chemist" and have more recently moved over to programming as a "biologist". I wonder what "sociologist" programmers will (or do) develop with?

<comment><USER3> Agree. I'm surprised this article hasn't been more popular or at least prompted more dicussion.

The point about moving away from doing everything from scratch to outsourcing via APIs is interesting from a business perspective as well as programming.

There a number of startups appearing that are nothing more than a mashup of existing technologies that can be harnessed via API.

</comment></comment><comment><USER4> Found this article to be mostly uninformed fluff, unfortunately.

One point: although "Ruby and Javascript" don't have compilers to yell at you, there are static analysis tools which accomplish that to various degrees.

</comment><comment><USER5> The use of 'void *' is sloppy? Since when?

<comment><USER6> Apparently sloppy = abstract.</comment></comment>