Programmer Competency Matrix - <AUTHOR> http://www.starling-software.com/employment/programmer-competency-matrix.html <comment><USER2> Those who are interested in this subject and want to discuss it, you may be interested in seeing the points - good and bad - that have been made on previous occasions that this has been submitted. Here are a few previous discussion:

<http://news.ycombinator.com/item?id=232192> (indiangeek.net) 75 points 4 years ago 33 comments

<http://news.ycombinator.com/item?id=554338> (indiangeek.net) 83 points 3 years ago 33 comments

<http://news.ycombinator.com/item?id=1022394> (indiangeek.net) 67 points 2 years ago 40 comments

<http://news.ycombinator.com/item?id=1949915> (starling-software.com) 155 points 1 year ago 106 comments

<http://news.ycombinator.com/item?id=2823912> (indiangeek.net) 12 points 1 year ago 1 comments

<http://news.ycombinator.com/item?id=3434350> (starling-software.com) 2 points 9 months ago 0 comments

Discussions like this can be of value, and no doubt some of the newer people on HN may have points to add. OTOH, many points will already have been made, so seeing the past discussions may be of benefit.

<comment><USER3> 111 comments 1 year ago had 155 upvotes, not 1. The upvote count patterns are interesting nevertheless :)

<comment><USER2> Sorry, I simply cut-n-paste from the search results - I'll go fix that now.

Thanks for the heads-up - much appreciated.

 _Added in edit: Now fixed - points and comments. Thanks again._

<comment><USER4> > _I simply cut-n-paste from the search results_

Ah, we have a level 0 on the HN matrix here apparently.

<comment><USER2> Well, to be honest, I have deep-scripted accessing HN, but I tend only to use it when doing big data crunching on trends and similar. When providing cross- references I've usually just put the IDs - actually counting points, comments and age is something I've not done for a while.

Don't think I'll bother again, I get enough grief and hate-mail just from trying to point people at earlier conversations.

<comment><USER5> I really appreciate it when earlier conversations are pointed to. (The additional points/comments/age numbers are wonderful, too, if you feel up to providing them.)

I wonder if a bot could be written to automate this? There are lots of horrible reasons one could have for writing a HN comment bot, so I'm hesitant to even suggest it, but this seems like a really useful/good bot.

<comment><USER2> I have one, I used it, I got significant hate-mail, I decided I didn't need the bile I got, I discontinued it.

<comment><USER6> Actually, I can't tell you how many times a HN comment like yours has turned me on to an ultra-valuable related discussion that I would never have seen or thought to look for.

So don't let negative comments (perceived or actual) speak for us silent appreciaters!

<comment><USER2> Thank you - much appreciated.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> This is someone's (very arbitrary) opinion in a grid. I hope you don't actually interview people. Figuring out someone's skill level is a soft science; it takes months after a hire to really know where someone is.

Some people I know are great at fixing bugs but have narrow language experience. They're damn valuable. Others integrate well into teams and provide great product feedback. Where's that on here?

Experience isn't measured in years but in knowledge gained. You really start seeing it 4-5 years in, some people move forward, some don't.

There are so many holes in this. I feel like whoever wrote this has a strong desire to reduce people to a number. It's not only terrible, it's a great way to fool yourself.

<comment><USER8> Some people have 10 years of experience. Others have 1 year of experience 10 times.

<comment><USER9> What a great quote. I wonder where it came from.

<comment><USER10> This is a quote I use very often! As with many other great quotes, I think it originates in English from Trevanian's _Shibumi_ (when Otake-san, the Go master is giving his final teaching to Nicholai):

Do not fall into the error of the artisan who boasts of twenty years experience in his craft while in fact he has had only one year of experience — twenty times.

Most probably Trevanian has drawn upon some other source, tough I've never been able to track that.

If you haven't read this fantastic book I strongly recommend it.

<comment><USER11> It's an ok potboiler that relies ever-more improbable plot devices.

It's also impossibly incorrect politically -- so transparently so that I think it's essentially harmless. But if you don't understand why some people would feel that why you really should watch your step.

</comment></comment></comment></comment></comment><comment><USER12> I'm not sure this has anything to do with "competency". When I was in high school, I implemented a kd-tree for a ray tracer using photon mapping. I knew that my n^2 algorithm was slow and that there was probably a better way, so I Googled and figured out how to write code for such a structure.

Since I'm not a programmer (I'm a chemical engineer), I don't instantly know all of the orders of various algorithms (which according to this article makes me "incompetent"). But you can bet that when it comes to implementing something, I'll research and figure out the most efficient algorithm available and use it.

I'm not sure being good at programming has much to do with having a whole bunch of CS knowledge; rather, it's being able to figure out what knowledge you need to know -- when you need to know it.

<comment><USER13> I agree with this - e.g not being able to bust out a dynamic programming implementation perfectly on a whiteboard doesn't make you a bad programmer - more important is to understand, "hey, looks like when I divide this problem up it has overlapping subproblems - let me research using memoization or dynamic programming". But that still assumes you have the right terms in your head to be able to do the research - and to me that's where some fundamental knowledge of algorithms and data structures comes in handy.

<comment><USER14> True. But note that the matrix isn't a binary "good/bad". If you know the concepts that underly dynamic programming and can look it up when needed, great. If you have used it before and can bang out an implementation without looking it up, well, that's even better.

</comment></comment></comment><comment><USER15> Here's mine:

Level 0: Saw the post title and didn't think he needed to look at it because he was so confident in his abilities already.

Level 1: Read the post, noticed a lot that he didn't know but didn't intend to do anything about it.

Level 2: Read the post and went out and learned everything in it that he didn't know.

Level 3: Read the post, figured out which things from it he didn't know but might need to or want to, and learned those.

Oh, and a special level -1: Read the post, noticed a lot of things that he didn't know, and decided the post was stupid because clearly he's an amazing programmer and the post didn't agree with that self-assessment.

<comment><USER16> Nice.

Is there a level for "Skimmed the first part of the post and then went back to working on his/her startup" ?

<comment><USER17> No but there's one about commenting on the post in a way to make oneself seem clever. I have hereby attained it.

</comment></comment></comment><comment><USER18> Interesting. One thing that caught my eye was "someone who knows most of the tools from Scott Hanselman's power tools list." I had never heard of Scott Hanselman, nor his power tools list and had to look it up. Frankly, it looks useless to me as a programmer and looks more like a list of desktop gadgets, and watered down versions of tools you get for free with most Linux distributions. He also doesn't mention Linux (or OSX), nor Emacs nor vim. The great majority of the tools are Windows specific or Windows only (I caught one or two that weren't).

<comment><USER19> Knowing most of the power tools mentioned in this book <http://shop.oreilly.com/product/9780596003302.do> would do a better job as a discriminating factor, in my opinion.

<comment><USER18> That's one of my favorite books. I know it isn't, but I've often imagined that the tool on the front is a hole hawg ;)

</comment></comment></comment><comment><USER20> Many are biased (objects > functions), several are not useful and a few are plain wrong.

Meh.

<comment><USER21> Agreed. I'm not sure how 'Source Code Version Control' and 'Build Automation' is anywhere near the words 'Software Engineering'. And 'Automated Testing' isn't a given at a lot of shops.

Professionally, I've never set up version control. It's there when I start the job, it'll be there when I leave. I've branched and merged but a lot of times this is handled by a build manager. Build automation I have never expressly done - a shop that needs continuous integration is typically one where it is already set up.

Seriously, unless the job is 'Build Manager' then the question should be just used to gauge what you might have to teach them. 'Have you used source/version control software? Which ones?' 'Do you have experience with build automation tools?'.

<comment><USER22> Let me get this straight, you do not know how to set up version control, therefore knowing how to set up version control is useless?

It's not hard to start a Git repo you know, it's something you can pick up in under a day.

<comment><USER23> It wouldn't be useless but it could be unnecessary. I don't anticipate ever needing to set up version control, unless it's my own business. Any place I work will almost certain already have it set up, and if not, someone else will be tasked to set it up.

<comment><USER24> I actually like to put many of my documents under version control. Also, if you do any project as a hobby, using a vcs would probably help a lot.

</comment></comment></comment></comment></comment><comment><USER25> I personally believe I fall into "n (Level 2)". I recently completed a degree in Software Engineering, and despite programming for a long time before that I would have ranked myself at "n2 (Level 1)". The degree pretty much brought me and everyone up to scratch with "n (Level 2)" in terms of knowledge. Having actually tried to work with some of my classmates - who also achieved first class degrees (some with higher grades than me despite little prior programming experience) - they're mostly terrible programmers.

As far as I'm concerned, this table is purely for working out a programmer's education, not competency. Experience and passion are the only ways I chose to judge my colleagues.

</comment><comment><USER26> "able to identify NP problems" ? Most of the problems we face are NP. Maybe he was meaning NP-complete problems. See <http://mathworld.wolfram.com/NP- Problem.html> for the definition.

<comment><USER3> _NP-hard_ is the actual class of problems you want to identify, although people who can identify NP-complete problems have no problems spotting NP-hard ones.

</comment></comment><comment><USER27> Inexperienced programmers consult documentation a lot and the more experienced you become, the _less_ you consult documentation? If this were true, the initialism "RTFM" would never have been coined.

<comment><USER28> That one seems totally backwards to me. Experienced programmers should be able to read and understand more technical documentation such as language specs or RFCs and use it to figure out the answers to their problems. Inexperienced programmers are the ones that sak for help at the first sign of trouble.

</comment></comment><comment><USER29> Here is a simpler matrix

\- can the candidate do the job? (github projects, portfolio, hands-on test)

\- when they talk, do you understand?

\- when you talk do they understand?

\- do they sit all day and read HN / Stackoverflow?

\- on the other hand, they never heard of the above

\- do they get the job done or try to make it perfect?

\- do they fit the culture of the company

\- can they learn a new skill quickly

give 1 point for each of the above and you got a much better matrix (if you want to hire people you won't need to fire)

</comment><comment><USER30> As usual, already written in the bible: <http://www.joelonsoftware.com/articles/fog0000000038.html>

</comment><comment><USER31> Where's the interpersonal skills section?

Attitude section?

<comment><USER32> Those are part of the "existing-in-modern-society-as-a-human-being competency matrix", along with bathing and wearing pants as often as possible.

<comment><USER5> I think there are programming-specific ones.

For example, on attitude: Learning to be patient. Patient with planning the right solution, selecting the right libraries, fixing bugs, dealing with other developers who may not be as experienced as you, etc. I think patience is an overlooked "programmer's virtue."

<comment><USER33> Actually, impatience is the virtue. <http://threevirtues.com/>

<comment><USER34> Ugh. A pox on people who write programs that "anticipate your needs". They get it wrong more often than right, slowing you down, blocking you, or interrupting you and breaking the flow.

And to make matters worse, those features tend to be baked into the system such that you can't disable them.

Impatience is no virtue.

</comment></comment></comment></comment></comment><comment><USER35> The original is here: [http://www.indiangeek.net/wp- content/uploads/Programmer%20co...](http://www.indiangeek.net/wp- content/uploads/Programmer%20competency%20matrix.htm)

</comment><comment><USER36> This is technically not a "bad" measure of performance, but the real danger is the word "competence." Imagine I am hiring two candidates, one has log(n) (level 3) competence across these charts, while the other has very mediocre/poor competence. What if I am working on an application is not performance-critical? What if the "incompetent" guy/girl works twice as fast, because he does not worry about crafting coding-works-of-art? What if the "incompetent" guy is three times cheaper to hire, and eager to prove him/herself, thus putting in extra hours voluntarily? What if I am hiring for a problem with a very specific domain, and expertise is required in only one area?

<comment><USER37> You don't need a wizard for every task. There's nothing wrong with taking someone from the left side of the matrix. You just have to consider what they'll be working on and what resources are available to them (ie. mentors, etc.) Hiring decisions are comprised of tradeoffs, just like everything else in engineering.

</comment></comment><comment><USER38> This is a rather narrow viewpoint of what makes a great programmer in my opinion. One completely missed element here is the ability to track organizational complexity. For example, you have a site with 100,000 lines of code. Making one change somewhere affects 10 other things elsewhere, each of which might affect another 10 things. Keeping track of all that is what I'm talking about. Same skill you find in great chip (integrated circuit) designers. Personally I think it's far more important than some of the concepts here, which focus mainly on algorithm-level stuff.

</comment><comment><USER18> Interesting. One thing that caught my eye was "someone who knows most of the tools from Scott Hanselman's power tools list." I had never heard of Scott Hanselman, nor his power tools list and had to look it up. Frankly, it looks useless to me as a programmer and looks more like a list of desktop gadgets, and watered down versions of tools bundled with most Linux distributions. He also doesn't mention anything for Linux (or OSX), nor Emacs. The list is very Windows specific.

</comment><comment><USER39> Thank goodness B-trees are considered the most-advanced of the data structure knowledge. My com sci classes never did a great job of making those applicable.

<comment><USER40> B-trees are great! They have better locality than binary trees --- that's why filesystems are universally based on B-trees: seeks are slow. As cache effects become ever more important, B-trees are starting to become important even for problems that fit in main memory.

<comment><USER41> And as data and disks grow, we'll see write-optimized structures like LSM trees and Fractal Trees in this matrix as well.

</comment></comment></comment><comment><USER42> I land solidly in the right-hand column, particularly on the systems programming front: I've made major contributions to the NT kernel. I'm not sure this matrix has all that much significance: Google rejected me after the in-person interview: do they want people who can sort N items in O(1) time, work 30 hours per day, and create new scripting languages over lunch?

<comment><USER43> Apply again. Google rejects a lot of good programmers on their first application. It's not a perfect process.

</comment></comment><comment><USER44> I would like to see a section that includes things like identifying/clarifying customer requirements, finding business solutions, controlling scope, organizing a project around some methodology, leading a team, etc. I see the lines for Problem Decomposition and Communication, but I think there is something much broader that is being left out.

</comment><comment><USER45> Either where I went to school had an exceptional CS program, or most CS programs don't cover what my run-of-the-mill state school covered during my undergrad. About 10 years ago I came out of school at around a level 2 or 3 in almost all of the areas on this grid. I would wager pretty much any 3.5+GPA student in a decent CS program should be the same.

I didn't pursue being a professional developer so I've lost most of these things, but I remember having worked on almost all of these things (such as they were at the time). Some of these things I haven't heard mentioned _since_ my undergrad!

Interestingly, we never studied functional languages, but did cover Prolog. This has definitely created a deficit in my thinking and consideration about certain problems/languages. If that's any indication this matrix can give insight into the abilities of people who simply haven't covered certain topics.

</comment><comment><USER46> This is the stupidest thing I've seen on this topic. Are we trying to eliminate good candidates? Because that's what this will help with.

If programmers spent half the time they spent evaluating others on being better programmers we wouldn't have any problems.

</comment><comment><USER47> I really like the idea of a way to measure my knowledge and skills. My preference would be for a directed knowledge graph, where nodes list the knowledge and skills fundamental to the nodes they connect to. This way, I could start from the bottom of the graph and place a checkmark on each node that I felt comfortable with. The end result would be an outline of what I needed to learn next to progress.

Alternatively, if I were in a rush to learn a specific item, the tree would give me a roadmap for getting there.

I'd be interested in a wiki-like interface for building such knowledge graphs, so that they could be built quickly and completely.

</comment><comment><USER48> I'm not sure if "has used git" should be log(n)-- I'm only barely a programmer, and I've never used CVS/SVN, but I use git to version control markdown files.

<comment><USER22> Using Git requires a bit more brainpower than using CVS or SVN. If you know how to use Git, you definitely know how to use SVN.

<comment><USER48> I've never branched, or performed a merge. I use git as a pure CVS clone.

</comment></comment></comment><comment><USER49> Cool -- I actually hit n or log(n) on a couple of these. Neat! Most of the rest I'm right at the n^2. Guess I still have a long way to go.

</comment><comment><USER50> "Checks all arguments and asserts critical assumptions in code "

Wait, I just finished code complete, and it told me that you specifically _don't_ check all arguments throughout your program. Instead, you choose a list of classes at that will check arguments for input as a firewall, and assume all classes behind those logically will receive the correct, sanitized input.

Did I misunderstand this?

<comment><USER18> If you're paranoid (eg, programming embedded where it absolutely, positively, should not fail), you _always_ check _all_ arguments and return values. Sometimes libraries shipped from the vender are _not_ perfect, sometimes the vendor's compiler will do something, _ahem_ , unexpected with your return value in some edge case where you've exceeded the stack. I'll agree that wrapping with inlining is a good idea to make code readable, and these days you usually don't have to worry about a lot of hardware limitations (especially in the general case: desktop/Android/iOS/web/etc).

<comment><USER40> > programming embedded where it absolutely, positively, should not fail > [means] you always check all arguments and return values

Let's distinguish "checking" invariants, preconditions, and postconditions from blithely propagating logic errors as error codes all the way up the stack. If you're programming in "paranoid mode", as it were, and you detect inconsistent state, you should _abort the program_. If you don't, you're in uncharted territory and you have no idea how your program will act. If you abort the program and restart the system, you'll probably end up back in a useful state.

Many, many times, I've seen code like the following:

        HRESULT hr;
        hr = DoComplexThing();
        if (FAILED(hr)) { RaiseFailFastException (...); }

Say someone passes a NULL pointer to a function somewhere deep inside DoComplexThing's implementation that doesn't expect NULL. If this function helpfully "checks" all parameters, detects the NULL, and returns E_POINTER, E_POINTER will probably propagate all the way up the stack to the top level, where we'll abort the program. Now you have a crash report, but you have no idea where the problem actually _is_.

If this function had instead not checked the NULL pointer and crashed or explicitly asserted its preconditions instead of treating contract violation as a runtime error, the problem would be a lot easier to diagnose.

In general, you can separate problems into two classes: logic errors and runtime errors. Logic errors are indications of the program being written incorrectly. Your program should blow up when it detects one. Runtime errors are errors that could conceivable occur at runtime due to reasons outside the program's control (e.g., removing an SD card). Only runtime errors should be propagated using your error reporting mechanism of choice.

</comment></comment></comment><comment><USER51> I'm amazed that Oz surface up in this matrix. The number of people actually program in Oz for production purposes must be less than 100. It's a lovely language and I can miss somethings one could do in (Mozart)-Oz that I can't do in Haskell.

</comment><comment><USER52> It covers a good section of the competency area and kudos for putting it together. It's certainly much better than the puzzle type questions.

It's kind of generous in its rating for advance though. May be there should be a level for expert. But good work.

</comment><comment><USER53> So in the end I figured out you can't be no good programmer if you don't know prolog.

<comment><USER54> yes.

</comment></comment><comment><USER55> This probably isn't very useful for evaluating an interview candidate as a whole, but I like how comprehensive it is in terms of technical knowledge. It's a useful roadmap to evaluate your own technical competency too.

</comment><comment><USER56> This reminds me that once upon a time scholars thought that art was measurable.

</comment><comment><USER57> I'm sure it's a perfectly good topic to discuss, but I wonder what's motivating this sudden trend of "You must be this tall to call yourself a programmer" posts?

</comment><comment><USER58> Why do I feel like "basic competency for someone with a few years experience" is the top end of this grid? Seems like some low bars.

</comment><comment><USER59> According to this, I'm a worthless newb.

Sounds pretty accurate. :)

<comment><USER60> I work with PHP and javascript so I already knew I was scum...

<comment><USER18> Hey now, don't be so hard on yourself - at least you work with Javascript ;)

<comment><USER60> I did teach myself enough Perl to write what was possibly the worst flat-file bbs script ever, ages ago on Tripod. >_>

</comment></comment></comment></comment><comment><USER61> Now if you could write tests that correctly determine which box a candidate fits in, you would have something very powerful indeed.

</comment><comment><USER62> Ha, wow, I think am a terrible computer scientist and software engineer, but probably a decent to maybe even good programmer.

</comment><comment><USER63> Unfortunately none of the skills mention the ability to create something that people ACTUALLY USE.

</comment><comment><USER64> maybe it's just because i'm mostly an algorithms guy, but the bar for "O(log n)" systems programming seemed to be somewhat higher than that for datastructures and algorithms. and having "author of framework" be on the same level is another huge leap.

</comment><comment><USER65> Someone has way too much time on their hands.

</comment><comment><USER66> What is a new grad expected to be n, n^2?

</comment><comment><USER67> > _automated testing_

I love how they non-chalantly squeezed "automated testing" in there... because to really do well in that area you need so much more than the average handful of junit-or-whathaveyou tests and the whole "comes up with good test cases" is just extremely shallow. While developers do well to implement some simple unit tests, automated testing includes SO MUCH more they just left out. It is a whole software project and includes a lot more people than just "programmers" and "testers".

Another personal pet-peeve of mine are those endless suggestions of how to pick and hire developers, how can you get the very best of the best for the likely average salary you are offering to begin with - yet we have a continuous stream of coding-related horror stories.

It doesn't take any magic, it doesn't even take a catalog of weird questions that might work for google. Ideally go with recommendations, people who know people that are good. Or pick the interesting CVs and cover letters, avoid the buzzword rally. Then interview them and talk about what they have done so far and go into some details there, scratch on the surface, get a feeling for what they got going on. By now you should have quite a good idea. Then hire them on a temporary contract and see how they do. Be ready to give the new-comers and graduates a chance and be ready to invest in them, both in trainings and compensation.

IF you are in a position where you just cannot afford to put good faith in a common sense choice for a candidate then you should not be hiring in the first place because that means you either don't have the money for one more hungry mouth anyway or your project is so far into the death-march that another pair of hands is not going to rescue the sinking ship even if you get a true kernel-hacker to work on it.</comment>


Show HN: HTML5 Holiday Canvas  - <AUTHOR> http://www.swiperoo.com/ecard/ecard2.php

<comment><AUTHOR> FYI should work on your iPad also</comment>


Italian Mafias killing less but infiltrating more businesses - <AUTHOR> http://www.japantimes.co.jp/news/2015/12/16/world/crime-legal-world/italian-mafias-killing-less-infiltrating-businesses/#.VnJptq88Kh9 <comment><USER2> People and movies focused on the killings, while the real nightmares were and are in the business, public sector and work environment. When you don't do what a politician wants you to do, he sends you the tax authorities or some other random authority of which Italy is full of, and he'll get what he wants. Do you realize how bad this is? The Public Forces are used to enforce mafia- like behavior. It's bad, it's widespread, it's cultural and it's stopping any kind of real development or innovation. The problem is that Italy never really got out of feudalism.

<comment><USER3> Gomorrah [http://www.imdb.com/title/tt0929425/?ref_=nv_sr_2](http://www.imdb.com/title/tt0929425/?ref_=nv_sr_2) does a really good job of portraying this aspect.

<comment><USER4> So does the Romanzo Criminale series [http://www.imdb.com/title/tt1242773/](http://www.imdb.com/title/tt1242773/)

</comment></comment></comment><comment><USER5> Bikers have done this too, here they own development corps which are just rackets to exploit city tenders. Nobody else can bid or else you find your yards and equipment robbed or destroyed. Police don't do anything because propping up org crime is part of their management strategy of the failed war on drugs. Young violent upstarts who heat out the city with public shootings are better handled by the dominant org crime group so they let them racketeer it up so long as they promise to reign in the lesser groups that make the police look bad with public violence. Since these young upstarts need the money laundering services only mafia/bikers can provide on that scale they generally fall into line.

They also are propped up in the prison system in turn for helping the guards establish order in overcrowded units. Guards do set ups to make sure the dominant group always has an upper hand in any conflicts, like 4 associates being "accidentally" put in the same class/program with a rival of another group.

<comment><USER6> I'd like to know more about this, do you have any references to articles about the subject?

<comment><USER5> I'll post some later tonight when not swamped at work crunch time, what would end this is legalizing most of the black market since all you need to make a million dollars right now is a handful of drug delivery drivers and a couple of guns to shoot at and rob all the other drivers. Cities can then break up the entrenched corruption since they don't need a "boss crew" to keep order. Prisons won't be overcrowded with drug convictions either.

There will always be gangs but they won't have access to millions in capital to fund huge street wars thus less incentive for new recruits to join them.

<comment><USER7> Sounds most intriguing. Please do post.

<comment><USER5> Here's one example, cops ask gang to police their members [http://www.canada.com/story.html?id=5ccc1c20-c1ef-4dc5-a4f9-...](http://www.canada.com/story.html?id=5ccc1c20-c1ef-4dc5-a4f9-ae0f74a3aa5d) and shortly afterwards said members start "losing status" (getting killed) [http://www.theglobeandmail.com/news/british-columbia/ex- hell...](http://www.theglobeandmail.com/news/british-columbia/ex-hells-angels- member-shot-dead-outside-vancouver-home/article1376917/?service=mobile) said member also ran a tender corp.

The prison beating lawsuits are all here [http://blogs.vancouversun.com/2015/11/26/violent-attacks- in-...](http://blogs.vancouversun.com/2015/11/26/violent-attacks-in-b-c-jails- increasing-and-victims-are-suing-the-government/)

There's also a well known illegal drugs shack called the "blackdoor" that operates with impunity because before they arrived the streets were a free for all of violence, during the Olympics the police asked them to close so the media wouldn't discover it and embarass them.

Then of course there was all the Quebec mafia corruption probes into illegal city tenders that went nowhere.

<comment><USER7> Fascinating. Thanks for posting.

</comment></comment></comment></comment></comment></comment><comment><USER8> My guess is that in a "professional" organized crime family if you have to whack someone it means someone (possibly yourself) screwed something up very badly. Killing someone is extremely high risk and is probably regarded as sloppy operations.

It sort of mirrors the way governments have gotten more advanced with repressive and tyrannical techniques. Unsophisticated governments will whack people or arrest them and throw them in bottomless pit prisons, etc. Sophisticated totalitarian regimes use nudge theory and very advanced astroturf type propaganda, and when they have to 'whack' someone they _discredit them_ and assassinate their _character_ and _credibility_ rather than martyring them with a physical attack. Much cleaner, more efficient, and in the end far more effective.

<comment><USER9> > My guess is that in a "professional" organized crime family if you have to > whack someone it means someone (possibly yourself) screwed something up very > badly.

Maybe, maybe not - a lot of those groups operate via intimidation, and there's nothing like a gruesome murder to underline that.

<comment><USER10> Yes but one messy and brutal murder is more efficient than a bunch where the body disappears and no-one ever hears about it.

The line is of course somewhere between "maximum intimidation" and "might annoy enough people to have the police actually do something" I guess.

</comment></comment></comment><comment><USER11> This trend started a long time ago when they realized it's easier to hide millions of dollars behind companies people expect to bring in millions of dollars. The smaller ones often used chain stores that did a lot of cash business but also plenty of volume. The larger ones just partnered with banks that looked the other way at where the money was coming from. Sometimes they start a bank (BCCI) geared toward like-minded individuals. A side-benefit of all this is much less heat on them and [in U.S.] a statute of limitations on any given crime that wasn't murder.

So this is nothing new except to the writer apparently.

</comment><comment><USER12> One interesting fact is that the mafia was reduced to meaninglessness during the facist era in italy. I deduce from that that there can be only one villian in town.

<comment><USER13> The US liberators fixed that and made sure that Sicily was to become one of the special statute regions. They don't send any collected VAT to the central state to this day. And VAT in Italy is 22%.

<comment><USER14> After WW2, there was a period of time when it was seriously considered that Italy might come under communist control. Most famously, you have Operation Gladio[1].

If I recall correctly (and I may not, because this is probably colored by some History Channel conspiracy documentaries), there were contingency plans where Sicily would secede to allow NATO to retain a foothold in the Mediterranean, with the CIA relying on Cosa Nostra connections with the Sicillian mafia in order to effect this...

Yep, I must have been watching that in between a special on the Mayan calendar, and one on how Stonehenge is a pre-historic alien spaceport...

[1] [https://en.wikipedia.org/wiki/Operation_Gladio](https://en.wikipedia.org/wiki/Operation_Gladio)

<comment><USER15> There's no shortage of international scumbags that big foreign powers have made deals with and supported in their atrocities in order to get their diplomatic/geopolitical games going.

</comment></comment></comment></comment><comment><USER16> I don't think people realize that Mafia is not anymore what it used to be 50 years ago. They are well-established in different regions across europe (you wish it was just an italian thing, right?) which are generating a tremendous income with drugs and prostitution, but the real deal is when this capital gets re-invested in legitimate business - again across europe. Do you really think that once you get into the range of the tens of millions any financial institution is going to turn down easily any investment? They won't.

So, I think it's so simplistic to picture mafia as gangsters robbing and extorting businesses and dealing drugs. I rather picture them as shirt-and-tie business men traveling to Milan, Munich, London, Luxemburg and Geneve looking for investing opportunities.

<comment><USER17> "So, I think it's so simplistic to picture mafia as gangsters robbing and extorting businesses and dealing drugs. I rather picture them as shirt-and-tie business men traveling to Milan, Munich, London, Luxembourg and Geneva looking for investing opportunities."

So where does the "organized crime" kick in?

</comment></comment><comment><USER18> Interesting that this is hosted on a Japanese news site. Remember that Japan has their own Yakuza.

<comment><USER19> Don't really get your point... Almost every country has some sort of organized crime.

<comment><USER20> The Italian mafia is evolving to operate more like the Japanese Yakuza do. The Yaks are so "infiltrated" into Japanese business and civil service that the Japanese just consider them a part of ordinary life. Only recently have law enforcement and some businesses been cracking down on the Yakuza presence.

<comment><USER19> I was under the impression the Italian mafia is very mainstream too.

<comment><USER21> As an Italian entrepreneur in Milan I must say: absolutely not! We are ensnared with inefficiencies often requiring 'networking', nepotism and clientelism, but the reek of the Mafia is nauseating and avoided at all costs. It's about, of course, but nobody I know or know of 'accepts' it.

<comment><USER19> Is it more acceptable in certain regions? I guess Milan is a pretty important city, so I can't see the Mafia being tolerated. But what about for small shops in villages etc.?

<comment> the-dude Milan: North Mafia: South

<comment><USER22> That's definitely an oversemplification of a complex problem, as anyone can understand. There is mafia also in the North of Italy, just a quick lookup to the news will do.

Really, Italy is a young and complex countries with regional differences that are almost ununderstandable to foreign people. There are many "mafias", more or less violent, more or less connected with the economic and political powers, it's not simple and it's not dicothomic, there is a very big grey area.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> I guess they finally figured out there is more money ripping people off being above the law as a banker or politician than below it as racketeers.

<comment><USER24> The headline reads like a scene from the last episode of Fargo s2.

<comment><USER25> I Ctrl-F'd "Fargo" upon reaching this page. Recommended viewing, for anyone who hasn't watched it.

</comment></comment></comment><comment> guard-of-terra Once they don't do killings, traditional business should be more efficient and therefore drive them off.

Mafia should only be able to thrive in underground activites like predatory loans, drugs and such.</comment>


P2P becoming smaller fraction of Internet traffic - <AUTHOR> http://www.wired.com/epicenter/2009/10/p2p-dying/

<comment><USER2> That graph is somewhat misleading. P2P clients use randomised port numbers now.

 _By contrast, P2P downloads arrive in random chunks, making it hard to know when a download will finish — which is when it's actually possible to start watching anything downloaded._

When software appears like that chinese football streaming software with "click and watch p2p", then there will be some serious headaches in the movie/tv industry. Hopefully, hulu and the likes will have taken up that space by then.

</comment><comment><USER3> Pretty graphics says ~0.5% for P2P. Tiny paragraph in the middle says 18%. The author deceives.

The graphic may accurately show that three of the ports popular with P2P programs are getting less use. These are a meaninglessly tiny portion of the total P2P level.

The result is a graph showing a factor of 5 drop for what is probably a factor of 2 drop in P2P's share of internet traffic.

<comment><USER4> Forget to read the rest of the paragraph?

"In fact, according to its sensors, peer-to-peer traffic still accounts for about 18 percent of all traffic. (That's by looking at packets — by protocol, P2P fell to less than one percent of traffic, but file sharing applications mask themselves in order to evade technical blocks.)"

The graph is useless, but what is presented is that the traffic dropped from 40% (per-packet detection) of internet traffic to 18% (per-packet detection) of internet traffic. This actually highlights the fact that advancing techniques of masking traffic are either working exceedingly well or are not of importance with regards to the use of P2P software.</comment></comment>


How to burn the most money with a single click in Azure - <AUTHOR> https://mijailovic.net/2020/03/28/azure-money-burning/ <comment><USER3> A few years ago my startup was killed by a AWS mistake that ran overnight. The irony: my AWS expert at the time had made exactly the same provisioning mistake at his previous job - so I figured he'd never make a $80k mistake again. It turns out - his mistake with my startup was even more impressive. More positively - he did help shell out with me to cover the cost & overnight we were out of money. The mistake shocked me so much, and I've since heard _so_ many stories of similar mistakes. The event hit me so hard I went back in time to PHP and shared hosting. Not kidding.

<comment><USER4> They should give you the option to set a hard limit across your entire account, to prevent you from accidentally spending more money than you have. "If I try to spend more than $5k in a month, something has gone wrong, don't let me do that."

<comment><USER5> Seems like circuit breakers should be a standard safety feature for automatically infinitely scaling computers.

I would rather my whole system shut down and be unusable while I investigate vs. auto-scale and charge me a bill I can't cover.

However, searching around it seems like I can only get alerts when a $$$ threshold is passed, but AWS won't take any action to stop computing or anything. Please prove me wrong.

<comment><USER6> >I would rather my whole system shut down and be unusable while I investigate vs. auto-scale and charge me a bill I can't cover.

The counterargument is that you get a usage spike (which is often a _good_ thing for a company), and AWS shuts down _everything_ connected to your AWS account without warning.

I'm not necessarily sure that optional/non-default hard circuit breakers would be a bad thing. But it certainly appears not to be a heavily demanded customer feature and, honestly, if it's not the default--which is shouldn't be--I wonder how many customers, or at least customers the cloud providers really care about, would use them.

<comment><USER7> The usage spike is very very rarely worth the cost. That's a pipe dream the cloud providers sell to cover up the fact that these scenarios are sweet sweet profit for them and nothing more. There are very few businesses where making more money is just a matter of throwing some more compute at it.

Nearly every customer (i.e. all of them with a budget) would make use of circuit breakers and it would make Amazon absolutely $0 while costing them untold amounts. Are you really surprised Amazon hasn't implemented them?

<comment><USER8> > Are you really surprised Amazon hasn't implemented them?

It become harmful to them though. At a certain point people feel the hit and avoid the service. Having people spent a little more accidentally and go 'oh well, oops' is the sweet spot. An unexpected $80k which kills the company is bad for everyone.

<comment><USER9> This almost feels like banking fees. A dollar here, a dollar there. In this case it's a couple of thousand here and there until you can't afford it anymore lol.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> Azure has, built in, hard price/cost limits but doesn't allow the public to use them. For example if you have MSDN subscription credit you get a hard limit of up to $150/month, but you yourself cannot pick a bespoke limit to use the service more safely.

Kind of makes me annoyed. I'm sure enterprises don't care/want unlimited. But solo practitioners, and people new to the platform would love a default e.g. $5K/month limit (or less).

Feels like these services just want people to "gotcha" into spending a bunch of money without simple safety nets.

PS - No, alerts do not accomplish the same thing, by the time you get the alert you could have spent tens of thousands.

<comment><USER11> This is only anecdotal, not personal experience, but I've read online and have had friends "oops" away large sums of money on AWS, and for the most part they seem to have at least gotten a partial discount when they contacted customer support.

I strongly suspect opaque pricing and high/nonexistent limits are more about getting large organizations to transition to the cloud seamlessly (i.e. not completely caring/realizing what they're getting into for any particular migration/deployment).

Tricking personal users into spending thousands by accident probably doesn't net much money compared to enterprise spend and runs the risk of alienating people who then can go into work and recommend against using a particular platform, having been burned by it on their personal accounts.

<comment><USER12> > "oops" away large sums of money on AWS, and for the most part they seem to > have at least gotten a partial discount when they contacted customer support

As a counter-datapoint, we accidentally left a Redshift cluster up idling for two weeks before we started getting alerts, and after numerous attempts have failed to get compensated in any way. The reasoning was that, well, it was what we requested and they had to allocate compute power to it (which we didn't use).

All in all a very frustrating experience and it makes me fairly cynical of all these "I got my money back without problems!" comments.

(For what it's worth, it was about $4k of costs which was a lot for us at the time)

<comment><USER13> It's also unnecessary. Give users cost controls, then you won't need the current mess of hoping support will write off $$$ of mistakes. With a risk of bankrupting a small shop if support doesn't help it drives risk averse users towards less dynamic offerings.

Isn't AWS supposed to focus on the virtuous cycle of saving customers money (or at least reducing AWS supports need to write off customer mistakes)?

<comment><USER12> I always have the feeling of a bit of "randomness" with these kinds of compensations. It makes sense, as it's difficult to "codify" these types of things, lest they get abused and you might as well just lower your prices at that point.

AWS is a large organization; I believe this type of stuff highly depends upon your "entrance" into the organization, i.e. the account manager. We were probably just unlucky with our Redshift troubles, but it did eventually trigger a move to Google Cloud / Bigquery, as the pay-as-you-go method seemed a bit safer (although it's still too difficult imho to accurately estimate the costs of queries).

</comment></comment></comment></comment></comment><comment><USER14> Just a couple of months ago, we were blindsided by a massive AWS bill after turning on encryption for logging (an ask from security team). The encryption relied on using KMS, but because it was a serverless setup and each time a lambda was initialized it would grab the KMS keys. Later we found out that invoking that many KMS calls while doesn't cost that much, however does invoke CloudTrail logs which are quite costly. Sometimes it's hard to model something like that. Following this experience KMS team made some changes to how their service/pricing works given its tight coupling to CloudTrail. We also stopped using KMS and simplified the log encryption approach by writing logs directly into an encrypted S3 bucket.

<comment><USER15> We are using two $3.5 AWS lightsail instances to handle hundreds of thousands of customers :)

No hidden or unexpected costs.

<comment><USER14> Can you share a bit more about the architecture and stack? And a few more details such as Roughly how many TPS do you serve from this setup? Does it scale well? Are response times consistent?

<comment><USER15> It's all written in Go + Postgres. Nothing else. Not sure what TPS is.

No problem with scaling.

Stack Overflow runs on a single machine. Use a fast compiled language and a simple stack. No need in $6k/mo AWS bills which are the fashion these days.

<comment><USER16> Your response gives me zero confidence that you know what you're talking about. Having "hundreds of thousands" of users tells us nothing (which is why TPS was brought up). Also if you do have a high concurrent user count then there's no way there have been "no issues scaling". There are always scale points in systems that actually grow.

<comment><USER15> Since 2015, no issues. You can choose not to believe me.

<comment><USER17> You're choosing not to try to convince anyone that what you're saying is true. Why bother posting at all, in that case?

</comment></comment></comment></comment></comment></comment></comment><comment><USER18> I see a lot of people here saying how they "get burnt" by unexpected increase in their bills, because some guy who was supposed to manage the cloud made a mistake. That's not the real burn, the real burn is when you pay $8K a month on cloud services for apps that could run on a laptop in a basement. And you pay that monthly, for years. And before devops guys slash me with "scaling", "fault tolerance", "redundancy" and what not, remember that your little website isn't a multi million user app and all the issues that you think are fixed by simply moving to the cloud can also be fixed by running your app on a laptop (for free).

I used to handle infra for a client a few years ago (the stack was built by another employee before I got there) and it always amazed me why these people pay $10-20K every month to AWS for their app with 500 users a day. Needless to say that AWS was also causing a lot of headaches with their instance management, network maintenance and service unavailability. I know cloud is cool, but business should probably think twice before going for it.

<comment><USER19> Very true, some companies boast about 10,000 registered users. When in reality there's only 100 daily active users...

I think the problem is that companies want to _appear_ bigger than they are. Saying they use _AWS Cloud Stuff_ is a selling point to investors who have a distorted understanding of how scaling works...

The cloud service companies are marketed to entice people into buying services they don't need... yet/ever

</comment></comment><comment><USER20> These basic gaps in cloud hosting providers' tooling have created a rather large cottage industry of companies that exit only to make the tools to fill in the gaps -- like VPC cluster management, security compliance, whatever.

The perverse side of it is, it costs to host the third party tooling as well, so cloud providers get more money from you setting up these tools so that you don't burn away all your money. However, they don't get more money from plugging the holes in their own tooling. So they have no incentive to fix it on their own.

<comment><USER21> Perverse is the right word here.

Perverse incentives are why things are the way they are. This will only change when a large enough competitor defects.

</comment></comment><comment><USER22> I got burned by AWS billing. I played around with some tutorials and before I knew it I had a $100 credit card bill.

Even working out what I was paying for proved really tricky. Turning it all off involved crawling through a bunch of opaque control panels.

I'm definitely never considering AWS again for a personal project. It's too dangerous. I'm not a company and I don't have corporate level budgets.

<comment><USER23> I get around this in GCP by having a script that spins up a brand new project whenever I want to play around with stuff. When I am done playing around with things, I have a 2nd script that deletes the project and thus all the associated resources that are in use in that project.

<comment><USER24> Why do you need a script? With CloudFormation (and I assume terraform), you create a file containing all of your resources in a "stack" and you delete the stack when you're done.

</comment></comment></comment><comment><USER25> We had an instance when some consultant dev wrote a lambda function which gets triggered when a new image is saved in S3, takes it resizes it and re-save it in same S3.

For non-technical guys, that is recursion. We were surprised to see $5000 bill in 10 days.

<comment><USER26> You got lucky, if it saved _two_ different sizes for the each image...

<comment><USER27> I'm actually interested how AWS handles exponential cases like this. They must throttle it somehow, otherwise something dumb like this would be causing availability issues everyday.

<comment><USER25> Unfortunately, there is nothing inbuilt in AWS to mitigate these conditions, except for monitoring for "anamolous" behaviour.

<comment><USER27> I mean there must be something, because otherwise taking down AWS would be as simple as the above. After 50 steps, you've created a quadrillion images.

<comment><USER25> They surely have very high upper limits on S3 files.

Lambda functions have limit on concurrent functions you can run.

</comment></comment></comment></comment></comment></comment><comment><USER28> Takeaways from this and other people's experiences in the comments (apart from making sure such mistakes don't happen in the first place):

• Providers should let users set hard spending limits.

• Providers should offer a channel for investigating and waiving charges for honest mistakes if reported soon enough.

• Perhaps you should sign up with throwaway billing details so you can continue on another account till you sort it out. Morally and legally unsound but probably the better alternative to "killing your startup" I guess.

</comment><comment><USER29> About a year ago I spent $80 making a single query using BigQuery. There was a large public dataset and I did a query that spanned several months of data. My query was something like SELECT fields FROM table.* WHERE date = ... the problem was that WHERE still scans data from ALL date partitioned tables then returns a filtered result. What I should have done is FROM table.date - <USER2> I am glad I checked the cost before I hit it with more queries. At $5/TB scanned is remarkably cheap for small data but not for large data. In my case it was 16TB.

Still, it was pretty amazing that the query returned data within ~2 seconds.

So be careful kids. Public datasets don't cost anything to store, but your cloud account is going to get pummeled for your exploratory data analysis and if you haven't set any billing controls or alerts you're in for a nasty surprise.

</comment><comment><USER30> Which cloud or physical server hosting providers provide either user configurable hard spending limits or the option to (only) pre-pay? The big 3 cloud providers obviously don't, and neither does DO, I believe (you can pre- pay but not as your only payment option).

For personal hobby projects taking your chances on customer support goodwill in case you rack up a bill that would bankrupt you due to a wrong click seems kind of insane.

So what are good alternatives?

<comment> why-el Can't think of any, but managed hosting tends to do better. For instance if you are doing your testing in Ruby, I'd suggest hosting on Heroku while you do all your preliminary work, where it's clear(er) what the cost will be, then move to AWS later when the bank account is, well, beefier. :)

<comment><USER30> Bonkers! Surely there must be a market for this? Also, I get that the selling point of heroku is kind of AWS for dummies, so you are probably slightly less likely to mess up because of that, but it's still some auto-scale-this, click together-that kinda of service. As far as hosting models are concerned, getting a dedicated box and running the stuff you want seems far safer, no?

<comment> why-el Agreed, it's a matter of degree. You can auto-scale up to a certain amount, at least in Heroku, so in practice you can limit it, say up to 8 servers or something, but nothing beats your own box if you can afford the opportunity cost.

</comment></comment></comment></comment><comment><USER31> Does AWS still not have a way to stop accidentally huge bills? How is this not intentionally negligent on Amazon's part at this stage?

<comment><USER32> The fewer system parts running "if" statements in delivering your service, the better.

The use case of delivering the service to a consumer of the service happens all the time, the use case of "oh, I didn't understand how this works and foot-gunned" is relatively rare.

AWS can eat that cost less expensively than maintaining the "if" statements inline on every request.

The most reliable code is code you don't write at all.

<comment><USER21> You can run an awful lot of if statements in a microsecond these days.

<comment><USER32> At scale, every microsecond matters.

But it's less the CPU time, and more the complexity.

<comment><USER21> That depends entirely on if the microsecond is in the embarrassingly parallel part of the workload or in the sequential part.

</comment></comment></comment></comment></comment><comment><USER33> Oh boy this why I love digital ocean ! There is never any surprize. Simple and intutive UI. Free alerts, free firewall. you can see all resources on single dashboard. not like aws where you forget to terminate instance from different region and screwed up in one night.

</comment><comment><USER34> I wonder if reserving all available services on a hacked account is the next DOS attack.

<comment><USER35> Already been done for a Defcon talk. Limits were added.

</comment></comment><comment><USER36> Not that I'd want production infra to be wiped because I went over a spending limit by $1 - but can't you setup a billing alert that goes to an alarm action to terminate all your EC2 instances as soon as the billing alert triggers? Or a lambda function that iterates through your account and deletes everything?

I'm not sure why AWS would build something like this though themselves. To stop spending money EVERYTHING must be deleted (s3 / glacier / etc). If someone in accounting loads the wrong budget amount you lose all your data.

Amazon's focus seems more on making sure your data is kept and available.

<comment><USER37> Biling alerts are not real time.

\- Some guy

<comment><USER36> Good point - I think they are on a 6 hour average delay from what I've seen. An area for improvement it would seem for sure.

</comment></comment></comment><comment><USER9> At what point would a VPS no longer meet your needs? Let's say a standard startup, not something that will reach TikTok traffic, or something like Slack.

</comment><comment><USER38> The easiest way I can think of is to got commit your AWS keys to a public repo.

(This actually happened to me a while back - a new employee created a 201st repo, which our private repo paid limit was 200, so the new repo was automatically made public and he had keys in his code. We had thousands of bitcoin mining instances launched using the keys and cost $75,000 super fast.

We caught it really quickly and AWS dropped the cost.

</comment><comment><USER39> It's worth mentioning here that if you find yourself accidentally running up an AWS bill you can get in touch with their customer support, explain the mistake and ask for amnesty.

I once accidentally ran up a $2.8k bill and after explaining the situation they added a credit to my account to cancel out the charge at the end of the month. They obviously review it case by case but it's definitely worth a shot.

</comment><comment><USER40> Why do none of these cloud services have spend caps?

Why not have a simple "budget" setting and have a setting what to discuss when it's exceeded. Options could be "shut down most recently started resources" or "shut down everything but don't delete any data", and "delete everything".

<comment><USER41> Azure has this in their Cost Management page, available for each subscription. You can set a budget and get notifications about when you are reaching its limit. Not sure about the power it has with re to shutting things down.

<comment><USER42> I believe you can hook into apis to shut things down, but it's not super straight forward.

There are many Azure services that can't be set to zero dollar billing without data loss, so I'm not sure how Azure could deal with those in a unified manner.

</comment></comment></comment><comment><USER43> I turned on a POC of a platform on AWS last month that stood up 30,000$ worth of AWS spend for it self... now where near this but the ease with which it came online was terrifying

</comment><comment><USER44> this part made me laught:

> Our next candidate is Azure Databricks. I have no idea what it is, and I > don't even care! All I know is that it's pretty expensive, and that's > exactly what I need

</comment><comment><USER45> This is hilarious.

But serious question: the original tweet is about a single instance that costs over 3 million dollars.

Is that genuinely a single _physical_ instance? Like is it even technically possible to build a traditional single physical server of CPU+RAM+disk that costs that much?

Or, being a database server, is it some kind of clever abstraction that actually splits it up physically (e.g. relying on the fact that different database threads might be able to get away without sharing memory)?

<comment><USER46> I'm not sure exactly what processor AWS uses, but assuming you match the db.r5.24xlarge's 48 cores, the Xeon Platinum 8160 is the closest match for the processor - ~$5k. The ram is likely ddr3 16gb ecc memory, so 768gb at $50/stick (you might be able to find a better price) is $2400. As for disk, the price of the instance doesn't include disk size since it's an EBS-only server.

The actual price comes from SQL server enterprise edition. A mysql db.r5.24xlarge multi-az prorated for 3 years is $219,551, which is ~$600 less per month than on-demand pricing. However, sqlserver-ee is 2,782,588. I believe this comes from per-core pricing for sql server. [https://docs.google.com/spreadsheets/d/e/2PACX-1vQZT7wl1yvav...](https://docs.google.com/spreadsheets/d/e/2PACX-1vQZT7wl1yvavctIYchaxHAI_BEct1lKRnl3A_Z8D9xBK-u1eqxy__S9w2kB9BucnLq8iZIYb- YGUZiN/pubhtml?gid=0&single=true)

Note - all prices were North Virginia, the original tweet was regarding servers in Bahrain.

<comment><USER45> Oh wow. That makes much more sense then, that it's mostly SQL Server licensing.

I was vaguely aware that some server licensing is per-core, but I never realized it could add up to costs at that level.

Thanks!

</comment></comment></comment><comment><USER47> A co-worker created a website for a charity and got some free Azure budget for it that would have been plenty for a year. After half a year, though, it was gone because something innocuous that he'd accidentally activated ate up all his budget.

Sorry I don't have more detailed data than that, but it certainly alerted me to the fact that you need to be really careful with these cloud services.

</comment><comment><USER48> Could people familiar with the "Cost Management and Billing" functionality on both Azure and AWS, share here relevant coherent thoughts (i.e., mini-review) on feature comparison, pros/cons, etc. for these two platforms? I'm especially interested in this from the perspective of a multi-tenant SaaS provider/vendor.

</comment><comment><USER49> Well, not sure if this counts or if it's a cheat, but you could do the following: Find the single most expensive click, and then a consulting company to implement it, and click _their_ button to sign the contract; Still just a single click and Boom: 20% to 200% increase over Azure cost alone.

</comment><comment><USER50> Existential question about cloud usage: at what point does the risk + cost + knowledge/consultation + DevOps staff required to efficiently manage a cloud provider outweigh the risk + cost + staff of running your own data centre?

</comment><comment><USER51> If your startup infrastructure scales "to well" then the new attack is not DoS but a CoS - cost of service attack. Trigger a cloud bill so high it bankrupts them!

</comment><comment><USER52> I wonder if they have started courses in 'cloud accounting for engineers' at universities yet? Snark aside, it would be useful for business oriented IT degrees.

</comment><comment><USER53> Just quickly glanced over some comments on this thread, found it's a bit funny that most comments are about AWS, mean while the original artichle is about Azure.

</comment><comment><USER54> I burnt $5,000 by putting some servers in the wrong region. Network traffic within the same region was free, but across regions was not.

</comment><comment><USER55> This is the reason why billing alerts exist. Just use them. You can't make joke of money.

<comment><USER21> I know this is common advice, but I expect history will very justifiably tag this kind of talk as victim-blaming.

I think fundamentally small and middle-sized companies are building websites improperly (cargo culting companies that can afford million dollar mistakes), which surely exacerbates the situation.

But we used to be able to count on a person providing a service not to overcharge you because it was bad for business. When they didn't, it was news.

</comment></comment><comment><USER56> Magic</comment>


Data Broker Lets You See Their Dossier About You Starting Today - <AUTHOR> http://www.forbes.com/sites/adamtanner/2013/09/04/800-pound-gorilla-among-data-brokers-lets-you-see-their-dossier-about-you-starting-today/ <comment><USER2> "Acxiom offers marketing and information management services, including multichannel marketing, addressable advertising and database management. Acxiom collects, analyzes, and parses customer and business information for clients, helping them to target advertising campaigns, score leads, and more." [0]

An advertising company wants me to enter my name and address and make sure the database they sell has the correct information on me.. hmm... I think I'll pass.

0\. [http://en.wikipedia.org/wiki/Acxiom](http://en.wikipedia.org/wiki/Acxiom)

<comment><USER3> Last 4 digits of SSN is particularly flagrant. I could almost believe (if I wanted to give them the benefit of the doubt, which I don't, particularly) that they are using name and email as good-faith identifiers to let me opt out. After all, they need to identify me with _something_. But what could they possibly be doing with SSN unless they have offline credit score information obtained from some third party, which they couldn't previously link confidently to my advertising profile, and now they can?

<comment><USER4> Is a 4 digit SSN number suffix really enough to be helpful to them in that regard? I guess they could find all "jcater"s with those 4 digits and make assumptions, but that is far from unique.

<comment><USER5> Yes.

ZipCode + SSN is enough to identify you, even without your name, close to 100% of the time. Toss in your name and I'd be amazed they'd ever got a false positive. I'm [honestly] not sure why they even bother asking for a mailing address other than to go from 99.9% confidence to 100%.

Fyi, there is ~8k people [on average] in a zip code. The odds of your name and SSN repeating inside a zip code is low. Name tho? Name could happen with reasonable odds. I used to know someone in my zip code with an identical first/last [not middle] name.

<comment><USER6> I'm going to give you the benefit of the doubt and assume you misread the post you are replying to, which referred to the last 4 digits of a SSN, and incorrectly assumed we were talking about full SSNs. Because otherwise your statement is absurdly wrong.

If there are 10,000 possible 4 digit combinations and 8,000 people in a ZIP code on average, the odds that someone else will share the last 4 digits of my SSN is 1-(1-1/10000)^7999 which comes out to 55%. So in reality, more often than not, you will share the last 4 digits of your SSN with someone in your zip code.

Also, there are a few zip codes with 100,000+ people in them, and many zip codes with 50,000+ people in them. These zip codes account for many millions of people. You are guaranteed that nearly every resident will share the last 4 digits of their social security number with at least one other person in these zip codes.

<comment><USER5> Yes, I was rushing and didn't read all that clearly.

</comment></comment></comment></comment></comment></comment><comment><USER7> From my account it is pretty clear who is sharing data with them. They have 5 purchase records, and think I am into furniture and linens. Overstock.com and Jennifer Convertibles are guilty of selling out their customers.

Amazon is clearly not selling them information.

Overstock.com is now on my douchebag list.

<comment><USER8> I noticed it keys off of address and name for purchases. Created two accounts, one is home address which i never ship to and the other is my girlfriends house which had most my purchases except high value items which i ship to work. First account was completely empty on data. Second account had information filled out, seems to use purchases to determine interests. The vendor in question was one of the largest party supply e-commerce stores.

</comment></comment><comment><USER9> I work with a client that uses Acxiom data.

My contacts say this is a real and legit attempt to get out in front of potential regulation. Because of the regulations around credit data, it is derived from a variety of non-credit sources. The only information they are likely to use from this site is the email address. In general, they do not trust user-provided information.

I think the reaction they are hoping for is "I guess that's not so bad" but what I think they are going to get is "wow, you don't know anything about me."

Oh, and they really do honor the opt-out stuff. So few people do it that it actually improves the response rate of advertisers because the people who care enough to opt out are never going to respond.

I wish they put a hit counter on it. I'd love to see how much traffic they get.

<comment><USER10> One of the companies I work with uses various data brokers. They have more than you realize. Especially if you have a vehicle and a house.

<comment><USER9> Only if you have a loan. The regulated credit data is pretty solid. The unregulated stuff.... not so much.

</comment></comment></comment><comment><USER11> So in order to see the information they may or may not have on you they ask for your name, full address, last 4 ssn, date of birth, and email... if they didn't have any data on you before they do now.

<comment><AUTHOR> I had precisely the same reaction in the comments of the article. [http://t.co/g6LpJvdGzL](http://t.co/g6LpJvdGzL)

<comment><USER12> This reminds me of the fake IQ quiz games, where if you take the test, you fail.

<comment><AUTHOR> You'd think one of the leaders in the personal data broking field -- online and off -- would be able to go about this a bit less ham-fistedly.

But I'm also surprised as many hacker newsers have submitted their info as it sounds. So, if hamfisted works, hamfist.

</comment></comment></comment></comment><comment><USER13> I can't believe they bought off Forbes to market this scam. This is not about data disclosure, this is about collecting even more data in the name of "educating" you. You will be giving away most important pieces of your information to them in trying to know whatever half ass information they might supposedly have. If they really wanted to disclose fairly then they can just ask for email and do verification by sending link in email. They shouldn't need birth date and precious last digits of your SSN.

<comment><USER13> Before anyone types away their information, you _should_ read this paragraph. This is a huge scam, a clever one:

 _In the privacy policy for the new site, Acxiom says it will allow users to edit and suppress information about themselves. However, to see their file, users must give up personal data and pass an authentication exam. That means giving your address, email, last four digits of your Social Security number and date of birth. Can they use that information? "The information you provide may be shared within the Acxiom Corporation family of businesses," the company says._

</comment></comment><comment><USER14> I highly recommend requesting a LexisNexis Risk Solutions Full File Disclosure [1] at least once a year. You have to mail in copies of an ID and proof of address, but I found signs of identity theft that didn't appear on other credit reports.

[1] [https://personalreports.lexisnexis.com/access_your_personal_...](https://personalreports.lexisnexis.com/access_your_personal_information.jsp)

<comment><USER15> Interesting. Are there services (like LifeLock or something) that will aggregate these types of information for you regularly automatically?

</comment></comment><comment><USER16> I've worked with somebody who came from Acxiom's grid computing division -- who has left after they were sold of to EMC. Essentially they had a fairly advanced grid computing infrastructure (comparable to Map/Reduce or other Scatter/Gather) for what seems like a while before Google has. Compared to what was then (2007) Hadoop's State of the Art (mind you, we were working together at Yahoo) it seemed quite impressive.

Now I wonder if the reason for their sale was prescience about open source grid processing -- which would have been against the grain at the time (this was before Hadoop was universally adopted even at Yahoo).

What's also interesting is that the sale indicated they also believed that they were to have no competitive advantage in further development/deployment/maintenance of data infrastructure. This is unlike the Internet firms I've worked at where even if the entire stack was open source, continued in-house development and operation of the stack was vital to the business. At first that surprised me, but then again compared to, e.g., Internet advertising data their data is never anonymous, much cleaner, and smaller in absolute volume (number web pages each unique client visits in a week -- which is a number greater than each unique individual -- visits vs. number of items each individual purchases from a store in a week).

<comment><USER17> Wasn't mapreduce developed at google around 2003? From what year are you comparing Acxiom's system?

<comment><USER16> As far as I understand, they've been doing grid computing since the mid-90s at Acxiom (the person I talked to joined in 1996/1997, afaik).

</comment></comment></comment><comment><USER18> Instead of signing up and cleaning their data for them, or "opting out" by installing their special cookie which continues to track you around the internet, opt out the right way: block all ad networks via a browser extension like Disconnect or Ghostery.

More options and tools here: [http://lifehacker.com/the-best-browser- extensions-that-prote...](http://lifehacker.com/the-best-browser-extensions- that-protect-your-privacy-479408034)

<comment><USER19> Not really. What most opt-out cookies do is that they have a "DO NOT TRACK" or "IGNORE" in the cookie fields.

Let's say you run a DMP, you think you can be smart by tracking people with cookies that have a "DO NOT TRACK"==True field.

Very soon though, you will be deluged by more noise than signal, and it quite literally becomes useless. It's honestly easier for companies to not track people who have a opt-out cookie.

Source: Wrote a prototype DMP for my company once

<comment><USER18> I was curious as to what Acxiom's cookies actually look like after opting out. With a fresh Chrome Igcognito session, following through the opt out process, I get 20 cookies. Plenty of unique GUIDs perfectly capable of tracking me through the internet, and no occurances of "DO NOT TRACK" or "IGNORE". Screenshot here:

[http://img443.imageshack.us/img443/950/d2ys.png](http://img443.imageshack.us/img443/950/d2ys.png)

Also, can you define DMP? I did do some googling first, and I'm not sure which acronym you're referring to.

EDIT: Found it. DMP stands for Data Management Platform. For anyone else: [http://www.adopsinsider.com/online-ad-measurement- tracking/d...](http://www.adopsinsider.com/online-ad-measurement- tracking/data-management-platforms/what-are-data-management-platforms/)

</comment></comment></comment><comment><USER20> I'm 21, and apparently married with an 8 year old daughter. I also drive a truck, own a home that I bought in 05/2008, and have an amex card.

Interesting that my profile is so specific, yet utterly incorrect.

<comment><USER21> Very interesting, you might want to check your credit report, your identity may have been stolen..

<comment><USER22> Also possible, their algorithms are incorrectly linking data from another person's records. Still a good idea to check, of course.

</comment></comment></comment><comment><USER23> I would not give this company the last 4 of my SS to see what they know about me. My SS and the confirmation information they are seeking is worth far more to them than any info they will give me.

In fact this whole site reeks of a scam, similar to a "Free personality test" or "Love IQ" test online that is really just a way to gain valuable personal information, except instead of playing off social anxieties to trick people into taking a test they are playing off privacy anxieties to get people to divulge more information...

</comment><comment><USER24> Looking at my data, it's so so so tempting to opt-out, but then again I'd rather see targeted ads than 'one crazy trick to get rid of my belly fat' ones....

<comment><USER25> Care to share what kind of info they had and how accurate it was generally?

<comment><USER26> Mine was pretty bad. They had some very basic pieces of public information wrong.

This is clearly not the same data I had when I used their web service at a startup I worked at in 2000. They had a service you could provide a phone number to and it would return a little story like this:

"Bob Smith from Portland, Oregon uses in-home pregnancy tests, drinks imported beer, and listens to alternative rock radio stations."

None of those sorts of rich details are in my data.

</comment></comment></comment><comment><USER27> _Finally, an excuse to relate an Acxiom anecdote!_

As a budding CS major I had a job with Acxiom. Most of it involved running phone books through sheet fed scanners, collating a truly massive amount of publicly available information. A lot of work was done on a custom OCR engine to parse all of this data.

<comment><USER25> >running phone books through sheet fed scanners

Wow. I had to take a minute to let that sink in.

</comment></comment><comment><USER28> None of the data they had about me, besides what I had just entered, was correct. This, I suppose, is encouraging.

<comment><USER29> You may have inadvertently improved their accuracy by 100% as a primary source.

<comment><USER30> Indeed. I would have run CCleaner first and use IE (which I never use) to make sure they didn't get any new info out of me. The only thing they could get at that point is my IP address unless I went through a trusted VPN.

</comment></comment></comment><comment><USER5> Ya, just scroll to the bottom and 'install the opt out cookie' is what I did. No clue if they actually honor it but figured worth a shot. ;)

No way in hell I'm giving them my social, address, etc. XD

</comment><comment><USER31> Can't authenticate me based on my information (steady address for 2+ years, on credit reports, drivers' license, etc)... and to fix this I can photograph and encrypt my DL/passport?

:(

</comment><comment><USER32> to see what others may know about your magazine subscriptions, enter your social here ___-__-____, and address here __________, (which we'll share with our trusted partners)

/s (kinda)

</comment><comment><USER33> At the bottom of their new user signup page is a link to an "opt out" section. If believed, they allow you to install an online cookie and mail them forms to stop Acxiom from tracking you online and selling your data.

[http://acxiom.com/about-acxiom/privacy/us-consumer- choices/](http://acxiom.com/about-acxiom/privacy/us-consumer-choices/)

</comment><comment><USER15> _" instead of getting a great offer on a hotel package in your favorite vacation spot, you might see an ad for the latest, greatest weight loss solution."_

Translation: "Fuck you, we'll show you obnoxious weight loss ads instead"

</comment><comment><USER34> Well, I'm happy they're totally wrong about me. Married ?! Two children ?! German ?! At least they got Male right...

</comment><comment><USER35> I also do that. I can tell you everything about you, if you want to test me, please send me:

Your full name.

Your date of birth.

your zip code. Your street. The number.

your social security number.

Your marital status. Please specify the sex of your kids and also their ages o I could contrast with my database better and inform you if you are wrong.

Your job.

This is a classic storyteller or talker with your ancestors trick.

</comment><comment><USER36> > _They have files on 700 million individuals_

Allright, let's sign up...

> _State: ZIP:_

So... USA only? Okay, let's see [http://www.census.gov/population/www/popclockus.html](http://www.census.gov/population/www/popclockus.html)

> _316,606,919_

Not quite 700M, calling BS.

<comment><USER37> That's a 300M population...today. Assuming avg US life expectancy of 75 years, that's another 4M in turnover for every year that they have data for, plus I assume records of another ~150-250M records from those that have only stayed in the US temporarily and aren't reflected in census numbers.

</comment></comment><comment><USER38> I just entered my info. The have my birthday wrong, think I am married and have a kid. They also got my credit card types wrong (should be the easiest thing to collect too), and were way off on my purchasing behavior. This is not that scary.

</comment><comment><USER39> Nada. Zip. Zilch. I know it's a good thing, but kinda disappointing in a way.

</comment><comment><USER40> Don't do it! I entered all that information and it said I needed to manually authenticate. Basically, they want me to provide even more information. Can't opt out and I just gave them more info...

</comment><comment><USER41> Pointless. They had no data on me. Now I've given them more than they ever had on me and their "opt-out" maze doesn't delete my info. _face-palm_

</comment><comment><USER42> > If you want to get the best advertising delivered to you, based on your > actual interests, start here.

They make no compunction, though, that's not what I want.

</comment><comment><USER43> let me review this... an advertising company wants even more data about me... remind me those anti-virus software... let me scan your computer for free and boom even more spywares .... IP+browser info+SSN+full name+ location....that's gold !!! don't know how many people is going to complete the form but that's a great marketing campaign... scare people to create a sense of urgency...

</comment><comment><USER44> Good to see how inaccurate, sparse, and almost completely useless to marketers my record is. Being an immigrant on a work visa has its advantages!

</comment><comment><USER45> All they had were my vehicles. They also think I'm a 72 year old African American. Nothing on my home, shopping, etc.

</comment><comment><USER46> I'm not going to sign up to opt out. I have a feeling it is going to trick me to give my personal data myself.

</comment><comment><USER47> Well thankfully they have literally no data about me, apparently.

</comment><comment><USER48> "It's a trap" \- Ackbar

</comment><comment><USER49> Submitted new user form at www.aboutthedata.com/... "We'll fix the error soon!" WTF? Is this a phishing site?

<comment><USER46> seems like you got tricked :(</comment></comment>


Ask HN: What do you like or dislike about goal setting? - <AUTHOR> <comment><USER2> This is timely as I was thinking about this yesterday. I was reading about goal setting and reviewing.

I think the easy part is the setting, the reviewing is more of a challenge.

Review your goals once a week? Ok, great - I have to pick a time and remember. Pretty minor thing, why don't I just schedule a weekly reminder alarm? Yes, why don't I...

The thing is, for some reason, it just doesn't get done.

I do believe the solution is quite simple. I put my goals somewhere, and something reminds me to look at one or two. Maybe I prioritize them a bit and set goal dates. Do I need sub-goals, let's not go crazy just yet.

I'm assuming you are looking to create an MVP? I think the challenge of a site is to say 'come back and review your goals". Oh, I can hear your response already "I'll just send you an email or notification to remind you", and you will probably have lost half the your target if you do that.

I don't have a solution, I think there is a nice opportunity here, and a nice way to learn about people.

On a side-note, I created an MVP app last xmas ([http://bucket52.com](http://bucket52.com)) which was less about setting goals and more about creating memorable events, but you're in the right timing for goal setting. If you hit around xmas/NYE with a goal setting app, it is a good time to get things to spread.

</comment><comment><USER3> I take solace in goals and goal settings is what I call "a 2,000 year problem", meaning humans have been grappling with problem for at least 2,000 years without full resolution.</comment>


Seven ways the world is not designed for women - <AUTHOR> https://www.bbc.com/news/world-us-canada-47725946 <comment><USER2> The rubber gloves at the supermarket are way too small for man hands. So there.

</comment><comment><USER3> This article is trash.</comment>


Google Hangouts Outage and G Suite Status Dashboard (Discussion Prompt) - <AUTHOR> Apologies if there is already a thread submitted by now.
It appears as though Google has been suffering some heavy outages for the past 30 minutes, and yet their G Suite Status Dashboard remains unchanged.  This feels similar to what happened when AWS experienced their historic outage several months ago whilst their dashboard did not indicate any service problems.
Is there a particular reason why such dashes are so slow to update?  What is the point of such dashboards if they do not update within a reasonable time?  Here, defining reasonable time as 30 minutes, although I think I'm being generous.
Link to G Suite:  https://www.google.com/appsstatus <comment><USER2> Still down and having clients call me about it. Googleapis keeps throwing a 503 on and off. Why does the status page not update? How is this 99.999999 uptime?</comment>