How To Get Media Coverage For Your Startup: A Complete Guide - <AUTHOR> http://onstartups.com/tabid/3339/bid/80121/How-To-Get-Media-Coverage-For-Your-Startup-A-Complete-Guide.aspx <comment><USER2> While the OP raises some good points, the letter which got them mention (on Mashable) doesn't even talk at all about what "buffer" is.

For example this paragraph ("paragraph one"):

"We have some big news for you. With the new release of Twitter.com, we have just released a way to post retweets via Buffer right from Twitter.com at a better time. It works seamlessly via installing our browser extension. You will have now, next to "reply" "retweet" and "favorite" a new option to "Buffer"."

Well that paragraph makes the assumption that the writer already knows what buffer is (and I will assume the website address (bufferapp.com is located in the email header).

While the writer (if interested) will certainly visit the site, a short description of what buffer is is essential.

My experience with doing this over the years (with mention in the WSJ, NYT, (multiple times) as well as online media (and blogs) has yielded the following formula:

1) Send occasional emails to writers before you need them.

2) In those emails compliment the story and add your perspective.

3) Invite the writer to contact you if they are doing stories on (something you know about) that is, to consider you a resource. By doing #1 your name will be in front of them and they might remember you and contact you.

4) Do this over the course of time. Then when you want to pitch the company they will pay attention to your email and message (which of course still needs to be well written) because they are familiar with you.

That is what has worked for me.

<comment><USER3> "What buffer does" is right there in the text you quoted:

> a way to post retweets ... at a better time.

<comment><USER2> Nope. Not clear enough for someone busy who doesn't want to think.

This, from their about page (it should really be on the home page as well) is the type of clarity you need when pitching:

"Buffer makes your life easier with a smarter way to schedule the great content you find. Fill up your Buffer at one time in the day and Buffer automagically posts them for you through the day. Simply keep that Buffer topped up to have a consistent social media presence all day round, all week long."

And, this is really the key point:

"Simply keep that Buffer topped up to have a consistent social media presence all day round, all week long."

</comment></comment></comment><comment><USER4> SAI, former VentureBeat reporter here, few observations:

1) What is Buffer? If I don't know what the company does, I'm going to pass it off to the tips folder and it will probably die in obscurity.

2) The "saw the post you wrote yesterday" is not the approach you want. I cover social games — be aware that I cover social games and make it relevant to that. Reading my last three posts is not enough to tailor a pitch, because we write about a lot of things and they aren't all necessarily on our beat.

3) Seriously, introduce yourself. Don't do it with a pitch — let me know who you are and what you do. We do Q&As all the time with companies that have no news. You are 100x more likely to get published if I know who you are, what you do and why what you are doing is important.

4) How does this relate to normals? Most Twitter users don't actually Tweet. This is useful to me, but not necessarily my readers — who are of utmost importance to me.

5) Know the publication you are pitching. Blind pitching everyone is a waste of time — if you are a social game developer, go for Inside Social, GamesBeat, TechCrunch, etc. Fast Company is not going to listen to you.

6) We have to rush through literally hundreds of pitches each day. Just because we phrased something differently than you'd like, if it is still factually accurate, you're going to upset a reporter if you try to nitpick on wording. I once encountered a PR person that was yelling at me because I didn't call an online deals site a "mobile platform" and, instead, an "online deals site."

7) Also, we do want to know what you think. Entrepreneur and founder input is valuable in just about any story. If some big news happens — Steve Jobs resigns, for example — let us know what you think. When Jobs passed away, the first person I heard from was Box.net's Aaron Levie, who told me it was such a gargantuan loss and he was basically his idol and what made him want to become an entrepreneur. That, in of itself, is a story because Levie is running a company with a val of more than $500 million.

8) Don't be afraid to have a personal relationship with reporters — we aren't going to screw you, because blowing a relationship is worthless in this industry.

I'll add more as I think of it.

<comment><USER5> Really appreciate the feedback on the post, I'll be sure to implement them.

Re 1.), you are right, the writer already had heard of Buffer beforehand, I should have made that clear.

Awesome stuff regarding 3.), I think I'm not doing this enough by far and will do it next time!

Anyways, really great to have these thoughts here, I think it'll be an awesome addition, for Buffer's next round of news coverage. :)

</comment></comment><comment><USER6> ShoeMoney wrote a great post about getting press... A must read if you are serious about getting press...

[http://www.shoemoney.com/2010/09/14/getting-press-for- your-w...](http://www.shoemoney.com/2010/09/14/getting-press-for-your-website- application-or-service)

<comment><USER7> > Last night, one of my really good friends sent me a email and asked me if I > could call up Mike Arrington over at TechCrunch or Pete Cashmore at Mashable > to cover their new iPhone application.

> (Trying hard not to sound like a dick) I basically told him that is not how > it works.

That is _exactly_ how it works. A huge number of stories that bloggers write come from a small number of sources.

The best way to get published it to become a source and a connector, or to know one. It isn't that different to how getting an investment works - It is about being a participant in the ecosystem.

If you have been written about by a blogger or reporter before don't be afraid to introduce other startups to them that you find interesting - most good bloggers will press you on doing this. This is how people become sources and connectors.

<comment><USER6> This is actually exactly his point...The best way to get coverage is to nurture a real relationship with that reporter.

You are focusing on the self-serving part of this post, which is Shoemoney doesnt want and wont put his relationship on the line for your product or company...Especially because he gets a million requests just like that.

Sure, you can try to convince your friends who have the relationships to get you coverage, but your best bet, and the way your friends probably developed the relationship is by nurturing a real relationship with these reporters.

</comment></comment></comment><comment><USER8> Does anyone else find the whole process of engaging the tech "media" a giant distraction from developing their core product and trying to engage real customers? Chances are if you have product that sounds "cool" to these companies you are going for a market that you are unlikely to win.

You are are far more likely to be successful targeting a business that is outside the consumer "cool zone". The consumer markets are so heavily saturated there's little chance for new products to really be noticed, and individual consumers aren't as profitable as targeting businesses unless you are lucky enough to be the next Facebook (you aren't.)

Unless your product customer is likely to be a hacker/geek/tech biz type that would read Mashable/Techcrunch/RWW/etc.. Don't waste your time trying to get coverage there, instead find reliable channels to talk directly to your customer and route them to your service.

Find heavy's in that vertical and appeal to them, especially ones that have a lot of influence (online or IRL.) Find blogs that match your vertical and get them to cover you, they will probably be happy to do it, and they probably will send you more link juice because they don't post 100+ articles a day.

Unless your PR is sending you real SEO heavy links, and/or putting a giant dent in the collective consciousness of your potential customer base, its worthless.

</comment><comment><USER9> "Really loved your post yesterday on how..." There's useful advice in this post, but recommending people turn the first paragraph in their email to a blogger/reporter into something akin to comment spam may not be your best bet if you want to get media coverage.

</comment><comment><USER10> As far as who to pitch and who not to pitch, I think it's worth a mention that a startup will have more success pitching the blogs in its own niche before going after general tech blogs.

For example, if your startup is a service based on WordPress, go to some of the popular WordPress news blogs and influential WordPress people before you try to pitch Techcrunch and the like.

</comment><comment><USER11> Excellent article, very beneficial with some good tips. Plus it brings out some actual writers here to give their insider knowledge which is really nice.

However, I wonder if this will cause the strategy and tactics to become less successful, at least in the short run. And by that I mean, if every entrepreneur that reads the article goes out and those this, it's just going to become the same noise that the reports see every day and they'll learn to ignore it...

What I expect to happen is that there will be an uptick of this type of activity by start-ups that are looking to get some publicity for their companies. It will probably last a month or so, and then die down as they realize that no, it's not really easy to make these connections and to make them be really successful you have to be genuinely interested in the writer and his stories, Think of it as the HN bump.

Any writers seeing this already? Or am I way off base?

</comment><comment><USER12> The article mentioned getting contacts in the press. I'd like to mention that I run a news site that also gets published in Google News, and often write and post all sorts of press releases.

News stories about your startup are possible too. Contact me via my email address in my profile for details.

</comment><comment><USER13> If we've learned anything from this week it's post something sexist to receive a lot of negative press (as it should).

</comment><comment><USER14> Solid advice. I launched <http://blisscontrol.com> a few days ago and it's been covered by Lifehacker, TechCrunch, Cnet and plenty more blogs. Most of the tactics mentioned in this post were ones that I used so definitely recommend using these tips.

<comment><USER15> How did you go about making first contact with TC?

</comment></comment><comment><USER16> this was a great post, and the fact that much of the information is repeated elsewhere in posts indicates that 1) there is a scalable strategy for pr that works and 2) not enough people are using it! i covered some of my advice for pr for startups here, as an extension to my post on mixergy on pr for startups [http://startupstella.com/2011/10/14/dos-and-donts-of-pr- for-...](http://startupstella.com/2011/10/14/dos-and-donts-of-pr-for- startups/)

</comment><comment><USER17> Love the detail in this. Nailing the pitch and finding who to send it to are so obvious, yet most people don't know how to properly do so. Great advice here.

</comment><comment><USER18> Do you think its better (for media coverage purposes) to create a blog for your startup or to simply create a blog for yourself?

<comment><USER19> Your startup should have a blog, to connect with your users and potential customers.

</comment></comment><comment><USER20> I'm biased, but this is one of the most down-to-earth startup PR articles I've ever read. Practical and useful.

Nodded my head a bunch of times when I read it -- which is why I agreed to run it as a guest post on OnStartups.

<comment><USER21> Thanks for helping us find it! :)

</comment></comment><comment><USER22> Just get on Shark Tank. Worked for MisoMedia.

</comment><comment><USER23> Ah, a post I can comment on with authority. I'm a local reporter with a heart for tech. Perhaps I should blog a little more about getting media coverage for young (and not-so-young) entrepreneurs.

At least for local papers (and probably TV too), if you're not local, don't bother with emailing them. They'll pretty much disregard your email altogether (You probably WOULDNT be surprised by how many emails we get in a day begging for coverage for this or that. But it is a lot. Probably a few dozen for a midsized paper).

Find some way the product is really cool. And I'm talking, COOL. You might get a business reporter to give you a brief mention or a two paragraph something or another, but if you want a full shebang write up, you have to paint a picture as to WHY your product is the bomb. And don't give us your pre-written media shpeal. That might get you an article, but being organic is going to be a much better bet. Personally, I like to HEAR the enthusiasm for your product come through in your email. People who are extremely passionate about their product usually make for good stories regardless of the product. The folks at mashable write on tech for a living. So if you have a novel idea there, they're probably going to write about it. They may not be excited about writing the story, but they'll still do it; it's their job (kind of like whether the education reporter LIKES any given story they're writing about, they still have to do it because it's their job).

I guess if i could sum it up, I'd say talk to a reporter in an email or a phone call like you would tell your best bud about this awesome new thing you've got going on. In short, why are our READERS going to give a crap about your product? Keep in mind, we write stories. So we like to hear interesting stories. Maybe how you and yoru cofounders found each other or how you came up with the idea (a big maybe). yes you're trying to sell us on covering your product, but don't approach it that way.

(And as a PS, if you have journalist friends, DON'T beg us to write about your product whenever you see us. Just forget we're reporters. If you have something going on that we know about and think our readers might be interested in, We'll ask you about it) ;)

Maybe that will help some of yall out a little :)

<comment><USER2> "At least for local papers (and probably TV too), if you're not local, don't bother with emailing them."

If you're not local then you need to find a user of your service that is local and base the story around them as the reason to write about it.

Say before twitter was famous the story would be "After the fire in her hotel room, Megan stayed in touch with her family in Topeka by a new service called "twitter" which allowed her to..."

You'll note that on traditional media when someone appears on a national TV show from the home town they almost always do a feature in the local paper.

That's the angle that I have taken which has worked in the past.

<comment><USER23> Yes there is a lot of truth to this. But you'll need more than just "so and so in your area uses it." in the sense tho that there has to be a local connection, you're right. But like I said before, there has to be a good STORY behind the pitch for coverage.

National Tech news blogs are a little different. They're going to focus on products and services. Local news outlets want stories the readers will like. (after all, we're in the game to keep readers, which is the bread and butter for our advertising rates).

Edit, now if your idea has a local business angle, pitch it as a business story. Also, logical moron above has excellent advice. I wholeheartedly echo his point number 6, the one about phrasing (I think it was 6)

</comment></comment></comment><comment><USER24> Very interesting and useful. Thanks!!

</comment><comment><USER25> Sensible, practical, and very clear advice - this is a must-read for founders and startup marketers.</comment>


Microsoft fairly and squarely beat Amazon in $10B Pentagon cloud contract - <AUTHOR> https://www.marketwatch.com/story/microsoft-fairly-and-squarely-beat-amazon-in-10-billion-pentagon-cloud-contract-2019-12-03 <comment><USER2> > Disclosure: The writer owns Microsoft stock and holds no shares in Amazon.

On the one hand, this would be a reasonable investment strategy if one sincerely believed in the stuff she's saying. On the other hand, the things she's saying would be strategically correct given her holdings.

A bit of a credibility chicken-and-egg.

</comment><comment><USER3> This article goes completely off the rails fairly fast. It is meant to be an article about a DoD contract but brings up global cloud coverage (for the DoD in the US?), HoloLens (???), and hybrid cloud without a real justification for why any of that would be relevant to the topic at hand.

This is a low quality article with a clickbait title. There's no substance here. It almost feels like an article written by a bot splicing different generic company information together.

</comment><comment><USER4> That is asinine to say AWS doesn't do hybrid cloud, or that hybrid cloud means AWS Outposts. AWS has a whole suite of services for building hybrid clouds and bringing cloud ops to data centers. Hybrid cloud for Azure has mostly been a sales strategy for MS to sell cloud services to their existing customer base. AWS has 50% marketshare of cloud services compared to Azure's 15%.

</comment><comment><USER5> The writer is a typical stock speculator not a real technical analyist. This is just another mesh of buzzwords to get stocks going for momentum trading

</comment><comment><USER6> Fairly and squarely. Pentagon contact. FFS.

</comment><comment> chupa-chups I'm not sure I'd trust something standing the test of time based on Donald Trumps decisions ;-)</comment>


Why we need to create careers for research software engineers - <AUTHOR> http://www.scientific-computing.com/news/news_story.php?news_id=2737 <comment><USER2> Working in research as a software engineer can be very frustrating. I wrote the majority of the analysis code for one of the chapters of the IPCC 5th Assessment and yet was told my name couldn't be put anywhere as I wasn't a 'scientist'. I also consistently see people who have never written much code put in charge of projects where the majority of the work/research is writing software. These projects end up with no code repository use, poor quality control, zero testing, limited OO use - all the things that a professional would take for granted. Worse, they don't perceive the difference between a postdoc who can write code and a professional developer with years of experience. So they hire the cheaper option and end up with the usual 'ball of mud'.

<comment><USER3> Some would argue that no OO use is a good thing (assuming you are referring to object oriented programming).

Expanding on this, writing research code is quite a bit different than writing production code. In particular, requirements can change quickly and drastically. This means that very open designs where you don't hide data (e.g. as is common in OO programming) are often the best designs.

<comment><USER4> If your research code is a big ball of mud then all it's results are meaningless. And all your research should be rejected by any competent journal. Sadly, journals tend to ignore such huge issues.

But, hey it's not like reproduceable results are important in science or anything.

<comment><USER5> The more computational journals are keeping track of this. I've had a manuscript returned for revision because I didn't have enough unit testing in the code. Wet-lab journals, on the other hand, probably let a lot of stuff go on the computation and statistics than they should.

</comment></comment></comment></comment><comment><USER6> This is pretty much my job. I develop climate analysis software at a DOE national lab. We have a pretty reasonable career ladder for the "research software engineer". A quick sample:

Me: Recent graduate (2012), early career. All dev, all the time.

Lead Dev: ~15-20 years experience, minor project management tasks, mostly just dev work. Gets author credit on scientists' papers every once in a while for his help in generating visualizations.

Principal Investigator: 30-40 years experience, created the software package that we develop in the early 90s, almost entirely project management.

Our organization doesn't expect publications from us (though they're a perk at raise time). There are a few conferences that we present at annually, some posters to make, but for the most part, I just write software that helps scientists do their jobs better. I'm just down the hall from some of the leading minds in climate science, and it's really a great environment.

Of course, our lab varies wildly internally; any different project could have a drastically different experience, and different departments would have very different expectations.

<comment><USER7> There are no promotions between entry-level and lead dev? I don't know that I'd call it a reasonable career ladder if the second rung is 15 years away.

<comment><USER8> Are regular promotions that important? As long as I get keep getting to do interesting work and keep getting reasonable raises I'm quite happy with my job title not changing more than once a decade.

<comment><USER7> I suppose not. I'm just used to compensation and responsibilities being tied closely to promotions. There just needs to be some way that the organization regularly recognizes people for doing a good job, so however that materializes is fine.

</comment></comment></comment></comment><comment><USER9> There is a chicken and egg problem here, also.

Many labs don't have any really competent software people in place, which means they don't have effective mentoring and training of new people as they join, which means they don't have a solid mechanism for generating competence in this area (unlike the focus area of their research).

A lot of research code is writing in fits and starts by a revolving door of post docs and graduate students, who don't understand software development, supervised by people who also don't understand software development. The results are really hit and miss, dependent mostly on individuals desire to learn these things outside of their area, or on the luck of hiring the right student or post doc who already happens to know quite a bit (but this probably isn't why you hired them). Some labs know they have this problem but have a structural issue with funding - they can't possibly come up with a competitive salary. Others don't even realize it's a problem.

You end up with a lot of people coming out of these labs with deep domain knowledge but only rudimentary software engineering skill. From an industry point of view it becomes hard to hire many of them because they will need the same amount of supervision and mentoring as an entry level programmer for the first year or two, but if you're running an R&D group you often don't have enough mentors to go around... so there is a shortage of people with 3-5 years of industry experience and an oversupply of those with 0.

</comment><comment><USER10> As a recently departed academic, to vastly generalize, the only people who are research software engineers are perennial postdocs who don't realise yet that their career is a dead end and they're being significantly underpaid for their skill level. They also won't develop their skills adequately in such an environment, lacking good standards and mentorship.

Academia needs to stop taking advantage and accept that it should pay double, so they can actually hire inwards from the private sector and gain good knowledge and skills, or somehow increase the prestige and power to be equivalent to other PI positions. I don't the latter is attainable [1], and I'm sure plenty of people would be happy to just be paid well rather than leave for data science/finance/building web apps.

[1] E.g. If you're in physics, and not doing physics, you're fighting a losing battle to get respect and that lectureship/professorship. Everyone doing coding as their main job has to pretend to do research as well.

</comment><comment><USER11> While I do find it unfortunate that software developers in research are underappreciated, I wonder if making an official title for this will really solve the problem. One issue is that if all you are doing is writing code, you could do the same thing in industry, and there's really no way academia is going to match that kind of salary.

I don't really see the issue with how it appears to work right now though? My understanding is currently if your research requires you to program, then you program it yourself, or collaborate with others. Thus, being a good coder actually helps some people publish more. At least this is what I saw with people in the more practical research topics like systems research, but even in theoretical research.

On the other hand, I suppose this is a problem for people who are not in CS. However, I imagine these people just hire programmers to do the work in the same way they might hire lab assistants. In any case, I'm not really convinced just putting a new title or career track will help.. how is this career track any different from just a normal software development track?

It seems more like the real issue is that you are hiring a postdoc when what you really want is just a software developer, but perhaps they only want to pay post doc salaries?

<comment><USER12> I actually have "research software engineer" on my resume and you're right, it doesn't do much. In fact I don't think this is uncommon, at least in the US. Myself and former colleagues indeed have no trouble working in industry positions.

However one thing to note is that many cases, research programming has specialized requirements that are rare for most industry positions and domain scientists. For example, numerical analysis to ensure stability of models, distributed computing to scale massive simulations or analytics, algorthims and visualizations to explore datasets, NLP, ML, etc. Some of these skills domain scientists develop but for the most part these are likely going to require someone with a CS background.

</comment></comment><comment><USER13> summary: if you're a researcher and spend all your time writing software rather than publishing your research career suffers.

I see three solutions:

1) Have the universities band to together to fund a software source for research. Something that would hire software developers who would provide tools for researchers who were with member universities. Pay a market salary for those programmers and release everything under a sharing friendly license, both for recognition and to aid in reproducing results in other labs.

2) Mandate a single source for all software acquisition through some software consulting company and add riders to all research grants that provide some number of "software supplies" for the team.

3) Get rid of the whole publish or perish paradigm and restructure scientific research so that everyone is accorded rank based on the novelty and repeatability of the research done in the lab.

</comment><comment><USER14> I think the problem is not the lack of "career path" or a title, but that the is little credit given to code. (To the point, that usually no public code is being shared, even if the publication bases on it!) Why?

\- sharing code may mean risk of being scooped (instead of milking the same code for years),

\- cleaning up code takes time, which can be used for writing another publications.

In short, as long as publications (i.e. papers accepted to prestigious journals or conferences) are the only credit in academia, quality programming is lost.

See my longer answer: [http://academia.stackexchange.com/a/23238/49](http://academia.stackexchange.com/a/23238/49)

</comment><comment><USER15> I started studying electrical engineering after being a self-taught software developer for a few years. I'm basically writing all the software for the project groups I'm in, and I love it. I like this kind of code much better than your web app. (Though Matlab is a bitch, no namespaces, array indexes start from 1, etc.) I hope that I can make a living researching and building electronics and software.

<comment><USER16> I've worked both in industry and at a research institution (albeit not very long) and I've learned a bit of a love hate relationship with matlab. It's really awesome for prototyping complex math-based algorithms as an intermediate step before writing a battle-ready implementation. There's a ton of libraries so you can think in terms of high level concepts, and the matrix math is very natural, as is Gpu integration which is a huge plus. Once it's in code, you can iterate quickly relying on libraries, and even create a gui very quickly. When you settle on a final implementation, you can do a 1:1 port to the optimal runtime to make the code more modular, extensible, and fast.

Once you get used to the indexing and quirks, it's a prototyping dream. In a day I can do what would take weeks to implement in C++ or even Python, and I can decide if the algorithm is good enough to warrant a full port. Once I make that call, it's trivial to decide which parts of the algorithm I should work on optimizing for speed during the port because you can analyze perf of each step. Granted, I do a lot of image processing which is sort of an ideal matlab use case.

I totally agree though, writing software to solve real world problems is much more riveting than writing endless web apps (I've done both.)

</comment></comment><comment><USER17> I've interviewed for a number of these types of jobs in the last year or so. The pay is just too poor to even think about it. One large university with a super well-funded and growing medical school in a huge mega-city was looking to hire someone with 5+ years of deep experience in both Python and C/C++, as well as some familiarity or past education in statistics and machine learning, to be the software development lead for a set of research groups. They weren't willing to go above $90k/year.

In another case, I interviewed with a school of public health at an Ivy university. It was in a medium-to-large city on the east coast, where the cost of living was certainly higher than national average, but not insanely high. Even still, they were looking to pay in the range of $75k for someone with significant years of experience in the scientific Python stack.

Not only does this fail to compare even with start-ups that aren't located in major tech hubs, but also when compared against salaries offered by established companies in the same cities, it's almost a 2x pay jump (more in some cases), for the same skill set.

To boot, these lab/university environments tend to have many of the same political issues that any other organization has. Sometimes they are _slightly_ more generous with vacation time and work/life balance, but not always. But you are generally also working with a lot of research staff who see "programming" as a nuisance that gets in the way of their research, rather than seeing programming as the medium of expression of their research. So when you lobby for using good tools, good workflow practices, and try to teach people how to write reasonable software, you usually get a lot of pushback that people just don't like doing it that way, and since you're the super low- status programmer, your opinion generally doesn't count for much, even if you can back it up with more substance.

You also carry some employment risk when projects are funded by grants. Even large, well-established, multi-year grants can be suddenly changed or reduced when there is significant political change in the funding organizations. And a lot of the universities, at least, are unwilling to hire these kinds of expensive programmers as permanent university employees.

You'd probably have a much better career if you instead went to work as a programming-literate business analyst in some part of the university's fundraising / business departments.

In the end, these places can only afford developers with very little experience, or people who don't understand the cost of living differential in the city they are moving to, or I guess are independently wealthy and just prefer to be a low-status programmer in one of these labs.

Edit: Another thing that struck me -- in a lot of these university research situations, the programming talent that they need to hire is genuinely just harder to find / harder to replace than even the research talent you would be working for.

Since there's a huge surplus of Ph.D. and post-doc labor, you could staff the domain expert research side of things much more easily than the programmer side -- especially if you need someone who is truly full-stack to do things ranging from migrating legacy research data to a cloud platform all the way up to designing fancy UI demos with d3.

If you look at another set of industries where the lower-level contributors are much harder to employ than the manager -- professional team sports -- you see that the market dictates that the lower level experts (the actual players on a sports team) almost always command more salary than the coach. There can be exceptions, like Phil Jackson (but even he didn't earn more than his best players) and there will be backup players who earn less. But the general idea seems to hold true.

This means that a lot of these scientific labs and universities will eventually have to face the market reality that the domain expert labor is just not worth as much as the software design labor. Or else they can just keep on compromising on the quality of the software and try to hold on to a worldview where scraping by with bottom-of-the-barrel software development is good enough. It might work for them for a while longer, but I doubt it will work for too long.

<comment><USER18> This is a really good analysis. I think that as it stands, the best hope for filling these jobs is to get a highly talented postdoc who will accept this role in exchange for assistance with his or her academic career aspirations (another way to do it is more insidious, which is where a PI stretches out a doctoral student's completion time to hang on to his or her talented and cheap programmer a few more years).

Of course, this means getting authorship (doesn't have to be first, but it needs to be there), recommendation letters, and so forth. It is also a temporary situation, since what is happening here is that the postdoc is doing software development for well below market rate in order to get a boost in climbing an academic career ladder.

Unfortunately, this scenario makes it even less appealing for a software developer who isn't part of the academic ladder. Because they aren't getting any special boost, career-wise, it just becomes a demanding job with long hours and low pay.

The setup is virtually guaranteed to result in very low quality software. Post docs may be brilliant people, but it's unlikely they will learn to write high quality, maintainable software in a transitional period between grad school and a faculty position, especially since it's something they'd like to leave behind as quickly as possible. And it's equally unlikely that someone talented at software development will stick around in an organization like this where they will never get the chance to lead a project and make a decent salary, especially when those opportunities exist outside universities.

The idea of a research programmer role is a really good one but I'm not sure I can really see faculty accepting it. It would be a major cultural transition for them.

</comment></comment><comment><USER19> Ironically we're all discussing this on a platform created by a research software engineer 25 years ago.

<comment><USER20> Please expand?

<comment><USER21> I assume this is a reference to the fact the www was created at CERN. http was written to help scientists share documents.

<comment><USER20> Exactly, it wasn't a research project. An engineer needed to solve a problem. He happened to work in a research institution. Had he been a researcher the web wouldn't have happened.

</comment></comment></comment></comment><comment><USER22> So universities should hire Software Engineering professionals and have them build tools for the research ecosystem? They can definitely afford it.

</comment><comment><USER23> They mean "programmers who support research projects in other fields", like lab techs. Not people doing research in software.

</comment><comment><USER24> I think I have this sort of career.

I take theoretical paper and turn it into practical implementation. I make money on consulting. Lot of independence and freedom.

I dont think it is possible to do such thing in academia. Software guys I know in astronomy are underfunded and on short contracts. Most guys will leave for commercial company before starting a family.

<comment><USER25> Can you give some details on what you do? I think it's pretty interesting (and rare) that you can make consulting money by taking a paper and turning it into an implementation. What field are you in, who do you work with, etc.? Are you independent?

I'm doing the same sort of thing at Google. I've been a software engineer for 13 years or so and I switched over to a research group in 2014. There is definitely a culture difference, even though Google is one of the places which has the least "pure" research. All research teams have software engineers pretty much, and they mostly work with the same codebase that engineers do.

<comment><USER24> I work on database software. Right now I am working on queues. I am independent. It started as hobby opensource project, people asked me for help and I started charging them. Now it will be 3 years since I started working full time on this.

</comment></comment></comment><comment><USER26> Dr. Pamela Gay wrote an article about the difficult state of non-tenure jobs in research and academia: [http://www.starstryder.com/2015/11/12/dreaming-of- success-in...](http://www.starstryder.com/2015/11/12/dreaming-of-success-in- an-age-of-diminished-possibility/)

It's the same problem for software devs, postdocs, research assistants, adjunct professors - basically everyone except tenured university professors. We're not rewarding the right things.

<comment><USER27> You are missing the career bureaucrats in universities. They have won. The vice chancellor at our university get more than AUD 600 000. Not bad for a small university in the middle of nowhere.

</comment></comment><comment><USER20> I think this is done much better by e.g. Google already today. Academia has have their 50 year run with software research and the end result is quite disappointing.

I have been peripherically involved with some recent academic research projects in Europe. The way I see it it's basically a way of keeping senior academics occupied and also diverting tax revenues to EU companies for no good reason. It's also a breeding ground for corruption - who chooses which companies get these pointless deals?

<comment><USER28> > Academia has have their 50 year run with software research and the end > result is quite disappointing.

You might want to tell that to the students and faculty at MIT, Berkeley and CMU who wrote the papers and much of the software powering your shiny new macbook and iphone.

<comment><USER20> Most of them were students, not academic researchers . The sole reason they were there was that it was the only way to get to computers.

Since you mention MIT: I guess one example is how much effort Marvin Minsky's influence wasted in AI research and how he single-handedly killed more relevant research in that area for decades.

What I was trying to get at though is what I have seen in the EU. Basically they don't even attempt to do basic research - most of the money goes to building product-like software systems, but without any realistic customer feedback loop (they already got the funding from oblivious tax payers thousands of miles away!). It's just wanking.

<comment><USER29> >without any realistic customer feedback loop

This is so so true (I spent some time working as a post-doc in England). There are tables filled up this very moment with dignified scientists arguing about bikesheds in CRUD applications that wouldn't get three upvotes here on Hacker News. And remember: if you can't open it in MS Excel, that makes it "Big Data"!

All joking aside though, what universities are doing with their postdocs is abusive and immoral and they're going to regret killing the golden goose when they've squandered all the prestige in these positions.

</comment></comment></comment></comment><comment><USER30> I thought I was going to read about the real problem with the lack of research into software development. Research software engineers is a secondary issue.

<comment><USER31> [http://herpolhode.com/rob/utah2000.pdf](http://herpolhode.com/rob/utah2000.pdf) is about lack of computer systems research, and it touches on the problem you mention.

</comment></comment><comment><USER32> Would this new career involve some kind of real scientific credit?

</comment><comment><USER33> Did they put software plus engineer in the same sentence. Uh oh!!</comment>


Can home buying be brought into the digital age? - <AUTHOR> http://www.betaboston.com/news/2015/12/17/can-home-buying-be-brought-into-the-digitial-age <comment><USER2> At leas in the country where I live it's all about trust. People have it in their mind that it's more trustworthy to buy from a realtor, and that the realtor is in some way liable for possible problems with the property.

None of this is true in practice however, a realtor has just about no liability in any direction.

It's just a habit for people to use and "trust" realtors, it seems weird for some people if the owner is selling the property himself. That he is possibly witholding information, lying and trying to "gouge" the buyers somehow because he has a "mental bond" with the property.

It's ridiculously easy to sell your appartment or property here, it's all about the image. And you may actually even get a better price using a realtor. So who's winning? The buyer, seller, or realtor? Or is everybody "profiting"? :)

So I don't see this as a technical issue to "disrupt". But if some of these apps would become popular and "accepted", then the trust issues with selling your property yourself would go away (I don't know what the point is of having dozens of "uber" like agents running things though, like explained in the article.). Buying an appartment or house is not about maximizing to the last dollar, but about getting a fair and trustworthy deal with as few problems as possible with the property after the deal is done.

</comment><comment><USER3> My best guess about the direction of the real estate market is that algorithmic valuation, and virtual currencies will allow homes to be bought and sold instantly by eliminating the 2 market problem of buyers and sellers. This will be done through the introduction of a third party which holds the property in an "un-owned" state.

The virtual currency is created for the previous owner when properties enter this state, and destroyed when a buyer uses the currency to buy a property. This currency essentially becomes a real estate backed currency which floats in relation to national currencies as the real estate market fluctuates.

</comment><comment><USER4> Trust, privacy, and convenience (someone assists the touring) are the main problems much more than liability, legal or financing process.

In US, the MLS controls the majority of house listing market, Zillow a pioneer in online house listing has been in constant battle with MLS.

Rental market however could be different as most are younger and apartment.</comment>


Y Combinator Learns Basic Income Isn't So Basic After All - <AUTHOR> https://www.wired.com/story/y-combinator-learns-basic-income-is-not-so-basic-after-all/amp <comment><USER2> A study of basic income with 3000 or even 3 million participants is pointless. If the local sovereign currency issuer isn't on board it's not sustainable. By analogy rent control works extremely well in Germany thanks to the national mietspiegel, but fails (for the general population) when implemented at the submunicipal level. Sure if you have a rent controlled apartment in a desirable area it's great, but it sure hasn't controlled average market rents.

Rent control has the same effect on household balance sheets as basic income. After all, guess where most of that basic income is going to go.

</comment><comment><USER3> [https://news.ycombinator.com/item?id=17851511](https://news.ycombinator.com/item?id=17851511)

<comment><AUTHOR> Oops. I posted this. Thanks for pointing out the dupe.

</comment></comment><comment><USER4> Basic income is conceptually the same as social security. If you want basic income, you should be thinking about lowering the retirement age.</comment>


3 killer features that are coming in Ruby 2.0 - <AUTHOR> http://blog.crowdint.com/2012/11/05/3-killer-features-that-are-coming-on-ruby-2-0.html

<comment><USER2> I'm excited, you can also try it on Heroku as of today! <http://news.ycombinator.com/item?id=4744573></comment>


Show HN: Browsergram, a one hour Chrome extension that Instagrams everything. - <AUTHOR> https://chrome.google.com/webstore/detail/browsergram/nfimhjenbdbaofikdffcinpepgpmcmcb

<comment><USER2> Haters gonna do the thing that haters are naturally best at.</comment>


Nesticle Source Released - <AUTHOR> https://mobile.twitter.com/forestillusion/status/1108489892680470528 <comment><USER2> It definitely was the coolest emulator when it was released.</comment>


Gentoo developers start udev fork - <AUTHOR> http://www.h-online.com/open/news/item/Gentoo-developers-start-udev-fork-1752421.html <comment><USER2> Looking through the email threads I found this: Kay Sievers more or less broke udev for a substantial amount of users by prohibiting kernel modules to load firmware in user-space. Sievers, Hartman, et al consider this reasonable as the drivers, which do this, should be fixed. Torvalds, Cox, et al consider this no good reason to break things. So at least Alan Cox seems to support the forking:

> Just fix udev, and if you can't fix it someone please just fork the last > working one.

[http://thread.gmane.org/gmane.linux.drivers.video-input- infr...](http://thread.gmane.org/gmane.linux.drivers.video-input- infrastructure/49758/focus%3D1368822)

</comment><comment><AUTHOR> Don't miss the link to Greg Kroah-Hartman's G+ page [1], it's pretty funny.

[1] [https://plus.google.com/111049168280159033135/posts/R387kQb1...](https://plus.google.com/111049168280159033135/posts/R387kQb1zxc)

<comment><USER3> Those are priceless.

There is that old phrase "take your ball and bat and go home" which a child was unhappy with how a game they were playing was going they could just leave. Now when someone gets criticized for their design skills they take their changes and try to build an entirely new ballpark, ticketing system, franchise rules, game rules, and seat numbering scheme.

I'd think it was funny if I wasn't dealing with the whole drm/kms/dri/x11/gles battle to see who can build the stupidest way to plug this together for my own, otherwise fun, projects. Makes me pine for Microsoft to port DirectX to Linux, how sad is that!

<comment><USER4> I don't think that the current direction of absorbing everything into either systemd or X11 is the right direction for the Linux ecosystem to take. I find it particularly amusing that, just as Wayland was being created, other developers decided to move input handling out of an independent daemon, hal, and into XInput2.

I hope that the Gentoo developers eventually settle on a single, lightweight udev daemon that does one thing and does it well, so that it remains useful outside of the Gnome+Xorg+systemd desktop stack.

<comment><USER5> hal is long deprecated and never provided input handling. XInput2 is an extension which you can use to get support for multiple pointers and better keyboard state. The input handling is still done (and was for a long time) by XKB in the X11 server, XInput2 is just a different API.

<comment><USER6> In the dim and distant past, I could set the keyboard layout used when X first started (in particular, the layout used for my login screen, KDM) in xorg.conf. Not pretty, but it worked.

Then HAL came along, and I couldn't do that, but there was an undocumented XML file that I could use to set my keyboard layout. Even uglier, but fine, after enough googling I found out how to do it.

Then HAL was deprecated, and now I don't know how to set that keyboard layout and can't find out (after logging in is fine, but having the correct keyboard layout is if anything more important when typing a password than in general use). I asked Matthew Garrett and he asked why would you ever want to change your keyboard layout?

This is why I don't use linux any more.

<comment><USER5> See [https://wiki.archlinux.org/index.php/Configuring_keyboard_la...](https://wiki.archlinux.org/index.php/Configuring_keyboard_layouts_in_X)

Also, why would one ask mjg (who doesn't do X11 stuff AFAIK) instead of googling? :-)

</comment></comment></comment></comment></comment></comment><comment><USER7> Probably the best explanation of the situation (found on reddit)

[http://www.reddit.com/r/fossdrama/comments/13il1o/gentoodev_...](http://www.reddit.com/r/fossdrama/comments/13il1o/gentoodev_can_get_pissed_on_everything_from_names/c74e7n7)

<comment><USER4> Calling a fork "hate-driven development" is unproductive. Further, I found the linked dismissal of Trinity[0] to be as personal, misguided, and "hateful" as it accused the Trinity developers of being.

[0] [http://blog.martin-graesslin.com/blog/2012/02/having-a- look-...](http://blog.martin-graesslin.com/blog/2012/02/having-a-look-at-the- oldnew-desktop-environments/)

<comment> qu4z-2 "[...] the reason that users have trouble with KDE Plasma on "lightweight" systems is not because the hardware is "lightweight" but just old."

It sounds like he's dismissing older hardware out of hand. I'm not pleased with this.

<comment><USER8> My read on it is he's saying that KDE 3.5 isn't great on older hardware either, so what's the point?

</comment></comment></comment></comment><comment><USER9> Slightly OT but hilariously spot-on about many gentoo-isms: <http://funroll- loops.info>

<comment><USER4> Many of the quotes on that page were reasonable questions and statements. Adding a picture of a van with more tailfins than a 1950s roadster doesn't change that.

<comment><USER10> Agreed. I still don't understand why the comments about USE flags are supposed to be comical. I haven't used Gentoo for a while now because the downsides outweigh the plusses for me, but I think USE flags are definitely one of the plusses.

 _Watching shit scroll by for hours makes me a Linux expert overnight!_

No, but when this or that package failed to compile and I didn't even have X yet let alone Firefox (for Google)- _that_ was when I first really started learning about Linux.

<comment><USER11> I don't use Gentoo anymore (mostly because I don't want to break my Macbook Pro's fussy EFI bootsector nonsense) but installing it on machines a couple of times taught me more about Linux (indeed Unix-ness) than any previous activity.

</comment></comment></comment></comment><comment><USER12> >their fork is designed to allow a system to be started even if the /usr/ directory hasn't been mounted yet.

What a terrible reason to fork. This feature is useless. Putting /usr/ on a separate filesystem made sense 20 years ago when storage capacities were small, but not anymore.

On Fedora, /bin, /sbin, and /lib are symlinks to their counterparts in /usr. I applaud them for this effort to simplify Unix and move away from cargo cult ritual.

<comment><USER13> Is _cargo cult_ the correct term? I have never seen it used to mean "doing things a certain way because they have always been done that way."

<comment><USER5> <http://en.wikipedia.org/wiki/Cargo_cult_programming> says:

Cargo cult programming is a style of computer programming that is characterized by the ritual inclusion of code or program structures that serve no real purpose. Cargo cult programming is typically symptomatic of a programmer not understanding either a bug he or she was attempting to solve or the apparent solution (compare shotgun debugging, voodoo programming).[1] The term cargo cult programmer may also apply when an unskilled or novice computer programmer (or one not experienced with the problem at hand) copies some program code from one place and pastes it into another place, with little or no understanding of how the code works, or whether it is required in its new position.

I think that applies here.

<comment><USER14> So we're saying that every Unix and Unix-like implementation in recent times that supports and indeed advocates [1] placing /usr, /tmp, /var, and so on on seperate filesystems have done so because the developers are unskilled novices with no understanding or experience of the problem at hand? Sorry, I think it's entirely the wrong term.

[1] <http://www.openbsd.org/faq/faq4.html#Partitioning>

<comment><USER12> Yes, that is what I'm saying.

Security: Adding a layer of security on a per-filesystem basis is not sufficient or necessary. It is only conceivably useful as a backup plan when the actual security system has failed, and at that point it is not sufficient to prevent compromise.

Stability: Quotas are a superior solution.

Speed: Not an issue for /usr, which is rarely written to.

Integrity: We have journaled filesystems now. If you lose a filesystem it is almost certainly because the device has gone bad and needs to be replaced. For last-ditch system recovery, an initrd solution like Dracut works better, since it can solve problems involving the root partition.

Size: Modern computers can boot from any size partition.

fsck: Preventing fsck is no longer useful. It doesn't take an hour like it used to. If there was a power interruption I would prefer to fsck all my filesystems anyway.

</comment></comment></comment></comment></comment><comment><USER15> This is fun: <http://twitter.com/mjg59/status/270323838468890624>

<comment><USER4> What's wrong with wanting to avoid glibc dependencies? The #define approach is not the way to do it, but why not make a udev that works with other C libraries and/or can be compiled with -std=c99?

<comment><USER16> Because glib is a decent library of data structures and algorithms that will just need to be reimplemented.

<comment><USER17> Also, glib != glibc

<comment><USER16> Bah. Poor reading comprehension.

</comment></comment></comment></comment></comment><comment><USER18> So let me get this straight: now we have two branches of udev that are maintained by ... unconservative[1] ... people, and no usable one. Hopefully this will be resolved before it gets out of hand. I think the situation is especially iffy for distros like Debian who are committed to long term support of whatever they choose.

[1] I didn't want to write "idiots", partly because I actually respect what the systemd camp is trying to do.</comment>


Ask HN: What phrase is a search query most likley to begin with - <AUTHOR>

Hi, How can I gauge and contrast the number of internet search queries that start with "How to" vs "How do I" vs "How do you". I am curious to find out what phrase people are most likely to preface a question when typing into a search engine. Thanks as always. <comment><USER2> According to the Google keyword suggestion tool, the global search volumes for those phrases are:

"How to" - 226m "How do I" - 9.14m "How do you" - 11.1m

<comment><AUTHOR> Thanks. I should have thought of using that.

<comment><USER2> We also have a bunch of internal data from our q&a search engine. I wouldn't want to publish that publicly, but feel free to email me (address in my profile) and I'd be glad to share.

</comment></comment></comment><comment><USER3> You could start with the Google Adwords keyword tool or traffic estimator. They will give you a gauge to work with, but you have to take their numbers with a little hesitation. You can also check out Wordtracker or Keyword Country to gauge search terms.</comment>


Ask HN: RHEL based or Debian based? - <AUTHOR>

Over the years I have worked with both systems and I generally prefer RHEL based systems, mainly due to familiarity. HN, what are the pros/cons to working with either system? <comment><USER3> I went for years as a RHEL/Centos guy, but over time I've become frustrated with the age of packages supplied. I know there are "solutions", in the form of the EPEL repository, RPMForge and the like, but in the end, they become a stopgap, and lead to all manner of issues with dependencies.

I've recently grown pretty enamored with Ubuntu 12, and the ease of package management. For me, the growing integration with LXC (which I love for the benefit of "light" VMs / isolation) is becoming buttery smooth.

Support has always been a question, but in 8 years of experience with approximately 50+ servers at multiple locations and clients, I've not once needed to contact RedHat.

<comment><USER4> I agree. Sometimes you simply must have recent versions of certain packages, and it's much likelier that you'll be able to do that out-of-the-box on Debian/Ubuntu.

</comment></comment><comment><USER5> I also use CentOS on servers. It just feels more solid to me, but this is probably more me than it. I also use Fedora on the desktop so I have a consistent package management experience.

Even if I switched to Ubuntu/Debian on the desktop, I'd probably keep CentOS on the servers. While its packages may be old, most things still compile on it.

A long long time ago I took a preference to RedHat/Fedora (well, pre-Fedora) as more software compiled by default on it than on Debian/Ubuntu.

I did once venture over to Ubuntu with the promise of clean upgrades from release to release. In truth, it was no better than Fedora, and I found issues hard to fix.

</comment><comment><USER6> I use Centos on servers. I've always found the Redhat model to be easier to manage servers with. I use debian on embedded systems because it can more easily be made read-only and smaller. Debian is more "Linux-y". Redhat makes lots of changes to the kernel and filesystem layout that make tinkering with the low level system harder.

</comment><comment><USER7> In my experience CentOS/Redhat is always a pain in the ass. I find Ubuntu/Debian awesome primarily because of the package management.

</comment><comment><USER8> I prefer Arch Linux, it's closest to FreeBSD in spirit but still running on Linux kernel for compatibility reasons.

</comment><comment><USER9> We just installed a new machine with the new Luna desktop from elementary os [1]. Looks awesome so far. Might just tilt the balance in favour of Debian on the desktop.

[1] <http://elementary.org>

<comment><USER5> The correct URL is: <http://elementaryos.org/>

</comment></comment><comment><USER10> A quick search of Dice dot com reveals the following:

Red Hat - 991 RedHat - 769 CentOS - 383 RHEL - 350 Ubuntu - 250 Debian - <USER2>

I love Ubuntu on my personal laptop, but RHEL, CentOS, and Fedora are what I've used at work most.</comment>


The Register now behind invisible HTML5 Bitcoin paywall - <AUTHOR> https://www.theregister.co.uk/2017/04/01/invisible_bitcoin_paywall/ <comment><USER2> What date was that article published again?</comment>


Anomaly Detection of Time Series Data Using Machine Learning and Deep Learning - <AUTHOR> https://www.xenonstack.com/blog/anomaly-detection-of-time-series-data-using-machine-learning-deep-learning <comment><USER2> TLDR: This is a high level overview of all the terms in the topic (Anomaly, Time Series, Deep Learning). If you have a passing familiarity with the problem space, there is nothing new here. There is an ad for xenonstack services at the bottom.

<comment><USER3> It taught me about the AnomalyDetection R package, which was nice.

</comment></comment><comment><USER4> > This is because the oscillations are dependable upon the business cycle.

No, no, no, no. The "business cycle" is a misnomer; it's not a cycle. It doesn't fit any descriptions of a "cycle" except for the fact that it sometimes goes up and sometimes goes down.

Saying that it's a cycle implies you can time downward trends like crashes or predict their intensity, which, anyone in finance or maroeconomics will tell you is impossible

<comment><USER5> _Saying that it 's a cycle implies you can time downward trends like crashes or predict their intensity, which, anyone in finance or maroeconomics will tell you is impossible_

Just because something is hard doesn't mean it's impossible; You can predict anything with a degree of certainty. Then there are things like uncertainty and radical (Knightian) uncertainty.

Any chartered actuary would tell you the same if you could afford one, and they will know quite a bit about business cycles and how to model them, too.

<comment><USER4> Except you won't in macroeconomics. The simple problem in macroeconomics is lack of data -- history happened only once so you can't test counterfactuals.

You can try to build a model, but the confidence intervals for the next big crash are going to be either useless (way too wide) or bullshit if what you're trying to predict is the next market crash. Building a classifier around it machine learning style will not fundamentally help, either.

You can try it: here is an open source, serious, full scale macroeconomic model by the NY Fed [1]. Try to predict the next large crash with it and report back

[1] [https://github.com/FRBNY-DSGE/DSGE.jl](https://github.com/FRBNY- DSGE/DSGE.jl)

<comment><USER6> This would be the same macro-economic theory that even macro-economists will now grant is fundamentally flawed.

The rest is left as an exercise to the reader.

</comment></comment></comment></comment><comment><USER7> If you're interested in more research level techniques, check out some of the work that's been done using minimum description length (MDL). It basically seeks to minimize the size of the model plus the size of the residual data (that the model doesn't predict) and uses Huffman compression as a proxy for the Kolmogorov complexity of a string:

"Discovering the intrinsic cardinality and dimensionality of time series using MDL" [https://pdfs.semanticscholar.org/2049/50b3cd9cf2eef52f957df6...](https://pdfs.semanticscholar.org/2049/50b3cd9cf2eef52f957df626e01cad7adcef.pdf)

A few papers that build on that one show how to incorporate anomaly detection (in the sense that the anomaly is incompressible).

<comment><USER8> And more on research-level techniques:

Ahmad et al. "Unsupervised real-time anomaly detection for streaming data": [http://www.sciencedirect.com/science/article/pii/S0925231217...](http://www.sciencedirect.com/science/article/pii/S0925231217309864)

Anomaly detection benchmark (dataset and evaluation): [https://github.com/numenta/NAB](https://github.com/numenta/NAB)

</comment></comment><comment><USER9> Since this article is very high level, are there other resources that show how to apply some (or all) of these topics in actual code or on a sample set of data?

<comment><AUTHOR> [http://machinelearningmastery.com/blog/](http://machinelearningmastery.com/blog/) has some good tutorials on time series and ML in general

</comment></comment><comment><USER10> The first few words of this title sound way too sci-fi. Very disappointed when I read the whole thing together.

Edit: supposed to be a light-hearted observation, I haven't read the article!</comment>


Show HN: The Polyglot Bible Typography Project - <AUTHOR> http://dodecaglotta.com/ <comment><USER2> I ordered my copy. This is typographical craftsmanship of the highest level. Thanks for this marvel reflecting a huge amount of dedicated labour and what must have been months of research.

</comment><comment><USER3> This is beautiful!

<comment><USER4> It is! It's both calming and exciting to look at. The perfect typography just gives me chills.

</comment></comment><comment><USER5> I'm not sure what this is supposed to be, but I did not find it inviting to read.

<comment><USER6> I have similar thoughts, it is not directly obvious and it is kind of hard to read, very heavy. But I think it is a new typeface/font + a print of a bible in 12 languages using this font.</comment></comment>


Xkcd: M87 Black Hole Size Comparison - <AUTHOR> https://twitter.com/xkcd2/status/1116051854700548097 <comment><USER2> Why not post to the original content?

[https://xkcd.com/2135/](https://xkcd.com/2135/)</comment>


China: Surviving the Camps - <AUTHOR> http://www.nybooks.com/daily/2016/01/26/china-surviving-camps-cultural-revolution-memoir/ <comment><USER2> Hypothesis: The psychology behind the turmoil caused by the red guards might be very related to contemporary phenomena in academic settings and on social media, where the rhetoric of social justice is perverted and turned into a weapon to antagonize anyone in the perceived out-group. (Of course, the magnitudes of the resulting problems are vastly different.)

<comment><USER3> Group dynamics can result in some awful things. It's expressed anywhere from kindergarten, highschool to politics, online bullying, conspiracy theories, etc. So, yeah, they have things in common, but as you say, the outcome in China was grievous and devastating.

Thankfully accounts still survive and if I recall correctly, there have been a couple of Chinese movies which broach the subject, although not certain if done in Hong Kong.</comment></comment>


New Puzzles to Tell Humans From Machines  - <AUTHOR> http://www.nytimes.com/2009/05/24/business/24novelties.html <comment><USER2> It's been tried: google "kittenauth". The problem is that you need to collect the images from somewhere, so the keyspace is too small.

<comment><USER3> Right, but for kittenauth you need to "burn" 9 images to achieve one authentication. (Three kitties and six non-kitties, as I recall correctly.) With the rotational thing, a single source image can be algorithmically manipulated to generate dozens or hundreds of challenges -- kitty at a 90 degree angle to the left, kitty at a 90 degree angle to the right, etc.

Figuring that, conservatively, you can get a dozen challenges out of an image, that makes this technique about a hundred times more efficient than kittyauth in terms of how many challenges you can issue for the size of your library. Then you pair it with a large set of images ( _cough_ Flickr Creative Commons license _cough_ ) and you're off to the races.

(The Flickr Creative Commons option isn't abused to nearly its potential by startups.)

<comment><USER2> I'm not a CG guy, but I'm guessing it's pretty simple and efficient to recognize which of a set of images a given image is a rotation of. So the ability to create lots of rotations doesn't buy you any security.</comment></comment></comment>


Using Meteor to Build a Twitter Wall for LegalTech New York 2013 - <AUTHOR> http://www.matb33.me/2013/02/02/ltny-twitter-wall.html

<comment><USER2> Great stuff!</comment>


How security flaws work: the buffer overflow - <AUTHOR> http://arstechnica.com/security/2015/08/how-security-flaws-work-the-buffer-overflow/ <comment><USER2> One of the best explanations ever written is the now classic "Cult of the Dead Cow issue #351 - The Tao of Windows [NT|95|98] Buffer Overflow" @ [http://www.cultdeadcow.com/cDc_files/cDc-351/](http://www.cultdeadcow.com/cDc_files/cDc-351/)

<comment><USER3> >but if you don't have automatic bounds checking like Java, I guarantee you that those 'A's are going somewhere unfortunate.

20 years later and we're still using C++. I hope Rust takes off. The status quo today is terrible.

<comment><USER4> Also of note is the lack of investment in exploit mitigation at the operating system and compiler level. C++ and C are not going away from our systems any time soon. Yet exploit mitigation technology still remains on the sidelines of the security world while bug squashing, AV, and IDS are flush with cash.

<comment><USER3> How realistic is a compiler-level change that is 100% transparent and reverse compatible with existing codebases? I suspect this is a tough nut to crack without making fundamental changes to the language and perhaps invalidating past codebases. You might as well move to a different language then.

I suspect we'll have a Rust compiler that compiles fast applications with a very minor performance hit before this happens to aging languages like C/C++.

<comment><USER5> The question is not just when we will have the tech, though. It is also when the tech will have an impact. A perfect Rust compiler only improves the safety of new (or rewritten) code.

<comment><USER6> I daresay that some codebases might _rely_ on overflowing behaviour even if they don't know it. I've worked with a codebase where assumptions about unused RAM were that it was always zeroed out. Moving to a different platform (Linux) changed those assumptions, and there was lots of weird behaviour in the codebase because of that.

I'll bet there are lots of legacy codebases that would actively resist using such a compiler technology for C/C++ because of unknown behaviours happening in their codebase (and because there is no test suite, or just poor test coverage).

<comment><USER5> Certainly, though usually that winds up perceived as brittleness moving between compiler versions in general. Ideally, the tech we are discussing could help make it easier to understand those unknown behaviors.

But even if not, and even if this accounts for 60% of existing C/C++ codebases, improving the remaining 40% is a huge win that would take tremendously more time to reimplement in Rust.

This is not to say that I am not also very excited about Rust.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Alternatively (just the concept, this should not work as outlined on modern systems): [http://phrack.org/issues/49/14.html](http://phrack.org/issues/49/14.html)

<comment><USER8> I used to use Aleph One's article as a kind of shibboleth when interviewing candidates for security researchers. I'd just casually say "Smashing the stack..." and see if they would fill in the rest of the title.

Not the best data point, but showed pretty quickly who had a self-taught (and usually practical) understanding of security vulnerabilities vs an academic understanding.

<comment><USER9> That's a class move and you are probably a cool person to work with.

<comment><USER9> That was not sarcasm.

</comment></comment></comment></comment><comment><USER10> The worse is that this error only happens because we are trying to save 4 bytes. If all arrays would also store their own size we could eradicate this bug with a limited memory impact.

<comment><USER11> You have overlooked the CPU cost. Given that most memory objects already have at least an over-approximate size _somewhere_ , and how we are more than happy to annotate stacks with canaries, I really don't think anyone is as concerned with the memory cost of adding a size to every array as the CPU cost of adding a read and compare to every array access.

<comment><USER12> Intel is adding hardware array bounds checking called MPX in skylake, so this could become a reality.

<comment><USER13> There already was a BOUND instruction. Nobody ever seems to remember this when they propose adding such an instruction.

</comment></comment></comment></comment><comment><USER14> There were so many critical vulnerabilities attributed to that... and in spite of putting tons of measures this is still potential danger. I knew projects decided to use Java instead of C++ (which got array boundary checks) mostly to limit security surface.

Maybe if we could start CPU architecture from scratch, array with sizes would make more sense? Maybe even we could done it so memory segmentation would no longer be needed: [https://en.wikipedia.org/wiki/X86_memory_segmentation](https://en.wikipedia.org/wiki/X86_memory_segmentation)

Haven't figured out the details, but maybe we can even gain some performance that way.

<comment><USER15> Friendly reminder: it's not just a choice between C and Java. In order to get bounds checking on arrays it's not necessary to give up the advantages of native programming and use a virtual machine.

Rust is obviously a great example. So is Go. Bounds checked arrays in languages that compile to native is not a new thing.

<comment><USER16> Swift is another new language with it. Its getting to be the norm now.

</comment></comment></comment><comment><USER17> _" The buffer overflow has long been a feature of the computer security landscape. In fact the first self-propagating Internet worm—1988's Morris Worm—used a buffer overflow in the Unix finger daemon to spread from machine to machine."_

I thought the daemon exploited was sendmail? ... quick search later ... so the fingerd was used. Good explanation here: ~ Eugene H. Spafford, _" The Internet Worm Program: An Analysis"_ [http://spaf.cerias.purdue.edu/tech- reps/823.pdf](http://spaf.cerias.purdue.edu/tech-reps/823.pdf)

</comment><comment><USER18> Are "buffer overflows" possible in systems without without virtual memory?

I know how I would answer this question but I am curious how others would answer it.

EDIT: s/possible/known to occur

<comment><USER6> As long as you have the ability to address memory directly, I would say that it's possible. I'm not 100% sure about how memory is allocated though. Outside of virtual memory, are the actual chunks of memory sequential, or would overflowing an offset get you a chunk of another process? (I imagine the answer might be implementation dependent)

<comment><USER15> It's more important to have bounds checked access and to disallow pointer arithmetic. If you do that, pointers (addresses) are fine.

</comment></comment></comment><comment><USER19> Seriously this is old news 20 years too late and not even good enough, checkout Phrack article, "smashing the stack for fun and profit" written in 96. This is what unleashed the flood gates. With sample codes to boot.

</comment><comment><USER20> The Burrough's architecture (1961) had a CPU that bounds-checked arrays & pointer access. Also differentiated code and data to prevent... all kinds of security issues. Total overhead in memory was 6% w/ almost zero overhead in CPU in such a design if you run security-checks in parallel with rest of pipeline while only committing a write if security check is OK. This is the modern approach.

[http://www.smecc.org/The%20Architecture%20%20of%20the%20Burr...](http://www.smecc.org/The%20Architecture%20%20of%20the%20Burroughs%20B-5000.htm)

No matter what naysayers claim, the ability to inexpensively immunize a system against buffer and stack attacks has existed for decades while being deployed commercially on _1960 's-1980's era hardware_. The market went for what was backward compatible with IBM, UNIX, and later DOS software plus highest raw performance at lowest price. Systems designed for excellent security and maintenance (esp written in HLL) had low market share with many getting acquired, disappearing, or having security advantages removed in later releases. It's a market thing, not a technical thing.

Here's a modern example, the SAFE architecture, that's being developed by BAE & academia.

[http://www.crash-safe.org/assets/ieee-hst-2013-paper.pdf](http://www.crash- safe.org/assets/ieee-hst-2013-paper.pdf)

It's actually more heavyweight as it tries to fix... computing in general haha. It specifically mentions buffer overflows and stack risks with the ways they try to immunize against them. The atomic groups with associated processing are actually lightweight a la Burrough's model. They alone provide quite the bit of protection seeing how many types of attack leverage pointer, array, or stack flaws. The other tag will cause more of a hit but could do type enforcement of calling functions. Is also very helpful given interface errors underlie most attacks on systems. I've suggested porting Oberon System and compiler to this architecture to get a fully-usable box with automated protection. And see what attacks can still get through.

In any case, buffer overflows have been preventable since before they even had a name past being an example of "insufficient argument validation" (MULTICS security evaluation). They only exist because (a) market wants them to exist via how it votes with its wallet and (b) legacy software refuses to take the hit that automated protections created. The latter situation improved greatly with tools such as Softbound + CETS which enforces full safety on C code with average 50% hit, limited tools such as Control Pointer Integrity that protects pointers with a tiny hit, and tools such as ASTREE/SPARK that prove C/Ada code free of many types of errors that lead to crashes or hacks. Pretty much no uptake.

There are counterexamples though. I've seen a reverse stack on x86, a DNS server (IRONSIDES) written in SPARK, security kernels that extensively used segmenting for OS-level POLA, a web application server reducing risk w/ FSM's + a separation kernel, and recently MirageOS building a whole stack in a language that prevents so many issues at compile-time while designing for isolation in partitions. So, we can do it better, some are doing it better, and I encourage others to do the same. Look for work that knocks out issues, use it, improve it, and get our baseline up to at least OK instead of FUBAR (current status).</comment>


Storing hundreds of millions of simple key-value pairs in Redis - <AUTHOR> http://instagram-engineering.tumblr.com/post/12202313862/storing-hundreds-of-millions-of-key-value-pairs-in <comment><USER2> The the code the article links to for zipmap.c (<https://github.com/antirez/redis/blob/unstable/src/zipmap.c>) is _rather_ literate.

I haven't dug extremely deeply into the sources for many F/OSS projects; the code I'm interested in reading has been inevitably opaque (at least to my inexperience). This particular source file (and maybe the rest of Redis?) is really good. I think I'll be taking many more looks at Redis (code- and usage- wise) in the future.

<comment><AUTHOR> Back when we were getting started with Redis, the readability / concise nature of the project was one of the things that most excited me about it (here's what it looked like around then: [https://github.com/antirez/redis/tree/0b420168b485d0a9c4b66d...](https://github.com/antirez/redis/tree/0b420168b485d0a9c4b66d0a6c341597fb155947))

<comment><USER3> I'm curious, what in particular makes you say that code is "readable"?

Coding style varies dramatically from person to person, and I don't mean this as a criticism of antirez, but any code which doesn't have at minimum a one- line comment before each function explaining its purpose immediately fails the "readability" test for me. Obviously this isn't a problem for you, so I'm curious to hear what your tastes are.

<comment><USER4> Hi cperciva, I actually think that comments are not a so important part of code quality. I tend to add comments where my code risks to be not clear by itself, and the zipmap.c code is indeed more commented than my average code since it is all about encoding stuff in a binary blob string, playing with pointers, and so forth. So actually too much comments may even be a sign that something is bad about the code.

IMHO good code should be readable since the purpose of different files, functions, statements, data structures, should be obvious (at different levels of course), and every time it is not obvious there should be a comment helping the reader to understand what is going on.

My idea is that programmers with time develop a feeling about when a comment is needed. For instance a comment is needed all the times you are writing something that avoids a specific problem but you'll likely not remember why it was needed in a few weeks. Other times comments are useful since the flow of the function is complex and there is no easy way to refactor it into many pieces, so comments help to organize the function in smaller conceptual parts, and so forth.

There is no absolute rule, but the reality is that IMHO the test is simple to do for external people: good code is easy to understand and modify without being an expert of that code base.

This topic is a good idea for a blog post, since I thought a lot about this issues lately. For instance if you want a place in Redis where code should be improved is in the handling of blocking operations: there are a few things in that code that are absolutely non obvious even adding comments, and you either are a lot "into it" or you'll not have an easy time understanding it. I'm planning a refactoring of that piece of code.

<comment><USER3> _I tend to add comments where my code risks to be not clear by itself_

I used to take that position, but I've started adding more comments in order to avoid "mental stack overflows". Suppose I'm reading function A and trying to understand it, then I find a call to function B which doesn't have any comment explaining what it does; I then go look at function B, and find it has a call to function C which is equally lacking in commentary; and by the time I've read the code in function C to understand what it does and gone back to function B to understand what it's doing I've completely lost track of what I was looking at in function A.

Of course, if you already know what most of the code is doing you don't run into such stack overflows because whatever code you're looking at is probably only calling functions you already understand. But for people who are new to the code -- or people who haven't looked at it for a couple years and have forgotten most of the details -- I think asking people to read the code to figure out what a function does is too much of a bar to understanding.

<comment><USER5> In my opinion, if you can't figure out what a function does by its name and its parameter names, it is a poorly named and thus poorly documented function.

I think function and variable naming is the one of the most important aspects of programming. Without good naming, you can easy double the amount of time it takes to edit and extend functionality.

<comment><USER6> A function may have the perfect name at the time you wrote it. It may make perfect sense within the context that you initially conceived of it. However, after some time away from it, when you're trying to mentally rebuild that context, it may make as much sense as def foo().

Good naming is important, but you also have to know the context, which is more difficult to remember.

<comment><USER7> If it only makes sense within the context it was conceived under, that's not a perfect name. A function name should make sense for someone with understanding of the domain, but no understanding of the code. When that's not possible, comments definitely help.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> 1 million pair using 16 MB is about 16 bytes per pair, which is perfectly fine but nothing impressive.

The dataset is static, so a simple naive solution would be to create a big array sorted by key. Assuming both photo and user IDs use 4 bytes each, this would result in about 2GB of data. Then use binary search to lookup values.

However, if we really want to reduce the size, we could build a finite state machine from the dataset (maybe reverse the values to increase the level of shared suffixes) which should reduce the size by an order of magnitude.

<comment><USER3> _The dataset is static_

If I read the article correctly, existing entries won't change but new entries will be inserted.

<comment><AUTHOR> Yep, ~30 inserts per second go in, so it's not static.

</comment></comment></comment><comment><USER9> First, I love Redis :-)

Second, this functionality seems to be a stop gap to support old users who may be using old clients. So they need an array of 300 million elements each containing an integer of 12 million or less. Assuming 32 bit values (24 would work but.. efficiency), that's a 1,144MB array which, in theory, could be stored as a file and cached through the filesystem cache.

I wonder how the performance of that would stack up against Redis. The convenience of Redis being a network daemon out of the box is potentially the big win here, though the memory usage even in the optimized case seems to be around 4x given that it's doing more than the task necessarily requires (from my interpretation of it - I could be wrong!)

</comment><comment><USER3> _Best of all, lookups in hashes are still O(1), making them very quick._

How quick is "very quick"? I was hoping to see some performance benchmarks, not just memory usage benchmarks.

<comment><USER10> _Best of all, lookups in hashes are still O(1), making them very quick._

Based on the zipmap code (linked below), a zipmap is implemented as an array of adjacent (key, value) pairs. Lookup in a zipmap is actually linear search. There is no hashing. The lookup will run in time proportional to the number of entries in the map.

 _We found this setting was best around 1000; any higher and the HSET commands would cause noticeable CPU activity_

This shouldn't be surprising either, as the redis documentation states that "If a specially encoded value will overflow the configured max size, Redis will automatically convert it into normal encoding." Their higher level hashing strategy of dividing the media id by 1000 guarantees that 1000 is the maximum number of entries in any zipmap. Setting hash-max-zipmap-entries to anything lower than 1000 means some of their zipmaps will be converted to normal redis key/value encodings.

<https://github.com/antirez/redis/blob/unstable/src/zipmap.c>

<comment><USER11> Wow I didn't realize that redis zipmap lookups aren't actually O(1). I found a document that describes zipmaps as "very memory efficient data structure to represent string to string maps, at the cost of being O(N) instead of O(1) for most operations. Since the constant times of this data structure are very small, and the zipmaps are converted into real hash tables once they are big enough, the amortized time of Redis hashes is still O(1), and in the practice small zipmaps are not slower than small hash tables because they are designed for good cache locality and fast access."

<http://redis-docs.readthedocs.org/en/latest/Hashes.html>

</comment></comment></comment><comment><USER12> I love companies like Instagram who take the time to share their scaling issues and experiences.

</comment><comment><USER13> You can find similar examples -- ruby code for the "exact" implementation as described by Instagram -- of this and other redis, memory optimization(s) at the Redis site: <http://redis.io/topics/memory-optimization>

</comment><comment><USER14> Lookups are not really O(1), they're O(number of keys per hash) as long as the hashes are zipmaps. When they become full blown hashes the memory usage increases.

Still, this is a very good way to store a lot of key/value pairs in Redis.

<comment><USER15> Big-O notation refers to _asymptotic_ behavior. The zipmap encoding of hashes only matters for small values bounded by a constant, so hash lookups are still expected O(1) time in Redis.

</extreme-pedantry>

<comment><USER14> Your point would be valid if the memory gain observed was not directly dependent on the fact that hashes are zipmap-encoded. So the trade-off here is between the constant factor of time complexity and the constant factor of memory complexity.

</comment></comment></comment><comment><USER16> Why use clear text numbers? Most of the time, you're going to be using large numbers, so binary pack them as save more space.

i had the same issue, normal storage was 1.1gb of space, HSET down to 200mb and binary packing every integer down dbl() bought it right down to 163mb of memory (32bit instance). For that 163mb, I was slicing a md5 of the field for the hset prefix, packing that and then using the remainer as the hset suffix. (due to the data format of the input field)

<comment><USER17> Internally, redis stores integers as 64-bit binary values, not strings: <http://redis.io/commands/incr>.

<comment><USER18> I don't think it does that for keys and values in hashes though. The only int- related optimization it does in the collections is integer sets.

</comment></comment></comment><comment><USER19> Why use the whole Media ID as the key within the bucket, rather than just the last three digits?

<comment><USER20> I think they were trying to keep the hash sizes to ~1000 elements each key. At a 3 digit key you would be increasing the hashes to 300k elements each. You should augment the scripts and see if that has an impact on the performance, would be curious to see.

</comment></comment><comment><USER21> The hash data type is probably the most awesome and underused feature of redis. Here is a small gem I wrote to expose it a little better in ruby: <https://github.com/lyconic/redis-native_hash></comment>


Security Tents - <AUTHOR> https://www.schneier.com/blog/archives/2013/11/security_tents.html

<comment><USER3> Invented by Mel Brooks in the 60's: [http://hilobrow.com/wp- content/uploads/2010/07/cone-of-silen...](http://hilobrow.com/wp- content/uploads/2010/07/cone-of-silence-1.jpg)

<comment><USER4> Churchill might have scooped Brooks.

Winston Churchill's secure telephone line from his Cabinet War Rooms to Washington DC was housed in a small closet. Rumors were that this was Churchill's private bathroom and everyone left it alone:

[http://en.wikipedia.org/wiki/SIGSALY](http://en.wikipedia.org/wiki/SIGSALY)

[http://www.flickr.com/photos/anthony- gurr/10782926364/](http://www.flickr.com/photos/anthony-gurr/10782926364/)

</comment></comment><comment><USER5> Nothing new. There's a picture of the inside of one of these tents[1] and Wikipedia covers them[2].

[1] [http://www.bbc.co.uk/news/world-us- canada-12810675](http://www.bbc.co.uk/news/world-us-canada-12810675)

[2] [https://en.wikipedia.org/wiki/Sensitive_Compartmented_Inform...](https://en.wikipedia.org/wiki/Sensitive_Compartmented_Information_Facility)

<comment><USER6> True, but what he mentions, and what you can see in the photo is that they forgot about the floor.

Unless the security tent comes with a floral printed, carpeted floor...

<comment><USER5> Why do you think that all these technical people would forget about the floor? So the tent comes with a crappy carpet, so what?

</comment></comment></comment><comment><USER7> > US security officials demand that their bosses - not just the president, but > members of Congress, diplomats, policymakers and military officers - <USER2> > such precautions when travelling abroad because it is widely acknowledged > that their hosts often have no qualms about snooping on their guests.

"No qualms" indeed.

</comment><comment><USER8> All those people living in basements lined with tinfoil were right all along.

</comment><comment><USER9> fwiw, this was reported in the local press when he was here (chile), although i can't find anything now. that was a couple of years back, so this isn't new.

</comment><comment><USER10> Next up: Guantanamo-style "secure hoods" containing smart "Google Glass"-style glasses.

</comment><comment><USER11> The Cone of Silence lives.

</comment><comment><USER12> startup idea:

it sounds increasingly like there could be market demand for a commercial product equivalent</comment>


AudioBox: Why We Moved to DigitalOcean - <AUTHOR> https://www.digitalocean.com/blog_posts/audiobox-why-we-moved-to-digitalocean

<comment><USER2> I heard about this from one of the Audiobox guys. One of the few good "OMG WE NEED TO UPGRADE RIGHT NOW COS THE SERVER's DYING" stories... :-)</comment>


Ask HN: What is this type of design called? - <AUTHOR>

When I scroll down the top bar stays in place and the pages are divided into sections. What is this design called? Are there any open source free themes like this for bootstrap and etc?<p>https://www.dropbox.com/ http://actionherojs.com/ <comment><USER2> Single-page HTML5 design?

Topbar CSS = background:fixed

You just build <div> with different color with width=100%

I googled for you - [http://www.instantshift.com/2013/10/14/free-responsive- html5...](http://www.instantshift.com/2013/10/14/free-responsive- html5-templates/) \--- go to around the 6th one, free Download, exactly the same as Actionherojs.com

<comment><AUTHOR> that's a gold mine. thank you!

question, for your [http://sharemoto.io/](http://sharemoto.io/), how did you make that automatic typing cursor? That animation of icon is simply genius. plus the auto scrolling when you scroll down is a neat feature. Is there a library for that as well? thanks again.</comment></comment>