How to crawl a quarter billion webpages in 40 hours (2012) - <AUTHOR> http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-billion-webpages-in-40-hours/ <comment><USER2> This was written in 2012, Its even easier these days by using SQS and Cloud Formation. 250 Million is a small number you are better of first going through Common Crawl and then use data from crawls to build a better seed list.

Common Crawl now contains repeated crawls conducted every few months and also urls donated by blekko.

[https://groups.google.com/forum/m/#!msg/common- crawl/zexccXg...](https://groups.google.com/forum/m/#!msg/common- crawl/zexccXgwg4w/oV8qeJnawJUJ)

<comment><USER3> Common Crawl contains the HTML?

I wonder how this is legal and considered acceptable. I wish I knew how even Google and others get away with scraping content, saving it, and utilizing it for profit without sharing any revenue with the original webmasters.

I know people can opt out of crawling, for those that actually respect that. But still, am I the only one who feels like this is wrong?

I guess I have this view that your domain is yours, and you invite the public in like an open house. It's my house, my property, and the door is open, where people can come in and look around at my stuff. But the expectation is that only locals will arrive, in small number, and they'll be good guests. If someone is breaking the lock on the bedroom door and going through the private drawers, that's wrong. If someone is taking photographs of everything, to then create a virtual tour of my house they charge for, that's wrong. The expectation is you're being nice by providing free and open access to information you created and own, and people should behave courteous to that.

Then if you as the webmaster choose, you can provide an API, or database dumps for people to download, along with the licensing terms. That is when it feels right for people to do things like this with the data, because you intentionally provided it through a non-personal interface.

To me the web is still a personal interface. I expect humans to use it, in an ordinary human-like way where it is somewhat ephemeral and courteous. I feel like Google cheated their way to success, and Common Crawl is stealing to rise their position in an unfair similar manner.

These all seem like parasites to me. They didn't create anything, they just steal it en masse.

There's so many businesses like that, such as Domain Tools that gets rich by hoarding everyone's contact details from WHOIS: [http://whois.domaintools.com/commoncrawl.org](http://whois.domaintools.com/commoncrawl.org)

They have a screenshot history they won't ever delete even if you ask nicely. Here is a picture of Common Crawl from 2011: [http://thumbnails.domaintools.com/domaintools/2016-01-08T19:...](http://thumbnails.domaintools.com/domaintools/2016-01-08T19:20:04.000Z/juwC52sCoU2vfhzE8XBhOgzFINQ=/commoncrawl.org/fullsize/39bc3b10013e1f43ccfec67ed654dbd3/1322786356.jpg)

<comment><USER4> > I wonder how this is legal and considered acceptable. I wish I knew how > Google and others gets away with scraping content [...]

Well, here's the answer: "transformative" reuse of content is explicitly permitted under copyright law. Simply reproducing the content and charging for it would not fall under this provision, but building an archive of publicly available information is - quite appropriately, permissible.

There was recently a very large court case regarding this principle and its application to Google Books. Google won, by demonstrating that their search index is not equivalent to and does not affect the market for the original work - a "transformative" use.

Sharing is good. Publicly available works achieve their aims only by being consumed by others - anyone who publishes a work free of charge should expect it to be, and remain, publicly accessible.

<comment><USER3> I don't think "Sharing is good" is true in the real world. If you apply that as a blanket statement, you'll end up in trouble.

What is legal is not always ethical. I think there's an interesting story there about how Google is legal, if someone doesn't automatically assume it should be just because it is.

The text online isn't always similar to a published text of the past. There is a personal overlap today that changes the rules. Such as this text I'm publishing right now. Forgetting about all the legalities and technicalities, I still feel like it is different than a page published in a book. I still feel like I should have the power to edit or delete it whenever I want in the future, yet Hacker News disagrees and removes my right to modify it, forever capturing it as if it owns it, not me. I still feel like this text is more transitory, where its relevance is mostly right now, and if it were deleted in a month it would be fine, because it's mostly just chit chat.

Certainly we could live in a world where everyone has microphones transcribing everything they ever say, which is transmitted to Google, and provided to researchers, where all kinds of uses could emerge. But that's a different world than the one where we've developed rules for today. Right now, I feel like most things I say are in passing, and should not only disappear, but won't spread where someone is capturing and propagating it beyond my control.

What control do I have over my text that is in this Common Crawler database? What if it captured information that was considered to be ephemeral in the website's context, and ripped it out of its home where it's now part of this collective publication, where anyone can use it for anything?

Sharing could be good in a world where people are not selfish and malicious. But in this one, many people will use whatever data they can get their hands on for selfish and malicious purposes, that do not benefit you, the author, in any way. I bet a large percentage of use for that Common Crawler database was harmful to society, such as for helping spammers generate fake content.

<comment><USER5> > "These all seem like parasites to me."

Your impression is wrong. Search engines and other services based on web data provide great value to society. They don't create documents they link to, but they deliver relevant links to people's queries. That's a great service. Without the search engine service, people may not even find the web page. That's why large portion of website owners and webmasters are glad search engine crawlers visit them and even expect indexing to databases to be fast and smooth.

If you publish anything on your web, you're facilitating free use and duplication of it in the whole world. If this was not your intention, but you still published your stuff on your web, you misunderstood the original intent and reality of the Web for sharing information.

There is a widely known standard of communication between robots and web sites called robots.txt standard. It is a file where you can state your intent to restrict crawler downloads. There is also html tag <meta name="robots" content="noindex,nofollow"> that signalizes to crawlers your wish that the page should not appear in search engine results. If you want to prevent people from accessing and using your documents, use these. Both Google and Common Crawl seem to obey them. If you want to _make_sure_ nobody accesses and uses your documents, don't publish them on the Web.

There is no practical way to achieve your documents are accessible only for some limited period you want. If you release them to the world, you always lose control over their distribution and use.

</comment></comment></comment></comment></comment><comment><USER6> Web crawling is just like most things: 80% of the results for 20% of the work. It's always the last mile that takes the most significant cost and engineering effort.

as your index and scale grow you bump into the really difficult problems:

1\. How do you handle so many DNS requests/sec without overloading upstream servers?

2\. How do you discover and determine the quality of new links? It's only a matter of time until your crawler hits a treasure trove of spam domains.

3\. How do you store, update, and access an index that's exponentially growing?

Just some ideas.

<comment><USER7> I've been working on something similar and I have ran into some of the issues you mention. As you correctly pointed out, quality and post processing is also relevant to not crawl irrelevant/spam sites, which can be HUGE! The work presented here is cool but it does not address the whole picture. Having a crawler that takes quality and user feedback into account is the hard part. Not to mention if you are being polite with the requests... we need to scale but not ignoring Robots.txt

So crawling a billion of links in X number of hours is not trivial but not that hard specially with cloud infrastructure like AWS, it's just a matter of a good enough implementation and how much money one wants to spend on it.

</comment></comment><comment><USER8> No one ever talks about a particular topic though when it comes to web crawling etc. How do you avoid all the "bad" sites as in, really bad shit? The stuff that your ISP could use as evidence against you when in fact it was just your code running and it happened to come across one of these sorts of sites. How do you deal with all that? That is the only thing stopping me from experimenting around with web crawling.

<comment><USER9> I used to do some crawling on comcast using what is now IBM Watson Explorer. I got a ton of phone calls from them. It sounded like those calls would go away if I just paid a little bit more for business class service.

</comment></comment><comment><USER10> old HN comments ( 3 years ago ) [https://news.ycombinator.com/item?id=4367933](https://news.ycombinator.com/item?id=4367933)

</comment><comment><USER11> I feel like more companies are building their businesses around web crawling and parsing data. There are lots of players in the eCommerce space that monitor pricing, search relevance, and product integrity. Each one of these companies has to build some sort of a templating system for defining crawl jobs, a definition of parsing rules to extract the data, and monitoring system to alert when the underlining HTML of a site has changed from their predefined rules. I'm interested in these aspects. Building a distributed crawler is easier than ever.

</comment><comment><USER12> This isn't particularly difficult anymore. The most interesting challenges in web crawling around turning a diaspora of web content into usable data. E.g., how to get prices from 10 million product listings from 1,000 different e-retailers?

</comment><comment><USER13> I don't understand his hesitancy in releasing his crawler code. I imagine there are plenty for people to access and alter for malicious use if they desired, so why does releasing his such a big deal?

</comment><comment><USER14> Is "quarter billion" used to make it sound like a bigger number? Even "half" is aggressive, imo.

<comment><USER15> I was going to ask this very question. Seems like the appropriate way would be to say 250 million. I mean you wouldn't say I got a quarter hundred dollars or quarter thousand...</comment></comment>


Ask HN: Is the CTO in a software company, the smartest programmer? - <AUTHOR> <comment><USER2> A CTO isn't necessarily a programmer at all. For example, if a company makes software for gene sequencing, their CTO could be a scientist.

And good managers try to hire people who are smarter than they are.

</comment><comment><USER3> I'll mention what someone told me what a CTO's job is _NOT_ to me. You can disagree

\----

CTO's should not be developing your app with your developers.

> He can be reviewing the code; or giving ideas to members and helping the > organization by giving them direction

Lastly it maybe unhealthy sign for your CTO to be coding with your developers like he is another developer. He has other tasks that need done than just coding</comment>


Google to give staff 10% raise - <AUTHOR> http://online.wsj.com/article/SB10001424052748703523604575605273596157634.html

<comment><USER2> Nice. Now if you want a 10% raise this year, it's easy to say, "everyone at Google got one..."

<comment><USER3> If you are performing well at a company as big as Google, you get 10% raise easily every fiscal year. "easily" as in the company is doing well. (It's a common sense that companies are less likely to give raises if the company is not making much money) If you happen to be asking for a raise, you are either working for a wrong company or not performing well compared to your peers.

<comment><USER4> I wish. It's more commonly some sort of good-cop bad-cop routine, where the line manager really appreciates your work and fights hard for you against the upper level manager, who is just so stingy about raises in general. If you're stellar, you wind up with something way better than your teammates, but an insult compared to what you could command on an open market.

I think on some level, they're constrained by collective bargaining agreements. Maybe they hope you've been with the company so long you only see the people around you. Maybe _they_ only see the people around _them_.

There was a real technical superstar of a guy a few years ago. I didn't work with him, but a coworker did. He went to management and said, "Give me a 17% raise or I'm leaving." They didn't and he did. The fallout was cataclysmic.

I left a project about three years ago, and there was widespread panic when I announced my decision to leave. I took four months to pack things up and leave them in a stable state, but you cannot totally mask the absence of talent. A coworker asked, "Is there anything this program could do to make you stay?" I answered, "No." He said, "How about a 200% raise?" I said, "Oh, I'd stay another year for that." Judging from the fallout over that year, it would have been an efficient option. But they either didn't have the power or didn't have the perception.

<comment><USER2> _He went to management and said, "Give me a 17% raise or I'm leaving." They didn't and he did. The fallout was cataclysmic._

I wish this happened more often. Then people would just automatically receive, as a raise, whatever it costs to hire a replacement. The problem is that people are really too nice about their compensation and don't realize that they have a business to run -- themselves.

(Man, now I want to get a business card that lists my title as "Technical Prostitute".)

<comment><USER5> Why 17%?

<comment><USER6> Maybe so the raise could ultimately be 10-15% and the company would think they'd "won."

</comment></comment></comment></comment></comment></comment><comment><USER7> "The company also began testing a mathematical formula to try to predict which employees are most likely to leave, based on factors like employee reviews."

Does anyone have more information on this?

<comment><USER8> I don't have more info on that statement, but it reminds me of this:

        When a company is filled with engineers, it turns to
        engineering to solve problems. Reduce each decision
        to a simple logic problem.

From here: <http://stopdesign.com/archive/2009/03/20/goodbye-google.html>

<comment><USER9> It'll be interesting to follow this up to see how engineers solve (or not) HR problems.

</comment></comment></comment><comment><USER10> "Mr. Schmidt wrote that company surveys indicate salary is more important to Google employees than any other component of pay, such as bonuses or equity. He added the company was moving a portion of employees' bonuses into their base salaries, so they would receive some of it in every paycheck."

Not sure, if this a 10% raise really.

<comment><USER11> The wording was a little bit unclear to me, but I interpreted it to mean that, _in addition to the raise_ , they're changing how their bonuses are delivered- rather than as one lump sum, they're being spread out over each paycheck. I could be mis-reading, though.

<comment><USER12> This is correct. The two changes are independent, and stack.

</comment></comment></comment><comment><USER13> This strikes me as misguided. Surely they cannot be equally concerned about the loss of any of their 23,000 employees. Wouldn't you be better off giving substantially larger raises to your higher performers? This way your high performers are pissed off and your low performers are happy, is that the outcome you want?

<comment><USER14> Implement a ranking system and standardized tests. Count lines of code and apply it as a bonus multiplier. Make sure to only measure things directly assigned to a person, while ignoring 'off-the-book' things like bringing new hires up to speed, code review, etc. Devote a large portion of the bonus to a 'face-time in meetings' metric. Put everyone on an artificial bell-curve measuring whim in exquisite detail.

20% of staff-time should be taken off anti-pagerank gaming efforts and redirected at anti-bonus-metric gaming.

<comment><USER15> Counting the lines of code an engineer has written is a _terrible_ road to go down. I can write 500 lines of good code, or I can abstract it into a 5,000 line heap of unmaintainable garbage.

<comment><USER16> That was the point ;)

</comment></comment></comment></comment><comment> a-googler Some asshat leaked the "CONFIDENTIAL: INTERNAL ONLY" memo. Good work, (presumably) newly unemployed person. You won't even get the goods.

<comment><USER17> With presumably 23,000 people getting that email, unless the person (if only one did) was not careful at all, I don't think they could be discovered. (I'm not sure that anyone would really care either)

<comment><USER18> Google found and terminated the employee responsible.

<comment><USER5> Got a link for that?

<comment><USER18> No, but Google notified employees internally that the person responsible was terminated. If you have a friend at Google just ask.

</comment></comment></comment></comment></comment><comment><USER19> Google's base salary is so low compared to other companies... they had to do this eventually.

<comment><USER20> Citation?

<comment><USER21> Glassdoor.com seems to provide supporting evidence of this.

</comment></comment></comment><comment><USER22> Seems like a desperation move. Can you imagine the exciting, dynamic Google of 5 years ago thinking ""Uh.. give them slightly more money?" was a great solution to this problem?

<comment><USER23> Almost every perk at Google I had at Microsoft. One of the biggest things keeping me from being fully invested in securing a position at Google is knowing that the salary is approximately the same at Google while the cost of living in Silicon Valley is much higher. You may not think about salary (beyond having enough to do what you like) but it is a very real maker/breaker for me.

<comment><USER24> You don't need to move to Silicon Valley to work at Google. Google has a large engineering office a few miles away from Microsoft's Redmond campus.

<comment><USER23> True. They also have one in Chicago. However, that doesn't mean you get to work on anything you want. Different teams are in different places and you usually get assigned to a team based on where you work.

<comment><USER24> Yeah, I heard that Google tries to colocate teams. I recommend that you ask Google what they do in Chicago and other locations that are acceptable to you. There might be something that interests you.

</comment></comment></comment></comment></comment><comment><USER25> [http://www.businessinsider.com/google-bonus-and- raise-2010-1...](http://www.businessinsider.com/google-bonus-and- raise-2010-11) claims that _In addition, Google will also give each employee an additional raise equivalent to 1X the employee's target bonus for the year._

My understanding is that Google "target bonuses" are much, much more than 10% of salary. I've heard 30%-40% as not uncommon. So does that mean that people are actually getting a much bigger raise than the headline indicates? (Or is the "target bonus" something else again?)

<comment><USER26> My understanding is that the target bonus is more in the 15% range but that there are multipliers (company and personal) that most of the time turn that ~15% into the 30-40%.

</comment></comment><comment><USER27> 10% doesn't make up for the fact that GOOG stock already peaked way back in 2007, which means any Google stock options granted at today's prices are worthless to new employees.

Google is essentially powerless to compete with Facebook's stock option potential value.

<comment><USER28> This logic seems dumb, Facebook is a big company, if you're joining Facebook now, even as a star hire, are they really going to throw enough equity at you to make you a millionaire in some unknown future IPO? And you have no idea when the IPO will happen, it could be 10 years before you can liquidate. And you might get screwed over on some dilution or the strike price or something currently undisclosed, private companies are notoriously secretive about their financials.

It's just too many unknowns. I don't think it makes sense to plan your life around hitting this options jackpot.

<comment><USER27> It's the difference between your stock options being worth $0 (in the case of Google), and they being potentially worth 10X (in the case of Facebook). I'd say 10X is a pretty big difference.

<comment><USER28> $0? GOOG is at $624, I'm sure there are a lot of googlers sitting on options with $300-500 prices.

<comment><USER29> Considering the repricing [1] that occurred earlier this year, yes.

[1] [http://googleblog.blogspot.com/2009/01/announcing-googles- em...](http://googleblog.blogspot.com/2009/01/announcing-googles-employee- option.html)

<comment><USER30> This is from Jan 2009, not this year, and I'm sure you're aware of the fact that the downfall affected the whole economy, not Google in particular.

<comment><USER29> Oops. Memories fade. At any rate, my point is that a lot of people have options with a low strike price. I'm not sure what the downfall of the economy has to do with this fact.

</comment></comment></comment></comment></comment></comment></comment><comment><USER31> I heard the top performers are getting the 10% raise and a bonus equal to 40% of base pay.

<comment><USER32> I'm not sure if bonuses have gone down recently, but a few years back (when I was there), bonuses of 40% were pretty common.

Even though some people said that Google paid "below market," when you take into account that kind of bonus (and some stock options), I think I did better than I would have pretty much anywhere else.

</comment></comment><comment><USER33> Google felt peer pressure from the Federal Reserve and decided to do it's own QE

</comment><comment><USER34> Indication of inflation on its way?

</comment><comment><USER35> This does make the battle for talent look pretty dramatic, doesn't it?

<comment><USER36> If "battle for talent" is focused on a battlefield being fought by large, enterprise companies vs. startups and more risk-adverse organizations, sure. I'm not convinced that people who opt for Google/Facebook are the right people for young companies look to take big risks with potential (vs. guaranteed) rewards.

</comment></comment><comment><USER37> For real entrepreneurs, the type of people Google needs to retain, a 10% raise is meaningless. I will say it's a nice PR stunt.

<comment><USER38> What about really real entrepreneurs?

<comment><USER33> the really real ones are imaginary

</comment></comment></comment><comment><USER39> Non-US person here. Is 10% raise supposed to be big? We get 10% raise minimum at my country (inflation is around 12.5% here)

<comment><USER29> You probably know the target inflation rate in the U.S. is slightly less than two percent, so a ten percent raise in a year is an 8% real raise. In your country it is a -2.5% real change in salary. So yes, of course a 10% raise is a bigger deal here than it is where you live.

Now, is it life changing? Not for me. Maybe if you were only making $20,000 a year, or if you were spending right up to the edge of what you make, an extra 10% of breathing room would make a huge difference in your life. And sure, some people if they change job function or get promoted in a U.S. company can get anywhere from 20-40%, all the way up to multiples of 100% raises. But even though it is not out of this world, a raise like this is nice. It's also nice to get something for "nothing" above and beyond what you're already doing, which is the effect a blanket raise must necessarily have.

</comment></comment><comment><USER40> A</comment>


Taliban join the Twitter Revolution - <AUTHOR> http://www.guardian.co.uk/world/2011/may/12/taliban-join-twitter-revolution

<comment><AUTHOR> The twitter account mentioned in the article: <http://twitter.com/alemarahweb></comment>


The One Issue the Left and Right Can Agree on – Monopolies - <AUTHOR> https://newrepublic.com/article/152680/one-issue-left-right-can-agree <comment><USER2> The benefits of a more assertive Congress would be profound. To take just one industry, Congress could save the average American family $10,000 each year by implementing straightforward health care pricing rules that would have all medical providers paid at Medicare prices. Now imagine if it fought corporate concentration across the economy.</comment>


Open-Sourcing Yahoo's Pulsar, Pub-Sub Messaging at Scale - <AUTHOR> https://yahooeng.tumblr.com/post/150078336821/open-sourcing-pulsar-pub-sub-messaging-at-scale <comment><AUTHOR> Github page: [https://github.com/yahoo/pulsar](https://github.com/yahoo/pulsar)

Pulsar backs major Yahoo applications like Mail, Finance, Sports, Gemini Ads, and Sherpa, Yahoo's distributed key-value service.

On the scale front:

\- Deployed globally, in 10+ data-centers, with full mesh replication capability

\- Greater than 100 billion messages/day published

\- More than 1.4 million topics

\- Average publish latency across the service of less than 5 ms

</comment><comment><USER2> I worked at Yahoo but have never heard of "Pulsar" before. Was this known as "CMS" internally?

<comment><USER3> Yes.

</comment></comment><comment><USER4> I wonder how this compares to Kafka and what tradeoffs were made.

<comment><USER5> I'm one of the Kafka authors, so admittedly my view might be slightly biased

Here is a quick comparison of Kafka and Pulsar:

\- Kafka is a complete streaming platform vs a messaging system which is what Pulsar is. Through Kafka Connect ([http://www.confluent.io/blog/announcing- kafka-connect-buildi...](http://www.confluent.io/blog/announcing-kafka- connect-building-large-scale-low-latency-data-pipelines/)), it has support for connectors to stream data between various sources and systems. Through Kafka Streams ([http://www.confluent.io/blog/introducing-kafka-streams- strea...](http://www.confluent.io/blog/introducing-kafka-streams-stream- processing-made-simple/)), it has support to do stream processing and transformations over Kafka topics.

\- Broad adoption base: Kafka is very widely adopted across thousands of companies worldwide. [https://cwiki.apache.org/confluence/display/KAFKA/Powered+By](https://cwiki.apache.org/confluence/display/KAFKA/Powered+By)

\- Tunable durability and consistency knobs on the producer: The Kafka producer API allows the application to either wait until a message is fully committed across all replicas or just the leader. This allows applications to make the right tradeoffs for throughput vs durability. One size does not fit all.

\- Performance and efficiency: Kafka supports zero-copy consumption allowing the consumers to read large amounts of data at high throughput. To the extent that I understand, Pulsar with its legder-broker model does not support zero- copy consumption.

\- A lot of the reasons quoted for creating Pulsar are features that exist in Kafka and are used in production:

\-- Kafka has multi-tenancy support through user-defined quotas (See this [http://www.confluent.io/blog/sharing-is-caring-multi- tenancy...](http://www.confluent.io/blog/sharing-is-caring-multi-tenancy-in- distributed-data-systems))

\-- Kafka has support for authentication, authorization, user-defined ACLs (See this [http://www.confluent.io/blog/apache-kafka-security- authoriza...](http://www.confluent.io/blog/apache-kafka-security- authorization-authentication-encryption/))

\-- Kafka has support for geo replication. In fact, that is the most common use case for Kafka in several companies. (See this [https://engineering.linkedin.com/kafka/running-kafka- scale](https://engineering.linkedin.com/kafka/running-kafka-scale))

\-- Latency: The end-to-end latency from publish to consume can be very low in Kafka (<10ms).

\- Support for millions of topics: To the extent that I understand, both Pulsar and Kafka use ZooKeeper for metadata management. That is the main bottleneck for supporting a large number of topics and likely the same tradeoffs apply to both Kafka and Pulsar as a result.

\- Storage model: The length of a partition in BookKeeper and hence in Pulsar is not bounded by the capacity of a server. So you have the ability to add servers to accommodate a workload spike.

This is merely a quick overview. There might be more aspects of this comparison that I'm missing.

<comment><USER6> How do you compare the geo-replication capabilities of Kafka vs. Pulsar?

<comment><USER7> Added some info on geo-replication in Pulsar at [https://github.com/yahoo/pulsar/blob/master/docs/GeoReplicat...](https://github.com/yahoo/pulsar/blob/master/docs/GeoReplication.md)

</comment></comment></comment></comment><comment><USER8> I'd like to see a blow by blow comparison of Twitter DistributedLog vs Kafka vs Pulsar. Particularly focusing on what each solution means by "geo- replication" ,what geo consistently guarantees are made, how operable the geo rep features are in practice etc. At first glance those interested in not losing data and interested in geo-replication and constituency are better off going with DistLog rather than either of the other two solutions.

However if all you care about is a single DC then K is simpler as long as you are happy for consumers to track offsets.

But if you want broker side delivery tracking then perhaps go with Yahoo or kafka with lazy commits of offsets to zk.

</comment><comment><USER9> Interested to find/read any comparisons to TIBCO EMS, software rolling out for large scale use in government circles.

</comment><comment><USER10> Before Obligatory "What is so different about this from Kafka?"

Edit- Got wrong product.

from the looks of it, it just seems to be a slightly different take on Kafka. From what I gather, looks like Pulsar allows for scaling of producers/brokers independently?

<comment><USER11> There is an another system from eBay named Pulsar [0] for Analytics. They should do some research before selecting a project name. Quick idea - Quesar ?

[http://gopulsar.io/index.html](http://gopulsar.io/index.html)

<comment><USER12> Quasar probably? Quesar is cheesy :)</comment></comment></comment>


Blast and fire hits major virus lab in Russia, where smallpox is stored - <AUTHOR> https://www.rt.com/russia/468919-russia-virus-lab-blast/ <comment><AUTHOR> Smallpox is apparently stored in this lab</comment>


How to secure an Ubuntu 12.04 LTS server - <AUTHOR> http://www.thefanclub.co.za/how-to/how-secure-ubuntu-1204-lts-server-part-1-basics

<comment><USER2> Should "change ssh port" even really be on the list?

<comment><USER3> I moved the front door of my house to the back, haven't had any burglars brake in. Can't argue with that!

<comment><USER4> The the only reason _I_ change my ssh port is to reduce log file noise. Not as a security mechanism.

Oddly enough, I have also run across networks where outbound port 22 was blocked. Handy for that too.

<comment><USER5> Me too. I've used a different ssh port for eight years. Not a single log entry from ssh brute forcing idiots during this time.</comment></comment></comment></comment>


The Manhattan Project of Marijuana: Lab Testing and Standardization - <AUTHOR> http://www.eastbayexpress.com/ebx/PrintFriendly?oid=936926

<comment><USER2> This operation has the loose, fly-by-wire feeling that startups do, and it's also trying to do something very technical (and illegal). You get the feeling from reading the article that, while they have a great vision, the actual execution is tentative, start-stop, and incremental. It sounds like fun.

I think they could really help advance research in this area. They are doing something necessary and very helpful, and there is tremendous upside.

But heck if I'd want to be doing it.</comment>


Exhibit 8: Excerpts from AirBed and Breakfast's Application to Y Combinator - <AUTHOR> http://i.imgur.com/55yP1CZ.jpg <comment><AUTHOR> This is from the Harvard Business School case study on Airbnb. Surprisingly I've never seen their application online before so I thought I'd share this valuable information with everyone.

You can read the original case study at this link: [http://www.hbs.edu/faculty/Pages/item.aspx?num=40931](http://www.hbs.edu/faculty/Pages/item.aspx?num=40931)</comment>


Yahoo CEO Mayer Apologizes For Mail Outage That She Says Affected 1% Of Users - <AUTHOR> http://techcrunch.com/2013/12/13/yahoos-marissa-mayer-apologizes-for-mail-outage-that-she-says-affected-1-of-users/ <comment><USER2> "After a week of Yahoo Mail outages that began four days ago"... how does that work?</comment>


You can't parse [X]HTML with regex - <AUTHOR> http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454 <comment><USER2> I've always loved that post but the truth is that while you can _not_ parse [X]HTML with what I'll call a "regular expression", by which I mean the formal CS definition [1], you can with a suitable "regex", by which I mean what most folk mean by the term "regex".[2]

[1] [https://en.wikipedia.org/wiki/Regular_expression#Formal_defi...](https://en.wikipedia.org/wiki/Regular_expression#Formal_definition)

[2] PCRE engines support recursive matching etc. but perhaps the most illuminating example is regex in Perl 6 such as this JSON grammar (a Perl 6 grammar is a class containing Perl 6 named regexes): [https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Gra...](https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Grammar.pm)</comment>


Show HN: PubStorm – Free CLI-based static web hosting with Let's Encrypt support - <AUTHOR> https://www.pubstorm.com/ <comment><USER2> Are you guys sustainable, given your free tier has sufficient functionality for many use cases?</comment>


Saving generated files on the client-side - <AUTHOR> http://eligrey.com/demos/FileSaver.js/

<comment><AUTHOR> MIT licensed source code: [https://github.com/eligrey/FileSaver.js](https://github.com/eligrey/FileSaver.js)

In case it's not clear, this allows you to generate a file from client-side JS, and allow the user to save the generated file _without first having to upload the generated file to the server_.</comment>


Why Firefox 4 isn't abandoning Windows XP - <AUTHOR> http://www.geekwire.com/2011/firefox-4-engineers-decided-stick-windows-xp <comment><USER2> Hardly surprising, but it's an interesting situation. The only company in the world that wants and needs Windows XP to disappear is Microsoft; for everyone else XP is an enormously bountiful source of users and revenue. It would be suicidal for Mozilla or Google to say "we won't support XP", just as it's fatal for Microsoft to linger on an 11-year old platform that eats into their revenue stream. Interesting dilemma.

<comment><USER3> How close are projects like Wine or ReactOS to full XP compatibility? Is there an opportunity here?

<comment><USER4> ReactOS is just WINE on top of a custom Windows-like kernel. All of the real action in reimplementing Windows occurs in WINE, afaik.

It's pretty far from "full XP compatibility", but a lot of stuff works just fine. I'd love to see WINE considered a big investment center for people like Canonical. I think WINE and open graphics drivers are the two biggest parts holding back desktop Linux. Even if WINE could get .NET compatibility, that'd be a huge step forward (right now it has trouble running any of MS's .NET frameworks correctly). The reality is it's going to take a lot of money and especially time to cause the lock-in of Win-only apps to fully regress; it'd be a lot faster to work with it and cause almost everything to work perfectly on WINE. Office and .NET are the biggest hold-ups atm.

The problem with major investment in WINE is that as soon as MS thinks that suing CodeWeavers/any other WINE investors or distributors would be less harmful than allowing the project to continue without the extra attention, they're going to drop some big patent litigation all over that. MS is not just going to sit back while WINE plays a fundamental role in killing Windows -- as soon as it becomes serious, look for a major, relentless attack by MS.

<comment><USER5> AFAIK, ReactOS is also able to use Windows drivers. And the best option you've got for .NET-compatibility is Mono.

But yes, it will never be Windows and will always have technical and legal problems.

<comment><USER4> Yes, the ability to use native drivers on ReactOS stems from writing a kernel that mimics the Windows kernel API/ABI as closely as possible. The ReactOS userland, however, is provided primarily by WINE. You can think of ReactOS primarily as an open kernel compatible with Windows's kernel API, and WINE as an open userland compatible with Windows's userland APIs. ReactOS isn't rewriting WINE, they use WINE extensively. The main unique work of ReactOS is the Windows-compatible kernel.

Mono is helpful for pure .NET directly from a Unix environment. However, Mono is more of an open .NET development environment than a .NET compatibility environment; while Mono can run a lot of .NET apps that are made for Windows, that's almost incidental of compatibility with the CLR bytecode. There are important implementation differences that Mono is aware of and has no intention to correct because, unlike WINE, Mono is not really a compatibility layer.

Additionally, many Windows programs launch .NET programs directly, and afaik it's not very easy to get them to use Mono by default. For instance, I recently purchased C&C 4 on Steam, but it's a pain to make it work because circumventing the .NET-based launcher wants me to include an EA ID (I could use a fake auth server but just don't feel like putting that much effort in), and launching directly doesn't work because of the .NET-based launcher (.NET doesn't work). Further, more and more products are relying on .NET components, including Visual Studio, Office, and other big things. I think getting the .NET runtimes working well should be one of the main goals for WINE devs.

</comment></comment></comment></comment></comment><comment><USER6> So what's all this nonsense about XP being 10 years old and so customers who bought it not being worthy of a newer browser? Do you know when Microsoft actually stopped selling XP to customers?

Trick question, they haven't. Despite several passed deadlines the latest plan, as of July 2010, is to continue selling downgrade licences to businesses up until two years after Windows 8 ships!

<comment><USER7> Right. Very large companies are still tied to XP, and therefore Microsoft is. IBM's standard load is still XP, and I haven't seen any Vista or 7 plans yet. I do suspect that they'll skip right to 7 when it's considered "mature" and they allocate the resources to clean up the enormous technical debt accumulated with the in-house applications.

</comment></comment><comment><USER8> The elephant in the room is that Chrome is compatible with XP and this might the real FF competition. Heck, even Safari works on XP.

<comment><USER5> Why is it an elephant?

Firefox 3.x works perfectly fine on WinXP too.

</comment></comment><comment><USER9> According to a friend that just got a new work laptop, Windows XP is still standard issue at IBM. I imagine plenty of other large companies are in the same position.

</comment><comment><USER10> Neither Chrome nor Opera are abandoning XP either.

All three will support hardware acceleration on XP.

Only one that left tens of thousands (maybe millions?) of XP users out in the cold is Microsoft.

</comment><comment><USER11> i think this is a smart move by mozilla. i hope they see a significant uptick in browser share on those platforms.

i'm sure it wasn't easy to dedicate all the extra work up front and over the life of the project...

<comment><USER12> Also will probably bleed into future platforms - people stick with what they're comfortable with, so once they switch to FF/Chrome/Safari they'll likely seek out that browser on whatever new machine they get after their hardware dies.

</comment></comment><comment><USER13> It's at a point where I don't see anything wrong with going either way.

I don't think IE9's abandonment of XP is going to slow the adoption of new web technology all that much, something that Mozilla seems to suggest. There is a lot of momentum behind them with mobile devices supporting them so well. Still, 40% of web users, even if those are among the lowest value web users, is worth developing around.

Mozilla isn't around to make a profit, so spending on XP users makes more sense. Abandoning them would have also made sense.

<comment><USER14> Mozilla wants its cake and to eat it too. They'll whine that IE9 doesn't support XP, and then whine that IE9 isn't modern. If it's not modern then it really doesn't matter if it supports XP or not.

I personally can't stand XP as an OS. But I do understand Mozilla from a business perspective. Although I wish they weren't so sanctimonious about IE9 not supporting it since they really don't care and I think it just continues to make Mozilla look like whiners rather than doers, e.g., Google/Chrome.

<comment><USER15> Mozilla is trying to position FireFox a as superior alternative to IE9. Therefore, there's no cognitive dissonance at all with Mozilla saying that 1) IE 9 is worse than Firefox because it doesn't run on XP and 2) IE 9 isn't really modern while Firefox is.

Mozilla doesn't want IE 9 to be modern, but more than that, it doesn't want users to think that IE 9 is modern. Mozilla doesn't want IE 9 to have any compelling features whatsoever.

A negative marketing campaign will always backfire to some portion of your audience.

<comment><USER6> Mozilla do want IE9 to be "modern", that's entirely in line with their mission as a non-profit that promotes the open web. It's ironically Microsoft that's scared most of a truly modern IE9.

What Mozilla certainly don't want is marketing BS confusing users into thinking IE9 is modern, while it holds the web back and drags its feet about updates. Nor does it want any browser controlled by Microsoft to have dominant market share while Microsoft still sees the web as a threat that it can fight mostly by go-slow tactics like having IE6-8 hanging around preventing new web technologies hitting critical mass.

They don't even have spellchecking in their browser because the Word guys feel threatened.

<comment><USER14> _They don't even have spellchecking in their browser because the Word guys feel threatened._

I suspect it may have more to do with the fact that some features are easy to do as addins. There are several spell checkers available for IE. Given that the IE team seems resource constrained, it seems more useful to work on things like perf, compliance, security, than features that are easy to add by 3rd parties. I'm not saying it wouldn't be nice to have, but given that IE9 still has HTML/CSS gaps, I'd much rather they work on those.

<comment><USER16> Why would the IE team be more resource constrained than the Firefox team, who managed to add a spellchecker years ago? I'll admit that I've not gone looking, but I've not ever heard of anyone using one of these third party IE spellcheckers - it's just been a feature which I expect a decent browser to have that IE clearly lacked. Surely that perception hurts them more than the cost of a few developer-months to take one of their existing spellcheckers and drop it into IE?

<comment><USER6> The IE team is resource constrained because it's not in Microsoft's interests to build a good browser. If it was in their business interests, like XBox, then they could throw tens of billions of dollars at it. But it's not, so no spellchecker and no WebGL and no IE9 for XP.

<comment><USER14> You do realize that the XBox team is also resource contrained. For example, you can't run any XBox or PC game on an XBox 360. Had to cut something. There's not even a web browser in the XBox 360.

Everything is about tradeoffs and constraints. You may not be familiar enough with the industry to know that they're there (ppl think that MS and Apple have enough money to simply do everything, but they don't), but trust me, these products have serious resource constraints across the board.

<comment><USER6> Strangely, you seem to think Xbox trying to conquer the living room without bothering to include a browser supports your case that Microsoft loves the web, but just consistently runs out of money across its various products before getting around to supporting it properly, whereas I would have had that right up there with the lack of spellchecking in IE as a clear example of Microsoft quietly hoping the web will go away if they ignore it long enough.

I'm not asking or expecting them to support everything and anything with their minuscule bank account and negligible market power. I'm just pointing out that the web is kryptonite to Microsoft and their PR to the contrary is laughable. If you want to believe it's simply mismanagement on a massive scale that leads them to fail at the web year after year then good for you, but I personally think it's just smart self-interest on their part.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> It's hard to make money selling stuff to cheap people, and it's hard to gain users by chasing people who are ok still using XP, for whatever reason.

It's nice that Firefox is still supporting XP, but I don't think the decision should be based on the % of web users still on XP. It should be based on the % of them likely to be interested in downloading a new browser.

</comment><comment><USER18> I sort of take the Apple philosophy: Thou Shalt Upgrade Thy Software.

Keeping software modern is a good thing.

<comment><USER19> Forcing an OS upgrade is in ms's interest since its likely people will likely stick with the built in one, which is Microsofts browser.

Keeping backwards compatibility is in mozillas interest since people are likely to change their browser than their OS. This means xp users are likely to switch to mozilla than Microsoft.

</comment></comment><comment><USER20> You can't always make decisions based on dollars and cents, sometimes its about just being good to your users and that is one thing that Microsoft has proved incapable of over the years.

For them its always business ... which is why I hope they eventually lose the browser battle.. They act like people on Windows XP didn't also pay money for their copies of windows ... but all this will do is give people a solid excuse to use other browsers on Windows XP as IE 8 starts to age.

Thanks for letting Chrome and Firefox an unencumbered run at your market share Guys!

<comment><USER21> Maybe they are having problems, like Apple, with spending money on old operating systems?

Windows XP users would never pay for IE 9, and Apple claim they can't give XCode 4 away to users from an OS released 18 months ago.</comment></comment>


Debian Hardener - <AUTHOR> https://github.com/CrystalX127/Shield <comment><AUTHOR> Would anybody mind just checking out this Debian Hardener I've created.

<comment><USER2> I see a lot of problems here. Consider scaling back what you are attempting to do.

\- The auditd rules you're using look highly application-specific. Installing them blindly is probably not appropriate.

\- Some of the changes made by this script are not persistent. In particular, changes made using "mount" will not persist beyond a restart, and changes to permissions on files installed by packages will be overridden when the package is updated.

\- Your script attempts to disable USB storage, FireWire, and some filesystems (like HFS+). These steps may be unexpected and undesirable for desktop systems.

\- Your script attempts to write to paths which will not exist on a Debian system, like "/etc/sysconfig/init".

\- Many of the snippets used by this script make assumptions about the base system configuration, like the names and purposes of its network interfaces. These assumptions need to be checked.

\- The IPtables rules installed by this script are extremely questionable. They include a number of sample rules (like one that allows IP forwarding on weekdays between 9:30 and 10:30 PM, one that blocks packets containing the strings ".com" or ".exe", or one that blocks certain traffic from 18.104.22.168), some rules which will limit the functionality of the system (like the rate- limit rules for port 80, or the rule to "force fragments packets check", which actually _rejects_ fragmented packets), and some rules which do nothing at all (like the unreferenced "port-scanning" chain, or a rule that attempts to "block new packets that aren't SYN packets"). Additionally, the rules will block all traffic from/to RFC1918 networks, which will block all network connectivity on any system that's on a private network (like a home network or a VPC).

\- The three (!) firewalls which this script will attempt to install (iptables-persistent, shorewall, and ufw) are incompatible with each other.

\- The "core file permissions" snippet does nothing, because it attempts to change permissions on files in the current working directory.

\- Some of the features that this script prompts for (like "revert_/root_permissions") are simply unimplemented, and will throw an error if selected.

<comment><USER3> also to add some items (of course this is all my opinion, so ignore it if you want):

\- break up the script into modules and work on one at a time until its solid \- don't assume that items like unattended-upgrades are desirable. In general, where you want to install more software than the user already has installed, it should be an optional module or prompted. Without that I wouldn't use this script for that reason alone. \- some people have a purpose in using firewire and USB. Make it an option (my example: high end studio cameras) \- consider reviewing [https://wiki.mozilla.org/Security/Guidelines/OpenSSH](https://wiki.mozilla.org/Security/Guidelines/OpenSSH) and the release notes for the version of OpenSSH you are configuring. I wouldn't use your configuration. \- why ask how many users to enforce diskquotas on and then ask for which user to enforce a disquota on? Simpler would be to read /etc/passwd (or count /home/<users>), provide a list to mark and then do it.</comment></comment></comment>


Smear Campaign Ramps Up Against Protect IP Opponents - <AUTHOR> http://www.techdirt.com/articles/20110705/02174514963/smear-campaign-ramps-up-against-those-who-believe-free-speech-is-more-important-than-hollywoods-obsolete-business-model.shtml

<comment><USER2> I wonder how far we can go with placing all value on the short term _at the expense of_ long term. I hesitate to make any predictions because I would have expected to see serious fallout before now. Looks like people like Mr. DuPuis want to find out though.</comment>


New York Magazine's Boy Genius Investor Made It All Up - <AUTHOR> http://observer.com/2014/12/exclusive-new-york-mags-boy-genius-investor-made-it-all-up/ <comment><USER2> Between this and the Rolling Stone U.Va-rape story...this has been a bad month for New York-based magazine reporting (and a boon for media critics). The NYmag reporter insists she made no error, because her lede paragraphs talks about a "rumor". Yeah, but the point of being a journalist is to find out the truth behind rumors, not just to say, "Hey, did you know that there are rumors about a high school boy?"

The reporter claims she saw "bank statements"...the validity of which seem to be in question if the boy has recanted everything. I wonder if she got taken in by the good-ol edit-the-HTML-of-official-documents-with-the-Web-inspector trick?

On the upside, at least this teenager doesn't have to bear eternal Internet shame for being part of a public prank/lie. His name is pretty much Google- proof, or at least as Google-proof as someone named "Jesus Christian" would be. That said...I just Googled "Mohammed Islam" and was pretty surprised to see all the top results relating to the young man at hand. I would've thought that Google's algorithm would've associated such a query with Islam the religion and the prophet Muhammad.

<comment><USER3> Yes, this story made me cringe really hard as I was reading it last evening. First off, if the reporter just bothered to speak with _only one_ person who ever traded derivatives, she would have learned this was a complete hoax. Simply put, even if the kid started with $1m and ended up with say $60m after two years (being generous all around given the context), that would still amount to 6,000% return. Here is why that couldn't happen -

 _He made directional outright bets._ In other words, say he bought call options betting for the price of the underlying security to go up. This is akin to buying underlying security with massive leverage. If the bet goes your way, you could make a massive return (for example, in Oct '05, I made ~1,000% one-day return on a very small amount buying Google out of the money calls expiring the next day and selling them in the money after stock picked up following earnings report). This is an incredibly dumb strategy and it is completely unscalable to the figures quoted in the article. First, public/listed options are incredibly illiquid in most stocks/securities, and particularly at strikes that could yield this return. This would limit him to making really small bets. Second, you would need a catalyst for a significant price move, which doesn't come too often, so most of the time you would just bleed your option time-value ("theta decay"). Finally, and most importantly, no one is lucky 100% of the time and this type of directional strategy is sure to backfire on you and cost you ~100% of principal (trust me, been there, done that)

 _He traded volatility._ This is effectively a hedge fund strategy, where you buy the option and short the delta of the option in underlying security ("delta hedging"). The hedge gives you protection against the price moves in the underlying and you effectively make money for small moves in the security, regardless of whether it goes up or down (due to "convexity"). Most of the time, you earn pennies on the dollar as the price fluctuates a bit and you trade your delta, and perhaps some of the time you make a bit more if the price moves a lot ("gamma event"). Now, here is the kicker - this is not fool proof either. The strategy trades the risk in the price of the underlying security to the risk in volatility - in other words, if you buy volatility high and sell low, you will still lose money. Most listed options have notoriously bad bid/ask volatility spreads and retail investor would unlikely make any money consistently and reliably enough following this strategy. Needless to say - no way to scale it to the numbers quoted in the article using listed options. Hedge funds and derivative desks at banks mostly trade in OTC (Over The Counter) options which are bespoke bilateral contracts with a ton of protective provisions (averaging periods, stock borrow protection, illiquidity protection, etc) and they still have to take a view on volatility and they do get it wrong sometimes. Even for professionals who are wicked smart and do this day in and day out, the only reliable way to make money is to buy volatility on the cheap in a special situation.

There are a few other strategies that come to mind (e.g. Black Swan) but there has been either no catalyst to make it work or they would be inaccessible to a high school kid from Queens.

The second part that really made me cringe were all the negative references to SV by the kids and by the reporter in her subsequent Tweets. Reality of the situation is that, whatever you may think about early stage bubble in SV, SV tech companies actually create real value, have actual products and serve actual customers/users.

Finally, what is sad about all this is that Mohammed appears to be a smart talented kid and I can't help but think he tainted his prospects quite a bit with this debacle. On the other hand, all this makes a hell of a "what I learned from a failure" college essay.

<comment><USER4> Ditto.

I was once told that in vol trading, if you're "making more than 60% a year, you're doing something wrong". The point being that with any option spreads, there is a risk-reward ratio, professional's care about money management than profit. You're trying to earn on the theta decay or vice versa, facing theta decay and banking on gamma moves. So you're either facing the death of a thousand paper-cuts with option time premium eroding your bankroll, or you're facing the LTCM catastrophic loss with the wrong kind of risk management.

Even when you trade directional, this past year, both realized and implied vol has been low, with VIX bouncing between 11-20 with median at 13 or 14; you can make lots of money long OTM options in 2008 or flash crash, basically you're playing Nassim Taleb's "long-tail"; this past year simply doesn't warrant that kind of performance unless you're doing some kind of OTM buying on biotech companies and betting correcting on a majority of your catalyst events.

<comment><USER3> Yep, you are spot on about how it works in the real world. Incidentally, you may get all three (i.e. small profits from daily delta adjustments + theta decay + gamma upside) on some trades, but the bottom line is that it always requires a vol view when you enter into the position, which lends it self to being wrong sometimes. There, you hit on another interesting point - even with all of the mess brewing in Ukraine, realized has been relatively low (and probably way below desks' expectations) this year, which I would bet caused some underperformance across the vol desks this year.

Don't know what was realized vol in oil lately (likely high given recent price action, have no sense of the shape/magnitude), but perhaps that's where he's seen some returns in simulated trading. Needles to say simulated trading is very, very different than the real world, with bid/ask spread dynamics, options illiquidity, collateral requirements, funding costs, borrow costs, etc.

</comment></comment></comment></comment><comment><USER5> Honestly, this story is so unbelievable that it leads me to believe that many journalists have no idea how investing or the stock market work. Even if he started with $100,000 to invest, it would be an insane annual yield. A yield so high, that even crazy frauds like Bernie Madoff wouldn't promise it.

It's not that you couldn't make $72 million dollars by investing in the stock market in a few years, but rather that it would usually take a rather large principal to do so.

This isn't the same as the UVA rape story, as that took months of reporting and fact checking. That's one of the biggest journalistic mess ups in the last decade or so, and will have far reaching consequences for Rolling Stone. We'll be discussing the UVA rape story for the entire next episode of my podcast.

This is the more run-of-the-mill story about how a lot of journalists have so little knowledge about things outside of journalism that they get duped by these stupid and false feel good stories. This is more embarrassing than harmful.

<comment><USER6> It seems that many "journalists" these days can't be bothered to learn much about anything that they "report" on.

I miss the old days of "two sources or we don't print".

<comment><USER7> Just a few days ago the New York Times took a google translate of an Italian Huffington Post article as the source for a papal proclamation.

</comment></comment></comment><comment><USER8> If anyone fron NYMag is reading this -- I'm the son of late Nigerian king Joseph Mabadu and would like to share my billion-dollar fortune with him if he can help me transfer my wealth out of the country. As a show of good faith, I'm also willing to hand him over my title for the Brooklyn Bridge. Please call urgently to discuss...

</comment><comment><USER9> The average person would be blown away at how sloppy (and really non-existent) fact-checking has become in the Internet age. Most people on HN seem to be a little bit pessimistic about it, but it's likely way worse than you think. Of course, we know that random people retweeting garbage on Twitter aren't fact- checking, but I'm talking about the largest media organizations around -- they just don't care enough to check.

We don't need to look any further for this than ISIS. Not only is ISIS controlling the whole world's talking points by carefully crafting what news they'll release, but they're spreading stories all the time that are completely bogus. Look, for example, at this story about ISIS crucifying one of its own members for corruption: [http://english.alarabiya.net/en/News/middle- east/2014/06/27/...](http://english.alarabiya.net/en/News/middle- east/2014/06/27/ISIS-crucifies-one-of-its-own-in-Syria-for-corruption-.html). It all started with a tweet from a now-deleted Twitter account, because ISIS wanted to spread the idea of how intense they are and how strongly they fight their perceived evil. So they plant a story about how they crucified one of their own members. Every media organization knows that's free pageviews, so they report it.

But look at the feature photo in that story, which constitutes most of the "tip" from the tweet. Do a reverse image search and you find the original source: It's a Danish Roman Catholic reenacting Christ's crucifixion in the Philippines. [http://d.pr/i/183Sf](http://d.pr/i/183Sf). The story of the ISIS crucifixion was made up out of thin air, based on a photo that was easily discoverable (and in the AFP database), but now the lie is in every mainstream media publication I can think of, from CNN and The Telegraph to The Guardian and the New York Times. They didn't even have time for a reverse image search. And ISIS wins.

A lot of what my startup does is fact-check stuff that's going on in the media (it's a newsroom for the Internet - [https://grasswire.com](https://grasswire.com)), and every media organization (and I mean _every_ ) from Al Arabiya and the New York Times to Newsweek to Rolling Stone and New York Magazine are quite frequently full of inaccuracies and intentional lies. Sometimes people catch these lies, a few people freak out for <24 hours, and we all forget about it.

The notion that a publication's relevance is based purely on its accuracy is, in my opinion, false. We don't have the time or energy to process which media organizations we trust and which have mislead us, because, unless we're ruthlessly sitting behind a computer fact-checking stuff, we probably don't even know that they're making mistakes in the first place. For every one person The Daily Mail pisses off by publishing its false stories, 100 people are reading it and saying, "Wow, The Daily Mail does some daring reporting." Very few people care anymore -- we don't have time to.

In Trust Me, I'm Lying: Confessions of a Media Manipulator, Ryan Holiday talks about how easy it is to get lies to spread throughout the Internet, and thereby all of mainstream media. The premise is very simple. You find someone at the bottom of the press pyramid - the person responsible for putting out 4 or 5 articles a day so that his or her publication can get enough pageviews to stay alive. There's not a chance that person can do any sort of verification. You send them a "tip," get a few smaller blogs to write about it, and the snowball starts rolling.

These publications all read each other, and most of them just want to pick up stories from each other and ride the wave of whatever's going viral. That's not just cat videos; that's stories about ISIS and Russian fighters in Ukraine and what we consider "hard news." The stuff we all talk about in the United States is mostly driven by PR folks who are carefully planting stories that push their agenda forward. In most other countries it's even worse. Few of my Russian friends are even aware of the fall of the Rouble and the pending collapse of their own economy.

I don't know what the solution is (I'm trying to find/create one), and I don't want to pretend like this has only happened since the Internet came into existence, but I find most people put way too much trust in the press machine we've created. I would like to think that somewhere there are independent reporters and people who are digging up stories, but I've come to realize that most stories aren't ever dug up - they're pitched.

<comment><USER10> There are websites that fact check politicians, call them out on big lies, give them ratings etc. Why isn't there the same for mainstream news?

<comment><USER9> I don't know; I'm trying to create one ([https://grasswire.com](https://grasswire.com))

</comment></comment></comment><comment><USER11> No fucking shit. Seriously guys? Was this a surprise to anyone?

If it's too good to be true, then it probably is. Words to live by.

<comment><USER12> On the other hand, Facebook _did_ buy WhatsApp for $18bn.

<comment><USER13> When they were bought, WhatsApp had hundreds of millions of users every month in parts of the world most companies barely reach.

This was a high school student supposedly beating the market to the tune of $72m during lunch period.

</comment></comment></comment><comment><USER14> The real shame is that this article would have still been great _even if she 'd done the fact-checking and found out he was lying._ You've got two kids in suits buying apple juice and caviar, trying to convince a grown woman that he's worth $72 million. It's the makings of a great Vice article. I would have loved to read about these two kids as they try to pull a highly-transparent confidence scam while all the adults in the room know they're lying.

</comment><comment><USER15> One thing I'm still confused about: if the parents have literally excommunicated him/functionally (temporarily) disowned him, who's idea was the PR firm, and moreover, who is paying for it?

<comment><USER16> I doubt he's literally disowned. This sounds like some hyperbole on his part, as well as theirs. Look at the quote:

> So what did your parents think when they're reading that you've got $72 > million? > Mohammed Islam: Honestly, my dad wanted to disown me. My mom > basically said she'd never talk to me. Their morals are that if I lie about > it and don't own up to it then they can no longer trust me. … They knew it > was false and they basically wanted to kill me and I haven't spoken to them > since.

He didn't say they haven't spoken to him; they didn't say they did disown him. I've had my parents angry at me, to the point where they've yelled and said "I could kill you!" \- but they're not literally going to murder him, and they're probably not literally disowning him.

<comment><USER15> I didn't mean to suggest they were literally disowning him — that's why i said "functionally" and "temporarily".

</comment></comment></comment><comment><USER17> This is the really dangerous consequence of chasing after stories that have nice narratives. If a story fits your ideological narrative perfectly, you should be twice as skeptical - real life stories are messy, imperfect, and very rarely perfectly illustrate whatever theory you have in mind.

<comment><USER18> > If a story fits your ideological narrative perfectly, you should be twice as > skeptical

That's exactly what Richard Bradley said (first prominent person I know of who publicly doubted the Rolling Stone UVA story; he was Stephen Glass's editor at George magazine)

[http://www.richardbradley.net/shotsinthedark/2014/11/24/is-t...](http://www.richardbradley.net/shotsinthedark/2014/11/24/is- the-rolling-stone-story-true/)

"The lesson I learned: One must be most critical, in the best sense of that word, about what one is already inclined to believe."

<comment><USER7> On another forum I was on, there was a guy with the handle Japhy who coined Japhy's Law: that which you most want to be true is that which you should be most skeptical of.

It came up _all the time_ in political discussions as various participants would pull in sources showing how their side was right, but would crumble when subjected to the slightest examination.

</comment></comment></comment><comment><USER19> Did anyone actually believe the story when they first read it? My first thought was "Yeah, right."

<comment><USER20> I doubt anyone here believed it. The rest of the Internet? All bets are off.

Look at how many Buzzfeedy hoaxes get passed around like wildfire on Facebook, week after week, often by the same people in your circle. Some people -- and by "some," I mean a disappointingly large proportion of people -- don't think critically about what they encounter on the web. For them, it's not about the article. It's about the headline, and how the headline makes them feel about themselves, about the world, or about their friends. By the time they see a catchy or feel-good headline, they're already racing out to share it with everybody. If they actually get around to reading the article attached to the headline, the reading is incidental.

</comment></comment><comment><USER21> In my opinion the VAST majority of people never fact check, and most of the remaining people don't fact check unless the reporter's views strongly conflict with their own. In the current case being discussed, some embarrassment, and probably some figurative spanking are the aftermath.

In other more serious cases (crime, armed conflict, group terrorism, state terrorism) the aftermath of such a pattern is much more dramatic.

The hard truth is that media outlets need money to survive, to get that money they do two things: publish News/views that won't upset people paying for ads; and publish scoops that sell. No one is going to read a magazine or news "paper" that is broke.

But the hard question is: how do you effectively encourage properly done objective journalism? Is there a Reddit style karma scheme for news outlets? I imagine this will be effective if taken into account as an SEO signal.

</comment><comment><USER22> It looks like at least one story is still up: [http://nymag.com/news/articles/reasonstoloveny/2014/mohammed...](http://nymag.com/news/articles/reasonstoloveny/2014/mohammed- islam-stock-trading/)

Can't find a reaction yet.

Edit: they updated it. "We presented it as a rumour all along".

<comment><USER23> _An unbelievable amount of money for anyone, not least a high-school student, but as far as rumors go, this one seemed legit._

</comment></comment><comment><USER24> Good on these Stuy kids!

The media will print anything, and they deserve to be messed with every now and then.

All in all, it was a really funny prank!

<comment><USER25> I don't understand why anybody would be blaming them. This is 100% on the author of the story and NY Mag. They published the story, it's their responsibility to fact-check it. I for one think it's hilarious that they got this story run at all, although I can understand the stress of being at the center of a hoax and all of the attention it brings.

</comment></comment><comment><USER26> People enjoy pretending _hard journalism_ is the rule rather than the exception. Having the internet & cable news destroy it is just another part of the fantasy TV shows like The Newsroom enjoy telling.

</comment><comment><USER27> Is there anything in place to ensure accountably of news releases in news organizations? I'm pretty sure that if enough of this happens; the reputation of the news organization will suffer.

<comment><USER28> > Is there anything in place to ensure accountably of news releases in news > organizations?

Who do you want ensuring accountability? A government? Another private entity?

<comment><USER27> Say I'm a news Agency with multiple broadcast mechanisms and I continually post incorrect information that in turn ends up destroying private citizens lives; Is there any recourse that private citizens can take against the organization?

For example, can you take legal action for defamation of character?

</comment></comment></comment><comment><USER29> There are no journalists anymore. Just a bunch of social media junkies rushing to be "first".

<comment><USER30> The average news consumer expects bite sized, outrageous chunks of information to wash down with lunch – it's a two way stream of degradation.

</comment></comment><comment><USER31> Didn't we just have the bogus story of the kid who made $300k on penny stocks?

</comment><comment><USER32> How much do PR firms cost? and how well off do you have to be to afford one?

<comment><USER33> The fact the name of the PR firm is mentioned in the article probably means it's (at least in part) PR for the PR firm as much as anything.

</comment></comment><comment><USER34> And the real winners here? The media.

</comment><comment><USER35> This feels slightly "Black Mirror" but, to me, it's inevitable. It's a charismatic story. It's somewhat of an advertorial, not just for a person, but for an idea: that someone at Stuyvesant (which is full of high-talent, mostly middle-class, students because the truly wealthy go to schools like Dalton that deliver Harvard if you show up for class, and Stanford as a consolation prize) can make $72 million on his wits alone. I would want to believe it, were I not knowledgeable enough on finance to smell the bullshit.

The current corporate world is built on lies, repeated often enough until they become truth (reputation, prestige, credibility). There are facts, and occasionally those hit back against an aggressive lie; but most of what drives economic life (especially at the individual level) is brand and reputation and social proof. How many corporate executives got there without lying about their CVs, polishing their career histories and accomplishments, and generally manicuring their reputations using a number of tools including deceit? Probably zero. Occasionally, some corporate executive gets caught in a lie and is scapegoated, usually because he was stupid and lied about something easily refutable like where he went to college... even though they all do it.

The sad thing here is that we're talking about a teenager who (a) stood to gain next to nothing and (b) probably didn't intend for any of this to happen at all. The disgusting thing is that, if he takes a fall for this, then he'll have fallen not because of any deception but because of social class. A middle-class kid makes the middle-class mistake of (possibly) lying about something that can be refuted, and gets burned. An upper-class kid gets a $5 million seed round on parental connections and Stanford Welfare (a more pernicious lie, because it looks like merit) and gets away with it, being allowed to play CEO for _years_ until "adult supervision" is brought in to administer a company that he _still_ owns.

<comment><USER36> Whoa, slow down there. Your post is littered with assumptions.

 _Probably zero_ How do you know this, exactly?

 _even though they all do it_ Again, how do you know this?

 _An upper-class kid gets a $5 million seed round on parental connections and Stanford Welfare (a more pernicious lie, because it looks like merit) and gets away with it_ Simply making up a hypothetical scenario does not support whatever point you are attempting to make here. Is there a specific example you are referring to? And besides, what exactly is this hypothetical upper- class kid "getting away with"?

</comment></comment><comment><USER37> wait.. is this story a lie too?

</comment><comment><USER38> _sigh_

</comment><comment><USER39> This sounds suspiciously like the plot of a Farrelly brothers movie.

</comment><comment><USER40> Yes because ordering freshly squeeze apple juice at a place of eatery denotes baller status..

the author is crazy

<comment><USER41> Well he's under 21 and a Muslim, so can't really drop $10K on the goose.

</comment></comment><comment><USER42> He pulled numbers out of thin air to artificially inflate returns and paint a rosy picture of his personal economy? I'm sure the US Federal Reserve is already actively recruiting him.</comment>


Show HN: Dropbox Apps can be cool - BoxySign.com - <AUTHOR> http://boxysign.com

<comment><USER3> Wait what?

You tell me of this service. You promote it here.

You tell me of its features.

You encourage me to "connect" it to my Dropbox account and thus authorize your app to my dropbox account.

ONLY AFTER I CONNECT YOUR APP TO MY DROPBOX ACCOUNT DO YOU REVEAL A SIGN UP FEE.

I think your methods are misleading and frustrating at best. And moving on to unethical and possibly more.

You should explain the fee prior to "connecting" to anyone's dropbox account.

Is there any reason I shouldn't complain about your practices to dropbox?

<comment><AUTHOR> I apologize for it being misleading and frustrating. Thank you for your comments.

I'm updating the home page to more clearly reflect the pricing now.

I'll welcome any other thoughts you have to make it clearer as well. I'm considering plans to have the payment form and Connect to Dropbox button as one sign up form.

(I should not that connecting via Dropbox is NOT giving me access to your entire Dropbox. It only grants me access to your Apps/Boxysign folder - <USER2> you can simply delete from your machine to disconnect from my app. A lot like Twitter and Facebook Connect.)

<comment><AUTHOR> Ok, I have this as a start ready to go - <http://dl.dropbox.com/u/190299/new- home-page.png>

I'm just waiting for <https://registry.npmjs.org/> to come back online so I can push to heroku.

</comment></comment></comment><comment><AUTHOR> I like the idea of Dropbox Apps. A user can consume your web service without even leaving Dropbox (almost). To demonstrate the concept I built <http://BoxySign.com>.

\- Drop an unsigned document in your Dropbox BoxySign folder, and you receive a link.

\- Share that link to receive a signature.

\- The signed copy is then delivered directly to your Dropbox BoxySign folder on your computer.

You can do all this directly from your Dropbox folder, but there is also a web interface which the video demonstrates - <http://boxysign.com/video>

Additionally (depending on how you architect it), it saves big on storage costs. You can store pieces of data in hidden files on the user's Dropbox.

I think this has neat potential for many types of little Dropbox Apps. Thoughts?

</comment><comment><USER4> Certainly a cool app. It's a bit worrying you can store hidden files in someone's Dropbox folder though.

<comment><AUTHOR> Yes a bit. It's isolated to your app folder though. Additionally, Dropbox reviews your app and approves it or not.

</comment></comment><comment><USER5> You should show the video more prominently. cool idea.

<comment><AUTHOR> Thanks. I think you're right.</comment></comment>


Using Slack as a Personal Knowledge Hub – Preslav Rachev - <AUTHOR> https://www.linkedin.com/pulse/using-slack-personal-knowledge-hub-preslav-rachev?trk=pulse_spock-articles <comment><USER2> "knowledge hub", in this case, apparently means a collection of logs on various topics.

1) create your own team

2) create lots of channels (don't screw this up)

3) put thoughts into the channels as you have them and use integration to automatically pull other stuff into the channels

4) review everything you've logged in Slack

5) put the condensed thoughts into Evernote</comment>


Ask HN: Mental Warm Ups to Start the Day - <AUTHOR> Usually when you exercise, you start off stretching and doing warmups before your actual workout.  What do you guys do to warm up your mind at the start of a workday or coding session? <comment><USER2> Meditation is the best thing for this. I also like to use a heart monitor to measure my HRV when meditating - it gives me an idea of my physical & mental performance today compared to my baseline.

A physical workout can also be good. Lifting weights (consistently!) makes a difference long term in my mental performance, but the biggest difference for me is going for a run. The runner's high is real, once you get to a place where you're comfortable with the distance you're doing.

There's scientific research that keeping a journal has a positive effect on your mood. Especially if you explicitly try to write about gratitude.

</comment><comment><USER3> I do as many pushups as I can in one go, and solve my speedcube a few times. Gets the blood flowing and the brain working.

</comment><comment><USER4> Work out(physically).

</comment><comment><USER5> This is important to everyone.</comment>


Ask HN: How can a site like GitHub make money when it pours data? - <AUTHOR>

Gargantuan quantities of data must be stored and pour out of a site like Github.
How can they have done this economically whilst bootstrapping?
I find it hard to see how companies can make money when they have to pay for data traffic and storage. <comment><USER2> They have paid plans which are quite pricey compared to the cost of data.

A repository isn't likely to cross the 1 gb threshold easily. They also email you if it does.

According to A16Z's podcast or something somewhere they have been profitable since inception so it was not much of a problem.

Keep in mind traffic and storage are cheap, and cheaper still in bulker quantities. 1 TB of data is about $30 a month on S3. If each repository averaged 50 mb one user's $7 a month subscription could support nearly 5,000 repos. 50mb is also quite large for a repository btw.

If each was about 5mb that's nearly 50,000 repositories that could be supported cost wise with their cheapest $7 micro plan.

<comment><USER3> Which points out that when your use case for Github is as offsite backup and/or a central repository, self hosting on Amazon is a potentially cost effective alternative.

</comment></comment><comment><USER4> GitHub Enterprise + $100M from VCs goes a long way.

[https://enterprise.github.com/home](https://enterprise.github.com/home)

[https://www.crunchbase.com/organization/github/funding- round...](https://www.crunchbase.com/organization/github/funding-rounds)

<comment><AUTHOR> But they started with no external funding.

</comment></comment><comment><USER5> When Github first started, it became immediately profitable because they had paid plans from the get go and they weren't dealing with that much data at first. They didn't have pull requests or issues or anything like that at the start, it was just a push/pull remote server with a web viewer.

After that, they grew enough to keep up with growth in the company and features until they got the Andreseen investment and threw growth into overdrive.

</comment><comment><USER6> [https://github.com/pricing](https://github.com/pricing)

<comment><USER7> yep i have a micro plan currently.

I can almost guarantee that I am using less than 5Mb throughout my Github, so I don't doubt they are making money.

</comment></comment><comment><USER8> Github data is not "gargantuan". They have 9 million users. They put limit to repository size to 1GB. My estimate would be that most repo don't even get to 10% of this. Plus this is mostly source code and text files that compresses very nicely. Given this information, I would estimate that Github data can fit in under ~5000 2TB hard drives. Cost of hosting data would be around $5 million per year. I bet their employee cost is much larger than this (my estimate is $50M).

</comment><comment><USER9> But they mostly store text, compared to video streaming or imgur that is quite small amount of data per user I guess

</comment><comment><USER10> If you can do some minor work, then you can make money on the internet with sismo money boom. Do a Quick Google search if you haven't heard about it yet.</comment>


Legacy of Steve Jobs - people who matter - And it's not Tim Cook - <AUTHOR> http://microreviews.org/legacy-of-steve-jobs-people-who-matter-and-its-not-tim-cook/ <comment><USER2> Bob Mansfield is missing</comment>


Automatic Docker Service Announcement with Registrator - <AUTHOR> http://progrium.com/blog/2014/09/10/automatic-docker-service-announcement-with-registrator/

<comment><USER2> I ran across a similar thing a few months ago named docker-gen[1]. It listens to the docker event stream in the same way Registrator does, but instead of talking to an external service it regenerates a config file from a template and then optionally runs a command. The canonical example is regenerating a host-wide nginx proxy config when containers start and die, and then restarting nginx.

Just goes to show how versatile docker can be by exposing that event stream to arbitrary processes.

[1]: [https://github.com/jwilder/docker- gen](https://github.com/jwilder/docker-gen)

</comment><comment><USER3> Brilliant. progrium is really good at identifying the single purpose components out of that ecosystem.

<comment><USER4> Thank you!

<comment><USER5> No thank you.

Truly outstanding suite of tools for integrating Consul with Docker.</comment></comment></comment>