Hundreds of Bitcoin Wannabes Show Hallmarks of Fraud - <AUTHOR> https://www.wsj.com/articles/buyer-beware-hundreds-of-bitcoin-wannabes-show-hallmarks-of-fraud-1526573115 <comment><USER3> \- The space is littered with fraud.

\- Wallets have bugs that lose hundreds of millions of dollars

\- Energy consumption forecasted to hit 0.5% of the world's electricity consumption by end of 2018

\- Promoters yelling "decentralize everything" while exchanges, development teams and mining are all highly centralized

\- No legal use case found to date

Blockchain is a parasite on society.

<comment><USER4> Despite all of this, I don't think we've hit peak blockchain fetishism yet. There's still a lot of room to make solved problems worse with a blockchain.

<comment><USER5> There is a lot of social capital to acquire by making solved problems worse (e.g. "reddit redesign") so blockchain will probably see much wider adoption soon.

On the bright side, there will be a lot of social capital to acquire by "fixing blockchain" with revolutionary "centralised databases" afterwards :)

<comment><USER6> Yes I think some of the coins caught onto this with DPOS. Have a couple dozen block producers who will run fast servers and can be voted out. Then don't worry about total decentralization and your performance is 100x the competition.

</comment></comment></comment></comment><comment><USER7> In the end, the "reputable" ICO-backed companies will mostly (perhaps entirely) turn out to be no better. Enron was a real company that did real things. During the time it operated, hundreds of aspiring future Enrons were born in the penny stocks, took down a few thousand in profits, and died. Skeptics of Enron could have pointed to those overt frauds and said "sure, the space is littered with frauds, but look at Enron over here; there are real companies too". In the end, it didn't matter; in fact: Enron was the more disastrous of the two threats.

<comment><USER8> Interesting comparison, but I think it breaks fairly quickly. If Enron were in fact like Bitcoin, they could not have committed institutionalized and systemic accounting fraud. The books are fully open and transparent. It's fair if you think that Bitcoin will fail at becoming anything significant in the history of the Internet, but calling it a fraud seems unjustified.

<comment><USER9> I read the person you're replying to as talking about cryptocurrency backed companies, not cryptocurrencies like bitcoin in general.

</comment></comment></comment><comment><USER10> Early stage is just not very well suited for dispersed ownership. The investors need to do a lot of work to dig up and verify the information and this only makes sense if they have big stakes in the company. This is a general problem with crowdfunding ([https://medium.com/@zby/proof-of- work-8d8265def194](https://medium.com/@zby/proof-of-work-8d8265def194)). Then there are specific problems with ICOs - which have no legal meaning and give no guarantee to the investors at all.

</comment><comment><USER11> There are many replies in this thread that seem to be forms of 'yeah but there are issues with other stuff too!'

Not a convincing rebuttal.

</comment><comment><USER12> It's the gold rush that killed crypto-currencies. They are dead, but their gold rush enthusiasts are going to hang on for a decade hoping to recoup their pride.

<comment><USER13> Last gold rush for crypto was in 2013 so ~5 years.

<comment><USER14> There was definitely a substantial influx of new (often clueless) investors in the past 12-18 months.

<comment><USER15> When Bitcoin hit around 14K my mom had heard all about the rise on MSNBC/CNN. So she asked me how to buy bitcoin. She needed to replace her radiator. She had about 500 and needed 1200 so she thought it was a reasonable gamble. So I had her send the money to me and I told her I would invest for her.

I did not..

But I did inform her that when it crashed I just stuck her money in my savings account. And nothing was lost. I was willing to eat the loss if it had gone up when she wanted to cash out.

And I did buy her a new radiator. I don't have to give a shit about mothers day for the next decade.

<comment><USER16> Not sure what your point is here. Predicting price movements is not an easy task, even for experienced traders. If you bought at 14k and sold at 20k, you would've made money. Nobody really knows the direction the market is going, otherwise we would all be rich now. Some trades make better profits than others based on experience, but for the most of us it's just pure luck.

</comment></comment></comment></comment></comment><comment><USER17> One of the most under-appreciated problems regarding the "soundness" of blockchain tokens is that they are trivial to create and so everyone does so. A given blockchain token is only nominally distinct from a potential fork of itself that literally creates blockchain money out of thin air.

The bitcoin-cash fork is a perfect example of how blockchain tokens can be trivially summoned out of the void with identical properties to the parent token which spontaneously "creates value" when the sum total value of the fork and the parent exceeds the value of the parent pre-fork. What this tells us is that blockchain markets are very inefficient and the vast majority of money in the system has near zero understanding of the intrinsic properties that underpin the token's ability to even exist as a thing that can be traded on an exchange. Once the market is saturated with an understanding of how these tokens are actually infinitely abundant and not at all scarce as we know it, we are going to see a catastrophic correction.

</comment><comment><USER18> It basically boils down to the world is full of suckers.

I'd like to find alternatives to Bitcoin and Ethereum that are actually innovating. For most coins, forking is the extent of their technology.

<comment><USER19> This! I wish other projects tried harder.

</comment></comment><comment><USER20> Article is non-paywalled for once.

Exhaustive analysis of ICO white papers shows about 1/4 contain major red flags. The bar the WSJ set is pretty low (as in easy to pass without raising flags). Any white paper that didn't plagiarize long passages, re-use others' headshots or fail to name principals, or fail mention risk basically passes.

It's also worth noting that the multi-billion dollar figure cited for total funds raised through ICOs is about as reliable as the ICOs themselves, since it's based on self-reported figures from ICOs that frequently appear out of thin air with multi-million claims of funds raised so far.

<comment><USER21> And that's not even considering the many white papers that are original work, but completely wrong. Many of them propose fundamentally incorrect and exploitable systems. But of course none of the papers are peer reviewed, so if the idea sounds cool to some armchair tech nerds on reddit, off to the races you go!

Want to see something truly scary? Pick a coin, any coin, that has had a major vulnerability discovered by researchers in a peer-reviewed publication. Then go onto that coin's subreddit and watch in amazement as a bunch of security illiterate redditors bend over backwards to invalidate the paper's claims with conspiratorial nonsense and cries of spreading FUD.

Blind loyalty is bad enough in meatspace. In the cryptocurrency realm, it's simply dangerous.

The crazy thing is that any one of these coins could have a double-spending vulnerability, which by its very nature would likely be undiscoverable. There could very well be some clever hacker somewhere exploiting double spends and slowly bleeding money out of exchanges.

<comment><USER22> By the nature of the technology, there's no double spending on a blockchain.

Because every node has the same database. All and every one of the miners maintain a copy of what's essentially the same DB, the same distributed ledger.

But there are plenty of stolen keys and seeds for wallets and if someone gets your wallet, all your money will be moved to someone else's. This, sadly, happens frequently.

<comment><USER21> 4 days later, #1 front page of HN: [https://www.ccn.com/bitcoin-gold-hit-by- double-spend-attack-...](https://www.ccn.com/bitcoin-gold-hit-by-double-spend- attack-exchanges-lose-millions/)

<comment><USER22> Yes, I was flabbergasted by this one.

</comment></comment></comment></comment></comment><comment><USER23> FYI: Beware Investors! At the Bits & Blocks Press Bookself I've collected books/booklets to warn about blatant crypto frauds and fudge. See Best of Bitcoin Maximalist - Scammers, Morons, Clowns, Shills & BagHODLers - <USER2> The New New Crypto Ponzi Economics [1] or Crypto Facts - Decentralize Payments - Efficient, Low Cost, Fair, Clean - True or False? [2] or Get Rich Quick "Business Blockchain" Bible - The Secrets of Free Easy Money [3]

[1]: [https://bitsblocks.github.io/bitcoin- maximalist](https://bitsblocks.github.io/bitcoin-maximalist) [2]: [https://bitsblocks.github.io/crypto- facts](https://bitsblocks.github.io/crypto-facts) [3]: [https://bitsblocks.github.io/get-rich-quick- bible](https://bitsblocks.github.io/get-rich-quick-bible)

</comment><comment><USER24> Can someone smarter than me try to write a formal proof that you can only have 2 of the following: distributed, fast, secure?

<comment><USER25> There's decades of academic work in the form of "impossibility" results showing that handling byzantine fault tolerance in distributed consensus is alot more computationally expensive than omission/crash fault tolerance. The following (FLP impossibility result) being one of the most famous ones:

[https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf](https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf)

<comment><USER26> Yes, that's the root of the problem. A deceptively simple sounding result.

</comment></comment></comment><comment><USER27> To be fair, How many startups listed on Angel list ar in fact frauds?

No, not just illegal stuff like intershipsp for no pay but more major stuff like outright fabrication of founder bios, etc.

<comment><USER28> Legitimate frauds? Or "Fake it till you make it" type frauds who think their idea is legit and just inflate some numbers and their capabilities to get the money they need to prove it?

What matters more, it seems to me, is that both the cryptocurrency and tech startup world are filled with bad investments which are going to make the next economic bust exponentially worse.

<comment><USER29> > think their idea is legit and just inflate some numbers and their > capabilities to get the money they need to prove it

how is that not "legitimate fraud?"

<comment><USER30> Because there's at least an actual idea, and an intent to execute on it. A lot of the fraudulent ICOs, on the other hand, don't even have that -- their real business plan is "take the money and run".

</comment></comment></comment></comment><comment><USER31> The anti blockchain sentiment on HN is something like the anti-Trump hysteria in the period leading up to/immediately after the election. Tons of grandiose "it's SO horrible" posts, tons of downvotes.

I predict it will shake out about the same way also.

Blockchain and bitcoin aren't going anywhere in the near future. Some people have, and will continue to have, a use (for whatever reason). The "I hates it precious!" screeds won't change this. In the end it will turn out it's not the savior the proponents claimed it would be nor the demon incarnate the detractors claimed either but just another instrument of the powers that be.

Why folks can't take less emotional views on these two phenomenon is beyond me. It really is. You don't like Bitcoin? Fine, ignore it. I simply don't get the mouth frothing hatred.

<comment><USER32> I think the negative sentiment is largely because of how much the bitcoin evangelists promise. There's a lot of hype and lofty goals tossed around but cryptocurrencies have been around for nearly a decade (If I remember bitcoin started early 2009) and they mostly seem to be used for fraud and speculation now. I'm not saying that the technology is fundamentally useless but the mismatch between delivery and retoric has made me sour on it.

A good more concrete example of this is the DAO from Ethereum. A lot of verbage was wasted beforehand saying how it was going change everything about corporations and investment but as far as I can tell little has changed.

<comment><USER33> You mean mostly media talks about fraudulent use cases. Or otherwise, how do you know mostly they are used for fraud?

<comment><USER32> The reason I think it's mostly speculation and fraud is that the only people I know who have crypto currency in real life have it as a speculative investment and that combined with the fact that it doesn't work well as a payment processor (historically it's been slow and expensive and even mainstream companies that used to take it seem to have often discontinued support). I actually don't think they are mostly fraud, I think they are mostly speculation and fraud is just the outcome ofof a speculative market with little regulation. I would be happy to hear about more use cases but most I've heard about have either been small toy use cases or "in the works".</comment></comment></comment></comment>


OPP: Other People's Problems - <AUTHOR> http://www.elidedbranches.com/2019/05/opp-other-peoples-problems.html <comment><USER2> I've not heard of Camille Fournier until now, but glad I found her, and this article is quite timely for me.

I'm dealing with several issues right now in trying to "take ownership" of things that aren't mine.

With our EDI system, I only get called in when the "real" owners are too busy, but when I do, I find their entire architecture is a foot gun that makes development way harder and more error prone than it should be.

With our (customer facing) admin system, the entire thing has a UX that I think easily triples the work/time/number of clicks to do anything, and just begs for many routine things that bog users down to be automated.

The problem I _do_ own, the mobile apps, is greatly hampered by the back-end that drives all of the other problems that I don't own, so all problems including mine could benefit greatly from working on _that_ problem as well.

But when I talk to other engineers, they (a) don't see the problems being as impactful as I do, (b) don't believe that I can solve the problems in the first place, and (c) ultimately don't want to put any time or effort into learning about the problems or my proposed solutions, even though putting in a little effort now could save us from regularly having to stay late and fix fire drills like we do now.

I've even gone to the length of building prototypes to demonstrate both (a) and (b), and in some cases they relent somewhat on (a), but continue to talk as though solutions aren't possible... even as they're looking at a live, running prototype.

I'm at a point where I either need to let it all go and just "do my job" (even though I feel the quality of work I can do on my own stuff is being limited by OPP), OR mobilize the internal end users by showing them my prototypes and getting them on my side (to pressure the engineers) which somehow feels a bit sleazy, OR quit and find another job.

<comment><USER3> This is a tough situation to be in. I've been there myself and it's not fun. If I were in this same situation again, I would ask myself a few questions:

* __Who is my customer in this situation? __It could be your boss, your boss 's boss, an important stakeholder, end users, etc. Make sure you have clarity on your customer so you can focus on their problems first. If you have find you have multiple "important" customers, then prioritize the most important first. This is likely the person who has the most influence on your immediate success.

* __Are any of the problems you identified "your" customer's problem? __If so great, time to get to work. If not, consider reprioritizing. You should reach out to your customer and get their feedback on priority. Be careful not to lose sight of the big picture.

* __How do I know the other engineers aren 't right? __If you work with smart people who know how to get things done, then you should take their feedback seriously. Maybe they 're right and you're wrong. Always validate your assumptions.

* __What would happen if I just solved the problem myself? __If you can then do, seriously. Submit a PR with your solution or build a proof of concept. Default to taking action. Don 't wait for someone else's approval to get shit done.

* __How can I measure and bring transparency to these problems? __You 'd be surprised how powerful emails with data are. If you can measure the problem (e.g., MTTR, rate of failure, response time) and more importantly the impact on customers, then you may be able to indirectly "motivate" others to solve their own problems. Send regular updates to managers, Directors, VPs, Execs, anyone important who will listen.

* __At what point should I just let this go? __Time box your efforts and once that time period has been reached, just let it go. This very well could be a situation where the "problem" is not the top priority. You may have to let this one burn so you can focus on other things.

<comment><USER4> These are good points but

> Don't wait for someone else's approval to get shit done

This may be a great idea, but mishandled it can truly piss people off. See my other reply to this. Turn the diplomacy up to 11 if you plan to try it.

<comment><USER2> Both you and jonathanfoster have raised some great points here.

In a way, the working prototypes I've built (many of which could be production ready with little extra work) have been my "just get shit done" approach, both to prove my solutions to myself, AND to help make the argument to the rest of the team.

What's tough is that I'm not getting any hard no's on anything... after I build and show working prototypes of my ideas (many of which could be deployed to production with little extra work) I usually get a pat on the back followed by a "we'll discuss getting this out there when we have more time to learn it", and then it goes nowhere.

Next step would be deploying the stuff to production, but I have a good hunch I'd be the only one using the stuff, which wouldn't actually fix the problem from the customers' perspective if only half the system works as they'd expect (especially if new features continue to be written by the other engineers using old systems).

Ego might be part of this, though, so I think being extra diplomatic and trying to frame these ideas as "other people's ideas" might get me further.

Barring that, I might go straight to the customers (starting with our internal users and maybe broadening from there) to make sure my ideas are actually worthwhile to them.

If I can get a hard no from either my co-workers or the customers, I'd be more willing to either back off or find another place where I fit better. But as of right now, I see our customers asking for things we can't deliver, and I have a drive to fix it, but I don't think there's a way forward without my co- workers' and superiors' buy in.

Thanks for all the ideas!

</comment></comment></comment></comment><comment><USER5> I think there are different ways of looking at this. IMO a CTO who is concerned with "fixing problems" and is working on actually fixing problems is just wrong.

The CTO is in a unique position where they have the power and I think responsibility to be looking at the "meta-problems". Have I done enough to enable others to fix these problems? As soon as you, as CTO, start getting into the muck to actually try to fix the problem you've gone too far or you haven't surrounded yourself with the right people.

<comment><USER6> Hey there

Why throwaway for this comment?

<comment><USER7> Perhaps working as a CTO and didn't want to be recognized as such

</comment></comment></comment><comment><USER8> I think it's also important to approach it with some humility about whether you fully understand the contours of the problem. Also I think you need to prove that you can and will do something about one problem before you start talking about all the other things you want to fix. Otherwise you have no credibility and others could reasonably think you're just complaining.

</comment><comment><USER9> _> > If you're reading this looking for advice, you're probably a go-getter. You consider yourself a responsible person, who cares deeply about doing things right._

In many environments, the tough part about resolving to pick your battles and leave problems to other people is maintaining the image of a responsible go- getter in order to protect your career. "Tending your own garden" translated to the language of your manager may be "limited vision", "not a team player", "does not display leadership qualities" and "not engaged". Regardless of whether or not you or anyone on your team actually has broad influence on anything, you have to _act_ like you _think_ you do to appear engaged.

<comment><USER10> There's a trade-off in perception, because there are two approaches that different kinds of leaders like. One approach that some favor is the "tell it like it is" approach, where anything is up for criticism by anyone. Another is the "if you're not part of the solution you're part of the problem" approach, where if you bring the criticism, you'd better bring the solution and be willing to see it through.

These approaches both have strengths and vulnerabilities. The strength of the "tell it like it is" approach is that big problems can't fly under the radar, and ideas and criticisms are shared across organizational and disciplinary boundaries. Its vulnerability is that clever people can curry favor by harping on known problems outside their responsibility to fix. Sometimes if a problem is hard enough, the people fixing it don't look like they're doing anything, and it may appear that they don't even appreciate the problem, because they're likely to respond defensively to having the problem announced over and over again. A hard problem is a goose that lays golden eggs for a loud critic, who looks more and more "right" the more they harp on it, and the people who are actually working on a solution lose credibility, without which they can't devote resources to the solution. So a boss who favors the "tell it like it is" approach must be careful not to create an atmosphere where words count more than action.

The strength of the "if you're part of the solution" approach is that it encourages an enterprising spirit and rewards go-getters instead of talkers. Its vulnerability is that big problems may not get talked about at all. Someone might be the only person who sees a problem but assume everyone knows about it but aren't talking because they don't have a solution. Or someone might see a problem and not want to mention it because they don't want to invite pressure on themselves to solve it.

In other words, these approaches are both pathological if they become consistent enough to define the atmosphere in a company. Good leaders have to constantly rebalance them to fit the occasion.

</comment></comment><comment><USER3> Camille Fournier literally wrote the book on Engineering Management so when she speaks, I always listen [1]. Every technology leader will reach a point where there are just too many problems to solve and OPP is a great mental model for prioritization.

OPP is also a forcing function for leadership. It forces true leaders to step up and make the hard choices.

If you're faced with OPP, here are a few things I found useful in my career.

## Do Important Work for Important People

The best way to be successful in any organization is to do important work for important people. Important work for unimportant people will get you no where. Same is true for unimportant work for important people.

Take a look at your OPP and ask yourself:

* Is this work for someone important?

* Is this work important to that person?

Both answers should be yes, otherwise it's just OPP.

## Let Fires Burn

Once you decide the work is OPP, then you need the courage to say no. You must let that fire continue to burn without it distracting you. Masters of Scale has a good episode on this topic [2]. Easier said than done of course. I found Stoic practices to be very helpful here [3].

## Customer Obsession & Ownership

OPP should always be evaluated through the lens of the customer. Bottom line, the customer is always the most important person and they trump all. True leaders are obsessed about providing a better customer experience and they're willing to pay the price in order to do so.

If you have OPP that's important work for someone important, but it's not important to the customer, then you may just have to let that one burn too. And once you make that call, you have to own it. Always take responsibility for the decision and defend it on the customer's behalf.

I've found Amazon's leadership principles to be invaluable when making these type of tough decisions [4]. It's no coincidence that Customer Obsession and Ownership are #1 and #2.

[1] [http://amzn.to/2niWhO9](http://amzn.to/2niWhO9) (Camille's link not mine)

[2] [https://mastersofscale.com/selina-tobaccowala-let-fires- burn...](https://mastersofscale.com/selina-tobaccowala-let-fires-burn/)

[3] [https://www.amazon.com/Obstacle-Way-Timeless-Turning- Triumph...](https://www.amazon.com/Obstacle-Way-Timeless-Turning- Triumph/dp/1591846358)

[4] [https://www.amazon.jobs/en/principles](https://www.amazon.jobs/en/principles)

<comment><USER4> I need to read the articles and the comments properly, but this

> It forces true leaders to step up and make the hard choices.

reminded me of a discussion on the radio about the mess of a situation detroit car manufacturing had got into. Someone summarised, saying that the bosses spent decades making easy choices instead of the right choices.

</comment></comment><comment><USER11> Damn. These kind of problems are the exact ones which drive me mad almost every week. Welp, looks like it's time for a new job (not that I haven't considered that already, but now some random article on the internet backed me up :P).

<comment><USER12> You're down with OPP?

<comment><USER13> For those that don't know it's a hip-hop reference to 'other people's p...y' or 'other people's p....s' (OPP in both cases). In other words, "Are you fine with pursuing physical relations with people in committed relationships?"

The related song is by group 'Naughty by Nature'.

<comment><USER11> Ha, didn't get that right away, thank you ;)

</comment></comment></comment></comment><comment><USER14> The last P? Well, that's not that simple.

</comment><comment><USER15> Not to be confused with SEP: [https://hitchhikers.fandom.com/wiki/Somebody_Else%27s_Proble...](https://hitchhikers.fandom.com/wiki/Somebody_Else%27s_Problem_field)

</comment><comment><USER16> [https://xkcd.com/1134/](https://xkcd.com/1134/)</comment>


The Pirate Bay is now hosted in North Korea - <AUTHOR> https://thepiratebay.se/blog/229 <comment><USER2> So.

A traceroute to thepiratebay.se is kind of amusing.

From my home (Sweden), the packets seem to go to Frankfurt, then New York, take a link via an ip which reverses to intelsatone.net to a cambodgian ip (500ms latency right here), then reach the ip 22.214.171.124, assigned to Star Joint Venture Co Ltd. Who seem to be a legitimate North Korea internet provider (or, well, as legitimate as it gets, coming from North Korea). The rest of the traceroute doesn't ping back (edit: 6 hops, which could stay in NK, or lead you back anywhere in the world).

If it's a joke, it's a very elaborate one.

<comment><USER3> I wouldn't be surprised if some of the ICMP responses are forged - It seems disadvantageous for the site to have such a long path, since each hop has the potential for attack.

<http://www.thoughtcrime.org/software/fakeroute/> can be seen as previous work that it is practical to do something like this.

<comment><USER4> No it's not forged as can be seen from the AS path in the global routing table.

<comment><USER5> Yes, it is forged, as explained by tuomasb.

Furthermore, a round trip time less than about 60 ms between Europe and North Korea is impossible, assuming the data is traveling at the speed of light. And we measured much less than that.

</comment></comment></comment></comment><comment><USER6> This makes me upset. Its upsetting because while yes, information freedom is a very important topic and needs to happen, but it does not trump human freedom. I'm not sure if this is a straw man or not - but how can they justify working with an entity that has such a horrid human rights record?

<comment><USER7> Yes, definitely, we should not stand by while companies host their content in countries known to use torture to extract information, hold prisoners with out trial, hold prisoners with out charge, allow their leaders to execute citizens summarily, execute minors and the mentally retarded, incarcerate people at a rate higher than any other country on earth merely in order to serve as slave labour for the state and state sponsored corporations.

Even worse is many of the citizens of this 'republic' have been brainwashed by a compulsory education system that they actually live in a democratic republic.

<comment><USER8> If you think you can legitimately compare the living conditions, political environment, and human suffering in North Korea that with the USA, you might have also brainwashed yourself (it happens when you take everything you have for granted).

In NK you'd be lucky to get 1 meal a day. Over here poor people are morbidly fat.

And at the end of the day, you're not a prisoner, if you hate it here so much, you can leave this country any day you want to. But can't say the same for people in NK.

<comment><USER7> How is it that you think I was talking about a country other than NK? Is there a country other than NK with such a piss poor human rights record?

Generally when someone murders one person we regard them as a horrid individual we don't say their not that bad because they only killed 1 instead of 10 like that Ted Bundy guy.

I didn't say I was living in a police state, the country I live in hasn't executed anyone in 40 years, and we certainly didn't think it was a big deal to destroy our nuclear weapons arsenal, stop stockpiling nuclear weapons, or ban landmines. I simply said we should stop hosting websites in countries with numerous years of history of horrid human rights abuse. Imagine living in a country where as recently at 1960 that certain classes of people were forbidden from eating lunch with other classes of people.

I find your definition of police state interesting, would you say that NK would stop being a police state with different emigration policies?

<comment><USER9> And what country are you from?

Edit: Oh I get it, if you you don't say what country you can pretend they are perfect. Brilliant.

<comment><USER10> > the country I live in hasn't executed anyone in 40 years, and we certainly > didn't think it was a big deal to destroy our nuclear weapons arsenal, stop > stockpiling nuclear weapons, or ban landmines.

From all countries with nuclear weapon 40 years without death penalty match only UK.

<comment><USER11> Except the UK is nowhere near destroying their arsenal; in fact, they are trying to figure out how to replace their current missile boats. The country described by that line does not actually exist.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> The Pirate Bay is not concerned with your judgment. The Pirate Bay is concerned with making socio-political statements and providing access to their site, seemingly in that order. Discomfort with the situation is exactly what they are going for.

It should make you uneasy that they have to go to North Korea to keep the site online. While they are not martyrs for an easy to grasp cause, and their definition of 'free speech' may fly in the face of yours, they are doing their absolute best to keep alive what they think is important. This has recently resulted in much irony. Irony that I'm sure they're proud of.

You should hate this. That's the point. This wasn't done so people could keep downloading movies illegally, this was done to make a statement, to get you to think. So please, ice your knee, and think.

<comment><USER13> It's a joke. THAT'S the point.

</comment></comment><comment><USER14> fale@machine:~$ tcptraceroute -f 128 -m 128 thepiratebay.se Selected device venet0, address 18.104.22.168, port 40771 for outgoing packets Tracing the path to thepiratebay.se (220.127.116.11) on TCP port 80 (www), 128 hops max 128 thepiratebay.org (188.8.131.52) [open] 51.673 ms 49.002 ms 47.187 ms

That server is in Germany, no way it's possible to have 50ms to NK. Also traditional traceroute has 500ms+ RTT.

They are faking/spoofing the ICMP responses. They are also prepending their route advertisement with corresponding AS paths to further disguise it.

From TeliaSonera looking glass <http://lg.telia.net/>

184.108.40.206/24 *[BGP/170] 02:10:36, MED 0, localpref 150, from 18.104.22.168 AS path: 2914 39138 22351 131279 51040 I

AS39138 is probably the real upstream provider of TBP. They peer with AS51040(TPB network) and TPB router prepends AS22351(Intelsat) and AS131279(North Korean ISP) into it's AS Path before advertising it to AS39138.

<comment><USER15> Yeah, and why is AS39138 in /all/ the AS paths I tried on that site, when there other ways to AS22351

<comment><USER14> Exactly. Here is a collection of route-servers and looking glasses which tell you what path a route from ISP x to IP y will take. <http://www.bgp4.as/looking-glasses> You will see that every single route to 188.8.131.52/24 will travel through AS39138.

</comment></comment></comment><comment><USER16> Well it's nice to know that "one of the most important movements in Sweden for freedom of speech, working against corruption and censorship" wouldn't let concerns over something as insignificant as totalitarianism deter them from "forming a special bond" with a state to distribute warez.

<comment><USER17> They acknowledge the irony in their own post - the only way that they can promote freedom of speech is by forming a partnership with the least free country in the world, _not_ by forming an alliance with the supposedly 'free' countries in Western Europe/North America.

In other words, the Western countries' belief in 'freedom of speech' is starting to seem as shallow as the word 'Democratic' in the 'Democratic People's Republic of Korea'.

<comment><USER18> Piracy is not freedom of speech. It is the copying and syndication of the intellectual works of others. I suspect more HNers would be upset about piracy, if they weren't using SaaS business models which are not as vulnerable to it.

It's really easy to brush off the fact that hundreds of people worked to make that movie you downloaded for free, especially when you never have to look any one of them in the eye.

These days with things like netflix, itunes, and amazon prime, the only excuse one can make for participating in piracy is destitution. I totally understand those who wouldn't have been able to acquire the goods anyway doing so by using the pirate bay and other services. In this sense, they're kind of a decentralized information welfare program. Most countries have this already, and it's called a library. Too bad most libraries haven't caught up with the information age yet. So the piracy services have been filling a very important spot.

I'd strongly urge everyone not to use the pirate bay. North Korea is an oppressive country that would nuke the US in a heartbeat.

It makes me sick that so many people idolize these guys.

Before you rationalize your piracy in response to my post, please sound out the word: "rational" "lies"

<comment><USER19> "Piracy is not freedom of speech."

Of course it is. It's the free exchange of information. If you don't support piracy you don't support absolute free speech.

For the record, I do not support absolute free unrestricted speech as it applies to exchange of (for instance) child pornography. I am happy to admit this.

<comment><USER20> Perhaps, "free speech" is the best term to use here. I think you refer to free culture, and/or free knowledge.

<comment><USER19> Is the transmission of information (1s and 0s) from one person to another not 'speech'? Would it be a free speech issue if a picture or movie was censored?

I don't know. I do know I've been berated as anti freedom of speech when I've explained my ethical issues with (me personally) running darknet nodes, because I'm unwilling to let CP be transmitted over my resources.

</comment></comment></comment></comment></comment></comment><comment><USER21> I expected more from HN than this knee-jerk support for TPB due to its self- promulgated association with "freedom of information." Freedom of information, in the specific context of The Pirate Bay, means denying content authors the freedom to choose channels of distribution for their work. It means repackaging content, giving it away "for free," while raking in advertising revenue. It means diverting money away from the entire chain of content creators (e.g., writers, actors, directors, extras, special effects artists), marketers, distributors, etc., and solely into the pockets of TPB's operators.

It's OK to support freedom of information, and also recognize TPB for what it really is.

<comment><USER22> Why should authors and not "consumers" (man, how I hate this word) choose distribution channels?

<comment><USER21> When the author chooses the distribution channel, the channel is essentially part of the asking price. People are then free to gauge whether or not the asking price is too high, and opt not to buy. Ultimately, the choice of distribution channel becomes determined by both parties, the buyers and sellers, by agreeing on a price.

On the other hand, if the consumers collectively decide to subvert this process and set the price at zero, where does that leave the author? Their message is "I'm going to take your work and pay you nothing for it. Get back to us when you can offer us a better deal than something for nothing."

Which option seems like a better system to you? Technology has made taking content and paying nothing for it a frictionless transaction. Many people here seem to believe that since technology has enabled it, or made it so easy, or made it so difficult to regulate, it therefore must be a natural right that we've had all along, but can only now fully enjoy due to the miracles of modern technology.

If some group on the supply side gains a monopoly, the government in theory steps in and regulates to protect the consumer from price gouging. This notion appeals to us, that when one party in the holds all the cards, they shouldn't use exploit that power to price gouge. Yet when the consumers suddenly hold all the cards, we turn a blind eye to imbalance in power.

I'm not saying that movie studios and record labels should be protected so that they can cling to old distribution methods. But expecting authors, or anyone on the supply side, to offer a price as good as free is not a reasonable expectation.

</comment></comment></comment><comment><USER23> Nuke tests haven't caused us to invade North Korean but if this happens the MPAA could get it done.

<comment><USER24> as strange that may sound, it might become true.

</comment></comment><comment><USER25> If North Korea really is involved in this (and the traceroutes seem to indicate it is), let's turn the irony up a notch and turn the spotlight on a few North-Korea-critical torrents - now served through the very censor-happy country that is getting criticized:

Children of the Secret State - North Korea <https://thepiratebay.se/torrent/6441601/>

Inside - Undercover in North Korea <https://thepiratebay.se/torrent/6136295/>

Comrades and Strangers: Behind the Closed Doors of North Korea <https://thepiratebay.se/torrent/6899842/>

The Vice Guide to North Korea <https://thepiratebay.se/torrent/5378291/>

National Geographic Explorer ~ Inside North Korea <https://thepiratebay.se/torrent/6110419/>

It will be interesting to see how serious their new overlord is about that "freedom of speech thing". My guess is that sooner or later, some of those films will suddenly vanish from the site.

</comment><comment><USER26> This is a pretty interesting bit of gamesmanship. In a single stroke it ties together what is perhaps the greatest threat to global stability today (PRK) and where is perhaps the most ridiculous non-threat to global stability (pirated movies) and sets them as equals.

By doing that, it illustrates just how ridiculous the current copyright situation is.

Nicely played Piratebay, nicely played.

<comment><USER16> Of course the other way of looking at it is that the creators of TPB think that threats to their freedom to profit from placing advertisements around links to copyrighted material trump the threats to the freedom of the North Korean people. A lack of PR perspective so perverse it almost makes the RIAA and MPAA look like the good guys.

</comment></comment><comment><USER27> >by Kim Jung-Bay

Of course it's serious.

edit: I thought it's a joke... but the traceroute trully ends in North Korea with the 18.104.22.168 IP.

So ... maybe it's actualy legit.

I don't know what to trust anymore.

</comment><comment><USER28> Sure, just like the airborne Raspberry Pi drone servers.

[http://arstechnica.com/tech-policy/2012/03/pirate-bay- plans-...](http://arstechnica.com/tech-policy/2012/03/pirate-bay-plans-to- build-aerial-server-drones-with-35-linux-computer/)

<comment><USER13> Did HN just turn stupid? I can only shake my head.

The idea that it would even be feasible for TPB to host in North Korea is ridiculous. The fact that they already have a history of pranks about their hosting should be a big clue.

</comment></comment><comment><USER29> Absolutely uncool. No matter what you think about current IP law - and let's face it, it's a bag of worms - getting into bed with North Korea is inexcusable. On top of everything else, and here "everything" means institutional murder and oppression, it kind of suggests a lack of morality on TPB's part.

</comment><comment><USER30> Sergey Brin, Dennis Rodman, and now the Pirate Bay? This is the weirdest PR campaign ever

<comment><USER31> Eric Schmidt, not Sergey Brin.

</comment></comment><comment><USER32> >"...we have been invited by the leader of the republic of Korea..."

That's the wrong one, though.

Republic of Korea = South

Democratic People's Republic of Korea = North

<comment><USER17> Is it possible that people in the DPRK refer to themselves as the 'Republic of Korea' in an attempt to deny/disqualify the existence of any other 'Republic of Korea'? (ie, 'The one true Republic of Korea' or something like that).

<comment><USER33> Why do communist countries insist on calling themselves "democratic"? If you're such a firm believer in that philosophy, why not have some pride?

<comment><USER34> Communism and democracy are orthogonal. Your question is why _dictators_ represent themselves as democracies, but I don't think that's a very complex question; it's propaganda 101.

<comment><USER35> For a while it was even visible on both sides of the Korean peninsula: a left- wing military dictatorship in the North referred to themselves as the "Democratic People's Republic of Korea", while a right-wing military dictatorship in the South referred to themselves as the "Republic of Korea". Neither country actually operated as a republic, but both seemed to like the term (and did have some superficial trappings of a republic, like legislatures). The South is now an actual Republic, though, since around 1987.

Similar situation for some years in the Taiwan Strait, where the not-a- republic People's Republic of China faced down the also-not-a-republic Republic of China (Taiwan).

<comment><USER32> Right. It's kind of seen as a "given" now that South Korea and Taiwan are the democratic, American-friendly, "good guys" and the DPRK and PRC are the authoritarian, oppressive "bad guys," but back in the '60s through the '80s before the ROK and ROC democratized, the situation was much murkier. Because of the Cold War, those regimes were considered allies solely by virtue of not being Communists, but they were also very unfree. Perhaps the winding down of the Cold War had something to do with their falling out of power and being replaced by democracies.

</comment></comment></comment></comment></comment></comment><comment><USER36> I don't like it. Putting the web's biggest P2P website in the hands of one of the most authoritative dictatorial regime - what could go wrong? Security issues anyone? Seems fishy at best, and if true it is counter productive: it will not reflect well on TPB...

<comment><USER37> Same as with Gerard Depardieu. He/they made choice not to be in one kind of dictatorship so they escaped to other. So it seems like pretty subjective stuff (personally i think it's wrong), but it's their choice so who cares. Why do you care?

Here is what they think:

"This is truly an ironic situation. We have been fighting for a free world, and our opponents are mostly huge corporations from the United States of America, a place where freedom and freedom of speech is said to be held high. At the same time, companies from that country is chasing a competitor from other countries, bribing police and lawmakers, threatening political parties and physically hunting people from our crew. And to our help comes a government famous in our part of the world for locking people up for their thoughts and forbidding access to information."

Is this false?

</comment></comment><comment><USER38> I'm sorry, but is this a joke? Or were they seriously offered server space in North Korea as some sort of political/PR thing?

<comment><USER39> The route goes to NK

traceroute thepiratebay.se output:

      12  INTELSAT-IN.car1.NewYork1.Level3.net (184.108.40.206)  127.129 ms  121.898 ms  121.857 ms
      13  188.8.131.52 (22.214.171.124)  214.763 ms  196.291 ms  210.602 ms
      14  188.8.131.52 (188.8.131.52)  697.258 ms  711.336 ms  693.061 ms
      15  18.104.22.168 (188.8.131.52)  696.368 ms  699.046 ms  702.013 ms

126.96.36.199 Seems to be an actual IP in NK[0]

[0] <http://bgp.he.net/AS131279>

<comment><USER40> I get a German IP, 18.104.22.168 and the tracepath results take the route nowhere near Asia.

<comment><USER41> From a Verizon FiOS connection in Massachusetts, I get:

        $ traceroute thepiratebay.se
        traceroute to thepiratebay.se (18.104.22.168), 64 hops max, 52 byte packets
        ... snip ...
         3  ae0-0.bos-bb-rtr2.verizon-gni.net (22.214.171.124)  4.940 ms  3.924 ms  4.962 ms
         4  0.xe-8-0-0.br3.nyc4.alter.net (18.104.22.168)  13.551 ms  15.600 ms  14.553 ms
         5  188.8.131.52 (18.104.22.168)  14.996 ms  13.661 ms  15.424 ms
         6  ae-2.r23.nycmny01.us.bb.gin.ntt.net (18.104.22.168)  14.953 ms  16.048 ms  14.726 ms
         7  ae-6.r21.frnkge03.de.bb.gin.ntt.net (188.8.131.52)  109.988 ms  106.394 ms  107.488 ms
         8  ae-1.r02.frnkge03.de.bb.gin.ntt.net (184.108.40.206)  105.110 ms  108.378 ms  104.977 ms
         9  xe-3-2.r00.dsdfge02.de.bb.gin.ntt.net (18.104.22.168)  129.906 ms  141.522 ms  141.731 ms
        10  188.8.131.52 (184.108.40.206)  105.035 ms  104.582 ms  105.151 ms
        11  * * *
        12  * xe-0-1-0-3.r02.frnkge03.de.bb.gin.ntt.net (220.127.116.11)  143.231 ms *
        13  xe-0.level3.frnkge03.de.bb.gin.ntt.net (184.108.40.206)  161.436 ms  125.598 ms  159.774 ms
        14  * vlan90.csw4.frankfurt1.level3.net (184.108.40.206)  262.171 ms  270.225 ms
        15  ae-82-82.ebr2.frankfurt1.level3.net (126.96.36.199)  197.361 ms  214.021 ms  272.423 ms
        16  ae-61-61.csw1.newyork1.level3.net (18.104.22.168)  219.914 ms  244.709 ms  214.996 ms
        17  ae-21-70.car1.newyork1.level3.net (184.108.40.206)  232.323 ms *  203.640 ms
        18  intelsat-in.car1.newyork1.level3.net (126.96.36.199)  319.879 ms *  219.131 ms
        19  rvs-rt0003_fe-0-0 .intelsatone.net (184.108.40.206)  289.920 ms  357.010 ms  332.387 ms
        20  22.214.171.124 (18.104.22.168)  832.528 ms  844.070 ms *
        21  188.8.131.52 (22.214.171.124)  797.110 ms  839.233 ms  859.813 ms
        ... snip ...
        40  * *^C
        $ whois 126.96.36.199
        ... snip ...
        inetnum:        18.104.22.168 - 126.96.36.199
        netname:        STAR-KP
        descr:          Ryugyong-dong
        descr:          Potong-gang District
        country:        KP
        admin-c:        SJVC1-AP
        tech-c:         SJVC1-AP
        status:         ALLOCATED PORTABLE
        mnt-by:         APNIC-HM
        mnt-lower:      MAINT-STAR-KP
        mnt-routes:     MAINT-STAR-KP
        remarks:        -+-+-+-+-+-+-+-+-+-+-+-++-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        remarks:        This object can only be updated by APNIC hostmasters.
        remarks:        To update this object, please contact APNIC
        remarks:        hostmasters and include your organisation's account
        remarks:        name in the subject line.
        remarks:        -+-+-+-+-+-+-+-+-+-+-+-++-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        changed:        firstname.lastname@example.com 20091221
        source:         APNIC

        role:           STAR JOINT VENTURE CO LTD - network administrat
        address:        Ryugyong-dong Potong-gang District
        country:        KP
        phone:          +66 81 208 7602
        fax-no:         +66 2 240 3180
        e-mail:         email@example.com
        admin-c:        SJVC1-AP
        tech-c:         SJVC1-AP
        nic-hdl:        SJVC1-AP
        mnt-by:         MAINT-STAR-KP
        changed:        email@example.com 20091214
        source:         APNIC
        $

KP is North Korea. And their IP, 220.127.116.11, is indeed German. And telephone country code 66 is Thailand.

Edit: <http://en.wikipedia.org/wiki/Internet_in_North_Korea>

<comment><USER40> Hmm, tracepath and traceroute give me differing results. With traceroute the last hop I get is 188.8.131.52 which is apparently Cambodia. traceroute takes me from London to Germany to New York to Cambodia. It's a strange route.

</comment></comment></comment></comment></comment><comment><USER42> Can anyone repost the text here for those of use whose work filters block tpb?

<comment><USER43> PRESS RELEASE, NEW PROVIDER FOR TPB

FOR IMMEDIATE RELEASE, 3 MARCH 102, 평양 (PYONGYANG).

The Pirate Bay has been hunted in many countries around the world. Not for illegal activities but being persecuted for beliefs of freedom of information. Today, a new chapter is written in the history of the movement, as well as the history of the internets.

A week ago we could reveal that The Pirate Bay was accessed via Norway and Catalonya. The move was to ensure that these countries and regions will get attention to the issues at hand. Today we can reveal that we have been invited by the leader of the republic of Korea, to fight our battles from their network.

This is truly an ironic situation. We have been fighting for a free world, and our opponents are mostly huge corporations from the United States of America, a place where freedom and freedom of speech is said to be held high. At the same time, companies from that country is chasing a competitor from other countries, bribing police and lawmakers, threatening political parties and physically hunting people from our crew. And to our help comes a government famous in our part of the world for locking people up for their thoughts and forbidding access to information.

We believe that being offered our virtual asylum in Korea is a first step of this country's changing view of access to information. It's a country opening up and one thing is sure, they do not care about threats like others do. In that way, TPB and Korea might have a special bond. We will do our best to influence the Korean leaders to also let their own population use our service, and to make sure that we can help improve the situation in any way we can. When someone is reaching out to make things better, it's also ones duty to grab their hand.

Posted 24 mins ago by Kim Jung-Bay

<comment><USER21> "...being persecuted for beliefs of freedom of information."

Seriously?

The Pirate Bay's "freedom of information" posturing is a complete farce. The site exists for the sole purpose of making money for its operators. Messages like this one, whether satire or not, expose the site for what it is. It makes money by ripping content from the distribution channels chosen by the content authors, repackaging it, and giving it away for free while raking in advertising dollars. In the process, they deny content authors the freedom to select channel of distribution that will allow them to be compensated for their work.

Technology has paved the way for improved means of distributing content, but this is not it.

The only upside of TPB is that it will disrupt the existing content delivery chain enough to force some innovation in a space dominated by a few very large and very stagnant players.

<comment><USER44> > In the process, they deny content authors the freedom to select channel of > distribution that will allow them to be compensated for their work.

Pre-supposing one supports intellectual property.

</comment></comment></comment></comment><comment><USER45> Hosting a site already hated by US authorities in a country that the US considers to be an enemy is probably not the best idea. My guess is that a US Senator or Congressman is somewhere right now drafting a bill that would prohibit US internet providers from providing access to sites hosted in North Korea.

<comment><USER46> hey, they could probably buy the implementation from iran!

<comment><USER47> Why? They sold it to them.... (Ok, Nokia did....)

</comment></comment></comment><comment><USER48> EDIT: My previous comment was totally wrong. This is legit. The Pirate Bay (AS51040) is announcing routes via Ryugyong-dong (AS131279)

<comment><USER14> See <http://news.ycombinator.com/item?id=5319720> AS Paths can be modified pretty easily [http://blog.ioshints.info/2008/02/bgp-essentials-as-path- pre...](http://blog.ioshints.info/2008/02/bgp-essentials-as-path- prepending.html)

</comment></comment><comment><USER49> According to this blogpost [https://rdns.im/the-pirate-bay-north-korean- hosting-no-its-f...](https://rdns.im/the-pirate-bay-north-korean-hosting-no- its-fake) it's not really hosted in NK.

</comment><comment><USER50> The irony here is just too much to swallow. Of all possible places, a party campaigning for greater freedoms ends up in NORTH KOREA. I used be proud of defending tpb. Now I'm really not sure.

</comment><comment><USER51> Stupid move, so much that words fail me. "Korea" (as they call themselves, they don't acknowledge the autonomy of the South) has been looking for months for investors to replenish their economy in the face of mounting sanctions. TPB is giving them exactly what they need to become a greater menace to the region.

</comment><comment><USER52> > The Pirate Bay has been hunted in many countries around the world. Not for > illegal activities but being persecuted for beliefs of freedom of > information.

How noble. Good thing that they found refuge in the country that basically is one big concentration camp. Can't make this stuff up.

</comment><comment><USER53> Holy Crap. This is... unprecedented.

So North Korea will now have access to the entire Western World's culture catalog? For Free?! I wonder how Kim Jong Un will exploit that.

And now anyone who downloads a torrent from TPB has potentially been compromised? Or is that alarmist?

<comment><USER54> _So North Korea will now have access to the entire Western World's culture catalog?_

Understand the difference between the torrent file and the actual data and all shall be revealed.

<comment><USER53> Does having access to the torrent file not give you access to the actual data?

<comment><USER54> <http://en.wikipedia.org/wiki/Torrent_file>

I suppose it gives access in the same way that North Korea visiting Pirate Bay gave them access before. There is a widespread belief that Pirate Bay is some giant repository of a bunch of pirate content, and it simply isn't true. Well, it's true that it's content that aids pirates in finding and downloading that content, but pirate bay itself is quite lightweight.

<comment><USER53> I understand the difference. I was just assuming that since NK invited TPB they were planning on using it within NK. It also makes sense that if you were planning on downloading a large collection of torrents you would want the best possible access to those torrent files, as opposed to crawling for links.

<comment><USER54> Ah okay I understand. I don't think this has anything to do with access inside North Korea however, and if it is actually hosted in North Korea it is likely done purely as a middle-finger to the West, not as some sincere support for filesharing or anything like that.

</comment></comment></comment></comment></comment></comment><comment><USER55> This is probably a practical application of what we saw earlier with the star wars traceroute.

Trace 22.214.171.124 to see the effect if you haven't seen it already.

</comment><comment><USER56> This is hilarious if true (nslookup points to 184.108.40.206 in germany, so I have my doubts). Hopefully (and this is really idealistic of me, because I suspect the reaction will be entirely negative) this opens up a larger conversation on internet freedoms in an absurd way beyond SOPA/CISPA/whatever internet freedom-stripping bill-of-the-day is debated on capitol hill.

</comment><comment><USER57> Isn't it one month early for April fools? Oh wait...

Seriously this is mind boggling! I can't believe it has come to that!

<comment><USER58> It is, isn't it? UK is a first world country. They've blocked TPB. NK is an oppressive regime. They've blocked everything.

I guess that's the modern difference between "rogue" and "first world" as far as censorship goes. The former block because of threats to the political regime. The latter because of threats to corporate profits.

</comment></comment><comment><USER59> Yeah, well, good luck with that.

We have a client in a chinese data center where they would just turn off the power randomly because the government said so.

Not to mention the unscheduled blackouts.

</comment><comment><USER60> Does NK have enough bandwidth for something like this? They just have one terrestrial link from PRC, right?

</comment><comment><USER61> Just. Feeling. Surreal.

But, the irony of the _Politics makes strange bedfellows_ does not seem lost on them.

</comment><comment><USER62> Ironic that no one in the country pirate bay is now hosted will be able to access its contents. The free world has to go to the totalitarian world to access data and the totalitarian world have have to come to the free world to access the data.

</comment><comment><USER63> Do you really want North Korea to know exactly which copyrights you've violated and which banned books you've read? I mean, your torrent client sends that information to the tracker. (But then, it also sends it to other peers.)

</comment><comment><USER64> I know it's TPB so I should have known this, but beware of this link at the office. If you don't have a good ad-blocker installed it will throw up some non-HR friendly popups!

</comment><comment><USER65> Someone send them this story <http://www.northkoreanrefugees.com/2007-09-atbirth.htm>.

</comment><comment><USER66> Prophecy fulfilled :)

<http://openclipart.org/people/worker/1325935795.svg>

</comment><comment><USER67> Well, I can certainly point to this article next time someone tries to tell me how awesome the HN comment section is.

</comment><comment><USER68> What's really ironic is that a lot of the content hosted on the Pirate Bay is banned in North Korea.

</comment><comment><USER69> Apparently North Korea has more information freedom than rest of the world.

<comment><USER70> unless you live there.

</comment></comment><comment><USER71> And they're down again!

</comment><comment><USER72> Mirror, anyone? I can't get to it on Comcast.

</comment><comment><USER73> Was this Dennis Rodman's idea?

</comment><comment><USER74> This is so fantastic.

</comment><comment><USER75> the enemy of my enemy is my friend

</comment><comment><USER76> the site isn'r working for me...

<comment><USER77> Most likely your ISP is censoring you.

<http://www.bbc.co.uk/news/technology-18833060>

<http://news.bbc.co.uk/2/hi/uk_news/7770456.stm>

</comment></comment><comment><USER9> Posts like this show that however smart we generally believe HN users to be, on some fronts it is a bag full of stupid.

</comment><comment><USER78> I realize this is propaganda but it's working on me. I'm starting to like Kim Jong-Un. First Eric Schmidt, then Dennis Rodman and now TPB. If these are the kinds of friends he wants to make, I'm a fan.

This is a masterful 'fuck you' to the establishment and at the same time an olive branch to the internet generation all over the World. The people who will be running the World in 20, 30, 40 years - when he is still in power.

<comment><USER78> The only two things I dislike about HN:

-It's a crapshoot or manipulation game to hit the front page

-People downvote logical arguments they simply disagree with

</comment></comment><comment><USER24> My first reaction: GET OUTTA HERE!! YOU GOTTA BE KIDDING!

but oh well, I have a wiggly feeling in my mouth each time I read this, I don't know, whether this is funny or world changing or ironic or god knows what. The internet has melted a new barrier.

Cheers to THE BAY!</comment>


Sony to push ahead with PS5 launch - <AUTHOR> https://www.ft.com/content/ab54e477-eb02-4256-839e-c1b0174b79dd <comment><AUTHOR> Also covered in PC Mag:

[https://www.pcmag.com/news/sony-confirms-ps5-will-launch- in-...](https://www.pcmag.com/news/sony-confirms-ps5-will-launch-in-2020)

and in BGR:

[https://bgr.com/2020/05/13/ps5-release-date-sony-says- holida...](https://bgr.com/2020/05/13/ps5-release-date-sony-says-holiday- season-launch-plans-unchanged/)</comment>


This architecture tastes like microarchitecture [pdf] - <AUTHOR> http://wp3workshop.website/pdfs/WP3_dunham.pdf <comment><USER2> If you look at the context of the paper, it makes a little more sense. This is a one-day workshop that's effectively a retrospective look at computer architecture over the past 50 years and asking "what's next?" So the real point of the paper is the somewhat meandering discussion of "why exposing microarchitecture is bad" that's the first part of the paper, with the latter part being a suggestion of how to take that advice to the future. That said, it's (IMHO) the worst of the bunch.

The point the paper is trying to make is that exposing microarchitecture to the ISA is a decision that ended with problems. It then claims that some of the bulwarks of modern ISAs still act too much like exposing- microarchitectural details, namely the presentation of a finite register set. It sort of doesn't bode well when your references to "this is totally a fruitful idea to go down" are all limited to papers in the 1990s.

</comment><comment><USER3> Memory is the bottleneck... so let's use memory ever more intensely.

-_-

Seriously, hardware accelerated context switching is much more of a viable option rather than saying "screw registers, we'll make memory do _absolutely everything_. To make matters worse, in most languages you can't tell where the pointers are going before you AG them, which results in pointer chasing pain.

Stateless ISAs are interesting, but I seriously doubt the way forward is to make memory do everything for us.

Also, the biggest cost of context switching is not in the register file migration, but in the flushing of the TLBs, look to SASOS+VIVT to address that, not whatever this is proposing.

<comment><USER4> One point the authors make is that registers are memory. I'm not sure they're saying screw registers, just that they're counter to portability and in general only a yet faster cache for programs to use. The other point they seem to make is that a secondary reason we have architected registers is that renaming over an arbitrarily large space was (and in some ways is still) difficult. By using graph coloring and reg allow in compiler, we narrow the encoding space before execution and give the hardware a much simpler problem. Recent research results on register less arch show that there are quite a few benefits. There's a lie in calling it registerless though, it's still there, just transparent and hence portable.

</comment></comment><comment><USER5> Raising the abstraction level of the ISA and making more logic part of proprietary CPU microcode is exactly the wrong direction to go in for security. I'm not sure how the authors can possibly claim this will be beneficial for security. Thankfully, recent events (Meltdown and Spectre) have made the flaws in their philosophy abundantly clear.

>When considering the period of rapid evolution that microarchitecture is about to face with the end of lithography scaling, abstractions that are free from underlying microarchitectural influence are critical to minimizing future disruption.

Ironically completely accurate! If chipmakers can successfully sell us on super-high-level ISAs (moving the scheduler and hypervisor inside their proprietary chips, as this article seems to suggest), they will be able to lock us in and easily prevent any "disruption" of their business model.

The greatest advantage for innovation in ISAs in the past 20 years has been the development of Linux and GCC, which allow any new chip to get a huge amount of working software with relatively little porting effort. Moving more logic out of this open source software portability layer and into the proprietary chip will just make it harder to build new chips from scratch.

<comment><USER6> I don't think that's what they're suggesting. This is more about replacing most register references within opcodes with stack pointer offsets, for faster context switching. It only replaces the register-spilling part of a context switch, along with the register-allocation code in a compiler.

It sounds a bit like SPARC, but more flexible? And not particularly difficult to port to.

<comment><USER2> > It only replaces the register-spilling part of a context switch, along with > the register-allocation code in a compiler.

The compiler would still need to stack slot coloring. And, as I noted elsewhere, they also seem to suggest dropping memory coherency, which makes the stack here just a register file with single instructions that load and store the registers to memory.

</comment></comment></comment><comment><USER7> Welcome to the club.

* The Myth of Sufficiently Smart Compiler (SSC)

* The Myth of Sufficiently Smart Virtual Machine (SSVM)

* The Myth of Sufficiently Smart Instruction Set Architecture (SSISA)

All these ideas have the same coal. How to preserve and use high-level information when transforming code to the lower level in a way that gives maximum performance.

In a hypothetical dreamland where where all these exist, compilers, hypervisors, virtual machine monitors, microkernels,operating systems, ISA's and microcode would generate "sufficiently smart" stack that provides performance increase.

[http://wiki.c2.com/?SufficientlySmartCompiler](http://wiki.c2.com/?SufficientlySmartCompiler)

[http://wiki.c2.com/?SufficientlySmartVirtualMachine](http://wiki.c2.com/?SufficientlySmartVirtualMachine)

</comment><comment><USER8> The author basically says "we've been doing it wrong" and doesn't provide an alternative.

<comment><USER4> Meh, for a retrospective paper, I think the authors probably provided more hints as to the future than they really needed to. I feel like they captured the exact reasons why ISA is so difficult to do right. As for "doing it wrong", I didn't take it that way. I think the authors lament the influence of what they seem to feel is a mantra that dictates the programmer must deal directly with the hardware versus providing a sufficiently abstract ISA. Building circuits for ML accelerators, etc. is actually damned easy, exposing those to the programmer in a portable way that does not require rearchitectng the program every time you change the accelerator is tough. I literally loathe porting for Intel b/c the AVX insn behave differently and are essentially architecting in the microarch, passing the complexity directly to the programmer. I'd much rather the Risc-V solution or the ARM solution.

</comment></comment><comment><USER9> Tl;dr, as far as I can tell: Context switch performance can be improved by abolishing registers. Also, here's some indications that this won't _necessarily_ completely annihilate processor performance.

<comment><USER10> Which sounds a bit like implementing the LLVM IR as an ISA. You don't have registers, but instead you get an infinite number of temporary values.</comment></comment>


Ask HN: Biz folks learning how to hack? - <AUTHOR>

It's funny when I think about the topic of my question now.
Traditionally, engineers, hackers and techies would seek to pick up managerial skills to advance their carrier, well at least in stupid boring-suited-corporate-fortune-1000.
Don't get me wrong, I believe leadership skills are indispensible for founders. However, the rapid development of technology and internet is
pushing the other way around – Biz folks learning to hack.
It's been pushing me since awhile now, and before jumping off the cliff, I really would appreciate your help in where and how to start climbing up with a bag of programming/hacking skills.
A bit about myself, I am from Algeria. I am not entirely a "suit" person, but my focus has been on leadership, management and language-
teaching throughout the past 7 years. I co-founded a social start-up
of an international student organization.
Funny enough, I first joined university majoring CS but then later switched to statistics – don't ask me why, I have no idea how I made that decision, I guess it was because I started to be attracted to
business and leadership. Any way my 1 year CS at university allowed
me to gain some knowledge in Pascal algorithms and math.
I am not seeking to learn hacking simply because I have to but rather I want to – It is beautiful to have that ability to help build the pr- oduct you have in mind (YC folks make a strong point here).
I do have 2 business ideas, and I am thinking of approaching 2 of my techie friends to possibly apply for the upcoming YC summer batch but I am more heading towards bringing the idea to life first then
apply next times to YC, allowing myself to gain some hacking skills.
So Hackers, how should I start this journey off?
Merci d'avance
P.S: programming skills I am looking for tend to be around web development. <comment><USER2> I think the "be your own CTO" advice for non-techies starting out with just an idea is always solid. I had a similar programming background to you and for over a decade I was working on the business side of things, successfully launching product from sales, marketing and business development roles.

However after having technical co-founder issues twice I recognized I needed to learn how to, at least in the early days of testing an idea, be my own technical co-founder. How to hack the initial stages of an MVP together in order to test things out.

I learned Ruby on Rails back in 2010 and haven't looked back since. It's easy to pick up and there is so much good information out there on how to learn you can quite quickly, if you're smart and committed to doing so, build up from being an absolute beginner to a reasonable intermediate-level programmer able to put out a functioning minimal viable product.

It won't be perfect, and you'll want to bring in the star programmers later. But it's not something you'll ever regret.

A simple analogy which you may appreciate as an Algerian. As an English speaker if you go to Paris and only speak in English you won't find a Frenchman who understands a word you say. If you attempt your best French, however poor, you'll find people talking back to you in English and willing to help. Same with engineering talent. Every engineer is approached all the time by non-technical people with "an idea" who "just" need it implementing by a techie. Unsurprisingly these approaches don't get taken up often. However if you have a product that is viable and working and now needs making better, more scalable, more secure, you have something that you can engage other engineers in.

<comment><AUTHOR> Thanks a lot, I very much like the analogy :) You covered it all.

To add more, even if you have star techie co-founders, it doesn't make much sense to be that business person with no understanding of programming, not even the basics. And even if the business requires enterprise sales.

Practically, it would affect brainstorming at early start up stages for example.

</comment></comment><comment><USER3> If you re going to start today, just learn javascript and nodejs. Like thar you will only need to be manipulating 1 language in addition to html and css to pump out webapps. Look into node's express framework. It's great for bootstrapping apps.

<comment><AUTHOR> Thanks a lot, I guess I need to start first with HTML and css

</comment></comment><comment><USER4> There are many good paths to take, and I'd avoid discussions about the "right" language to start with. Pick any language and run with it. If, down the road, you need to use a different set of technologies, then that time will come. But right now, it's time to learn. If you decide to learn Ruby, feel free to email me if you need any help getting started. It's my HN username at gmail. Good luck!

<comment><AUTHOR> I tried to email you. I got a delivery failure. HN firstname.lastname@example.com right?

<comment><USER4> adelevie at gmail

</comment></comment></comment><comment><USER5> I'm surprised this hasn't been mentioned already, but as a business-side person with an interest in "hacking", I find Codecademy to be extremely valuable. You can learn all kinds of programming languages there.</comment>


Python web scraping resources - <AUTHOR> http://jakeaustwick.me/python-web-scraping-resource/?mc_list=python <comment><USER4> A nice write up. At a previous company I built a solution (was forced into java but ultimately the same process...) that used many of these techniques. Some suggested next steps/additional enhancements if you need to do this repeatedly and at scale.

Implement global throttling on a per domain basis.

Consider some abstraction. I implemented an abstract fetcher (with a number of concrete fetchers that were runtime selectable), and an abstract/concrete parser. Compose a Scraper with these two things. Allow for a runtime switch that will determine which fetcher to use (javascript enabled, straight requests, etc.). If you want to get really fancy, in your database of urls, you can flag urls that need to use a heavier, full fledged browser.

For the fetcher, use the selenium bindings. We tested phantomjs and chrome and chrome outperformed phantom. It might have been the java bindings (ghostdriver) but w/e, something you just have to test for yourself. Once we settled on chrome, I built a chrome plugin to block ads and other unrelated calls. This adds LOADs of time. Its pretty tricky but you can inject in a list of well crafted regexes and it drops initial load times dramatically.

For the parser, you may want to consider a fallback system. Often times the particular piece of data you want (say a title) can be found in a handful of places on the page. It will make your parsing much more reliable.

Compose a 'bot runner' from the Scraper. We had json documents that described fields we were abstracting and all the fallback rules used to locate the needed data. Lastly, the bot runner can be expanding to include things like navigation and other fancier tricks.

If you go for broke, build a system for generating bots, think chrome plugin.

Don't forget, pruning dead URLs is a tricky little problem but an important one.

To scale this whole operation linearly, we used a queue (redis at first, eventually kafka) and storm. Storm allowed us to arbitrary expand and contract our bot runners.

Scraping is a problem that just about everyone encounters and alot of the most standard solutions seem to really fall short. Your article is an excellent start.

<comment><USER5> Curious what was the catalyst to switch from Redis to Kafka? Reliability that a given message was received and processed and replaying that to a different consumer in the event of failure?

<comment><USER4> mostly just scale. Keeping your queue in memory isnt necesarilly a requirement unless you have a lambda arch or a realtime requirement (which we started with). For our larger operations, kafka, which is disk backed, distributed with a little more ease and cheaper was a nice option. For reliability, we were using some of storms primitives as well as the mechanisms inside qless, the queuing library we were using on top of redis. [https://github.com/ChannelIQ/qless-java](https://github.com/ChannelIQ/qless- java)

</comment></comment></comment><comment><USER6> I would highly recommend Scrapy if you plan on doing any serious scraping: [http://scrapy.org/](http://scrapy.org/).

<comment><USER7> I wouldn't. I used this for a project and then quickly regretted it for the following reasons:

* XPath selectors just plain suck.

* The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it.

* Scrapy does not play well with other projects. You _can_ integrate it with django if you want a minimal web UI but it's a pain to do so.

* Tons of useless features. Telnet console? wtf?

* It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself.

* For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up).

IMO this is a classic case of "framework that should have been a library".

Here's what I used instead after scrapping scrapy:

* mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining.

I'm largely happy with the outcome. The code is less straitjacketed, easier to understand and easier to integrate into other projects if necessary (you don't have the headache of trying to get _two_ frameworks to play together nicely).

<comment><USER8> Hey,

A good feedback, thanks!

> XPath selectors just plain suck.

Scrapy supports CSS selectors.

> The item pipeline is way too straitjacketed. You have to do all sorts of > fucking around with settings in order to make the sequence of events work in > the (programmatic) way you want it because the framework developers > 'assumed' there's only one way you'd really want to do it.

Could you plese give an example?

> Scrapy does not play well with other projects. You can integrate it with > django if you want a minimal web UI but it's a pain to do so.

This is true. But it is a pain to integrate any even-loop based app with another app that is not event-loop based. It is also true that Scrapy is not easy to plug into existing event loop (e.g. if you already have twisted or tornado-based service), but it should be fixed soon.

> Tons of useless features. Telnet console? wtf?

Telnet console is a Twisted feature; it came almost for free, and it is useful to debug long-running spiders (which can run hours and days).

> It's assumed that the 'end' of the pipeline will be some sort of > serialization - to database, xml, json or something. Actually I usually just > want to feed into the end of another project without any kind of > serialization using plain old python objects. If I want serialization I > probably want to do it myself.

If you don't want serialization then you want a single process both for crawling and for other tasks. This rules out synchronous solutions - you can't e.g. integrate a crawler with django efficiently without serialization. If you just want to do some post-processing then I don't see why putting code to Scrapy spider is worse than putting it to other script and calling Scrapy from this script.

> For some reason DjangoItem didn't really work (although by the time I tried > to get it to work I'd kind of given up).

This may be true.. I don't quite get what is it for :)

> IMO this is a classic case of "framework that should have been a library".

It can't be a library like requests or mechanize for technical reasons - <USER3> make crawling efficient Scrapy uses event loop. It _can_ (and should) be a library for twisted/tornado/asyncio; it is possible to use Scrapy as a such library now, but this is not straightforward; this should (and will) be simplified.

> * mechanize - to mimic a web browser. I used requests sometimes too, but it > doesn't really excel at pretending to be a web browser, so for that reason I > usually used mechanize as a drop in replacement. * celery - to schedule the > crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - > because xpath selectors suck and jquery selectors are better. * python > generators - to do pipelining.

Celery is also not the easiest piece of software. Scrapy is just a single Python process that doesn't require any databases, etc.; Celery requires to deploy a broker and have a place to store task results; it is also less efficient for IO-bound tasks.

<comment><USER7> >Scrapy supports CSS selectors.

Still far inferior to JQuery selectors.

>Could you plese give an example?

The example I'm thinking of is when I was trying to create a pipeline that would output a skeleton configuration file when you passed one switch and would process and serialize the data parsed when you passed another. It was possible but kludgy.

>But it is a pain to integrate any even-loop based app with another app that is not event-loop based.

That's not where the pain lies. It's more the fact that it has its own weird configuration/setup quirks (e.g. its own settings.py, reliance on environment variables, executables).

>If you don't want serialization then you want a single process both for crawling and for other tasks. This rules out synchronous solutions - you can't e.g. integrate a crawler with django efficiently without serialization.

I don't really want scrapy doing process handling at all. It's not particularly good at it. Celery is much better.

Using other code to do serialization also doesn't necessitate running it on the same process. You can import the django ORM wherever you want and use it to save to the DB. I know you _can_ do that - but, again, kludgy.

>It can't be a library like requests or mechanize for technical reasons - <USER3> make crawling efficient Scrapy uses event loop.

I get that. It should have been more like twisted from the outset though. The developers were clearly inspired from django and that led them down a treacherous path.

>It can (and should) be a library for twisted/tornado/asyncio; it is possible to use Scrapy as a such library now, but this is not straightforward; this should (and will) be simplified.

Well, that's good I suppose. I still think that it focuses on bringing together a bunch of mediocre modules for which, individually, you can find much better equivalents. Also, (unlike django) tight, seamless integration between those modules doesn't really gain you much.

>Celery is also not the easiest piece of software.

The problem it is solving (distributed task processing) is not an easy problem. Celery is not simple, but it is GOOD.

>Scrapy is just a single Python process that doesn't require any databases, etc. Celery requires to deploy a broker and have a place to store task results; it is also less efficient for IO-bound tasks.

A) You can use redis as a broker and that's trivial to set up. I always have a redis available anyway because I always need a cache of some kind (even when crawling!).

B) My crawling tasks are never I/O bound or CPU bound. They're bound by the rate limiting imposed upon me by the websites I'm trying to crawl.

C) I'm usually using celery anyway. I still have to do task processing that DOESN'T involve crawling. Where do I put that code when I'm using scrapy?

<comment><USER8> > The example I'm thinking of is when I was trying to create a pipeline that > would output a skeleton configuration file when you passed one switch and > would process and serialize the data parsed when you passed another. It was > possible but kludgy.

I don't get it - how is creating a configuration file related to the processing of the items, why would you do it in items pipeline?

> It's more the fact that it has its own weird configuration/setup quirks > (e.g. its own settings.py, reliance on environment variables, executables).

It is possible to create a Crawler from any settings object (not just a module), and Scrapy does not rely on executables AFAIK. But this all is poorly documented. Also, there is an ongoing GSoC project to make settings easier and more "official".

> I don't really want scrapy doing process handling at all.

Scrapy doesn't handle processes, it is single-threaded and uses a single process. This means that you can use e.g. a shared in-process state.

> Using other code to do serialization also doesn't necessitate running it on > the same process. You can import the django ORM wherever you want and use it > to save to the DB. I know you can do that - but, again, kludgy.

You can't move Python objects between processes without serialization. Why is using django ORM kludgy in Scrapy but not in Celery?

> The problem it is solving (distributed task processing) is not an easy > problem. Celery is not simple, but it is GOOD.

You don't necessarily need distributed task processing to do web crawling. Celery is a great piece of software, and it is developing nicely, but you always pay for complexity. For example, I faced the following problems when I was using Celery:

* When redis was used as a broker its memory usage was growing infinitely. Lots of debugging, found a reason and a hacky way to overcome it ([https://github.com/celery/celery/issues/436](https://github.com/celery/celery/issues/436)). The issue was fixed, but apparently there is still a similar issue when MongoDB is used as a broker.

* Celery stopped processing without _anything_ useful in logs (and of course Celery error sending facilities failed and I didn't have external monitoring) - it turns out an unicode exception was eaten. A couple of days of nightmarish debugging; see [https://github.com/celery/celery/issues/92](https://github.com/celery/celery/issues/92).

* I implemented an email sender using Celery + RabbitMQ once. I think I was sending email text to tasks as parameters. Never do that (just use MTA:)! When a large batch of emails was sent at once RabbitMQ used all memory, corrupted its database, dropped the queue; I haven't found a way to check which emails were sent and which were not. This was 100% my fault, but it shows that complex setup is not your friend.

Crawling tasks differ - e.g. if you need to crawl many different websites (which is not uncommon) you will almost certainly be IO and CPU limited. Scrapy is not a system for distributed task processing, it is just an event- loop based crawler. I'm not saying your way to solve the problem is wrong; if you already use celery it makes a lot of sense to use it for crawling as well. But I don't agree that going distributed turtles all the way down with celery+redis+DB for storage+... is easier or more efficient than using plain Scrapy. A lot of tasks can be solved by writing a spider, getting a json file with data and doing whatever one wants with it (upload to DB, etc).

</comment></comment></comment></comment></comment><comment><USER9> I wish every post/tool out there on scraping covered obeying robots.txt. It's a crap standard, but it's what we've got.

"Just ignore it" is a great way to identify yourself as a crappy netizen.

<comment><USER10> I'm glad that all of the sites you target want your scraper to access them. The goal in many cases where one would use a scraper is to access information not provided in an API or otherwise encased in HTML. Most of their robots.txt are "User-Agent: *\nDisallow: /\n"

<comment><USER9> Then we have no right to scrape that content.

Why is there an implied right to scrape?

</comment></comment></comment><comment><USER11> As a self-taught Python and Ruby programmer, I really appreciate this.

</comment><comment><USER12> Clean and direct write up. It reminds me when I used to work on crawling/scrapping. Covered most of the topics that needs to know for web crawling.

</comment><comment><USER13> Very nice, thanks for posting =).

Can people suggest any additional resources/reading on scraping/crawling as well?

I was hoping to experiment with it in GoLang, but there doesn't seem to be much on crawling/scraping with GoLang, except for GoQuery ([https://github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery))

</comment><comment><USER14> Little distributed web scraper project I created a while back if anyone is interested / needs resources : [https://github.com/Diastro/Zeek](https://github.com/Diastro/Zeek)

</comment><comment><USER15> Really nice. I was just going to comment on how similar this was to scraping I did during my days working in SEO, and then I saw your username! Long time no talk. What's up man? Sounds like we should get in touch.

</comment><comment><USER16> indepth article! Had to learn most of the stuff the hard way, could need that a couple of weeks ago:) Proxies nowadays are really cheap. Isnt ignoring the robot.txt opening doors for suing? Scraping copyrighted material should be avoided too in my opinion, but i guess that only matters if you get caught:)

</comment><comment><USER17> I need to scrape an eccomerce site this weekend, and this will be a great resource to keep bookmarked. Thanks.

<comment><USER18> Check out [http://diffbot.com/products/automatic/product/](http://diffbot.com/products/automatic/product/) to do this fully automated.

<comment><USER19> I'll self-promote too then. [https://screenslicer.com](https://screenslicer.com)

<comment><USER20> very cool! probably one of the first web scrapers I've run across with a front end where you can easily show users / investors etc what this does. and it seems to work pretty well. really like your output display as well.

</comment></comment></comment></comment><comment><USER21> python selenium bindings are also nice if you absolutely have to deal with information put together by javascript code.

<comment><USER22> I actually cover this (very briefly) here: [http://jakeaustwick.me/python-web- scraping-resource/#thesite...](http://jakeaustwick.me/python-web-scraping- resource/#thesiteisshowingdifferentcontenttomyscraper)

I should dedicate a section to it though, will stick that on my to-do list.

<comment><USER23> I've always opted for Ghost.py rather than selenium as I've found it uses less memory and pretty capable. Admittedly my scrapes are normally pretty targeted and not over 10k+ sites.

One query I had from your piece: "When I've found myself in the unfortunate place of getting my proxies banned before on certain sites, they have been more the happy to switch them out for new IP's for me."

If you're getting banned from sites, is it not time to leave them alone? If they don't want your traffic (which the admin/system has judged as too much), should you really be circumventing it? With that and the robots.txt bit (which admittedly you justified), you've got to be careful not to slip into a bit of a grey area with scraping, which people regard suspiciously in the first place.</comment></comment></comment>


Machinery to extract ChangeLogs from git repositories - <AUTHOR> http://allmychanges.com/

<comment><AUTHOR> We wrote this project during last DjangoDash.

I'm wondering if somebody is interested in a service to track changelogs of various libraries?</comment>


Ask HN: Why don't we have academy awards for software? - <AUTHOR>

Aside from the webby awards, and other web/design specific awards, turing award for computer science achievements, I haven't seen (or Googled enough) to find one award to rule them all.
I wonder what attempts what arguments for and against are for such an award. The award should be voted by the top of the software community, which will be academia, startups, VCs, enterprise (yes, we have to), students, and select independent people.
the awards will be for things like, best open source project, best software design, best commercial product, best mobile application, best SaaS, best bootstrapped startup, best linux project, best Ruby on Rails project, best Scala project, best Python project (and so on)
The benefit of such awards are superficial but
- people like getting awards - people like award shows - will help promote good software projects
- Hollywood has them, why not us? - will encourage people to learn SE or CS
Now please tell me why is that a crazy idea that won't work, (I thought of a few reasons, but I chose to ignore them) <comment><USER2> Its a nice idea, I'm sure there are tons of people who go unrecognised for what they've done. However, I can't help but think it will make the pseudo celebrity nonsense going on in the tech-sphere an order of magnitude or more worse than its already become. Imagine the collusions that would ensue from certain "news bloggers" and others in the SV cespool with awards in the mix. I remember this little gem: [http://www.realdanlyons.com/blog/2012/02/13/hit- men-click-wh...](http://www.realdanlyons.com/blog/2012/02/13/hit-men-click- whores-and-paid-apologists-welcome-to-the-silicon-cesspool/)

</comment><comment><USER3> There are the CODiE awards

<http://www.siia.net/codies/2013/>

I built a pruduct that was nominated in the educational category several years ago. The awards event was really fun but otherwise I don't see it as a way to attract more CS/SE students.</comment>