The Extraordinary Popular Delusion of Bubble Spotting - <AUTHOR> http://online.wsj.com/article/SB10001424052970204621904577017960729384948.html

<comment><USER2> The book mentioned, "Extraordinary Popular Delusions and the Madness of Crowds," is a great read (at least the first half) that I would recommend to anyone who wants to be able to spot a bubble.

The free eBook is here: <http://manybooks.net/titles/mackaych2451824518-8.html></comment>


I Wasn't Paid - <AUTHOR> http://iwasntpaid.com/ <comment><USER3> This is not a way forward.

1). Work with better clients. You can have invoice collection problems at $50k, too, but they're much less likely than at $500 and you have _much_ better options for... escalation methods at that point.

2). If you're not a bank, stop taking on so much credit risk. Businesses can deal with substantially sterner payment terms than you'd think to offer. If your clients balk, see #1. (Thomas has some suggestions here, too, many of which reduce to "Work for Thomas' clients and charge enough that dealing with their purchasing processes is worth your headaches rather than chasing deadbeats for Snickers money.")

3). Showing up on iamachumpconsultant.com will not enhance your professional reputation, bill rate, or client pool.

<comment><USER4> Patrick, I completely agree with you, which is why I have never publicized my difficulties with problem customers.

But something in my reptilian brain really still likes this. Let me tell you why...

You say "Work with better clients." Working with better clients means avoiding the bad ones. Avoiding the bad ones means being able to identify them. Being able to identify them means being warned.

It's almost like seeing an obstacle in the highway and wanting to warn others going in the opposite direction. But how do you do that? I once had the idea of putting a giant tattoo "Avoid me." on my bad clients' foreheads.

The biggest problem with bad clients isn't that they took you, it's that they never go away; they just keep popping up over and over, only to take advantage of others.

One former client of mine is the perfect example. He never paid his bills, he was extremely abusive, he was often very unethical, and would do _anything_ to make an extra buck. He would pop up all over the country with a different name, starting the cycle all over again. I recently noticed he just got out of jail with a brand new name and web site. If only I could find a righteous way to warn others, "Run the other way!"

<comment><USER3> If you want to perform (for lack of a better word) a mitzvah for the community, rather than picking out a single abusive customer, use your experience and teach people how to pattern match.

Memo to young Rails developers: "I have an idea for a website. If you build v1.0 I'll pay you _a whole thousand dollars_ and you might get equity if it works very well." has never had a happy ending. (Slight overstatement there, but it won't weigh on my conscience.)

<comment><USER4> _"I have an idea for a website. If you build v1.0 I'll pay you a whole thousand dollars and you might get equity if it works very well."_

If only all the assholes were so easily self-identifying.

The real problems are the charming, smooth talking, dream clients offering cool projects, top dollar, and paying on time...until they don't and decide to start making your life hell because you're in so deep already. They're no dummies.

<comment><USER5> Agreed. I had a project a couple years ago, just like this. Everything started off fine -- solid contract, good communication, etc. Payment was solid for the first few months, then it stopped. I was accused of putting together templates with no functionality behind them, and they demanded their money back. When I informed them where they could put their demands, so to speak, they threatened litigation. I had done the work and knew the contract was solid, but I couldn't have afforded a lawyer. Thankfully, they soon disappeared... But not before stealing their customers' funds. Ugh.

In retrospect, there were a few warning signs fairly early on, but it's hard to see that when you need the money to eat.

<comment><USER6> _In retrospect, there were a few warning signs fairly early on_

Care to elaborate to teach the communtity?

<comment><USER7> Hindsight is 20/20, but typically any sign that they're weird and irrational (trail of angry developers/no business model), or that they'll try and screw you out of a fair payment (won't do commencement payments) are a red flag.

Even if you can't afford a lawyer, I would have something along the lines of terms and conditions, what happens on non-payment (usually work stops), etc. The "Fuck you, pay me" is a pretty good introduction to the field.

For web design you'll have milestone payments on commencement, initial design (with a couple of rounds of changes), conversion to HTML (with no changes) and installation on the site. Payment terms are 7 days, or as short as you can get away with. If they need more changes than agreed, or they change their mind, then they pay the balance and you requote for the new work. If they decide not to use your work, then they pay the balance. If they delay on any part of the work (eg. won't provide content for six months) then they pay the balance and you do the work later on. You get the idea, I'm sure.

If they balk at any of your conditions without a good reason, that's usually a sign that they'll be trouble and that you'll need to keep them on a tight leash and/or handle them with kid gloves.

On the flip side, if they're a good client, good payer and you're happy to work for them, then give them a lot of leeway. But the deadbeats get held to the letter of the agreement.

<comment><USER6> _trail of angry developers_

Hehe. Reminds me of a former client (sorta) who would always wind up working for assholes. Everyone he had worked with turned out to be an asshole would couldn't do what he wanted. Turns out he was the asshole.

It's almost impossible for everyone else to be an asshole. It's like the old poker tip. If you look around the table and can't find the sucker, it's you. If you think everyone is an asshole, you're probably an asshole.

<comment><USER7> Bonus points if you find angry rants from the previous developer in PHP comments :)

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> What if this happens:

I work with a difficult dev/designer for a couple of months trying to bootstap a product. In the end it's not working, we agree to go our own ways, but I keep the name of the product. For what ever reason they get shitty with me, and a month or two down the line when I'm working with someone else they post about me on 'iwasntpaid' - not just once, loads of times - they know a lot about me so they can make it look like a string of unhappy dev/designers are annoyed and I can never be sure who it was posting this.

Would the site owner let me/help me remove these? Will they be asking for proof? If not I can see them getting in some interesting legal battles pretty quickly.

I think something like: <http://youthoughtwewouldntnotice.com/blog3/> is great - but it is totally open and honest - you state who you are and why you feel you've been ripped off.

This anonymous sniping isn't going to help anyone. If someone rips you off write a blog post about it, tell people you work with, or better yet talk to the offending party in person.

This feels like bitching-meets-4chan. At least <http://www.grouphug.us> tries to keep both sides anonymous.

<comment><USER9> FYI: <http://youthoughtwewouldntnotice.com/blog3> leads to a giant ad for some Canadian pill site.

<comment><USER8> Really - no adverts when I visit it?

Edit: Some sort of IE7 hack/virus/attack? (sorry can't reply to your message for some reason)

<http://i.imgur.com/r8u2I.png> \- screenshot is with adblock turned on, but I also tried it with it turned off and didn't get it.

<comment><USER9> I'm on IE 7 (don't ask), so perhaps the ad is the site's way of telling me "F U"?

Edit: Screenshot - <http://i.imgur.com/5lnJ7.jpg>

<comment><USER10> Your computer probably has malware. Run ComboFix and that should fix the issue.

<http://www.bleepingcomputer.com/combofix/how-to-use-combofix>

<comment><USER9> I'm on a highly secured military network. While it's not impossible that there's malware on my computer, I think it's more likely that something's up with the website.

Edit: There's no choice of browsers on this network. It's IE 7 or nothing.

<comment><USER11> May I suggest that you refrain from surfing the Internet using IE7 while on a secured military network.

<comment><USER9> Unfortunately there is no other option. You use the software you're given, and that's it.

<comment><USER12> If you're browsing random sites that could well be infected, even if you weren't using IE7 the fact that you're on a "secure military network" doesn't stop you from being just as vulnerable as anyone else browsing random sites. Sounds to be like the security of your network has at least one hole in it.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> I was involved in the case involving Carlos 'Storm' Martinez, who is an amazing post sound mixer and a stand-up guy. I am a little perturbed to see him mentioned on the site.

The design of the site in question was good, no doubt, but it was also riddled with a stolen Neutraface2 font face and the HTML didn't validate. And there was no RoR backend; the whole site was hosted on a GoDaddy shared server.

From my understanding of the dispute, the site was delivered many months late, and then the designer doubled his price. When Storm refused to pay double for late work, the designer linked in an overlay from a file hosted on Dropbox that announced 'This site was stolen'. That was the 'kill switch'. The designer also made a few blog posts and tweets bragging about his 'kill switch', which he later took down.

This lead to a panicked call from Storm to me to move the site over to my servers until the dispute could be resolved. A few days after that, I received a very aggressive e-mail from the designer threatening to sue me for stealing his work.

What iwasntpaid.com needs is links to conflict resolution and contract negotiation books, which are important freelancing skills that the designer sorely lacked. You never introduce yourself to a fellow freelance web programmer with "Hi, my name's Jeff, and I'm going to sue you over this dispute I have with another party".

Given the designer's history of posting complaints about how he's been screwed out of $500 and then taking them down once he realizes that it might be bad for his reputation, I wouldn't be surprised of that post about Storm gets taken down as well.

Also, from my web-stalking around this issue, I wouldn't be surprised if the hours that the designer put in trying to smear Storm across the internets, billed at the designer's normal rate, exceeded the payment he was expecting from Storm.

tl;dr: Settle on a rate before you start doing work, and if it's not a flat rate, keep the client up to date with projections of the final bill.

PS Carlos 'Storm' Martinez and his team at Creative Mixing are amazing, talented people and I would recommend them to anyone looking for post-sound work. It saddens me to see the post on iwasntpaid.com

<comment><USER14> >and the HTML didn't validate

Stupid as it is to go down the publicly naming route, what has the HTML validation status of the code got to do with anything? This is a measurement that only means something to guys who measure things - it has nothing to do with business generated, traffic, or the user experience of a website (the true yardsticks of good work).

<comment><USER13> True, it doesn't have anything to do with anything. It was just a pet peeve of mine.

We need to adhere to standards, dammit!

</comment></comment></comment><comment><USER15> One dangerous thing that stands out is this "kill switch". VERY bad idea. Now you are liable for damages.

The way to handle this is: When you get the judgement against your "client", slap a lean on his properties equal to the amount owed (and throw in your court costs). That inclues his accounts. All you need is an order from the judge. That's it.

That's about all you can do really. Is to go after his assets. Also I'd let the BBB and your local Chamber know. You'd be surprised how effective both these organizations can be.

<comment><USER16> I wonder if it's legal to state upfront in your contract that 'technical measures' are implemented in your work to prevent theft of service (or whatever you call the use of the work product without payment), and are to be deactivated & removed following completion/final payment?

The software licence-key approach seems somewhat similar, and I've yet to come across legal proceedings "because we didn't pay the guy and our 30-day demo ran out", although I'm sure it could get more complicated for services rather than actual shrinkwrapped products.

On the other hand, you've got the fact that it's laid out in the contract (which is presumed read, rather than just clicked through like a shrinkwrap EULA), and the interesting potential for DMCA/WIPO96'alike anti- circumventation proceedings against clients who refuse to pay and extract the kill-switch code to reactivate.

In that theme, is:

        final_invoice_paid = false;
        if (!final_invoice_paid) { exit() }

a valid "means to descramble a scrambled work, to decrypt an encrypted work, or otherwise to avoid, bypass, remove, deactivate, or impair a technological measure, without the authority of the copyright owner"?

My immediate thought is yes, assuming you're still the copyright holder (and you have a copyright assignment on payment clause in contract), but jurisdictional work-for-hire terms might screw you.

<comment><USER6> Reminds me of the talk "Fuck you, pay me", where they talk about how the client only gets copyright assignment upon reciept of final payment. Thus, if they stop/don't pay, then they do not have copyright on the work, and hence are guilty of copyright infringement if they use it.

<comment><USER17> It is always a good idea to keep ownership until final payment. This also increases the chance that you'll see money when a client goes bankrupt and is sold or undergoes a restructuring ("doorstart").

(nb: all based on NL law)

</comment></comment></comment></comment><comment><USER18> Similar situation, but closer to $9k. Filed a lawsuit in court - <USER2> waiting_ 3.5 years later (NYC). I suspect it'll take a long to collect even after a judgement.

Yes, don't extend so much credit - lesson learned (the hard way).

I consulted with a number of people debating whether or not to blog about this situation with the person's name. In the end I chose not to at that time, but realized later that this person had another non-payment suit pending which had been filed the day before I started working with him. Had I researched a bit more, I might have found that (but then again, I might not have - you often don't think to do court record searches on potential clients). I ended up _not_ disclosing his name in my blog, but did disclose the location I filed a suit. Anyone wanting to know can go check the public records and find out for themselves. Felt like a decent middle ground, and it's not irrevocably commiting myself to having been the "name and shamer". But without more naming and shaming, this behaviour by some people will continue. I found out later that this guy had also stiffed multiple other parties, who were then coming to me for advice about how to collect. "File a NY lawsuit" was the only answer I had.

Perhaps some public naming and shaming would either correct this person's behaviour or at least ward off others from working with him in the future. _one_ person complaining about no payment can be written off as a crank. 5 devs in a 5 month period, all from different areas, would be harder to dismiss.

<comment><USER19> Another middle ground would be to name and shame just to thing that are factually part of the public record: "I am suing John Smith in court for non- payment of $9k for work done 3.5 years ago". Don't go into fiery detail, keep it to the facts that are part of the court record.

Of course, the hard part is keeping all the opinion stuff out of the blog posts around the fact-only one.

<comment><USER18> right... well, that's almost what I did. saying where it was, letting someone google my name in nyc and look for court records. i've only sued one person in nyc - it wouldn't be that hard.

But, publicizing on my blog, yeah. In some ways I think the moment's passed. If/when we ever get to court, I'll release the name then, unless there's some gag order (am I being too dramatic?)

<comment><USER19> The difference is that if someone is googling John Smith, they won't find your blog post without his name on it.

Thankfully I've never had to experience court on either side - it looks like it gets expensive very very quickly.

<comment><USER18> I'm representing myself, so it was only the trip to NYC, filing fees, and hiring a processor. The processor could _not_ get this guy, so we filed a notice saying he'd attempted 3 times. I suspect this guy has avoided process servers before, but not sure.

Yes, the googling aspect doesn't help too much. Interestingly, having to remember back to 2008, this same situation happened with someone else they'd contracted. This particular contractor was in tennessee, and was owed ... much less than me. He blogged about it by name, and the NYC guy apparently (per the TN guy's story) went ballistic threatening to sue and such, and said he would pay when the post was taken down. So... it was taken down. Then never paid. The TN guy eventually gave up through these bullying tactics. I'm not going to give up.

It's so far in the past now that, while $9k would be great, I don't need it, and won't settle for half, nor will I settle for any conditions about not blogging about it.

</comment></comment></comment></comment></comment><comment><USER20> While I sometimes wish something like this existed, it seems ripe for abuse and/or open to some sort of libel law violation should it get out of control.

<comment><USER21> Isn't it already explicitly illegal to post lists of customers who haven't paid in your store? IANAL, but I seem to remember that the case law is already pretty settled on this one.

<comment><USER22> illegal where? Perhaps in some juridictions - which one are you referring to?

<comment><USER23> The difference will be whether its stated as factual and is factual or stated as opinion..

As I understand it if its factual stated and grossly wrong the poster can be charged civilly

<comment><USER24> Whether truth is a defence against libel also depends on the jurisdiction.

</comment></comment></comment></comment></comment><comment><USER25> Anonymous accusations, no verification and guaranteed reputational damage to those accused: This is a bad idea.

If you have legal claims, press charges instead of going down this road.

Edit: the first post, an anonymous accusation (by the site's creator?), disappeared shortly after the site showed up on HN. Also, now suddenly submittals are subject to review (by whom?).

</comment><comment><USER26> In general this sort of site gets sued a lot. For the exemplar site see RipOffReport which has almost more lawsuits than posts it seems. So if you run it for any length of time be sure to have a good legal team already in place.

The other thing is that there is an endless supply of bad business deals gone wrong so any individual deal that went south may make the person venting feel better (until they are sued) but they don't contribute to the overall success rate of other folks (except perhaps if they are also dealing with a specific individual or business mentioned).

Contrast this with the Nolopress (www.nolo.com) guys who, by example, provide both tools to successfully navigate what may be new territory and provide suitably anonymized examples which demonstrate the problems that you want to prevent. That is a positive outcome to a bad experience, which is shared learning and signs of things to avoid.

<comment><USER27> I worked on a similar project a while back and definitely got legal threats constantly. Not one ever followed through but I grew tired after a couple years of dealing with it.

</comment></comment><comment><USER28> ZenCash is new startup that is solving the problem the right way. It can connect with your invoicing service (FreshBooks, Harvest, Blinksale) and then they will contact the client when the refuse to pay. They work with a collection agency so now we the lone web developer can actually force someone to pay.

<http://www.zencash.com/>

</comment><comment><USER29> Not the best route to handle this. Make sure you have a decent lawyer on hand (and a good contrct) when dealing with delinquent clients. Something like this will guarantee that you'll never get paid.

<comment><USER30> At what point does it make sense to involve a lawyer? $5,000? $10,000? Isn't it expensive to legally fight for your payment?

<comment><USER31> Stipulate in your contract that the losing side has to pay the legal fees for the other party. That way getting paid doesn't cost you more.

<comment><USER32> I used to work for a company that put a late-payment clause right in the contract. If you didn't pay us you were on the hook for a fee plus interest on the outstanding balance. So essentially it was like owing money to the IRS.

As far as I know nobody ever gave us trouble about payment.

<comment><USER33> > I used to work for a company that put a late-payment clause right in the > contract. If you didn't pay us you were on the hook for a fee plus interest > on the outstanding balance.

This is so common in the business world -- terms -- that there's a short hand minilanguage for describing your terms on invoices, purchase orders and contracts.

Here's a decent list: [http://www.businesslink.gov.uk/bdotg/action/detail?itemId=10...](http://www.businesslink.gov.uk/bdotg/action/detail?itemId=1073791761&type=RESOURCES)

</comment></comment></comment></comment></comment><comment><USER34> Actually collecting isn't that hard (in the U.S., at least). I won a similar case in small claims and the defendant didn't pay. All I had to do was file a form for the Sheriff's office to collect from him and pay the filing fee (I think around $45, which was then added to the amount to collect). I provided the defendant's bank account info, and when he was unresponsive to the police, they withdrew the amount from his account into an escrow account and then cut me a check for the amount owed and mailed it to me.

This can probably be a little more challenging if the account doesn't have sufficient funds in it, but I was happy to see that the system worked. It was a learning process for me (first time I had to do it, and thankfully, knock on wood, the last), but in the end, justice was done and I got paid.

</comment><comment><USER35> This is basically a rite of passage for all freelancers/lawyers/service workers unfortunately. My rule of thumb: If someone doesn't want to give you at least 20% upfront for a project, you don't want to work with them. Always give yourself checkpoints in the project for partial payments instead of a lump sum at the end.

</comment><comment><USER9> A much more useful approach would be to educate people about protecting themselves from non-payers. Eg, use an agreement or contract, check references, etc. That would be far more beneficial than a collection of libel and rambling.

For instance, the entry from Jan 31 says the person worked for _four_ months without being paid! Why would you work for four months straight without a single intermitent payment (or an advance)?

</comment><comment><USER36> I agree with others on the legal view, but also, to me this site just focuses on sour grapes. As a result, I can't see anything positive of helpful coming out of it. I'm not going to search a blog each time I get a client to see if they've been written about here. And aside from that, this site will only ever tell one-side of the case.

What if the developer did really poor work, missed every deadline by weeks, or the client still says there's work to be done before being paid? All this site shows is angry developers complaining about less-than-ideal clients. For me, that's the kind of talk you have over beers at a bar — you say your piece and then you move on — not something you permanently record into a collection on the web.

<comment><USER37> > What if the developer did really poor work, missed every deadline by weeks, > or the client still says there's work to be done before being paid?

Then make a place for that kind of post too. Perhaps even create an index that correlates the two accounts, so that people can see both sides of each story, and weigh the evidence for themselves.

</comment></comment><comment><USER38> This fantastic speech by Mike Monteiro ( <http://vimeo.com/22053820> ) explains how you can avoid ever having to use this site as a last resort to get your grievances resolved.

<comment><USER31> This is an absolutely brilliant talk. And if you do any freelance or contract work you can learn something from it.

The quote: 'when you hire a lawyer [to handle contracts] you switch from being a design amateur to a design professional' really spoke to me, and it applies equally to programmers.

<comment><USER39> I was about to watch it but the page starts out with an F Bomb, if I wanted to aspire to that level of communication I could have stayed in the Navy.

<comment><USER40> You'd rather not hear profanity than learn how someone managed to eradicate basically all of their client-payment problems?

</comment></comment></comment></comment><comment><USER41> Everyone here is saying it, anonymously calling others out by name is wrong and profesionally destructive.

This is why we pay lawyers: [http://www.dvafoto.com/2011/04/fuck-you-pay-me-a- discussion-...](http://www.dvafoto.com/2011/04/fuck-you-pay-me-a-discussion- of-adventures-in-contracts-negotiation-and-payment/)

</comment><comment><USER42> My Dad had a folder labeled 'Deadbeats' at this small consulting firm. It was in the conference room - anybody was free to look at it. In the field he was in, word got around quickly so there was a few tardy clients that paid but stipulated that he remove their transgressions from the 'Deadbeats' folder.

<comment><USER33> Sounds like he also needed a "Deadbeats Who Eventually Paid Up" folder as well.

</comment></comment><comment><USER43> We are currently bootstrapping our startup with some consulting work in iOS and Rails. This is the process that we follow:

1) We discuss the project with the client stating our hourly rates clearly from the beginning and that we also work in our startup so we won't be a "full-time" developer 2) We then make a clear estimation of time and deliverables for the project + time for testing/debugging/QA + time for calls + time to prepare for publication to the App Store. 3) If they accept we make a clear contract with several payments and deliverables. 4) We ask for 20% to 40% upfront payment and then the rest divided by time and deliverable completion. 5) The last payment is conditioned to App Store approval (in case it is iOS) 6) We only give the source code after the last payment is done 7) We keep sending weekly updates usually via TestFlight (when iOS) and use a Google Doc for any observations that they have 8) We have discovered that some clients also want time spent so we always use toggl as time tracking tool.

I know there are still risks but at least with the upfront payment and deliverable payments you always keep healthy cashflow, the clients feel they are protected and you feel protected too.

This way it's also easier to detect a red flag, in that case you can just stop your development until the payment is done (We actually did that once now and it worked really well).

<comment><USER44> I completely agree.

That and a (very) detailed list of requirement and goals is the best way to go.

</comment></comment><comment><USER45> Upon reading this article I seem to think that in most cases it's actually the developers fault. Don't be retarded; create a contract for the work you do with your potential employer.

I've been doing freelance for quite some time; and did have two clients try to screw me over; however due to the contracts which I setup ahead of time, and having a lawyer send a simple note upon their threat of not paying, quickly remedied this non-paying issue.

1) Cover your ass 2) Setup a reasonable Payment schedule 3) Don't be stupid

</comment><comment><USER46> > People who feel that they have been wrongly accused can now send in feedback > that they have been wronged and we will look into it

That sounds like a bad idea. You either publish submissions through the website, or you don't publish at all. If you manually take the 'feedback' and turn it into a story, you're now the publisher and lose the legal protections of the Communications Decency Act.

Your site was a better idea, legally, before you posted that.

</comment><comment> e-dard I'd do something about the kerning of the h1 element, it's not particularly pleasant in my opinion.

</comment><comment><USER47> They should force the developers to put their names. That way people know which developers to stay far away from. Don't go near anyone that would post to something like this.

<comment><AUTHOR> thanks, we have started to enforce it alr :)

</comment></comment><comment> k-mcgrady I just read the first story on the site. Taking someone to court over $1000 is a waste of time. $1000 may be a lot of money to you (it is to me) but I have faced this problem twice and you are much better spending your time on new projects and forgetting the client.

I have found the key to getting paid is milestones. I will not work with someone unless they are willing to pay 50% of the total before I start. This way if they cancel the project you are getting some money for your time. I also make clear to the client they do not need to pay the final 50% until they have the project from me. This way they need to trust me enough to pay the first 50% and I need to trust them that they will pay the final 50%. I've been using this system for the past 12 months and haven't had any problems with clients since.

There is a good thread on Programmers Exchange about charging clients (particularly for freelancers): [http://programmers.stackexchange.com/questions/63042/recomme...](http://programmers.stackexchange.com/questions/63042/recommended- payment-schedule-for-freelance-development/63062#63062)

</comment><comment><USER48> I think this issue is very significant these days, with the boom of outsourcing and remote freelance work. For scenarios with no down payment requirement, there's no guarantee the service provider will receive payment at all (and no guarantee the rest of the payment will be paid for the down payment scenario either). And for scenarios where a down payment (or full payment) is paid, there's no guarantee the service will ever be delivered to the requester. And how about scenarios where the wrong service was delivered? And how about if the service provider and requester are not in the same country (how do you go about taking legal action)?

I am a co-founder of a nascent third party payments aggregator (TPPA) called PayGuard. We are currently building our Beta, but our system specifically targets such scenarios. And with such capability, our system inherently solves the same issues with material products as well (a market PayPal currently dominates, though it does so extremely unsatisfactorily).

</comment><comment><USER49> One of the big problems with web consulting and clients that don't pay is that unlike other business situations, these clients often lack physical assets and storefronts that can be used to collect on a judgement from a court. Traditionally, if you had a small claims judgement and it's not paid, you could have the Sheriff or the like serve notice or even put a lien on assets.

When a client's business is entirely virtual, there's a risk they'll simply change their mailing address or otherwise dodge a judgement, and it can be very difficult to enforce the claim (or at least not economic to do so.) I've seen this happen a number of times in recent years in situations where the amount ranged from $1500 to $10,000.

</comment><comment><USER31> Ultimately it's your fault as a contractor if you don't get paid. And a site like this does nothing to teach developers, freelances, designers, and the like how to avoid these pitfalls which is why I'll never use it.

Why is it your fault? Because _as a professional_ you've not taken the time/money to have a contract created to cover your ass. Contracts exist to protect both parties from each other. And it is your professional duty to know what's in your contract (not a piece meal that you dug up on the internet.) You should be able to explain to your clients what is in your contract, why it's there, and how it benefits them and you.

<comment><USER44> Contract are only really usefull for big company and with big accounts.

For private freelancers and small/medium jobs suing someone for breach of contract will probably not be a viable option.

The amount of time, effort and money that you need to sue someone and win sometimes is seample not feasible.

Yes, having a contract written by a laywer may work as a deterrent for some clients. But if they don't accept you still need to file a lawsuit.

And there is also the problem that sometimes they simple can't pay. They look and act like they have all the money in the world, but they are actually betting all they have in a website because "making a website is easy and I will make millions with my great idea".

Then even if you win they are simply going bankrupt and you will never see nor your money nor the ones that you spent on lawyers.

<comment><USER50> It is foolish to do "contract work" with no contract, period.

The other key deterrent is to keep a handle on your deliverables until the client pays you. That does not mean "kill switch" in an app running on someone else's servers, it means (whenever possible) _don't put the deliverables on someone else's servers until they pay you_.

If the work depends on their pre-existing code or assets, get the dependencies into your private and properly secured development environment (with a tightly scoped license and NDA to protect the client).

And make sure your contract says that until they pay you, everything you create is your property. You can't claw fees out of a bankrupt client, but you can at least keep exclusive rights to your work.

</comment></comment></comment><comment><USER51> This reminds me of ol' F'd Company. Can't be sued for libel if the story is true.

<comment><USER52> Yes, you can. You can't _lose_ a libel suit if you're able to prove what you said is true, and it goes to decision.

I am not being pedantic. The difference between the two statements is huge, because the amount of time and money it may take you to prove your case could easily bankrupt a person.

<comment><USER51> Good point.

</comment></comment></comment><comment><USER53> How about instead, we have a pool of customers which are GOOD CUSTOMERS but which you in the future might want to move on from, for various reasons? Other developers can pick up the slack -- and some work! They would also be able to have a transition period. This website could even make a small commission from the handoffs.

The best way to avoid bad options is knowing where to find a lot of good ones. Then the others have to prove themselves to you, and you can take them or leave them :)

</comment><comment><USER30> How about making it a paid subscription service instead? I would subscribe to that.

Developers have their identity confirmed (to avoid anonymous business-bashing) and can freely discuss bad (and possibly good) experiences with businesses.

The membership agreement could explain that no emotional or biased comments would be allowed and would encourage some level of explanation of why the client did not pay along with proof in the form of bounced checks, etc...

</comment><comment><USER54> In an ideal world this is great but anonymously posted data only could increase problems with people posting one or many stories about others they just want to harm for personal reasons.

While self naming probably increases the chance of legal issues I think people should have the choice to say I am the person he did this to to help credibility. And if one have it all documented there should be nothing to be worried about.

<comment><USER55> It strikes me that the anonymity is mostly pointless here. It may be meant to protect the accuser of taking a hit to their own reputation, but the accused can just post the accuser's name and their own side of the story in comments or on another site. So in the long run, there wouldn't be much benefit to posting anonymously.

We had a discussion about this last weekend at She's Geeky. There are some women's lists that pass around information like, say, managers at {big company} who are known harassers and should be avoided. If we stick to backchannels to share this type of information, we aren't helping, perhaps, the very people (new to a company or a scene) who need it most. If we put the information in public, we suffer the blowback on many fronts; and our vulnerability is greater than that of the accused. I tend to come down on the side of greater transparency, but it may be that the most privileged person still "wins" -- except, as someone pointed out earlier, when 5 developers report similar experiences.

</comment></comment><comment><USER56> This is what nasty bad ass debt collectors are for. They'll collect (really they will) and then take a cut. Carlos won't know what hit him. Get some recommendations for good debt collectors in your local area. You are wasting your precious time trying to collect this yourself. Employ them and then you can inform Carlos that it is longer his debt to you but his debt to some burly tattooed biker.

</comment><comment><USER57> I have started to put a clause in my own contracts that's paraphrased like so: I do some work for you and submit it to you for verification. You have x days to verify it. If you fail to verify it or tell me a scheduled date for verification by that date, it goes on github for the world to see and use for free. I.E. "F __* U, Pay Me."

<comment><USER58> Interesting approach, but what stops the client from using this to his advantage? "Hmm, if I fail to pay, the code becomes open-source, and then I can use it for free."

<comment><USER57> Depends on how much they want to deter competition. This particular company was a hardware video switch; competitors could easily port the same app to their machines and then the client loses the advantage they had previously.

</comment></comment></comment><comment><USER59> I recommend to ask for payment at pre-defined milestones. Waiting to get paid at the very end is just asking for trouble.

</comment><comment><USER60> This strikes me as an exercise in futility.

If you can properly document that you should have been paid and haven't, go to court. Posting on a website won't do anything to actually recoup the cost.

If you _can't_ prove that, you're setting yourself up for a libel claim.

So what exactly is the gain of posting there, except giving in to a temporary desire for revenge?

</comment><comment><USER61> Anyone know what indian libel laws are like?

</comment><comment><USER62> How long before this gets sued for libel?

<comment><USER46> The site wouldn't be liable. A lawyer would tell his client that.

[http://en.wikipedia.org/wiki/Section_230_of_the_Communicatio...](http://en.wikipedia.org/wiki/Section_230_of_the_Communications_Decency_Act)

Anyone could post libelous statements on the 50+ million blogs and forums out there, yet you've never seen "Warn HN: Monitor your blog comments for libel" -- that's because you can't be treated as the publisher or speaker, so can't be held liable for what was said.

<comment><USER63> This, and the fact that the truth can not be libelous. As long as you're not misrepresenting a claim, you're fine.

<comment><USER17> That depends on your location. In NL the truth can get you in a lot of trouble.

Also not that the TOS tries to make the submitter liable for all damages _and_ cost : <http://iwasntpaid.com/terms_of_submission>

<comment><USER63> If you stole money from me, I sued you and won the lawsuit, and then published an article about how you stole from me, that could be construed as being libelous?

<comment><USER17> No, because our law separates defamation into two parts.

\- Libel (laster) where allegations are not true or the defendant should have known they are not true.

\- Smaad where allegations are true, or it's reasonable for the defendant to assume they are true.

In your example, you could be convicted of Smaad, unless you could show that you

1) acted in the public interest.

2) or needed to defend yourself against eg a counterclaim

3) or did not intend to harm someones good name

See <http://lexius.nl/wetboek-van-strafrecht/boek2/titel16>

<comment><USER63> Interesting. Thanks for the explanation.

</comment></comment></comment></comment></comment></comment></comment><comment><USER64> Well, it's a nice idea but I think this is high risk. In the very distant past when I was young and naive there were occasions on which I didn't get paid for commercial work, so I can sympathize with that. However, naming and shaming rogue customers could just be a recipe for libel lawsuits.

</comment><comment><USER65> on the iwasntpaid.com site:

I was eager that the service would let me search for the client names and get reviews on them quickly. And providing a very basic template for each story for the author to enter metadata around the story. I think that would be a much better experience for readers and contributors.

<comment><USER66> > I was eager that the service would let me search for the client names and > get reviews on them quickly.

That's what I was expecting -- I tried to search for "Martinez" and "Padbury"; two of the names prominently featured in the front page stories right now, and got a "0 results" page for both terms.

</comment></comment><comment><USER67> How about freelancers who never complete work but still get paid? it happened to me.

</comment><comment><USER68> just for argument sake: 1 - freelancers union is working very hard to pass the Freelancer Payment Protection Act (S4129/A6698) that grant freelancers the same wage protection as traditional employees and require the Department of Labor to pursue freelancers' unpaid wages holding deadbeat executives personally liable for up to $20,000 and jail time. 2 - on the union website you can find the Contract Creator, a free tool that allows you to easily create a strong (and nationally accepted) contract to protect yourself. my2cents

</comment><comment><USER69> Just build a kill switch into the code. If they don't pay, use it.

</comment><comment><USER70> The lesson here is never piss off someone with a blog and an audience. His name will forever show up in Google as the guy who didn't pay up.</comment>


Ask HN: How to get Ads fit the Content? - <AUTHOR>

I use Google AdSense to display ads on my web-app (dezidr.com). Unfortunately, the ads don't really fit the contents. Take this as an example: http://dezidr.com/en/home/show_option/295.<p>Do you know how to get better ads? Is there a way to optimize AdSense? Are there ad-services with better targeting? Thanks for any idea! <comment><USER2> Take a look at [https://www.google.com/adsense/support/bin/answer.py?answer=...](https://www.google.com/adsense/support/bin/answer.py?answer=23168&topic=8441) Maybe it helps.

</comment><comment><USER3> I'd like some info on this too.</comment>


Apply HN: Horde.tv – Live stream any show or YouTube video - <AUTHOR> prototype: http://www.horde.tv
15 second gif walkthrough: http://i.imgur.com/7AGRMnF.gifv
Why:
Live streaming is everywhere (Livecoding, Meerkat, etc), and the interactive experience of watching live + realtime chat is great. But all these apps focus on live broadcasts when there are so many existing videos that would be exciting to watch with a live audience (ie: Bob Ross on Twitch). Not to mention, these existing videos tend to have much higher production value, compared to some live streams on periscope and others.
We've created a site where anyone can create a live channel of their favorite videos. Everyone is synced to the same second of the video as it plays so you can chat in realtime. Videos are embedded directly from YouTube and Vimeo (original creators still benefit from YouTube ads and revenue).
Growth:
We've partnered with a handful of popular YouTubers to create their own live channels from their YouTube videos. Many YouTubers love the idea of increasing viewership and having a chance to showcase their videos to a live audience.
Who's behind it:
We're a team of two engineers who have worked on several apps together. Hobby YouTuber with over 15 million views.
Progress:
We've built the prototype and are launching it for the first time right now. Neither of us are UI designers so please excuse the mess :) we're open to comments & suggestions! Or you can email us: firstname.lastname@example.com
TLDR: Horde lets you watch youtube videos and tv shows like you're watching a tv channel (with realtime chat). <comment><AUTHOR> We've opened up access so anyone can create a channel. It's pretty easy: just pick a channel name and add videos.

Here are some live channels we created from our favorite shows:

VSauce, Numberphile, Veritasium: [http://www.horde.tv/channels/mind-blowing- science](http://www.horde.tv/channels/mind-blowing-science)

Interviews from Startup School: [http://www.horde.tv/channels/startup- interviews](http://www.horde.tv/channels/startup-interviews)

TEDx: [http://www.horde.tv/channels/ted- talks](http://www.horde.tv/channels/ted-talks)

</comment><comment><USER2> This sounds really fun - obviously Twitch's Bob Ross stream was an incredible experience. (I work at Twitch)

Do you think that youtube will provide enough engaging content for people to start getting into the platform? How could you go about getting 'real' popular content onto the platform, and if this works how can you compete with Twitch?

<comment><AUTHOR> Awesome to hear feedback from someone at Twitch - to answer your questions:

\- Absolutely. As the largest video platform, there's a ton of great content produced on YouTube everyday. Not to mention, a growing number of legally available shows (including Bob Ross!) on YT. YT is also just a starting point.

\- We're working with several notable YT creators to bring them in as partners. In comparison to Twitch, we're focused on pre-existing content that can be watched as live channels.

</comment></comment><comment><USER3> The concept is neat, had a chance to view briefly. Can you comment on any 'gatekeeping' type considerations for age/region/content in regards to either the videos being broadcast and/or the comments being written? In an ideal world I wouldn't be asking such things, but in context of the internet at large, I think it'd be neat to hear your perspective(s) and potential avenues to address them.

<comment><AUTHOR> Currently channels are meant to be available to everyone with no age/content restrictions from our part. Any inappropriate content will be flagged and removed.

However, since videos are sourced from platforms like YouTube and Vimeo, they may impose their own restrictions such as unavailability in certain regions. We've started adding more video platforms to ensure there's content available for all regions.

</comment></comment><comment><USER4> As a fan of Vsauce, CGP, and several conference channels (JSConf) this looks promising. I definitely see the value in having live chat alongside the video.

But instead of YouTube videos, I think this is a better fit for watching shows like Netflix/Hulu live (not sure if they allow embedding). I regularly tune into new episodes on Hulu - this would fit perfectly.

<comment><AUTHOR> Thanks for the feedback. We're starting with YouTube & Vimeo videos because there's an endless pool of great content to start with and embedding is built- in. Not to mention, content creators continue to benefit from ad revenue.

We'd love to add Netflix shows for live streaming in the future. We've had a lot of people suggest the same.

</comment></comment><comment><USER5> Can this be used for broadcasting small events live to people who couldn't attend physically? Although I'm not sure realtime chat will be of much use there.

<comment><AUTHOR> Yes, we even had a stream go up yesterday for Kobe's final NBA game. Realtime chat was fun to watch as the game unfolded.

</comment></comment><comment><USER6> Cool idea though I'm not seeing the chat on mobile

<comment><AUTHOR> The realtime chat is disabled on small mobile screens because there just isn't a lot of space. The chat is enabled for tablets and larger screens.

</comment></comment><comment><USER7> Not a fan of the lack of volume controls

<comment><AUTHOR> We're still figuring out which direction to go one several features. We disabled the controls bar (incl. vol) so each channel feels more like a live stream where you can't pause/skip through the video. However, you can still use the keyboard to control volume (up/down keys)

But we're going to give it try: we've re-enabled the control bar. Thanks for the feedback

<comment><USER8> Be very careful about adding/removing features based on one comment in a forum.

<comment><AUTHOR> I agree. However, we're open to trying suggestions that are reasonable and feasible for us to implement without tying up too much time. His/her suggestion took less than 30 seconds and has a relatively minor impact on the site. We'll change it back if we don't feel its a good fit.</comment></comment></comment></comment>


Web Scraping: Bypassing "403 Forbidden," captchas, and more - <AUTHOR> http://sangaline.com/post/advanced-web-scraping-tutorial/ <comment><USER2> Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

<comment><USER3> How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?

<comment><USER4> You reverse engineer the application, or you run it in a debugger.

If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions.

If it features HMAC signing, you decompile the app, find the key, reverse engineer the algorithm that chooses and sorts the paramaters for the HMAC function, and rewrite it outside the app. If the key is generated dynamically you reverse engineer that too, and if it's retrieved from native .so files you're going to have a fun time, but it's still quite doable.

All they can do is pile on layers and layers of abstraction to make it painful. They can't make the private API truly private if it requires something shipped with the client.

<comment><USER5> The initial idea was to make your life simpler by parsing JSON instead of HTML. Now we are decompiling binaries. Somewhere on the way, we got lost.

<comment><USER6> Once you do the one-time work of pulling out the key, you can just add something like, "secret_key=foobar" to your requests, and you're back to happily parsing JSON.

If they keep changing it up, I'm sure you could automate the decompiling process. The reality is that this technique is security by obscurity at its core, and is therefore never going to succeed.

<comment><USER7> Skype is probably one example where it took developers 10+ years to figure out how the app worked.

<comment><USER8> Did it take that long to do it or did it take that long for someone to care to do it? I mean, it's Skype.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Better solution: pay target-site.com to start building an API for you.

Pros:

* You'll be working with them rather than against them.

* Your solution will be far more robust.

* It'll be way cheaper, supposing you account for the ongoing maintenance costs of your fragile scraper.

* You're eliminating the possibility that you'll have to deal with legal antagonism

* Good anti-scraper defenses are far more sophisticated than what the author dealt with. As a trivial example: he didn't even verify that he was following links that would be visible!

Cons:

* Possible that target-site.com's owners will tell you to get lost, or they are simply unreachable.

* Your competitors will gain access to the API methods you funded, perhaps giving them insight into why that data is valuable.

Alternative better solution for small one-off data collection needs: contract a low-income person to just manually download the data you need with a normal web browser. Provide a JS bookmarklet to speed their process if the data set is a bit too big for that.

<comment><USER10> > pay target-site.com to start building an API for you.

When has that ever worked?

<comment><USER11> I can't cite specific examples because the ones I know about formed confidential business relationships, but I can say with confidence that this works All. The. Time.

That said, if you're some small-time researcher who can't offer a compelling business case to make this happen, then it won't be worth their time and they're likely to show you the door. [Note: my implication here is that it's not because you're small time, but it's because by the nature of your work you're not focusing on business drivers which are meaningful to the company/org you're propositioning].

Edit: Also be warned that if you're building a successful business on scraped personal info, you're _begging_ to be served w/ a class action lawsuit (though take that well-salted, because IANAL and all that jazz).

</comment></comment></comment><comment><USER12> Scrapy is indeed excellent. One feature that I really like is Scrapy Shell [1].

It allows to run and debug the scraping code without running the spider, right from the CLI.

I use it extensively to test that my selectors (both CSS and XPATH) are returning the proper data on a test URL.

[1] [https://doc.scrapy.org/en/latest/topics/shell.html](https://doc.scrapy.org/en/latest/topics/shell.html)

<comment><USER13> A few things turn me off about Scrapy is that it feels over engineered for what it does. Why do I need an entire framework?

I'm taking on technical debt to access data I don't have programmatic access to.

CSS/Xpath are very fragile. You most likely will be changing them in the future.

<comment><USER14> > _CSS /Xpath are very fragile. You most likely will be changing them in the > future._

Genuinely curious what the alternative is

<comment><USER13> I've been doing research on this but it's not clear whether this problem is a pain for enough number of businesses to justify further investments.

I often feel like web scraping is a commodity without understanding any of the inherent technological complexities and challenges.

Very discouraging field to be in, especially when people claim to have pain but are unwilling to pay very much for it or show appreciation for the effort that goes into it.

edit: thanks for the downvotes. perfect illustration of how innovation is punished and unrewarded in this field.

<comment><USER15> FYI, I only downvoted you after you complained about downvotes.

<comment><USER16> I only downvoted because there was no alternative offered, just complaining about how underappreciated scraper creators are.

 _> perfect illustration of how innovation is punished_

I see no innovation in your post.

The complaining about downvotes was just the icing on the cake, cementing the downvote.

</comment></comment></comment></comment></comment></comment><comment><USER15> Here's an idea (although probably an unpopular one around here): if a site is responding to your scraping attempts with 403s -- a.k.a. "Forbidden" \-- stop what you're doing and go away.

<comment><USER17> This is a very obvious thing to say. Perhaps it's needed to be said, I don't know -- It's just a very obvious counter.

<comment><USER15> Yeah, it _seemed_ obvious... but judging by all of the comments here on how to "bypass" 403s, it actually wasn't obvious at all.

<comment><USER17> I meant it is obvious in that everyone knows that. But they'll still want to bypass it. So everyone is completely aware of it.

</comment></comment></comment></comment><comment><USER18> The web scraping tool of my choice still has to be WWW::Mechanize for Perl.

P.S. I wrote a WWW::Mechanize::Query ext for it so that it supports css selectors etc if anyone is interested. It's on cpan.

<comment><USER19> Same in Ruby land. That with inspector gadget and you're golden.

</comment></comment><comment><USER20> I have done a lot of scraping in Python with requests and lxml and never really understood what scrapy offers beyond that. What are the main features that can't be easily implemented manually?

<comment><USER21> Pluggable parsers, automatically good error handling and spidering functionality (finding and queueing new links to scrape), great logging, progress stats, exports, pause/resume functionality, and a million other goodies that are seemingly "trivial" but really you don't want to rewrite them every time you write a scraper.

edit: Especially if your scraping jobs take a LONG time - days and weeks, this stuff is extra handy. Might I add a great debugging environment (scrapy shell), error handling, rate limiting, respecting robots.txt, so much more.

<comment><USER22> How much benefit does the spidering/progress/pause/resume functionality give if you're not just spidering every link on the site, but have complex logic to determine exactly which links to crawl and in what order? Does Scrapy provide convenient extension hooks to change the crawl algorithm?

<comment> un-devmox I haven't had the need to use pause/resume but I do incorporate logic (not necessarily all that complex) in determining which links to crawl. It is very easy to do within each spider especially with how the framework uses generators. It is also easy to extend the pipelines for pre and post processing.

As others have said, managing a project with Scrapy is super easy and highly configurable with sane out of the box settings.

</comment></comment></comment></comment><comment><USER23> I'm curious what others use to scrape modern (javascript based) web applications.

The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work.

For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved from the server.

This worked fine but required some effort to implement. Is there an open source solution which is as helpful as Scrapy but solves the issues provided by modern javascript websites/applications?

With tools like Chrome headless this should now be possible, right?

<comment><USER24> Disclaimer: I'm a co-founder of Apifier [1].

It's not an open source, but free up to 10k pages per month. And it can handle modern JS web applications (your code runs in a context of crawled page). You can for example scrape API key at first and then use internal AJAX calls.

There's also a community page [2] where you can find and use crawlers made by other users.

[1] [https://www.apifier.com](https://www.apifier.com) [2] [https://www.apifier.com/community/crawlers](https://www.apifier.com/community/crawlers)

<comment><USER13> interesting. are you seeing any product/market fit for this?

<comment><USER24> We see a lot of users who needs data from the web or APIs for sites which doesn't have one. Just not all of them can code and we have to scale custom development.

<comment><USER13> Are these developers? Business people? I'm curious because we've been searching for a tool like this for a while but ultimately management thought it was a bad idea to rely on scraping, there's simply no replacement for a REST api.

<comment><USER24> Both - developers on a free plan using own RSS for sites without one and business people (mainly startups) building their products on top of Apifier.

Typical use is an aggregator that needs common API for all partners who are not able to provide it. So they have running API on Apifier in an hour. It might break once in a while - than you have to update your crawler (not that often if you use internal AJAX calls).

<comment><USER13> I see, so there's not much value beyond startups and bootstrappers.

I feel like it's a hard sell to enterprises. Scraping is viewed inferior to an API so it makes sense for enterprises to just pay the target website for access to the data.

<comment><USER24> It's also hard to get direct access to the data.

But you're right it's a hard sell to enterprises although we have some (e.g. real estate developer creating pricing maps)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER25> I use greasemonkey on firefox. Recently, I have written a crawler for a major accomondation listing website in Copenhagen. Guess what? I got a place to live right in the center in 2 weeks. I love SCRAPERS I love CRAWLERS.

<comment><USER26> Well the problem is when someone scrapes ALL the good listings then pre- purchases them for resale at double the cost.

<comment><USER27> How is it different than paying 50+ low-wage remote workers to "scrape" the phonebook for you and then using the information acquired for profit?

<comment><USER28> One difference is that Feist v. Rural Telephone says that the data in a phonebook can't be copyrighted.

[https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R...](https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._Rural_Telephone_Service_Co).

<comment><USER27> What about using those employees to "crawl" the web for you then?

<comment><USER28> I suspect it's roughly the same as a crawer -- same issues of fair use, TOS/CFAA, etc -- but likely there's no expectation that humans will read and follow robots.txt.

</comment></comment></comment></comment></comment></comment><comment><USER29> I use Java with simple task queue and multiple worker threads (scrapy is only singlethreaded, although uses async I/O). Failed tasks are collected into second queue and restarted when needed. Used Jsoup[1] for parsing, proxychains and HAproxy + tor [2] for distributing across multiple IPs.

[1] [https://jsoup.org/](https://jsoup.org/) [2] [https://github.com/mattes/rotating-proxy](https://github.com/mattes/rotating- proxy)

<comment><USER29> Hardest part was synchronization

\- to end the main thread only if all tasks are done

\- when every running task can produce multiple new tasks

\- with limiting the maximum number of running threads

\- always running the maximum nubmer of threads if possible

semaphores to the rescue

<comment><USER30> Doesn't ThreadPoolExecutor take care of all of that if you store the returned Future from the submit method? Then you just have the main thread wait for those.

</comment></comment></comment><comment><USER31> Note that in some places this constitutes breaking the law.

<comment><USER32> How is this any different from Google doing it? It is okay for Google to crawl the Internet, but not okay to crawl Google Play? Google raising such an objection would be an ultimate irony.

Edit: On second thought, I guess you are referring to overcoming 403s and Captchas?

<comment><USER33> Unauthorized access, if you access the service in violation of their TOS then potentially they have a case against you. I'm not aware of it ever going to court in a case where they didn't also send a cease and desist.

<comment><USER34> I'm afraid that as long as explicit agreement is not required to make a TOS binding we'll be dealing with this crap.

<comment><USER35> Yes, but see _QVC v. Resultly_ , where the robots.txt was considered binding, not the human-readable TOS.

We are getting small wins, but it's going to be slow going until we can get Congress to adjust both the CFAA and the Copyright Act, or until we can get SCOTUS to seriously alter the way these acts have been interpreted with reference to internet access.

<comment><USER13> Not quite. [https://www.law360.com/articles/757906/qvc-website-crash- sui...](https://www.law360.com/articles/757906/qvc-website-crash-suit-against- resultly-gets-trimmed)

Basically, all damage claims are null because QVC & Resultly never entered into a mutual agreement. You can write whatever the fuck you want in your ToS but it's not law binding.

> Judge Beetlestone also rejected QVC's claims that Resultly violated the > Computer Fraud and Abuse Act by knowingly and intentionally harming the > retailer when Resultly caused the shopping network's website to crash, > reasoning that the tech company and QVC both could only earn money if the > site was operational.

I see you are back on the FUD train surrounding web scraping but there's only very specific case where your fears materialize: "When you receive C&D from said website, do not continue scraping". Such was the case for Craigslist vs 3Taps.

Please do not cite legal resources and grossly twist realities to spread FUD. If you don't want to be web scraped, simply do not put it online.

<comment><USER35> >Basically, all damage claims are null because QVC & Resultly never entered into a mutual agreement. You can write whatever the fuck you want in your ToS but it's not law binding.

I'm pretty sure that's what I said re: the ToS? That's only _one_ element of the case (breach of contract). You are correct that in this case, browsewrap was not considered applicable. There have been a few other cases where it wasn't too, as in _Nguyen v. Barnes & Noble, Inc._, but there have been cases where it _was_ , as in _Hubbert v. Dell Corp._. Also note that most cases re: browsewrap do not challenge the viability of automatically entering an agreement by clicking around the site, but rather argue that the notification was simply not prominent enough. It could be worked around by moving the notice into a more prominent location on the page.

The other element is CFAA, and referring to the wide-open robots.txt helped Resultly establish that they were attempting to act in good faith and were not maliciously damaging QVC's systems and not exceeding authorized access to the computer system.

>Please do not cite legal resources and grossly twist realities to spread FUD. If you don't want to be web scraped, simply do not put it online.

I'm not trying to spread FUD, I'm just trying to make it clear that the legal situation is precarious. Google has clearly shown that if you are able to build your coffers and reputation faster than you can incur lawsuits, you can win on this. In fact, lots of big companies begin that way, and become big companies merely because they were lucky enough to get big enough to stand up for themselves before the legal threats started coming in the door.

It's understood that you'll be scraped if you put it online. That doesn't mean scraping is legal.

You may be confused here -- I'm not a publisher trying to stop people from doing this. I'm an entrepreneur whose business depended on scraping data from a specific source. That business got destroyed when they chose to dispatch their law firm against us.

The point of repeatedly discussing this on HN is to make the legal situation clear so that people work to change it, and to make sure people who are going into similar ventures are informed about the legal risks associated with them.

As I said on another post, I am not a lawyer, and this is according to my layman's understanding. No one should misinterpret my posts as legal advice. I'm not going to copy and paste this disclaimer into every post I make because it should be implicit, and a few IANAL disclaimers is plenty.

<comment><USER13> Well the difference is pretty clear between your case and the rest. Developers scraping a website isn't going anywhere. A business reliant on scraped data is making money off of it. That will lend you in precarious situation more.

My criticism was that you mixed in service providers and tool providers that enable businesses to make money off scraped data-the vendor cannot be held responsible for misbehaving clients, the best it can do is cut them out when requested by external parties. Toyota doesn't appear as witness to vehicular man slaughter cases. It's a car to take you from A to B but it's not Toyota's fault if the customer runs over something between those two points and not it's intended design (QVC vs Resultly).

It also doesn't help that there are pathological web scrapers who simply does not have the money to do anything fruitful so they will bootstrap using any means necessary and plays the victim card when they are denied. This particular group is responsible for majority of the litigations. People who otherwise have no business by piggybacking off somebody else using brute force to bring heat to everyone involved.

<comment><USER35> >Well the difference is pretty clear between your case and the rest. Developers scraping a website isn't going anywhere. A business reliant on scraped data is making money off of it. That will lend you in precarious situation more.

Developers presumably scrape websites because the data is of some value to them, frequently commercial value. Google's entire value proposition is based on scraped data, and it's one of the most valuable companies on the planet. The way the data is used is not necessarily relevant to whether the act of scraping a web page violates the law or not -- several more basic hurdles involving access, like the CFAA and potential breach of contract depending on whether the facts of the case are such that the court holds the ToS enforceable, have to be overcome before the matter of whether one is entitled to utilize the data obtained becomes the hinge.

>My criticism was that you mixed in service providers and tool providers that enable businesses to make money off scraped data-the vendor cannot be held responsible for misbehaving clients, the best it can do is cut them out when requested by external parties.

 _3Taps_ is one of the most prominent such cases and it was just the type of tool that you're claiming wouldn't be held accountable. 3Taps's actual client was PadMapper, but since 3Taps was the entity actually performing the scrape, they were the party that was liable for these activities.

The lesson we've learned from 3Taps is that scraping tools _might_ be OK if they strictly observe any hint that a target site doesn't want the attention and cease immediately, but there's really no guarantee either way.

Most people won't sue if you adhere to a C&D, not because they _couldn 't_ do so and win, but because it's much cheaper to send a C&D and leave it at that, as long as that settles the issue moving forward. Litigation is very slow and expensive.

<comment><USER13> 3Taps became liable because they put their neck out for PadMapper even after they received written letters.

It was a poorly executed business strategy because they were up against powerful legal team.

Best thing to do if you receive C&D or requests to stop scraping, best to not continue and just let that customer go.

<comment><USER35> You _can_ be sued (and lose) for damages incurred by illegal activity whether the aggrieved party sends a notice or not. It's not the plaintiff's job to let you know you're breaking the law, and they're entitled to damages whether you know you're breaking the law or not.

In fact, it's _assumed_ that defendants weren't intentionally breaking the law, which is why when it's clear that they _were_ , courts triple the actual damages for willful violations. [0]

If a reasonable person wouldn't realize that they were "exceeding authorized access", that probably limits a potential CFAA claim, but that's it, and that's not only the potentially perilous statute when you're a scraper. In the QVC case, Resultly got lucky that QVC did not have an up-to-date robots.txt; otherwise, they very well may have been on the hook for multiple days of lost online revenue, despite their immediate cessation upon receipt of a C&D.

Again, you are more than welcome to take your perspective and run with it, and it's plausible that no one will get mad enough at you to sue over it. That doesn't change the law.

I would assume that 3Taps pursued this litigation not because they had special love for PadMapper, but because they felt it was important for their business to be allowed to scrape major data sources and thought they'd be able to win. Pretty sure Skadden was their law firm so they gave it an earnest try, but ultimately lost.

[0] [https://en.wikipedia.org/wiki/Treble_damages](https://en.wikipedia.org/wiki/Treble_damages)

<comment><USER13> You can be sued for crossing the street. You can be sued for flipping the bird and someone happens to get aneurism from it. You can be sued for writing what you just wrote!

<comment><USER35> Yes, but if you fight it adequately, you won't lose. If you get sued for scraping, it's quite likely you'll lose, as the law has numerous pitfalls for scrapers, including things as basic as regarding RAM copies as infringing.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER36> We all do this, but how legal is it? If people end up in prison for pen testing without permission, how safe is to intentionally alter the user-agent, circumvent captchas, javascript and other protections? Can that be considered as hacking a site and stealing the data?

</comment><comment><USER37> Proposition: 99% of scraping use cases are eliminated if the scraper agrees to subsequently abide by the target's terms of service.

<comment><USER38> Googlebot doesn't abide by 99% of websites' terms of use.

<comment><USER39> But that's "different" because they've built a $600bn company off it.

<comment><USER38> More that the websites actually want to be found by someone.

<comment><USER40> The over-reliance on google search is both a blessing and a curse for the web. Today, google IS basically the web, a centralized version of it.

</comment></comment></comment></comment></comment><comment><USER41> I've used antigate for captchas and ether Tor or proxies for 403s before. Usually the browser header alone does not help for long.

<comment><USER42> Anticaptcha and deatbycaptcha are some others. But it mames me feel sad to use them, as it exploits cheap labor overseas.

<comment><USER43> Most of the time they use OCR, humans are unreliable and rarely used.

<comment><USER41> no, at least antigate doesnt. When you hit recaptcha with known proxy urls (or generally hit it a few times per hour) the captchas get so bad that no OCR would be able to solve it, even humans struggle

<comment><USER42> yup, exactly. i tried tesarract before (nothing too fancy), it didn't have problems solving it, but at some point it became really hard.

I think part of it is how you crawl (phantomjs, for example, seem to hit captcha almost every time), but things like ip&proxy usage could make this trigger more often.

</comment></comment></comment></comment></comment><comment><USER44> Good article! I been doing scraping for the last 10 years and I've seen a lots of differents things to try to avoid us. Also, I'm in the other side protecting websites to ban scrapers, so funny!

<comment><USER17> I'm in the same position for the first time (protecting against scraping) and honestly I'm kind of blind right now. Which is weird because of how much scraping I've done (okay not that much). Any tips or tricks or blogs you know of off the top of your head for protecting your site?

<comment><USER45> Virtually everything can be easily defeated. The only outfit I've consistently seen put up a good fight is Distil. They do it by acting a little like Cloudflare. They put their servers in front of your www facing endpoints and use ML to mine their global client traffic to identify bot signals (aided by some aggressive in-browser javascript fingerprinting).

<comment><USER46> Yeah, Distil is the first outfit I've encountered where they've got the model to make it really hard to _reliably_ bypass. It comes down to "I can spend a significant amount of time trying to bypass this, and I would, but they would likely identify and block me again within a few weeks at most.", and it's not worth it when it's only _part_ of what I need to do to scrap some data, and it's their entire job, and they can afford to hire multiple people.

The economics are in their favor, and I make it a point not to fight economics when I recognize them, it's rarely sustainable.

<comment><USER17> Distil is really interesting.

</comment></comment></comment></comment></comment><comment><USER47> What if the target page is blocking by IP address and if even with 20 different IP addresses you wouldn't be able to fetch all the data you need in a month?

<comment><USER45> Professional proxy services. Price, IP pool size and quality vary hugely but if you're not trying to scrape an aggressively defended target and don't need to make more than a handful of requests per second, 100K IPs will usually be more than enough to circumvent most rate limits and a pool that size can be rented for under $100/month.

<comment><USER48> Interested to know where to get 100k proxies for $100/mo. Can you give some options?

</comment></comment></comment><comment><USER49> Have you seen the sentry antirobot system I can't remember the name exactly but it's a hosted solution that randomly displays captchas, when it senses suspicious(robots) crawling. It's a nightmare, because after you solve 1 captcha it can display 4 more one after the other. They also ban your IP, so oyu need IP rotators. Any workarounds? ic3cold

</comment><comment><USER50> What if they use that before:after thing where the content takes say a couple of seconds to appear so when you try to scrape the site it appears that nothing is there. I have only used HTMLSimpleDom scraper with PHP at this point.

</comment><comment><USER51> Sometimes it's also necessary to spread requests over numerous IP addresses.

</comment><comment><USER52> The first part seems like a very long-winded way to say "don't use the default user agent".

The captcha was unusually simple to solve, in most cases the best strategy is to avoid seeing it in the first place.

</comment><comment><USER53> Enjoyed learning this and playing with it. What would you recommend storing this sort of data in? Not too keen on going with the traditional MySQL.

</comment><comment><USER54> Nice overview! The "unfortunately-spelled threat_defence.php" just uses British spelling though.

<comment><USER55> What's wrong with British spelling? It's also the English spelling using in India, Australia, New Zealand etc. By pure numbers, more people may spell it defence than defense. Americocentrism is quite annoying from the other side :)

<comment><USER54> I'm not saying anything's wrong with it. The "unfortunately named" bit is from the article, and I'm just pointing out that the author's snark is ill-placed.

</comment></comment></comment><comment><USER56> too bad it's named for ovine prions.

</comment><comment><USER57> Try lynx

</comment><comment><USER58> Great article!</comment>