Ask HN: First tiny freelance job questions - <AUTHOR> I'm a freshman in college and have agreed (informally) to help a local small business convert a large excel file into a format their new inventory software understands. I'm meeting with them in a week to determine whether their request is feasible, and then discussing terms of engagement if it is. We've discussed compensation and I've said I will charge a fee for the consultation and then some additional rate if there is any other work that can be done.
After this point, though, I'm lost. I don't know what to charge, how to invoice, or what kind of contract I need to write up. There is lots of info about this around the Internet, but it seems to be aimed at more established developers doing larger projects in higher cost of living areas. If I walk in with a 10 page contract asking for $300/hour they'll probably politely decline.
That in mind, does anyone have any advice on how to proceed? <comment><USER2> Keep it simple, this sounds like a simple project. Estimate how long it will take you to do the work, charge them a flat fee, and get half of the agreed upon amount up front.

$300/hour is way way way too much, even for a professional developer. You have no experience or formal education as a freshman, your rate should be max $50/hour.

You don't need any contracts, just get half of the money up front and if they screw you over, well at least you got that. When you give them the quote for your work, have each party sign the quote - that's your contract.

<comment><USER3> One of my projects started something like OP's project did - a small gig, replacing a complex Excel with a simple software. The Excel was a way more complex than I initially thought - business processes usually are, so take that into account if you offer a flat fee. Despite the flat fee for the initial project, I managed to do the replacement. After that the project grew and grew, and become fundamental part of their business - basically their ERP - and I ended up working on it for over 3 years. It was good money, but a bit boring in the end.

I recommend that you set market rate as your base (target) rate. Ask around what the market rate is in your area, but it is likely more than $50/hour for short gigs like this. However offer them a significant discount, for example 50%, for the first X0 hours - and say that you offer it because this is your first project and it's win win for both parties, but you don't want to work with that rate forever.

If the project grows in importance, and you get a lot more work from them, it is easier to negotiate new hourly price, when they don't expect that your standard rate is the discounted rate.

</comment></comment><comment><USER4> As someone who has been there they will likely view what you're doing to be equal to what a freshmen summer temp might do. Thus asking for professional standard pay has a chance of going poorly. And to be fair as a freshmen you might have less then one year experience in programming.

If someone were to give me advice before I started at a young age I'd have wanted them to tell me to keep my expectations low and treat it as a learning experience and to derive as much enjoyment from it as possible.

<comment><USER5> I was thinking the same. Our Salesforce / Marketing cloud consultancies are in the same sort of range as described. I am doing the job for a fraction of the price (well my normal wage), but it is taking me a fair bit longer, as I am having to learn a fair bit about these (awful) systems as I go.

<comment><USER6> What do you dislike about SalesForce? We're currently looking at CRM's and that is one of the choices, so I'm curious about your thoughts on it.

<comment><USER5> As a developer its really tedious to work with.

I have to integrate our booking system with it, sending everything via the REST API. Its got loads of restrictions (only send 1000 records per file and such like) that make it really slow. It takes around 1 week to do a similar task via salesforce as a day long task doing Django to Django.

A lot of the functionality need to be done over their web interface - so again slow. Mass deleting records again has limits of 250 per delete.

I am sure an expert in Salesforce would be able to do things more efficiently than me, but its unpleasant enough to work with, that I don't especially want to become one.

</comment></comment></comment></comment><comment><USER7> I'd keep it simple, pick an hourly rate that works for you, and just bill the initial meeting at that rate. Get them to agree to the rate informally before you pass around any paperwork. Don't write your own contract, use a standard one, and keep it short. I can share my half-pager with you if it would help.

<comment><AUTHOR> _I can share my half-pager with you if it would help._

If you wouldn't mind, it would be nice to get a real-world example.

<comment><USER7> Long [https://dl.dropboxusercontent.com/u/6821374/Independent_Cont...](https://dl.dropboxusercontent.com/u/6821374/Independent_Contractor_Agreement.docx)

Short [https://dl.dropboxusercontent.com/u/6821374/Independent_Cont...](https://dl.dropboxusercontent.com/u/6821374/Independent_Contractor_Agreement_Short.docx)

Be sure you understand everything in whatever contract you sign, and consult with a lawyer if you're unsure of anything.

<comment><USER8> Thanks, this will help me a lot. Are there any others around that I can access somewhere? Googling didn't help.

<comment><USER9> This is late, but checkout hellobonsai.com ... I'm a freelancer and use it for all my clients. Customized pre-made contracts + built in invoicing and payments through Stripe, Bonsai takes a very minimal fee too..

Made my life so much easier..

</comment></comment></comment></comment></comment><comment><USER10> In the world of freelancing/consulting/contracting:

1\. No matter how you price your services, what terms you offer in your contract, or how competent you are, there's a significant chance that the _potential_ client will decline.

They will decline because they are just window shopping. They will decline because someone cheaper comes along. They will decline because it turns out to appear not to be worth the money. They will decline because someone more expensive comes along...or the owner's nephew, or someone who is simply a better fit. They will decline because a customer canceled an order, the bank loan didn't come through, a key employee quit, a new hire was made, or your shoes were the wrong color.

2\. It's hard to do at first, but there's no point in investing too much in any one proposal. 'Just' listen to the _potential_ client, make a decision on rate and terms and present it.

3\. When dealing with amateurs -- by which I mean people who don't frequently contract for the sort of work you do -- you will often be competing with free. You will often be up against other similarly unrealistic expectations.

4\. When dealing with non-amateurs it's often a good idea to consider what happened to the person who did the sort of thing you're doing for them previously.

Specific advice to your situation:

It is not unlikely that the company had the option to pay the inventory software vendor to convert their data over, or that the vendor could provide a list of qualified consultants to do the conversion. And that the company chose not to pursue that route.

If this was an advertisement to students rather than something that came through your social graph via 'warm' and 'organic' relations, it is likely that the company is seeking to get the work done on the cheap. Because that's frequently the motivation for hiring students to do 'professional' type work. Unfortunately.

Finally, keep in mind that it is highly unlikely that this is the last or best or most important piece of work that will come to you in your career. The most important thing, whether successful or not is to learn a bit about business, a bit about human nature, and to improve your 'spidey sense'.

Good luck.

<comment><USER11> As someone who has freelanced a lot, I wanted to say great post. I hope OP takes your "most important thing" to heart.

</comment></comment><comment><USER12> There are a lot of resources out there for contracts-- which you should definitely be using-- andFreelancer's Union also has some good tax and contract guides-- [https://www.freelancersunion.org/resources/](https://www.freelancersunion.org/resources/) (They're free to join.)

The main thing to remember is that if the total value is over $600 and you're reporting the income, the company will likely ask for a W-9 from you and issue a 1099 at the end of the year. Then, you will be charged the 15% self- employment tax on top of federal and state taxes. A lot of beginning freelancers don't figure this into their rates, and they end up with a much bigger tax bill at the end of the year than anticipated.

I would recommend charging hourly, $50-75/hr is likely reasonable in your area, but providing the business an estimate of the range of hours-- say 10-15, or 20-25, etc so at least they know ballpark in the beginning what to expect. If you check in with them midway and say "hey this is ending up to be a lot more complicated, just so you know it will take more hours" then they'll appreciate it and be more likely to pay you quicker with no quibbles or "surprises."

</comment><comment><USER13> Do you run a business with a DBA or will they be writing a check to you personally? How well do you know the client?

I would be fair and honest when you do the consultation - in terms of what you charge and in terms of the feasibility part. Without more info, I think an email can be enough at this point just confirming details in writing, including the consultation fee and anything else as this gig sounds quite informal right now. Others may disagree and recommend an LLC or some long contract as there is risk involved.

Only after the consultation can you figure out how to structure a contract. Based on what type of effort and time frame is needed. Whether to go with a total project fee or charge hourly and you will get a better feel for how this client does things.

An email might go like this:

"Hi...,

Follow up to our conversation...Consultation to get your new software compatible...should be 1-2 hours on site...The flat fee is $200. Please make the check payable to Freshman Paidsworder...I am available Wednesday and Friday next week to come out there...Please let me know if this works for you and what day/time is convenient..."

Keep in mind, how do you think this small local business handles other contracts (eg HVAC repair job), do they have a bookkeeper or are you working with the owner directly?

If this type of work is something you would like to keep doing, try to make it positive/easy for you (be yourself) and make it positive for this client (word of mouth/reference/right thing to do).

Good Luck, Congrats.

P.S. $300/hour is not _that_ crazy if it is one hour of work total ;)

</comment><comment><USER14> I would charge them a flat fee for time spent on discovery. Say a few hours so you can gauge the effort. Then write up a description of what you think constitutes the "deliverable" and effort/cost involved for fixed cost. That way they can't tack "one more thing" on the end without more money.

<comment><USER15> This is a good place to start; I've done this for several freelance projects, start with a "Feasibility Study" and charge a fixed price for it.

This gives you time to investigate the effort involved in the work. The deliverable is a short document outlining what the scope of your solution is and ideally how much it will cost (for small projects clients generally prefer a fixed price). Pick a number that you think you would be happy doing the work for and you think the client will be willing to pay- this could be based on estimated time x rate or simply a number you are comfortable with.

If this is agreed then you can move forward with doing the actual work.

</comment></comment><comment><USER6> I've always been interested in doing side jobs, but unsure how to agree to a rate and bill businesses/people based on what they want.

Some people want a website built or fixed up or apps - can anyone offer some simple resources/advice to go about doing that?

<comment><USER16> Double your day job hourly rate. So if you make $100k a year, that's $50/hr, and your freelance rate would be $100/hr. When you factor in benefits (insurance, 401k) and the self-employment tax of 15%, plus the cost of downtime to do self-promotion and marketing, it works out to about double your employee rate.

<comment><USER17> I've always heard that 100k is about $50/hr but I don't think that's right.

salary 100k

self employement tax + 8k ~ 108k

health insurance + 16k ~ 124k

401k match + 3k ~ 127k

3 weeks of vacation and 10 holidays is 47 weeks of work.

So ~$2700 per week which is $67.5 per hour

(this doesn't include disability/unemployment/life insurance)

<comment><USER16> It's just raw salary divided by 2000 hrs (50 weeks a year * 40 hr weeks). It's obviously never fully accurate for any one person, but that's just how the world calculates something that isn't inherently hourly. If that's what you're worth an hour then double $67.5!

</comment></comment></comment></comment><comment><USER18> A few ways to determine your fee-- can you estimate the value created by the work? Man-hours saved? Revenue generated? What sort of budget did the client have in mind? Is there a benchmark rate individuals who specialize in this area?

Suggest reading Alan Weiss, still the gold standard on building a consulting practice > [http://www.goodreads.com/book/show/260218.Million_Dollar_Con...](http://www.goodreads.com/book/show/260218.Million_Dollar_Consulting)

</comment><comment><USER19> I've done a bunch of little freelance stuff. Money is always great, and I'd follow the advice already given if that's what you're looking for.

On the other hand, it might be worth considering a trade for labor for whatever it is they sell. When I go this route, I usually end up getting way more in trade than I would have in cash. But again, this is only if they're selling something that you may want.

</comment><comment><USER20> Be very careful with intellectual property issues. Make sure you decide what part of the work you'll do belong to you or them. Discuss that with them upfront, in a diplomatic way (so you don't sound too pretentious). That's very important if you "think" the open source way, where we can share some stuff. Some people are absolutely against that mindset.

</comment><comment><USER21> Contracts just say I x agree to pay n dollars per t time for work. That both sign. And then sometimes have clauses if party 1 or 2 doesn't pay or do the work.

You have two options imo (probaby...ianal,...), give a contract that has rate on it and then your timesheet, or just contract and invoice them after.

You can do a simple contract off google.

Just do 1 Job at a time imo

</comment><comment><USER22> It's ok to be honest about where you are at: telling them that you don't know what to charge, invoices, contracts, etc. Then you guys can work together on finding solutions for all of that. And the pay rate can always change later on.

</comment><comment><USER23> Converting a large excel file into a format for their new inventory system is valuable to them. But it might not be that complicated for most developers.

They would probably agree to a fixed fee more easily than a higher hourly rate. There is some risk here if you're not confident how long this will take.

Figure out what your target hourly rate should be based on what you want to make or feels fair. Then setup a fixed fee price that works for you.

I would propose three phases.

A. Initial meeting, they provide a small sample of their existing file (a few lines) along with the same data in another excel file manually converted to the format their new software requires along with the full excel file they want to convert. Also ask for rules/requirements, special cases for the conversion.

B. Setup a sample 50 to 100 lines as an initial test, that you will convert and provide them as a test to verify the conversion is working properly.

C. Once they ok part B, convert the entire document.

So maybe part A: $50, part B: $50, part C: $150.

Prepare a contract ahead of time with the Phases and amounts included so they can sign off on it at your meeting. A simple one or two page contract should work. You can probably use the short version of the contract eschutte2 posted. Just replace the hourly rate section with a fixed fee section.

I would invoice them $250 payable by check when you deliver the final excel file(note this in the contract). Companies usually don't like writing multiple checks for small amounts.

If you do part A and part B and the conversion isn't working out, invoice them $100.

Let them know that you will provide support if there are issues with the conversion at no additional cost if it's an issue outlined in their original rules/requirements for the conversion. If it's a new rule/requirement you can provide that for an additional fee.

They will probably ask you for a W-9 tax form for their accounting department so they can send you a 1099 for taxes, required if they pay you $600+ for the year. You'll probably be operating as a sole proprietorship which is ok, I'd recommend getting an EIN (free online IRS) so you can use that instead of your social on the W-9. You could prepare this ahead of time or send with your first invoice.

I would include a clause in the contract that the deliverable is the converted excel file, the scope of work is converting the excel file and that you retain the exclusive rights to code developed to make the conversion.

Google convert excel for 'name of their inventory software' if it's a popular search/topic. You might be able to setup an online conversion tool that you could charge companies a one time fee for converting their file online. Might be a fun project for you and a potential money maker.

Good luck with your first freelance project.</comment>


Karmurl - Give Feedback on Websites to Get Feedback - <AUTHOR> http://karmurl.com/

<comment><USER2> It's a good idea, and a couple of bits of feedback I've received have been useful.

It would be good to be able to modify my submission after it's been added.

I'd also like the opportunity to respond to others' feedback—could be an option when leaving feedback.

<comment><USER2> Some of the feedback I've received has been excellent, and I plan on sharing it with my team members. The ability to download all the feedback for a submission as a .txt or .pdf file would be very helpful.

<http://karmurl.com/give?s=9okt8061otbq7o56> if you'd like to leave feedback for <http://www.localphone.com/>

</comment></comment><comment><USER3> I cannot have the feedback window open AND look at the site at the same time. A real popup (not that javascript usability break... (grumpiness not directed at you specifically)) would work better.

A report button is missing, I had "drupal.org" once with a nonsense question. And another site asking about some spammy stuff ("what are your thoughts on online marketing" i think))

A function to reshow the question would be nice

aargh, this is addicting.

</comment><comment><USER4> Wow.

Why has nobody ever thought of this before? You could go a step further and make 'karma points' a commodity! Pay a dollar for a feedback karma and take 50 cents for every karma point you want to give up. Would be nice. Add paypal to make it work nice and easy.

P.s. Although both the idea and the 'easy' bit in paypal integration could be controversial.

</comment><comment><USER5> Definitely need to be able to edit/add to your response and would love if you could respond to others feedback. Maybe it's just me but I feel like the 1 to 1 feedback system is a little tedious and could lead to feedback like "Great site" so whoever it is can see somebodies feedback on their site. I'd love to see a gallery view as well where I can see a screenshot of the site and click it to give feedback. What's the business model?

<comment><USER6> It was a midweek side project so there was never a business model in mind. I tried starting up <http://kennelboard.com>, but it never really took off (selling and developing is hard on your own), so I'm now looking for a job and hoping that this will help me swoon potential employers.

<comment><USER4> Hey, great job! A real test of an idea could often be it's potential to generate revenue.

I understand that it is a side-project and you may have different motivations.

But I guess it is even more reasonable for you to try implementing a business model. There is not much to lose (given you don't end up putting desperate adverts all over the app... Duh!) As an example, the 'karma-as-commodity' model might just work.

Point here being that I don't see something as valuable to a potential employer as a person who can write amazing apps and make some solid money out of it.

</comment></comment></comment><comment><USER7> I actually liked using this, really friendly and easy to use interface. I would definitely use this for a personal website or a small project

</comment><comment><USER8> Really cool idea! Great way to receive feedback for new concepts/designs. I look forward to using it!

The direct link to the submission is also a great feature: <http://karmurl.com/give?s=47msbc5r9kl7mx2l>

</comment><comment><USER9> Nice concept, but I tried removing my submission and re-submitting it with a proper request, but it won't let me add.

Keep getting an error that Submission could not be added :(

</comment><comment><USER10> Hey congrats this site looks awesome! I did DomainPolish and I'd love to talk some time about what you learned from it if you're interested. I love what you did with the design.

</comment><comment><USER11> the modal/unmoveable feedback overlay is quite annoying, to send a proper comment I needed to recheck and all the time it required closing and reopening it.

Also, it seems the site has a terrible potential to become some spammy/free- traffic engine, which I'm not sure can be controlled.

But apart from this, it _is_ a really cool idea and I like it a lot.

</comment><comment><USER12> I've been using Karmurl for the last couple of days. It's the simplest and most useful app I've seen posted here. I've already exchanged 30 pieces of feedback.

Feel free to give me some more and I'll return the favour - <http://karmurl.com/give?s=uyg28o3kbh94lszw>

</comment><comment><USER13> Really addictive, well done !</comment>


The Disability Gulag (2003) - <AUTHOR> http://www.nytimes.com/2003/11/23/magazine/the-disability-gulag.html?pagewanted=all <comment><USER3> This resonates. When my vision problems were first diagnosed (~1976, UK) my parents were explicitly told not to 'make too much fuss'about it or I would be taken away and placed in 'special' care. Being stealth mainstreamed came with its own baggage of course, though perhaps mercifully I was unaware of this Faustian bargain until recently.

I was lucky, I learned enough 'hacks' to pass for 'normal' most of the time which saved me from the thing that makes the gulag in its many forms so awful, which is that many people, even the kindest most well meaning people, have a predictable first response to almost any kind of disability or serious illness which is to immediately strip an individual of some of their agency.

There's a spectrum from over solicitous behaviour all the way through to making people shit in a pan where everyone can see but it all comes from the same place. /narcissistic navel gazing

Oh, pro tip: kinda covered in the article, but subtly. If you _really_ want to piss off someone who has a disability, call them 'inspiring' to their face. Bonus points if you squeeze a shoulder and wrinkle up your face. Surprised me too.

</comment><comment><USER4> Great article, loved the bits about making the senators uncomfortable, haha.

My significant other works in the field of disability care. It's chronically underfunded, but one of the main issues as explained to me is the quality of the labour pool. It's hard to find respectable people who are OK with cleaning poop and inserting catheters and are also OK with working shifts, sometimes 12 hour ones. And then doing that for minimum wage.

So when I read something like "...they still can't select their own assistants or decide where they live or with whom." I am tempted to think that it's not the fault of the aid organizations. They're trying to make the best of their finances and the labour they have available to them.

<comment><USER3> Somewhat, yes. My own SO works in post 16 education, at the college their is a vocational 'health and care' track. These are the people who will credentialed to work in state sponsored or provided care. I would not want many of them to look after any one I cared for.

I think part of the point the article is making RE this is that given (amongst other things) the variable quality of this labour pool (and the regulation it attracts), the ability to select outside of it - difficult as this may be - <USER2> valued.

<comment><USER5> I think this is partly a knowledge problem.

People don't have knowledge that this even really exists, or what the expenditures they're asked to fund are actually providing (or not providing as the case might be).

A transparent (open) budget from taxes to payment might help, as well as an easy way to compare the typical compensation for this task to others in society. At that point a more informed, and hopefully more humane, decision can be reached.

</comment></comment></comment><comment><USER6> Date: 2003.

<comment><USER7> Thanks; added.</comment></comment>


Setting up an HA Kubernetes cluster with private networking in AWS - kris-nova https://www.nivenly.com/k8s-aws-private-networking/ <comment><AUTHOR> I wrote a similar article a while ago; except using Rancher - Which adds extra features for infrastructure management. [https://blog.baasil.io/how-to- install-kubernetes-on-aws-d9fb...](https://blog.baasil.io/how-to-install- kubernetes-on-aws-d9fbbc04e816#.cnqfv841r)

</comment><comment><USER2> Good lord this is so much easier than it was 6 months ago. Really glad to see the barrier of entry going down to get these clusters built. Kubernetes has been paramount to our engineering growth at FarmLogs.

<comment><USER3> Yah I agree. K8s install has been the bane of the projects existence.

<comment> kris-nova Happy to know we are finally "here" with deploys/installs!

</comment></comment></comment><comment><USER4> kops is pretty great for managing the lifecycle of a k8s cluster. It even supports outputting Terraform code, if that is your Infrastructure-as-Code tool of choice.

<comment> kris-nova One of the best features of kops, is it's ability to be open ended in generating what the user wants! The private networking piece of Terraform codegen is right around the corner, and rumor has it, it will be used in some new API tests

</comment></comment><comment><USER5> ECS tasks now support IAM roles; how does this work for K8s equivalent? Do you need to use IAM user credentials for app credentials?

<comment><USER6> IAM credentials for ECS tasks are implemented with a transparent proxy to the metadata API, and there are similar projects for Kubernetes.

The one that's most popular I believe is kube2iam: [https://github.com/jtblin/kube2iam](https://github.com/jtblin/kube2iam)

Using IAM roles then boils down to setting an annotation such as `iam.amazonaws.com/role: my-db-access-role` on a pod (assuming the correct trust relationship has been configured).

</comment></comment><comment><USER7> kops is nice... but too tied to AWS.

I'm really liking the direction that kargo is taking. It leverages a well known toolset - ansible - to build out its functionality.

kargo is compatible with everything from bare metal to AWS... with the caveat that when using it in AWS, you have to use a different provisioning tool like Terraform.

This is a feature.. not a bug.

<comment><USER8> Kops actually isn't tied to AWS, though we only support AWS right now. We'll likely be adding GCE support next; user requested features on AWS have taken priority so far though...

<comment><USER7> @justinsb - oh I didnt know that. For example, we are struggling to use kops on bare metal. However, I quite understand your AWS feature priority.

<comment><USER8> So right from the start I wanted to make sure it wasn't locked to AWS. We used to have GCE support enabled (and continue to have it on a branch, though obviously it has bit-rotted as we've added more AWS feature support) But we built GCE support early so we could be sure that the abstraction wasn't AWS specific. And we have bare metal support on another (more experimental) branch. Bare metal is much more challenging, largely because on clouds we can run etcd in HA mode with automatic instance replacement, which isn't really possible on bare-metal, so it looks we have to settle for HA with operator intervention when an etcd instance fails.

So I actually think kargo is a great choice on bare-metal - and it also does a great job of running on multiple clouds and bare metal. But the design of kops isn't particularly tied to AWS, though it does match better with clouds.

</comment></comment></comment></comment><comment><USER9> This is a nice write up, but we honestly needed to stand up kubernetes on- premise and in public/private clouds. This is why we've been using this: [https://www.youtube.com/watch?v=B7nMFVaOOi8](https://www.youtube.com/watch?v=B7nMFVaOOi8)</comment>


Super 8 Languages for Making Movies - <AUTHOR> http://dl.acm.org/citation.cfm?id=3110274 <comment><AUTHOR> This is a language for video editing. The language can be found at: [http://lang.video](http://lang.video)</comment>


Ask HN: How to Disrupt? - <AUTHOR>

Thoughts, ideas and suggestions?
We have great articles on "How to become a hacker":ESR and "How to start a start-up":PG. I though it would be interesting and useful to explore "How to Disrupt?"
Disruptive ideas should be push enough to make then your new need. Now days we use google, for almost everything online search, bookmarking, history, cache, research... essentially google disrupted market so much that it has become a daily need(if you are online). <comment><USER2> Take something that is capital intensive and make it free.

</comment><comment><USER3> Uh, you already asked that:

<http://news.ycombinator.com/item?id=1578019>

<comment><AUTHOR> Due to time difference I didn't get as many responses as I was expecting from the community. Hence the repost.</comment></comment>


Interview with Paulus Schoutsen, Creator of Home Assistant, about Its Future - <AUTHOR> https://hasspodcast.io/ha046/ <comment><USER2> whoa.</comment>


The Web Is Dead, Long Live the Web - <AUTHOR> https://humane.computer/the-web-is-dead-long-live-the-web/ <comment><USER2> I'm curious why can't the html standard be defined in form of source code (of a parser and a render), rather than human language.

the trouble developers have to support many browsers is because of the implementation variations.

a recent issue I had on Edge is that if the border color is #FFFFFFFF, it is rendered as black, but on other browsers, this is white.

What if the standard itself is defined as a common source code, and not owned by any company?

the standard implementation could be a dumb one without any hardware acceleration. It only needs to define the correct behavior. Any customized implementation has to conform to it.

<comment><AUTHOR> An interesting idea! I think that was one of the original goals of the VPRI project ([http://vpri.org](http://vpri.org)) — very high-level executable specs that encoded the meaning rather than worried about performance. IIRC, their 2D graphics library was about 45 LOC ([https://raw.github.com/wiki/damelang/nile/socal.pdf](https://raw.github.com/wiki/damelang/nile/socal.pdf))

<comment><AUTHOR> The question still remains: what should the standard look like? And I think right now there'a not enough agreement on what the Web should be to make standardisation effective.</comment></comment></comment>


Crew describe climate of fear at Cathay after Hong Kong sackings - <AUTHOR> https://www.reuters.com/article/hongkong-protests-cathay-pacific/crew-describe-climate-of-fear-at-cathay-after-hong-kong-sackings-idUSL4N25F1OH <comment><USER2> They don't even have to impact any actual protesters. Just get the word out that your life / career might be impacted and that will be enough to scare people.</comment>


So, you want to learn bookkeeping - <AUTHOR> http://www.dwmbeancounter.com/tutorial/Tutorial.html <comment><USER2> If you don't know the basics, how to do you know when your accountant or bookkeeper is making a mistake, or giving you bad advice? How to do you know if they are optimizing their advice to make their jobs easier, not yours?

I stared an LLC last year. It it took two months to find an anchor client, and they started paying net 60 (so no income for four months). Compared to the year previous, my income effectively dropped 25%. The accountant figured estimated payments that would require me to pay every penny I made during the first half of the year. That would mean 6 months without income. He was simply taking last year's tax obligation dividing it by the number of quarters left, not the actual tax obligation. Did I mention he talked me into an LLC because the administration was simpler? He forgot to mention how much more in taxes I would be paying compared to an S Corp.

I'm now reading every tax book I can get my hands on, and am reading accounting textbooks. No one will watch out for your money like you should.

<comment><USER3> How are you paying more in taxes for an LLC compared to an S Corp?

<comment><USER4> The other two replies are incorrect (I have up until this year run our business as an S-Corp).

With an S-Corp, you pay your owners a reasonable salary. $50-70k is reasonable for our field according to our CPA. Any profits or losses from the business pass through to your personal income tax via a Schedule K1. The profits are taxed like normal income, they are _not_ taxed as capital gains.

The tax savings come from the fact that the distributions you draw from the profits of the company are not subject to FICA (medicare, social security). This saves you upwards of 13%.

If you draw more than your profits for the year (say you run an accrual method of accounting, not cash, so you recognize revenue after services have been fully rendered, rather than when the cash comes in and you can have a large bank balance but no profit for that year), the excess is essentially double taxed with capital gains. A painful mistake, so track your AAA appropriately :)

<comment><USER2> Interesting. When I ran my S Corp, my goal was to avoid estimated tax payments. At the end of the year, the accountant would calculate the required taxes and I would issue a "net zero" check that only paid taxes, or a bonus check with a enormous withholding. The remainder was taken out as a dividend.

Taxes paid via withholding are treated as if they are paid in through the course of the year which avoids estimated tax payment penalties.

<comment><USER4> Yeah, I just do the quarterly estimated payments. If your business is constant (not growing) it's easy to handle via extra withholdings. If you are growing, they at least make it easy so that if you pay an estimated tax that matches the prior years tax owed (paid quarterly) you avoid any penalties, even if you have to make a lump payment to settle up the difference when you do your taxes at the end of the year.

We have a pretty sophisticated business intelligence platform written in R that gets us estimated revenues within 10%, so I know how to budget through the year for taxes owed. I just pay my known quarterlies and keep enough in the bank to handle the remainder at year end.

</comment></comment></comment></comment></comment><comment><USER5> This was the blog post that finally unlocked double-entry for me:

[https://martin.kleppmann.com/2011/03/07/accounting-for- compu...](https://martin.kleppmann.com/2011/03/07/accounting-for-computer- scientists.html)

<comment><USER6> Good article, I submitted it here: [https://news.ycombinator.com/item?id=11240888](https://news.ycombinator.com/item?id=11240888)

<comment><USER7> Refer to [https://news.ycombinator.com/item?id=2298471](https://news.ycombinator.com/item?id=2298471) (5 years ago)

</comment></comment></comment><comment><USER8> Accounting is the language of business. If you go into business, you should either know it or be willing to learn. It's not good enough to have someone specialize in it for you unless you want to be handed your ass when it comes to anything regarding business valuation--which concerns more than raising capital and m&a. For a business with resources, capital allocation is perhaps the most important concern and accounting is the toolset you work with.

<comment><USER9> Could you elaborate more on why it is not wise to have someone specialize in it for you? I can certainly see how knowing accounting can help when going from MVC to capital. At what point should you rely on someone who is more knowledgeable in accounting?

<comment><USER8> It's wise to work with accountants, but it's not enough. There are a lot of characters in business, and in order to know when they're full of shit (because sometimes _they_ don't even know), you need to know how to count beans by the rules.

For example, a public company might report itself as profitable for some period based on operating earnings alone. But, you read their annual report and see they took a loss after tax with some extraordinary items. If you don't know what that means, then you don't know what it means--It really means they didn't make any money for the period, but they're going around talking about it like they did. I should mention that's all elementary stuff--there exists a whole field called "Forensic Accounting" where accountants work to spot fraud in financial reporting.

And small business owners can get mixed up too. They talk about their big revenues but don't mention they have a lousy 5% net margin. It can extend beyond trying to bullshit you--they can get into the habit of bullshitting themselves thinking they're ballers.

</comment></comment></comment><comment><USER10> I think there's probably room for a great one day course (interactive, examples, etc.) to show tech people starting businesses the important points of law, finance, management, etc. Positive and negative examples (so people internalize the need), followed by principles, followed by clear guidelines, and some sample policies.

It's way more effective for me as a learner to understand "why" first, then the specifics of how -- most accounting, law, etc for non-experts is presented as hard/fast rules on what to do, with no real motivation.

<comment><USER11> I think you should make it at least 1 day per topic. I can surely give you a great introduction to business finance in one day. Even though some sort of support mechanism is needed so that you retain the material learned (say email support for 3 months or similar).

On another note, reading few VCs blog and how many of them try to teach basics of financial statements in their blogs, this seems to be enough of a problem that they would be very happy with founders taking a good course in finance.

</comment></comment><comment><USER12> I'm in the "definitely do not learn bookkeeping" camp. Even to many accountants, bookkeeping is different than accounting.

Bookkeeping is the data entry of accounting - degree or professional designation are not required.

Accounting is taking the data and trying to reflect an accurate story to the financial statement reader - the data visualization.

Finance folks are readers of financial statements, where you interpret the data viz to make decisions. Business owners should be readers too. The decisions regarding working capital, capital structure, budgeting, etc. are the important parts. Not debits and credits.

It does help finance and business owners to learn accounting, but bookkeeping is a step too far - especially when there is many other higher priority things to learn. To put it in a tech analogy: a Python dev could learn C to gain a better understanding, but assembly is step too far.

<comment><USER13> I disagree. And I agree. What I disagree with is that you shouldn't learn bookkeeping. I agree that you should learn accounting.

The problem is that you do need to know the basics of bookkeeping to learn accounting.

And seriously, learning bookkeeping takes maybe one or two weekend and you will use what you've learned the rest of you business and professional life.

If you can't commit at most two weekend of your life to something as important as the basic building blocks of finance and accounting, then you probably shouldn't consider running your own business.

And yes, bookkeepers and accountants will cheat you. The stories are numerous. And the marks are always executives and entrepreneurs who never bothered to learn bookkeeping and accounting.

Finally, basic bookkeeping knowledge (as expressed in this tutorital) is crucial for making informed evaluations of accounting software (which is honestly going to do most of the actual 'bookkeeping' for you).

<comment><USER12> Bookkeeping is the wrong place to start learning the basic building blocks of finance and accounting. It's too detailed and step-oriented. You should start at the high-level of reading financial statements and dig deeper as you go. Try to see how financial statements flow by using everyday concepts.

If you are trying to spot mistakes an accountant made, you won't spot it by going through accounting entries. If you are trying to see how your business is doing, you won't know from knowing debits and credits.

You can spot an accounting mistakes or see how your business is doing by comparing the financial statements to what you are seeing in the business in real life. "Why is my net income look great, when the cash in the bank is so bad?" Or, "why does my cash in the bank not match the cash in the financial statement?". By asking these questions you learn accounting much more effectively. And your accountant should be able to explain that to you convincingly and in detail - if not get a new accountant (In fact, it is best practice for corporations to switch accountants every 4 years or so to catch mistakes).

<comment><USER14> The accounting equation, and theory underpinning book-keeping, is the right place to learn about accounting. But this is literally an hour's work.

I agree that you don't need to learn how to 'do book-keeping', in the sense of knowing your way around the UI of some specific accounting software.

But, unless you spend that first hour, you will need a _lot_ of explanation from your accountant in order to understand why income != cashflow. If you _have_ spent that hour, then your accountant simply needs to tell you: "your 'accounts receivable' went up, as you haven't converted those invoices you issued into actual cash". Actually, they wouldn't need to even tell you this, as it would be obvious from the balance sheet.

</comment></comment></comment></comment><comment><USER15> A fun and dynamic blog to understand accounting for entrepreneurs:

[https://www.odoo.com/documentation/functional/accounting.htm...](https://www.odoo.com/documentation/functional/accounting.html)

(part of the Odoo documentation, an open source accounting software)

</comment><comment><USER16> "So you want to learn bookkeeping?" No. No you really shouldn't. This is commodity, undifferentiated work and will potentially cost you hours (or tens of hours) per month. If you are a business making any sort of money you should go hire someone like Bench.co to do this for you.

<comment><USER17> It's funny, this is the exact opposite advice I've heard from most business owners. They all have recommended learning enough accounting to make sure that you understand what's going on with your books so that you're never fully trusting someone else to handle your money.

<comment><USER18> > _It 's funny, this is the exact opposite advice I've heard from most > business owners._

Because a substantial portion of HN considers all parts of a business, besides the product, a waste of time. Marketing? Finance? Human Resources? That's for the plebs who can't program. Meanwhile, this place consistently features debates conflating balance sheets and income statements, revenues and profits.

Seriously, if you a considering starting or running a business, and don't understand the foundations of book-keeping as laid out in this tutorial, you should feel bad. It's not like you require a two-year diploma; understand the basics.

</comment></comment></comment><comment><USER19> If you're looking for a handful more resources [https://bench.co/syllabus/](https://bench.co/syllabus/) is a decent start.

</comment><comment><USER20> Can anyone recommend a good Canadian/Quebec alternative to Bench? I'd love to use them, but they're US only.

</comment><comment><USER21> As someone who started a business structured as an S-Corp and running it without a professional bookkeeper, I found this site immensely helpful.

I do have a tax advisor, but I was on my own for day-to-day bookkeeping. It was well worth it to spend a few hours on this site absorbing as much as possible.

</comment><comment><USER22> There's an amusing bit of Javascript in that page to cause it to print as a single blank page. Never seen that particular "dark pattern" before. At some point, he appears to have commented it out and replaced it with a single CSS statement that does the same thing.

<comment><USER23> Which only serves to prevent the technically unsavvy. In Firefox: Shift+F2 "screenshot --fullpage" then print the screenshot of the page.

I've also never seen someone do that before, good spot.

What's interesting is that the CSS media query doesn't work for older browsers. So users using IE7/IE8 can still print the pages.

</comment></comment><comment><USER24> i've always done my own paperwork and filled out the ol' tax returns and company returns.

the biggest difficulty i have is the language, which wastes much of my time in google wondering what these /stupid/ terms mean when they involve concepts we learn at a very young age and nothing complicated at all...

i'll be bookmarking this and trying to drill it into my head to avoid it... but i still find it sad that there is an entire industry around accurate counting, summation and trivial sums propped up by red-tape and confusing language.

</comment><comment><USER25> offline for right now, archive => [http://archive.is/rHh8q](http://archive.is/rHh8q)

</comment><comment><USER26> I'm having flashbacks of undergrad.</comment>


Ferret – Declarative web scraping - <AUTHOR> https://github.com/MontFerret/ferret <comment><USER2> I think we are confusing two things here.

The OP probably meant this tool as something useful in his situation and shared it with others, in case someone else needs something like that. He was probably thinking along the lines of "If someone needs that, it's fine, let them discover something like that exists and use it". People over here are viewing this as a one-size-fits-all solution for web scraping and pointing out valid reasons why it's a bad idea to use it like that. I think that we should accept that this tool might be good for some, but completely unnecessary for others and we shouldn't criticize it for not being useful for our purposes.

One situation where this tool has clear advantages over other solutions is client-side scraping. If we made an app for ios/android/Windows/whatever that runs on devices owned by end-users and crawls data upon request, perhaps from multiple websites, having the crawlers written as external scripts would be extremely useful. That would allow you tu push updates to the crawlers separately, immediately after a website changes its layout, without the need to update your app. Making a gallery of downloadable crawlers for more sources would, probably, aslo be possible. The limitations of that language are very advantageous in this situation, as crawlers are mostly sandboxed and can't destroy your filesystem, steal your data etc. This tool also allows keeping the crawlers separate from the app. That would allow people to create a global npm-like repository for crawlers working regardless of what programming language you use (provided someone wrote an implementation of this tool in your language). Imagine an use-case like building a books price comparator app in, let's say, Java for android and swift for ios, and maybe even c++ because some libraries still run Windows xp and would like that app to be available, and being able to download the crawlers for Amazon and tens of local bookselling websites that would work in all of those apps, without a need to write them yourself. If used right, this tool could actually allow programmers to imagine that there's actually a semantic web as originally imagined and write services that interact with various websites in surprising ways, without thinking about how the interactions are done on a lower level.

<comment><AUTHOR> Thank you very much for your valuable feedback and I'm glad that someone has finally got the idea :)

</comment></comment><comment><USER3> I'm sorry but there is nothing new here? This seems like a backwards step if anything.

Usually when web scraping, I can just load in the HtmlAgilityPack (c#), point it at a URL then write some functional code to extract the necessary data.

Even better, I'll examine the website in Fiddler and hope they have a data- view separation going on, and be able to just intercept the json file they load instead.

Worse case scenario I need to dynamically click on buttons etc, but this can usually be handled by selenium, or if they detect that just roll a custom implementation of CefSharp (again not hard, just download the nuget, and it lets you run your own custom javascript).

A new, more limited, language (with no IDE tools) is not the way to go. If anything a better web scraper just make the above processes I mentioned more seamless, for example combining finding/selecting of elements in chrome with codegen.

<comment><AUTHOR> The main advantage of this over your approach with HtmlAgilityPack is that Ferret can handle dynamic web pages - those that are rendered with JS. And also, it can emulate user interactions. But anyway, thanks for your feedback :)

<comment><USER4> I think AngleSharp can handle JS and it's not that different from HtmlAgilityPack. [https://github.com/AngleSharp/AngleSharp](https://github.com/AngleSharp/AngleSharp)

<comment><USER5> 14th of July this year: [https://github.com/AngleSharp/AngleSharp/issues/693](https://github.com/AngleSharp/AngleSharp/issues/693). I'm not sure there's much JS support.

</comment></comment></comment></comment><comment><USER6> I think I have never seen so many negative comments before. Yes, OPs idea might be a bit unnecessary but it surely looks interesting. It could mature to a very interesting project. And depending on your business, if you rely heavily on crawling, having a specific language for it might help with making code more uniform

</comment><comment><USER7> What makes this language "declarative"? It looks pretty imperative to me.

<comment><USER8> I'm not actually seeing much code controlling the flow of the program.

<comment><USER7> I mean, I can remove control flow constructs from an imperative language and call it declarative, but that wouldn't be very useful.

What is the advantage of it being declarative? At least for the example, the equivalent imperative code is about the same length.

<comment><USER8> I don't think there is much advantage in being declarative in this case because whenever I need to scrape stuff I have to do a lot of edge case handling and I want to be in control, but it does seem to be a recurring dream to make some sort of web query language (based on my memories of Old Dr. Dobbs issues)

</comment></comment></comment></comment><comment><USER9> To be honest, I am not sure why this needs yet another query language.. :(

<comment><AUTHOR> Actually, it's not yet another language :) The syntax is taken from ArangoDB - AQL [https://docs.arangodb.com/3.3/Manual/](https://docs.arangodb.com/3.3/Manual/)

</comment></comment><comment><USER10> I was expecting an ML-driven framework where you write the HTML you want to scrape, and the framework diffs the trees and attempts to extract the information from the target tree as best it can to match your input tree. That's what pops into mind when I think of "declarative" scraping.

      LET google = DOCUMENT("https://www.google.com/", true)

      INPUT(google, 'input[name="q"]', "ferret")
      CLICK(google, 'input[name="btnK"]')
      WAIT_NAVIGATION(google)
      LET result = (
        FOR result IN ELEMENTS(google, '.g')
          RETURN {
            title: ELEMENT(result, 'h3 > a'),
              description: ELEMENT(result, '.st'),
              url: ELEMENT(result, 'cite')
          }
      )
      RETURN (
        FOR page IN result
        FILTER page.title != NONE
        RETURN page
      )

Looks an awful lot like:

      const { document, input, elements, waitNavigation } = require("your-library")
      const scrape = () => {
        let google = document("...", true)
        input(google, "...", "...")
        click(google, "...")
        waitNavigation(google)
        return elements(google, ".g")
          .map(r => {...})
          .filter(p => {..})
      }
      scrape();

Am I missing something here? I don't see anything declarative about the the first one over the second; both of these look identical and rather imperative to me. Is "declarative" becoming a buzzword (thanks to React, maybe?), or am I missing something?

</comment><comment><USER11> This looks all wrong. Page scraping is not accessing a data source in a way that means a query language makes any sense. The moment you need to interact with the page and admit there's a dom under there it breaks the idiom.

And why it's remaking variable declaration I don't know, and why is the for loop so verbose? If you insist on a query language go the whole way and remove repetition and syntax complexity because that's the only thing that could actually add value.

<comment><AUTHOR> DOM is a representation of some data. Which means, you can extrapolate the data and then manipulate it. The language itself has nothing related to the DOM. All DOM operations are implemented via functions from standard library.

"Good artist copy, great artist steal" I'm trying to be a good artist trying to not invent a new brand language (I'm not that smart), so I just picked up (copied) an existing one that fits better for dealing with complex structures like trees. So it is AQL - ArangoDB Query Language. [https://docs.arangodb.com/3.3/Manual/](https://docs.arangodb.com/3.3/Manual/)

If you have any suggestions how to improve the language - you are very welcome.

<comment><USER12> How about using an existing language, like Python? You can make a really great DSL using Python, and then people have access to all the other Python language features that they already know, and the stdlib that they already know, and 3rd-party modules they already know..

<comment><AUTHOR> I could, if I knew Python pretty well :) But I've done it in the way I needed it to be done. I wanted to have an isolated and safe environment that would allow me to easily scrape the web without dealing with infrastructural code.

<comment><USER12> Yup, I get it, I want that too, but I don't want to learn another language just to do that :/

</comment></comment></comment></comment></comment><comment><USER13> I think this could go further in terms of making it declarative.

A simple declarative approach could taking this:

        LET google = DOCUMENT("https://www.google.com/", true)

and instead of thinking about it as an action (get this page), think about it as giving you an object. The result is a tuple of the URL, the time fetched, and maybe other information (like User-Agent). This helps with exploratory scraping, where you want to be able to repeat actions without always re- fetching the documents. And you'll be constructing a program, unlike a REPL where you always write the program top-to-bottom, including all your intermediate bugs.

Changing DOCUMENT() is easy enough. Things like CLICK() are a bit harder, though if you extend the data structures you can have a document that is the result of clicking a certain element in a certain previous document. Again to do it the first time you have to actually DO the action, but later on perhaps not. And you'll be constructing interstitial objects that are great for debugging.

Then what could make it feel really declarative is having more than one presentation of an execution. You can package up a scraping, and then you can answer questions about WHY you ended up with certain results.

<comment><AUTHOR> That's what you can do right now :)

[https://github.com/MontFerret/ferret/blob/master/docs/exampl...](https://github.com/MontFerret/ferret/blob/master/docs/examples/input.fql)

Document, returned form DOCUMENT() function, represents an open browser tab which allows you to do all interactions with the page.

<comment><USER13> Well, that's what I'm saying... right now, making it represent an open browser tab with a specific state and where everything DOES something isn't declarative. But it could be declarative if you changed how those commands are implemented.

Or, to phrase it another way: if the program represents a PLAN then it's declarative. If it represents a series of things to DO then it's imperative. It seems like it's doing things, but it could plan things with the same syntax.

<comment><AUTHOR> Oh yes. The reason if this is that for now the language itself is DOM agnostic, it's just a port of an existing one. ([https://docs.arangodb.com/3.4/AQL/](https://docs.arangodb.com/3.4/AQL/)) . So, the entire DOM thing is implemented by standard library which is pluggable. In the future, I might extend the language to make it less DOM agnostic by introducing new keywords for dealing with that. But for now you have to move document object around. Which is not that bad, because you may open as many page as you want in a single query.

</comment></comment></comment></comment><comment><USER14> In late 2000 at the tail end of the bubble bursting, there was a search engine company trying to build on a platform of push notifications instead of crawling.

I know that at my current company the plurality if traffic comes from crawlers. We don't want to throttle them because that's biting the hand that feeds but it sure sucks.

I wonder often, how many crawlers you need before it's cheaper for a website to volunteer up when pages change or new ones arrive.

<comment><USER15> Maybe Pointcast? [https://en.wikipedia.org/wiki/PointCast_(dotcom)](https://en.wikipedia.org/wiki/PointCast_\(dotcom\))

</comment></comment><comment><USER16> Parsing the html or traversing DOM is the easy part. Doing request queues, ip rotation, data quality management, exponential backoff etc. on scale is much harder.

<comment><AUTHOR> PRs are welcome :) There is gonna be a separate project within the organization that would do all these things and even more. It's just beginning :)

</comment></comment><comment><USER17> Does anyone else find the narrative style in the README (memes, Internet language, etc) obnoxious?

</comment><comment><USER18> If you're looking for a slightly-more-native way to declaratively scrape (the data-binding aspect, at least) in Go based on CSS selectors, I wrote a wrapper around GoQuery that just uses struct tags for mapping. You don't have to learn a new language, and it should feel familiar if you've ever written CSS or jQuery. I've found it helps reduce a lot of boilerplate and makes things a lot easier to come back and read than what was previously a lot of nested GoQuery functions, selectors, etc.

Might be helpful and in a similar vein. :-)

[https://github.com/andrewstuart/goq](https://github.com/andrewstuart/goq)

</comment><comment><USER19> Web scraping "at scale" ends up being a lot more complicated than blinding firing HTTP requests. Scrapy, for example, supports arbitrary "middleware" that can, for example, follow 301 Redirect, respect robots.txt files, follow sitemap.xml files, etc.

To what extent is this supported (or, to what extent do you plan to support it?) Similarly, since the front-end language is essentially a compiler, would it be possible to write an alternative "backend" (e.g. something that distributes requests across a cluster)?

<comment><AUTHOR> This package is more like a runtime. There are plans to create a dedicated server, where you would be able to store your queries, schedule them and set up output streams like Spark or Flink. For now, it does not respect robots.txt. But it can be easily added.

Out of the box, there are not scaling mechanism yet, since the project is WIP. But, it's written in Go, which makes it pretty fast.

One idea of how you could scale it is to run cluster of instances of headless Chrome, put proxy/load balancer in front of it, and get Ferret a url to the cluster. It will treat it as a single instance of Chrome. The only problem, you would need to differentiate request from CDP (Chrome DevTools Protocol) client, and once a page is open, redirect all related requests to the same Chrome instance.

</comment></comment><comment><USER20> This is imperative scraping, and not as easy as just working with XML (what the page is anyway).

<comment><AUTHOR> One of the goals of the project is to hide technical details and complexities that follow modern web scraping, especially when you deal with dynamic web pages.

<comment><USER20> Which is already very easy with puppeteer

I feel in this case, it's solely to stay within a Go ecosystem which seems to be counter to what matters, business value.

<comment><AUTHOR> Yes, puppeteer and ferret use same technology under the hood - Chrome DevTools Protocol.

But, the purpose of the project is not to "be better than". No, the purpose is to let developers focus on the data itself, without dealing with technical details of underlying technologies. You can treat it as a higher abstraction on top of puppeteer / CDP.

It does not really matter whether it's written in JS or Go. main goal is to simplify data scraping and make it more declarative (even though, some people say it's not a declarative language :))

</comment></comment></comment></comment><comment><USER21> I think the name could be a problem? Or is this related to ferret the c library with ruby bindings? [https://github.com/dbalmain/ferret](https://github.com/dbalmain/ferret)

<comment><AUTHOR> Nope, just thought that ferrets are cool :)

</comment></comment><comment><USER22> This reminds me of Russ Cox's toy Webscript language: [https://research.swtch.com/webscript](https://research.swtch.com/webscript)

</comment><comment><USER23> yetanotherscrapingframework

<comment><USER24> It's not so interesting to me that people continue to feel the need to build a scraping framework (it's an excellent beginner project because it encompasses so much of web development) but why HackerNews finds scraping frameworks so interesting. There seems to be a scraping framework at the top of HN every week or two.

<comment><USER25> Personal theory, as someone who has done a good bit of scraping over the last decade:

Extremely common problem space, with a lot of tantalizing opportunities for a "platform" or "shared language" where one doesn't exist. I see existing tooling like BeautifulSoup, Scrapy, Selenium, as isomorphisms to the "near hardware level" of our problem-space-tool-stack, whereas the abstractions and higher level logic is often defined per the use case.

On top of (or perhaps because of) that, one often writes a lot of boilerplate, but when it comes to genericize, one often ends up writing the tool that genericizes within their problem space, and for all the tantilizing opportunities, the number of "not quite fully intersecting scraping problem spaces" (and associated tradeoffs/different paradigms) is far more massive than I considered when I started any of my own scraping tools.

This has lead me to take a very opinionated view with my own tooling, wherein I build for _ONE SPECIFIC RECURRING SCRAPING PRIMITIVE_. (in my case, treating the whole world as a stream of events. You want something that can be more or less massaged into that? Cool; maybe something for you. If not? Probably want to look at a different set of tools)

</comment></comment></comment><comment><USER26> I think comments here are unfairly harsh. I really like this innovative idea of having a dedicated language. If it can see client-side rendered HTML (e.g. React, Vue, etc...), that would be a whole another level for me since I don't think this has been made before.

<comment><AUTHOR> It can! :)

Even more - it can interact with these pages! Here is an example of use of Google Search page: [https://github.com/MontFerret/ferret/blob/master/docs/exampl...](https://github.com/MontFerret/ferret/blob/master/docs/examples/input.fql)

<comment><USER26> Wow, thank you. Now I am absolutely in love with your tool!

<comment><AUTHOR> Great! ^_^

</comment></comment></comment></comment><comment><USER27> I am currently using [https://www.npmjs.com/package/proxycrawl](https://www.npmjs.com/package/proxycrawl) web scraping system that offers anonymous scraping too. But they do not have a Go library yet. Does the system support anonymous scraping too like over a proxy? I'd love to use it if I can scrape URLs with proxycrawl API like with this example: [https://api.proxycrawl.com/?token=DsFuFiigAZ2Wm6U1BPh7Zw&for...](https://api.proxycrawl.com/?token=DsFuFiigAZ2Wm6U1BPh7Zw&format=html&url=https://www.amazon.com)

<comment><USER28> Token?

<comment><USER27> I do not use tcp token, so I could share it for free, i use the api for javascript token which gives me dynamic content more.

</comment></comment></comment><comment><USER29> The best way to scrape already for many years is using a headless browser plugin. For example phantomjs with nodejs. That in combination with tor or a large proxy pool is unbeatable by all other alternatives.

<comment><AUTHOR> This is how it works under the hood. But everything is wired for you ;)

</comment></comment><comment><USER30> Why is this a language instead of a library on top of an existing language?

Here's what it would look like as a JavaScript (Node.js or browser) library:

        let g = getDocument("https://www.google.com/", true);

        g.input('input[name="q"]', "ferret");
        g.click('input[name="btnK"]');

        g.waitNavigation();

        let result = g.elements('.g').map(({
          title: result.element('h3 > a'),
          description: result.element('.st'),
          url: result.element('cite')
        }));

        return result.filter(i => i.title !== null);

<comment><USER31> I'm not sure why you're being downvoted. As far as I can tell from the examples, there is nothing that this language brings to the table that couldn't be implemented instead as an API on top of an existing language.

<comment><AUTHOR> That's true. The difference is how much efforts is needed to do that using API.

What it brings is just a higher abstraction of that API which lets you easily to get work done.

<comment><USER31> Do you have a more involved example where Ferret really shines, as opposed to a library with a similar API in JS or another common language? I really don't mean to be negative, but I just don't see how Ferret is any easier to use than something like Nightmare[0]. That said, I'm wondering if it's an issue of communication more than anything, so maybe a different example than the one in the readme would help.

[0]: [https://github.com/segmentio/nightmare](https://github.com/segmentio/nightmare)

<comment><AUTHOR> You are fine, I totally understand your scepticism. And you are right, there are definitely issues in communication.

First of all, I built it for myself. I needed a high level representation of scraping logic, which would run an isolated and safe environment. Second, I needed to be able easily scrape dynamic pages.

So, what I got is: \- high level, declarative-is language, that hides all infrastructural details, which helps you to focus on the logic itself. that helps you to describe what you want without worring about underlying technology. Today, I'm using headless Chrome, tomorrow I will use something else, but the change should not affect your code. \- full support of dynamic pages. You can get data from dynamically rendered page, emulate user's actions and etc. Heck, you can even write bots with it. \- embeddable. now, I have only CLI, there are plans to write a web server where you can save your scripts, schedule them and set up output streams.

But the main idea is to provide high level declarative way of scraping the web. I'm not saying you can't do that with other tools. I'm just trying to come up with something more easy to work with.

Regarding examples, the project is still WIP, so as more complex features I get, more complex examples I get. Here is more or less complex, getting data from Google Search. It's not that difficult, but it showcases the core feature of work with dynamic pages.

[https://github.com/MontFerret/ferret/blob/master/docs/exampl...](https://github.com/MontFerret/ferret/blob/master/docs/examples/input.fql)</comment></comment></comment></comment></comment>


Amazon Ending Prime 20% Off Games Discount - <AUTHOR> https://www.amazon.com/b/ref=as_li_ss_tl?node=17911060011 <comment><USER2> Not only that, but Twitch Prime's no-advertising benefit is disappearing. New prime subscriptions or renewals after September 14 will no longer have that feature. Instead, it will cost $108/year to purchase separately.

[https://news.ycombinator.com/item?id=17805691](https://news.ycombinator.com/item?id=17805691)

</comment><comment><USER3> Not surprised considering Best Buy just recently decided to discontinue their Gamers' Club Unlocked program which offered a 20% discount on games:

[http://forums.bestbuy.com/t5/Gamers-Club/Gamers-Club- Unlocke...](http://forums.bestbuy.com/t5/Gamers-Club/Gamers-Club-Unlocked- being-discontinued/m-p/1138597#M32725)

</comment><comment><USER4> Will amazon be lowering their price for prime because they are offering less for the membership?

<comment><USER2> To the contrary, they also intend to charge an additional $108 per year just to keep an ad-free Twitch.tv experience (which ceases being a Prime benefit on September 14).

</comment></comment><comment><USER5> Yet another reason im glad I cancelled my prime membership

<comment><USER6> Yep, I let mine expire this summer. It would have been 50% more expensive than when I first signed up for it. If I want two day shipping I'll pay for it and it will be significantly cheaper than a Prime subscription.

<comment><USER5> I mainly cancelled mine due to the worsening quality of everything I tried to buy from amazon. Its gotten so bad I pretty much look away from Amazon when online shopping whenever possible. Just as an example, I purchased a supposedly new pair of $300 headphones. They arrived in the mail, obviously opened previously, and covered in q sticky residue (dont know what it was). That was a prime featured item.</comment></comment></comment>


Ask HN: How do you choose a startup to work at as an early employee? - <AUTHOR>

I am an engineering undergraduate currently in my final year (New Delhi, India). Soon I'll be graduating and working in a job. I've always found working at startups more rewarding than working at large or mid size tech firms. I am interested in joining a startup as an early employee. But the problem is how do I identify a startup which works with cutting edge technology and has the right people to do the job? That is how do you identify a startup that could be equivalent of Google, Dropbox, Microsoft, Stripe and the likes. There is tons of data here at Hacker News, Angel list, Crunch Base, TNW index but I find it hard to filter out the noise from the signal. <comment><USER2> i've been there myself.

        - pick an area that interests you ( or are willing to spend some time).
        - search for startups/roles with those keywords
        - perhaps mail the founders

Lot of startup employees, give juicy traction info at conferences

check out the talk proposals and speaker profiles at conferences like [http://fifthelephant.talkfunnel.com/2015/](http://fifthelephant.talkfunnel.com/2015/)

FWIW you can ping me. bosky at helpshift dotc0m

~B</comment>


All or Nothing vs. Keep What You Raise - <AUTHOR> http://avc.com/2014/11/all-or-nothing-vs-keep-what-you-raise/

<comment><USER2> At first glance this made sense but after a second read it doesn't.

> _3) Creators who choose all or nothing are more committed to the project, > the raise, and the process. Keep what you raise often attracts dabblers._

I feel this one needs more evidence to back up the claim. I'm sure there are plenty of serious teams who are just as committed on either approach. Just because someone put a number on something doesn't mean they can actually deliver if they hit that goal.

> _4) All or nothing raises more money. The amount of money that is raised > every year in crowdfunding via the all or nothing model dwarfs the amount of > money that is raised in the keep what you raise model, except in the > charitable giving category._

That may be little more than a function of Kickstarter being larger than the others. Not really a helpful datapoint if you're trying to compare the the two approaches.

> _A founder can't go out to raise $5mm and then say to the investors "well we > only got $2mm of capital committed, but we are going to close on that next > week." That just doesn't fly_

Isn't this exactly what happens in a 'party round' or a rolling close? We may never know what the the founders were _trying_ to raise, since they'd close at 2MM and announce it as a success. I'm aware that I'm comparing Angel to VC here but my point is that some investors are ok with approach.

Oh, and the ending is unnecessarily insulting.

<comment><USER3> i'm not sure the comparison to vc is valid either.

I've heard of people starting kickstarter campaigns who specify a lower-than- needed amount in order to gain momentum that comes from successfully passing the funding mark. However, if they didn't get to an unspecified, possibly more realistic ( and larger ) funding amount they would cancel the project. clever hack!

not sure the same this can/is done in vc..?

</comment></comment><comment><USER4> We ran a Kickstarter campaign last year and we specifically wanted the All or Nothing model because our campaign _made promises_ \- and we only wanted to be held to those promises if we had a certain level of interest.</comment>


The Fall of Perl, the Web's Most Promising Language (2014) - <AUTHOR> https://www.fastcompany.com/3026446/the-fall-of-perl-the-webs-most-promising-language <comment><AUTHOR> Previous discussion: [https://news.ycombinator.com/item?id=7288884](https://news.ycombinator.com/item?id=7288884) (~3 years ago, 100 comments)</comment>


The Evolution of Character Codes 1874-1968 (2012) [pdf] - <AUTHOR> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.96.678&rep=rep1&type=pdf <comment><USER2> "By January 15, 1915, the Western Union Telegraph Company had begun using a printing telegraph system that combined aspects of the Murray and Morkrum codes. It used Murray's codes for the letters and controls, but generally followed the Morkrum conventions for which figures should be paired with which letters. Like the Morkrum code and the later English Murray code, the Western Union code used separate line feed and carriage return characters instead of a single line character."

Damn you, Western Union!

<comment><USER3> "Even before X3.4-1967 was published, there was already interest in two more minor revisions. First, the ISO code had since its first draft allowed the use of character 0/10 for new line as well as for line feed, but ASCII had not. On July 5, 1967, John B. Booth proposed that ASCII also include this dual meaning."

I fear it will never be resolved at this point.

<comment><USER4> Oh, it's resolved. It's just we don't agree on the answer.

</comment></comment></comment><comment><USER5> When was this paper published? There is no date on the paper itself, and different sites gives different dates.

I'm wondering because it incorrectly states that Unicode is a 16-bit code, which used to be a common misconception in the 90's.</comment>


Chinese billionaire Wang Jianlin warns of 'biggest bubble in history' - <AUTHOR> http://money.cnn.com/2016/09/28/investing/china-wang-jianlin-real-estate-bubble/index.html <comment><AUTHOR> So where can I buy a credit default swap on Chinese real estate?</comment>


The loss of jobs at the Carrier Plant visited by the president - <AUTHOR> http://www.newyorker.com/news/news-desk/what-its-like-to-get-laid-off-at-the-carrier-plant-trump-said-hed-save?google_editors_picks=true <comment><USER2> That article and the events and reactions it describes is a neat little encapsulation of the entire Trump presidency. This is much more important than a minor sense of schadenfreude. The challenges facing the president are really hard and complex, simple solutions often don't get us anywhere - we have to deal with reality.</comment>


Ask HN: Alternative to Snipplr.com? - <AUTHOR>

http://snipplr.com/ is a little webapp that let's you save and share snippets of code, it has an open API and plugins for many IDEs and code editors.
It was originally created by Tyler Hall (http://clickontyler.com/) on 2006, who sold it (I suppose) to the guys behind http://raven-seo-tools.com/ in early 2008.
Since then, the site has only been updated in order to add more and more ads and on top of that, it has been completely taken over by spammers.
In its most recent update -and despite the huge spam problem- they have just added even more ads, so I have finally decided to take my snippets out of there.
I'm currently considering http://pastie.org/ and http://gist.github.com/ as alternatives, but none of them offers the core functionalities that I'm looking for.
I would like it to have:
- Public/Private snippets. - Advanced search and browse by language - TextMate integration. - Comments and/or Versioning
Any recommendations? <comment><USER2> I really like <http://gist.github.com>

</comment><comment><AUTHOR> I will go for Github's Gists, then.

Thanks for the replies.

</comment><comment><USER3> tinypaste.com does most of that, sans textmate integration.</comment>


Customers now wary of security holes in connected devices, survey finds - <AUTHOR> http://venturebeat.com/2016/01/04/mobile-device-sales-slow-customers-grow-wary-of-security-holes-in-connected-devices-survey-says/ <comment><USER2> With connected devices being deployed with complex software (Linux) and hardware vendors who have no auto-deployment plan for security updates, I think consumers _should_ be wary.

<comment><USER3> Completely agree, though I'd love to see the actual questions asked, as I would be shocked if near 50% of consumers ranked security as a concern without prompting.

Personally, my waning interest in IoT devices is due mostly to lack of increased utility. A new fridge that displays my Google Calendar doesn't really offer me much additional value, at the cost of adding another failure mode to the device. Having had several newer appliances with failed touch-pads or electronic modules, I'm loath to buy a device that has _even more_ fancy electronics.

As for phones and tablets, I feel like I've reached the point where new iterations don't offer much additional value. Roughly equivalent to how I felt about PCs/laptops 7-8 years ago. Battery life isn't getting better (without going large), screen resolutions are great, cameras are great, and the devices are fast enough for the apps I use most (email, calendar, phone, camera, Flickr, and Kindle).

<comment><USER4> they don't share the question asked, but the PDF has a little more: "Out of the consumers aware of hacker attacks and owning or planning to own IoT devices in the next five years, 18 percent decided to terminate the use of the devices..."

[https://www.accenture.com/_acnmedia/PDF-3/Accenture- Igniting...](https://www.accenture.com/_acnmedia/PDF-3/Accenture-Igniting- Growth-in-Consumer-Technology.pdf)

Knowing the government actually snoops, and that hackers could also snoop will take the triviality out of a non-compelling connected device.

<comment><USER5> I wouldn't assume that most Americans know anything about the NSA snooping. Last Week Tonight ( John Oliver ) did man-on-the-street interviews last year to find out how many people knew about Edward Snowden. The number was approaching zero.

I would take a bet that the NSA is just as well known.

</comment></comment></comment></comment><comment><USER6> Even if most people were aware of these things, marketers seem to have gotten pretty good at convincing people that everything is totally secure, because most people lack the technical knowledge required to even being to evaluate their claims. "It's secure because we use ENCRYPTION!" would probably be good enough for most.

<comment><USER2> I think frequency and ease of updates can serve as a reasonable proxy for security. By that measure, for example, you should think long and hard before you ever buy a Samsung device that markets itself as 'Smart'.

<comment><USER7> Agree, though pretty much every other player patches on par with, or worse than, Samsung.

Realistically the standard is "is it made by Google, Apple, or Microsoft", because pretty much everyone else is shite at it.

</comment></comment></comment><comment><USER5> Good. I never saw much benefit from IoT talking to the outside world. Device to device in the house, sure.

Externally - nope.

</comment><comment><USER8> I do not trust random devices that plug into my home network. All connected devices are on a 'guest' network and firewalled (openwrt) from my other machines.

When I go to a cafe with my laptop, I run it in fully firewalled mode (no incoming connections of anysort).

At home, I want my machines to be more promiscuous. With the promiscuity comes various concerns. Whether it is my 1Password safe, my health and financial records, whatever - I don't want some $30 connected device to connect to my home network and publish a port to the outside world that permits tunneling into my inside network.

Unfortunately, outside of sandboxing/firewalling, I do not have the time to implement a tracking system to see what devices doing what.

Anyone else feel more safe with connected devices at home? What do you do?

<comment><USER9> Not really which is why I have not really gotten started with the whole IoT thing. I plan to use vlans to segregate as well as iptables.

I'm not fond of the arcane nature of iptables and started researching only enable unidirectional initiated traffic from my normal network to the isolated IoT network (IoT gets access to internet) yesterday before giving up as it was too late. Suggestions welcome.

</comment></comment><comment><USER10> If the vendor is collecting every bit of personal data that they can get rather than an unknown third party doing the same thing, is it a security issue?

<comment><USER11> Absolutely. If big name stores, health insurers, and even banks can barely keep their machines secure, then I have no faith that Random J. Company, Inc. is going to keep their customer data secure.

<comment><USER10> That's a good point. I meant something a little different (but wasn't 100% clear): Why is Random J. Company's access to and use of the data any less of a security breach than some unknown third party accessing and using the data?

Someone might say that user consent is a difference, but I doubt more than a very few users understand the scale of data collection, much less the implications.</comment></comment></comment>


73 Rules of Spycraft (2009) [pdf] - <AUTHOR> http://www.oss.net/dynamaster/file_archive/100102/0a947a77d762061cc87ec541c2d2dcc7/2010-01-02%20Dulles%20on%20Tradecraft%20via%20Srodes.pdf <comment><USER2> Rules 1, 2, 3 and 4 are like the first 2 rules of Fight Club.</comment>


Should the U.S. roll out nationwide free Wi-Fi? Depends on which administration you ask. - <AUTHOR> http://technologizer.com/2008/12/11/national-wi-fi-plans/

<comment><USER2> This is one of the dumbest ideas I've heard. WiFi has a typical range of 32m indoors and 95m outdoors. With a range of 95m, each access point would cover 0.0283385 km^2. The lower 48 states have a land area of 7,663,941.71 km^2. So, we'd need over 270M access points to cover the continental US. But that's an underestimate since you'd have to have a lot of overlap given that if you put two circles next to each other (oo), you would have gaps. So, you'd probably need around a billion. Also, remember that you'd have to run power and data lines to all of them (very expensive) and pay for the power (hey, a billion devices using as little as 10kw per year would cost a ton). It just isn't a feasible project. Heck, to provide indoor coverage you'd need around 10bn. WiFi is barely a decent idea in the densest of cities.

Now, if you want to propose that the US use the spectrum freed up by the discontinuation of analog TV in a few months for free nationwide wireless, I'll listen. Those signals can go for a hundred km! That moves it from being a stupid proposal from someone who doesn't understand how radio works to something really workable. Now, because of power limitations in battery operated devices, you'd have to place sites every few miles like cellular operators do, but that's still going to make it a matter of, say, 50-100k sites vs. 10bn.

But wait, wouldn't that just make the government another wireless carrier? Pretty much. Maybe the government could do it cheaper, but I'm not so sure I'm into that. I mean, carriers are already competing hard core to get wireless internet to people (all of them are around $60/mo). Would the government be able to roll out an entirely new network just for data cheaper? Would it be more effective?

If we've concluded that WiFi is a ridiculous proposition, then people would have to buy equipment to use with this new technology (whether it be WiMAX, EV-DO, HSDPA, HSPA+, or LTE). That's the real appeal of WiFi, the idea that we already have the equipment for it. Anyway, we can already buy that equipment from carriers. Would we get it cheaper from the government? Likewise, we can already get service from carriers. Would the government be able to provision it cheaper?

There is the case to be made that wireless is a natural monopoly and so you see efficiencies by having one regulated (or government) operator rather than having competing companies duplicating each others efforts in building a network. However, the government isn't going to compete with them directly. It just won't happen. The companies have invested hundreds of billions in developing networks.

So, I think the best we can hope for is that the new white-space created by DTV is allowed for open use (ala Google's push).

</comment><comment><USER3> Would you trust this network anyway? I wouldn't. It would (by necessity) be a public network, meaning no encryption, meaning everything you do can be viewed by your neighbors. I imagine it would become a major source of phishing/spam emails, since it's basically impossible to block a semi-skilled user from an open wireless network.</comment>


Show HN: TimeBend, a novel productivity app for iPhone - <AUTHOR> https://itunes.apple.com/us/app/timebend/id525224894?mt=8

<comment><AUTHOR> Please see the post on Time Bending (<http://chetansurpur.com/blog/2012/10/time-bending.html>) for more on the productivity technique that this app is based on.</comment>


What companies can do quality startup intro video for a good price? - <AUTHOR>

I am looking for some recommendations on consulting companies / motion designers that can do short intro video for product. Must take American Express. <comment><AUTHOR> Anything?</comment>