Ask HN: How do you handle customer support on your site? - <AUTHOR> Hi,
Customer support can make or break a site but for those with small businesses, handling customer support can be a real nightmare (esp. technical and billing questions). You can of course hire full time employee(s) but that is not an option until your site is making a sizable income, or you are funded.
So for the rest of us:
- what are the ways to handle customer support?
- Do you do it yourself, outsource it to some other site / person, or have hired person(s) do it? - What is the average time it takes you to resolve customer queries?
- How much budget do you allocate for handling customer support? <comment><USER2> My companies have supported X00,000 users and ~8k paying customers over the years with a maximum team size over that interval of, ohh, 1.25 or so. It's quite achievable by a small team.

I did all my own CS, over email, for the first 6 years. This was approximately 4 years too long. Time to first response (TTFR) was generally on the order of "I'll get back to you by the next business day on a best effort basis" \-- I hit that on about 98% of customers for the first few years and then probably slid later, particularly when sick or overwhelmed.

My first (B2C) product had a lot of not-very-critical CS issues which also weren't very complicated to resolve. After having done that for 6 years, I hired a VA firm in the Philippines who staffed me with a single person for ~4 years. I think it was 20 hours a month for $300 or so, IIRC. I acted as tier 2 support for her. She probably head off more than 90% of inquiries. We used Snappy (BeSnappy) for CS software -- I enjoy it rather more than I do support over email, and it was easy enough for her to get spun up on.

I've also done B2B support for my other SaaS product, which has fewer users, fewer customers, fewer incidents, and _radically_ higher maximum-criticality- of-a-ticket. Again, did it all by myself for the first few years. Eventually it became obvious that a supermajority of our support was onboarding for new customers, so when I hired a sales manager we mutually decided to roll that into her job description. Our process is she firewalls me from any ticket she can and, if anything survives, she dumps it into Slack as a morning roundup every evening. I either get her instant answers or tell her "#3: ETA tomorrow."

My sales/support person works near my customers timezone-wise, so they generally get very quick "Thanks for the email. I'll ask my tech guy about that and get back to you." acknowledgement for the harder issues (and very fast resolutions to the easier ones). This makes for mostly happy customers, although to-be-fair most of them thought I was pretty responsive back when they wouldn't even get first contact until a day after they had sent in the ticket. I'm not competing with having a super-responsive startup team on speeddial; my customers expectations' are set by bureaucracies which take weeks to acknowledge receipt of the first of three things required to get the ball rolling on opening a ticket.

Fully-loaded cost for my sales/CS person is on the order of "a few thousand dollars a month." for what I'd estimate as approximately 0.25 FTEs. (Worth _every_ penny in decreased stress level for me.)

Customers occasionally ask for 24/7 phone support. I quote Enterprise rates for anyone who wants that. No takers, thankfully.

I feel like the median experience of my customers getting CS from the team and myself is probably worse than it was when I was doing 100% of the CS, but it is not obvious to me that is true of their _perceived_ experience, and we've managed to successfully manage expectations from customers about how much of my personal time is included for $29~$199 a month. (A _particularly_ good thing as I won't be involved in that business much longer.)

Anecdata: 80% of CS incidents take < 5 minutes to resolve, 15% take < 20 minutes, and 5% take Way Too Long.

Favorite tricks for CS:

You're going to get the same issues _a lot_. Keep a list of them. Fix the ones that are caused by fixable product issues, and keep fixing them until those go away.

Build self-help tools for the issues which are high-frequency low-value-added like, most obviously, password resets. (Yes, there are still companies without automated password resets. I know someone with a master's degree who carries a beeper on Saturdays due to their company's refusal to spend money on tweaking an application to allow password resets.) Routine billing inquiries, etc are often also candidates for this.

You're now looking at a long tail of issues, but it _still_ has a fat head of the distribution. Process-itize those issues which are amenable to it: you want to have a Google doc which explains exactly the company standard response to X, including what to say (templates optional) and what, if anything, needs to be done in internal tooling. Do not extemporize a solution to the same problem 100 times: cache it and save extemporization for places where the team's brainsweat actually adds value over "the right answer delivered very efficiently."

Final thoughts: your expectations for CS are not your customers' expectations. Your expectations for your company's CS are not your customers' expectations. Your desired outcome for a CS incident is not your customers' desired outcome. I can keep banging this drum for a while. It's important.

<comment><USER3> What are the differences between your customer support desired outcome and your customers' desired outcome of CS?

<comment><USER4> I don't know what Patrick was thinking of specifically, but I know many people will call customer support looking for handouts rather than support. They'll say, "My $5000 Whatchamacallit has a blemish on its whozit. This is unacceptable. No, you can't fix it. I want a new one free."

<comment><USER3> So, should we bend to our customers' wishes or only provide actual support?

</comment></comment></comment></comment><comment><USER5> \- Just use email.

\- As long as possible, handle all requests yourself.

\- If people keep having billing issues, use a different solution for billing. I use Fastspring, and issues are rare. And I can just forward billing related issues to Fastspring support.

\- If you keep having to deal with technical problems, fix your product. Ideally it shouldn't need customer support. If it is so complicated that customer support is required, charge at least $100 per customer.

\- Write good documentation. This is so often overlooked. People generally contact support as a last resort, and will usually read all documentation in advance. If your docs explain it, they won't need to contact you.

\- Time: it usually takes me 10-30min to answer one support request. I don't have a time budget, I try to answer as quickly as possible. Being responsive makes a great difference to your customers, and they will be much more likely to buy your product and recommend it to others.

\- However, to limit the time per week I spend on customer support, I do everything I can to make sure people don't need to contact me at all. I try to fix every reported issue permanently so that I never have to deal with it again.

<comment><USER6> I agree with most of what you said, and try to follow a similar approach as well. Especially pre-empting support issues by fixing the product. That said, I'm not convinced about documentation - a large percentage of enquiries and support calls we get could have been resolved by checking the documentation, or even just reading the landing page of the website. I'm still not sure how to efficiently resolve these.

<comment><USER7> With documentation it will only be used if it is visible. If they have to exit your program and go to a website and search it won't be used. If you have a prominent (not hidden in a menu) search for help option it will be used. Assuming the search function works well.

<comment><USER6> The documentation is HTML, but it's linked directly in the app, so you can get relevant information and tutorials about the function you're using.

Most of the time these inquiries are related to "can it do X", rather than "how do I do X" \- my best guess is that they want something in writing (therefore passing the responsibility for making the claim to us), rather than doing their own reading and confirmation. It still takes a fair bit of effort to answer all of these.

</comment></comment></comment></comment><comment><USER8> You start by doing it yourself. This is super important. Doing support brings you closer to your customers to understand their pain points so you can build a better product.

Technical question that they don't get? Either make it more explicit in your product, on your sales page or in your documentation. Once that's solved you should better understand customer needs and have fewer support tickets coming in.

Billing can be a little more painful but with a solid FAQ and some email templates over time this should resolve itself pretty quickly.

Non-product related enquiries (usually like billing) are the first ones you might outsource to someone else to handle. You can use services like influx.com to handle this.

</comment><comment><USER9> 1) Sign up for intercom.io. It's genuinely the best customer support tool I've used.

2) Get the app and get a notification for every user's question. Try to answer them the second you see it.

3) Don't hire a dedicated customer support person. Stay close to your customers and build a relationship.

4) Watch this talk by Patrick Collison from Stripe: [https://www.youtube.com/watch?v=nnllRegL_NI](https://www.youtube.com/watch?v=nnllRegL_NI)

5) Build out an FAQ

6) Make sure your customers are happy.

<comment><USER10> I also wholeheartedly recommend intercom.io

<comment><USER9> They are, hands down, my favorite company at the moment. Been a loyal customer of them for about a year now. One of the few companies I will go out of my way to recommend to people.

</comment></comment></comment><comment><USER11> I'm the co-founder of a company that provides "elastic customer service" (influx.com) and I'm happy to share with you what I see happening in practice.

Most founders start out doing customer themselves. This is absolutely the right thing to do for reasons outlined elsewhere in this thread.

When the business grows, the the founder(s) can't continue handling 100% of customer support, both because there other priorities and because the volume grows beyond what the founders can personally handle.

Generally in this "transition phase" time-to-first-response grows steeply.

I wrote about my experience providing customer support on a open source project, you can see where the "wheels started to fall off" \- ie time to first response climbed steeply:

[http://moniker.net/2013/07/10/two-open-source-emails-a- day-f...](http://moniker.net/2013/07/10/two-open-source-emails-a-day-for-five- and-a-half-years/index.html#time-to-first-response)

At the time, I felt bad about TTFR growing steeply. I now understand that it's a really common state of affairs.

So at some point, the founders need help, and they either hire or outsource, or do a combination of both.

You asked about "the average time it takes you to resolve customer queries". A typical time-to-first response before Influx starts working with a client is 12-15 hours, with a outliers at 1-2 days.

On "stay close to your customers" \- it's 100% correct but a harder question is "how do I stay close to my customers and scale at the same time?". The answer depends on the size of the business. I see people using a combination of metrics and qualitative insights. The metrics look after big picture health and qualitative insights involve customer support staff bubbling issues back up to the product+dev teams.

</comment><comment><USER12> Great businesses provide great customer service

Some principles worth following:

0) Be human - personable, friendly and honestly there to help

1) Be honest and transparent with technical issues

2) Be fair with billing issues

3) Be effective. Fast responses don't help if the question isn't answered

4) Empower your team to do the right thing - most big co's fail here.

5) Keep the feedback loop between yourself and your customers tight - everybody on the team should help out on support. Lots of valuable insight can be gained here. While outsourcing makes "support questions go away" it also cuts off an important feedback loop.

<comment><USER13> These are good points, but for point 3, I chose a hybrid solution. Reply quickly that your team is looking into the issue/ticket and reply later with the real answer.

From personal experience with other's support systems, sending support emails and not getting an immediate feedback doesn't feel great

<comment><USER5> Make sure that the initial answer is from a human, though. Your customers need to know you've received and understood their requests. An automatic reply doesn't help much.

<comment><USER12> Agreed. Automatic acknowledgement emails tend to result in disrupting phone notifications with very limited useful value.

Where auto acks are are valuable, however, are for emails received outside of business hours. This sets expectations that you may not receive a response until the next business day.

</comment></comment></comment></comment><comment><USER14> Yes, have a good FAQ which becomes your canonical reference point. [http://dabase.com/blog/How_to_create_a_FAQ_that_does_not_suc...](http://dabase.com/blog/How_to_create_a_FAQ_that_does_not_suck/)

I have a fairly complex Fastmail email setup in my organisation to share the support mail folder with my colleagues. Ultimately all people added to support can see all past "conversations" and join the conversation. Huge PITA to setup though. Still prefer it as it's not locked into the semantics of some ticketing system evolving to be email.

</comment><comment><USER15> Support for WP All Import is one of our biggest selling points. We get a lot of support requests in part because our software is complex and WordPress has a low barrier to entry.

We have three part time employees and one more in training all to handle support. We did, however, start where you are. Here's my advice:

\- Just use email. We tried Zendesk, Groove, and Helpscout. Helpscout is far and away the best.

\- Your reps need to be smart, talented, and close to the business. That means you need to handle all support inquiries until you can afford to hire someone. When you do hire someone, make sure they're 5x more technical than you think they need to be. We hire developers as support reps.

\- It takes us on average 2 replies to close a ticket. A reply takes us on average 6.5 minutes to write (some take 30 min, some take 30 seconds). Our tickets involve lots of debugging and weird stuff, so YMMV.

\- Answer every support request as fast as possible. Our customer happiness ratings have a lot to do with how quickly the reply was sent.

\- Treat support like a focus group. In a perfect world no one would ever need to ask you for help. If you get people asking the same question over and over, change your product to answer their question for them.

\- Docs are a last resort. It depends on the product, but often documentation is a band aid for bad UI. That said, it's good for SEO.

\- As you grow, keep support close to your ear and heart.

<comment><AUTHOR> > We hire developers as support reps.

Wonder why I never thought of it. Hiring a programmer to do customer support is a really good idea since for the sites we have a dedicated customer support guy (outsourced) they often come back to us to explain trivial questions like "how to embed a code", "how to publish on site X", etc which can be easily solved when someone knows a little programming. They can also offer some creative solutions which instead of just following a script we give them.

I will def keep this in mind when we need to hire next. Thanks.

<comment><USER15> It doesn't make sense for every product, but for us it's absolutely necessary. In general I think it's really helpful to have your support reps be 5x more technical than your average user.

</comment></comment></comment><comment><USER16> When you're a small business, it's important to establish repeatable methods that customers and your support team can follow easily. Do not over complicate. Find the single most efficient and scalable delivery method and perfect it. Start with email based support. Build your reputation as a dependable, knowledgeable and responsive machine through that single channel. You'll later be able to expand to the other channels but not until you can afford it.

Depending on the size of your business I would suggest some tips from this blog post:

[https://www.monsooncommerce.com/2015/11/customer-service- sof...](https://www.monsooncommerce.com/2015/11/customer-service-software- companies/)

</comment><comment><USER17> I think a chat widget on your site is a convenient way to offer support that works for a single-person company or a dedicated support team. We use Zopim on our site. It's the only way that I know of were one support person can offer real-time help to multiple customers at the same time. (or at least give that perception as long as not too many people are chatting at once).

Having a knowledge base and/or documentation is a great because a lot of people do prefer to help themselves. That cuts down on the volume of support calls too, so it's an easy win.

I personally think that support and sales go hand-in-hand so, especially when you're getting started, you need your customer's support experience to be excellent.

</comment><comment><USER18> To me it is key that no-one gets between me and the customer. That feedback is essential for improving my products. So I've done all my own customer support for the last 10 years. The challenge is how to provide great support without being swamped.

I probably spend 30-60 minutes per day replying to support emails (I haven't measured it). Most take less than 5 minutes.

Here is an article I wrote that summarises what I've learnt: [http://successfulsoftware.net/2012/08/21/tips-for-great- soft...](http://successfulsoftware.net/2012/08/21/tips-for-great-software- technical-support/)

</comment><comment><USER19> Be real. Make relationships with customers on first name basis. Specially when the customer is from SMB or a startup.

I have my customers on my skype/email/intercom. We have looong threads with them, discussing features and related things.

</comment><comment><USER20> I run romanhaze.com - we sell eliquid for vaporizers.

Support and questions are handed through email. We have a contact form and various internal bits will direct a customer there with preselected subjects (e.g. "Question for Order #n" ).

I would never outsource it from the company, and want to have me and my co- founder primarily do it. It's the closest contact you have with users who are experiencing problems. If you're big enough, you're going to need dedicated support people, but I think founders should still have a grasp of what customers are asking you about.

</comment><comment><USER21> I run www.nusii.com and we are still quite small and we handle Customer feedback ourselves. Se use it to get to know our customers. We have some saved replies we use over and over again, and we link to tutorials where we can. I think we take 30/45 minutes a day to do customer feedback.

I must say, great customer service helps churn a lot. Helping them out actually makes them a happier customer and it is a great moment to start a good relation. See it as a opportunity and don't outsource it too soon!

</comment><comment><USER22> Intercom is by far the best customer support tool I've used (and I tried 10+).

We use it both as a widget on our landing page and our support email redirects there, which allows us assigning tickets effectively to the team member responsible for their resolution.

Interesting fact: Because our median response time on Intercom is around 12 minutes we convert about 80% of all inquiries (from visitors to paid clients). 80%!

Bottom line: customer support makes a huge difference, so you should definitely do it yourself.

</comment><comment><USER23> This is something I'm really passionate about. I'm a solo founder of a service (SSL installs) that has a disproportionate amount of onboarding technical support issues that absolutely have to be resolved or people immediately quit.

When I started, I was really concerned that support requests would be overwhelming. It's a direct time sink (time you could be using for marketing or development) and when you're getting off the ground it is really difficult to predict how much time it will take. To some extent being 'swamped with support' is a nice to have problem as it implies that you at least have customers or at least people trialing your product.

So, it is with a fair deal of surprise that I've come to really enjoy providing support for ExpeditedSSL, that support has become a huge driver of word-or-mouth referrals and has driven any number of big user experience improvements.

WHY DELIVER GREAT SUPPORT Beyond, just trying to keep your customers sort of generically happy, there's two really important reasons to try and nail support:

1\. Support Effort is Weighted to New/Trial Users

To get a new user of your software you have to attract visitors, teach them what you do, maybe email them a few times and then after they signup and you've expended all of that effort and expense it's crazy to throw away an opportunity to keep them.

Early user support is incredibly lucrative as it's the difference between keeping a trial customer and them leaving. A 5 minute email to a new user that keeps them around for a year of service can easily earn you hundreds of dollars.

2\. Good Support Is How You Get Fans

With rare exception, people seem to get more upset over support issues (unresponsiveness) than from actual issues.

Most places are so bad at support that if you're able to respond quickly, fix their problems and in general not act like some stiff support robot you'll make people really happy and huge fans.

This effect is so pronounced that I almost wish we could have some minor issue occur in the onboarding process that we could reach out to people about.

This is an area where you as a startup actually have a profound advantage over larger companies as you're likely much more knowledgeable about your product and definitely more invested in a user having a good experience than some random for-hire support person at BigCo.

WHAT TO SUPPORT It's easy to lump all post sale activity for a customer into 'support', but splitting it out into a few broad categories helps address the underlying issues in each.

Technical Support

This is what most people consider "support". A user reaches out with an error and after some troubleshooting you tell them how to fix it.

Onboarding/Usage

Not necessarily that there is an error, but perhaps they don't know what to do next or need assistance in getting their data loaded or preferences setup.

Customer Service

Typically billing, renewals and anything else regarding the service outside the core functionality.

HOW TO DO SUPPORT

EMAIL

Your primary means of supporting users is likely to be email. It's asynchronous but lets you respond quickly and starting out will likely be sufficiently organized to keep things moving without a full blown helpdesk application.

Secondly, it lets you be extremely precise and copy & paste friendly in a way that phone calls can't match. Ex: we often have to help people set their DNS entries and it's much easier to email someone the following than explain it.

Please set your www CNAME to fugu-2034.herokussl.com

Email is in fact so predominant as a support tool that you can actually go quite far with managing support requests in your email client before you need a dedicated support app/service. Modern email clients that nicely group emails into threads are more than adequate to get started.

A few months after starting I switched from a dedicated Gmail account to Fastmail and it made a profound difference as just receiving and responding to emails was faster by about 10 minutes on each side.

SKYPE & PHONE

We primarily use Skype and Phone calls as an escalation method. If someone is really turned around conceptually as to what needs to happen to get things working; often a short phone call can put things straight.

I'm also always on the lookout for users phone numbers in their email signatures. I'll sometimes just call them back if they email in with a question or problem as this is such a huge inversion of their expectations (being left on hold with instrumental covers of ABBA tunes playing and a disembodied voice that talks about "call volumes" and how you're "very important to us") - that they'll hopefully feel really positive about the experience.

CHAT

I had really high hopes for chat, and had tried using Olark for several months, but whatever combination of user experience and expectation that came through, just did not get any takers.

I'd really like to offer real-time chat as a support channel and may try it again in the coming year.

TWITTER

Some developers still look askance at Twitter "If I'm building a business, why do I need to know what people had for lunch. Har Har." \- but it is undeniably a support channel as people now have an expectation of being able to say: "Hey COMPANY_NAME - I'm having trouble" and get a response.

Further, while all the other support mechanisms are at least within your control (aka you're not going to get support phone calls if you don't have a support number) - Twitter is most definitely not. People will tweet about your product or service and the issues they have with it at the drop of a hat.

To help these people you need to monitor Twitter, I setup a custom column within TweetDeck that searches for ExpeditedSSL and "Heroku SSL". Which makes this a breeze.

WHAT TO SAY Whatever the support channel that a request comes in from, I try to incorporate the following elements of what I've found makes a great support experience:

1\. Apologize for them having had to waste their time: "Sorry you have to deal with this hassle"

2\. From the context make sure that they know it's a knowledgable live human that is invested in fixing their issue; not a NLP auto-responder script, not an intern with a list of set support responses.

The easiest thing to do is just look at their email address or signature for their name and put that in your salutation.

3\. Explain what they need to do in the exact way that they need to, this means that I'm not cutting and referencing things like "set the dns cname to example.com" \- but actually put their domains, names, servers, etc. into the instructions.

4\. Try to explain what caused the issues. We sometimes have issues with our upstream cert providers, sometimes weird things happen with the Heroku API, once I forgot to buy more credits (I have to purchase certs ahead of time). I just try to be really honest about what happened.

5\. Sympathize - we're really lucky that all of our customers are genuinely competent developers. But they don't typically do DNS work or other config tasks - so they feel bad about not knowing what to do.

6\. Anticipate - beyond just fixing their current issue, point them to the next step that they need to accomplish.

MAKING A SUPPORTABLE SERVICE Some of the highest ROI time I've spent on ExpeditedSSL was in user experience reviews. I'd ask a friend out to coffee (or random people on Twitter to join me on Skype) and just ask them to try and get through a SSL Install. I'd watch their progress and flich and be incredibly embarrassed as they got stuck on seemingly "obvious" steps.

Some of the companies I've worked for have spent tens of thousands of dollars on doing customer feedback and user review studies.

So with that in mind, it's quite reasonable to consider every support case as a free, mini lesson on UX that a real actual user has sent you. As a direct consequence of support cases we've:

\- Added a "Test Email" button the the approver addresses

\- Automatically force the best option for 'www' vs naked domain names

\- Improved the DNS checking to say both what should and what should NOT be done with DNS

\- Added post install instructions for configuring the most popular app-stacks for forced ssl.

Together this forms a feedback loop, where as you improve the product you get fewer and fewer support issues that you can then proportionally handle better and better. To help encourage users to give feedback I always put my title as 'Developer Support' so that they can feel more free to complain to me about product issues than if I had my CEO or Founder hat on.

<comment><USER24> > A few months after starting I switched from a dedicated Gmail account to > Fastmail and it made a profound difference as just receiving and responding > to emails was faster by about 10 minutes on each side.

Would you elaborate on this point please? 10 minutes?! Each side?!

<comment><USER25> Presumably greylisting? does gmail do that? [https://en.wikipedia.org/wiki/Greylisting](https://en.wikipedia.org/wiki/Greylisting)

<comment><USER26> I used OpenBSD's spamd (not to be confused with spam-assasin'g binary). Spamd greylisting is extremely effective but hard to configure properly. It assumes that every SMTPd[1] out there respects the RFC: When a mail is rejected for whatever reason, should be sent again using the prior IP address to get whitelisted. Google and nearly all corps/mail providers switch IP address when they re-send an email. It's probably a measure to circumvent IP-based blocks, but it breaks OpenBSD spamd's implementation. If you're in a hurry or expecting an important email, you're out of luck. It might take 12-24 hours before the mail finally reaches your mailbox, which is simply not acceptable.

That said, the amount of spam using spamd fell to nearly 95% on the other hand I had to whitelist manually 81 IP addresses in ~ 6 months.

</comment></comment></comment></comment><comment><USER27> First of all, _billing_ questions need to be segregated to sales or accounts receivable, as they are not technical support issues.

Secondly, how many of these queries are related to customer education / misunderstanding of product use vs. actual software defects? The former may reflect weak or inaccurate documentation or conflicts in design.

</comment><comment><USER28> Intercom.io for all your support and customer tracking needs. It also handles DRIP[1] e-mails and doubles as a sales live chat.

[1] [https://en.wikipedia.org/wiki/Drip_marketing](https://en.wikipedia.org/wiki/Drip_marketing)

<comment><USER29> Don't do this if you consider your "customers" to be human beings (vs. resources to be exploited). Also, don't do this if you want them to keep paying attention to your desperate and cynical begging. If you "drip" them, most will just add you to their spam filters.

<comment><USER30> Just to clarify: Intercom itself is amazing and the best way on the Internet to support your customers and treat them like humans. The problem is drip campaigns, which is only a small portion of Intercom. (And, Intercom's are triggered by actions, which makes them better than the average drip campaign).

(Nope, I don't work for Intercom. Just a huge fan.)

<comment><USER31> Intercom might be amazing, but the huge security issues it usually introduces aren't.

I'd rather not let a bunch of random companies inject arbitrary code onto my website.

<comment><USER32> What about externally hosted JS libs or ad/tracking scripts? Yes, you can do without all of them, but I don't think that's the case on most sites.

<comment><USER31> You can (and mostly, should) do without externally hosted JS libs.

Ad/tracking scripts is where it gets interesting though, although I'd argue for in-house analytics ads are often a major source of income and therefore business critical.

In the end it just comes down to trust, but keep in mind that if you trust Intercom you're also trusting www.icb.co.uk (which just happens to have several persistent XSS vulnerabilities in their ticketing system that could be quite trivially be used to hijack intercom.io), AWS, gandi, google, linode (seriously?), optimizely.

And that's just a quick 5 minute audit of the critical infrastructure providers for intercom.io, I strongly believe that if any of these were compromised the attacker could trivially take over any site that is including intercom code on their page. And don't forget, this could also be done recursively, for example optimizely uses edgecast. An attacker that compromises edgecast would be able to insert code into intercom.io.

Google fares much better against such inspection, as it relies much less on external infrastructure providers.

And since there's no real reason for intercom.io to require users to remotely include their code, I'd avoid using them until they fix their product.

<comment><USER32> Thanks for elaborating.

What do you mean by them "fixing their product"? Make it self-hosted/server- side?

<comment><USER31> Only the javascript code (and any html) would need to be self hosted. It could still communicate with intercoms API servers, just not load code from remote locations at runtime.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER33> We keep it simple, accepting only email and phone support.

\- For emails we use Zendesk (zendesk.com) \- For phone support we use Toky (toky.co)

(I'm the founder of Toky, and we dogfood our solution :))

</comment><comment><USER34> I use email, but I also make a FAQ entry to address every question I get. I got the idea from the 4 hour work week.

</comment><comment><USER35> HelpScout + outsourced tier one support

<comment><USER36> NEVER outsource support. Never-ever lose focus on your customers needs by adding another layer of people and software.

<comment><USER35> There's lots of support functions you can outsource. You just have a preconceived notion of what "outsourced support" means.

</comment></comment></comment><comment><USER37> Disclaimer: We run a Heldesk software, Busibud.com which maps out a company's customer support process and helps them outsource only the mapped out part. The helpdesk software is free to use.

I've been helping companies setup their customer support processes for some time now. Initially, founders/core team members must do customer support on their own. This should happen till your team has some visibility of the process (what kind of questions you get and how to reply to them).

\</comment><comment>- What are the ways to handle customer support?

You can do it synchronously (chat or phone call) or asynchronously (email). Usually, async customer support requires lesser money/time because things can be structured. But, before you have some visibility of the process and are still finding p/m fit, I would suggest you talk to people.

\</comment><comment> Outsource vs do it yourself?

Before you know your process, do it yourself. No one can decide how things can be answered. Once the process has been mapped out, a lot of the startups I know of ask every team member to pitch in for customer support. This may lead to better quality but leads to huge costs as well. I've seen salesmen spending 1-2 hours everyday on customer support answering what they call "dumb questions". You can outsource it as well, which comes with problems of its own. For starters, it will be very hard to find an outsourcing partner who will be ready to deal with extremely small volumes.

Even if you do manage to outsource, your process will now be managed by someone else. Ad-Hoc changes become difficult to communicate and the people giving out support for you will also be giving out support for other companies....hence leading to quality issues. Unfortunately, most outsourcing companies still rely primarily on training to ensure quality. They don't use the technology they should be using. Thus, quality will depend on the infrastructure of the outsourcing partner (training managers, quality managers, etc).

Once you reach a decent size, hiring people and creating your own customer support team is an option but is usually more expensive (HR/payroll/training/infrastructure). But, it also leads to better quality. Do remember that customer support teams have a huge turnover rate...people keep going and coming in. Thus, training and quality monitoring costs can be huge!!

\</comment><comment> What is the average time to resolve queries?

I've seen a first response time of 5 minutes and also first response times of a few days. Don't know what the average is but depends on a lot of factors. The one doing it in 5 minutes usually has happier customers.

\</comment><comment> How much budget do you allocate for CS?

This is a tricky question. Even huge companies have a problem with this. The amount of budget to be allocated would depend on a lot of factors. What I would suggest is that your get started with a particular level of resolution time (which would determine your budget) and then try to optimize it using a series of A/B tests with retention/return being the metric being monitored (the metric too can depend on a lot of factors). Once you use these metrics, you would have a better idea of how much budget you should allocate. I might be wrong, but Busibud.com is perhaps the only customer support tool that I know of that lets you A/B test customer support strategies.

<comment><USER37> When it comes to tools and customer support problems of a growing company, here is a post I found to be accurate for a lot of the cases I've seen: [http://blog.outreach.io/why-we-switched-from-intercom-to- zen...](http://blog.outreach.io/why-we-switched-from-intercom-to-zendesk-and- almost-chose-desk-com/)

</comment></comment><comment><USER38> I run [http://www.wordfence.com](http://www.wordfence.com) \- We provide customer support via Freshdesk for our paid customers (a ticketing system) and on the wordpress.org forums for our free customers that use our open source product.

I initially did CS myself along with my co-founder and it is tough work. The hardest is the context switching overhead. One minute you're coding or marketing, and then you have to switch into doing 2 or 3 hours of customer support. What is really tough is at the end of the 3 hour stretch, a customer with a real gnarly issue comes along and you have to do 30 to 45 minutes of actual work researching it to help them. You have to dig deep to go that extra distance.

Once our revenue scaled we could hire our first full time customer support engineer and wow. It was so awesome to have someone offload this and let us focus on all the other stuff (which we would eventually have to hire for too!). We have a team now, some permanent and some contractors.

So there's no easy way to do it and because it's a human intelligence task, it's super time consuming and super expensive. But I can't overstate the importance of great customer service. We are known for it in our space and it has really paid off. Competitors have dropped the ball in a big way and we have picked up the slack through excellence in customer service.

A few things I learned:

\- Reply early, reply often. It's very important to most customers to simply be acknowledged.

\- Everyone has a bad day and 90% of the time, a polite response to a very angry customer will get you a surprisingly polite and grateful response back.

\- Twitter is great. Reply to customers, interact, but direct them to ticketing or forums for full customer support service.

\- Ticking system is essential. It's the only way to reliably track issues. Support via email and you will eventually drop the ball and it will bite you.

\- A FAQ will save you a huge number of support tickets. Put it front and center, maintain it well, make it as user friendly and easily navigable as possible.

\- Only hire people who speak your customer's language as their first language. Even the smallest grammatical errors are giveaways of offshoring and will irritate customers.

\- Depends on what kind of biz you run, but in our case our customers are surprisingly understanding when we have an autoresponder saying we will be providing limited support over the holiday season or over weekends. So pick your hours, and just communicate clearly what they are.

Hope that helps.

Edit: You asked about average time to handle customer queries. A ticketing system gives you that kind of reporting along with a ton of other metrics. I would think the average time to close a ticket varies depending on the business and product complexity and customer sophistication.

</comment><comment><USER39> I unlist my contact details, and only people who can do a WHOIS contact me. Then I talk to those like a regular person. If the system is designed optimally, it should be inherently supportive and most people won't need help. But it totally depends on your specific business and goals, which you didn't explain.</comment>


How to Seem (and Be) Deep   - <AUTHOR> http://lesswrong.com/lw/k8/how_to_seem_and_be_deep/ <comment><USER2> _If you want to sound deep, you can never say anything that is more than a single step of inferential distance away from your listener's current mental state._

This is important for more than just sounding deep. Many communications require calibration to the audience to be effective. An advocate who neglects this appears to be a crank, speaking mostly to indulge himself rather than convince others. A teacher who neglects this confuses and eventually loses his students.

And yet, on the net, where any communication can be excerpted out-of-context and forwarded to new audiences, every calibration will be 'wrong' for some readers. Opponents of a viewpoint or teaching will, in fact, be motivated to forward low-context excerpts to alternate unintended audiences, to discredit rival speakers.

I don't know an easy solution. Maybe audiences will wise up about this cheap forward-out-of-context maneuver, and demand better of their aggregators/forwarders/filters. Or maybe persuaders will be pushed to specialize in certain narrow ranges of the inferential chains, and avoid saying anything to other audiences that could possibly be redeployed to alienate their usual audience. (For example, mass majority politicians.)

<comment><USER3> There's a third possibility: technology could begin to easily (or automatically) provide context, without audiences first wising up to the particular problem you describe. There are plenty of incentives for developing this sort of thing -- e.g., you'd revolutionize the news industry.

</comment></comment><comment><USER4> I remember more than a few all-nighters, back in my college days, on big papers due the next day. Nevermind that I'd neglected to start them earlier; the problem at hand was coming up with a thesis that was provocative enough to be engaging on two fronts. First, it had to engage me in order to keep me writing. Second, the product of my writing had to engage my professor in order to stand out from the pack. Being "deep" was an easy trick to accomplish both degrees of engagement.

I didn't have a conscious method at the time, but when I look back at my madness, I see a few oft-used patterns. The first was to pick a statement in a text -- preferably one held as common knowledge about the subject -- and disagree with it. Another technique was to take that same statement, and graft on some out-of-left-field, tangential implication from it (my college pal Mary Jane was great at helping me with this kind of thinking). Yet another trick was to take two authors or arguments believed to be in opposition to one another, and claim that both were more closely aligned than commonly assumed. There were endless variations on all of these methods, but all of them shared the same logical scaffolding outlined in this article.

All of them were problematic.

You see, when you start with the explicit _goal_ of "being deep" or "making a provocative argument," you run a huge risk of bullshitting for bullshit's sake. And if you're a great speaker, or a deft wordsmith, your bullshit will sound clear and striking enough to convince anyone who hears it. Even yourself. Soon enough, you become the sort of person who can start with pretty much any claim, no matter how superficially absurd, and then bend sources, and arguments, and logic, and everything shy of thermodynamics, to prove it out. This is intellectually lazy, because the entire point of writing a paper, or proposing a theory, or researching a topic, is to do the up-front legwork of hypothesis-test-refinement. Deliberate provocation skips the "test" portion of that pattern, opting instead for style over substance. It predetermines a conclusion, then makes the conclusion so shiny and pretty that everyone assumes a great deal of thought went into reaching it.

All of this is, of course, tangential to the author's point. It's also based on personal experience -- though I have been around enough wise-ass buddies in my day to know that I'm not the only world class bullshitter ever to hammer out a college paper. But my point is that I find the idea of setting out to be "deep" a bit troublesome. Deep _thinking_ is fantastic. But "deep" presentation is easy to achieve without deep thinking, and it passes muster 99% of the time. It's like Frodo's ring: very powerful, very easy to slip on, and very hard to take off.

<comment><USER5> The article didn't do much for me, but this comment did. What you write about rings very true in my experience of writing papers and having arguments without a clear goal. One of the better HN comments I've read for a while.

<comment><USER4> Thank you.

The big moral challenge for me was not that I could bullshit my way, quickly and successfully, through any single paper. Rather, the problem was that this type of work became _very_ easy and, eventually, routine. I found myself applying the principles of "deep" bullshit to my actual problem-solving in real life, and they led me astray more than once. I had to take conscious stock of what had become a subconscious pattern, excise it painfully, and learn to rebuild my thought process.

So I guess my overall point is that "How to Seem Deep" and "How to Be Deep" can be two very different things. The trick is recognizing when one is seeming deep, and when one is being deep, and not letting the ease of the former replace the hard work of the latter.

</comment></comment></comment><comment><USER6> It's funny, because this seems like it gets you a lot of upvotes on Hacker News as well. The top comment on an article will often be a coherent and (seemingly) original rebuttal of the articles main points. And then the top response to the comment will be a coherent and original rebuttal of the comment.

</comment><comment><USER7> What? This article should be titled how to sound sophisticated (possibly pretentious) but not "deep".

You want to talk deep? Think things through. Go for the WHY of everything - from technical systems to relationships to art to everything else you can see, touch, feel, and comprehend. After some time, you will develop basic foundations of why and how systems (technical and people) work. Then you will want to generalize everything into rules and laws. After that into natural laws. Pretty soon you'll be seeking the first principles in everything. Only then can you be deep.

Once you have _some_ understanding of the world around you, you'll want to apply it, test it out. See what works and what doesn't. DO THIS! In fact, this is arguably the basis of an entrepreneurial mindset.

\-----

"Seem" deep?! YA OK.

 _Think wrongly, if you please, but in all cases think for yourself._ -Doris Lessing

<comment><USER8> You should really pay attention to the last two paragraphs.

<comment><USER7> I did read the last paragraph. IMHO, Its too cyclic (link wise) and ambiguous to be of much use. But if it really was what the authors main agenda was, then props to them...except why in the very last paragraph? What is everything before it? What is this 'seem' business?

You shouldn't write down every answer in the book and hope the reader will pick the right one. Thats a multiple choice test, not an informed article.

</comment></comment></comment><comment><USER9> If you want to seem deep attempt to learn to speak E-Prime fluently, you'll have to use a lot more vocabulary to get your point access.

<http://en.wikipedia.org/wiki/E-Prime>

<comment><USER10> Seconded. Take it with copious grains of salt, but I've found that it improved the quality of my writing significantly. I no longer reach for the laziest verbs. More importantly, I reduce the number of unsupported assertions, and I find myself writing better justifications and evidence.

</comment></comment><comment><USER11> >Human brains only run at 100Hz and I responded in realtime, so most of the work must have been precomputed.

Is the 100Hz some metaphor I'm not getting? Or is it something that's actually a fact? If it's the latter, does anyone have more informations about this, because it sounds really fascinating and I'd love to learn more about it.

<comment><USER12> It is roughly correct, different neurons propagate signals at different speeds but it is in the ballpark. However, the brain is massively parallel so the slow clock is not evidence for it precomputing stuff.

<comment><USER13> We can evaluate a lot of things that are one inferential step away, because of the parallelism. But we're very bad at bridging more than a few inferential steps because parallelism is almost of no use there. It's very difficult for a human brain to think a few chess moves ahead, but incredibly easy to dismiss a thousand obviously stupid moves in a fraction of a second. The human brain has to precompute almost everything in order to function.

</comment></comment></comment><comment><USER14> Pretty sure Eliezer missed the #humblebrag tag off that post.

</comment><comment><USER15> Stop caring about what other people think

<comment><USER16> I will once I'm Warren Buffet rich and can move to my own island.

<comment><USER17> I hope your island has nice healthcare, fiber, and 3G towers.</comment></comment></comment>


What 'Once in a blue moon' literally means - <AUTHOR> http://www.thisisbrandx.com/2009/12/once-in-a-blue-moon-this-new-years-eve.html <comment><USER2> Pretty cool. Question though - the definition of a blue moon is for there to be 4 full moons in a season.

(Learned on HN):

<http://news.ycombinator.com/item?id=1008761>

which cites: <http://en.wikipedia.org/wiki/Blue_moon>

If that is the case then this is not a blue moon as the first 3 happened in the fall and the 4th is after the solstice. Right?

<comment><USER3> If you had read the Wikipedia article for yourself and you'll find multiple definitions of a blue moon.

To save you the effort, _The Farmer's Almanac_ used the definition you just cited, but "2 full moons in a month* is the more commonly used definition.

<comment><USER2> Yeah you are right.

Or simply gone to nasa.gov:

<http://science.nasa.gov/headlines/y2009/29dec_bluemoon.htm>

"The modern definition sprang up in the 1940s. In those days, the Farmer's Almanac of Maine offered a definition of Blue Moon so convoluted that even professional astronomers struggled to understand it. It involved factors such as the ecclesiastical dates of Easter and Lent, and the timing of seasons according to the dynamical mean sun. Aiming to explain blue moons to the layman, Sky & Telescope published an article in 1946 entitled "Once in a Blue Moon." The author James Hugh Pruett cited the 1937 Maine almanac and opined that the "second [full moon] in a month, so I interpret it, is called Blue Moon."

</comment></comment></comment><comment><USER4> This one's pretty cool since it's happening on the last day of the year and all. Although, by definition, they can only happen in the last few days of the month, so it's a little less special than I first thought.

I also remember a few years back, there were two blue moons only two months apart, something that only happens every couple decades. There was a full moon in early January, a blue moon on Jan 31st, then the tiny month of February managed to fit in between full moons, so the next full moon happened on March 1st, with a blue moon later that month.

</comment><comment><USER5> * hey?), some believe that a full moon -- not to mention a blue one -- can cause strong emotions to flare. Some evidence shows that arrests for petty crime and public drunkenness increase 5% during a full moon*

Anyone know why? Any ideas on the causal effect? Pure chance?</comment>


Startups clear out cash registers to usher in retail revolution - <AUTHOR> http://venturebeat.com/2013/03/28/startups-clear-out-cash-registers-to-usher-in-retail-revolution/

<comment><AUTHOR> This article is mostly about Square; it really doesn't discuss what other startups are doing in the "retail revolution".</comment>


Ask HN: What are some solutions for addressing income inequality in SF? - <AUTHOR>

I would like to pick the collective HN brain on how we can address income inequality issue not only in Bay Area but also in other cities around the world.
According to Brookings Institution(http://www.brookings.edu/research/papers/2014/02/cities-unequal-berube) SF ranks in the top in terms of income inequality. SF Chronicle compares SF with Rwanda for having the similar income inequality - Gini Coefficient (http://www.sfgate.com/bayarea/article/Income-inequality-on-par-with-developing-nations-5486434.php)
What are some solutions that can address this problem?
A few people are getting together to discuss about this issue tomorrow.
http://www.notforsalecampaign.org/reinventsf/ I am looking for different perspective from HN community to address this issue.
Thanks in advance! <comment><USER2> I think I have a way to cure poverty. What if we created a national auction system for low wage labor. The person is paid $11.15 an hour(poverty rate). Their time is auctioned off. They can refuse an offer but the difference between an offer they take and the highest offer is subtracted off their pay. They can work less than 40 hr a week but again subtracted off. And it's paid for by a national sales tax. So how much would cost? To pay the 10.5M unemployed cost 234.3B. Subtract unemployement benefits 116B. Bring Minimum wage and lower to $11.15/hr. 3.6M cost 28.8B The rest of the people in proverty 31.9M assuming avg pay at 9.15 hr. 127.8B. Total of 275B.

National Sales tax of 3.05% on 9T personal consumption(assumes not taxing unprepared food, housing etc.)

<comment><USER2> My guess is it could be done at a city or state level with similar % sales tax. As for medical services I think you people pay a % of their income for insurance.

</comment></comment><comment><AUTHOR> Including links for easy clicking:

Brookings Institution Research: [http://www.brookings.edu/research/papers/2014/02/cities- uneq...](http://www.brookings.edu/research/papers/2014/02/cities-unequal- berube)

SF Chronicle article: [http://www.sfgate.com/bayarea/article/Income- inequality-on-p...](http://www.sfgate.com/bayarea/article/Income-inequality- on-par-with-developing-nations-5486434.php)

</comment><comment><USER3> The only proposed solutions seem to be taking from the people earning more through taxes and giving it to the people with less, reducing the "inequality".

I would like to solutions that don't involve this.

</comment><comment><USER4> SF is geographically unique as is NYC... as concentration of wealth focuses on such areas it has a exponential effect on value of goods and services in such an area. There is not much you can do about it.</comment>


My battle to question doomsayers about the Great Barrier Reef - <AUTHOR> http://www.foxnews.com/opinion/2018/02/08/science-or-silence-my-battle-to-question-doomsayers-about-great-barrier-reef.html <comment><USER2> This guy has presented extremely little evidence in the article to support his argument. That, coupled with it being on a tabloid leads me to doubt him. A quick search on his name just reveals some recent disagreement he had with JCU but, again, very little supporting evidence from his side. Can anyone find more?

<comment><USER3> Well, from his homepage/portfolio on JCU he doesn't seem a "casual passer by" or an "armchair critic":

[https://research.jcu.edu.au/portfolio/peter.ridd/](https://research.jcu.edu.au/portfolio/peter.ridd/)

The article (co-authored with Piers Larcombe) that backs the statement in the interview that seemingly started it all is linked to among the other papers related to the litigation:

[https://platogbr.wordpress.com/serious](https://platogbr.wordpress.com/serious) misconduct /

it is this one (pdf):

[https://platogbr.files.wordpress.com/2017/12/larcombe-and- ri...](https://platogbr.files.wordpress.com/2017/12/larcombe-and- ridd-2018.pdf)

It doesn't seem to me at first sight particularly "aggressive" or "anticollegial", right or wrong I have seen several similar arguments about the "bad quality" and the "non-reproducibility" of studies:

ABSTRACT Research science used to inform public policy decisions, herein defined as "Policy-Science", is rarely subjected to rigorous checking, testing and replication. Studies of biomedical and other sciences indicate that a considerable fraction of published peer-reviewed scientific literature, perhaps half, has significant flaws. To demonstrate the potential failings of the present approaches to scientific Quality Control (QC), we describe examples of science associated with perceived threats to the Great Barrier Reef (GBR), Australia. There appears a serious risk of efforts to improve the health of the GBR being directed inefficiently and/or away from the more serious threats. We suggest the need for a new organisation to undertake quality reviews and audits of important scientific results that underpin government spending decisions on the environment. Logically, such a body could also examine policy science in other key areas where governments rely heavily upon scientific results, such as education, health and criminology.

</comment></comment><comment><USER4> Thanks, this really got the old gears turning.</comment>


Recursive Language Revolution 70k Years Ago the Romulus and Remus Hypotheses - <AUTHOR> https://www.biorxiv.org/content/10.1101/166520v9.full <comment><USER2> It's interesting, but I'm very suspicious of statements like

"Without PFS, Neanderthals must have been simply wrapping the hides around their bodies like a poncho" or "As discussed above, building an animal trap is impossible without PFS"

Spiders can build traps without any language. Birds can build quite complicated nests without PFS. Yet we couldn't because we lack a particular language construct?

I'm pretty sure some things my dog does would be described as impossible without PFS by these authors :) Like burying a bone in one place when other dogs see it, then reburying it in another place when the other dog is in house :)

</comment><comment><USER3> > as many as 18% of modern individuals exhibit PFS disability

wow, almost fifth can't combine even two objects in a novel way (and they can't learn to do it).

It sounds unbelievable

<comment><USER2> Yup. And according to the paper it means 18% of modern population couldn't sew a cloth or prepare a simple animal trap.

Sounds like bullshit TBH.

</comment></comment><comment><USER4> Several things:

-No genetic explanation at any point during the paper. It's all and well theorizing about evolution this genetics that, but if you don't have a mechanism, or even a putative explanation with actual genes involved and the evolutionary pressure behind them, don't expect to be taken seriously bt evolutionary biologists.

-The paper makes the assumption that a gap exists between the apparition of language and that of elaborate constructions, as opposed to us just not knowing more about it. It is entirely possible that humans from 300000 years ago were able to make figurines but we couldn't find any.

-What the hell is this doing on biorxiv?

-What the hell is RIO? I've never heard of this journal, and I don't think many people have (IF=0.8).

Overall this reads much more like a blog post than an actual article.

</comment><comment><USER5> Among other notions, Lingua ex Machina [1] discussed possible evolutionary pathways which allowed complex language (nestable subject verb object clauses) to arise from simple language (just verb object statements).

I found that book on a lark and barely understood the content. I was tinkering with knowledge representation and semantic web stuff at the time. The utility of subject verb object clauses stuck with me and seems to come up frequently.

I'd love some ELI5's for how this paper advances our understanding.

[1] Lingua Ex Machina: Reconciling Darwin and Chomsky with the Human Brain [2001] [https://www.goodreads.com/book/show/252045.Lingua_Ex_Machina](https://www.goodreads.com/book/show/252045.Lingua_Ex_Machina)

</comment><comment><USER6> I'm only half way through the paper, but it's a very interesting read. One thing I don't get though; I'd have thought that the dative case would be very useful, if not essential, to humans living a hunter-gatherer lifestyle, but the paper seems to suggest that the necessary neural machinery to support the grammatical case only came into existence 70k years ago. What am I misunderstanding here?

<comment><AUTHOR> The paper says that Hunter Gatherers existed well before 70,000 years ago, but this genetic mutation enabled the invention of recursive language, particularly spatial preposition. This enabled more abstract communication and planning, so for instance, Romulus and Remus on their first Buffalo hunt, dug a pit and covered it with branches in a location where the buffalo would naturally run to, when chased. They trapped far more buffalo than the typical zero or at most one - of traditional chase and spear method - ie they were far more successful than anyone before them. The benefit of this enhanced collaboration far outweighed the cost of slower brain development in children, so the mutation rapidly spread and dominated.

</comment></comment><comment><USER7> Previously posted

[https://news.ycombinator.com/item?id=22324236](https://news.ycombinator.com/item?id=22324236)

</comment><comment><USER8> Really interesting but poorly written (for a scientific publication). Had a few sentences that were like "big leaps" logically. I'm no scientist but this read differently from most papers I've come across.</comment>


Free services for a broken mobile startup - <AUTHOR> https://medium.com/@xeo_it/free-services-for-a-broken-mobile-startup-1c011863d10#.4wnry7s4d <comment><USER2> Nice post with tons of great resources, thanks!</comment>


Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic - <AUTHOR> We're Ken, Nate and Matt, co-founders of Speedscale (https://speedscale.com), a tool that automatically generates continuous integration (CI) tests from past traffic. Carefully scaling rollouts to ever larger groups of customers is the safest deployment strategy, but can take weeks. Even for elite DevOps organizations up to 15% of changes to production can result in degraded service [1] [2].
We met as undergrads at Georgia Tech and come from a DevOps and operations background so we've seen this first hand. Each of us has over 15 years of experience building high-reliability systems, starting in the early days with satellite earth station monitoring. As interns we once wrote a bug that caused a 32 meter antenna to try to point down through the earth, almost flattening the building we were in. It was a great environment to learn about engineering reliability. We leveraged this experience to tackle monitoring Java app servers, SOA, SaaS observability and cloud data warehouses. What if we could use a form of observability data to automatically test the reliability of new deployments before they hit production? That's the idea that got us started on Speedscale.
Most test automation tools record browser interactions or use AI to generate a set of UI tests. Speedscale works differently in that it captures API calls at the source using a Kubernetes sidecar [3] or a reverse proxy. We can see all the traffic going in and out of each service, not just the UI. We feed the traffic through an analyzer process that detects calls to external services and emulates a realistic request and response -- even authentication systems like OAUTH =).  Unlike guessing how users call your service, Speedscale automation reflects reality because we collected data from your live system. We call each interaction model a Scenario and Speedscale generates them without human effort leading to an easily maintained full-coverage CI test suite.
Scenarios can run on demand or in your build pipeline because Speedscale inserts your container into an ephemeral environment where we stress it with different performance, regression, and chaos scenarios. If it breaks, you can decide the alerting threshold. Speedscale is especially effective in ensuring compliance with subtle Service Level Objective (SLO) conditions like performance regression [4].
We're not public yet but would be happy to give you a demo if you contact us at email@example.com. Also, we are doing alpha customer deployments to refine our feature set and protocol support - if you have this problem or have tried to solve it in the past we would love to get your feedback. Eventually we'll end up selling the service via a subscription model but the details are still TBD. For the moment we're mainly focused on making the product more useful and collecting feedback. Thanks!
[1] https://services.google.com/fh/files/misc/state-of-devops-20...
[2] https://aws.amazon.com/builders-library/automating-safe-hand...
[3] https://kubernetes.io/blog/2015/06/the-distributed-system-to...
[4] https://landing.google.com/sre/sre-book/chapters/service-lev... <comment><USER2> For a little background story on the satellite antenna. I was building a monitoring device driver for an Antenna Control Unit (ACU). It's like a 4 rack-unit computer with special hardware for talking to the antenna motors (azimuth, elevation and polarization). After sending it a command, the device froze up, so we rebooted it. The CMOS battery was dead so when it came back up the date was wrong, but I did not notice. I sent it a command to reposition and it began moving to point below the horizon... The bad date meant that it had the wrong geo location position for itself. Well it turns out "below horizon" is really important because the building was just a structure to hold up the antenna and it was going to crash into the ground. Fortunately someone ran in and hit the STOP button while I was staring at the monitor. That day I learned that monitoring and alerting is important stuff.

<comment><USER3> Dead CMOS batteries suck.

</comment></comment><comment><USER4> Using products like this in the pastI've run into a pretty simple issue:

\- Request 1 is a post that generates a "todo" with a random id "5y22"

\- Request 2 is a "get" for /todo/5y22.

That works in production, but on replay of the traffic:

\- Request 1 generates a different random id, "86jj".

\- Request 2 is still a replayed "get" for /todo/5y22, which is now 404.

How does your tooling handle this nondeterminism in replays?

<comment><USER2> For one thing we look at both inbound and outbound traffic and we treat it separately. That use case looks different if we are trying to "test" TODO or if the TODO service is a backend that our app relies upon.

So if you mean that we want to "test" TODO, our analyzer looks for data in subsequent requests that was provided by the system-under-test (SUT) in previous responses. A common example of this is an HTTP cookie. The SUT gives us a session id through the Set-Cookie header response. So in a subsequent request we use the cookie from the app, not the one that was recorded. This has been done in general way to look for tokens.

Of course nobody is perfect so we'd love to see your real world app and test our algorithms against it. :)

</comment></comment><comment><USER5> This sounds great! We actually discussed doing this at the very start of CircleCI (we had a partnership with an exception handling service, but we never executed on it). Coincidentally, my current company, Dark ([https://darklang.com](https://darklang.com)) is based around a similar concept -- using live traffic as an assistant as you're writing code.

<comment><USER2> For sure we see a lot of synergy with the CI systems. One of our alpha customers is using CircleCI (no surprise there). They have an issue where devs deploy services on top of each other in staging and accidentally take it down for their internal users. So Speedscale lets them detect their new build is not a good candidate to deploy to staging.

Thanks for sharing the info about your project, I am checking it out on GitHub right now. :)

</comment></comment><comment><USER6> Congratulations on the launch Ken!

Ken worked as a top SE at New Relic and has been involved in the Atlanta Web Performance meetup for many years. It was a wonderful surprise to see you all on HN

<comment><USER2> Thank you Billy, we are very excited to be making progress with Speedscale. I miss seeing you and the rest of the crew from the ATL Web Performance meetup.

</comment></comment><comment><USER7> Hi Ken, I have been researching your project for a while and would love to get early talk before Demo day starts. Will attend next Mon/Tue anyways, but timing for decision is critical in YC. Wanna have more time dig into SpeedScale. Really like you guys' solution can potentially bring huge value to complicated production release (how many times SpaceX tests even just combustion chamber pressure before the real launch?:). Your solution seems really a systematic one by the 3 pillars you mentioned and also go under the hood (vs Recording. 1st principle wins). Trust your 32M antenna lesson is scary enough long lasting in memory. :) I was physics background, and yes, definitely you should learn where is Panic Button before poking around in a sophisticated lab.:)

Anyways, was trying to email you to founders at your domain but no response. Can you reply to my email? Handle is lli at my company domain (also in my profile). Would love to have a quick chat this week to get well warmed up before thee Day.

</comment><comment><USER8> Real traffic exposes wrong assumptions in code, which cannot be caught with unit or integration testing. Awesome to automate this kind of testing. I encourage people to invest heavily in setting up artificial environments to replay historical data.

One benefit of the artificial environment is efficiency (can cover e.g. a week of historical data); this also requires mocking out time with simulation/replay time. The integration with data-platforms makes a big difference as it means data-scientists can help set up the right test scenarios.

</comment><comment><USER9> This is cool, is the traffic curated in any way? Like if the database isn't initialized, do you start with create requests before moving on to GETs for those IDs? Also does this only support HTTP or does it support other protocols as well?

<comment><USER10> Ultimately, the idea is to mock the database itself so we just return whatever the real database returned during the recording. We don't have to run create commands because we aren't actually managing a real database's internal state. We "only" need to accurately return the responses the database gives the system-under-test for a particular GET sequence. During the alpha we are limiting support to HTTP/s but protocols like MongoDB, redis, MySQL, etc are on the backlog. Until we have more database support we're asking alpha customers to deploy test data in a test database, which seems to be a fairly normal part of the CI process for big apps.

</comment></comment><comment><USER11> I love the look of this, I cant even remember all the times I've run across applications out in the wild that were built without any thought about testing until long after the original devs are gone. All you have is production traffic and reverse engineering. It's a real pain, and I would love to have more tools to attack this problem when it comes up.

So how open is that alpha? I'd really love to try this out.

<comment><USER2> Reverse engineering protocols has morphed from a hobby to a full time job. I've spent a lot of time going through logs, looking at data and knowing that I am not looking at the TCP level request and response. So we are trying to solve that problem. :)

Please send a note to firstname.lastname@example.com so we can get the details of your environment and determine if it is a fit for our alpha.

</comment></comment><comment><USER12> Is this like goreplay[1]? How are you different?

1\. [https://github.com/buger/goreplay](https://github.com/buger/goreplay)

<comment><USER2> Yes, I really like goreplay and it was an early inspiration for our sidecar. It uses gopacket (like tcpdump) to collect data, we currently use the TCP proxy route because we can use TLS libraries for HTTP/S support. There are similarities in that both can collect and replay data.

One of the early observations from my co-founder Nate was there are 3 key ingredients to testing in a SOA environment [1]

* Automation

* Dependencies

* Data

While goreplay has a form of automation, it doesn't help you with dependencies (no mocking), and the data is either streamed or locked in a special file format. Of course like any open source project there's pieces of the solution but you have to assemble them. For instance there is no UI, no reports, no pass/fail assertions, no integration with CI system, etc. I'm by no means an expert, there is likely a path to combine with other open source projects to fulfill those.

Our stuff isn't perfect, but we primarily see overlap with the automation capability of Speedscale and the goreplay project.

[1] [https://speedscale.com/2020/02/06/triplethreat/](https://speedscale.com/2020/02/06/triplethreat/)

(edited formatting)

</comment></comment><comment><USER13> Congratulations on surviving the 32 meter odyssey and living to launch!

Wondering how do you deal with stateful services? You mention an "analyzer process" for external services, what about internal services?

It seems this would work well at a single service level, but would it also be possible to apply the analysis at both a single service and a group of services? Some form of unit-testing _and_ integration-testing at a service level...

<comment><USER2> Awesome thank you for the note. Fortunately there was a "stop antenna" button which saved the day lol.

We've been down the path of stateful services before and actually reflect the proper state in our responder. Because we control the test that is being played against the system-under-test, we understand the sequence and order of calls that will be made to the downstream system as well.

In addition, the analysis actually captures all outbound services at once. We are able to identify each separate hostname that is invoked and mock them all out as a group. One of our first alphas was stunned that it auto mocked 7 backend systems on the first try.

</comment></comment><comment><USER14> Sounds great, for using this in a financial environment it will need an option for data anonymization, I'm not sure how can you identify what needs to anonymized without human interaction though.

<comment><USER10> You're spot on :). We got this feedback from one of our financial services alphas so we built a DLP rules engine to cover it. That wasn't enough. So we offered to integrate with Google DLP. Still no. So in the end we architected for a split-plane architecture (similar to DataBricks) so big customers can host their own data but we can manage the control stack. It's not something we're doing during the alpha but it's part of the plan. Would that work?

<comment><USER14> I'm only the end user so I can't judge the details

</comment></comment></comment><comment><USER15> I had a similar idea, but from the frontend. I think it would solve a few edge cases where frontend logic is needed to complete full state-based interactions. But I think there's now some solutions out there for this. Good to see some traction here though, since automated testing and regression testing is still a very difficult and manual thing to set up and keep updated. Lots of opportunity to make it less painful.

</comment><comment><USER16> There's another company that has a bit of an earlier start doing this: ProdPerfect (www.prodperfect.com)

"Reach new heights with weightless test automation. ProdPerfect is the first autonomous end-to-end (E2E) regression testing solution that continuously identifies, creates, maintains, and evolves E2E test suites via data-driven, machine-led analysis of live user traffic."

<comment><USER17> These are very different; one is clickstream based and the other sits on the networking layer.

<comment><USER16> Whoops! Failed to read past the first paragraph. Also should have added a "similar" in there. Thanks for pointing this out.

</comment></comment></comment><comment><USER18> Super Excited about this. Data Variance is hard to solve and the bucket of data that your helping folks access is impressive...

</comment><comment><USER19> Can you run tests from different geos?

<comment><USER2> You can run it in your own environment. If you're running Kubernetes we provide an operator that orchestrates the test runs. If you are using docker we give you containers that you control with ENV VARs.

Do you have more background on the multi geo use case?

<comment><USER19> We run a multi-geo service (Fly.io). Replicating user load on distributed apps is hard.

Containers with env vars are easy though!

<comment><USER2> Yes I recently went through a similar use case with one of my alpha users. They wanted to run the reverse proxy and playback as docker containers spread through their environment. Will drop you an email with more info...

</comment></comment></comment></comment><comment><USER20> Terrific stuff and glad this is becoming industrialized at scale.will keep you guys in mind as we expand

</comment><comment><USER21> Much needed! Congrats on the launch.

</comment><comment><USER3> Congrats on the launch, y'all!!!

</comment><comment><USER18> Sweeet.... cant wait to get my hands on it and leverage it

</comment><comment><USER22> Love it!

</comment><comment><USER23> does it work for gameserver-style multi-client websocket communication?

<comment><USER2> One of our early inspirations was a video game company that would replay recorded gameplay against new builds of a game server. And our proxy assembles each request and response in the order they were received, even if they go with different user sessions. For the alpha we are focused on API type calls over HTTP/S, but websocket implementation is definitely something we are tracking.

</comment></comment><comment><USER24> Congrats on the launch!</comment>


Your Middle-Aged Brain Is Not on the Decline - <AUTHOR> http://www.npr.org/2016/03/15/469822325/forget-about-it-your-middle-aged-brain-is-not-on-the-decline <comment><USER2> I'm a bit disappointed in the article. The author explains the tests they use to measure her supposed increase in intelligence and how in many cases, older "brains" are preferable in various professions (e.g. the Supreme Court, surgery (possibly), air traffic controller) but doesn't really talk about any practical changes in her life.

Rather she talks about how she still forgets the keys. So is there no perceived difference after her $64,000 experiment?

</comment><comment><USER3> Why is 50 considered "middle age"? That implies life expectancy of 100.

I would say 40 is middle age.

Regardless, I applaud the work on trying to increase fluid intelligence. It's a truly hard problem and if humanity can crack it we would all be better off for it.

Sometimes we focus so much on AI that we forget that we already have about 7 billion human intelligences, most of which are severely underutilized.</comment>


How Pandora ships 20 features in 90 days with 40 engineers  - <AUTHOR> http://www.fastcompany.com/3026201/lessons-learned/how-pandoras-40-engineers-nabbed-70-million-monthly-users

<comment><USER2> Was already posted as "What would be stupid not to do in the next 90 days?" iirc.</comment>


Ruby Is The Future - <AUTHOR> http://enfranchisedmind.com/blog/posts/ruby-is-the-future/

<comment><USER2> The author claims that monkey-patching private methods to expose them publicly is representative of Ruby's beauty, and specifically:

"... later upgrades to the library don't need to be re-fixed: the one-time patch will automatically get re-applied by application code automatically!"

Anyone who fails to realize how monkey-patching can and will break both subtly and catastrophically across library upgrades has no business advising anyone regarding language choices.

[Edit] I hate April Fool's Day. Depressingly, the author's satire is so subtly aligned with common arguments as to so thoroughly fool.

<comment><USER3> Just because it's a satire doesn't mean the title in incorrect. I would argue that Ruby is, in fact, the future for the same reason that Python is the future.

As for monkey patching, yes, it's ill-advised in general just as global scope and gotos are ill-advised for similar reasons. However, monkey patching is a powerful feature if used responsibly as Chad Fowler has noted ("The Virtues of Monkey Patching"):

[http://chadfowler.com/index.cgi/Computing/Programming/Ruby/T...](http://chadfowler.com/index.cgi/Computing/Programming/Ruby/TheVirtuesOfMonkeyPatching.rdoc,v)

Now admittedly, not every programmer is as competent or responsible as Chad Fowler or the programming world would be a very different place but the point is there's a right way and a wrong way to do it.

Werner Schuster's article on InfoQ ("Ruby's Open Classes - Or: How Not To Patch Like A Monkey") has more good advice on how to do it the right way:

[http://www.infoq.com/articles/ruby-open-classes- monkeypatchi...](http://www.infoq.com/articles/ruby-open-classes- monkeypatching)

<comment><USER2> As noted in Werner Schuster's article, many languages (such as C# or Scala) do support type-safe, non-ambiguous, non-conflicting extension of existing classes, demonstrating that extension of existing classes can be safely implemented and does not require the use of dangerous monkey-patching.

<comment><USER4> dealing with the type system in scala can be a pain and is a hit to productivity in the early stages. The question is if the hit in the early stages leads to more productivity in the later stages. I'm not convinced that it does.

I've done a lot of ruby, and I've had monkey patching bite me a few times. However, fixing the problem was never that hard (two days max) and is clearly overshadowed by the productivity gained from the power ruby gave my teams.

<comment><USER2> How can one effectively argue against an entirely subjective judgment of difficulty in terms of dealing with a type system?

Anecdotally, I find that leveraging functional language programming features coupled with type inference and polymorphism leads to considerable productivity gains as code correctness can be ensured through judicious use of types -- without requiring extensive testing or programmer effort -- while excess verbosity can be eliminated through the use of FP features such as type-safe anonymous functions, currying, etc.

<comment><USER3> How do you deal with excess verbosity in C#? I know this is a separate argument from the monkey patching issue but for me the conciseness and elegance of the Ruby syntax versus C# is the primary reason to choose Ruby.

<comment><USER5> Which version of C# are you thinking of?

In C# 3, the verbosity level is much less than in C# 2, where it's slightly less than C# 1.

I don't see where Ruby syntax is particularly more concise than C# 3's.

Well, let me clarify "particularly". I don't find myself feeling like I'm suffering under C#'s syntax, relative to Haskell's. It has about the same amount of syntactic overhead: writing types for method parameters, and when defining data structures. Occasionally I have to explicitly write a type parameter to a function.

<comment><USER3> "I don't see where Ruby syntax is particularly more concise than C# 3's."

1.upto(10).each{|x| print x}

I'm curious - how would you print 1 through 10 in C# 3 by comparison?

<comment><USER6> I still find this clearer:

      for x in range(1, 10): print x

<comment><USER7> I'm nitpicking, but your range should be range(1,11).

<comment><USER6> Every nit picked is valuable :)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> I'm going to dig through HN archives or wait until tomorrow to read any news.

</comment><comment><USER9> Just two words: "Hopefully not..."

<comment><USER3> Why? Because of monkey patching, syntax, performance, the community or some other reason such as you prefer Python or have an intuitive, ineffable and unexpressable disklike of the language or because Matz is from Japan and they bombed Pearl Harbor?

<comment><USER10> I think (and I mostly code Ruby these days) it's not least because of comments like this.

<comment><USER3> Out of curiosity, what has your experience been like? Do you have any major gripes with the language or are you still on a Ruby honeymoon?

<comment><USER10> I love Ruby. I love that code can be so short and yet easily readable. It might be slower and perhaps more memory-hungry than other languages, but for my needs that's usually not an issue.

What I do miss are good, _well documented_ , stable libraries. For example, I currently process RSS/Atom feeds with FeedTools, whose own creator says he's tired of maintaining (I totally understand him and am extremely thankful for the time he put into it). I assume if there were more people coding Ruby, there would be a better chance someone would step up and take his place. So, having a community that is more welcoming to newcomers would eventually benefit us all.

By "open to newcomers" I don't mean that Ruby-celebs should stop posting snarky comments at each other in their blogs. This is pretty negilible. I mean stuff like having better documentation, for example, online, free, in googleable format (rather than screencasts or print books). IMHO PHP3 annotated online docs back in 1999 were significantly more usable than what Rails has to offer now.

<comment><USER3> "I mean stuff like having better documentation, for example, online, free, in googleable format"

Do you mean for the core language? Are you not happy with <http://www.ruby- doc.org/> ?

<comment><USER11> Plus you can do a custom Google search like this:

          http://ruby-doc.org/q/<your search string>

<comment><USER3> Best thing I learned today. Thanks.

<comment><USER11> My pleasure.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> martoo on the equivalent Reddit post had the best thing to say about this:

 _"There needs to be a special term for an attempt at an April Fool's Day Joke, which is in all of its points true and then ends up looking like a joke made by an author at his own expense."_

[http://www.reddit.com/r/ruby/comments/8966h/ruby_is_the_futu...](http://www.reddit.com/r/ruby/comments/8966h/ruby_is_the_future/)</comment>


Microsoft Sued Over Windows 8 Tiles - <AUTHOR> http://www.eweek.com/enterprise-apps/microsoft-sued-over-windows-8-tiles/

<comment><USER2> What good can come out of a lawsuit like this, which sounds like it will fail anyway? As usual, the only people that benefit are Lawyers.</comment>


Are You the Fool at The Table? - <AUTHOR> http://steveblank.com/2011/06/17/are-you-you-the-fool-at-the-table/ <comment><USER2> I think Steve is on the money with this article. In particular, with this key proposition:

"The issue is whether the next 100+ tech IPO's carried by this bubble will be worth their offering price in 8 years."

Ask yourself that question. We're going to see a lot more IPO action, and 8 years from now, I think many will be gone.

As an aside, I strongly recommend Steve's book "4 steps to the epiphany." It's the equivalent of a graduate school level course on entrepreneurship.

<comment><USER3> I read the first 40 pages of "4 steps to the epiphany" at [http://www.stanford.edu/group/e145/cgi- bin/winter/drupal/upl...](http://www.stanford.edu/group/e145/cgi- bin/winter/drupal/upload/handouts/Four_Steps.pdf). It's remarkably dense reading but for someone who was not around during the 2000 bubble, it contains valuable information about what happened.

</comment></comment><comment><USER4> "There are five types of participants: Smart Money, the Shills, the Marks, the True Believers and the Promoters." I want this to be the start of action movie then you get to see each type of character and how the situation plays out.

</comment><comment><USER5> Slightly off-topic: looking at his graph, I remembered the idea that internet traffic is fractal-like in time domain. <http://c2.com/cgi/wiki?InternetTrafficIsFractal>

...because the graph looked just like a fractal fragment to me. That coincidence, or are stock prices indeed fractal-ish in time domain?

<comment><USER6> _> are stock prices indeed fractal-ish in time domain?_

Guess what Benoît Mandelbrot studied before he started doing pure maths? ;)

EDIT: for bonus points, guess why he fell out with his old friends in the 90s.

<comment><USER7> Did he suggest that the quants at hedge funds were merely lucky rather than supremely clever?

[http://liberalironist.wordpress.com/2010/10/08/book- review-t...](http://liberalironist.wordpress.com/2010/10/08/book-review-the- quants-part-ii-the-meaning-of-models-statistics-and-social-science/)

[Edit: I have read _The Quants_ it's an interesting book]

<comment><USER6> It's to do with the asian and russian crises of the late 90s, which he felt showed that the black-scholes models were wrong and inferior to his own multifractal view of the market. After those crashes, he wrote articles and books attacking those ideas.

[http://www.scientificamerican.com/article.cfm?id=multifracta...](http://www.scientificamerican.com/article.cfm?id=multifractals- explain-wall-street)

[http://books.google.com/books/about/Misbehavior_of_markets.h...](http://books.google.com/books/about/Misbehavior_of_markets.html?id=9w15j-Ka0vgC)

<comment><USER8> When you model equities as log-normal ie. log(stock price) is normally distributed, and then use the geometric brownian motion to model the underlying, which spits out derivative prices using Black-Scholes. The problem is that normal distribution is thin-tailed ie. 5 sigma events are extremely rare. So use a fat tailed distribution - which is what Mandelbrot did. He used a power-law ( Pareto ) distribution because its infinite variance permitted wider swings in price than the Gaussian. However the simplicity of using a Normal distribution is what makes Black Scholes so robust. If a thousand statisticians look at market data over a time window & model it to fit a lognormal usaing MLE, they'll come up with approximately the same parameters, so calibration is relatively easy. In fact this is one of the standard exercises in any MFE pgm - to calibrate a derivatives model corresponding to underlying data over a time window. With Mandelbrot's Paretian distribution, calibration is virtually impossible. No two people get the same parameters given the same data points on a Paretian model. There is quite a bit of literature on this very topic, in both Taleb's last book & elsewhere. Mandelbrot points out how given different time windows, the Paretian distribution can be calibrated to fit virtually anything, but the parameters will change wildly.

Essentially, given the choice between an inaccurate robust simple Gaussian model with high predictive power and a supposedly accurate but un-usable Paretian model, financial engineers choose the former & add fudge-factor explanations ( eg. the vol-smile ) to augment the data.

1\. [http://blogs.reuters.com/justinfox/2010/10/18/why- didn%E2%80...](http://blogs.reuters.com/justinfox/2010/10/18/why- didn%E2%80%99t-people-in-finance-pay-attention-to-benoit-mandelbrot/) 2\. <http://en.wikipedia.org/wiki/Fat_tail> 3\. [http://brokensymmetry.typepad.com/broken_symmetry/2009/08/wh...](http://brokensymmetry.typepad.com/broken_symmetry/2009/08/what- mandelbrot-and-fama-both-got-wrong.html)

<comment><USER9> Thank you. I understand things a bit better now.

</comment></comment></comment></comment></comment></comment><comment><USER10> Mr. Blank makes some great arguments, and I tend to agree with him.

What's fantastic is that we're so recently removed from a bubble (in housing) that it's entirely familiar. Take his arguments, like the "five types of participants", and apply them retroactively to that bubble. Perfect fit.

Denying the next bubble is futile. They exist; always have, always will.

<comment><USER11> Not just a housing bubble but a bubble in the exact same industry sector as well!

Of course, a bubble is fundamentally a gross mismatch between future expectations and actual returns. That's why there's no learning from the past, and why it's always "different this time."

More importantly, bubbles don't occur in a vacuum. The 1990's tech bubble wasn't just because the internet was new, and everyone thought it was going to be great. The 1990's also featured low interest rates, the repeal of Glass- Steagall, and other loose policies conducive to asset bubbles. Sound familiar?

Such policies force money to chase yield, which means risk. As it does so, the market sends false signals about the true value of its companies, which reinforce themselves until some "Black Swan" event reveals the imbalances. In 2008, that event was the fall of Lehman Brothers.

As I noted below, if the national government continues to tighten its fiscal and monetary policy, then this bubble will be over before it really began. If, however, the government reloosens after the end of QE 2, it will most likely be full steam ahead.

</comment></comment><comment><USER12> Interesting article, but I found a contradiction:

"Smart Money" ... "but they don't hype it, talk about it or fan the flames."

"Promoters" ... " they are a small subset of the Smart Money" ... "They loudly tell the Marks and Shills that everything is just fine, enticing them to buy into the bubble,"

Promoters can't be a subset of Smart Money because Smart Money doesn't talk about it, but Promotors talk about it very loudly.

For a couple years now, I've been looking at the market and trying to figure out how to take advantage of the trends. Be 'Smart Money' in other words. I don't think I have the financial sense to make it happen, though.

<comment><USER13> I think what he meant to say, is that the _Smart Money_ typically does not talk about it or hype it up. However, there are exceptions - and _promoters_ are those exceptions. Perhaps he's making a moral value call, saying how most good investors aren't actively _trying_ to screw the public?

<comment><USER12> That could be. The words 'in general' would have fixed the contradiction.

And I don't think he's making a moral judgement... There are plenty of reasons to keep quiet about good investments that are all over the moral scale.

</comment></comment></comment><comment><USER14> The full second part of the Economist debate was linked (with Horowitz's response), if you're interested:

<http://news.ycombinator.com/item?id=2665389>

</comment><comment><USER15> _"the size and scale of these new markets have never been seen before; some of these applications and companies will reach billions of customers, generate unprecedented revenues and profits,"_

Statements like that always come with bubbles too. They are necessary, but not sufficient. The old dot com bubble was all about a new economy that defied standard valuations -- so you couldn't apply historical models to new these new "internet companies".

</comment><comment><USER16> For those who have been following the Economist debate, I'm curious who you think is getting the best of it so far?

</comment><comment><USER17> Wait, Steve is making the argument that the bubble only exists for new tech companies? Considering there only a handful of IPOs of social/web/mobile/cloud companies, how exactly will this supposed bubble form?

None of these stocks has done well. The market has corrected or is correcting all of them. If there is a bubble and we are on the upswing, it is the least impressive bubble ever.

<comment><USER11> I think a lot of conventional analysis of investments is looking more foolish than usual because the market isn't keying off what people think it is. Three major things have dominated market movements since the 2008 crisis:

1) Rock-bottom interest rates (and a depreciating dollar) 2) Quantitative easing 3) Enormous deficit spending

All of these things, through mechanisms both simple and complex, have driven money into equities and speculative investments such as start-ups.

While interest rates will remain low for quite a long time and deficits will no doubt continue, QE2 expires at the end of the month. Additionally, the proportion of the deficit is moving away from stimulus into structural spending such as Medicare. The natural response by the major market players is profit taking. Hence, the IPOs at bubble-like valuations (LinkedIn, Pandora, etc) are running into the teeth of a "smart money" evacuation of the stock market.

If the Fed and government overcome political pressure against further loose policy (always possible if the economy continues to weaken), then I imagine the LNKDs and Ps of the world will pop back up to the eye-watering valuations they attained at the time of their IPOs.

If national fiscal/monetary policy continues to tighten, then the bubble will have been ended before it really began. In which case, LNKD and P will continue to decline, and I would not be surprised to see some of the other hot properties like Zynga delay their market entries. The market for VC funding will also become substantially tighter.

</comment></comment><comment><USER18> Getting a little tired of "former entrepreneur" professors like Steve and Vivek playing the wet blanket in tech. It's too easy to be negative, especially with the first dot-com bubble so fresh in our minds. It's a lot harder, and more rigorous, to be positive in a frothy market. You have to be smart about it, but you still have to be in it.

<comment><USER19> It's harder and more rigorous to follow the herd in a "frothy market"?

<comment><USER18> It's not about following the herd - it's about being where the herd is (because that indicates there is something big there) but choosing your bets carefully and controlling what you can control. I think it's too easy for these people to throw everything out altogether like we should all just go home and not even try.</comment></comment></comment>