Stop Using Digital Ocean Now: The Aftermath - <AUTHOR> http://serdardogruyol.com/?p=137 <comment><USER2> Sorry but I must be blunt... What do people expect for < $8/mo service? Should DO spend a few man hours to fix the issue at hand (abuse or not) and very much burn the 7 months worth of service income from this account? This is an unreasonable expectation.

Something has to give. Yes we've all built and sold products and believe in providing an impeccable service worth far greater than the sum of its parts. Because we're in it to please everyone and build a reputation. But hosting is different. There are real costs for _not_ taking action (upstream null routing, blacklisting, chargeback fees, fraud, abuse, etc).

I'm definitely playing devil's advocate and yes I go above and beyond for my clients. Never have I kicked a client to the curb for abuse they haven't originated. But DO is unmanaged and acquires clients by the shovel. Something has to give.

For the record, I did publicly state here in HN that DO's business plan doesn't add up and this is one of the side effects.

<comment><USER3> I expected a budget service, but they've been anything but that. Their support staff aren't just reading off a script, their instances are fast and downtime non existent. In this case the user fucked up and was made part of a botnet, and DO were forced to protect their network. I'm happier running my services on DO than on any other host.

</comment></comment><comment><USER4> On the other hand, here's my experience with Linode.

Another host company claimed that one of my machines was doing port scan on their network. Linode opened a ticket and preemptively blocked all outgoing connections to the SSH port from my machine.

I had enough time to see what's going on and chatted with a very responsive support. The aftermath is that I moved all my data to a new linode, waited to for the DNS propagation and killed the old linode. No service disruption and no all-nighters.

I seriously can't recommend they enough. And yes, I'm aware of the security problems they had months ago, but I bet that they don't want any more damage to their brand and they're working very hard to no let that happen again.

<comment><USER5> Not all VPS service providers are treated alike. Linode is probably a mid-tier VPS. Their cheapest plan is $20, which is 4x more expensive than Digital Ocean's cheapest plan of $5 a month. I would guess that most of the ultra cheap VPS plans probably won't come with a network engineer to help you.

<comment><USER4> Agreed. If I'm trying something new, or just playing with, I could try a new and cheaper host ($5/m is less than a starbucks coffee where I live).

But if I'm running something that pays me money, there's no doubt that I'll choose the best provider. And the best provider, for my requirements, is only $15 more expensive.

</comment></comment></comment><comment><USER6> > _The things which i still don't understand.... Is my privacy is more > important than my user experience or happiness of the service ? Even if i > want them to tell what really happened ?_

Well, yes. It is completely appropriate that they aren't going to discuss your logs and other private details in public without your permission. You don't understand that, for real?

They have no _obligation_ to discuss your case in public at all, although it might be wise for them to do so and explain themselves, if you are generating lots of bad feelings for them.

(They do, I'd agree, have an obligation to discuss your case _with you_ and tell you why they closed your account. It's not really clear to me if they did so; it kind of seems like they did so, something to do with DDoS.)

</comment><comment><USER7> OP, can you please confirm whether or not you made this post quoting a reply from Digital Ocean in the previous thread, as [1] claims you did?

 _Now i 've received an answer from DO. I seriously dont know how i did a DDos._

 _Here it 's._

 _Greetings,_

 _Based upon the tcpdump results, I have again confirmed that your droplet was indeed performing a Denial of Service attack._

 _With this information, we are unable to restore services to your account._

If this is true, it is disingenuous not to mention this reply in your post.

[1]: [https://news.ycombinator.com/item?id=6439501](https://news.ycombinator.com/item?id=6439501)

<comment><AUTHOR> The thing is that they wanted to stay private when i posted this so i immediately removed and now waiting for them. By the way i received that response after 2 hours of the topic opening. They weren't answering me at all before HN post

<comment><USER5> So did your server get owned? Or was the DDOS attack a result of a bug in your code?

Also, why do you delete your posts on HN? Do you only keep posts that make your viewpoint look favorable?

<comment><AUTHOR> I don't know the first question's answer and that's what i am trying to learn. Second it's my first time participating in a HN topic that much i didn't know the etiquette here sorry for that.

<comment><USER5> I'm guessing that most people who run a server may not even realize when they get hacked. These people (you included) probably should not run their own servers and stick to PaaS solutions like Heroku or Google App Engine. It happens all the time to guys who think they can install & maintain Wordpress themselves.

You probably should have analyzed the issue before making a blog post about it. I have had servers hacked into in the past, but I wasn't about to defame a company over my own mistake of securing the server.

<comment><AUTHOR> You miss the point here. It's not about getting hacked or so. It's their way of handling it. Like i said they kill it first and then tell you the reason why. What's the point in it ?

<comment><USER8> The point is that your system may be actively attacking another system, and it's their responsibility to immediately stop the attack first and then contact you after.

If they _don 't_ do this, then they run the risk of having their netblock(s) blackholed by upstream providers or other networks, which is bad for all of their customers.

They aren't responsible for making sure your system is secure, you are. You're a sysadmin now; it's not just a toy, there's responsibility too.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> His server was probably broken into, someone used it for DDOS'ing, DO shut it down and customer has no clue what's going on.

</comment><comment><USER10> Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly.

\-----

All times are UTC.

Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later.

The customer informed us that it was a script that was crawling in the background.

We informed the customer that it may be a good idea to check through the virtual server to see if there were any signs of a compromise just in case.

The droplet was unlocked at this time.

A second UDP pattern was detected on 2013-09-24 12:27:09 and a ticket was opened 2013-09-24 12:27:14 to request more information from the customer.

Because this was already a second occurrence we had to do a more thorough follow up. Discussing the matter with the customer, he informed us that it was a mysql db dump script that was pushing data to dropbox.

He provided us a link to a github project that he wrote, we asked further questions. Specifically if you are writing a mysql dump remotely why are the packets being sent as UDP? Additionally if the final destination is dropbox that would be an SSL encrypted connection and again why would that transfer go over UDP?

We reviewed the code of the dump-to-cloud project and it was using the dropbox sdk, here is where the file transfer is initiated:

       def upload_file(file_name)

            client = DropboxClient.new(@access_token)

            file = open(file_name)

            puts 'Uploading file!! Please wait.'

            response = client.put_file("/#{file_name}", file)

            puts "uploaded:", response.inspect

        end

From the dropbox SDK here is where it sets the destination for the file transfer:

       def build_url(url, params=nil, content_server=false) # :nodoc:

            port = 443

            host = content_server ? Dropbox::API_CONTENT_SERVER : Dropbox::API_SERVER

            versioned_url = "/#{Dropbox::API_VERSION}#{url}"

            target = URI::Generic.new("https", nil, host, port, nil, versioned_url, nil, nil, nil)

            #add a locale param if we have one

            #initialize a params object is we don't have one

            if @locale

                (params ||= {})['locale']=@locale

            end

            if params

                target.query = params.collect {|k,v|

                    CGI.escape(k) + "=" + CGI.escape(v)

                }.join("&")

            end

            target.to_s

        end

The code that actually transfers the file from the dropbox sdk:

        def do_put(url, headers=nil, body=nil)  # :nodoc:

            assert_authorized

            uri = URI.parse(url)

            do_http_with_body(uri, Net::HTTP::Put.new(uri.request_uri, headers), body)

        end

The file is transferred via HTTPS since it is going to a secure service and HTTPS would rely on TCP for the data transfer, again to ensure that all packets are delivered.

Given that it was the second incident that a UDP traffic pattern was observed in less than 30 days and that the information the customer provided regarding the traffic did not match up, we made a determination that in fact it couldn't be this script that was generating the traffic.

All of this information was relayed to the customer that we did not believe that the traffic in question was related to this script because it would not rely on UDP, an insecure protocol to deliver files to a secure endpoint where data integrity was of the utmost importance.

Unfortunately, we could not unlock the account at this time because the information we received was not clear and we already had two incidents of outbound UDP traffic that appeared to be disruptive and abusive in nature totaling 1Gbps as if it were a denial of service attack, typically associated with UDP packets.

<comment><AUTHOR> Hello Ben, thanks for the response. Fırst of all at first ticket i told that the only possibility of having an UDP outgoing is that script that i wrote.

Other than that i've no other activity or script that can generate that much traffic. Haven't you even considered that my droplet may be compromised or being attacked ?

Instead of letting me know what exactly happened or which processes were running at that time you just locked the account and accused me.

Couldn't you even look at the access logs or so to see which IPs login into the droplet and then take your action later instead of closing it instantly?

<comment><AUTHOR> I was also tremendously happy with DO and their service. But what if you get your production apps down without even any notification and proper reasoning ? That's the thing which makes you feel insecure.

<comment><USER11> They did notify you.

>Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later.

Also, you should do a better job securing your server. It seems like the server was compromised.

</comment></comment></comment></comment><comment><USER12> I use DO for a messing around on a small web app I'm developing. Within 24 hours of having my droplet up the root password was guessed and my machine was used for some DDoS. Granted I was an idiot for not changing the password immediately but I definitely felt like DO should just use ssh key validation like AWS does right off the bat. That deters attackers from even trying brute force attacks in the first place.

Anyway, I checked the logs and pretty much the minute my machine was deployed a script was guessing my password (lots of failed login attempts for "root" and "oracle"). This probably means someone knows DO's IP addresses and their automatically generated password scheme (all lower case alpha characters of a fixed length).

I reported the incident and destroyed my droplet since there was nothing important on it. When I heard back from DO I basically got (paraphrasing here) "you should install fail2ban next time". Case closed. I'm not a big customer or anything so I don't expect premium support or anything but I feel like someone should have looked into the attack a bit more. Seems like a lot of people are experiencing the same thing.

I guess what I am saying is you get what you pay for (it is only 5 bucks after all).

EDIT: Still using DO. I was just a bit more careful next time I deployed a droplet.

</comment><comment><USER13> About a week ago I got to experience Digital Oceans very tight suspension policy first hand. What happened was that one of the accounts I manage was suspended. I had to go through a very long and detailed validation process before they understood that they had done wrong (they admitted to doing an error and apologized). But when my account was activated again my droplet had been destroyed..

After some more time they managed to resolve this and I'm again a happy DO user but I wish that they take a look at their policies. Just the fear of knowing that they can shut you down by mistake for a day or two is bad enough to not use them. They should have a policy where they at least call you and talk to you before they do anything.

<comment><USER14> These kinds of seemingly random suspensions at DO are starting to concern me. I've got a service with ~10K active users I'm migrating to DO as we speak, but man! I'm starting to feel like it might be professionally negligent to do so if they're trigger-happy with shutting down instances.

It'd be nice if DO had some way to communicate "hey, this VM matters to my business, PLEASE don't do anything stupid/automated without contacting me first", but that's probably too much to ask for the cost.

Gah. Back to AWS. This sucks. :/

</comment></comment><comment><USER15> Some months ago a tried to open an account with DO, but my account was blocked by DO before creating a droplet because my dad, who has the same name as I do, had already used the service. DO wanted me to verify my identity, address, etc.

I can't stand such friction, so I stayed with AWS, who have never had an issue with having a customer with very similar or equal name to another. Clearly, there is something wrong in DO's identity/fraud detection process, and even more wrong is the fact that they are locking user's production accounts without any warning at all.

<comment><USER16> I ran into the same problem just last week when I tried to make an account to try it out.

I added a credit card to my account without a promotion code, but later on found out about the new promotion they are doing (free $10). I realized that you cannot enter a promo code if you didn't do it the first time you add payment to an account, even if the account is otherwise still new (I didn't do anything with it after adding the payment card).

So I de-activated my account, and made another one, entered the same payment info but with the promo code this time. Then my account was instantly locked for using the same credit card, and I received an email shortly after citing section 2.6 of their ToS which prohibited users from using the same payment card on multiple accounts (even though my previous account was de-activated).

Their trial/promo code system really needs to be revamped as I spent way more time figuring out how to put the code in than actually trying out their product. Also their fraud detection system is perhaps too sensitive? I mean I should be able to re-use payment cards if my old account was de-activated...

<comment><USER17> They have really quick customer service; they probably would have added the promotion to your account if you had contacted them.

<comment><USER16> Probably. But at the time I thought it was a problem I could've taken care of in less than a minute, basically de-activate my account then create another one and add the code. In this case I chose not to contact them, because as fast as their support can be it mostly likely won't be faster than I can de- activate and make a new account. But I was wrong as my account was locked :(

Another thing is their ToS. I admit I didn't read their ToS prior to registration (most people don't I assume, but I won't use that as an excuse), but I did go back and read it afterwards, and it said:

"Users are restricted from registering multiple accounts with the same billing details without first notifying DigitalOcean of that intent to ensure that accounts aren't automatically flagged as possibly fraudulent and without notification accounts may be treated as abuse and/or fraudulent which would lead to suspension of service."

And I thought, even if I did read this I would've reasonably assumed that since my other account has been de-activated, I'm free to make a new one with the same billing information. Unfortunately I was wrong.

</comment></comment></comment></comment><comment><USER18> Wait... did I miss something or did you send snailmail and expect a response within 10 hours?

Also I am nowhere close to being a lawyer but I'm pretty sure if they disclosed your private information on here without your written permission you could sue them probably for more than either of these 2 posts are worth against them.

<comment><USER19> I think he scanned the letter and emailed it.

</comment></comment><comment><USER20> Sad to say, the same thing happened to me and the support messages were unhelpful to say the least. They have several copy-and-paste answers they use far too eagerly.

And it's asinine to ask someone to update billing information when they've locked your account from accessing the billing info page. Two of us have seen that with DO. Makes them look quite sloppy.

</comment><comment><USER21> _And now after nearly 10 hours or so i still haven't heart from DO._

There is a problem on the Internet is that people demand things NOW. Really? How long will it take for stuff to happen in real life (especially if you are dealing with government).

Some stuff does happen immediately (like registering or purchasing something), but stuff which requires human intervention is obviously slow. And it requires time.

Yes, you can have 24/7 response. But only if you hire someone (probably 3 persons) working for you round the clock which will mean paying thousands of $$ per month and not $5/month.

<comment><USER17> 10 hours is unreasonably long for someone who is running an application that other users need to access. People are not going to be happy if Farmville is suddenly not remembering all the cows they milked last time they were logged in. This is about keeping customers happy down the chain.

That said, I actually really like DO and I've only had prompt, helpful responses from their customer service. I don't build apps but it is an _excellent_ personal vps, and I still recommend them to my friends when they are looking for one.

<comment><USER5> If you're paying $5 a month, don't expect to have support turn around times under 1 day. If you're running Farmville, I would hope that you're paying more than $5 a month for your server.

</comment></comment></comment><comment><USER22> > Is it fair to tyrannically close someone's account up and accuse him or > further treat him as a liar ?

Isn't this what Google does on a daily basis to users of their various services? There seem to be no consequences for them and less than zero interest in improving any of it.

There are companies that can cause your business untold financial damage through these kinds of actions. For some reason they continue to evade responsibility in financial, moral and ethical terms. It sucks.

<comment><USER23> The difference is: you are the product for google and not the customer. Trust me, google doesn't treat advertisers the same way it treats free gmail users.

<comment><USER24> Google often shuts down advertiser accounts with little or no notice. You only have to search for "adsense" or "adwords" along with "banned". Some of those are with good reason - others are not.

</comment></comment></comment><comment><USER25> There may be all kinds of valid reasons to validate acounts, check documents and ownership of cards, etc.

None of those reasons excuse putting services offline while this validation is happening - first try validation, give it some reasonable timeframe, and only then cut or don't cut the service depending on results. Is it really so complicated?

</comment><comment><USER26> OK, after read the two blog posts, and all the comments here, this is my tough:

I've a poor's men backup solution too, from my personal droplet to gmail.

I use it for backup /etc, /root, /usr/local/(s)bin and /html, with excludes and rotations (about 15MB total), encrypted 4 passes with 4 different algorithms.

I did write two scripts, one for backup-restore (and encrypt/decrypt, secure deletion, html email generation, etc) and other to list-retrieve remote backups using IMAP.

On gmail I did put a filter to send all to the trash, so I get 30 days backup rotation. More complex rotations could be implemented using different cron tasks, with different config files, pointing to different accounts (srv-month- number@...), but for my personal VPS I don't need that, 30 days is enough.

But.

Now, I go to the digital ocean panel, to the "backups" tab, and I read:

       "Pricing is set at 20% of the Droplet's monthly cost (e.g. It will cost $1/mo. to enable backups for a 512MB Droplet)."

And I can only think: facepalm.

It's not only the price of the implementation time, that me and this other person with ruby/dropbox have spend...

Even if it looks like "you get a _gratis_ backup solution", really it may be more expensive (because of all the network bandwitch)

Well, at a side of the fun of implementing your own backup/restore scripts, you get:

* provider independence (if all DO is down, you still can restore from gmail or dropbox)

* no periodical costs (but remember the implementation cost maybe bigger the first time, then is just reuse and edit a few variables).

* Security (you control how and where the data goes, otherwise you can save and restore from the provider, but implementation, management, internal policy, budget, team or technology changes, etc around your data is up to you).

But, I will never call again my solution a "poor's man" solution, because counting implementation time and network traffic, is much more expensive than $12/YEAR.

Edit: formatting a list.

</comment><comment><USER27> Previous post was submitted here: [https://news.ycombinator.com/item?id=6438761](https://news.ycombinator.com/item?id=6438761)

</comment><comment><USER28> I left them long ago after super crap support.

No thanks and good luck.

<comment><USER5> Most online services that are under $5 a month rarely come with support. I feel bad for startups that charge so little per month because usually the less you charge, the more demanding the customer is. Virtualized servers tend to require more support than say, online accounting or email products.

People who go to Digital Ocean are going for cheap VPS solutions. If they had money to blow, they'd probably be with amazon or rackspace.

</comment></comment><comment><USER29> > And now after nearly 10 hours or so i still haven't heart from DO. Also now > that the HN topic is not on the front pages no one is getting updated about > the situation. That's why i wanted to write this post.

I won't speak for everyone, but I can personally live without having updates on the OP's Digital Ocean drama on my HN homepage at all times.

</comment><comment><USER30> > DigitalOcean requested me to send a paper which authorizes them to disclose > the information in public. It was midnight here in Turkey and i was asleep.

> And now after nearly 10 hours or so i still haven't heart from DO

What about timezone differences? You were asleep, they could be, too.

<comment><AUTHOR> That's a possibility. But at this time yesterday they were really really active.

</comment></comment><comment><USER31> that's rough

they obviously only care about conversion and not validating the user before hand in (any) way, so brutal.

From verizon I got a similar answer they couldn't discuss my own account with me after me giving them my social and all that - wtf. i hung up and tried another service rep and they gave me too much of my own personal info with out verifying any of my personal details and anyone elses' on my plans, which is kind of scary (Wtf)

also i really really like the idea of daily backups to dropbox, with like 1TB of backups i need a solution, and this seems like a great one. thanks for coming up with that

<comment><USER32> you're going to back up 1TB of data to dropbox?

</comment></comment><comment><USER33> Funny that DO's public response was to demand permission to discuss the case _publicly_ , when apparently they had not discussed it with the customer concerned!

<comment><USER5> I think they have. The customer in question simply didn't put it in his blog post. I feel as though the customer may be presenting the facts in such a way to make DO look bad. I don't see anywhere in his blog post about his DDOS attack. His server was shutdown because it was running a DDOS attack. My guess is that his server got owned...

People get disproportionally mad when their server gets shutdown, for a service that charges $5 a month for a VPS. Most people probably wouldn't get angry at Netflix if they cancelled their $7 a month account, but shutting down a VPS feels like a personal attack.

If you pay $5 a month for a service, don't expect a customer representative to hold your hand and debug your server troubles.

</comment></comment><comment><USER34> I've also had a bad experience with DO in production. Moved to Webfaction and have never looked back. They are just excellent in every which way.

</comment><comment><USER30> By the way, someone should watch this thread and start saving all posts by the OP. Last thread he deleted a lot of them.

</comment><comment><USER35> I'm a DO user, I created my account 2 months ago and I'm a happy customer.

</comment><comment><USER36> Isn't that a major violation of their SLA? Have you consulted with a lawyer?

<comment><USER5> Even if they violated their SLA, the most DO could do is offer a $5 refund, which isn't that big a deal. It's probably too costly to hire a lawyer over a $5 server.

Also, I don't think the SLA would include things like getting hacked, or running illegal services.

</comment></comment><comment><USER37> Never heard on DO but will use them for a project soon - looks great

<comment><USER38> Indeed, never though a $5 provider would go that far for a noob on their network. Looks like their monitoring system is good too, took them only a couple of minute to take him down, which is good.

</comment></comment><comment><USER39> is this... a joke?</comment>


Ask HN: I feel like an 'expert beginner' and I don't know how to get better - <AUTHOR> I really want to improve at programming but I feel I'm kinda of stuck in this 'expert beginner' phase. I'm familiar with basic data structures and their complexity (linked lists, arrays, hash tables, queues, trees), some basic theory (DFA, NFA, CFGs, TMs, lambda calculus etc), and elementary math (calc I, II, III, linear algebra, ODEs, discrete).
I don't feel particularly confident in any of these areas of study, knowing the minimum amount to get me by. Sure I know what a NFA is, but ask me convert it to a DFA and I'm lost. I lack 'deep', real knowledge and am desperate to acquire this mythical stuff before it's too late.
In terms of areas of interest, I am interested in type theory, FP, compilers, machine learning, expert systems and AI.
So knowing this, how would I best use say 8-12 hours a day, every day, to learn this stuff? Should I be reading papers all day? Should I be writing code? Should I be doing exercises in books? There's just so much material I need to get through that I am lost on how to actually spend my time getting better. <comment><USER3> Do you want to be a programmer or a computer scientist? They're both options. One of them ships software for a living, the other ships journal articles. (This is a joke grounded more than a little bit in reality.)

Your list of interests sounds like that of a computer scientist, but I'm allocating a little space to "Maybe your only exposure to computer programming so far has been undergrad CS and you're describing it in the only language anyone has ever taught you."

The best way to skill up as a programmer, if you're interested in that, is to ship more software. There is deep, mythical knowledge in programming, and it is acquired with sweat on your brow and a pickaxe in your hand chipping away at a coal face made out of Why Doesn't This Freaking Work.

You will note that most programs you work on build on things you have heard about in CS classes but do not implicate most of them directly. Compilers are very nice to understand. If you want to be a programmer, absent you making the goal of working on compilers your life's work for the next 10 years, you will probably not work on compilers. Most coal faces sound like something rather more like "A business has some fairly straightforward transformations to be made on two sets of data. There's plenty of opportunity for smart implementation choices and cleverness, but that mostly informs the small rather than the large of this project." (Cleverness in the large in programming is good selection of which coal faces to attack at all. After you get there people often stop calling you a programmer no matter how much programming you do.)

<comment><USER4> "There is deep, mythical knowledge in programming, and it is acquired with sweat on your brow and a pickaxe in your hand chipping away at a coal face made out of Why Doesn't This Freaking Work." That is a beautiful sentence.

<comment><USER3> Thanks, I try.

</comment></comment></comment><comment><USER5> Either build something, or go to the academy. If you build something, you'll do a lazy evaluation of the knowledge tree, learning just what you need to get the job done, which will be a very small subset of all that you described above, depending on the domain of your problem. However if you want knowledge for the sake of knowledge, get a master degree / PhD, it sounds to me it is exactly what you are looking for.

Also you don't sound line an expert beginner, you sound like a beginning expert, I'm programming for 10 years for a living, built a couple of money making startups, and doing my MSc at the moment, I can't say I know half of all that at an expert level ;)

Finally, accept that you don't really have to know everything, and more so, you can't really be an expert in everything, it's really hard but CS is a very wide field. You can't help but being a beginner expert on a wide area of topics, and only a "real" expert in a very narrow subject. Not all CS Phds are expert in everything, actually they are most likely expert in a very narrow set of topics relevant for their research.

Bottom line, either learn for learning and do it in a place that honors it (academy or independent / commercial research) Or build something that makes money, and the subset of human knowledge needed to make it work will be defining itself (you'll have to fight the urge to learn things that "you'll probably need later", and make it more "on demand" learning)

</comment><comment><USER6> To become a better programmer and not a better computer scientist:

Do your own thing. Build something you want to see built and you will learn oh so much. Programming isn't about how you implement something so much as it is for what reason. Think of something you want to see built and figure it out from there.

For example, I learned Python by trying to write an app that would take my Shazam tags and convert them to a Google Music playlist so I could more easily remember songs to listen to later.

Notice that I didn't write anything about SQLite or how slow Python's HTTP was when making the queries, because in learning Python, that wasn't important; those things were just implementation details that I only started thinking about after my application was demonstrably "slow." And more than that, I really didn't consider anything about lambda calculus and I don't know a single thing about NFAs or DFAs. I just wrote an app.

And I learned something, enough to get the job done. If you really want to learn about how to apply functional programming, learn Haskell. If you want to learn about compilers, write a compiler. You'll learn enough, because there's no way you can learn "everything" on a topic in Computer Science.

The field is much too broad, and you'll be way better off lazy-evaluating it than calculating it wholesale.

Tl;dr: If you want to be a better programmer (as opposed to being a better computer scientist) build and the knowledge will follow.

<comment><USER7> This, a thousand times this.

Don't go straight for an open source project unless you're really into it. Honestly - debugging other people's code is hell, I hate doing it (but frequently have to). By all means dig through other people's code to work out how something is achieved, but for people starting out, bug-fixing is a bore.

Make your own damn open source project! Pick something you love that has a problem and have a bash at fixing it. An here's the best bit: you'll still get to solve bugs, lots and lots and lots of them. Some of them will be easy to solve, some will take you a week before you realise you missed out a bracket.

You have to enjoy what you do or it won't stick. And don't be fooled into thinking you like something until you actually try to build something with it.

</comment></comment><comment><USER8> Simply put you have to start practicing. Try to solve problems with the techniques you've learned, and you will get a better understanding of their strengths and their weaknesses. That will help you internalize the theory you have absorbed. It does take time (this annoys most people) but it serves the exact same purpose as homework does in school, moves the understanding from the hippocampus out through the rest of your brain.

<comment><USER9> I agree with this, however I'd like to add to it.

When you build stuff for the purpose of practice also make sure you build it incrementally. For example, design a file format, build a parser/generator for it. After that, add networking, so two apps can communicate the contents of the file. After that add encryption (nothing fancy, just use a library), then add some TLS, and then, and then, and then.

I found this approach best because:

1) You learn new things (because you research best way to do stuff, experiment with code, etc), and 2) you learn from your mistakes, because as you add you might notice that certain things don't play with your new features, and you will be forced to improve them.

P.S: I know my example might seem a bit lame, but I couldn't think of anything else at the moment.

</comment></comment><comment><USER10> I'm always suspicious of the advise to work on open-source projects, or practising programming on your own, or online coding academies, etc. I feel these are without purpose and people soon lose interest since there is no goal (although helping out on an open-source project is certainly noble).

My advise would be to look at your own network (friends, family, etc) and find those who are in business and ask them about their pain; and there is always some pain that a business has. Then figure out a solution to their problem and program that. This serves 3 purposes: 1) It has a definable goal and purpose (solving the pain) as it's a real-life project. 2) You will learn tons about yourself, programming, and the business. 3) It could lead to either employment or a program you could sell to others and start a business.

As always, make sure you write-up a contract which states that the IP is yours. Hope for the best; plan for the worst.

<comment><USER11> > _I 'm always suspicious of the advise to work on open-source projects, or > practising programming on your own, or online coding academies, etc. I feel > these are without purpose and people soon lose interest since there is no > goal (although helping out on an open-source project is certainly noble)._

imho, if you work on an open source project and you find it (or your relation to it) being "without purpose," you're doing it wrong. You've picked the wrong open source project, the project is an undead zombie, whatever. If you do it right, it will most certainly ( _edit_ : "usually, at least") not feel "without purpose."

Perhaps consider looking at open source projects and your possible personal relation to any particular one from a kind of two-layer perspective:

1\. General idea behind the project; its general goals; and how you feel about them. It's probably a fruitless thing to work on something you don't believe in. On the other hand, finding something that shares your values to a great extent can bring out actual passion.

2\. This is (usually, sometimes) not enough: hence find _particular_ lower- level goals (these are the "ideological<->technical interfaces" so to speak, or, purely technical particulars (think of some very particular problem or proposed enhancement on a bug tracker.)) In my experience, the lack of _attainable_ (to you) lower level "what do i do" things (which you can use to deduce "how do i do", and then end up with a todo list of particulars, which always helps to curb general laziness (well, sometimes)) might run you into trouble (and maybe this was something the parent was actually commenting on, so we probably disagree on details only.) You have to also keep re-evaluating these lower-level particulars, and set out new ones, etc. (and this is not always easy, of course.)

A good combination of the two can be quite a powerful thing indeed. And if the project is lead by good people with lotsa experience, and if a wider community of smart folk is involved, you can learn a lot.

> _My advise would be to look at your own network (friends, family, etc) and > find those who are in business and ask them about their pain; and there is > always some pain that a business has._

But I agree that this is probably also a great idea, and sometimes more practical, etc. (also, learning to translate business (and sometimes it is an actual business) needs into solutions / technical designs is a very practical skill indeed)

</comment></comment><comment><USER12> Pick a Free Software project that you use regularly (possibly one without a CLA to begin with if you want to avoid paperwork), preferably one written in a language you already know, find the bug tracker, and find a bug (or wishlist item) that sounds interesting but _really_ trivial. (No, really. The first time you do this will probably be a lot harder than you think it ought to be. If it's not, no problem, pick something less trivial second time around.)

Download the code, figure out how to build/install it, and start to find your way around the codebase to try and figure out which bit of code is at fault/needs extending. When you get stuck, ask on the dev mailing list or IRC or whatever comms channels the core dev(s) have.

A lot of programming is not about designing something elegant and new. It's maintenance work, fixing bugs, extending functionality, adding new features. Sometimes adding exciting new features is a chance to design something elegant and new, but other times it's a bunch of repurposing and refactoring some features that are mostly-there under the hood, but need a couple of tweaks, and a small amount of really new stuff (but in the same idiom as the rest of the system) and exposing in a new way.

You'll really find out how to properly spelunk into a codebase (which is a complementary skill to just reading code), how code is used, and how it solves real-world problems.

If another dev solves the bug before you do, that's not a problem. The real purpose of the exercise was for you to learn, and only perhipherally to help the project. As a bonus, you can see how the other dev solved the problem, and how their solution differs from yours. Did they solve the bug at the "same level" as you? Was their fix a bigger or smaller change than yours?

If you get there first, great! Submit a patch to their mailing list, or a pull request to their git page, or whatever they use. Do not take their criticisms of your work as a personal insult. (If they do insult you personally - <USER2> almost certainly won't happen, but very occasionally does - that's another matter. Drop it and find another project. Life's too short to waste on asshats.) Rather, listen to exactly what they don't like about the way you solved the problem, use that to fix the problem in a way they will like, and re-submit.

Repeat.

</comment><comment><USER13> The most important advice that comes to mind is a bit proverbial, and from a board game so appropriately titled; "Go."

 _Lose your first 50 games of Go as FAST as possible! Don 't worry about winning or losing or finding the "right" move, just put some stones down, get used to looking at the shapes that come up, and get a good feeling for how the rules work. Of course, a consequence of that attitude is that you will lose most of those games, but it doesn't matter. Once you have a bit of experience under your belt, then you are ready to begin_

<comment><USER14> Any idea where that advice comes from, I've given similar advice in the realms of learning to throw pots (on the potters wheel); would be nice to ground that advice in a source as ancient wisdom if it is indeed that.

<comment><AUTHOR> Off-topic but I tried ceramics and found it to be the hardest thing I ever had to learn. Ceramics seems to be unique in that your creations literally get destroyed right in front of you with no way to go back. (I'm told ceramics people dream of VCS' in their sleep :)

<comment><USER14> Version branching physical items would be great in many areas of life.

</comment></comment></comment></comment><comment><USER15> Solve real problems - write programs that "scratch your own itch" \- and learn as you need to. It's hard to tell how well you're learning when you don't have actual results to go by.

Honestly though that set of interests doesn't seem like the sort of thing that lends itself to real problems; it sounds to me like you've already got more than enough CS theory. If you're looking to learn the kind of programming that will make you better at doing it professionally, you need to start making practical things. Find a hobby that could do with a website/app or something else in your life that can be automated.

<comment><USER16> Either that or work at a company that provides you a rich enough set of problems where you can apply your disjointed knowledge since it'll force you to synthesize it into a working solution.

</comment></comment><comment><USER17> I'm going to recommend if what you're doing isn't working for you, that you should try a different approach to see if it can get you the momentum you're looking for, in addition to what you've already been doing:

Take a break from focusing on the strategies and tactics of programming. No single algorithm, framework or language is going to unlock the panacea of potential inside of you. Most problems simply aren't that complex in the beginning.

Having the mindset of an innocent beginner always is the toughest thing to maintain to remain a problem-based thinker instead of a solution-based thinker.

The best way I've found is simply to solve problems the best I can, and when I learn what I could have done better, if the need is there, go and refactor it. No matter how great you are, or aren't today, what you write may look bad in 5 years because you have more experience.

Most things we build as developers become obsolete. It's a separate discussion but I'm not sure what you're trying to optimize, your skills, or a result in a project?

No developer is a a factory of churning out code or results at the same speed. Be less focused on the practice of programming alone and look at the results you're creating.

Software is as much a craft, an art, as much as a technical skill, and maybe it's something for you to explore the other fronts.

</comment><comment><USER18> I think some of it might just be that CS theory sounds intimidating. But if you paid attention in college (which it sounds like you did), you might find that papers really aren't that hard to read. The hairy bit comes up when you don't know enough math, but this is typically more a problem if you're trying to learn something specialized (e.g cryptography).

Also, there's a disconnect between academia and the workforce. You might invent a fancy new data structure that blows up in real use because it ignores caches. Or you might invent something that's cache-aware, but everybody's using scripting languages or the JVM and doesn't have that level of access to the hardware. And so on. If you're strictly speaking about becoming better at the work of programming (rather than CS), that stuff isn't that important. You will spend way more time, especially as you move up in seniority, interacting with people - whether it's designing your system to be understandable by humans, persuading others of your ideas, getting funding, etc.

</comment><comment><USER19> Seems like you fell into the typical CS trap. The problem is that Computer Science is not software engineering, and the dev skills you need aren't going to magically fall out from between the pages of CS books. What you need is more dev oriented knowledge and experience/practice.

A couple books that are worthwhile: "Refactoring" and "Rapid Development". That will teach you a lot of basic skills in terms of development process and how to improve the design of real systems (warts and all). Also, take a look at the Architecture of Open Source Systems, it'll acquaint you with how applications fit together.

Also, take on some projects. Pick something interesting and work on it. Pick some small stuff then move up from there. I'd suggest in your case eventually building a compiler. If you're interested in AI, build some simple games and work on building AI for them. There is nothing more important than actually writing code.

</comment><comment><USER20> I'd suggest drilling down into ONE subject area. If you want to become the world's foremost expert on NFA to DFA conversion techniques, go for it!

Forget about trying to learn EVERYTHING. Pick one thing, learn it well, repeat with the next thing.

As mentioned, the more you can do this within the confines of projects you feel passionately about, the easier it will be.

</comment><comment><USER21> Some remarks:

\+ There's a traditional sequence in skilled trades of apprentice -> journeyman -> master. A contemporary trade probably throws student on the front end and perhaps renames apprentice to 'intern'.

What separates a student from an apprentice/intern is the type of problems they work on. The same idea distinguishes the journeyman from the apprentice except that the journeyman is expected to successfully solve the problems they work on. Etc for the master.

What I am getting at is that there is a range of expertise and that what marks someone as an expert is the sort of problems they solve. But it's critical to realize that the context in which they solve those problems matters. This week thousands of students will be writing quicksort code. In 1960 it was the stuff of ...well CA Hoare probably was an expert programmer in an absolute sense, and perhaps a new journeyman among those who were programmers. Here again, context matters when talking about expertise...the one-eyed man in the kingdom of the blind and all that.

\+ It's not really clear what you mean by 'expertise'. An expert AI programmer is someone who has solved expert level problems - one in an academic setting earns a credential. In a vocational setting it's going to take several.

But in both settings the context is years, not hours per day.

\+ Sure you're free to define what it means to be an expert. Regardless of what definition you choose, the question remains, what does an expert do that you aren't doing? This isn't a question of research. It's not a question of methodology. It's a question of personal opinion - that is, what activities would you have to do to meet your definition of an expert. Maybe that's working at Google. Maybe it's earning a phd. Maybe it's writing a replacement for HotSpot.

\+ Once you know what sort of problems you need to solve to move toward being an expert, perhaps just trying to solve some of those problems is the place to start. I.e. what sort of problems does an apprentice or journeyman tackle?

Good luck.

</comment><comment><USER22> Trying to learn Machine Learning stuff I realized I didn't remember much Linear Algebra from my college days. Im also a beginner in Haskell. So I decided to crack open my Linear Algebra book and start implementing what I was reading in Haskell. The result of this was that my Linear Algebra knowledge that was all hazy now feels like its etched out in my brain like a digital circuit board :-) And my Haskell calf muscles are in much better shape. Im still a beginner in Haskell but I can now walk fast in it instead of just crawling like before. Hopefully in a year I'll be running in Haskell too.

</comment><comment><USER23> I've felt like this my entire life. You're in a good spot. You know a lot about what you don't know. You are prepared to learn these things as you need them to reach your goals. It's the people who feel like they _are_ experts who have actually stalled.

You can easily spend the rest of your life shaving these mythical yaks and when you are old and surrounded by mountains of yak hair, you'll feel exactly the same as you do now.

There is more to learn than you can ever hope to. Just learn what you need as you go. It's the key to all existential crisis: Try not to think so much.

</comment><comment><USER24> >Sure I know what a NFA is, but ask me convert it to a DFA and I'm lost.

Do you really think your average HNer knows how to do that? Or even know what that means and how you define it? I really don't think so :)

In my opinion you know plenty about computer science, certainly much more than your average programmer, but I understand how you feel. I think of myself as not knowledgeable enough and below average all the time.

You haven't mentioned your goal. Do you want to eventually work in any of the areas you mentioned? Or rather you want to learn this for your own intellectual curiosity?

</comment><comment><AUTHOR> PS: I guess what I'm looking for is a 'work-out' type procedure. Like practicing to run a marathon, you work out and run a lot (I assume). What should I do in order to 'work-out' my CS skills? Musicians practice, athletes work-out, what do computer scientists do in order to improve problem solving skills and knowledge?

Unlike for athletes or musicians, practice in a knowledge field (like CS), seems a little ill defined..

<comment><USER25> Consume and create. Depending on your aims focus on consuming either papers/books or code and producing the same.

</comment></comment><comment><USER26> There are few choices and/or some combination of the following:

(1) Start committing to Github daily. Make your own project and just go with it until you it's finished and learn what you need too.

My example project: [http://austingwalters.com/openbkz/](http://austingwalters.com/openbkz/)

(2) You can go through online courses in an attempt to learn more and become an expert, many courses are online. You could also start following blogs, read papers and replicate results.

If you are interested in maximizing learning: [http://www.scotthyoung.com/blog/](http://www.scotthyoung.com/blog/)

(3) The past month I have been writing a blog a day, it increased my productivity and forced me to really master topics:

My explanation on blogging: [http://austingwalters.com/learning-through- blogging/](http://austingwalters.com/learning-through-blogging/)

I recommend blogging, videos, or writing stuff down for "teaching" others because it really helps you master a subject and helps you think of ways to use those concepts.

</comment><comment><USER27> Love what you do and keep doing it, after 25 years I still aknowledge that I am a beginner in some areas and an expert in other.

Above all, always have fun!

<comment><USER28> I came here to say this, also have 25+ years experience. I've noticed that a lot of us with extreme experience feel the same way. In this last year, yet again, I look back at last year's code in horror. Every year it's been the same.. 20 years ago I was producing v. horrible looking code - but it still worked well.. all this zen stuff is fun, my ONLY gift was that I did things as well as I could at the time.

NEXT YEAR!! woohoo, I can't even imagine how good I'll be - this year will seem like crap.

Basically don't fret kid, we're all learning - chill and do your best.

</comment></comment><comment><USER29> When looking to improve your programming ability, it's important to ask for what end you are improving your skills. If your goal is to become a professional computer scientist, then you'll need to choose an area of focus. The field of computing simply has too many facets for anyone to master them all. If your goal is to learn, simply for its own sake, then all that matters is that you are enjoying the learning. It won't matter what technologies you use, or if what you're making actually works, so long as you enjoy it. In either case, you should choose something to learn that interests you. There are many highly sought after computer scientists, who specialize in computer vision algorithms, for example, who get recruited by companies all the time, despite knowing little to no software engineering and data structures. TL;DR: Learn anything, it will get you places.

</comment><comment><USER30> I've learnt so much over the years and barely remember any of it. I wonder whats the point of studying all this if I am going to forget 90% of it in couple of months. How do people deal with this? Should I convert everything I learn into some sort of cliff notes for rapid reacquisition of the knowledge at a later time?

<comment><USER22> Yes. Atleast that's what's worked for me. Once I started writing compact notes to help me remember stuff, I found that I still forgot that stuff after a while but re-reading the notes was a great optimization trick in re- learning/remembering the subject matter. I don't think this would've worked for me if my notes were verbose. There is probably an art to this, some sweet spot of information density to verbiage ratio.

Do other people do this too? It would be interesting to hear how other's solve this problem?

</comment></comment><comment><USER31> I recently got out of college I'm going through almost same situation. What I've learned is you are eventually going to forget many things after some time, and you can't know it all. Secondly, knowing theory is one thing, making use of what you know is another. You can grasp all theory you want, but the main point is how are you gonna put that into use? A good job where you get to solve interesting problems may help. I couldn't do so. What I'm doing right now is, pick up a subject area and build things from scratch. I feel that this is working for me.

Also, I see a pattern that more you are used to studying more quickly you tend to understand something new. I believe its something do with reading habits. So don't leave theory completely, but keep a balance b/w theory and practical.

</comment><comment><USER32> > I lack 'deep', real knowledge and am desperate to acquire this mythical > stuff before it's too late.

I mean, there are books on this stuff? Read them, do their exercises, and you will be fine.

I will say, most of this stuff now functions mostly as mind training. The specifics about low level computer systems, Big (O), parsing and compilers is the only part I see repeated use in my day to day.

>So knowing this, how would I best use say 8-12 hours a day, every day, to learn this stuff?

You should audit courses at a CS program taught near you. This stuff is chunky and the books are designed mostly for a classroom setting

</comment><comment><USER33> If you want to improve at programming: follow the advice about either making a personal project or working on an open source project. The fastest way to get better at programming is to program more.

However: it sounds like your interests are actually more in the realm of Computer Science than Programming, though: type theory, theory of computation, compilers, AI, etc. are less in the realm of programming and much more in the realm of computer science. You don't really need to know how an NFA or DFA works in most day to day programming, frankly. For learning Computer Science, I would suggest finding an advanced course with all of its material online (MIT OpenCourseware is a good place to start, or any of the other free online courses places), and working through a course on the topic. If it has prerequisites that you don't know any/most of the material from, then find a course on those. In my opinion, the other common options (reading through a textbook, reading papers, working on a project related and just learning what you need for that) all have flaws: textbooks are usually designed to accompany a course, which means they usually have much more than you need to learn a lot about the subject (and, in my experience, many are unbelievably boring and poorly written; if you want to go that route, make sure you find ones that are appropriate for self study). Reading papers is really interesting (and a lot more fun than reading textbooks), but without context or knowledge about the area, it's hard to evaluate the paper's meaningfulness/claims/etc., and hard to decide which papers are important to read. Working on just a project (e.x. just writing a compiler) leads to learning just enough to make that project, and not more. If you want to improve at programming: follow the advice about either making a personal project or working on an open source project. The fastest way to get better at programming is to program more.

Frankly: it sounds like you are decidedly not as much of an 'expert beginner' as you think you are. Familiarity with basic data structures+complexity, an understanding of theory, and an understanding of math through linear, ODEs, and discrete already puts you on a very firm grounding. (Outside of certain very specific parts of CS, you /do not/ need more math than that. If you're going into graphics research, knowing differential topology+geometry might be handy. I'm having a hard time thinking of things other than that, though.) From your description, you have more or less finished the 'beginner' stage, moved well beyond the 'novice' stage, and are moving towards becoming someone with a lot of advanced knowledge. Just knowing that you're interested in things like type theory+compilers/machine learning+expert systems+AI already puts you /way/ beyond being a beginner.

Shameless plug of someone else's stuff: if you're not sure where to start on the courses front, and want to start on the programming languages side of things (compilers, etc.), this course might be a good place to start: [https://www.udacity.com/course/cs262](https://www.udacity.com/course/cs262) I haven't worked through this course in particular, and I don't know its exact difficulty, but I took undergraduate programming languages from this professor and he's an /amazing/ teacher.

Edit: don't know why I had two copies of that written, but now there should only be one. Also, cryptography is another part of CS that you need more math (in this case, abstract algebra). (More multiple copies? I seem to be pretty bad at this "say things once" thing, and need to make some preference changes.)

<comment><USER26> Wow, awesome response.

<comment><USER33> I recently finished up my undergraduate education and realized there were certain large important parts of computer science I knew very little to nothing about (I stopped really taking CS classes to go take a lot of advanced math courses my last two years, which I'm not certain was the best decision, but it wasn't a bad one either), so I've been thinking a lot in the past few months about what my best options were for learning about subjects I'm weak in or knew next to nothing about. (E.x.: implementation of databases, OS design, AI, HCI, and other fields) Putting that into words wasn't too difficult, and hopefully it'll make figuring things out easier for someone else... I tried the reading papers and textbook approach, which is part of why I wouldn't recommend them.

<comment><USER34> >I recently finished up my undergraduate education and realized there were certain large important parts of computer science I knew very little to nothing about

Get used to it, the field is huge. You will only every have a surface level overview of "everything".

</comment></comment></comment></comment><comment><USER35> On the Computer Science side of things. You are spreading yourself too thin if you say that you are interested in type theory, FP, Compilers, and so on. Each one of these fields is a discipline in itself. You could spend years and years acquiring the 'deep' knowledge of one of those itself. Don't set yourself up for impossible goals. Try to focus and selectively figure out what you want to do and how much "depth" you want to acquire.

> mythical stuff before it's too late.

Too late for what? What is the hurry?

</comment><comment><USER36> Build something that is doable but will push your current skills. Choose a project that appears slightly beyond your current capability. That is the only way to truly learn.

</comment><comment><USER37> Build something. Resist the urge to use all that theory to make it optimal. Just make it work. From what you've posted, I'd suggest inventing a "toy" language and writing a compiler or interpreter for it.

One of my favorite sayings goes: "What's the difference between theory and practice?" "In theory, there is no difference".

It sounds like you have the theory covered pretty well.

</comment><comment><USER38> Maybe try something pretty. Information visualization with [http://d3js.org/](http://d3js.org/) Or make a game with cool AI.

Whatever you do, as long as there is a minimum amount of fun in it, you'll get better. And the more you know, the better you can identify what areas you have and/or want to improve in.

</comment><comment><USER39> 1\. Keep trying.

2\. Books and papers are great, but you should also try to find a mentor / guide / teacher / expert to talk about the issues you're facing and what you're thinking about. Schools are organized the way they are for a reason, even if they often fail at their intended purpose.

</comment><comment><USER40> Ship a product. More specifically, think something up, write it, ship it. Along the way learn what you don't know to finish it, but the first time you try this don't push yourself too hard. Once your done, post a "ShowHN" and we will probably tell you whats wrong with it.

</comment><comment><USER41> Get an internship at a company you think you will like to work at, something with 10-15 people I think is ideal. If you can't find one yet make some small projects help some friends or build something at a hacker space and try again.

</comment><comment><USER42> 1 read abd understand other people's code

2 build, then rebuild, your own systems

After 10,000 hours of this, assuming you keep climbing rather than just doing the same thing over and over, you will be competent.

</comment><comment><USER43> I tend to be pretty mercenary, but I think that writing software in exchange for money will be the most educational thing you can do right now.

</comment><comment><USER44> there's not limit for creativity.Programming is about creative. Look around your and pick one problem and create a project to solve it. Ask people around your ,it is good or not. I suggest you to take part time in non computer department and see how /anything can be improved via software you make/build.

</comment><comment><USER45> Make a project.</comment>


Show HN: Mixable ambient sounds to help you concentrate - <AUTHOR> http://asoftmurmur.com/

<comment><AUTHOR> This is the first thing I've ever 'coded' (if you can call it that), so it's pretty rough around the edges.

It's pretty similar to RainyMood [1] and Coffitivity [2] but with the added feature that you can play more than one type of sound at the same time and adjust the volume of each to find a mix that suits you. I was inspired to make this because that's something I ended up doing manually quite a bit.

It's HTML5 with jquery and jquery-mobile, mainly for the slider elements (which I didn't know how to make myself). I wish I could have avoided using jquery-mobile because it was quite a headache trying to deactivate a bunch of the default features and just use the slider.

It uses the standary HTML5 audio player. When the play button is pressed, all the audio streams play. When the value of each slider is changed, the volume for that player is updated with JS.

iOS devices don't support changing the volume via javascript in any fashion, so this won't work on any iOS device. I don't see a way of getting around that. It should work in most other modern browsers. Seems to work in IE10, FF, Chrome and the stock Android browser.

The only slightly tricky thing was getting seamless looping for the audio samples. The HTML5 audio player has a "loop" setting, but in every browser I tried it in there was an audible gap between the end and start of playback. Very annoying in this kind of application. To get around that, each audio track fades in and out at the beginning and end. When the player reaches the beginning of the "fade out" section of the main track, about 10 seconds before the end, it triggers a 25s "glue" track to start playing, which fades in as the main track fades out. When the glue track begins to fade out, it triggers the main track to fade in again at the beginning. The end result is something approaching a seamless loop with constant volume. I think it works quite well for ambeint noise, but it obviously wouldn't work for anything with a beat.

The sounds are all CC licensed samples from freesound.org [3][4] that I did my best to mix and clean up. In the unlikely event this ever became popular, I'd love to make some field recordings myself to get a bigger range of sounds in higher quality, but that's a bit of a pipe dream.

        [1] http://www.rainymood.com/
        [2] http://coffitivity.com/
        [3] http://www.freesound.org/
        [4] http://asoftmurmur.com/info.html</comment>


How I Used Amazon's Mechanical Turk to Validate my Startup Idea - <AUTHOR> http://harperlindsey.wordpress.com/2010/09/01/how-i-used-amazons-mechanical-turk-to-validate-my-startup-idea/ <comment><USER3> I'd caution making your survey "Hey would you use Web app A?" The Mechanical Turk users want to blaze through the survey as fast as possible and get paid. Clicking "Yes" gets them through the fastest and also lets them please the surveyor (the same bias as your friends or family).

Force them to make a choice. Present Web app A (real), Web app B (dummy), and Web app C (dummy). Make them rank which web app is best.

Set up the survey three ways: A, B, C. B, C, A. and C, B, A. Have a third of the sample take each survey. You would be surprised at the first choice bias with Mechanical Turk. Actually you wouldn't be surprised when you remember that these people just want to get the thing done.

Finally a good secondary survey is to make them rank order features for their worth. This helps find your MVP. (Read more about "conjoint analysis" if this interests you.)

</comment><comment><USER4> It's not clear from the article what kind of business model the Startup in question is planning to use. This is relevant, as the fact the 73% of random strangers said they would "use the service as described" doesn't give any indication that they'd be willing to pay for it.

Clearly, the author got his $27.50 worth; how much that's really worth, though, in the long term, remains to be seen.

<comment><USER5> Yeah, I've done this before. Of the 200 people I surveyed a much higher % (around 95%) said they liked and would use the service...and then did not. My experience with this stuff makes me think that people who do it generally have a perception that you are their "boss" and that they need to sugar coat everything they tell you.

TLDR: Be HIGHLY suspect of any results you get from mechanical turk (same goes for feedback army and others).

<comment><USER6> It applies to everyone. Not just mechanical turk folks. When someone says they 'will use' your product they are non-committal. (I have done this so many times for the fear of offending the person who is asking me). When its time to take out their credit cards then they backtrack. They are 2 different decision points. The later is what matters to startups.

<comment><USER7> _When its time to take out their credit cards then they backtrack_

And to be fair, even if I am trying to be super-honest, I find it very hard to decide whether _I_ would be happy to whip out a credit card from a concept pitch. It's not just a question of "would I pay for A", it is a question of "would I pay for A it it really rang my bell", and I can't tell whether you are going to ring my bell or not until I have tried your product _and_ your customer support philosophy.

When I look at my credit card statement, I see a bunch of companies not just whose product I like and use, but also a bunch of folks for whom I can say "I like the way they do business".

I didn't pay to cloud-store photos for over a decade, until I found smugmug. So what is the right answer to the question "would you pay to store photos"?

This isn't the kind of nuance you would get from an MT survey I don't think.

</comment></comment></comment></comment><comment><USER8> Strange use of Mechanical Turk if you ask me. You could've spent about the same in Google Adwords asking people to take a survey. At least then you'd have a survey of "people who are searching for important keywords related to my topic of interest" rather "people who are so technically capable that they are using MT".

"I also found out to my surprise that more men than women said they'd use the service. I fully expected it to be the other way around." 200 people is not a large sample size and I would caution you against making sweeping decisions about the future of a company based on this small size. It could be that the MT program attracts more men than women? I don't know how you set it up but you said earlier that you didn't really do any segmentation.

<comment><USER9> It is still waaaaaaaaaaay better than asking your family and friends.

<comment><USER10> Dude. No.

They're both 100% useless.

Mechanical Turkers are incentivized to give positive feedback because there's a chance (lurking in the back of their minds, surely) that people are emotionally sensitive and if they get criticized, they will be emotionally wounded and won't approve their HIT.

Do you know how I know this? I asked Turkers to give me feedback on one of my sites, and it was nothing but sunshine and razzmatazz smoothies.

You know you're building something people want when they vote with their wallets or attention (give you their email address, phone number...)

New Theory: hell, an even better test might be are they willing to "invite their friends" to tell them about it? People share things with friends because they like their friends - is it of sufficiently high value that they will get a "social credit" for sharing with their friends?.

<comment><USER11> I guess you could ignore the 73% that said they would use it. But note that 27% said they wouldn't use it and also stated "WHY they wouldn't use" it. This should be at least 1% useful.

</comment></comment></comment></comment><comment><USER12> People who use AMT for earning money may not necessarily be the right audience you are looking for.

<comment><USER13> I don't think that people use Mechanical Turk as their sole source for money. If I remember correctly, someone figured out that mechanical turk pays roughly $5/hour if you spend your whole day doing it. That's not very much.

Mechanical Turk seems to be filled with people itching for some community to participate in -- why not get paid for it at the same time?

<comment><USER14> I've run an MTurk survey as well and asked standard demographic information. My results are surprisingly in line with national averages. I ran the survey at the suggestion of someone with much more experience running MTurk surveys, who claims he has verified that the results are _better_ than typical phone and mall campaigns.

<comment><USER15> >I've run an MTurk survey as well and asked standard demographic information. My results are surprisingly in line with national averages.

Do MTurkers tell the truth though? Presumably one person can run a dozen fronts so that they can answer questions that require a specific demographic. Or they masquerade as elderly females to appear more trustworthy and shame you out of objecting if they've not done the job properly or whatever.

<comment><USER16> Do you think they would lie so as to mirror the national averages?

</comment></comment></comment></comment></comment><comment><USER17> I would take the "would you use this service" feedback with a grain of salt.

It costs you nothing to say "yes", but everything changes as soon as you start to charge for it.

Going from Free to $0.01 for any online product or service significantly changes your conversion numbers.

You might get better feedback standing on a corner downtown, asking if people would use your service... and charge them $1 for a voucher that would allow them to use it when it is ready.

PS: Please get out of stealth mode!

</comment><comment><USER18> For anyone else interested in this "quick, cheap, feedback" cycle, PickFu is another service that does this: <http://pickfu.com/>

I spent about $20 to get 200 responses for my question, but only ended up collecting 103 answers before the question expired.

None of this stuff is perfect, but employing a few of these approaches all at the same time will generate results that can be handy to correlate.

<comment><USER19> Actually, it didn't expire, it finished with 104 of 200 answers picking "B - Paypal" <http://pickfu.com/00PULC>

Also, just to clarify, it's $17 for 200 responses.

Thanks for trying out PickFu!

<comment><USER18> Justin, my apologies on that, I misread the screen and thought I only got 104 answers.

It was a great experience overall, and for micro-testing (where something like Usability testing is too much) PickFu is on my short list for future projects.

<comment><USER19> No worries. Other people have misread it too so we probably need to tweak the wording to make it more clear.

Glad to hear it!

</comment></comment></comment></comment><comment><USER20> This is Lindsey - who wrote the post. I wanted to clarify a couple of things, based on the feedback I'm seeing here. (btw. This was my first blog post :)). I am still in stealth mode (which is a completely different conversation that I'll be blogging about soon). www.Swayable.com will be a consumer facing free service. With that said, the Turk system will not work for a lot of startups that need niche specific data. I was looking for general consumers that are web/computer savvy and the Turk system fit the bill for what I needed. The most valuable feedback I got was actually from the text responses as opposed to the yes/no response. I will definitely follow up on this post as soon as I am out of stealth mode, and provide more specific data. Thanks for all the great feedback!

<comment><USER21> You are in stealth mode, yet willing to pitch the idea to hundreds of random strangers?

</comment></comment><comment><USER22> cool. My girlfriend had an idea about startup and we started discussing it. At some point I told her 'lets stop hand-waving and look at some data' . For ~10 bucks we got 110 good responses to 11-question survey on Mechanical Turk. It really gave us new perspective on some of our problems. That was much easier than asking our friends for feedback (which we did too).

</comment><comment><USER23> The post would be more interesting if we knew what questions were asked in the survey.

Edit: I meant how exactly were the survey looked like, including the service description. If it's a paid service, I highly doubt the 73% figure.

<comment><USER24> Me, too. I checked my stats after using MT for a site-survey. Most of the users were from the same geographical region, they gave very positive feedback, and they spent very little time on the site. Pair that with an understanding that MT only pays well if you do lots of little tasks very fast, and you start to get the idea that the feedback might not be the most useful.

<comment><USER24> You need to put in a text box: "Please describe, in your own words, the purpose of my site:" Then throw out all the responses that are incomplete or way off. The problem is that when I did that, there were too few useable results.

<comment><USER25> This is incredibly good advice. All MTurk tasks of a small size or that will be approved by a single person, should ask an open ended question, that requires a response.

This greatly reduces fraud, and shoddy work. And it has a tendency to highlight the really good workers (some of which are amazing!) - <USER2> your benefit as a requester.

</comment></comment></comment></comment><comment><USER26> Interesting article, but probably poorly timed submissing to HN. You could have used this article to promote your product, but you're in stealth mode so we get a "sign up to hear when my product is available" page, but still have no idea what your product is. Good article, bad marketing. Should have released this about a week after you launched the site :)

</comment><comment><USER27> I wonder how many of these 200 people were from India, which appears to be supplying a good chunk of Mech Turk userbase.

I also ran a couple of polls with similar intent on MT and then stopped doing that because results were skewed dramatically. For one there's no filtering by a geographical region and you bet your 10c-per-answer that you will get a _ton_ of replies from the poorest corners of the world. Secondly, responders _are_ afraid of you rejecting their answer and thus affecting their MT rating, so they tend to tell you want you want to hear and not what they actually might be thinking. Hence your 73% approval rate.

This sampling is not random, very far from it. You are effectively sampling MT community with its own dynamics, the community that is likely NOT to be representative of your own target user base.

<comment><USER28> I've been using CrowdFlower as an interface to MTurk, and it let me exclude specific countries. Your point about the secondary motivation is unfortunately valid, though.

</comment></comment><comment><USER29> Might be a bit off topic, but does anyone know a good alternative to MTurk for people outside the US?

<comment><USER30> I'm building an API on top of MTurk, <http://houdinihq.com>, that you could use. It's currently in alpha, but feel free to email me at email@example.com if you're interested in an api key.

<comment><USER31> Simple API looks really simple, something that I definitely like!

How do you plan to accept payments? How much would Houdini itself cost (per task, per month?)? Do you plan to accept non-US users?

<comment><USER30> Thanks! We also have some more advanced apis in the works, like asking multiple workers to do the same work and automatically determining the 'true' answer. From your point of view, they'll be just as easy to use though.

Pricing is still TBD. MTurk charges will be passed through directly to you with Houdini-specific charges either per task or per month.

Yeah, I'm planning on accepting non-US users.

</comment></comment></comment></comment><comment><USER32> I've seen weirder use for mechanical turk: <http://www.youtube.com/watch?v=D_CC5r5Wfm0> So, I guess why not?

</comment><comment><USER33> What if you asked for a follow up email address / phone number? The survey would then act as a screening device that would identify those who are really interested.

</comment><comment><USER34> In case anyone doesn't know what a Mechanical Turk is: <http://en.wikipedia.org/wiki/The_Turk>

</comment><comment><USER35> I think Amazon Turk could be better used for beta testing and early testing of word-of-mouth. First, ask some people to register on your site and then track what they do. Then, have them fill out a feedback form and tell you what they would have you improve. One iteration would cost you around $20. And if you do it once every 2 weeks or so, you can keep in touch with what your users actually want.

</comment><comment><USER36> Does Amazon say anywhere how diverse Turk users are? Maybe you ended up only surveying mostly one gender in mostly one part of the world?

<comment><USER37> When we were creating <http://markiter.com>

We did a blog post referencing a study:

<http://blog.markiter.com/#turk>

[http://behind-the-enemy- lines.blogspot.com/2008/03/mechanica...](http://behind-the-enemy- lines.blogspot.com/2008/03/mechanical-turk-demographics.html)

Which has been updated recently. Some high quality data presented in a clear way.

[http://behind-the-enemy-lines.blogspot.com/2010/03/new- demog...](http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics- of-mechanical-turk.html)

<comment><USER36> Thanks for that - this set is from Feb 2010

[http://behind-the-enemy-lines.blogspot.com/2010/03/new- demog...](http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics- of-mechanical-turk.html)

        * United States: 46.80%    (65% female)
        * India:         34.00%    (70% male)
        * Miscellaneous: 19.20%

</comment></comment></comment><comment><USER38> May be the survey data us worth useful if your startup's target audience consists of folks who earn pennies on the hour.

</comment><comment><USER39> Strange? Sure. Useful? Probably.. But I really like the novelty of looking at AMT as a group real people and not just commoditized, bite-sized, labor. Opens my mind up to problems that might be solved with AMT.

</comment><comment><USER40> Some good advice I've heard: When doing this kind of market research, don't ask people if they would buy, ask them to buy.

Of course, that's easier said than done in certain circumstances.

</comment><comment><USER41> Come back after you launch and tell us if the feedback was accurate- until then, you are half way through executing a (rather interesting, imo) experiment.</comment>