How Do You Find New Problems to Solve or Ideas for Products To Work On? - <AUTHOR> What is your thought sequence, particuarlly in a new venture that you're thinking about or have already started.  Do cool things just randomly pop in your head after whatever things you do through life?  As a follow-up, how do you actually validate whether or not you will actually move forward with it (market opportunity, personal problem, growth potential, ease/challenge of problem, customer validation)? <comment><USER2> I believe it is a way of thinking. I once told someone who asked me about the same question "How can you _not_ see new ideas here?"

I'm a problem solver, and when there is a problem from "keys locked in the car" to "too expensive to move mass to lunar orbit" I just start speculating solutions. Knowing that any problem is in fact a series of problems the other problems get solution speculations on their own, and out of those hundreds or thousands of ideas pop out things that occasionally are both implementable and monetizable.

To give an example of how I think, one day when I was taking a shower I recognized that I'm so busy thinking about things that I sometimes lose track of whether or not I've done one round of shampoo or two. So that suggested a solution where I could make a note after the first shampoo and after the second. And then looking what I had to work with I realized that I could algorithmically determine the issue if I left the cap open on the first go round and if I grabbed the shampoo and the cap was open it was round 2 so just close it. Basically transferring the 'state' of how many times I'd gone around the loop in the state of the cap. Problem solved, no cost. Maybe a nice thank you if I sent it in to "Helpful Hints from Heloise."

So looking around at problems they come in some common forms.

They are the result of something being too complicated thus requiring either specialists or training time.

They are the result of not managing state (like my shampoo problem) where the stage of the solution needs to be recoverable.

They are a result of an economic barrier, so its too expensive to do X which causes problem Y, can you do X more cheaply?

They are the side effect of a desirable thing, so doing X is desirable but has (or sometimes has) undesirable side effect Y. Here the solution makes X better because you find a way to mitigate Y.

They are problems of comprehension. The human brain has a finite capacity for maintaining state, so solutions that either abstract out chunks (and free up the brain for more relevant state) or eliminate complexity, win in those situations.

Then there are solutions that are not currently possible, but would be if 'X' existed. For example, if you had unlimited energy you could easily create liquid fuels out of the air. Sometimes its useful to think about those kinds of things and watch for the 'unlimited energy' appearing in the news (as it would if we suddenly mastered the fusion cycle for example). A variation on this I was doing back in the 80's as a network programmer at Sun wondering what sort of things would be possible when everyone had high speed network access. Consider the more probable new capability like 'easy access to space'. So there are folks like Virgin Galactic, SpaceX, etc who are working toward making access to space available to non-NASA types. What problems are those people going to have? Gucci barf bags anyone? How about the sex market for them? We know that nearly every nerd who goes into space, if they have a wee bit of privacy is going to be thinking how they try out sex in space. What do they need to do that? Bigelow wants to put inflatable hotels up there, what kind of luggage do they need?

And there are all sorts of geeky technical problems like "how do you make white space spectrum available to everyone? ", "What sort of cash drawer thing is useful for an NFC enabled world?", "How do I enable non-technical people to have all their purchased video available at their TV?" (clearly the media PC hasn't quite gotten the recipe right there).

If the market is small ask what folks with high levels of disposable income might want.

If cars can self drive how would you build a parking structure for them?

How can I build something that would auto-plug in my Tesla and save it from being bricked without thinking about it?

What does 3D printing mean for the toy business? Can you build a device which parents of young kids or day care centers could use to generate a steady stream of safe toys?

What would it take to make a solar powered device which recycles plastic bottles or cans into something which is more easily processed into new products? Can you build a device which powderizes these materials and then feeds them into feed hoppers for an industrial scale 3D printer? Bottles in one side, park benches out the other?

The thought sequence, venture wise, is always the same:

1) How many people have this problem.

2) How many other ways are there to solve it or live with it.

3) What would it cost to offer a solution?

4) How does the cost compare with the pain of the problem, and the cost of other options (including 'doing nothing'!)

You can use personal experience to work out some intangibles (like how many lines of code would it take and how long to write) and sometimes you have to construct small experiments which help inform the outcome. The key is that if you are ready to accept that your idea isn't feasible (productwise) to just move on. There are lots and lots and lots of ideas out there.

</comment><comment><USER3> There are tricks to get more of _any_ kind of idea, like keeping a notebook with you and jotting down suggestions to yourself during the day, then doing triage and record the best ones at the end of the day. Once you get into the habit of looking for ideas, you'll start to see them everywhere. Most will have fatal problems (and practice also makes you better at seeing these quickly), but some won't.

The next step is to break out of your personal routine and small circle of problems. If you're only building ideas based of your own life, and your life is generally pretty comfortable, you'll only be solving problems that are relatively meaningless in the grand scheme of things. So get out of your own neighborhood and immediate circle of friends, and learn how to talk to people everywhere. Everyone has problems (and ideas on how they should be solved).

For validation: you definitely need to put in the time talking with the people who you imagine most need what you're building. People have habits, and solutions of some kind already in place for any problem -- you need to give them a solution that better _enough_ that they'll change their habits for it. Find out in detail what they already do before you try to "fix" it.

<comment><USER4> _If you're only building ideas based of your own life, and your life is generally pretty comfortable, you'll only be solving problems that are relatively meaningless in the grand scheme of things._

There's so much truth in this. This combined with the other idea that you should 'scratch your own itch' can result in bit of frustration. Many (myself included) fall into this trap and find ourselves trying to dig deep to find problems we can solve. Sometimes we fool ourselves in thinking we have problems that we really don't have. Then you end up with _meaningless_ solutions or even worse, solutions looking for a problem.

</comment></comment><comment><USER5> After a handful of side projects that never got the traction I was hoping for, I came upon Bret Victor's recent CUSEC talk on principle based inventing.

Unless you're lucky enough to stumble on an idea that has instant traction, any path to success is going to be hard, with lots of periods of self/market- doubt, and lots of opportunities to quit. I think the only way to get through those times is pursuing an idea that stems from a deep principle you believe in. This is currently what I'm looking for: identifying my humanistic principles, and seeing what ideas are natural conclusions from it. No more random market based ideas.

<comment><USER6> Link to CUSEC talk : <http://vimeo.com/36579366>

<comment><USER7> Wow. Amazing talk.

<comment><AUTHOR> Agreed.

</comment></comment></comment></comment><comment><USER8> I keep a notebook on my night table with a pen to catalog my ideas as they popup. My ideas usually come as the result of me reflecting on the past day and letting my mind wander. I write everything down, even if it seems highly unlikely to ever come to fruition. I believe this is important as further ideas can be derived that are slightly more 'approachable'. I try my best not to force this process and let it occur organically. The notes are written as a stream-of-consciousness, I sketch and write to my hearts content hoping to capture the idea envisioned in my minds eye.

Validation: I don't validate my ideas as much as I should, this is a personal failing but its something I am working on. I do perform market research on each product and consider the challenges of getting a MVP out to consumers. I recently watched a talk by DHH at Startup School '08[1] which gave me a personal epiphany, sadly this should have been painfully obvious, that you can still do well aiming to satisfy the niche markets, chances are you will not be making the next facebook. While it is nice to dream of being the next big thing, I personally would be ecstatic to have a handful of people paying 5$ a month for a product that solved their problem.

[1] <http://www.youtube.com/watch?v=0CDXJ6bMkMY>

</comment><comment><USER9> I mainly work as a janitor so I think all day. Ideas come into my head constitly. The only problem was I would forget the the time I got home. So I started emailing my facebook account everytime I get a idea. Know the problem I have is I got alot of projects to work on. The good thing is most sound good in my head, but when I get started on them they suck so I ether drop them or add the good parts to other projects

</comment><comment><USER10> They come to me every few days. Ideas for inventions, ideas for startups, ideas for websites, ideas for designs. It's too much.

As soon as I start working on one, another comes to me. So it's kind of like torture. Ideas just keep coming and I find it hard to settle down and commit myself to one. I wish I had a whole bunch of money and just paid people to make them all. But it doesn't work that way.

I think it's my negativity and critisism. When I see something that's already been made, I look for its flaw. I almost always find something wrong or imperfect with it. So then I ask myself why it feels wrong, how it can be made better or different, and from that another idea and invention is born. Sometimes so different from its origin that it becomes a whole new different product or service. I mix and match different little fixes together and eventually they create a whole new product.

Other times, idea formation is just subconscious. It happens without thinking. It cooks in the back of your mind and when it's done it gets pushed towards your consciousness and you realize you have a "good idea". Where it came from, you don't know.

<comment><USER11> _As soon as I start working on one, another comes to me. So it's kind of like torture. Ideas just keep coming and I find it hard to settle down and commit myself to one. I wish I had a whole bunch of money and just paid people to make them all. But it doesn't work that way._

I can relate to that. I am having more than a dozen very good ideas(at least I think so). They keep coming and try to take my focus away from my current project.

<comment><USER12> what are some of those ideas u can share?

</comment></comment></comment><comment><USER13> The way I do it is two things. The first and most obvious (as most state) is to just live your life normally, and think about the things you are doing or observing. Why am I doing it this way? Is this the best way? X is so boring/tedious, surely there is a better way.

The second way is to read - not just articles or HN, but novels or non-fiction are another source for me. It functions in the same way as the first: chances are the main character or at parts in the story you will notice things that seem impractical. I noticed this while reading case studies at school, the authors usually explicitly point out that "this industry was fragmented" or so forth.

</comment><comment><USER14> Great question. I think the big thing is that you have to get exposed to different things. If you don't have your own itch that could be scratched for profit, then you need to find somebody else's. Some of the posters mentioned learning about different domains and niches; to me that's the key. If you are a problem solver with an entrepreneurial streak, you will find itches that need to get scratched just from talking to people about what they do.

The key is: how do you get exposed to new domains? I'm sure most business managers/owners could name 3 or 4 things they wish they could make better. However starting that conversation with them is difficult.

My wife does some direct sales stuff (Pampered Chef, Lia Sophia) and the key concept with their sales training is: get on the damn phone! Make 3 calls a night, that's it and your business will grow.

Idea generation is the same thing. Just think if you could have 3 conversations a day (heck, even 1 conversation a day) with people about what they do during the day. I bet you'd have multiple actionable ideas in a week.

It sounds easy and trivial, but it isn't. I know this works but I haven't done it yet. I'm not brave enough, charismatic enough, too big of a p __*y to just start up conversations with people, at least so far... But I think if you go to the core of most business startups, that didn't start with a guy/girl solving her own problem, you'll find that most of them started by putting themselves out there.

<comment><AUTHOR> Great points...I especially like "I'm sure most business managers/owners could name 3 or 4 things they wish they could make better. However starting that conversation with them is difficult."

</comment></comment><comment><USER15> Vision and principles is a good method. This is a great talk about that <https://vimeo.com/36579366>

Covers the inventions at Xerox PARC, how they were motivated by removing modes. Then how Brett Victor of Kill Math <http://worrydream.com/KillMath/> fame comes up with his ideas. They all stem from a single unifying principle.

First world problems though and no concern with markets and customer validation.

</comment><comment><USER16> Regarding your follow-up question, here are a few things I ask myself before moving forward with a new idea:

1\. Can I get the expertise/experience to actually create this? I may have the technical expertise to do it, but if I don't have the domain experience I may produce something that makes perfect sense to everyone except someone who actually needs the product. Not having the experience myself is fine, but I need a plan for how I'm going to get it or someone who can provide it.

2\. How would I market this? Having a good idea is one thing, but getting it in the hands of people who would actually use it is another. For me this will often mean partnering with someone who is already working with the customers I want to reach.

3\. What is the probability of success and what is the potential payback? It is easy to get so enamored with a problem that you stop to think about whether it really makes business sense or not. A project has a 0.1% chance of making $1,000,000, but a 99.9% of losing money is going to be less attractive (from a financial standpoint) than project that has a 25% chance of making $100,000.

</comment><comment><USER17> One of the ways is to go out of your niche. Out of the high tech stuff, and see what "ordinary" people with uncool professions need. Other option is to transfer ideas from one niche to another - it may look odd and weird at first but then lots of ideas will pop up.

I even do get some of my ideas from TV. From the news. From seeing a problem or someone's business model that can be transferred elsewhere.

<comment><USER18> I did this and I can see how "getting out of your niche" works but be careful that you know who your customers will be. I went into working with the real estate industry, particularly realtors. I now find myself really wanting to get into high tech because my personality clashes with a lot of my customers. For one, they are incredibly un-tech savy and 80% of them use some outdated windows computer with Internet Explorer 6 or something prehistoric.

I had to spend 30 minutes the other day explaining to someone how to download a picture and where it gets saved on her computer.

They also like to have meetings and conference calls on things that can be resolved with a 2 sentence email. There are also some things I'd like to be able to do like build relationships with my customers through social media and although many of them do use it, it's just not the same as interacting with a younger or more tech savy demographic.

So in the end, for my next business, I'm going to look towards finding a market where my customers are better suited with my personality and skills. I think it will not only be more enjoyable for me but also allow me to fully utilize what I am good at.

<comment><USER19> On the other hand, if you invest some effort to get more comfortable working with people like these, you could have access to a market that may have fewer competitors, since many tech-oriented people might have the same obstacles you did. And real estate agents may have real money to spend on a product that would make it easier for them to earn their commissions -- you could get a lot more money on a single sale than you could from selling iPhone apps to random people for $4.99 a piece.

<comment><USER18> Less competitors... Yes, quite possibly. Do real estate agents have real money to spend on product to make their life easier? Sure. Of course they do, but so do other people. Your point about $4.99 iphone apps, I'm not sure what that has to do with anything...

I also never said I wasn't comfortable working with people like this it's just that I wouldn't if I had a choice and knew what I know now. I do not agree with a lot of their work principles and I appreciate different things than they do.

For me, I'm still young and have the time to experiment in different industries it's not about jumping into the first market that has the least competitors, its about finding something I enjoy doing and am passionate about. I notice that is when I tend to do my best work.

I think finding a job or industry is like dating. You have a relationship with your customers and if they are 2 or 3 times your age and share no common interests, it can make for a not so great relationship.

</comment></comment></comment></comment><comment><USER20> I wrote a brief article about a way to select an idea from an idea pool assuming that you produced enough ideas. It's at <http://blog.databigbang.com/ideas-and-execution-magic-chart/>

Now, how do you find new problems? The first rule is (obviously) there are not rules. But you can improve your heuristics like using serendipity or strong market analysis.

Problems are everywhere, so this are the kind of questions I ask myself searching for opportunities:

\- Integration: Does technology X connects with technology Y in a seamless way? What can I do achieve integration. A subset of the (X,Y) is {('.net', 'java'), ('web', 'desktop'), ('e-mail', 'crm'), ('e-mail attachments', 'dropbox'), ('google analytics', 'excel'), ('gdata', 'odata'), ('mobile', 'desktop'), ('thunderbird', 'outlook'), ('mobile phone', 'asterisk')} You can add a relation R, like how can I R=migrate from 'thunderbird' to 'outlook' ?

\- Extending Features: Can I open different gmail/twitter/etc accounts in different browser tabs? Does Salesforce has all the features needed? Does Kindle export notes?

\- Development libraries: There are easy to use libraries to do X in the technology Y?

\- Changing industries: Does industry X have a problem that was already solved in industry Y?

\- Futurism: What we will need in one year? and in ten years? For example, do I need a voice based web browsing to use while I am doing other things? (like Siri does with e-mail). How I can integrate the information from a lot of sources (G+, Twitter, Facebook, CRMs, etc). Will the torrent protocol will be the future for streaming? in that case how can I add dynamic advertisement to torrent streaming? new games for toddlers (like the AI toy), and Turing test for toddlers?

</comment><comment><USER21> I'm still trying to figure out the answers to these questions. I get ideas about anything I pay enough attention to. Mostly this has happened in an academic context, so far. I had an idea that I intend to pursue in a startup- ish way, based on a way to modify to the bitcoin protocol to, roughly speaking, allow people to purchase time on the computational resources devoted to the blockchain lottery[1]. I got the idea when reading a thread on bitcointalk.org about a blockchain dedicated to cracking RSA keys. I went and had a shower shortly afterwards, and at some point during that suddenly it was clear to me how you could arrange such a blockchain in a secure way. It took a few months more thinking in my spare time to make the idea flexible enough to be useful, which sounds laborious, but was actually tremendous fun.

I'm still trying to figure out ways to assess the marketability of the idea. While asking people about the technical feasibility of the idea, I got some useful feedback on how it needs to be improved[2], but I'm still don't have a clear idea about how to reach potential customers[3]. My current plan is to survey people using GPUs for approximately embarassingly parallel Monte Carlo inference and optimization, because the computational constraints in my framework bear some resemblance to those of GPU programs, and I think GPU developers could migrate fairly easily.

[1] [https://docs.googlye.com/open?id=0B3qaT- ZL6aeKMWFhNmIwOGYtNW...](https://docs.googlye.com/open?id=0B3qaT- ZL6aeKMWFhNmIwOGYtNWM2Yi00ZTU0LTkxZjMtZGYzNjUwNWJhZTBm)

[2] E.g., [https://bitcointalk.org/index.php?topic=64421.msg757930#msg7...](https://bitcointalk.org/index.php?topic=64421.msg757930#msg757930)

[3] <http://news.ycombinator.com/item?id=3609991>

</comment><comment><USER22> To be honest. The best ideas come when you are not even trying hard at it. Like walking down the street, overhearing an argument, or my personal favourite - in the shower. But no matter what just keep a notebook.if that is too big, use the note pad on your phone. so that when even the smallest idea pops in your head (which you think is cool) write it down. It is not enough to just write it down either, don't stop mulling over it, for about an hour or so. (Because from a zen advice, if you go chasing your thoughts, you get too obsessed and lose opportunity for new and greater ones.) Consider a few things like possibility, implementation, improvement etc. After a while (say a day or two) read over your notes, if you still like the ideas, you just might have yourself a viable project.

<comment><USER23> True story, I have a whiteboard in my shower ha.

</comment></comment><comment><USER11> These are the keywords - See, Observe, Understand, Analyze. Finding news ideas or problems to solve is a way of thinking that anyone can learn. For me, it is now a part of my nature. I always try to be curious to know about things around me - much like a child. This world is not perfect and it will never be. There will always be room for improvement. I always make a conscious effort to think about what's missing and how can I improve anything.

Also, keep sharpening your axe by learning new skills whether it is technology, art, science, management, marketing or anything else. The more you know, more you will be able to find and notice brilliant ideas that will pass through you.

</comment><comment><USER24> I wrote about this in my blog:

* How to find ideas that make money: [http://paraschopra.com/blog/entrepreneurship/how-to-find-sta...](http://paraschopra.com/blog/entrepreneurship/how-to-find-startup-ideas-that-make-money.htm)

* How to validate your startup idea: [http://paraschopra.com/blog/entrepreneurship/validate-startu...](http://paraschopra.com/blog/entrepreneurship/validate-startup-idea.htm)

I am a big believer of market-based approach of doing startups. So, instead of trying to come up with a startup idea out of the blue, why don't you identify a market or an opportunity and then see what problems can be solved.

</comment><comment><USER23> Trying to force yourself to think of a solution to a problem your having almost always ends up disastrous, from my personal experience. The ideas might be good but I've never personally found a brilliant solution taking these methods. When an idea is sporadic and completely natural it encompasses the idea of simplicity and originality and these two ideas are an excellent platform to build products or solutions off of. I have had so many forced ideas that I regret moving forward with a few months down the road because the original idea wasn't a strong enough base to start from.

</comment><comment><USER25> 1) The starting point is always the observation of anomalies in an existing system or pattern. 2) You reason on those anomalies in your head, alone. 3) You come up with a possible response, alone. 4) You visualize in your head how this response would actually works. This should be practiced until you see it working smoothly from beginning to end. You do this alone. 5) Finally you set to work in order to bring this response into our physical world. Either alone or in a group-work

</comment><comment><USER26> Awareness, which may be one step above "Schlep Blindness:" <http://paulgraham.com/schlep.html> . Cultivate it.

In addition, read some and try to become aware of what others need and what others are working on.

Finally, when you become aware of problems or ideas, write them down. I'm a fan of small notebooks; other people use phones or computers or whatever. Look over whatever notes you've taken once a month, and once a year; look for patterns in those.

</comment><comment><USER27> I'm an artist. Most of my methods have clear analogs in what's already been mentioned. But here's something I've done that nobody else has mentioned:

I constructed an imaginary goddess and offered my skills to her.

A couple weeks later, I found myself drawing a Tarot deck completely out of the blue, spent a year on it, and was able to find a publisher for it who's distributed it internationally. I basically consider it my masterpiece (in the old sense of "this is the piece of work during which my skill went from 'journeywoman' to 'master'", not the modern sense of "a totally awesome pinnacle of the art that will survive for the ages").

Another artist friend of mine did a comic that was in part an offering to Ganesh - a statue of that diety figures prominently in the plot. It helped her work get noticed, an agent just sort of appeared in her life and now she's amidst a multi-book publishing deal with a major publishing house.

So, yeah, I know this is kind of wild for the technical atheistic crowd here. But... pray. Maybe to a well-known diety. Maybe one you made up yourself. It doesn't matter who as long as you can whole-heartedly believe in him/her/it/them for the duration of the ritual where you offer yourself and your skills up to Do Whatever. (I would however advise putting some care into your choice as this may result in a long-term working relationship.)

Then pretty much forget about it, just go about your business. You'll find a major project manifesting itself shortly. Consider it carefully as you do the initial doodles; ask yourself "do I want to spend the next year on this?" And if the answer is yes, then DO IT. You will no doubt gain a level in your primary skill and pick up some secondary ones as well.

(I don't know what happens if you turn down the first idea - I didn't.)

NOTE: This is NOT, IMHO, a route for the total novice. You HAVE to have pro- quality in at least one arena, because you need to be able to just crank on the core work of whatever the project is. Success is not guaranteed but it will be _really interesting_.

\---

Less esoterically, I will just repeat what others are saying: keep notebooks where you record every halfway-interesting idea you have, broaden your input to include fiction (how much of the modern age is inspired by sci-fi?) and technical work outside your field (like an artist reading HN!), help out on brainstorming other people's projects.

</comment><comment><USER28> 1) Work with a large company for a few weeks. You'll see hundreds of areas where there are inefficiency and problems to be solved.

2) ...

3) Profit.

<comment><USER29> Step 2, unfortunately, is more likely to be "axe these useless dead-weight middle managers who waste resources on turf wars and lie out of their teeth to screw over everyone else" than some neat technology.

</comment></comment><comment><USER30> Learn about a domain. Talk to lots of people in the domain. Ask about pain points, then imagine solutions. Do you have a friend that is an expert in any industry? Often domain experts will have ideas about things that would help their industry.

</comment><comment><USER31> I have done the same things that many people have noted in this thread. Dont try to get an idea, just get in a habit of looking at things and analyze if there is a better way to do it and if there are people willing to use your solution to the idea.

I was in tokyo last month on their bullet trains and suddenly this thing came to my mind that why dont I have a bullet train in States. My iphone is my note book, I keep writing stuff in it whenever I have an idea and in the end it goes to my blog from where I choose the best one I think and start developing it. Some of the ideas are here <http://nsingh.posterous.com/steal-these-ideas> and if you look they are mostly random things that we come across but still ignore them.

But personally I still think that you have to go out of your comfort zone to get different ideas to work on.

</comment><comment><USER32> <http://www.ideaoverload.com/> \- Everything About Ideas In One Place To Make It Easier For Everyone To Find New Ideas (including finished business ideas)

</comment><comment><USER33> Whenever I find myself doing something annoying, or tedious, or slow I ask the question: "Why am I doing this?". The solution to the problem usually stems from the answer to that question.

</comment><comment><USER34> _What is your thought sequence, particuarlly in a new venture that you're thinking about or have already started._

OK, in a venture that I've already started, I go back to our mission statement[1], and decide if the new idea fits the mission. If it does, I think about whether or not it would be a feature of a project we're already working on, or would become a whole new product. In either case, it has to be prioritized relative to the existing work. In some cases, it's easy enough to say "this isn't important enough to work on right now, so it goes in the backlog." In other case, it's not so clear, so you might want to get out and talk to people. I like to bounce ideas off a select handful of people I know and trust initially, and then - depending on the feedback - I might start talking it up to a wider audience, and/or blog/tweet/g+/etc. the idea to solicit discussion.

 _Do cool things just randomly pop in your head after whatever things you do through life?_

Sometimes, but I find that most of the cool ideas I have (well, the ones that I think are cool anyway) occur in response to reading or studying something new, or something that shifts my perspective. I'm a rabid reader anyway, and I try to stretch my boundaries by reading books that span a lot of territory. Even in terms of technical books... I mean, I'm a software guy by trade, but one of the last books I read was _Beyond Boundaries_ [2] by Miguel Nicolelis, which deals with neuroscience and brain/machine interfaces. And lying around here somewhere are some books on nanotechnology, philosophy of mind, artificial life, economics, etc., etc. I often find inspiration from reading something seemingly fairly unrelated to my day to day work.

I also find that talking to interesting people, doing interesting stuff, with other cool ideas, can spark a new idea. As such, I hang out at the local hackerspace[3] quite often, and just listen in, talk to people, swap ideas, etc. Going to user group meetings around new technologies and talking to people there can also spark ideas.

Another neat thing to do, is to follow the "incubator"[4] discussion list at the Apache Software Foundation. Just following what new and interesting projects are being submitted there can potentially spark some cool and creative ideas.

 _As a follow-up, how do you actually validate whether or not you will actually move forward with it (market opportunity, personal problem, growth potential, ease/challenge of problem, customer validation)?_

See above, but if something makes it as far as being seriously considered for a product / product feature, I'm a big fan of @sgblank's "Customer Development" approach. See his book _The Four Steps to the Epiphany_ [5] for details.

[1]: Previous HN discussion on missions: <http://news.ycombinator.com/item?id=3631611>

[2]: Author Website: <http://www.beyondboundariesnicolelis.net/~beyond/wordpress/>

[3]: <http://www.splatspace.org>

[4]: <http://incubator.apache.org>

[5]: [http://www.amazon.com/Four-Steps-Epiphany-Successful- Strateg...](http://www.amazon.com/Four-Steps-Epiphany-Successful- Strategies/dp/0976470705)

</comment><comment><USER35> good rough process:

1\. what is something that sucks?

2\. how would you improve it?

3\. bonus: how can you monetize that?</comment>


Salesforce releases language model bigger than GPT-2 large - <AUTHOR> https://github.com/salesforce/ctrl <comment><USER2> I am working on a guide (should be released tomorrow) to easily get it up and running for personal use. Here's my Twitter thread of current experiments with the model: [https://twitter.com/minimaxir/status/1173081315177975810](https://twitter.com/minimaxir/status/1173081315177975810)

I recommend reading the linked paper in the repo as it gives decent examples/instructions on how to use the model. Although the size and architecture is comparable to GPT-2, the emphasis on conditional generation differentiates it.

<comment><USER3> > running for personal use

how one can use it for personal use? In my understanding it will not fit into single GPU memory available to average person? Someone need to distill model first?

<comment><USER2> It currently fits into a P100, but _barely_.

</comment></comment></comment><comment><USER4> Wow, that's some license addendum:

> This software should not be used to promote or profit from:

> violence, hate, and division,

> environmental destruction,

> abuse of human rights, or

> the destruction of people's physical and mental health.

<comment><USER5> So Salesforce could never use it?

<comment><USER6> Not sure which of those you think they would be violating? And they own it, so the license doesn't apply to them, it applies to everyone else. (Apologies if you're making a joke and I'm ruining it)

<comment><USER5> You can very easily argue that Salesforce, by dealing with companies who do it, does all of the things that are forbidden by the license.

</comment></comment></comment></comment><comment><USER7> Anyone have a real-world use case for something like this? I must admit I'm having trouble thinking of any that aren't essentially deceptive. Because in my little biased world, I have no need of "text" per se, and what value any text has to me is closely linked to the fact that it came from a human.

<comment><USER8> Machine learning researchers aren't working on language modeling because they want to enable fake news.

They are working on it because it improves all downstream NLP tasks. See: [http://ruder.io/nlp-imagenet/](http://ruder.io/nlp-imagenet/). BERT, Elmo and XLNet all fall under this use case.

For example if you're trying to recognize speech or translate some text, it helps a lot if you can start off producing something that is statistically grammatical even if the content is nonsense.

</comment></comment><comment><USER9> From the blog post: "Beyond the technical work to develop this model, we've also taken several steps to anticipate and mitigate malicious use cases where possible."

From the preprint, this seems to be doing some review before release and having a code of conduct in the GitHub repo.

</comment><comment><USER10> The unicorn prompt is the new text generator lorem ipsum

</comment><comment><USER11> It was trained on 140GB of text on 256 TPUs for 2 weeks, the model being made of 48 transformer layers. I'm wondering when we will see a model trained on 1TB or 10TB of text.

<comment><USER12> I doubt training a scaled up transformer on 10TB of text will lead to significant improvements (btw, 10TB is about the size of all books in English in the Library of Congress). Image classifiers don't get _a lot_ better when trained on a lot more data than ImageNet. 140GB is probably enough to train a general model, which could be finetuned on extra data for specific tasks.

Text generators need a world model and situational awareness, something like a map and a GPS signal. So we are probably two major breakthroughs away from a machine that actually _understands_ something (or at least which seems to understand something, if you're philosophically opposed to the idea that a machine can understand something).

</comment></comment><comment><USER13> Could someone provide a high level summary of what this is for a technical person not conversant with the field?

<comment><USER14> Salesforce has created a computer program where you put in a small prompt, like "Wikipedia page about badgers" or "News article starting with the line, 'Donald Trump was impeached today'", or "French translation of 'I like pears'", and it tries to predict what the text will be. You can also run the program in reverse, where you put in a snippet of text and it predicts whether it came from Wikipedia or a mystery novel or the fitness subreddit.

Salesforce created the program by first writing some relatively simple linear algebra, then fiddling with the constants until the output happened to look right. Their program contains 1.6 billion constants, which is more than any other program of its kind.

This program is also special because Salesforce has released it publicly; other organizations, like OpenAI, have previously claimed that text-generation software is too dangerous to release to the general public.

<comment><USER15> > writing some relatively simple linear algebra

Except, that it wouldn't work if it was purely linear.

<comment><USER14> Right, yeah, it's linear algebra combined with a few non-linear functions. The point is that Salesforce didn't come up with an algorithm that generated English text by writing a grammar or thinking really hard about what sentences look like—all the functionality comes from the "training" process that set the constants.

</comment></comment></comment></comment><comment><USER16> > Advertisement

Yeap, This one is indistinguishable from reality

</comment><comment><USER17> Are there any hardware reqs to work with this?

<comment><USER18> In theory, no. But for any decent performance, you need a big CUDA capable GPU, as far as I know.

But you can try it on a CPU of course. (Maybe with some modifications; see this: [https://news.ycombinator.com/item?id=20977776](https://news.ycombinator.com/item?id=20977776) ; also if someone can get it working in Google Colab you get a GPU capable instance for free.)

</comment></comment><comment><USER19> Open AI did the right thing by not releasing their model; it's disappointing that researchers are so callous about the potential effects of their research in the name of progress.

<comment><USER14> I've never really gotten why AI types are so concerned about text-generation models.

Like, sure, I can kind of see why you wouldn't want to make the Deepfakes program public; it currently takes a lot of time, effort, and expertise to swap faces realistically in a video, and maybe we don't want to give every average Joe the ability to do that.

But pretty much everyone in the world can already pretty trivially write text. (I'm doing it right now!) And the "typical" generation output from these programs usually isn't very good—OpenAI had to try like thirty times for each of the prompts in their PR materials—so it usually ends up being less work to just write the fake news yourself instead of using the software.

My personal conspiracy theory is that all this talk of "the model is too dangerous to release" really boils down to "if we let people test out the model, they'll find it doesn't work as well as our PR team wants them to think it does".

<comment><USER11> I dunno, this time the text looks really good. I got as far as 5 or 6 phrases deep before it said anything silly. I would have been fooled if I red it in real life.

My guess is that they will perfect the transformer and its training process, curate the dataset and make this method really easy to use. Maybe it can do translation, math, even auto-complete code. That is only by iterating more on the current formulation of the Transformer.

But it is also possible that it is surpassed by something even better. This new language model could replace the inductive bias specific to the Transformer - the ability to "attend" to any part of the input text, with something more efficient, because Transformers are quite hard and expensive to train right now. Maybe the Transformer inductive bias is too general (like a fully connected network) and needs too much data, with a slightly different idea it could be made much more efficient and probably more convincing.</comment></comment></comment>


You are leaking your /etc/hosts DNS queries - <AUTHOR> http://qosys.info/etc-hosts/ <comment><USER2> Not quite sure what the point of this is.

Summary:

* Addresses not in your /etc/hosts are queried against your DNS server. * Firefox will do both IPv4 and IPv6 lookups (so will probably trigger an IPv6 lookup even if there's an IPv4 entry). * DNS queries are in the clear so can be snooped.

Essentially, this is how DNS is known to work?</comment>


How Homeownership Became the Engine of American Inequality - <AUTHOR> https://www.nytimes.com/2017/05/09/magazine/how-homeownership-became-the-engine-of-american-inequality.html <comment><USER2> I think it makes sense to abolish programs like the MID or first home buyer assistance that largely just increase home prices.

But it would certainly suck to take out a loan on a home now and be told next year that not only is the MID gone, but your home is less valuable now and you effectively just surrendered a chunk of your deposit.

Maybe phasing it out over by 10% a year over a decade would be realistic, but would still run into the realtor lobby.

</comment><comment><USER3> If you lump it in with the standard deduction its not so regressive.</comment>


How Optimizely Almost Got Me Fired (2014) - <AUTHOR> http://blog.sumall.com/journal/optimizely-got-me-fired.html <comment><USER3> A/B testing is high on hype and promise but low on actual results if you follow it though to actual metrics. I've done various forms of A/B throughout most of my career and found them to be cinsistent with the OP's results.

A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form.

That, or actually build a product that users want instead of chasing after pointless metrics. I mean, really, you think changing the color of text or a call-out is going to make up for huge deficiencies in your product or make people buy it? The entire premise seems illogical and just doesn't work. The only time I've seen a/b tests truly help was when it accidentally fixed some cross browser issue or moved a button within reach of a user.

Most of the A/B website optimization industry is an elaborate scam, put off on people who don't know any better and are looking for a magic bullet.

<comment><USER4> > A much better approach is to install significant instrumentation and > actually talk to users about what's wrong with your sign up form.

But but but telemetry is evil!

<comment> ross-life Telemetry in a specific product? Fine.

Telemetry in my OS* that has access to _everything_? No.

* That I cannot turn off.

<comment><USER5> What if the OS is the product?

<comment> ross-life An OS is a product of course, but to me it's on a completely different level to "just" an application product. It has access to everything, I want to be able to trust it and know exactly what it's doing. Telemetry I can't turn off ruins that. Having it is fine, just let me turn it off...

I like Windows and really like VS, I was often literally the only (non-VM) Windows user in a sea of OS X at several offices but Windows 10, the flip- flopping UI, the ads, and the telemetry pushed me to Linux and building my own desktops.

</comment></comment></comment></comment></comment><comment><USER6> [I'm a PM @ Optimizely]

We were asked about this article before, on our community forums, and one of our statisticians, David, wrote a detailed reply to this article's concerns about one- vs two-tailed testing, which might be of interest [3rd from the top]:

[https://community.optimizely.com/t5/Strategy-Culture/Let- s-t...](https://community.optimizely.com/t5/Strategy-Culture/Let-s-talk-about- Single-Tailed-vs-Double-Tailed/m-p/4220)

Additionally, since then, as other commenters have mentioned, we've completely overhauled how we do our A/B-testing calculations, which, theoretically and empirically, now have an accurate false-positive rate even when monitored continuously. Details:

[https://blog.optimizely.com/2015/01/20/statistics-for-the- in...](https://blog.optimizely.com/2015/01/20/statistics-for-the-internet-age- the-story-behind-optimizelys-new-stats-engine/)

</comment><comment><USER7> Disclaimer: I'm the director of data science at VWO, an Optimizely competitor.

In my view, the issue is not one-tail vs two-tail tests, or sequential vs one- look tests at all. The issue is a failure to quantify uncertainty.

Optimizely (last time I looked), our old reports, and most other tools, all give you improvement as a single number. Unfortunately that's BS. It's simply a lie to say "Variation is 18% better than Control" unless you had facebook levels oftraffic. An honest statement will quantify the uncertainty: "Variation is between -4.5% and +36.4% better than Control".

When phrased this way, it's hardly surprising that deploying this variation failed to achieve an 18% lift - 18% is just one possible value in a wide range of possible values.

The big problem with this is that customers (particularly agencies who are selling A/B test results to clients) hate it. If we were VC funded, we might even have someone pushing us to tell customers the lie they want rather than the truth they need.

Note that to provide uncertainty bounds like this, one needs to use a Bayesian method (only us, AB Tasty and Qubit do this, unless I forgot about someone).

(Frequentist methods can provide confidence intervals, but these are NOT the same thing. Unfortunately p-values and confidence intervals are completely unsuitable for reporting to non-statisticians; they are completely misinterpreted by almost 100% of laypeople. [http://myweb.brooklyn.liu.edu/cortiz/PDF%20Files/Misinterpre...](http://myweb.brooklyn.liu.edu/cortiz/PDF%20Files/Misinterpretations%20of%20Significance.pdf) [http://www.ejwagenmakers.com/inpress/HoekstraEtAlPBR.pdf](http://www.ejwagenmakers.com/inpress/HoekstraEtAlPBR.pdf) )

<comment><USER8> Could you point to the math/protocol that goes from a sample set, to a range "-4 to +34" and of course how one goes from a range to giving a single number?

I feel like the discussion on this thread is missing the underlying "source code"

<comment><USER7> Here are slides on the topic and the full math paper. But we dont go from a credible interval to a single number - we just report credible intervals. We find that to be the only honest choice.

[https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2...](https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2015/slides.html)

[https://www.chrisstucchio.com/blog/2013/bayesian_analysis_co...](https://www.chrisstucchio.com/blog/2013/bayesian_analysis_conversion_rates.html)

[https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic...](https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf)

<comment><USER8> thank you - bedside reading :-)

</comment></comment></comment></comment><comment><USER9> One thing I noticed that I haven't seen commented on yet:

The solution mentioned of running a two tailed test would not have solved the problem of a false result the author demonstrated through conducting an A/A test.

According to the image in the article: [http://blog.sumall.com/wp- content/uploads/2014/06/optimizely...](http://blog.sumall.com/wp- content/uploads/2014/06/optimizely-test.png)

The A/A test had: A1: Population: 3920 Conversion: 721 A2: Population: 3999 Conversion: 623

        Z-Score: 3.3
        2-tailed test signifiance: 99.92%

Looks like the one-tail vs. two tail test doesn't make huge difference in this case.

So, maybe a larger sample size would have seen a reversion to the mean, but given the size and high significance that would be unlikely (interesting exercise to try different assumptions to calculate how unlikely, with the most overly generous obviously just being the stated significance).

Yes, the test was only conducted over one day, but if it was the exact same thing being served for both, that shouldn't matter.

If there was a reversion to the mean due to an early spike, we would expect to see the % difference between the two cells narrow as the test kept running. You can see in the chart that the % difference (relative gap between the lines) stays about the same after 8pm on the 9th.

So if it's not the one-tailed test at fault, and it's not the short duration of the test at fault, what is?

Don't know.

I have seen in the past that setup problems are incredibly easy to make w/ a/b testing tools when implementing the tool on your site. I've seen in other tools things like automated traffic from Akamai only going to the default control, or subsets of traffic such as returning visitors excluded from some cells but not others.

Based on those results, I'd be suspicious of something in the tool setup being amiss.

</comment><comment><USER10> > This usually happens because someone runs a one-tailed test that ends up > being overpowered.

It always pains me a little when people doing research describe statistical power as a type of curse. Overpowered? Should we reduce it? The risk isn't having too much power, the risk is that someone will incorrectly interpret their Null Hypothesis Significance Test (NHST). They need to shift their focus to measuring something (and quantifying the uncertainty of their measurements), rather than think of "how likely was this result given a null hypothesis", whether that hypothesis is..

something is not greater than 0 (one-tail), or

something is not 0 (two-tail).

> You'll often see statistical power conveyed as P90 or 90%. In other words, > if there's a 90% chance A is better than B, there's a 10% chance B is better > than A and you'll actually get worse results.

This isn't necessarily true. A could be the same as B. Also, these tests are being done from the frequentist perspective, so saying "there's X chance B is better than A" is inappropriate, unless you're talking about the conclusions of your significance test (e.g. 90% chance you correctly detect a difference between them--a difference you assume is fixed to some true underlying value). Overall, being aware that a one-tail test is taking the position that nothing can happen in the other direction is useful, but a good next step is understanding what NHST can and cannot say.

This even a frequentist vs bayesian problem, since you could create situations where a person felt a study was overpowered in either framework.

</comment><comment><USER11> Don't just run tests longer - run tests for a pre-defined amount of time instead of "until you see a result you like".

<comment><USER9> Your comment hits the nail on the head here.

Standard statistical tests used in a/b testing are based on one check. If someone is checking repeatedly on a test until they get a 'significant' result, your chance of getting a getting a false positive is many X the stated significance.

Best practice - set a pre-defined end, and one or two defined early check-in points where only make an early call if result is overwhelmingly significant or if the business has fallen off a cliff.

<comment><USER12> That would be best practice if you insist on using null hypothesis significance testing and only wanted to use classical frequentist statistics. We really can do much better these days with multi-armed bandits, and by focusing on effect sizes and credible intervals rather than a yes/no answer to a hypothesis.

<comment><USER11> Sounds like I need to update my stats knowledge! Do you happen to know of a good place to start learning about today's state of the art?

</comment></comment></comment></comment><comment><USER13> I agree with the main point of the article, but I'm somewhat disturbed by the statistical errors and misconceptions.

>Few websites actually get enough traffic for their >audiences to even out into a nice pretty bell curve. If >you get less than a million visitors a month your >audience won't be identically distributed and, even >then, it can be unlikely.

What is the author trying to say here? Has he thought hard about what it means for "an audience" to be identically distributed?

>Likewise, the things that matter to you on your website, >like order values, are not normally distributed Why do they need to be?

>Statistical power is simply the likelihood that the >difference you've detected during your experiment >actually reflects a difference in the real world. Simply googling the term would reveal this is incorrect.

</comment><comment><USER14> > In most organizations, if someone wants to make a change to the website, > they'll want data to support that change.

So true and sad. In all the so called data-driven groups I have worked for, the tyranny of data makes metrics and numbers the justification for or counter to anything, however they have been put together.

> The sad truth is that most people aren't being rigorous about their A/B > testing and, in fact, one could argue that they're not A/B testing at all, > they're just confirming their own hypotheses.

The sad truth is that most people aren't being rigorous about anything.

</comment><comment><USER15> Here's a great article about how Optimizely gets it wrong: [http://dataorigami.net/blogs/napkin-folding/17543303-the- bin...](http://dataorigami.net/blogs/napkin-folding/17543303-the-binary- problem-and-the-continuous-problem-in-a-b-testing)

There are _many_ offenders. I've yet to see a commercial tool that gets it right.

Tragically, the revamp by Optimizely neglects the straightforward Bayesian solution and uses a more fragile and complex sequential technique.

<comment><USER16> The article you mention names RichRelevance, but there are others who implement Thompson Sampling or other forms of Bayesian Bandits, such as SigOpt [0] and Dynamic Yield [1]. Various adtech companies also use it underneath the hood.

[0] [https://sigopt.com/](https://sigopt.com/)

[1] [https://www.dynamicyield.com/](https://www.dynamicyield.com/)

<comment><USER7> Thompson Sampling is not a replacement for A/B tests. Unfortunately, the real world violates the assumptions of Thompson sampling virtually all the time.

[https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm...](https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html)

Bandit algorithms do have some important use cases (optimizing yield from a short lived advertisement, e.g. "Valentines Day Sale"), but they are not suitable for use as an A/B test replacement.

Also, I'd steer away from dynamic yield - I've found their descriptions of their statistics to take dangerous (i.e. totally wrong) shortcuts. For example, counting sessions instead of visitors as a way to avoid the delayed reaction problem and increase sample size (as well as completely breaking the IID assumption).

<comment><USER16> I love your posts, and completely agree that Bayesian Bandits are not replacements for A/B tests.

To be fair though, the realworld issues like nonstationarity and delayed feedback are also concerns for A/B tests (which you also bring up in your great post), and you can tweak the bayesian bandits to handle these cases decently.

How does counting sessions instead of visitors avoid delayed feedback? I read your post [0] but dont remember anything about that. Is it just that they say that after a session is completed (which is somewhat nebulous to measure in many cases), then you have all the data you need from the visit?

[0] [https://www.chrisstucchio.com/blog/2015/no_free_samples.html](https://www.chrisstucchio.com/blog/2015/no_free_samples.html)

<comment><USER7> Absolutely agree you can tweak Thompson sampling to handle nonstationarity, periodicity and delayed feedback. I even published the math for one variation of nonstationarity: [https://www.chrisstucchio.com/blog/2013/time_varying_convers...](https://www.chrisstucchio.com/blog/2013/time_varying_conversion_rates.html)

(I've also dealt with delayed reactions, but I've never published it, and probably won't publish until I launch it.)

Dynamic yield has the delayed feedback problem because users might see a variation in session 1 but convert in session 2 (days later). They "solve" this by doing session level tracking instead of visitor level tracking - <USER2> delayed feedback is now only 20 minutes (same session) instead of days.

The problem is that session A and session B are now correlated since they are the same visitor. IID is now broken.

<comment><USER16> Makes sense, thanks for the explanation!

</comment></comment></comment></comment></comment></comment><comment><USER17> Optimizely rolled out a huge update to the way it handles statistics called Stats Engine last year. That update resolves the issues discussed in this article. You can read more about Stats Engine here: [https://www.optimizely.com/statistics/](https://www.optimizely.com/statistics/)

<comment><USER12> Their Stats Engine does resolve the issues, but I have to laugh at their marketing materials calling it 21st century statistics, because even the new approach is comically behind the times. Looks like the poor schmucks accidentally outsourced the project to some really old-school one trick pony "let's throw some maximum likelihood theory at this" statisticians. I would've hoped even die-hard frequentists would see the value of a multi-armed bandit and the irrelevance of declaring a winner rather than quantifying the effect of each variant and the uncertainty around it.

Cf. the technical paper: [http://pages.optimizely.com/rs/optimizely/images/stats_engin...](http://pages.optimizely.com/rs/optimizely/images/stats_engine_technical_paper.pdf)

</comment></comment><comment><USER18> > some testing vendors use one-tailed tests (Visual Website Optimizer, > Optimizely)

> Most A/B testing tools recommend terminating tests as soon as they show > significance, even though that significance may very well be due to short- > term bias. A little green indicator will pop up, as it does in Optimizely, > and the marketer will turn the test off.

People pay brisk money for this?

</comment><comment><USER12> After reading the blog post and reading through the comments, it looks like people are drawing the wrong conclusion from this. The problem is not that AB- testing is overrated, doesn't work, is bullshit etc. but that Optimizely used to do it wrong.

<comment><USER19> Exactly. A/B testing is very useful in some situations (i.e. where traffic to the page is great enough, product-market fit is achieved, and the product is well-developed enough that any changes to it after the campaign is run will be minimal). But many companies, usually startups, scale too early and use the products poorly. Companies that sell these products have very little incentive to discourage use of their products by ill-informed users, so the misuse continues.

</comment></comment><comment><USER20> > Statistical power is simply the likelihood that the difference you've > detected during your experiment actually reflects a difference in the real > world.

This seems incorrect to me. Isn't statistical power the likelihood that the null hypothesis would generate an outcome at least as extreme as what you observed?

I'm guessing the issue has a lot more to do with peeking at the outcome and not correcting for it (and similarly running many tests)

[http://www.stat.columbia.edu/~gelman/research/unpublished/p_...](http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf)

<comment><USER21> Yes, this is incorrect.

Also the next sentence is wrong:

>You'll often see statistical power conveyed as P90 or 90%. In other words, if there's a 90% chance A is better than B, there's a 10% chance B is better than A and you'll actually get worse results.

Having a test w/ 90% power means that if A is truly better than B (for one sided) or A is truly different than B (two sided), then you'll detect it 90% of the time you run the test (on independent data).

</comment></comment><comment><USER22> In my experience, if you are seeing huge effects like 60% difference in conversion, etc, you probably did something wrong (i.e. too small sample, didn't wait long enough, etc) - I've never seen something this large by simply moving things around, changing colors, changing messages, etc.

Also, in general, the more drastic the changes are, the more of an effect you __could __have (up to some percentage). I.e. a small change would be changing a message or color, dont expect conversion to change by much. A large change would be changing from a flash site to an html one with a full redesign that loads twice as fast...

<comment><USER23> One of the better explanations of a/b testing I've read is that there are three types of uses on your site: those who will covert no matter what, those who will not convert no matter what, and those who might convert. Testing is for that final group. Thinking in those terms, you have to consider if such a large percentage fits into that "might" group. I've never personally witnessed a 60% improvement at any volume I'd bother testing on and I've been doing tests for many years.

I also think a lot of the "change the button" tests or "increased email sign up by x times" are dubious. Unless the conversion is someone giving you money, then you still have steps to go before you see real business improvement. There are lots of ways I can minipulate my traffic to make certain funnel steps look better and more optimized, but the only thing I really care about is what's coming out of the funnel. So all those extra email signups or button pushes mean nothing if the group who perform those actions on your b version still aren't interested in an actual purchase.

</comment></comment><comment><USER24> I'm increasingly convinced all statistical analysis performed by non-PhDs is no better than a coin flip. Maybe even worse.

My favorite example is still the quite popular Page Weight Matter posts. I wonder how close they were to abandoning a 90% reduction in size. I wonder how many improvements the world at large has thrown away due to faulty analysis.

[http://blog.chriszacharias.com/page-weight- matters](http://blog.chriszacharias.com/page-weight-matters)

</comment><comment><USER25> This is a huge problem with paid marketing as well. Many folks will look at a conversion rate completely ignorant of sample size and allocate thousands of dollars in budget to something which they have no idea performs better or worse.

The real problem (as you allude to in the article) is that the demand for accurate tools is not really there. Vendors don't build in accurate stats because only a tiny portion of their client base understand/demands them.

</comment><comment><USER26> There is much more to running experiments properly than it seems. While I'm not an expert on the statistics side, there are a few things I've learned over the years which come to mind...

1) Run the experiment in whole business cycles (for us, 1 week = one cycle), based on a sample size you've calculated upfront (I use [http://www.evanmiller.org/ab-testing/sample- size.html](http://www.evanmiller.org/ab-testing/sample-size.html)). Accept that some changes are just not testable in any sensible amount of time (I wonder what the effect of changing a font will have on e-commerce conversion rate).

2) Use more than one set of metrics for analysis to discover unexpected effects. We use the Optimizely results screen for general steer, but do final analysis in either Google Analytics or our own databases. Sometimes tests can positively affect the primary metric but negatively affect another.

3) Get qualitative feedback either before or during the test. We use a combination of user testing (remote or moderated) and session recording (we use Hotjar, and send tags so we can view sessions in that experiment).

</comment><comment><USER27> Interesting read, might be worth adding (2014) to the title though.

<comment><USER28> Why the downvote? The year is relevant as Optimizely has made various adjustments in the past 1-2 years to the way they handle statistics, one other user even purporting it addresses the very issues this article mentions.

<comment><USER29> One might wonder if a company which doesn't understand statistics, despite statistics being their sole reason for existence, is actually _capable_ of fixing these problems. If they don't understand what they don't understand, how can they address it effectively?

</comment></comment></comment><comment><USER30> Almost didn't click on this because the title seems (and is) clickbait-y, but it was actually a very useful read for me.

As a founder, I'm constantly hearing about A/B testing and how great these tools are. I'm not enough of a statistician to know whether everything in this article is true/valid (and would welcome a rebuttal), but the part about regression to the mean really hits home. Encouraging users to cut off testing too early means that you make them feel good ("Look, we had this huge difference!"), when in reality the difference is smaller/negligible.

I'll still do some A/B testing, but given our engineering/time constraints—and my inability to accurately vet the claims/conclusions of the testing software—I won't spend too much time on this.

<comment><USER31> Deciding not to AB test because of this article would be a huge mistake.

Learning from mistakes made by others, and avoiding them is what I would suggest as the take away.

I work for a successful ($50M+ revenue) bootstrapped startup. And one of the reasons for the success is that AB testing became part of the company's culture, as soon as there was enough data coming in for the tests to become useful.

AB testing is so important that we have built our own in-house framework that automatically gives results for our company specific KPIs.

</comment></comment><comment><USER32> AB testing is too simplistic. Even on my marketing team we have designed more complex metrics to look at a factors impact on multiple outcomes. The testing is still a straight forward chi square, but with a bit more depth.

</comment><comment><USER33> Wow, did not see that coming. This article actually confirms the cynical hypothesis I entertain - that most of the "data-driven" marketing and analytics is basically marketers bullshitting each other, their bosses, their customers and themselves, because nobody knows much statistics and everyone wants to believe that if they're spending money and doing something, it must be bringing results.

Some quotes from the article supporting the cynical worldview:

\--

"Most A/B testing tools recommend terminating tests as soon as they show significance, even though that significance may very well be due to short-term bias. A little green indicator will pop up, as it does in Optimizely, and the marketer will turn the test off. But most tests should run longer and in many cases it's likely that the results would be less impressive if they did. Again, this is a great example of the default settings in these platforms being used to increase excitement and keep the users coming back for more."

This basically stops short of implying that Optimizely is doing this totally on purpose.

\--

"In most organizations, if someone wants to make a change to the website, they'll want data to support that change. Instead of going into their experiments being open to the unexpected, open to being wrong, open to being surprised, they're actively rooting for one of the variations. Illusory results don't matter as long as they have fodder for the next meeting with their boss. And since most organizations aren't tracking the results of their winning A/B tests against the bottom line, no one notices."

In other words, everybody is bullshitting everybody, but it doesn't matter as long as everyone plays along and money keeps flowing.

\--

"Over the years, I've spoken to a lot of marketers about A/B testing and conversion optimization, and, if one thing has become clear, it's how unconcerned with statistics most marketers are. Remarkably few marketers understand statistics, sample size, or what it takes to run a valid A/B test."

"Companies that provide conversion testing know this. Many of those vendors are more than happy to provide an interface with a simple mechanic that tells the user if a test has been won or lost, and some numeric value indicating by how much. These aren't unbiased experiments; they're a way of providing a fast report with great looking results that are ideal for a PowerPoint presentation. _Most conversion testing is a marketing toy, essentially_." (emphasis mine)

 _Thank you_ for admitting it publicly.

\--

Like whales, whose cancers grow so big that the tumors catch their own cancers and die[0], it seems that marketing industry, a well known paragon of honesty and teacher of truth, is actually being held down by its own utility makers applying their honourable strategies within their own industry.

I know it's not a very appropriate thing to do, but I _really_ want to laugh out loud at this. Karma is a bitch. :).

[0] - [http://www.nature.com/news/2007/070730/full/news070730-3.htm...](http://www.nature.com/news/2007/070730/full/news070730-3.html)

<comment><USER34> >most of the "data-driven" marketing and analytics is basically marketers bullshitting each other

Most of the ones you hear about. You know, the ones who were SEOs or content writers or programmers before waking up one day and deciding to be marketers.

Marketing degrees have mandatory statistical courses. The good marketing programs take them very seriously. However a lot of schools focus on the communication side of marketing, which leads a very significant chunk of the marketing analyst positions to be filled by people with economics and accounting degrees.

The Internet is actually quite new and has just begun maturing. A lot of people working in digital marketing do not really understand what marketing is. When you meet somebody on the street and ask him what marketing is, he will describe advertising, or more precisely mar. communications. So when the internet became this giant medium for doing all sorts of commerce, big companies and schools couldn't fill the skill gap fast enough, so the gap was filled by self-learners coming from all sorts of backgrounds. When they wanted to build up their "marketing skills", naturally they defaulted to learning about marketing communication instead of the economics-orientated part of marketing. This is why digital marketers obsess with their site and drool over a/b tests and such. The web site is a communication medium. They've put themselves in this box equating marketing and communications.

Too bad for them, because graduates nowadays are digital native too, so they have no problem navigating the internet and learning html/css.

</comment></comment><comment><USER35> (as others have mentioned) Optimizely's newer engine uses ideas like Wald's sequential analysis. Here is my article on the topic: [http://www.win- vector.com/blog/2015/12/walds-sequential-anal...](http://www.win- vector.com/blog/2015/12/walds-sequential-analysis-technique/) .

</comment><comment><USER36> \tangent When I first heard of A/B testing, I thought of combining it with genetic algorithms to evolve the entire site. Just run it til the money rolls in.

Unfortunately, if it did work, it would probably be through something misleading or scammy. Therefore, you need some kind of automatic legality checking... which would be hard.

</comment><comment><USER37> Is anybody out there taking an approach of gradually driving more traffic to whichever option is winning out and never running 100% with anything specific?</comment>


A case for a four day work week - <AUTHOR> http://www.inc.com/jay-love/the-case-for-a-four-day-work-week.html

<comment><USER2> Excellent article, and especially good content in the comments on the site.</comment>


The music moves us, but how? (2018) - <AUTHOR> https://www.knowablemagazine.org/article/mind/2018/music-moves-us-how <comment><USER2> "We have no idea, but here are some areas that light up in an fMRI machine when you listen!"

<comment><USER3> /thread

I feel like this kind of "research" is the bulk of neuroscience papers that I see these days. Or they're at least the ones that get shared the most because they claim to explain things that haven't been explained yet.

</comment></comment><comment><USER4> Levitin also points out in "This Is Your Brain on Music" that

 _Only relatively recently in our culture [...] did a distinction arise that cut society in two, forming separate classes of music performers and music listeners. Throughout most of the world and for most of human history, music making was as natural an activity as breathing and walking, and everyone participated.

[...]

A couple of generations ago, before television, many families would sit around and play music together for entertainment. Nowadays there is a great emphasis on technique and skill, and whether a musician is "good enough" to play for others. Music making has become a somewhat reserved activity in our culture, and the rest of us listen._

I didn't really realize this until reading this passage. I am lucky enough to have learned piano at an early age, and have picked up other instruments along the way. Even so, I was hesitant to sing around others until I took a leap and joined a choir in college, and got in enough practice to gain the confidence to sing more.

Music is an incredibly important part of my life, and it makes me a little sad to see how often people are too embarrassed of their own ability to participate.

</comment><comment><USER5> Here's Tony Royster, jr doing some crazy things going in front of or behind the beat [https://www.youtube.com/watch?v=Yh8PATuI_NU](https://www.youtube.com/watch?v=Yh8PATuI_NU)

And here is a musical performance that actually makes many people cry [https://www.youtube.com/watch?v=r8KrpwqxE4g](https://www.youtube.com/watch?v=r8KrpwqxE4g)

</comment><comment><USER6> What I find interesting is that in some cultures people dance together (e.g. as couples or as a group), but in Western culture people mostly prefer to dance alone. And often people even zone out, focusing only on the music and ignoring other people.

<comment><USER7> That's a _very_ recent phenomenon even in the West. Solo dancing as a widespread cultural phenomenon was basically born in the 50s, in conjunction with rock'n'roll.

<comment><USER8> Is that because the music is cool and varied enough that it's now fun to just dance alone, but it wouldn't have been fun with older music?

Or because, with 1000 kinds of music today, nobody can figure out how to dance together to a given song?

Or did people dance alone all the time in the old days, but nobody recorded it?

<comment><USER7> There are likely many reasons, among them the fact that the invention of recorded sound made listening to music something you could do alone in the first place, and the general trend of individualization in the Western society in the 1900s.

When it comes to partner dancing, there exist modern social dances that can be danced to a wide variety of pop music, mostly descendants of the original 30s/40s swing dances like Lindy Hop and Jitterbug, and even moreso than those predecessors, emphasize improvisation and nontraditional lead–follow dynamics. West Coast Swing is probably the most popular example while by no means mainstream; partner dances simply aren't that fashionable even if there has been a resurgence of some sort in the past decades. (Coincidentally, I happen to be currently in Budapest, at one of Europe's largest annual WCS events!)

Of course, there's the fundamental fact that solo dancing requires next to no skill, while acquiring even basic partner dancing skills requires one to take a class, or at the very least "learn by doing" in an appropriate environment.</comment></comment></comment></comment>


Show HN: BudgetSheet: Import Bank Transactions from Plaid into Your Google Sheet - <AUTHOR> https://www.budgetsheet.net/ <comment><USER2> Can this work with any bank account in any country?</comment>


Computer Generated Math Paper makes it into Peer Reviewed Journal - <AUTHOR> http://marginalrevolution.com/marginalrevolution/2012/10/nonsense-paper-accepted-by-mathematics-journal.html <comment><USER2> It's not accepted. The referees were very kind and gave the benefit of doubt to the authors and gave them the option of revising the paper.

I think the Journal should be lauded for being open minded but principled. I think the authors should fess up and offer the reviewers a beer for wasting their time.

PS. Original post <http://thatsmathematics.com/blog/archives/102>

PPS. Well, after reading the response letter it seems the Journal was kind of eager to publish. In the Neurosciences I'm used to more guarded language from editors even for papers that have basically been accepted.

<comment><USER3> The paper was accepted, under condition that these comments are addressed. Considering that the paper contains no meaningful content whatsoever, it should have been outright rejected.

It seems that the journal is one of a number of recent "low tier" (putting it mildly) journals that mainly profit of the processing fee they collect to publish the paper.

I doubt this could have gotten accepted, even conditionally, by a more established journal with a reputation to lose.

<comment><USER4> Here is the paper in full: [http://blog.richmond.edu/physicsbunn/2012/10/19/math- journal...](http://blog.richmond.edu/physicsbunn/2012/10/19/math-journal- publishes-computer-generated-fake-paper/)

Well, if they rewrote the abstract, and proved their results, it would have meaningful content..

<comment><USER3> I've seen it. They'd have to rewrite the abstract, state an actual result, prove it and rewrite everything in between. The "paper" is just grammatically correct but non-sensical glibberish from start to end.

If a referee accepts this paper conditionally, he might as well accept a blank page on condition that the author will put something there.

</comment></comment></comment></comment><comment><USER5> This isn't a pervasive problem in the peer-reviewed math literature. It is a problem with one publisher that is widely regarded to lack legitimacy.

See <https://www.google.com/search?q=%22scirp%22+scam> for some examples.

<comment><USER6> Yes, the journal has been labeled a money-making scam: if you need publications (for your academic career), they'll publish your paper in exchange for large publication fees. The "peer-reviews" are basically token reviews. I don't think there's any larger meaning here about the state of mathematics.

See one of the blog post comments for details: [http://www.lrb.co.uk/blog/2012/10/17/paul- taylor/stochastica...](http://www.lrb.co.uk/blog/2012/10/17/paul- taylor/stochastically-orthogonal/comment-page-1/#comment-6010)

(And as random trivia, the four-color theorem was the first major theorem to be proved using a computer. Based on the headline, I was expecting something like that.)

<comment><USER7> _Neither Marcie Rathke nor the University of Southern North Dakota at Hoople is willing to pay the 'processing charges' levied by Advances in Pure Mathematics, so we will never know if the work would actually have made it to publication._

\-- How much $$, just out of curiosity?

<comment><USER8> $500, see <http://thatsmathematics.com/blog/archives/102>

</comment></comment></comment></comment><comment><USER9> The tittle first mislead me because I thought somebody generated a theorem- prover smart enough to find something worth publishing (note: I know nothing about computer generated proofs of theorems).

After reading the "article" I found it obvious it was void of sense. "Let \rho = A", what is A? And who on earth would bother to write 0 as tan(\intfy^{-1}) ?? There are also proofs "left as an exercise to the reader"! Amongs many others those are obvious clues that the article is just a joke.

But thanks to that I know that the journal "Advances in Pure Mathematics" is a scam...

</comment><comment><USER10> The title is misleading. I came in expecting a proof found by an automated prover being explained by a natural language generator. Instead, it's just another joke paper where meaningless content is generated by some random process.

<comment><USER11> I expected to see some mathematical theorem unexpectedly proved by Mathematica or something.

</comment></comment><comment><USER12> How long before an ignorant humanities scholar writes a book claiming that the "Rathke Affair" proves that all math is nonsense?

<comment><USER13> We have independent verification that math works, because it is used every day in millions of companies to perform a variety of critical tasks.

The targets of the Sokal Affair were on much shakier ground to begin with.

<comment><USER7> Whether Academia is full of shit is irrelevant to the validity of mathematics. That being said, the GP comment basically demonstrates a peculiar cognitive bias. A variation of a logical fallacy: "I'm not dumb, therefore I never do dumb things." Of which we see all too often grafted into politics: Viz: Science is objective, therefore my (self-interested political) viewpoints are also objective, infallible, and inevitably correct.

</comment></comment></comment><comment><USER14> The university of southern north Dakota at hoople? Yeah, it doesn't exist. <http://en.wikipedia.org/wiki/P._D._Q._Bach>

...making the whole thing satire.

</comment><comment><USER15> The reviewer did not even look at the references... They are certainly the best part.

</comment><comment><USER16> Not to worry. Someone with a graduate-level background in actual math won't make it very far into the title before deciding the paper is nonsense.

After that, it's all fun and games trying to assign meanings to things. "Arrows of Equations and Problems in Formal Applied PDE" sounds like what happens when someone gets frustrated with their homework to me. ;)

I'm really impressed with the quality of the lorem ipsum, though. If you don't actually read what it says, just kinda skim over the formatting, it's surprisingly believable.

</comment><comment><USER17> Funny. One would imagine that filtering stuff like this should be very easy, because stuff in math either makes sense, or not, at least to someone knowledgeable in the field.

<comment><USER18> I propose a simple experiment:

Print out a copy of this paper. Take it to the nearest university campus. Ask math grad students and profs to look it over. I suspect almost all of them will ask if it's some kind of joke, or simply tell you it doesn't make any sense.

That's the filter we expect to apply. But the journal this was submitted to _doesn't use a math filter_ ; it uses a "did you pay me $500" filter, which isn't really useful for identifying legitimate mathematics.

<comment><USER18> I performed a variant of my suggested experiment. I went on facebook and found a friend of mine, who has a B.S. in mathematics and did some coursework toward a Masters.

It took him about 5 minutes to identify it as a hoax. And he was distracted (with facebook, football, and his 3 year old.)

He's how the exchange went:

12:11pm ME: tell me what you think of [http://thatsmathematics.com/blog/wp- content/uploads/2012/09/...](http://thatsmathematics.com/blog/wp- content/uploads/2012/09/mathgen-1389529747.pdf)

12:11pm HIM: How in-depth a review do you want?

12:12pm ME: just give me your first impressions

12:14pm HIM: My first thought: "Holy crap, there's a ton here I don't even remember..."

12:14pm ME: second thought?

12:15pm HIM: Second thought: "Is this legit? It's all over the place..."

12:15pm ME: third thought?

12:16pm HIM: "This is starting to look like something thrown together to appear like a legit math paper."

12:17pm ME: Thanks. That's adequate. It actually is.

12:17pm HIM: Yeah, I was about to say I was 100% certain after one more glance.

EDIT: my wife and I both have advanced math degrees, and we spent the morning laughing at this paper. But since we already knew it was a hoax from the HN post, we weren't valid targets for this experiment. My friend didn't know anything about it before this chat, so he was a valid experimental target. I think his response is adequate to demonstrate that this paper wouldn't pass peer review by actual mathematicians working for a legitimate journal.

<comment><USER16> _we spent the morning laughing at this paper_

And how. I still can't breathe quite right. I still think differentiable category theory was my favorite bit, but I'll admit the recent paper by Pythagoras has appeal.

</comment></comment></comment></comment><comment><USER19> It says "provisionally accepted". What's the provision?

<comment><USER20> That the only thing left to do is to pay the "processing fees".

<comment><USER19> Oh, sounds like a very prestigious journal.

</comment></comment></comment><comment><USER21> What should give you pause for thought is not so much the 'quality' of the review but the fact that there is a ready market for such journals. Else they would not exist.

What drives this market? Can it in any sense be said to be good and in public interest?

</comment><comment><USER22> Reminds me of the story about someone submitting a fake or computer generated paper on Derrida and deconstructionist analysis to a number of literary journals and having it accepted. I can't find the link though, unfortunately.

<comment><USER23> The Sokal Affair?

<http://en.wikipedia.org/wiki/Sokal_affair>

</comment></comment><comment><USER15> Can it be that the journal reviews the papers using computer-generated referees?

</comment><comment><USER24> Yes, a gobbledydook author (the program, not its creator) passed peer review of gobbledygook reviewers.

The insinuation that the mathematical community at large had any involvement in this self-proclaimed journal is false, and then headline is linkbait unworthy of HNm

</comment><comment><USER20> This finally happened. Narrow specialization at its best.

<comment><USER8> No, that's not what happened: this journal is a scam, they publish anything vaguely math paper-like in exchange for $500.

<comment><USER25> Then why even bother giving a referee report? But I guess scammers do many strange things...

<comment><USER8> This sounds to me like asking "Why would a scammer do something that makes the scam more plausible?". The answer is, of course, to make more people fall for it. Their target is tiny (people that want to write math papers!), so they need to give at least the appearance of being legit.

<comment><USER25> It depends if the authors are in on the scam or not -- is it scamming authors of their money, or scamming universities who are checking CVs?</comment></comment></comment></comment></comment>


Arial versus Helvetica - <AUTHOR> http://www.swiss-miss.com/2009/09/arial-versus-helvetica.html

<comment><USER2> This inspired me to generate some automatic font comparisons using Mathematica: <http://cronus.ws/~mta/fonts/>

Any feedback is appreciated.

<comment><USER3> How about Time New Roman vs. Georgia? It's always puzzled me that they look much different on screen, but fairly similar when printed.

<comment><USER2> That's a good, subtle one: <http://cronus.ws/~mta/fonts/GeorgiaTimesNewRoman.html>

<comment><USER3> OK, so I guess the slight differences are exaggerated by the font hinting. But the printer, being much higher resolution, can show the similar font shapes more clearly. Cool.

</comment></comment></comment></comment><comment><USER4> The easiest way to tell is that Helvetica is obsessive about straight horizontal edges, while Arial tends to use angled edges rather unsystematically.

<comment><USER5> Except, for some reason, the capital R in Helvetica, which is a very curvy letter.

<comment><USER6> But the R only curves so that its leg meets the base at a logical angle. Arial's straight leg looks straighter, but if you isolate it as a piece itself then that straight part cuts off at a diagonal angle to maintain the flat bottom.

<comment><USER5> Oh, wow, interesting; I hadn't thought about it that way.

(The R does still have a little serif though, which always bothers me.)

</comment></comment></comment></comment><comment><USER7> <http://www.helveticafilm.com/>

About the film: Helvetica is a feature-length independent film about typography, graphic design and global visual culture. It looks at the proliferation of one typeface (which celebrated its 50th birthday in 2007) as part of a larger conversation about the way type affects our lives. The film is an exploration of urban spaces in major cities and the type that inhabits them, and a fluid discussion with renowned designers about their work, the creative process, and the choices and aesthetics behind their use of type.

<comment><USER8> By total coincidence I watched that movie yesterday. I enjoyed it, though it's mostly just a bunch of people's opinion about Helvetica/modernism/post- modernism. Doesn't good much deeper than that in typeface design.

Can't remember where I heard about it (possibly MetaFilter, or here?).

<comment><USER9> MeFi, it was posted there a couple of days ago including the film link. My favorite comment from the the resulting thread was 'Impact or go home, font dorks', which I have been unable to get out of my head since.

</comment></comment></comment><comment><USER10> Does anyone know of reputable research and blind tests that back up the common designer sentiment against Arial?

The reaction of many designers seems to smell of elitism and subjectivism more than anything to me, but I'll be happily proven wrong.

Are designers commonly aware of _why Arial exists_. I presume many designers must understand the very strong business & licensing reasons for Microsoft to have developed this typeface.

Do many designers also understand the distinctly different approach to pixel alignment in display faces, which some people prefer while others prefer Apple's approach (I subjectively prefer Microsoft's approach).

It's easy to pile onto the _I hate Microsoft and everything they touch_ bandwagon, but how much unbiased and objective research has been applied to the question of qualitative value to the target audience? Is an old typeface only good if it's designed by the Swiss?

Something is stinky here. There are so many subjective factors at play while the language surrounding debate rarely recognizes those factors.

[Update: As noted by DrJokepu below, Microsoft didn't develop the font originally. I think the underlying licensing and business considerations remain a similar discussion.]

<comment><USER11> It isn't so much an issue of taste (or sentiment) as much as it is an issue of the design itself. Typography is about math/geometry as much as it is about taste. Helvetica was designed first, meticulously. Arial came later, resembling Helvetica, but breaking many typographic rules.

I'm a designer and I'm not particularly (at least not inordinately) impassioned about things like this, but I do find it important to recognize the difference between _design_ and art. It's often less about subjective preferences and more about purposeful/thoughtful solutions—in this case, what constitutes acceptable typography.

<comment><USER12> Note that Arial is not based on Helvetica, it is more like a cousin of Helvetica. Arial's ancestor is Grotesque 215. There's an excellent article about that here: <http://www.ms-studio.com/articlesarialsid.html>

</comment></comment></comment><comment><USER13> This is blogspam, [http://ragbag.tumblr.com/post/187708731/arial-helvetica- on-f...](http://ragbag.tumblr.com/post/187708731/arial-helvetica-on-friday-i- hosted-a-screening) is the actual source.

</comment><comment><AUTHOR> Loved this article. Amazing how such a subtle difference can cause such a big difference in visual impact

<comment><USER14> Um, the fact that there is a HOWTO tell the difference tells me that there isn't really much of a difference in real-world visual impact. Sure, Helvetica kerns a lot better than Arial, but the vast majority of people _don't see the difference_.

<comment><USER15> It might not be obvious given a few characters, but the overall impact is huge.

People who don't know the difference, and can't articulate it, will /still/ know something is wrong, even though they can't put their finger on it.

<comment><USER16> Which one should we use so that people don't feel bad due to our choice of minutely different font?

<comment><USER15> Meaning, which is the higher quality font between Arial and Helvetica?

Helvetica is the winner. Geometry is much tighter, plus the kerning and hinting is far better, and you have the possibility of using the many faces in the family from ultra condensed, to wide, from light to black.

</comment></comment></comment></comment></comment><comment><USER8> Strange timing. Yesterday I watched the documentary "Helvetica" (2007) which is all about typefaces:

<http://www.imdb.com/title/tt0847817/>

<comment><USER17> Funny, I was just reading about the Baader-Meinhof Phenomenon recently.

<comment><USER8> Thanks for giving me the name of that phenomenon. I knew about it - wasn't pretending that it was some supernatural occurance - just that even if you know about it, it still feels a bit strange when it happens.

</comment></comment></comment><comment><USER18> For people more typographically inclined, there is a much analysis and history of Arial at: <http://www.ms-studio.com/articles.html>

</comment><comment><USER19> OK now i know the difference and how to spot Arial, but can anyone show me why is Helvetica superior to Arial ?

<comment><USER20> I was under the impression that Arial was a cheap knock-off of Helvetica. They took a similar font and reshaped it so it could stand in for Helvetica.

Clearly this is uninspired, much like a KDE theme trying to recreate Mac OS X. There's no actual law that says such a knock-off must be inferior to the original but a) it's off to a bad start, and b) even if it ties with the original it's got the taint of being a rip-off motivated by greed.

The full history of Arial, from a link someone else posted: <http://www.ms- studio.com/articles.html>

Note that Red Hat has in turn commisioned a font that is "metric compatible" with Arial: <http://en.wikipedia.org/wiki/Liberation_fonts>

<comment><USER21> If they wanted an exact knock-off they could do so perfectly legally -- the only defensible IP afforded to typefaces is the trademark on their names -- the set of shapes for use in typesetting cannot be copyrighted or otherwise protected.

<comment><USER20> The link I provided above covers this and suggests that the foundry that created Arial didn't want to be a blatant pirate, so instead did just enough to pretend they weren't ripping off Helvetica.

 _"What is really strange about Arial is that it appears that Monotype was uncomfortable about doing a direct copy of Helvetica. They could very easily have done that and gotten away with it. Many type manufacturers in the past have done knock-offs of Helvetica that were indistinguishable or nearly so. [...] It's quite possible that most of the "Helvetica" seen in the '70s was actually not Helvetica.

Now, Monotype was a respected type foundry with a glorious past and perhaps the idea of being associated with these "pirates" was unacceptable. So, instead, they found a loophole and devised an "original" design that just happens to share exactly the same proportions and weight as another typeface.

This, to my mind, is almost worse than an outright copy. A copy, it could be said, pays homage (if not license fees) to the original by its very existence. Arial, on the other hand, pretends to be different. It says, in effect "I'm not Helvetica. I don't even look like Helvetica!", but gladly steps into the same shoes. In fact, it has no other role."_

</comment></comment></comment></comment><comment> onreact-com I'm afraid most hackers need a chart showing differences between Arial and Verdana or even worse, Arial and Times!</comment>