it. And what I learned over this period of time is now I've reduced my pack to the one


MozCon 2014 - 16 - Marshall Simmonds - Keep the Focus on the Doughnuts

on the right, which is now 10 pounds of base weight, which is a heck of a lot easier, and


I brought the tools that I need, not the ones that I wanted. And the third thing that I


do when I'm out there is I count my steps, because it gives you something to do. And


we have a lot of data available to us. And if it's the Fitbit or if it's the Nike Fuel


Band or the Garmin, you can see ‑‑ and you become a more efficient hiker, because


I can see what I'm doing, how I'm doing it, where I'm doing it, and what I need to be


doing. And all this data is available to me. And it's extremely powerful, because now we


can turn this into this sexy data, right? And we're inundated with sexy data. It's everywhere.


Publishers are embracing this data. And it's becoming extremely useful. It's useful in


ways that we haven't even seen before, but we start to see publishers starting to utilize


this data in a way that tells stories. And these stories become powerful because it connects


us. So, for example, this is an incredible visualization


of a company, and you can find this at IP Viking, that's tracking in real time every


cyber attack that's happening around the world. It's the only time I'll say cyber, by the


way. But cyber attacks that are happening. It's gorgeous. It's beautiful. It's not actionable,


but it's a great promotional tool. We also have access to every one of the weather


satellites that NOAA makes available to us, so we can track what's going on with air currents,


temperature, specifically wind. If you're a pilot, you can zero down to the airfield


to see wind velocity. Very actionable, very cool data, and beautiful.


The New York Times has embraced this big data journalism and used it in a number of different


ways, whether it's talking about an avalanche, whether it's a dialect quiz that you may have


taken, or it's this, where they actually just took Google map data that everybody has access


to, took Google map data and located a specific neighborhood in Detroit. 43,000 properties


that are on the brink of foreclosure, and they took a statistic, a number, 43,000 or


10% that you hear on the radio that goes in and out, and they made a story out of it,


and they connect you to it, because you look at these people's homes and you see this is


a visceral experience. This is not just journalism. This is an experience that you can only have


if you're utilizing this data and collecting it in a way to make the story possible to


make and really connect with users. That's powerful data.


FiveThirtyEight that everybody has been talking about, Rand talked about it yesterday, they


did some incredible coverage that made big data cool around the World Cup. If you haven't


checked out, they have a bracket for America's best burrito, highly actionable. There's open


source crowd sourcing abilities with target map where you can actually, if you're monitoring


this, you can make a map that shows where radiation is at its worst in Japan, and that's


important actionable data. But by far the best, in my opinion, is this guy that collects


every set list the band Fish has ever played since 1983, and he makes it available. And


with just a little bit of massage, you can show that, I mean, who knew? You and I have


been playing hundreds of times more than an ACD bag. That's actionable. That's important


data. Now, throughout all this, we've seen all this beautiful, big, sexy data, and it


is. And it's actionable. But there's some bad news. The bad news is, SEO is not sexy.


It's not sexy at all. Why is that? Because socials grabbed all that light, right? They've


got tools like BuzzSumo, and they've got the top seeds of the world that take real time


analysis, sentiment analysis from across every different medium, and they compile it so you


can see it in real time and act on it. That is sexy. That's the shiny object that continues


to pull from our budgets as an SEO. Right? And we're constantly fending off these attacks,


too, because you get this irresponsible journalism that's doing just click bait, like BuzzFeed


is doing, saying that this year Facebook far surpassed in their network, which we have


access to, they have this data that Facebook has far surpassed what Google is doing, which


is totally wrong. And then you get this irresponsible journalism compounding on itself where the


Atlantic picks it up, Recode picks it up, and regurgitates this message that your managers


are hearing and they're reading. And then Comscore comes out and says 67% of the market


share is Google's. We won't tell you how, we won't tell you why, but it is. Your managers


are reading this, and if they're looking at their budget and they're trying to see, okay,


where do we allocate budget for 2014 and 15, we've got social on this side and SEO on this


side, social is being louder. And so our budgets are being taken because of this misinformation.


And so our job as SEOs, we're constantly combating this misinformation, whether it's at the company


level or if it's at the corporate level like this. So SEO isn't sexy. But we have big data.


We have big data, but we have to make it useful. We have all this data. Everybody in this room


has access to this data, but we have to make it useful and we have to make it actionable.


And you have to go to school. That's unfortunate, but you have to go to school. And the school


is the school of Annie Cushing. She's making SEO data sexy. So go do your time and learn.


And if you don't do your time, have somebody on your team go do the time to learn about


how to mark this up and make this data sexy, because it will help. Because if you treat


SEO just as a project, it will fail. The day your manager or your boss thinks the SEO on


the site is done is the day you start your path to failure. Because SEO can never be


done. There can never be a start, middle, and end point. It's a constant evolving process.


Google is constantly moving. Because you can't half-ass this. If you go in and just kind


of do SEO, it's going to fail. You have to be all in or out. If you're not using all


the data that you have available to you from your client list or from your company, you're


missing an opportunity. So part of what we do when we work in the enterprise level is


we validate. We work with SEO teams all the time. And our job is to point to them and


say yes, what they're saying, for whatever reason they've been marginalized, what they're


saying is true. We have to prioritize that appropriately. And so that's what I want to


do. I want to give you some validation about the data that we're seeing. Because there's


so much misinformation, we need to set that record straight. So the methodology we use


is across our network, we have access to about 250 sites. For this, we took 127 across 59


categories, 119 billion total page views, 28 billion visits, and 15 billion total search


visits. That's the data that we're going to operate and we're going to slice and dice


throughout the rest of these slides. So you're able to draw conclusions on this billions


and billions of data points. And that's important because, again, we have social that is taking


our budget and taking the attention away from the focus where it should be. Because Comscore


is wrong. They said 67% of that market share is Google's. That's not true. It's just not


true. It's about 86%. Unfortunately, Comscore won't tell us and give us any transparency


as to how they're pulling in that data. Well, we're looking at 1.3 billion search referrals


and 86% of that is going to Google. Now, if your marketing manager was to look at this


and to look at Comscore's number, any person worth their weight in salt is going to look


at that and say, we need our piece of that pie. 86% of traffic is going to, is coming


from Google? You bet. And that's exactly why Google is not correcting Comscore's number.


They don't want that number to be out. That's a huge number. That invokes all kinds of attention


they don't want. Comscore sets the ad rates. Do you want your boss thinking that Comscore


is correct in that sense? No. And if you drill down a little bit further on that data, it


breaks out where we've got 39% coming from direct, 39% search and 22% social across 4.4


billion search referrals. Search has never been stronger as well. We're seeing on average


about a 42% year over year growth. That's strong. And it hasn't seemed to change. We


see that, we see these numbers continue to increase. It doesn't mean social is not changing


though and it doesn't mean that social is not growing. Social is very important and


any part of a good SEO strategy should be walking hand in hand with social. You should


be working with those teams to ensure that there is a compromise and at least there's


a combination of strategies there. So we take all this data and we use it so we can get


some shit done. So that's the validation piece. So when you're going into this, we need to


change some ways of thinking. And we need to think about this in a different way and


that's Panda is your greatest ally. If you're scared about getting hit by Panda, if you're


concerned, you probably deserve it. This is the best way to get the attention that you


need to further your agenda, to further your SEO agenda. You want to future proof yourself,


that's the smartest thing that you could possibly do for your site is to prepare yourself against


all these changes that Google is making. Google is constantly moving the goal posts. That's


job security. That's why SEO is going nowhere. That's why SEO will never die because Google


keeps changing what's happening and they need somebody to interpret it. So we are the interpreters.


There's no smarter move than to future proof yourself so when your boss comes to you and


says what are we doing about Panda, you've got an idea. Or what can we do to prepare?


We've got an idea behind that. We've got an agenda that we want to push. That's the best


thing that could have happened to us. You want to see these Google's been making these


kind of changes for years. Go back and look at the Florida update. These changes happen


all the time. This isn't new. Panda isn't new for all of us for a lot of us in this


room. We've seen this. We just don't care about it. It just gets the noise out of the


way so we can get that traffic. So far as the strategy is concerned and the tactics


are concerned, as we pointed out, there's so much misinformation that the first thing


that we have to be doing is training. Reinforcing a message. Because as organizations grow,


as there's turnover, we have to continue to reinforce this message. And we should be training


at every level. The editorial teams, product teams, tech teams, why wouldn't you be talking


to the biz dev teams or legal to make sure that any contract that goes out the door for


syndication is future proofed or at least for the business.


You should be training everybody in the company, upper management, as much as possible. There's


so many things to talk about. You can talk about keyword research, video, images, how


to write title tags, how to write headlines, and you should on a monthly basis. Because


people get caught up in their deadlines and they forget about SEO. Editorial teams love


to forget about SEO. And they have to go back and retrofit. And it's too late. If you're


dealing with Google news, you're going to have to go back and retrofit.


And it's too late. If you're dealing with Google news, it's the first one that gets


in. And so how do we do that? We have to provide data. Right? We have to use this big data.


We have to control this data and give it back. We have to give it back to make sure that


we quantify the results and that we reward people for the work that they're doing. Because


they're the ones that are in the trenches doing the work and we have to quantify it.


Two tools that will change your life. One, Supermetrics Data Grabber for Google Analytics.


The other one is Omniture Report Builder. Supermetrics Data Grabber, which is fun to


say, is $300 a year. Omniture Report Builder is $300 billion a year. But if you have it,


you're fine and it does things for you. But Supermetrics Data Grabber, I bet everybody


in this room could get $300 in budget to help build this data feedback loop. Why? Because


dashboards are cool. Dashboards help us quantify results. They help us get in front of people,


get in front of management, get in front of all the upper and lower decks to report on


the progress. Red is bad. Green is good. What kind of charts? You can segment data. You


can show which a friend of mine has always said, I use this as a carrot or a stick. And


tell me if you put a dashboard in front of your manager, he or she would say, no, don't


ever send that to me again. It's valuable, easy to quantify data and you mark it up because


you can show what's happening with mobile, what's happening with the home page, what's


happening with your section fronts. And you should be sending this around on a monthly


basis, because if you're not, you're missing an opportunity. And if it's too hard, we built


one for you. Go grab our template, get Supermetrics, Superdata Metrics, well, Supermetrics Data


Grabber, and dump it in there. That's yours to keep.


So for a second, let's go back in time. 2012 image search, Google did this. You clicked


on an image. It would load your site in the background with a pop-out. 2013, they changed


the carousel. You clicked on an image. No more site, right? They give you attribution,


but no more site gets loaded. So what happens to traffic? It plummets. This isn't news,


right? This 2013, image traffic just tanked. And that really sucked. Because image traffic


was a great resource, is a great source of traffic that effectively came to a halt. And


so what did Google say about this? Pirafar said that Phantom visits were causing problems


for some Webmasters, and so we thought we can do away with that. That's gone now. This,


along with new options to reach publisher sites, is a net win for Webmasters. You want


to see the net win? That's the net win for Webmasters. Across billions of page views.


That's the net win. That's what Google's doing. They're making a land grab for that. Why?


Because Google used to be a search engine that had advertisements. Now they're an advertising


company that has a search engine. They want to keep you on the site. That's what they're


doing. We've tracked this. We've tracked this for 18 months now, and it looks like image


traffic is effectively gone. Sure, there's a little bit of it, and there's ways to protect


your content, because now we have to go on the defensive. We have to protect our content


from Google, which is very interesting to think of. And there's other companies that


have done it, like Pixabay. Go check these guys out and read their blog and what they've


been doing. They watermark all their images now. They watermark their images, and as it


turns out, they've actually seen some traffic increases. And Google's whacking at it. So


what do we recommend? We recommend that you watermark and get control of your data. If


you're using WordPress, check out the WordPress pick shield, because publishers love their


slideshows, but nobody's quite figured out how to do it. Do we use real canonical? Do


we use pagination? Do we use real previous and next? What do we do with these? Nobody's


figured out how to do it, because they suck. You get 80 different ways that you get 80


different comfortable pillows that you have to pick, you know, that you have to select.


Does it load pages? Do you do it dynamically? What can you do? We've tried real canonical,


which is technically a misuse of it, pointing everything back in traffic tanked. Sometimes


it did well. Google's wildly inconsistent with how they treat slideshows. One thing


we have found that works, though, is the view all. It gives the power back to the users,


and then you can optimize it. You can optimize this page easily, because these are thin content


factories. But if you point to a view all page and use the real canonical tag, that


works. Then you can optimize that content. And then you future proof. If it's e-commerce,


it's a little bit different, so I recommend that you go read Audette's blog. Adam Audette,


who spoke here last year, and the bit.ly URL is A-U-D-T-T-E, he wrote about what to do


with rel previous and next, because we can't get it to work. Sometimes, but not all the


time. It's inconsistent. Which leads us to our second problem, which is indexation. One


of the biggest issues that we come across with enterprise-level sites is effectively


and efficiently getting our corpus into Google's. And the takeaway here is we don't rely on


that on Google Webmaster Tools. Why would you rely on the data that the company is giving


you that doesn't trust you with your own keyword data? You're taking Google Webmaster Tools


for face value that they're telling you the health of your site. Take back your data and


get control of it. Take your XML site maps, get Screaming Frog, put it into list mode,


run it through, and look at the health of your site maps. You're looking for 301s, 302s,


404s, 410s, anything on there that Google doesn't like. Even if you download the API


feed from Google Webmaster Tools, you're not going to get all the information. So that


is a way to monitor the health. One of the most powerful queries you can use is you run


the site colon on your root domain minus www. Enterprise-level sites, you're going to find


staging servers that were pushed, development servers that were pushed to inadvertently


push to production, you're going to find duplicate content, you're going to find things that


were left over after a migration. It's a very powerful query, especially when you're auditing.


But you say, well, we got this, it's fine that we have duplicate content. We have system


checks in place that apply real canonical text. Let's talk about real canonical for


a second. Because what real canonical is by definition is a strong suggestion. It's a


strong suggestion, right? It's not a 301, it's not a redirect, it's a suggestion. So


what I have here is a query from the New York Times where I found 577,000 properly


implemented real canonical tags that were crawled as recently as last month. Do you


want Google spending all of its time and resources actively crawling real canonical pages? I


don't. That's not a good use. That's a crawl barrier. That's not where we want our time


used. We make the decision, the conscious decision to fix the problem. Real canonical


is a Band-Aid. And remember what Rand said yesterday about our dependency on Google?


Real canonical is Google dependent. So move away from the real canonical tag and fix the


problem. Fix the problem of duplicate content. Because a 301 is much better than real canonical


tag is. So you take the haircut, you take that 15% haircut and play the long ball there.


We have a client that has 40 million pieces of content. And they ‑‑ and it's good


content. Not all of it was good, but we cleaned up the duplicate, we cleaned up the overlapping


content. But they have 40 million pieces of content. What we said to them is let's look


at out of all that content where the traffic is going. And what we found was that the traffic


was going to very few pages. And so we proposed the idea, let's start removing content and


see what happens. We removed 90% of their site and traffic grew 40%. So in this situation


less is more. And I caution you that your mileage may vary with that. But it's an interesting


thought. That it's not about that shotgun method of producing as much content as fast


as you can, but it's producing good content. And if you get out of sorts, you need to future


proof. So these guys have gone a step farther and now whenever they produce content that


is under a certain word count or under a certain value, they'll slap a nofollow tag on it or


noindex tag on it. It might be usable to the users, but we don't want it. It's in the AAA


league until it's ready for prime time. And if it builds up and it graduates, then we'll


take that tag off. So those are the system checks that you can automate to ensure that


you are future proofed. So we can't talk about publishing unless you talk about Google News,


because Google News is an excellent side door into the Web results. And when I'm saying


side door, think about all the times you see that news module at the top of search results


when anything trends. If you can't get into Google News, you should try, because it's


getting more restrictive, but if you don't like what they're saying, like Cyrus said,


it's my favorite quote too. If you don't like what they're saying about you, change the


conversation. Find a way to get into Google News, but be ready. You've got to be really


ready. Go read Adam Shirk's blog about how to get into Google News. Because if you're


a publisher, if you have content that's worthy, you want to get in there, because there's


a lot of very interesting tools in there that you can use. So not a lot of people are going


to news.google.com, and that's what we're talking about right now. Give us a couple


months and we'll tell you how much traffic is coming from the Google News one box from


the Web results page. But for now, we're talking about just news.google.com. So if we look


at 11 of our top publishers and how much content that they produce, which is a lot in the last


18 months, and how many visitors come from that, 60 million are coming from just news.google.com.


That's a lot of people using it. That's a lot of users using it. That's enough for me


to justify this. And there's a lot of things going on that we don't necessarily need to


get into because there's a lot of data around it. But the Google News keyword tag, it's


useful, but it doesn't supplement the fact that you need to be writing good headlines


and you need to be training around good headlines and titles. There's the standout tag, which


looks cool, doesn't have a lot of traffic impact, and there's workflow issues because


you only get seven. But a little-known secret about this is that you get as many as you


want if you're using the standout tag for other pieces of content offsite. So if you


have sister sites, why wouldn't you be using it? Why wouldn't you be using the standout


tag as much as you can? And then there's the editor's pick. And this is for the people


that have customized their Google News experience where you can actually see, like we used to


do with my Yahoo, you can customize the modules. And there's no traffic impact by using this,


but it's just an RSS feed, so why wouldn't you for the 60 million visits that are coming


from news.google. It's a great opportunity. The one caveat to this is that if you don't


produce more than 80 words, Google will throw an error. Google News will throw an error


in Webmaster Tools and the content won't be indexed. And with Google News, it's a race.


Shitty content gets into Google News all the time. Parasitic journalism, the Huffington


Post poaches the New York Times content all the time because they're better headline writers.


No, that's not true. They are better Google News headline writers. And so they beat us


all the time. However, when you're watching a trending story, you may only have a sentence


or two, right? The other problem is that if you're running inline ads, we call them speed


bumps, if this ad appears before that 80-word mark, it throws an error. So you have to watch.


But that doesn't happen, especially if you're in a race for traffic. And finally, one of


the big tools that we like to use is News Whip. It's people powered. It's crowd sourced.


It's like BuzzSumo. It's like some of the other popular tools out there, but it's people-driven.


Google Analytics versus Omniture. This data is very interesting to me because these are


on sites that are running both at the same time. So the takeaway here is the blue line,


Google Analytics is underreporting. Omniture overreports. So if you're running Google,


if you're running Omniture, you absolutely should be running Google Analytics for a second


data point. Like Kyle said yesterday, what's better than one data point? Two. And as SEOs,


we're often asked to predict if we want resources, if we want head count, predict what those


changes that you're proposing, what they will make, what kind of traffic increase will we


see. If you have two data points, it's a heck of a lot easier. We all have to do that. We


all have to try to project. So if you can run two data points, absolutely. It's free.


If your boss says, oh, but Google will get our data, Google has your data. That's not


the problem. Ask Mike King what Chrome is doing when it's on your site.


If you're looking for competitive analysis, your new best friend is SimilarWeb. It is


eerie how accurate their information is. It's so eerie that they won't tell you how they


get it, which means it's valuable. And some of it is free and it gets expensive real quick.


But this is cool. Competitive analysis, that's your new best friend. The mother load of all


big data that you have access to is your log files, your server files. Everybody in the


room technically has access to it. Getting it is a different story because it's ugly.


It's ugly as sin and parsing it and making sense of it isn't easy.


For example, we took a big publisher and took 30 days of data, which was an insane amount,


it's like 28 gigs, just one month of data, and we parsed it and we said, how many times


did Googlebot request content from our site? And it came back and told us we found Googlebot


coming in 1.3 million times a day. For that same time period, you go into Google Webmaster


Tools and ask that same question and look it up, Google Webmaster Tools says 12,000.


Google Webmaster Tools is only going to tell you what they want you to know. Why would


they do any different? We know. They're only going to show us what they want us to know.


So if you want the true story, if you want what Google is doing on your site, get into


your log files. Because that's a big difference. There's a story there. There's a story that


we're going to have to pull out and we're going to have to visualize so we can act on


it, so we can tactically act on it. And your new friend is Splunk. Because Splunk will


take that data and you can upload it and it will help you parse it. You just ask, hey,


what is Googlebot doing and it will visualize that data for you. It will show you exactly


what Googlebot is doing. In fact, it will show you where Googlebot goes. And what was


surprising is that you'll find that what you think is happening isn't happening. Now, crawler


activity does not dictate popularity of the site and it does not dictate traffic. It's


just showing where Google is active. And what's interesting about this data that we found


is that this did not correlate at all with traffic. And it didn't correlate with popularity.


It just showed us where Google was going. I want to know that. If you're going through


a redesign, if you're redesigning the site, if you're into information architecture, if


you want to know how to cascade page rank, which we all do, this will tell you. Because


what we found was that, oh, the business section touches everything on the site, more so than


the home page, at least important enough that Googlebot is calling it and accessing it often.


That's powerful data. Get your data back and control it. It's out there. And you can do


something with it. That's extremely actionable. Because if we're launching a new product package


or some piece of content, guess where I'm going to feature it? I'm going to feature


it here and I'm going to feature it on the most crawled pages on our site. That's powerful.


And that's extremely actionable. And the pages that you're seeing there probably aren't the


pages you think they are. Because they're not the most popular and they're not the most


trafficked. They're just the most actively crawled. So if your site is too big for Splunk,


there's other tools out there. There's WebLog Expert. There's Exact Trend. And they'll visualize


all this data. They'll help you make that story that you need to connect with upper


management. And if you're interested in this, the starting point is David Sodomano's piece


that he did for the Moz blog two years ago. It's a great entry point. And the bit.ly URL


is just Moz log files. That's the starting point. The guy knows what he's talking about.


You should read it if you're interested in it. So the roadmap kind of looks like this.


You've got to constantly be training. You have to reinforce the message. SEO isn't the


most important thing in your organization to anybody else but you. You've got to create


those dashboards to quantify the efforts, to give that feedback loop. Grab our dashboard.


Read the blog. Watermark your images. Protect your assets because Google's not giving you


the traffic so you might as well protect it. You've got to play with real canonical and


real previous and next because it's not necessarily consistent. It's not doing what Google always


says it's doing. And with slide shows, just point it to view all. That's what your users


want. And you can't rely on what Google Webmaster Tools is telling you. We are past that point.


So take that data back. System checks to automate because sometimes less is more. Google news


is great. Gives us chocolate cake. Always run Google Analytics under Omniture. And don't


forget the real story, what log file analysis can tell you about your site.


So last thing is I just wish for everybody these rolling green hills on a downward slope


because that's the easiest. But you've got to have a lot of intense challenges because


it makes us better at what we do and it makes us better as people. So thanks for the time.


It's really nice to be here. Thanks.


I had to write this down. Google is an advertising agency that has a search engine. That's


fairly accurate at this point. Unfortunately. So you, I just saw people writing down resources


the entire time. The power of MozCon, we crashed your site, unfortunately.


Oh, that's awesome. It's back up now. I was so excited to hear you.


That's awesome. Thanks.


Vouch for a supermetrics data grabber. We use that at Moz. Isn't that great?


And if you don't want to put out the $300, they have a 14-day free trial.


No. Which is awesome.


Those guys work hard. $300.


So you talked, one question I had, it always comes up, we hear about this. You talk about


that site with 40 million pages where you removed content. And we hear this a lot of


people, you know, should I remove pages? No one ever really knows when the answer is yes.


What sort of, for sites without 40 million pages, what kind of questions should people be asking?


It's the value. Is there value to the user? The value to the user, they're going to find


those pages no matter what if there are endemic traffic on your site. But the question really


is, is it valuable to be pushing out in search? Is it valuable to spend the cycles to optimize,


to do the research, to have somebody do the optimization? That's kind of the weights and


balances that we look at. And so we were able to determine that there was a lot of content


that didn't meet that threshold. So we removed it and it worked. But it's a gamble. It's


a huge gamble. That's a scary gamble.


Rand. Oh, boy.


What? I knew it would be you. Hello.


It doesn't even sound like me. You look great.


I can't believe they played that video. I wanted to ask about the trust in Google Webmaster


Tools data. So, I mean, what you showed in the crawl on that site, is that an outlier


or are you finding that sort of order of magnitude or several orders of magnitude difference


in what's actually being crawled and what Google Webmaster Tools is saying being crawled


that different? And then the second part of that is what about keyword data in Google


Webmaster Tools?


Yeah, it's a good question. So the bigger the site, the bigger the delta. So that may


be that, because, look, we have an enterprise crawler and we know that every enterprise


crawl takes a little bit of manipulation, takes some massage. You've got to monitor


those dials. Google is no different. They're not infallible as a crawler. And so the bigger


the site, the more mistakes, the more the chance that they're not going to get all the


content that you want. They're not going to actively crawl the content, which is why indexation


is such a problem for big sites, to get a corpus into theirs. It's not easy to do. And


so it's economy is at scale at that point. But the other thing is your question about


I'm sorry, I forgot the key word data. Keyword data. So we kind of ignore it.


Now, if you talk to Vanessa Fox, she says no, but that's it's been average for personalization.


So what you're seeing in there is actually the source of personalization, the end result


of personalization. I don't buy it. I don't buy that at all, in fact. There's so many


there's so much misinformation. And I'm not saying that Google Webmaster Tools isn't important.


It really is. But it's just one data point. It's one data point in many. And that's 20%.


The other 80%, you got to go get. And it's a little blood, sweat and tears.


Q Can I ask the blood, sweat and tears on that getting that key word data, especially


with not provided, how do you guys do that at an enterprise level to compare against


Google?


A We don't. That data is not interesting to us.


Q Oh. Wow. Thanks.


All right. One more quick question for Marshall. Yeah, I just want to apologize to everyone


here, especially Rand for my underwear being out of where. But I will take care of that


later. So appreciate being called out on that. I just want to back up a statement about that's


funny. I'm sitting there and he's on the video and I'm going, that's me.


And you're tough to do. Yeah. Well, I'm Chuck because it's a poll. I'm on the golf course.


I don't know. We need a ruling on that. Yeah, I don't know. I've got dress pants. Sorry.


Are they pleaded or not pleaded? They are not pleaded. Good.


My wife would kill me.


You're on your way, sir. Thank you, sir. And I appreciate your untucked shirt. You know,


you could work that even though it was I couldn't. It was a point of contention in breakfast


yesterday. Talk to her. No talk. Yeah. So I just wanted to back you up on the point about


knocking out pages. We actually went through. We knocked out a thousand or so pages, but we


found we crossed them with social data. We crossed them with links from OSCE and also data


from Google Analytics. Most of those pages weren't getting traffic to start with. So we really


weren't losing a lot. What was coming into the site. But then our crawl efficiency went up


and we've seen improvement to back that up. It was scary as hell. But check the data and it's


okay. It works. And it is scary. But I would say then baby step it. Put the no index tag on it


first. Nothing happens. Then you can access. You don't have to just you don't have to cut


everything. You don't have to cut it off at the knees. Baby step into it. Thank you for sharing


Prioritizing Technical SEO Fixes — Whiteboard Friday

Hi Moz fans, I'm Oli King, I work here at Moz and I'm excited to join you today for


another edition of Whiteboard Friday. Today we'll be talking about prioritizing SEO fixes.


By now you are probably already familiar with the concept of technical SEO and if you aren't,


there are a bunch of resources out there including the Moz blog and other sites that you can check


out but technical SEO is probably the most important part of SEO because if you have


the best content on the web, you have the most backlinks, if your site is not technically


sound then you might not be able to get the best results from all of your efforts. So


your technical SEO is really the foundation of everything else you do. There's a bunch


of tools like SiteBob, Moz Pro and Screaming Frog that can allow you to analyze your technical


SEO issues involved but once you have the list of the issues you might not always know


how to prioritize your effort so today's session is to help you have a better handle on that


once you have a list of those issues. For your technical SEO, the first thing the search


engines will do towards your website is really to crawl it so you want to make sure that


you have your sitemap set up correctly and then you have to make sure that your robot.txt


is set up correctly so that the search engines can crawl your most relevant pages on your


site. Well once they've crawled your site, you want to ensure that the crawl budget is


spent towards indexing the relevant pages on your site so today we'll cover the pages


that you should be indexing and we'll also then talk about how you can fix those pages


or how to prioritize your efforts towards fixing those pages. So first thing, index.


The pages that you should be indexing are really pages that are important towards your


business. So what's your KPI, what's going to drive leads to your site, what's going


to drive traffic or are there strategic pages that you're trying to get more results from?


With that, know what those are and those are the things that you want to make sure that


are indexed and of course the larger your website, the more you have to pay attention


to this. Maybe earlier on when it's a brand new site, it's okay to index everything initially


but as your site grows, you might want to be more careful of that and of course you


don't want to index all the private and sensitive pages so think of your login page, privacy


policy pages. They should exist on your site of course but they don't need to be indexed


so you want to ensure that no index is set up for those pages and you can do that from


the very source by setting up your robot.txt to ensure that those pages are not even crawled


in the first place. So the pages that you should prioritize in terms of indexing for


your site are the iTrafficValue pages. So these are sites that are either you want to


get more traffic to us or are already getting lots of traffic and they could also be sites


that might not be getting lots of traffic but they are strategic in terms of they bring


quality traffic to your website or you expect them to bring quality traffic to your site.


Then iLinksValue, you want to ensure that the pages that are positioned on your website


to drive links are indexed or if they are already driving links, you don't want to mess


up that aspect and you can use once again tools like MozPro to analyze pages that are


getting links on your site so you can use the link explorer for that and make sure that


your internal links are set up as well, sorry, make sure that you're not messing up pages


that are driving value to other pages as well so even if they don't have external links,


if they are important in terms of internal links, they are important to you to index


as well. They add iKeywordValue as well if a page is getting a lot of your links, a lot


of your keywords, you want to ensure a page like that is indexed as well. So some pages,


the role in the user journey, some pages might not have much SEO value but they are still


very useful to your user journey so think of like your ELF pages, they probably won't


rank for a lot of keywords to be honest, however, if your customer goes to the search engine


and search for a solution to something, you want to direct them to those ELF pages and


you want to ensure that they are finding those easily so not a lot of SEO value in the traditional


sense but it's still very useful towards your user journey and also pages that have placed


in a very prominent position on your site should be indexed as well so your own page


most likely every link on your page should not lead to a dead end, it should lead customers


towards the intended target so no 404 errors on your own page or other pages that are very


important to your user journey. So a way to prevent that is you can set up a Google Analytics


custom report so that anytime there is an issue on a page like that, you are alerted


quickly. So these are the pages that you should be indexing, if there is something that you


think should be here, please let me know, this is not meant to be a perfect list but


this is based on my experience so far, what I've found but you might know something that


I don't so please let me know. So for the fixes, in terms of the fixes, this is how


you should prioritize what you should fix so these are the prioritization factors so


once again, page value, pages that are ranking, that are getting links, that are getting clicks,


you should prioritize fixing those quicker than other pages, pages that align with your


priority or that would have impact towards your KPI, you should probably prioritize those


pages as well and there are some pages that might not be doing quite well but they have


a high ranking potential so if a page is on page 2 and you know that it's going after


a keyword that is not so difficult, then you might want to prioritize fixing that page


so that you can start getting results a lot quicker and then by issue type as well, some


issues are worth going after and fixing quicker than other issues and the technical effort


as well, some issues are a lot easier to fix, if you are not technical, this is not for


you to determine, you might want to talk to your developers and for them to estimate how


hard or easy a page would be to fix, if you are technical, of course you can be the judge


of that yourself but in terms of the issue type, this is the list of the issues, the


common issues and how to prioritize them so you have the critical crawler issues, so if


you have a server error or a broken page, so the 4x6 errors, make sure you fix those


immediately because they are impacting your user journey and potential rankings as well,


the metadata issues as well, so things like missing description, those are things you


want to fix next after the pages are already accessible, redirect issues as well, no one


wants a redirect chain, so make sure that's fixed after you fix your metadata issues and


content issues are minor tweaks there, so they are important but they are very much


low on your priority, so if you are finding like you don't have the right keyword for


example in your URL, that might not always be very important to fix right away because


the efforts to fix that might not be worth the result in the sense of just fixing a URL


issue would actually result in doing an audit to make sure that you are not breaking anything


else on your site, so for something that might not have a lot of impact, it is leading to


a lot of work which is not super valuable to you, so yeah, so these are the things that


I consider when I'm fixing a site's technical issue, there are still many steps to this,


we are not able to cover all of that due to the time constraints, but things like for


example I mentioned the technical efforts, so you might want to use things like t-shirt


sizing for example to determine what issues are small, what are medium, large, extra large


and so on and you can also, depending on your project management tool, you might be able


to set up a sorting function so that you are able to do this automatically, so once you


upload your URLs for example into Google Sheet, you can be able to set up a script that allows


you to, once you've selected the effort, the issue type and the value to you, you should


be able to sort that automatically as well, so all of this work is a lot quicker, but


yeah I would like to know what you think about this and I would love to learn from you as


MozCon 2014 - 05 - Kyle Rush - Architecting Great Experiments

Who wore the green bow tie better? Me, right? All right. I'm just kidding, Cyrus. There's


room for two green bow ties in the world. Thank you all for having me back. I love coming


to Seattle. It's such a beautiful city and I love coming to Moscon. It's one of the best


conferences I get to attend every year. So last year I talked a little bit about the


success that we had on the campaign. I did a lot of examples. But what I've noticed in


the A B testing community is that nobody, we all talk about what we're testing, but


a lot of people don't talk about how we're testing it. And it turns out that that's very,


very critical to the success of your A B testing operation. So when we're thinking about and


brainstorming about what topics I was going to cover this year, I wanted to do a deep


dive into every stage of the experimentation process. And so I have to warn you guys, there's


statistics behind A B testing. I don't know if you all know that. There's a little bit


of math involved, so brace yourself. My suggestion would be if you have a five-hour energy or


a Starbucks on your desk, just chug that now because it's going to get a little bit technical.


So I apologize for that. But I've done my best to make it easy to understand.


And so the first thing is that I've made my presentation really easy to understand. When


you see this blue star, that means you should write something down that's on the screen


or tweet it. If you write it down, you'll just feel smart yourself. If you tweet it,


you'll look smart on Twitter. Either way, the option is yours. But don't forget the


blue star. You'll see it several times. So the first thing I want to cover are some


experiment concepts. This is the hard technical part. And then once we get through that, everything


will be easy, I promise. But it's really important to understand these concepts if you're going


to be doing A B testing. And so the first one is significance. You all have probably


heard this word before, probably called statistical significance. And so I want to briefly explain


to this, explain to you guys what this means. Significance is basically the risk of encountering


a false positive. You'll commonly hear this called a type one error. And just, you know,


a very basic explanation of this is that if you ran 100 AA tests, an AA test is where


there is no difference between either the control or the variation. Five of those will


inaccurately report a difference. And that's just the, you know, the basic bottom line


of statistics. 95% is the level that people usually choose. The higher you go, the less


false positives you'll actually get in your reporting. So that's pretty basic and easy


to understand. The next concept I want to cover is statistical


power. I always call this power. And this is just sort of like the opposite of significance.


It's the risk of encountering a false negative, which is commonly called a type two error.


And so to make this easy to understand, all you have to remember is that out of instead


of AA tests, it's easier to understand this one if you're doing AB tests where the variation


is actually different than your control. So if you ran 100 of these, 20 of the winners


when they are actual winners, they wouldn't be reported. That's a pretty bad problem,


right? I think everybody expects AB testing when they're doing that to be the end all


be all solution, you know, because everybody is like, well, let's just test it and we'll


figure it out. But it's not always the case that you're always going to get a reliable


result unless you're doing, following these standards. So 80% is sort of like the standard


that people use. And the same with significance. If you increase your statistical power, you're


going to get better results. All right. So that's sort of like the meat


out of the way. But another concept is tails in your hypothesis test. Nobody really knows


this. I feel like it's not covered enough in AB testing. If you already know it, that's


awesome. If you don't, I'm happy to share this with you because it's a very, very important


concept to understand. A one-tailed test is only going to tell you


if your variation is better. A lot of the AB testing software on the market today only


does a one-tailed test. There's a reason for this. And basically, the explanation of it


is that it's easier and quicker to find out if the variation is better, but it takes a


longer time and more sample size if you want to know if it's worse.


But this is why most AB testing software uses a one-tailed test. A two-tailed test tells


you if the hypothesis is correct in both directions, right? It will tell you if the variation is


better and it will also tell you if it's worse. Most of the time when I'm doing an AB test,


I only want to know if the variation is better because if it's not better, I won't deploy


the change. I'll just stick with the control. And I have a feeling most other people do


that out there. But if you're running a test that is important


for the bottom line of a company and maybe it's going to determine, you know, a change


in your marketing strategy in the long term, you might want to run a two-tailed test because


it's going to give you a little bit more accurate results and it's going to tell you if it's


worse and not just better. So that's the difference between one and two


tails. Oh, sorry. Yeah. Is the variation better or worse?


Another concept, this is one that you should write down. There will be a star for it in


a little bit, is MDE. And this stands for minimum detectable effect. You have to sort


of work through a little calculator and some math to understand how this works. But and


I'll show that to you right now. But the important thing to know, and here's


your first blue star, so tweet this or write it down, successful testing strategies are


based around the minimum detectable effect variable. And what this means is when you


have a certain sample size, the way that statistics works, there will be an effect size, which


is the difference between the control and the variation that you can reliably detect


in that experiment. If the difference is less than that, then it's not going to be a reliable


result. So naturally, you can see that it's very important to pay attention to MDE when


you're running experiments. All right. So the last concept that I'm going


to cover and then we're going to go into a little bit of math is sample size. And this


is pretty simple, I think, for everybody to understand. That's how many subjects are in


your experiment. All right. So what I want to do, well, okay,


another blue star. There's so many things. Always use a sample size calculator to calculate


your sample size before starting an A-B test. Most A-B testing software on the market does


not guide you through this. This is like an absolute key to running a successful experiment.


When I first started running A-B tests, I wasn't calculating my sample size, and I had


a lot of false positives. I had some false negatives. But I didn't know that until I


started calculating my sample size in advance. If you want to know exactly why this happens,


there's an article that you can read that if you just copy that bit.ly link right there,


it will take you right to it. But it's written by a guy named Evan Miller who publishes a


lot of great content and makes a lot of great tools for A-B testing. But I didn't want to


get into the math here because I think it's a little too heavy. But go ahead and read


that article if you want to know why you should always use a sample size calculator to calculate


the sample size before you start your experiment. All right. So here is the sample size calculator


that Optimizely produced. I used to use Evan Miller's sample size calculator, but the one


that Optimizely just released does one and two-tailed calculations, so I think it's a


little bit better. So I'm giving you a link here, but I also just gave you a link to Evan


Miller's website if you want to use his. Either would be fine. But if you do want to switch


between one and two-tailed calculations, I definitely suggest using the Optimizely calculator.


All right. So I was talking about earlier how we would go through a little bit of math


to understand what MDE is. And this is an example of how I think people often focus


on absolute lowest MDE. And the reason is because anybody will get up or, you know,


if you're having a conversation, anybody will say a 1% difference between the control and


the variation is important to the bottom line of your business. And that's certainly true.


I would love to be able to detect a 1% difference in my experiments, but the problem is, as


you'll see, is that you need a very, very high sample size to detect a 1% effect. So


let's walk through this. So I'm going to assume that, you know, the


experiment that we're going to run has a baseline conversion rate of about 4%. And that's just


the conversion rate over time, right? We're not running an experiment right there. We


get a 4% conversion rate on whatever action we're asking to happen.


We said that we want the absolute lowest, so I'm just choosing MDE. I don't want to


go like.0005. But let's just say 1%, because everybody here wants to be able to detect


it within 1%, right? Power we're setting at 80%, like we discussed before. Significance


is at 5%. These are pretty standard values for these. And we're doing a one-tailed test.


So we should need a pretty low sample size, right? Well, it turns out that we're going


to need 2,972,435 visits per branch. Okay? That's if we have an A B test, we have a control


which is one branch, and then we have a variation which is one branch. So we need double this.


So that's a lot of traffic. I want to see a show of hands here. Who has this much traffic


in, let's say, two weeks to a month? Okay. There's not a lot of hands. I don't know if


you guys can see, it's not a lot of hands. Notice my hand didn't go up. The company that


I work for does not have this kind of traffic. So this is a mistake that a lot of people


do when they're conducting their A B test is they think it's just going to tell them


the absolute truth within a very, very small difference like that. You don't have sample


sizes like that, so it's not going to tell you that. So let me show you how to rethink


minimum detectable effect. So there's a different way when you're calculating your experiments,


and that's to focus on time. So what you can do is say I get this much traffic in this


much time frame, and I really only want to run my experiment for, let's say, two weeks


or even a month, right? So you have to figure out what your MDE is based on that sample


size. So let's assume that one month of traffic is around 170,000 unique visitors. This is


something that's attainable in my company, so this is why I use this number. But I'm


going to show you how you can do this with lower sample sizes in a bit.


So the conversion rate here, I'm keeping that the same. It's 4 percent. It's a pretty standard


conversion rate, I think. MDE, I'm setting this to 6 instead of 1 this time. Now I'm


setting the power. That's the same, 80 percent. Significance is at 5 percent. That's fine.


And I'm using a one-tailed test. Here, my sample size now per variation is about 83,230,


which is now attainable for me, and I'm running this for one month. It's not possible for


the company that I work for for us to detect a difference in 1 percent. Our MDE in this


case is 6 percent. So that changes our experimentation strategy. We're not going to make little small


color changes to our pages expecting that to make a 6 percent difference, because likely


it's not. So this is going to radically change how we're thinking about the experiments that


we run. And as promised, I also want to talk about


how small startups can A-B test, because a lot of people think, oh, I don't receive a


lot of traffic, so I can't be doing any A-B tests, which is not true. You should not listen


to that advice. I'm not saying that A-B testing is the highest ROI for a small startup, but


if you want to test something, you certainly can.


And here's how you do it. So you have one month of traffic. Let's assume that a small


startup, I would say somewhere around 3,000 unique visitors, substantially less than we


had before in the other example. Keeping the conversion rate the same, 4 percent.


Our MDE now, I've changed that to 45 percent. That's much, much larger, right? Keeping my


power at 80 percent and my significance at 5 percent, I want to keep those at least at


that. I'm using, again, a one-tailed test. And now I only need 1,567 visits per branch.


So 1 percent is much different than 6 percent, and 45 percent is way different than 6 percent,


right? So this small startup does not have the luxury


of finding little small differences in their A-B tests. They have to make big changes,


which makes sense if you're a small startup, right, because you really shouldn't be focusing


on small changes if you need massive growth, right?


So what happens at a lot of startups is they'll redesign their home page or they'll redesign


their applications, and that's a pretty big change, perfect for an A-B test for them,


right? They can actually test that. And most of them make that change anyway, so there's


no reason they shouldn't test that. It doesn't take a lot of effort to conduct an A-B test


with the software that's on the market now. All right. So sample size. So this is one


little point that I want to mention, and this is a super important one, and that sample


size calculators tell you how many subjects but not which subjects should be in your experiment.


So you can't just blindly run your experiment for that many people and then, you know, shut


it off and say, okay, we're good. You have to actually worry about who is in your experiment.


And so this is called sampling, right? You want to choose who is in your experiment.


And my first note to you is that sampling can be really hard. It can be very easy if


you have a very simple site and your traffic is all the same coming in and it never changes.


Great. I'm very happy for you. You're going to have a very easy time A-B testing. If you're


doing a lot of marketing and you're sending a lot of campaigns, it's not going to be that


simple. So you have to consider things like weekday


traffic versus weekend traffic. So what happens if your sample size calculator gives you a


sample size that would, you know, probably take you about two days in weekday traffic?


That experiment is not going to account for the differences in weekend traffic. What I


see at my current company is that weekend traffic, our conversion rate bumps way up.


I don't know why it happens, but it does. And so if I'm not taking into account those


weekend days, then I'm not going to get great results, right? Because I could start my experiment


for just those two weekdays, stop it, deploy the change, and then I don't experience the


actual change that I observed in the experiment. And you experience something that's called


regression to the mean, which is where you expect a certain difference in your conversion


rate once you deploy the variation, but then it comes back down. And that's because you


didn't account for all the traffic. You didn't run it for a weekend, right?


So what we do then is you can actually just bump up your sample size to higher than you


wanted and let's say you run the experiment for two weeks or one week.


There's also the type of source that's coming into your Web page. It could be, you know,


campaigns. So you could be sending a lot of Google ad traffic in, and that's very different


than organic traffic. I think we all know that.


But what happens, you know, it's not often the case when the person that's conducting


the A-B test is, you know, very, sits right next to the person that does the marketing


campaigns and the guy that does the marketing campaigns all of a sudden shuts off the traffic


for a campaign because he's not satisfied with the quality of the traffic or the company


doesn't have the money. All of a sudden the sampling, the sample for


your experiment changed drastically, and that's going to affect the results.


So you have to like really, really monitor these things.


And there's also like returning versus new visitors, right?


That's, they're very different as well. And there's a million different segments of


traffic. So I think you guys just really, really pay attention.


I think the first natural question when you're going through an A-B test is where should


I test? Well, here's an example for you guys.


I get a lot of questions about A-B testing, and somebody came up to me once and said that


it was an e-commerce Web site, so this is the example that I'm using.


It wasn't this actual company, though. But they said that they were running a lot


of experiments on the home page and they would get statistically significant results, but


then they wouldn't see any change in the conversion rate for purchases, right?


So they would see people adding the product to their cart, and they would see people,


you know, visiting more product pages, and that's all great, but it wasn't contributing


to the actual lift in purchases. This is a hard problem, but I'm going to show


you how to get around that. Let's consider Zappos.


It's just the first Web site that came into my mind.


I've never actually talked to Zappos. I'm not giving you any secrets.


I don't know anything about the company besides I buy shoes from there.


So here's their home page. And let's just make some assumptions, right?


The home page sometimes is the most visited page on the Web site.


Sometimes it's not. Sometimes you're sending, like, campaign traffic


to another page, and sometimes that will be the more visited page.


But let's just assume on this Web site it's the most visited page.


It has relatively few conversions if people start on the home page and then check all


the way out, right? So if we started our experiment there, we


could assume that there would be relatively few conversions.


So let's go through the sample size calculations for this.


Let's assume a 2.5% conversion rate and about 100,000 monthly unique visitors.


I'm sure Zappos has way more than that. I'm just making these numbers up.


So if we put this into the sample size calculator, we're going to say we'll run this experiment


for one month. That was 100,000 visitors.


Our conversion rate is pretty low because it's the home page.


Let's put it at 2.5%. Let's put power because we're using good


standards at 80%, significance at 5. Also, just to mention, a 95% confidence level


is the same thing as a 5% significance level. And we're just going to do one tail.


Our MDE is 10%. It's not bad.


Not horrible. Like the small startup had 45.


This company has 10. So now let's look at a different page on


the Web site. This is the checkout page.


So we've got a nice little John Varvatos Oxford in here.


When I was doing this, I noticed that I had a $50 discount, so that's cool.


But here's the checkout page. This probably has some different stats behind


it, right? So it has probably much lower traffic because


not everybody that visits the home page ends up adding something to their cart and going


through the checkout screen. But the interesting thing is it's going to


have a relatively high conversion rate. Because if we start our experiment here, we


would expect that more than 2.5% of the people complete the checkout process.


So the numbers are going to be a little bit different.


Let's assume about a 50% conversion rate. I have no idea what Zappos' conversion rate


is. I'm just assuming this. It's got to be higher than the home page.


And let's assume that only 10,000 of those 100,000 people that made it to the home page


actually make it to the checkout page. What happens in this instance?


Okay. So we run the experiment for one month, 10,000


visitors. Conversion rate is 50.


Power is at 80. Again, we're using good standards.


Significance is at 5. We've got one tail.


And now our MDE is 5%. That's a pretty big difference, right?


It's going to be much, much easier for you to get a winning variation if your MDE is


5% than it is 10%. And so my message to you there in that little,


you know, arithmetic is that we just cut the MDE in half.


For some reason when I think that I did a good job, I think of this emoji, so I put


it right there. But that's what that means to me.


It's like, yes, we did a good job there. Our MDE is now lower.


And since we're testing on the checkout funnel, we'll likely see that change when we actually


deploy it. So the message here is if you're new to A,


B testing or even if you've been doing it for a while, start by focusing your A, B test


on the last step in your conversion funnel and you're going to get much more, you're


going to get much better results that you can take and report up to your boss.


It's not going to be, hey, I got more visits to this page but it didn't translate into


more purchases. It's going to be, hey, I increased our purchases.


All right. So the next question is probably what should


I test? Well, this depends on your MDE and the time


that you want to run the experiment. So I talked a lot about what you should test


or what we tested in my last presentation at MozCon.


There's the bit.ly link. If you want to look at it, feel free to take


a look. It's got a lot of good examples in it.


I'm not going to cover too much of what you should be testing because that was the last


presentation. But I also have another presentation that


you can look at that I gave at conversion conference this year, and that's called going


beyond the low-hanging fruit. And this is sort of a little bit more of advanced


practice A.B. testing. So you can visit that bit.ly link if you want


to see that. But I will, I think the one message I want


to leave you guys with as far as what you should test is that you should be fearless.


You've undoubtedly seen a lot of results from other companies and written those down and


been like, oh, okay, so they changed their font from black to green and that gave them


a 10% increase in the conversion rate, so it must give me a 10% increase in the conversion


rate. That's not always true.


Websites are different. They have different visitors.


They have different sources of traffic. So don't just assume that what you saw from


what we did on the Obama campaign or what you saw they did at Etsy or wherever, that's


always going to work for you. You should test that.


And so here's my sort of be fearless test that we ran at Optimizely.


Optimizely has a fearless mentality inside the company because we are an A.B. testing


company. It makes sense for us to be.


And so, you know, that's just one of our principles. And so when I got there, I wanted to run this


experiment just to see what would happen. And I called this terse versus verbose.


Terse meaning, you know, obviously that there's really no content there.


And then verbose meaning you're putting a lot of content in front of the user.


So here was our home page. It's sort of hard to see, but you don't really


know the details of it. We have this home page right here that has


on the left, that was our control when I got to the company, and it has a lot of content


on it, right? We've got which customers are using it.


We've got a quick little benefits section. We've got our press mentions.


And then we've got the footer in both. But what I did is I ran a variation that took


out all of that content because I felt like we were asking the user to focus on too many


things. And so on the left is terse.


On the right is verbose. On the left is verbose.


On the right is terse. And so the result for this was a 31% increase


in leads. That's great.


You don't always come across a 31% increase in leads.


This is something you might want to try at your company.


I think a lot of people have blogged about this.


People have blogged about doing the opposite and seeing the opposite effect.


That's really great. But you should be testing it.


Don't just assume that what I did at Optimizely is going to work for your company.


So that's just an example of being fearless that I want you guys to focus on.


I think it's being fearless because what I did there is I removed everybody's content


off the home page. And obviously there was a lot of stress within


the organization. The marketing team was like, wait, wait,


who's going to see the events now? We have all these events.


And every person in the company was like, oh, my content is being removed.


And then once the experiment was done, I showed them, okay, we got a 30% increase in leads,


and now this is why our home page looks like this.


But it's sort of being our fearless mentality. Oh, one thing that I want to mention is that


we got a very, very strong result in that experiment. We got a 31% increase in leads


at the 99.9% confidence level. It doesn't get any stronger than that.


All right. So now I want to talk about some stuff that


you can do before your testing. And not just that you can do, that you actually


should be doing. The first thing I want to talk about is goals.


You really should be measuring as many goals as possible.


And that means micro goals, which a lot of people I think don't measure just off the


top of their head, but you should be paying attention to these things.


And I'm talking about things like form field errors.


So if people are confused by your labels for your form inputs, you're going to see that


when you start tracking form field errors. Maybe you're asking them, like we did on the


Obama campaign, to enter credit card information. And it's really difficult for people to enter


credit card information, because it's like 16 or 15 digits, and human brains just don't


work like that. So you won't know where the pain points are


in your form unless you're measuring these things.


And you won't know how your variation affects those unless you're measuring those things.


So it could be that you run a variation that maybe doesn't have any effect in the conversion


rate, and then it actually increases form field errors, and then you deploy that, but


you didn't know that it increases form field errors, and you just made your form harder


to use. So you want to know these things.


You can measure things like time on page. And then there's the macro things that I think


everybody just measures off the bat. And, you know, if it's an e-commerce site,


that's like purchases or revenue, the thing that you're actually trying to optimize for.


You should always choose a primary goal. All of the math that we did earlier was focused


on MDE, but that's, you know, it's all based off of the baseline conversion rate.


That's for a certain goal. All goals have a different baseline conversion rate.


The conversion rate for purchases is going to be much different than the conversion rate


for form field errors. So you definitely want to be measuring both


of these things. But your experiment length and your sample


size should all be calculated off your primary goal.


Or you may choose otherwise, but just make sure you're measuring a lot of things.


And my other message to you is don't forget about down the funnel goals, like repeat purchases


or on the Obama campaign a real key for us was saving your payment information.


So after you donated, we wanted you to save your payment information.


Because the trick is that even though you don't make a change on the conversion page,


which is probably like a checkout page if we're thinking about an e-commerce website,


it can affect the follow-up action. Maybe you ask somebody to save their payment


information. Even though you're not making any changes


to the save your payment information page, you can actually affect it from the page before.


So you definitely want to make sure you're measuring everything that you possibly can.


So I want to show you why you should do this, right?


And this goes along with getting ideas to help you experiment with.


This is an example of success versus submit on a form.


So you can't really see this very well, but this is a screen shot of the Optimize the


Results page for an experiment that we ran, which I'll show you screen shots of in a minute.


But this is an experiment we ran on our conversion funnel.


The numbers that I've highlighted here, I'm just going to give you the original conversion


rate. It's 13.72%.


It's not bad. I'll take it. But what's interesting is this goal is for


pricing dialogue submit success. And that's just internal language for us, meaning that


the person actually purchased the product. So they went through and this is great.


This is a good customer or good visitor, I should say.


And then we have the next goal that's pricing dialogue submit.


So they actually submitted the form, but they didn't hit the success.


And this has a 17.15% conversion rate. That's a big difference, right?


You would want everybody that submits your actual payment form to get into the product


and purchase it and you want a charge to go through.


Well, there's a 25% difference there. So just by running this experiment, I now


have an idea where I can increase our purchases by 25%, right?


I can just go into that form field, add in all the validation errors, see which one has


the highest, and then slowly start attacking those.


If you go to your boss and you say that you increased conversions by 25%, you're going


to be a hero. So this is a very easy way that once you start


looking at all of your goals, you're going to get an idea of where the pain points are


in your funnel. So another blue star for you guys.


Measure as many goals as possible for every experiment.


There's no real cost besides the time investment, and that gets better in the long term, to


measuring a lot of goals, right? Just put as many on there as you can.


Sometimes I run experiments that have like 25 goals.


It's a lot of data, but you might as well have it.


The next thing I want to talk about is something that I do at Optimizely, and I actually just


started it at Optimizely. I wasn't doing it before.


But I call it a testing standards document. And the things that this testing standards


document covers are for every page in our funnel or for multiple funnels.


So like, say, the home page or our ads landing pages, I document all of the standards that


we need to conduct a test. So I provide in there the three-month monthly


average of unique visitors, because that's important to calculate sample size.


I put the stopping conditions in this document for each page that we're going to run an experiment


on. So that's generally speaking the sample size.


I write all of the goals that we usually measure for that page.


And I include for each goal, I put the baseline conversion rate.


I put the minimum detectable effect so that you can see that maybe your primary goal has


a much different MDE than like form field errors. And I put the visits per branch that


you'll need for the experiment. And I put the baseline conversion rate over


time. So you see each month I update this document


and you can see what's been happening to the baseline conversion rate each month.


And I just produced this one in Google Docs that you guys can download.


Here's the bit.ly link to it. It's open to the public.


You guys can comment on it if you have feedback. But this is the bit.ly link for, it's just,


you know, a basic template for you guys to get started with this.


It's almost an exact clone of what I'm using at Optimizely.


So another step in the experimentation process, and this one is super, super important, is


quality assurance. And I don't think, at least when I've seen


people conduct their experiments, I don't think that I really see many people focusing


on this. But it's super important.


You want to make sure that there's no bugs in your variation, because what happens if


you deploy your variation and you're testing it against the control and then all of a sudden


you see that the variation loses, but it's not because it's not better, it's because


you have deployed a bug in that variation, that's super frustrating, but likely you won't


know it until you go through the quality assurance phase.


I've definitely run experiments before where I expected something and got something very


different in return and gone back and looked at the variation and found bugs in it.


And it's a waste of time, because we've just now ran that experiment for, like, let's say


two weeks, and then there were bugs in the variation, so our results were not authentic,


and then we have to run the experiment again. It's not a good situation to be in.


Obviously you're not going to be able to catch every bug, but you should at least be trying


to find some. So definitely make sure that there's no bugs


in your variation. Equally as important is to make sure that


there's no bugs in your control. The same thing can happen with your control.


Maybe you release the variation and it fixes a bug that was in your control, and that's


the reason you get the huge increase in conversions. It's not going to hurt you in this case because,


you know, the variation has a higher conversion rate and you might deploy that and that's


nice, but you want to know these things. You want to know because the way that you understand


the experiment results will be different if there was a bug in the control that contributed


to it losing. You want to make sure that tracking works


correctly. There's a number of different bugs that could


be happening in your experiment, but one of the biggest things is that I've seen is that


sometimes your tracking is not working correctly. So when you hit an actual conversion event


in your conversion funnel, maybe it doesn't send that off to your A B testing platform


and maybe it doesn't work in Internet Explorer 10 or something, some obscure browser, and


that accounts for 30% of your traffic. You're missing out on 30% of your conversions for


the variation of the control there, and that's going to be really, really dirty results.


So definitely put a decent amount of time into quality assurance.


One other thing that I want to talk about before you start your experiment is eliminating


bias. Bias can happen on many, many, many different


levels, but I found a really good way to sort of prevent this.


And when I talk about bias, what I mean is that whoever comes up with the experiment


idea is the one that has an attachment to the experiment idea, right?


So I came up with the idea. I wanted to win.


I've invested time and energy and I sold the variation to everybody, meaning that, you


know, I argued for its prioritization and then it went live and I'm very interested


in the results. But it turns out that in that sort of situation,


you're biased towards the variation and you might stop the experiment early and you might


say, oh, you know, we're just barely at the sample size, you know, maybe 5,000 more visits,


but it's winning by a lot, so let's stop it. That's not the way you should be conducting


an A B test. And the medical community actually does a


really good job of this. And this is where I got the idea.


It's not my idea. It's borrowed from the medical, or I should


say the pharmaceutical industry. When they're developing prescription drugs,


what they do is they do double-blind experiments. And this means that the subjects who they


actually test the drugs on are given either a placebo or the treatment.


And the subject does not know which one they got.


They just know that they need to take the pill at whatever interval.


The researchers also do not know. So when they're looking at the results, they


don't know which group got the treatment and which group got the placebo.


And this helps because that means that they're not really looking for one group or the other


to win. They have no idea, right?


And so they're literally just looking at data to call a winner.


And so this is a really important part that you guys should also be focusing on.


And what I do at Optimizely is when I start an experiment, I'll ask someone on our team


to go through and switch the variations up so that I don't know which is which and they


don't tell me. And then at the end of the experiment, I call


a winner or not a winner, and then we just proceed that way, and I'll find out later


which one was the variation and which one was the control.


Another thing similar to the template doc is an experiment brief.


An experiment brief, you want to cover a lot of things, and you want to document this and


make sure that you have a lot of information for whoever might be reading your experiment


results after the fact. Maybe you leave the company and it just being


in your head is not really acceptable. So things that you should cover in an experiment


brief are what are your hypothesis or what is your hypothesis?


You want to describe the audience for the experiment.


What URL did the experiment run on? Was it new?


Was it returning visitors? Was it logged in, logged out visitors?


Was it campaign traffic? Was it not campaign traffic?


You know, this all needs to be documented because you'll have this situation happen


to you in hindsight that you'll be like, wow, I really wish that I remember how that experiment


was set up. And if you don't document it, you're never


going to find out. You want to document which goals you tracked.


I think it's easy to think that, oh, it's all documented in my A.B. testing software,


right? I can just log in there.


But what I've seen happen is that maybe your company doesn't want to pay for that A.B.


testing software anymore, so they switch to a cheaper one, and then all of your experiment


data is gone. And so you don't have it in there anymore.


So it's just much better to store it in things like Google Docs that you'll always have access


to. You want to document the stopping conditions


like we talked about in the doc before. Before I start any experiment, I send an e-mail


out to the entire company or whoever is listening, and I have all of this information in there.


And I say the experiment will stop at, you know, 100,000 visitors, and that's when we


stop the experiment. I also put screenshots in the brief.


I want everybody to know what the control looked like and what the variation looked


like. This is a really, really key thing, because


once you deploy that variation, you're probably going to run another experiment.


So over the course of a quarter, you might run, like, seven experiments.


It's really hard to remember what the original control was or what the control was at any


part of those steps. Sometimes, you know, experiment titles are


just not enough. You need to know detailed information, like


what exactly did the control look like, because you might come up with things later down the


line like, oh, actually, I think this actually contributed to the success of the variation


and it shouldn't have been in there. Maybe there was a bug.


Maybe, I mean, it could be anything, right? So you want screenshots.


And then you also want to document the QA summary.


What did you check during QA? Look through all of this stuff.


And here's a template for you guys. Go ahead and write down that bitly link.


All right. And then after you test, I want to cover briefly


what a statistical tie is. This means that you do not have enough data


to conclude that there is a difference. It does not mean that there is no difference.


This is a problem that happens when your MDE is higher than what you can actually, lower


than what you can actually detect. And so my little blue star for you on this


is that the overwhelming majority of your experiments are going to result in a statistical


tie. This is just something that happens.


You're going to have to get used to it. It's pretty humbling because you really think


that you're going to get in there and just start, you know, causing all kinds of havoc


on your conversion rate. But realistically, most of your test results


are going to be a statistical tie. And so I want to show you what I mean by


statistical tie. So here's an example of an experiment that


we ran at Optimizely. This is the screenshot for the test that I


was talking about before where we discovered that there was a difference in the submission


and the success form rate, or conversion rate, I should say.


So on the left, you have the control. And then on the right, you have the variation.


What we did here is this is our payment screen, and we put a little bit of an emphasis on


there being not a high commitment, right? So we said that you're going to start a 30-day


free trial, and we said that you won't be charged until after the trial ends.


So it sort of lowers the commitment that you have to take when submitting this form.


And the control, it's at the bottom, but it's just not really there's not a lot of emphasis


on it, right? So we wanted to let our users know that we


want you to try the product for 30 days. We think it's so good that you're going to


love it, but if you don't, you know, you can cancel it before.


It's just a free 30-day trial. Here's the result.


These numbers are probably really hard for you to see, so I'll read them up for you.


The chance to be baseline, which is the confidence level, is at 78.4.


It's not 95%. So this is not telling us that we have a winner,


right? The improvement is at 2.7%.


So this is just like one of the most typical statistical ties that you'll see out there.


You really just get a result that doesn't tell you that there's a difference.


There's not much you can do about it. You can try to extend the length of the experiment


to lower the MDE, but what can happen is that the difference actually lowers after that,


right? Because the nature of conversion rates is


they get more accurate as you have more data. So, you know, it's happened to me in the past


where I decide to extend the length of the experiment and then the improvement goes from


2.7% to like.5%. And so I just ran the experiment for longer


and still have a statistical tie. So that's one thing you have to account for.


Or you could just say, you know what, let's cut our losses and let's move on to the next


experiment, which is usually what I do, but it depends on, you know, the experiment that


you're running. One thing that you definitely always want


to be paying attention to is retesting. And what I mean by retesting is I'll just


explain it through this example. I think this is a sort of hilarious example,


and I love to tell this story. So in the test that we ran, somebody had the


idea at the Obama campaign because they were reading a case study on high-end restaurant


menu psychology. And I don't know if any of you have noticed


this, but when you go to a high-end restaurant, they remove the dollar signs from their menu.


And it turns out that when you remove the dollar signs, people spend more money because


they're thinking about it less as money and just like numbers, right?


It's like, oh, it's just coming out of my account. Who cares? It's not money.


It's great. It works. It works on me. I spend a lot of money at restaurants.


So we wanted to test this on our users. So we ran this test, and we were measuring


revenue. That was our primary metric.


And we got a 36% increase in revenue. Show of hands, has anybody seen a 36% increase


in revenue? No.


There's like no hands up. Yeah.


So our reaction was like, what? Like, we are geniuses.


We just saved the entire campaign. We increased revenue 36%.


I cannot wait to tell my boss, this is amazing. My whole life is fulfilled right now.


And then we started thinking about it a little bit more, and we're like, wait, 36% increase


in revenue? Like, that's kind of hard to believe.


If I'm going to go tell my boss that I raised revenue by 36%, I want to be dead certain,


because I don't want to look like the guy that was like, hey, revenue went up 36%, and


then they start looking at the reports and revenue didn't go up 36%.


I don't want to be that guy, right? So we ran the test again.


The second test was a statistical tie. So at this point, we're like, oh, man, that's


a bummer. We're feeling really depressed about our lives.


And we thought, well, I mean, the first result was so strong, let's just run it again, right?


Let's do a third test. We have the traffic.


That 36% is like really enticing, so let's really try to get at it.


Ran a third test, another statistical tie. So we came to the conclusion here that the


way that we conducted this test, it opened up a lot of vulnerabilities and what makes


the test results not reliable, and it's some of the standards that I was talking about


before. We weren't calculating the sample size.


We weren't using 80% power. We weren't using 95% well, we were later looking


at the results, we were looking at 95% confidence level, but we didn't calculate the sample


size using that. And so what happened to us is this situation


called regression to the mean that I mentioned before.


So it's not always the case that you have the luxury of retesting because maybe you're


that small startup and it took you a month to test that home page redesign and you don't


want to take another month of traffic, but look at your confidence levels, and if they're


not like 99.9% and if it doesn't make sense to you, you definitely want to consider retesting


it. It just takes more traffic and it's kind of


a bummer, but it's something that you should do.


Not all the time, but you're just going to have to make like a you're going to really


have to think critically about that and make a decision.


Another thing that is really helpful for getting buy-in for an experimentation program is sharing


your results. What we do at Optimizely is I will send an


e-mail, like I said, to the entire company and I'll tell them before an experiment is


starting. That you have to be really careful, though,


with because you don't want the subjects coming into your Web site to know that they're going


to be in an experiment. So we have this understanding at Optimizely


that we don't talk about our experiments until they've been like okayed to talk about.


So you certainly don't want to be tweeting like, hey, just started a home page test,


go check it out, because those people now know that they're in an experiment and you're


not going to get results that are reliable. But I share the experiment with the entire


company and sometimes when I have enough time I'll set up a voting form and they can go


in and they can vote on which variation they think is going to win.


People get super excited about that because it sort of includes them in the process.


And then I have this scoring board that says who has the most points, who got most things


right. And there's this very like complicated algorithm


that we've made up to calculate scores. It's like, you know, depending on how many


groups are in the experiment, if there's like five groups and you get the winner right,


then you get a lot of points. If there's only two groups and you get the


experiment right, you get much less points. And if it was your idea, so people can submit


ideas to us, then that's great, too. And if your experiment gets actually run on


the Web site, then you get some points for that.


If you if your variation actually wins, then you get even more points for that.


So everybody's really into this like voting and score board.


And so this is a great way to get buy-in. But even after the experiment is done, I reply


to that same e-mail and I tell everybody what the experiment results were.


And sometimes, you know, there's bugs in the experiment or there's a problem with the implementation.


And I always e-mail that out so that everybody knows that A-B testing is not as easy as just


like pressing a button, right. It's not really that easy.


You have to think critically about it. And so everybody finds this information really,


really helpful. And I found that it really helps get buy-in


for the testing program. So my along with sharing, my blue star for


you on this point is to always, always record detailed experiments in an archive.


Sending them out to the company in an e-mail is not the best way because it's hard to search


your e-mail. And so what you really want to do is use another


template that I put together for you guys. In your experiment archive, here are the things


that I cover. The experiment archive from the Obama campaign


is like a massive document because we ran over 500 experiments.


And there's like literally 500 rows in this Google spreadsheet document.


And it has, I just call it like the key information to the world, right, because it has so much


information in it that I use to be successful at any test that I'm running that it's just


an invaluable document to me. If that ever got like eaten in Google servers,


I don't know what I would do because it has so much information in it.


But here are the things that I record in the experiment archive.


I want to know the experiment date because I won't always remember that stuff.


And maybe I don't have access to my A.B. testing platform.


Or maybe somebody deleted the experiment on accident.


Believe it or not, that happens. You want to know the audience, right?


What URL did you start the experiment on? Who was the audience?


What kind of traffic was it? Was it campaign traffic?


Was it not? We've been talking about these concepts and


it's important to document them. Again, you want detailed screenshots.


You should be taking these before you start the experiment because it's really hard to


do screenshots after. Like six months down the line, you're not


even going to remember how to produce that control and how to produce that variation


because you've probably iterated on it so many times that it's going to be impossible


and then you're just going off of memory. And human memory is not something you want


to be relying on when you're talking about the bottom line of your company.


You want to document what the hypothesis was in this document so that not only do you have


a date telling you what the time the experiment started, it will tell you what it was.


You want to record the results. So was your hypothesis correct?


Was it incorrect? You know, what was the lift that you observed?


Was it significant at what level? Was it the 95% confidence level?


Was it 99.9? Was it 75?


Was it a statistical tie? Was it not?


These are things you want to document because the screenshots won't always tell you that.


One thing, though, that I want to throw in for screenshots is that you want not only


screenshots of your control and your variation, but also screenshots of the results on your


experiment. So when I finish an experiment at Optimize,


I actually screenshot the Optimize the results page.


But no matter what platform you're using, screenshot that page so that if you lose access


to that platform in the future, you still have the results, right?


Or maybe the experiment gets deleted, like I said.


You don't want to be in that kind of situation because the chances are that you're going


to run a lot of experiments that you really can't remember this stuff.


And somebody is going to come up to you and say, hey, remember that experiment that we


ran where you changed this to that and it was a winner?


And you're going to be like, no, I don't remember that.


And that happened to me all the time. I mean, if you're running 500 experiments,


there's no way I can remember the details of 500 experiments.


So on the Obama campaign all the time, people would come up to me and say, hey, have we


tested this? And I'm like, psh, I don't know.


Search the archive. And that's a great way.


And you also want a link to the experiment if possible.


So for every experiment that we run, there's a link to the actual experiment.


And it will open it up in Optimize or whatever platform you're using to look at the experiment


results. And that just makes that a little bit easier


so that somebody doesn't have to go into the interface and actually search for the experiment.


And the last thing that I include is just to sort of provide a fail safe in the future


if I ever want to look at the raw data again, is I provide a link to the results CSV.


It's a comma separated value document. Most A B testing platforms will allow you


to export the results in this format. And if they do, you should absolutely be


taking advantage of it. What I do is I store all this stuff on an


Amazon S3 bucket. It's really aimed towards developers, but


you don't have to be a developer to use this. There's a GUI, which is just basically a website,


that you can log into and create an S3 bucket. And then there's a drag and drop interface


and you can put all of this information in this document.


And then it's nice because it gives you a URL to visit it.


But you could also store these things in Dropbox or wherever you want to store them.


And here's the template for that. It's just a Google spreadsheet.


It's at this bit.ly URL. It's open to the public.


And I also turn comments on. So if you guys have any feedback or something


doesn't make sense, please comment on it and I'll get right back to you on that.


So that's all that I have for my presentation. I really hope that was helpful for you guys.


And I think now we're on to Q&A.


Thank you. Awesome job, man. Glad you came back. Lots of information here. So if people


were not able to write down all those bit.ly links, your deck will be available afterwards


and they can download all that stuff, right? Okay.


So for the person just getting started with A B testing, this is a lot. I might be intimidated


by some of this. Like I am going to screw it up. Is it okay for me to go ahead? And


what dangers do I need to look out for? Well, I mean, there's certainly you have to


think about a lot of this stuff, right? Because if you are not calculating your sample size,


and let's say you just pause your experiment to when it comes to significance, which a


lot of people do, it's not the right way to run an A B test. One of those bit.ly links