A Novel Approach to Scraping Websites

the H1 from the page, which let me access its contents, which now printed down there


A Novel Approach to Scraping Websites

in the console as well.


A Novel Approach to Scraping Websites

And then I wrote a different line of code to alter the HTML of the page.


A Novel Approach to Scraping Websites

I put a yellow dashed box around the same element, the H1, to show that that is working.


A Novel Approach to Scraping Websites

The point here is that you can write or paste code directly into the console and it will


A Novel Approach to Scraping Websites

run on the site.


A Novel Approach to Scraping Websites

But if you want to run some code more than just once, you're going to need something


A Novel Approach to Scraping Websites

easier than pasting it into the console every time.


A Novel Approach to Scraping Websites

And this is where I love using JavaScript bookmarklets.


A Novel Approach to Scraping Websites

I just looked this up and I actually published my first blog post about JavaScript bookmarklets


A Novel Approach to Scraping Websites

11 years ago today.


A Novel Approach to Scraping Websites

I'm still a fan.


A Novel Approach to Scraping Websites

Basically, you can create bookmarklets that don't point to URL, but instead run a bit


A Novel Approach to Scraping Websites

of code.


A Novel Approach to Scraping Websites

You do this by starting the URL field of the bookmark with JavaScript colon followed by


A Novel Approach to Scraping Websites

the code you want to run.


A Novel Approach to Scraping Websites

It means you can take little code snippets, store them in a bookmark and run them anytime


A Novel Approach to Scraping Websites

you want at the click of a button.


A Novel Approach to Scraping Websites

And it's great for bits of code that are never going to change again, though it can be tricky


A Novel Approach to Scraping Websites

to share code in this way.


A Novel Approach to Scraping Websites

So my third option, while slightly more involved, is a technique that I'm going to use today.


A Novel Approach to Scraping Websites

And again, we're going to create a JavaScript bookmarklet.


A Novel Approach to Scraping Websites

But instead of putting all our code in there, the bookmarklet is only going to do one thing.


A Novel Approach to Scraping Websites

It's going to create a new script tag on the page that imports a JavaScript file from elsewhere.


A Novel Approach to Scraping Websites

You can store that file on your own server, on Dropbox, on GitHub or anywhere else.


A Novel Approach to Scraping Websites

And then have a bookmarklet like this that adds it to the page as if the site itself


A Novel Approach to Scraping Websites

had put it there.


A Novel Approach to Scraping Websites

Your code now has all the permissions that native code written by the site's developers


A Novel Approach to Scraping Websites

would have.


A Novel Approach to Scraping Websites

So in the rest of this presentation, I'll focus on the code we will write in that JavaScript


A Novel Approach to Scraping Websites

file that you inject into the site using a bookmarklet and then we'll run from there.


A Novel Approach to Scraping Websites

I have three examples.


A Novel Approach to Scraping Websites

And I'll start with that challenge that was posed to me earlier.


A Novel Approach to Scraping Websites

How could we compare product stats from g2.com?


A Novel Approach to Scraping Websites

Well, I suggest starting with pseudocode or rather just writing out the process you would


A Novel Approach to Scraping Websites

go through if you were doing this manually.


A Novel Approach to Scraping Websites

So we might say that we have a list of products to compare.


A Novel Approach to Scraping Websites

For each product, you'd search for it on the front page of the site.


A Novel Approach to Scraping Websites

Then you'd look at the search result page, find the top results, then you'd visit that


A Novel Approach to Scraping Websites

page which is all about the product and you'd grab the ratings from the top section.


A Novel Approach to Scraping Websites

And while you're there, you can grab the specific ratings that g2 has about ease of use, quality


A Novel Approach to Scraping Websites

of support and setup from further down the page as well.


A Novel Approach to Scraping Websites

And then you'd put them all into some kind of table as an output, maybe in Excel or Google


A Novel Approach to Scraping Websites

Sheets or whatever you want.


A Novel Approach to Scraping Websites

So let's turn that into code.


A Novel Approach to Scraping Websites

And one thing I noticed that again, g2 already has jQuery installed.


A Novel Approach to Scraping Websites

So all the fancy features that I like using are ready to go.


A Novel Approach to Scraping Websites

Now here's the results page when you search for Zendesk chat.


A Novel Approach to Scraping Websites

I'll zoom in and when we look at the URL, we realize it's a really straightforward address


A Novel Approach to Scraping Websites

with a parameter called query.


A Novel Approach to Scraping Websites

So we know that if we replace the value of that query, we'll get to the g2 results page


A Novel Approach to Scraping Websites

for any product name search.


A Novel Approach to Scraping Websites

And we wanna look at the top result here.


A Novel Approach to Scraping Websites

That's when we fire up Chrome DevTools, inspect these elements and we find that each product


A Novel Approach to Scraping Websites

listing has a div with a class called product listing.


A Novel Approach to Scraping Websites

In each of those is a difficult product name and a difficult star rating and a link to


A Novel Approach to Scraping Websites

the specific page about the product itself.


A Novel Approach to Scraping Websites

So the first thing we're gonna do is crawl that search page, find the top results and


A Novel Approach to Scraping Websites

grab those bits of information.


A Novel Approach to Scraping Websites

Here's some code that looks a bit intimidating at first glance, but I'll show you three cool


A Novel Approach to Scraping Websites

features we can use to write powerful code really fast.


A Novel Approach to Scraping Websites

First I just created a variable up at the top there called search query that I set to


A Novel Approach to Scraping Websites

be Zendesk.


A Novel Approach to Scraping Websites

This was just gonna be useful while I tested things.


A Novel Approach to Scraping Websites

Then I used this jQuery function that we're gonna rely on again and again.


A Novel Approach to Scraping Websites

It's called get and it literally sends a get request to any page.


A Novel Approach to Scraping Websites

You give it a URL you wanna fetch and then it gives you a variable containing the entire


A Novel Approach to Scraping Websites

HTML of the target page.


A Novel Approach to Scraping Websites

You also add a function that processes the HTML however you want.


A Novel Approach to Scraping Websites

We're not navigating to that page, we're not changing anything visible in the browser,


A Novel Approach to Scraping Websites

we're just requesting the page in the background and sticking it in a variable.


A Novel Approach to Scraping Websites

Now let's take a moment to appreciate the power of running this as code on the site.


A Novel Approach to Scraping Websites

The website is making the request so we have no cross site scripting issues.


A Novel Approach to Scraping Websites

We have no authentication challenges.


A Novel Approach to Scraping Websites

All the cookies that the site has set in your browser are available and working.


A Novel Approach to Scraping Websites

So you get that HTML back in the blink of an eye ready for you to parse and investigate


A Novel Approach to Scraping Websites

however you want.


A Novel Approach to Scraping Websites

And then we can do things like use the.find function.


A Novel Approach to Scraping Websites

Similar to earlier, this lets us use the CSS selectors that we already know and love to


A Novel Approach to Scraping Websites

find things in the page that was returned.


A Novel Approach to Scraping Websites

Here I've started off by just finding the H1 of that page and writing it to the console.


A Novel Approach to Scraping Websites

But then I use.find again to find the very first listing on the results page and then


A Novel Approach to Scraping Websites

use it again within that element to collect the URL, the product name and the rating of


A Novel Approach to Scraping Websites

the very first listing.


A Novel Approach to Scraping Websites

You can see here that once we have an element we care about, we can then use the.text function


A Novel Approach to Scraping Websites

to tell us what the text of each of those elements is.


A Novel Approach to Scraping Websites

Let's grab this code and run it for real in the console.


A Novel Approach to Scraping Websites

There's my code.


A Novel Approach to Scraping Websites

I pasted it into the DevTools console while we're anywhere on G2 and it outputs a couple


A Novel Approach to Scraping Websites

of things.


A Novel Approach to Scraping Websites

Let's zoom in.


A Novel Approach to Scraping Websites

The first line is the H1 of the results page that I asked it to output.


A Novel Approach to Scraping Websites

The second is that object we created and it's correctly pulled in the URL, the name and


A Novel Approach to Scraping Websites

the rating of the product.


A Novel Approach to Scraping Websites

We can do a similar iteration now.


A Novel Approach to Scraping Websites

We can take the URL of the actual product page, do another GET request.


A Novel Approach to Scraping Websites

We can then look for those three metrics of how users review the product and extract the


A Novel Approach to Scraping Websites

numbers.


A Novel Approach to Scraping Websites

Let's look at that code.


A Novel Approach to Scraping Websites

Here it is, very similar.


A Novel Approach to Scraping Websites

We start with a GET request, but this time we're requesting the product pages URL that


A Novel Approach to Scraping Websites

we found earlier.


A Novel Approach to Scraping Websites

Then we use those metrics and we find them, they're not surrounded by any clean or semantic


A Novel Approach to Scraping Websites

markup so we have to hack our way around.


A Novel Approach to Scraping Websites

We use the contains pseudo selector to look for an appropriate div that mentions ease


A Novel Approach to Scraping Websites

of use or whatever else.


A Novel Approach to Scraping Websites

We traverse up and down the DOM tree a little bit, find the element we really want and grab


A Novel Approach to Scraping Websites

the score of it again using the.text function.


A Novel Approach to Scraping Websites

I pasted that code into the console.


A Novel Approach to Scraping Websites

Let's see what it gives me.


A Novel Approach to Scraping Websites

Excellent.


A Novel Approach to Scraping Websites

It's created this object called rating details that has those three scores.


A Novel Approach to Scraping Websites

So we can get data about a single product.


A Novel Approach to Scraping Websites

How do we output it?


A Novel Approach to Scraping Websites

Well, just like reading data from the page we're on, JavaScript lets us change the content


A Novel Approach to Scraping Websites

of the page and jQuery makes it even easier.


A Novel Approach to Scraping Websites

.text can change the text of an element,.html can change the entire HTML of an element and


A Novel Approach to Scraping Websites

.append will add HTML to the end of an element.


A Novel Approach to Scraping Websites

And we're going to combine these for our purposes.


A Novel Approach to Scraping Websites

Before we do any crawling, we're actually going to use the HTML function to overwrite


A Novel Approach to Scraping Websites

the entire contents of the page.


A Novel Approach to Scraping Websites

By running this code, we're replacing the HTML element of the page, which is absolutely


A Novel Approach to Scraping Websites

everything with the code.


A Novel Approach to Scraping Websites

This gives us a text area, a go button and a table with some headings.


A Novel Approach to Scraping Websites

When we run this, here's what the front page of G2 turns into.


A Novel Approach to Scraping Websites

And I know what you're thinking.


A Novel Approach to Scraping Websites

First, you're saying, wow, that's really powerful, but also, my God, that's ugly.


A Novel Approach to Scraping Websites

And you're right on both accounts.


A Novel Approach to Scraping Websites

This is ugly.


A Novel Approach to Scraping Websites

It's raw, unstyled HTML.


A Novel Approach to Scraping Websites

But this is for you and not for anyone else to see.


A Novel Approach to Scraping Websites

So does it matter?


A Novel Approach to Scraping Websites

This is going to help you get your job done fast and efficiently.


A Novel Approach to Scraping Websites

Who cares what it looks like?


A Novel Approach to Scraping Websites

Now, I'll also contradict myself here just a little, because later on, I will share something


A Novel Approach to Scraping Websites

with you that is a little better looking and a little more usable.


A Novel Approach to Scraping Websites

But for now, once we have a table in there, we can just put data into it.


A Novel Approach to Scraping Websites

What I've done here is you've got that input text area where you could type a load of product


A Novel Approach to Scraping Websites

names.


A Novel Approach to Scraping Websites

I've taken the contents of the text area using the.val function, and then I split it up


A Novel Approach to Scraping Websites

into an array by separating each new line.


A Novel Approach to Scraping Websites

Now we have an array of the product names you entered, and then I use this.each function


A Novel Approach to Scraping Websites

from jQuery.


A Novel Approach to Scraping Websites

It makes it easy to loop over that array one item at a time.


A Novel Approach to Scraping Websites

Each term that was entered, we use the append function to add a new row to the table.


A Novel Approach to Scraping Websites

And then we wrap all of this up in a click function.


A Novel Approach to Scraping Websites

This code selects the start button that we added and says, when it's clicked, here's


A Novel Approach to Scraping Websites

what to do next.


A Novel Approach to Scraping Websites

Which in our case is fill that table up with the names we entered and then start crawling


A Novel Approach to Scraping Websites

the site.


A Novel Approach to Scraping Websites

After that, populate table function runs.


A Novel Approach to Scraping Websites

Here's what we get.


A Novel Approach to Scraping Websites

Our ugly-ass table is now an ugly-ass table with five rows in it.


A Novel Approach to Scraping Websites

After we've done that, we kick off another each loop.


A Novel Approach to Scraping Websites

This time we'll loop over a set of elements from the page.


A Novel Approach to Scraping Websites

We'll select the table's rows, and for each row we'll run some code, which is basically


A Novel Approach to Scraping Websites

the code I showed you earlier that did the scraping.


A Novel Approach to Scraping Websites

So now we have all of this wrapped up in a JavaScript file.


A Novel Approach to Scraping Websites

We have a bookmarklet that injects the file into the page.


A Novel Approach to Scraping Websites

So here we are.


A Novel Approach to Scraping Websites

I tried doing a screen recording of this for you, but it all happened so fast.


A Novel Approach to Scraping Websites

So I've just screenshotted things instead to slow it down.


A Novel Approach to Scraping Websites

We're on the front page of G2.


A Novel Approach to Scraping Websites

We hit our bookmarklet and the page gets replaced with this.


A Novel Approach to Scraping Websites

And like I said earlier, I've actually made this look a little bit better than unstyled


A Novel Approach to Scraping Websites

HTML.


A Novel Approach to Scraping Websites

I found a tiny CSS framework and with just one extra line in the HTML here, we can include


A Novel Approach to Scraping Websites

the external CSS file that makes the buttons and the tables look a little nicer.


A Novel Approach to Scraping Websites

But whether you spend time doing that kind of thing for yourself, it's entirely up to


A Novel Approach to Scraping Websites

you.


A Novel Approach to Scraping Websites

It's also important to note here, as far as the browser is concerned, we're still on the


A Novel Approach to Scraping Websites

same website.


A Novel Approach to Scraping Websites

We still have permission to make requests from this site.


A Novel Approach to Scraping Websites

So you type some product names into the box and you hit the go button.


A Novel Approach to Scraping Websites

And quickly the code populates the table and starts scraping.


A Novel Approach to Scraping Websites

It goes to the search result page for each product name and grabs the URL of the product


A Novel Approach to Scraping Websites

page, then it goes to the product page URL and grabs the scores.


A Novel Approach to Scraping Websites

Ten page requests happen, all the data is collected, and one row at a time it's written


A Novel Approach to Scraping Websites

back into our table.


A Novel Approach to Scraping Websites

This takes almost no time at all.


A Novel Approach to Scraping Websites

And here we are.


A Novel Approach to Scraping Websites

Now, my friend that wants to compare the scores from a bunch of different software options


A Novel Approach to Scraping Websites

can see this table with all of the details.


A Novel Approach to Scraping Websites

Have a different list of products to compare?


A Novel Approach to Scraping Websites

Click it again, run it again.


A Novel Approach to Scraping Websites

I've published this code and there are comments all over it for you.


A Novel Approach to Scraping Websites

I can't imagine many of you will have exactly the same use case and need to scrape data


A Novel Approach to Scraping Websites

out of this site, but you can use this as a template for running the same process over


A Novel Approach to Scraping Websites

almost any other site that you might want to scrape.


A Novel Approach to Scraping Websites

The code and the bookmarklets are available at oosby.com.


A Novel Approach to Scraping Websites

Now I said I had three examples.


A Novel Approach to Scraping Websites

So what else can we scrape?


A Novel Approach to Scraping Websites

Well, that leads us on to chapter two, which I call when the scraper becomes the scrapee.


A Novel Approach to Scraping Websites

And who's the biggest scraper of them all?


A Novel Approach to Scraping Websites

Yeah, you're damn right.


A Novel Approach to Scraping Websites

Let's jump straight to scraping Google.


A Novel Approach to Scraping Websites

But not just for the sake of it, let's solve a real SEO problem.


A Novel Approach to Scraping Websites

We've all used Moz Pro or a site like it to crawl our site.


A Novel Approach to Scraping Websites

Here's my Moz Pro account for my old company Distilled.


A Novel Approach to Scraping Websites

And from the site crawl, I can see that Distilled has pages under folders like events, like


A Novel Approach to Scraping Websites

blog, like resources and training.


A Novel Approach to Scraping Websites

But my question is always, how many pages does Google know about in each of those folders?


A Novel Approach to Scraping Websites

How many pages of each type have they indexed?


A Novel Approach to Scraping Websites

A site crawl cannot tell you this.


A Novel Approach to Scraping Websites

And I'd love to know this for some competitor sites as well.


A Novel Approach to Scraping Websites

How are they structured?


A Novel Approach to Scraping Websites

And how does Google understand that structure?


A Novel Approach to Scraping Websites

Here's the process I've been through in the past to answer this question.


A Novel Approach to Scraping Websites

First you do a site colon search.


A Novel Approach to Scraping Websites

Here I'm doing a search for site colon Moz.com to find all the index pages that Google has


A Novel Approach to Scraping Websites

from Moz.


A Novel Approach to Scraping Websites

And then we look through that list of results to find the unique folders.


A Novel Approach to Scraping Websites

Then we write those down.


A Novel Approach to Scraping Websites

We've got Moz.com slash learning slash blog slash community.


A Novel Approach to Scraping Websites

How many pages does Google have indexed for each of those folders?


A Novel Approach to Scraping Websites

We do another site search.


A Novel Approach to Scraping Websites

Site colon Moz.com slash learn.


A Novel Approach to Scraping Websites

And there's the number.


A Novel Approach to Scraping Websites

Four hundred and forty seven pages of amazing learning content.


A Novel Approach to Scraping Websites

We do it for each of the folders we found.


A Novel Approach to Scraping Websites

And eventually we have a list like this.


A Novel Approach to Scraping Websites

It takes a while, but we get there.


A Novel Approach to Scraping Websites

But we want to find more of the folders of the site.


A Novel Approach to Scraping Websites

So we do another crafty little search.


A Novel Approach to Scraping Websites

We do a site colon Moz.com search and now we exclude all the folders we already knew


A Novel Approach to Scraping Websites

about.


A Novel Approach to Scraping Websites

Minus site colon Moz.com slash learn.


A Novel Approach to Scraping Websites

Minus site colon Moz.com slash community.


A Novel Approach to Scraping Websites

This gives us a search result with a whole bunch more folders that we can now put into


A Novel Approach to Scraping Websites

our list and then go start the process for them again.


A Novel Approach to Scraping Websites

This is the kind of thing that is arguably at the boundary of being very time consuming


A Novel Approach to Scraping Websites

and enough that you're not going to do it by hand for a load of different sites, but


A Novel Approach to Scraping Websites

what becomes potentially very useful is you can just hit one button and get it all done


A Novel Approach to Scraping Websites

for you.


A Novel Approach to Scraping Websites

So what would the code for this look like?


A Novel Approach to Scraping Websites

Well, first we'll replace the Google page again with our code.


A Novel Approach to Scraping Websites

Next when we go to Google and run our bookmark, look, this is what we get.


A Novel Approach to Scraping Websites

An input box, the bare bones of a table and a couple of buttons.


A Novel Approach to Scraping Websites

Again, I wanted this to be a little less ugly.


A Novel Approach to Scraping Websites

So it's one extra line of code to add a CSS.


A Novel Approach to Scraping Websites

What does the actual code do?


A Novel Approach to Scraping Websites

Well, this might sound familiar second time around.


A Novel Approach to Scraping Websites

First, we get the text of the domain name you typed into the box at the top and we add


A Novel Approach to Scraping Websites

a site colon to the front of that to make it into a search query.


A Novel Approach to Scraping Websites

Then we use jQuery's get function to go and request that page on Google that is slash


A Novel Approach to Scraping Websites

search question mark Q equals query.


A Novel Approach to Scraping Websites

And again, we're on Google and as far as the browser knows, this is Google doing its own


A Novel Approach to Scraping Websites

Ajax requests to other parts of the site.


A Novel Approach to Scraping Websites

This happens all the time and there's nothing out of the ordinary here.


A Novel Approach to Scraping Websites

Once it gets the cert back, it selects all the links on the page that points to the site


A Novel Approach to Scraping Websites

loops through them with dot each and extracts the top level folder.


A Novel Approach to Scraping Websites

If it's not seen that folder before, then it adds it to the table on the page.


A Novel Approach to Scraping Websites

So for example, you enter moz.com in that box, hit the button and boom, in the blink


A Novel Approach to Scraping Websites

of an eye, here's what you get.


A Novel Approach to Scraping Websites

A list of some of the top level folders that Google has indexed pages from.


A Novel Approach to Scraping Websites

But now we need to find out how many index pages it has in each.


A Novel Approach to Scraping Websites

So we loop over the list of folders and for each one, we get another Google results page,


A Novel Approach to Scraping Websites

this time for site colon domain dot com slash folder.


A Novel Approach to Scraping Websites

That's this get request here, which grabs this page.


A Novel Approach to Scraping Websites

And as you'll see at the top there, it has the thing saying about 7,400 results.


A Novel Approach to Scraping Websites

We inspect the HTML for that and see that it's helpfully wrapped up in a div called


A Novel Approach to Scraping Websites

result stats.


A Novel Approach to Scraping Websites

So with the result of that get request, we can use dot find to find the results that


A Novel Approach to Scraping Websites

div we can use dot text to grab the text from it.


A Novel Approach to Scraping Websites

Then we use plain old JavaScript to extract the number from that text.


A Novel Approach to Scraping Websites

We write that back into the appropriate row of our table and that's done.


A Novel Approach to Scraping Websites

And we move on to the next folder until they're all done.


A Novel Approach to Scraping Websites

Now it'll be no surprise to you that running dozens of site colon queries one after the


A Novel Approach to Scraping Websites

other is likely to trigger some bot sensors at Google's end.


A Novel Approach to Scraping Websites

And honestly, your first line of defense there is just to slow down a bit.


A Novel Approach to Scraping Websites

So instead of running all these get folder count functions at the same time, I use JavaScript's


A Novel Approach to Scraping Websites

set timeout function to run these one after the other a second apart.


A Novel Approach to Scraping Websites

This might be enough to prevent problems, but even so, you might sometimes see your


A Novel Approach to Scraping Websites

code suddenly stop working and that's because Google has you pegged as a non-human.


A Novel Approach to Scraping Websites

We don't have to stress.


A Novel Approach to Scraping Websites

We're not running a load of cloud servers and churning through IP addresses or whatever.


A Novel Approach to Scraping Websites

We are a human.


A Novel Approach to Scraping Websites

Our browser has all the fingerprints of being a regular user, so we can pass a capture no


A Novel Approach to Scraping Websites

problem.


A Novel Approach to Scraping Websites

Once you get blocked, you just open up another tab, go to Google, click the recapture button


A Novel Approach to Scraping Websites

to prove you're a human, and then carry on merrily collecting the data that you want.


A Novel Approach to Scraping Websites

Remember, it's not Black Hat.


A Novel Approach to Scraping Websites

It's just faster than the manual way of doing what you were going to do anyway.


A Novel Approach to Scraping Websites

Okay, so let's see how the whole thing looks when we put it together.


A Novel Approach to Scraping Websites

Here we are in Google.


A Novel Approach to Scraping Websites

Let's go click the bookmark and great.


A Novel Approach to Scraping Websites

So it loads the form and the table.


A Novel Approach to Scraping Websites

We type our domain into the box and hit the blue button.


A Novel Approach to Scraping Websites

It immediately grabs the first set of folders from Google and then one each second.


A Novel Approach to Scraping Websites

It does another search to see how many pages Google has indexed from that folder.


A Novel Approach to Scraping Websites

When it's complete, we hit the button again.


A Novel Approach to Scraping Websites

Here's another set of folders.


A Novel Approach to Scraping Websites

And the process continues.


A Novel Approach to Scraping Websites

And another.


A Novel Approach to Scraping Websites

So what have we done here?


A Novel Approach to Scraping Websites

We've built a very simple but very useful tool that runs entirely in your browser and


A Novel Approach to Scraping Websites

collects data from Google.


A Novel Approach to Scraping Websites

There are obviously some other really time-consuming ways of getting the same data, but with just


A Novel Approach to Scraping Websites

a few lines of code, we now have a brand new SEO tool that solves a particular question


A Novel Approach to Scraping Websites

that I've had for a long time.


A Novel Approach to Scraping Websites

Again, the tool is available here if you'd like to use it.


A Novel Approach to Scraping Websites

But more than that, I'd suggest you grab the code, take a look at it, and see how you can


A Novel Approach to Scraping Websites

make it bend to your will for doing other things you might want to do.


A Novel Approach to Scraping Websites

Okay, one final example to push home some ideas of things that we can do here.


A Novel Approach to Scraping Websites

What about some very specific SEO data?


A Novel Approach to Scraping Websites

I got this Slack message from my colleague Brittany Muller the other day.


A Novel Approach to Scraping Websites

She asked me to look at the web.dev site and particularly their implementation of Google


A Novel Approach to Scraping Websites

Lighthouse.


A Novel Approach to Scraping Websites

She said the Lighthouse API was the worst and could I please scrape the site?


A Novel Approach to Scraping Websites

I said I'd take a look and exactly two hours and one minute later, I messaged her back


A Novel Approach to Scraping Websites

to say I'd come up with something fun.


A Novel Approach to Scraping Websites

Let's dive in.


A Novel Approach to Scraping Websites

This is the app that Brittany linked me to.


A Novel Approach to Scraping Websites

It's a site made by Google to support web developers.


A Novel Approach to Scraping Websites

And this page is a tool that runs a Lighthouse report on any URL.


A Novel Approach to Scraping Websites

You put in the URL of any web page and it crawls it, renders the page, and discovers


A Novel Approach to Scraping Websites

a lot of information about it.


A Novel Approach to Scraping Websites

It takes about 15 to 20 seconds to run and then it comes back to you with all of these


A Novel Approach to Scraping Websites

data points.


A Novel Approach to Scraping Websites

This is the report for the MozCon marketing page.


A Novel Approach to Scraping Websites

It has these top level metrics about performance, accessibility, best practice, SEO.


A Novel Approach to Scraping Websites

And these are all just rollups of dozens of even more detailed metrics.


A Novel Approach to Scraping Websites

For example, under performance there, we see MozCon's page scores for things like time


A Novel Approach to Scraping Websites

to first contentful paint, time to interactive, blocking time, and loads more.


A Novel Approach to Scraping Websites

Let's just focus on these top line metrics.


A Novel Approach to Scraping Websites

I thought at first that if I want to crawl this results page and get all the metrics,


A Novel Approach to Scraping Websites

but I realized that what was actually happening here when you run a report is that the page


A Novel Approach to Scraping Websites

itself was doing an Ajax request to get the data from elsewhere.


A Novel Approach to Scraping Websites

So we fire up the network tab of Chrome DevTools.


A Novel Approach to Scraping Websites

We take a look at all the network traffic from this page and lo and behold, there's


A Novel Approach to Scraping Websites

a request that takes 15 to 20 seconds to complete.


A Novel Approach to Scraping Websites

So that must be it fetching the report data.


A Novel Approach to Scraping Websites

Let's dig into that request and see what's going on.


A Novel Approach to Scraping Websites

Well, we see three interesting things from the headers on that request.


A Novel Approach to Scraping Websites

The first is that it's actually making this request to another domain.


A Novel Approach to Scraping Websites

It's this appspot.com app that's doing all the heavy lifting here.


A Novel Approach to Scraping Websites

Okay, so we know we're probably going to end up making a cross domain request.


A Novel Approach to Scraping Websites

This might cause some problems.


A Novel Approach to Scraping Websites

There might be cross site tokens and stuff at play that are going to complicate matters.


A Novel Approach to Scraping Websites

Then we see that this time it's a post request rather than get.


A Novel Approach to Scraping Websites

Okay, we just have to remember that.


A Novel Approach to Scraping Websites

And then we see down here that it's basically just sending the requested URL as the post


A Novel Approach to Scraping Websites

parameter and nothing else of interest.


A Novel Approach to Scraping Websites

That means we don't have to worry about any of those cross site token issues and potentially


A Novel Approach to Scraping Websites

means we could actually request this Lighthouse data from any site we want to, not just from


A Novel Approach to Scraping Websites

the web dev site.


A Novel Approach to Scraping Websites

Okay, so we know that to make this request, we just post that URL and send it the URL


A Novel Approach to Scraping Websites

of the page we want information about.


A Novel Approach to Scraping Websites

Let's take a look at what the service sends back to us.


A Novel Approach to Scraping Websites

Well, this is where I think I actually yelped out loud.


A Novel Approach to Scraping Websites

There's no HTML being sent back.


A Novel Approach to Scraping Websites

It's just a JSON object, perfectly structured data.


A Novel Approach to Scraping Websites

Right there under the LHR key is an array of all the metrics we want to get.


A Novel Approach to Scraping Websites

So I thought we have this easy to parse Lighthouse report that I think I can request from anywhere.


A Novel Approach to Scraping Websites

What can we do with that?


A Novel Approach to Scraping Websites

Well, what data set would be improved by having performance and accessibility data piped into


A Novel Approach to Scraping Websites

it?


A Novel Approach to Scraping Websites

What about some of Moz's tools?


A Novel Approach to Scraping Websites

Hopefully most of you have at least one Moz Pro campaign up and running.


A Novel Approach to Scraping Websites

If you don't, you can bounce your way over to Moz.com slash pro for a free trial.


A Novel Approach to Scraping Websites

Get into your Moz Pro campaign, then go down to the analyzer feature, sorry, analyzer keyword


A Novel Approach to Scraping Websites

feature, select a keyword from your campaign, and here I'm analyzing the SEO agency keyword.


A Novel Approach to Scraping Websites

When you scroll down here, you'll get to the SERP analysis table.


A Novel Approach to Scraping Websites

Now SERP analysis is typically focused on two huge traditional types of ranking factors.


A Novel Approach to Scraping Websites

The link equity of a page represented by domain authority and page authority, and the keyword


A Novel Approach to Scraping Websites

optimization of a page.


A Novel Approach to Scraping Websites

Moz has all of those next to each other in our SERP analysis table right here.


A Novel Approach to Scraping Websites

But in addition to those factors, Google has said they don't want to send people to sites


A Novel Approach to Scraping Websites

have poor user experiences, which could mean that things like slow sites or sites with


A Novel Approach to Scraping Websites

poor accessibility will suffer in the SERPs.


A Novel Approach to Scraping Websites

So let's try to include those Lighthouse metrics in this very report.


A Novel Approach to Scraping Websites

I wrote a file and created a bookmark that uses nothing but the tech that I've already


A Novel Approach to Scraping Websites

walked through in this deck.


A Novel Approach to Scraping Websites

If you go to a page like this and click it, here's what happens.


A Novel Approach to Scraping Websites

First a bunch of additional columns and cells are added to the table, along with these run


A Novel Approach to Scraping Websites

Lighthouse links in each row.


A Novel Approach to Scraping Websites

When you click one of those, it requests the Lighthouse report for the URL in that row.


A Novel Approach to Scraping Websites

Since it takes like 20 seconds to run the report, I replaced the link with a little


A Novel Approach to Scraping Websites

loading gift to let you know that something's working.


A Novel Approach to Scraping Websites

And then the data starts appearing.


A Novel Approach to Scraping Websites

We'll come back here in a second, but let's take a look at the new code I wrote for this


A Novel Approach to Scraping Websites

feature.


A Novel Approach to Scraping Websites

Remember, we navigated to a Moz campaign SERP analysis page and added this.


A Novel Approach to Scraping Websites

With an array of objects that specified the Lighthouse features I wanted, I ran a loop


A Novel Approach to Scraping Websites

over it, then added a heading and a bunch of cells to the table, and then I added those


A Novel Approach to Scraping Websites

run Lighthouse links.


A Novel Approach to Scraping Websites

When you click any of those links, we go request the data from the AppSpot server.


A Novel Approach to Scraping Websites

We're using the AJAX function here, because it's a bit more customizable than the GET


A Novel Approach to Scraping Websites

or POST functions, and it's helpful when we're working across domains.


A Novel Approach to Scraping Websites

And then when it comes back, I loop over my features once again, and I go into the JSON


A Novel Approach to Scraping Websites

response to grab the particular data points I need.


A Novel Approach to Scraping Websites

It turns out that instead of a 0 to 100 range, they're giving us numbers on a 0 to 1 range.


A Novel Approach to Scraping Websites

So I just multiply the result by 100, round it off to the nearest whole number, and slap


A Novel Approach to Scraping Websites

that data into the appropriate place in the table.


A Novel Approach to Scraping Websites

Now let's take a step back here and see what we've got.


A Novel Approach to Scraping Websites

We now have a SERP analysis tool that presents more data in one view than has ever been seen


A Novel Approach to Scraping Websites

before.


A Novel Approach to Scraping Websites

Linked data, keyword data, site performance data, and accessibility all stacked up next


A Novel Approach to Scraping Websites

to each other.


A Novel Approach to Scraping Websites

If you've been looking at the ranking sites for a particular keyword and wondering why


A Novel Approach to Scraping Websites

is this well optimized website doing so poorly, then maybe you want to run something like


A Novel Approach to Scraping Websites

this and see if there's a non-traditional ranking factor coming into play.


A Novel Approach to Scraping Websites

Maybe a site that seems strong with lots of links and good optimization is actually really


A Novel Approach to Scraping Websites

slow or not very accessible, and that might be holding it back.


A Novel Approach to Scraping Websites

So while this is really useful, I want to make two things very explicit.


A Novel Approach to Scraping Websites

Firstly, this is not an official Moz product.


A Novel Approach to Scraping Websites

This is not a feature release.


A Novel Approach to Scraping Websites

This is not something Moz built.


A Novel Approach to Scraping Websites

This isn't something that we're going to provide any support for.


A Novel Approach to Scraping Websites

And in fact, my product engineers and designers are mostly seeing this for the first time


A Novel Approach to Scraping Websites

right now.


A Novel Approach to Scraping Websites

This is an example of the kind of powerful things that you can create in two hours and


A Novel Approach to Scraping Websites

one minute just by injecting JavaScript into a site.


A Novel Approach to Scraping Websites

And secondly, let me make it really clear that I didn't need to work for Moz to do this.


A Novel Approach to Scraping Websites

I'm not using any secret Moz knowledge about our code or our infrastructure here.


A Novel Approach to Scraping Websites

I'm not using any of our APIs.


A Novel Approach to Scraping Websites

I could have written this last year when I was an SEO consultant, as well as any of you


A Novel Approach to Scraping Websites

could.


A Novel Approach to Scraping Websites

One thing you can take away here is the realization that if you think of a genius idea to extend


A Novel Approach to Scraping Websites

your favorite tools or apps, you might not need to submit a feature request.


A Novel Approach to Scraping Websites

Maybe you can just hack around with the site a bit, inject code, grab data from as many


A Novel Approach to Scraping Websites

places as you need, and create the features or datasets that are most useful to you in


A Novel Approach to Scraping Websites

your role right now.


A Novel Approach to Scraping Websites

Of course, I also want you to take away a link to this page that includes the code for


A Novel Approach to Scraping Websites

this bookmarklet.


A Novel Approach to Scraping Websites

Like I say, it's not a Moz product.


A Novel Approach to Scraping Websites

It's not supported.


A Novel Approach to Scraping Websites

For all I know, web.dev will change their code in the next 24 hours and screw up the


A Novel Approach to Scraping Websites

way that I've been scraping this data.


A Novel Approach to Scraping Websites

But for now, if you have a Moz Pro campaign up and running, you should 100% try going


A Novel Approach to Scraping Websites

to a search analysis page in your campaign and give this a try.


A Novel Approach to Scraping Websites

To wrap up, I want to reiterate the items that you can put in your toolkit.


A Novel Approach to Scraping Websites

You can run JavaScript on any site as if it was native to the site itself.


A Novel Approach to Scraping Websites

Run it in the DevTools console, store it in a bookmarklet for later use, or use a bookmarklet


A Novel Approach to Scraping Websites

to import an external JavaScript file and store that wherever you want online.


A Novel Approach to Scraping Websites

When you're writing JavaScript for this kind of work, there are some jQuery functions that


A Novel Approach to Scraping Websites

you'll want to use a lot.


A Novel Approach to Scraping Websites

You can use get or post to other URLs on the same site.


A Novel Approach to Scraping Websites

You can select elements on the current page or find them in the page that was returned


A Novel Approach to Scraping Websites

from a get request.


A Novel Approach to Scraping Websites

You can iterate through a whole bunch of elements that you find, and you can extract or edit


A Novel Approach to Scraping Websites

the individual elements of a page, whether they were there to begin with or whether you


A Novel Approach to Scraping Websites

put them there.


A Novel Approach to Scraping Websites

If you're going to start hacking around with JavaScript, don't worry about making it pretty.


A Novel Approach to Scraping Websites

If it works, then it works.


A Novel Approach to Scraping Websites

I hope I've inspired you to think about some of the things and some of the ways that you


A Novel Approach to Scraping Websites

might be able to use this for yourself.


A Novel Approach to Scraping Websites

I'll post more of this kind of thing on Twitter, so follow me there, at Rob Usby, if you'd


A Novel Approach to Scraping Websites

like more free tools as I create them.


A Novel Approach to Scraping Websites

And I look forward to you tagging me to show me what you've created.


A Novel Approach to Scraping Websites

Have a great rest of your MozCon, and I'll see you all soon.


A Novel Approach to Scraping Websites


Whatever You Do, Put Billboards in Seattle – Getting Brand Awareness Data from Google

Rich stats of secret stash media once taught me that a lot of people are intelligent, but there's this special next-level intellect that few people possess, which basically allows them to beautifully articulate complex ideas to anyone.


That person is Robin Lord.


One of my all-time favorite talks of, dare I say, the last 10 years was by Robin at Searchlove, where he so gracefully showed the power of Python and Pandas data frames in comparison to Excel.


Robin is a senior consultant at Distilled, where focusing on unusual questions and tricky answers is just part of his job description. So let's give it up for Robin Lord.


New York, 1930s. A little boy called Joe is living with his parents in Brooklyn during the Depression.


They're so poor that they have to move at least once a year, because back then your landlord would give you your first month of rent free, and that was the only way they could survive.


And yet at six, Joe decides he wants to become a lawyer. So he works and he works, and when he gets older he takes day jobs to pay for night school until he can apply for Harvard and get in.


In Harvard, he's one of the shining stars of his class. Part of the Harvard Review, peers tell stories about everyone else in lectures furiously scribbling notes, while Joe sits and listens and absorbs and comes out with thoughts and arguments that hadn't occurred to anyone else in the room.


Hiring season rolls around, and when the dust settles just two people don't have a job. Joe is one.


The firms were looking for people who are smart and driven and well educated. Joe was all of these things, but even if you jump through all the hoops they set for you, someone could take you into a room and gently explain that the firm couldn't possibly hire someone who's Jewish.


Joe ends up working for a much smaller organisation that has to take whatever work comes through the door. A lot of that ends up being things like corporate takeovers, which are seen as too uncivilised for the major firms.


I'm sure this is a surprise to no one, but halfway through the 1900s things change. Suddenly corporate takeovers are big business.


At this point Joe is one of the leading experts in this field. He helps his firm grow from having just four lawyers to over a thousand. And people will hire the firm, even if they're not going to use anyone on the staff, just to be certain there's no risk of being in an important meeting and finding themselves arguing against Joe.


Joseph Flom died, named partner in the fourth largest law firm in the world. And you could be asking yourself, why is he telling us this story? But bear with me, my point's coming.


The thing to remember about Joe is yes, he was smart, he was driven, but most importantly, he was building up a set of skills that nobody knew was going to be very important. He was able to surprise people.


And that has a lot of parallels with changes that happened in our industry. So for example, when Joe was at his peak, so was the concept of brand. We had this idea that brand has value, even if we can't measure it. And some of the most important brand building campaigns came out around this time.


A little while later, a couple of small organisations began work. Google began its quest to completely catalogue the internet. Facebook began its quest to completely catalogue my aunt's means.


And to begin with, nobody really thought much of these little companies and the data they were offering. But as we know, over time, those companies grew until they ain't so little anymore.


And the people who've been playing around in the background, who knew how to work with social media communities, who knew how to work with search engines, they had an advantage. Like Joe, they were able to surprise people.


And around that time, a lot of us kind of locked away this concept of brand. Why would we try to measure something so big and messy when we know how many people are searching for our product? How many people are going to see the next tweet I sent? How many people land on my site and then convert?


Now, things are changing again. So those little companies have become some of the biggest monopolies we've ever seen. Marketplaces are getting more competitive, costs are going up to advertise. It's not enough to know how to get a site crawled and indexed when we're working against Behemoth companies.


The days of quick wins are waning, which means that as SEOs, we're more impacted by the performance of the business and the brand than we ever have been before.


As an example, we can all agree that Moz is a very successful company, but in 2018, Rand Fishkin gave a talk in which he said he wishes he'd been able to measure Moz's product brand because he might have seen competitors like SEMrush and Ahrefs coming.


Likewise, Tom Capper has shown that brand correlates more strongly with rankings than links do. In fact, he showed that in 2017, 2018, and in 2019.


Now, these talks hit home particularly hard for me because around that time, I'd started working with the APAC wing of a global B2B company.


It's our job to help 10 teams across 20 countries, 30 languages to perform in search. And we started with a lot of the usual stuff. Tech audits, keyword research, training, and dashboards.


We kicked off projects across Singapore and Melbourne and Tokyo. And every time we did, we asked, what does marketing look like for you?


And the things we heard back were things like blogging and emails and digital ads, but also things like local country partners, small events, opening new locations.


There was a whole host of things that these teams were doing that we weren't contributing to at all, and it was impacting our metrics.


We would see the organic conversions had taken a dip and look into it and realize that they dialed back events in that territory and conversion rate across the board had fallen.


Or we'd see that we'd started ranking for a couple of terms particularly well. And they would say, yeah, we've been pushing those terms quite a lot to our industry partners.


We think we're just getting interest and links we're not seeing.


We have competitors who are experimenting with driving up specific branded searches to improve their rankings. 35% of our traffic is branded.


So even if everything else is looking good, it can still take a dive if brand falls. And a lot of the stuff we're doing does contribute to brand.


None of us thinks that when we write a blog post, that's going to take someone from I've never heard of them to, yeah, I'll spend $10,000 a month, please.


This kind of activity has value because it contributes to brand.


And initially we thought, this is just a funnel problem. Blog posts and events, they're just very top of funnel, will improve our attribution and account for them that way.


But attribution is very hard to do. It doesn't account for things like offline events or people taking a long time to decide about something.


And it relies on things like cookies to track users over time. And we know that browsers like Safari and Firefox are targeting cookies so much they'd be safer on Sesame Street.


So our funnel is starting to erode. We're, as SEOs, more and more impacted by brand. We were bad at measuring it to begin with and we're getting worse.


So at this point I'm starting to feel a bit hot under the collar, right? But we look into it and we realize that, yeah, we're facing some real challenges as an industry, but without knowing it, we're ready for them.


We've been building up a set of skills and like Joe, like the upstarts of the 90s, we're ready to surprise some people.


In this talk, I'm going to take you through my journey to measuring brand the way I do, a broad way of getting brand insight and a more granular approach to get tactical brand info.


Now, I knew this was information we needed, so I started by finding any guide I could, but they resulted in frustratingly vague conclusions like, you have bad brand awareness in Japan, spend more money on brand advertising, master all relevant sales channels.


I can't say this to my client. They would say, I asked you to help improve the performance of my business and you're telling me to get the attention of the entirety of Japan.


You realize someone constructed a full-scale Godzilla in Tokyo and still couldn't get anyone's attention.


So we took a step back and we thought, okay, what do we want here? We want brand interest data.


And we could use surveys, but the problem with surveys is they're expensive, they take time and we can't ask everyone, so we're missing out on a bunch of data.


What's more, academics have said for years that we can't always trust what people write in surveys.


So even if it's part of the picture, our data can't just be self-reported.


What's more, we need to be able to do a competitor comparison. So it's not enough to be able to just say our brand is strong.


We have to be able to say it's stronger than these people or weaker than these people.


Now, as SEOs, for a while we've had a way of knowing how many people are interested in something.


We asked Google. We know, for example, that Kanye West is a very popular man.


And we know that Taylor Swift is even more popular still.


Normally we can ask tools like Stat and Moz and SEMrush, but they tend to report at country level.


So brand data, brand interest in New York is not the same as brand interest in Seattle.


So we need a way to break that country up.


And they also tend to report in blocks of numbers.


So if we get results back saying, yeah, you and all your competitors get a thousand searches a month, there's not much to work with.


So we need data that's not just self-reported, that lets us do a competitor comparison, so we can't rely on things like Search Console,


and gives us the granular data to break up a country and see the difference between brands.


Now, this is a pretty tall order. And as you can probably guess, I exhausted a lot of my options before I turned to the free tool I'm going to talk to you about now.


As a warning, when I put this on screen, you're probably going to think I'm a bit of a moron, that I've been wasting your time.


But honestly, by the time we finish this section, I think you'll agree that this tool is not being used to its full potential,


and a lot of people are missing out on valuable brand insight.


So when I was trying to make head or tail of my client in APAC, we put their brand into Google Trends.


This line shows the searches from my client in Google Trends in India over the last 10 years.


I didn't know anything about their industry in India, certainly nothing about it 10 years ago, but this tells me a really clear story.


We can also add in competitors and see the red competitor started in the lead and has lost that over time.


Their industry is getting more competitive. The yellow competitor was behind, but over the last two years, they've taken the lead.


This also lets us zoom in on specific activity. So for example, this spike here corresponds with a large influencer-led conference they ran.


This got a lot of press attention, really caught the eye of our team on the ground. But as we can see, after that spike, things kind of drop off.


And it's not until later on when we see their graph steadily rising, that's when they're running a series of smaller ongoing events scattered across the country.


Now, we already had a big conference planned. We didn't cancel that course, but we made sure that from the off, we planned in a series of follow-up events,


smaller, targeting the people we really want to talk to.


And before the COVID virus hit, we were starting to see some really nice changes in this graph and a boost in organic search volume.


Now, one problem we run into here is it's not always clear exactly what the competitor's done.


So we've added in this green competitor. Now, we can't pinpoint a specific activity.


What's more, the lines are far enough apart that we run into that same Godzilla problem.


How do we make an entire country pay attention to something else? Well, we don't.


We have to be more tactical than that, and that's where the maps in Google Trends come into play.


So this is exactly that same data, searches for these brands, but broken up at territory level.


We can see the green competitor is dominating, and the deeper the green, the stronger a hole they have.


We can also see percentage breakdowns by territory. So in Maharashtra, for example, out of all the searches for these competitors, 75% are for the one in green.


Now, one problem we run into here is not every territory is worth the same amount.


You know, 50% of searches in one territory can be worth the same as 100% in another.


So we needed a way to know where is valuable.


And this is where we turn to census data.


So every country seems to make census data freely available and tell you number of businesses based on location and industry and size or number of people based on age and demographics.


So if you have an idea of your target market, you can combine census data with proportions of search to work out which territory is worth how much to you and your competitors.


As an example, Bangalore looks like a good target for us, you know, growing tech hub.


But we could see that the green competitor had been strong there for a while.


It's actually their global headquarters.


And according to census data, there actually weren't that many relevant businesses.


So even if we're talking purely about branded organic traffic, we're not getting a return on investment.


Mumbai, on the other hand, well, the green competitor was investing, but they haven't always been strong.


The census data showed there's lots of valuable businesses there and lots of small to medium businesses, which we know the green competitor doesn't cater particularly well to.


So off the bat, we know that we can push out lots of messaging about small to medium businesses.


We can look at what people are saying on social media and in press around Mumbai to find out what matters to them, use that to generate content.


Now we start ranking for terms that people are searching, which is generating organic traffic, but also strategically relevant organic traffic.


We can take that content, push it back out to press.


They're going to cover it because it's targeting the kind of things that audience are interested in.


So now we're getting links and press attention and organic traffic and cut through.


Everything feeds into each other.


So Google Trends, often overlooked, is a really powerful source of brand insight.


It will give us an idea of the history of a territory.


It will tell us whether big events are working for us or our competitors, and it will help us find valuable territories and narrow them down with census data.


But it's not perfect.


It doesn't give us actual numbers.


We have to rely on proportions out of 100 and using census data.


It's not terribly fine grained.


We tend to work with relatively large territories and we only get brand level data.


And we know that a brand isn't just one thing.


So, for example, say we're looking at Sony.


You could know Sony for PlayStation, headphones, TVs, batteries, cameras, DSLRs, phones, watches, movies, semiconductors or multi application cell sources.


So if we're thinking about what data we want here, maybe it's not just brand level data.


Maybe it's product specific brand cut through.


This is where my more granular approach comes in.


And I was facing this problem when I happened to talk to my colleague, Rich Cotton, who's a senior member of our paid advertising team.


And he suggested getting this information out of Google ads.


Now, we know when we're working with Google ads, we get lots of lovely data like impression numbers.


How many times people have seen our ads and impression share.


Now, Google calculates impression share by looking at our targeting criteria, working out the total number of times we could have been seen, and then telling us the proportion of times we were actually seen.


So say, for example, we're bidding on mattresses in Seattle and Google tells us we got a thousand impressions and 50% impression share.


From that, we can estimate that we got 2,000 searches for mattresses in that time.


This is how we're going to get that more granular data.


Now, if you know Google ads, you could be thinking, but can we really believe impression share?


Because Google's really tweaked this formula.


Say we're bidding on mattresses, but we're not bidding high enough.


They'll remove a bunch of searches from those impression share calculations.


So it completely throws off our numbers.


We end up having to bid really high on really specific mattress terms, and it's not worth the while.


But in this case, we're looking for brand insight, which means two things.


First, we're willing to spend a little bit more.


You know, brand investigations can cost tens of thousands of dollars, and this is way cheaper.


Second, because we're able to be very specific with the searches we're focusing on, we can drive those costs down until they don't really matter.


Now, as you can guess, product-level brand data is impossible to anonymize.


So I've gone out and I've gathered data on fashion brands across West Coast America.


Now, you could be wondering, does this guy know anything about fashion in West Coast America?


No, I don't. I don't work with any of these brands.


But when you're doing this kind of analysis, you will have that context.


So you get an idea of how I approach this.


I bid on phrase match terms.


So the exact phrase is H&M dresses, J.Crew dresses, Zara dresses, Zara maternity, Uniqlo bikini, and a host of other besides.


I had separate campaigns running in L.A., Vancouver, Seattle, and Portland so I could get city-level data.


And you could be wondering, how much did all of this cost?


Because taking other people's branded traffic gets expensive.


But the thing to bear in mind here is we don't want their traffic.


We want their data.


So, yeah, it would be really expensive for me, Robin Lord, to bid on Zara sale and take their traffic.


Which is why, when I was bidding on Zara sale, I drove all of that traffic straight to Zara.


Most of the time, these brands weren't bidding on the terms themselves.


I didn't want that traffic. I didn't want to take anything away from them.


All I wanted to know is how many people are searching Zara sale.


Which meant that it cost me just three cents per keyword per day.


So we don't have enough time to go through all of the examples here, but we'll go through three and start with a very simple graph.


So this is the total number of searches for Zara dresses across L.A., Vancouver, Seattle, and Portland during February this year.


This is the total number of searches for Zara dresses, Zara jeans, Zara pants in those territories during February this year.


This is Zara compared to competitors.


Now, one thing we can see from this graph is Zara is smashing it.


They're getting way more product-specific searches than everyone else.


But if we turn to Google Trends, we can see that Zara is doing well, but their brand search actually isn't that much higher than everyone else.


So what could be causing people to search for Zara plus products so much more than everyone else?


One of the things that came up when I was looking into this brand is the site can be quite hard to use.


So to break down what could be happening here, say someone's on the Zara site and they're looking for dresses, but they find it quite difficult to use.


So instead, they just jump to Google and search for Zara dresses instead. Get the page they want.


The problem here is they're passing through Google, which is a place where many people advertise dresses, and they could go to a competitor, which has lost sales in and of itself.


What's more, Google is seeing someone land on the Zara site, not get what they want, come back to search results quickly and go somewhere else.


Now, it's controversial as to whether Google uses this data directly in search rankings, but we do know that Google's feeding more and more UX signals into the search results.


They're rolling out changes at the moment.


So even before we look at competitor versus competitor comparisons, this gives Zara some data to work with.


If they see that their generic branded search is roughly in line with competitors, but their product specific search is much higher, that gives them a sign that maybe something's broken in their UX.


They could run surveys on people leaving the site to see what they're struggling with, or use Twitter advanced search to reach out to social media users, find out what they're struggling with when they say the site is hard to use.


They can even offer these people incentives to get involved.


These are people who are already tweeting about the brand, who care enough to care that the site is hard to use, so they're likely to get their followers involved as well, which is free promotion.


We can also treat this as kind of similar to a tech order.


We're rooting out systematic issues that could be impacting performance.


So maybe this helps improve our rankings over time.


Even if it doesn't, it helps improve our conversion rate, which makes our organic numbers look better anyway.


But we didn't come here for brand insight. We didn't come here for UX insight. We came here for brand data.


So let's look at two of the biggest brands we have, H&M and Zara.


At a product level, it's kind of clear that Zara is winning across the board, except for in maternity.


There are way more searches for H&M maternity than Zara maternity, which is interesting given how much better they're doing everywhere else.


Now I asked around a bit about this, and overwhelmingly the response I got was, yeah, I'm not really surprised.


Zara doesn't really do maternity.


So that's kind of a boring thing for our data to point out, I suppose.


People don't search for something that someone doesn't do.


So if we look at H&M's maternity category, we see that they've got over 400 maternity products.


They've invested heavily in it.


But if we look at Zara's maternity category, which they do have, we can see they've got over 200 products.


So even if it's clear to us as consumers that people wouldn't search for Zara maternity, someone at Zara disagrees.


Now, if we just say Zara is doing worse for maternity, that gives them something to work on,


but they're still trying to change the opinions of the entire West Coast about a topic, which is a tall order.


So let's break that data down a little bit more.


These are numbers of searches for Zara maternity and H&M maternity across L.A., Vancouver, Seattle, and Portland.


Now, one problem we run into here is even though we can see Zara and H&M are doing best in L.A. and Vancouver,


we're looking at cities of very different sizes.


So we end up basically just tracking city size with search volume, unless we bring in our old-friend census data.


So if we do something as simple as divide these search volumes by the number of people in a city,


we get kind of per capita search number, proportional search.


Now we can see that they're both doing best in Vancouver.


So maybe Zara starts thinking about birth rates in Vancouver and whether it's an area to target.


H&M is doing second best in Portland.


Now, Zara is doing worst in Portland, so maybe they decide this is an area we need to focus on,


this is where we're losing out the most.


So in terms of outreach, they could focus on Vancouver and Portland, targeting mom bloggers and press to drive up interest.


That should increase interest in things like their category pages, maybe get them some links which should help them improve their rankings.


They could also do things like overbidding on maternity terms with PPC,


because the value that we're driving with these organic sessions, with these paid sessions,


yes, short term, we're getting sessions in sales.


But we're also appearing in search results and letting people know this is something that we do.


We are an option.


So it's going to have short-term impact on organic sessions,


but also much longer-term impact on branded organic sessions continuing to drive value.


As our final example, let's look at Anthropology and J.Crew.


They have roughly similar overall branded search, and it's kind of a mixed bag when we look at product, until we focus on plus size.


There are far more searches for Anthropology plus size than J.Crew plus size.


And to be fair, Anthropology really puts in the effort.


They're number one for plus size.


They have a number of category pages targeting plus size searches, which, as I say, drive organic sessions now and also change minds.


They show a variety of body types on their social media.


They sponsor events like CurvyCon, which let them have models on the runway and pop-up shops and got them featured in blogs.


They have Instagram influencers with over 260,000 followers talking about Anthropology plus size.


And it got them featured on Good Morning America, which has a morning viewership of over 5 million.


J.Crew, on the other hand, we can look at by doing something as simple as searching J.Crew plus size in Google News.


That'll tell us how press are talking about J.Crew plus size, and we can use the tools function to filter to a certain time period to see how that changed over time.


So, for example, in 2014, we can see that J.Crew aren't doing very well for plus size searches.


They're kind of getting slammed for the focus on the lower end of the scale.


Now, if we jump back to 2020, things have kind of changed.


J.Crew are doing considerably better.


The brands, the publications that were previously slamming them, are now talking about J.Crew as a brand that is changing modern fashion.


And Lisa Greenwald, their chief merchandising officer, has promised shoppers can expect to see their favourite styles in a wider range of sizes.


And that is what we see on the product page. You know, there's a variety of sizes on show.


The problem is, the product page itself doesn't make that very clear.


What's more, J.Crew have taken the decision that if someone's searching for dresses, they shouldn't have to search in a separate plus size dresses page.


If you want to dress, go to our dresses page, which I think is fantastic, but it means they haven't got any plus size category pages.


So if we search J.Crew plus size, as we see some people are, we get an ad for the homepage, resellers, and then a 404 deals page.


So we're probably losing traffic here.


What's more, if we click on that ad for the homepage, the models there don't really show a variety of body shapes.


If we look at things like the Twitter or Facebook ads library, we can also see what kind of advertising J.Crew is pushing out at the moment.


And again, there's a variety of styles, but not necessarily a variety of body types.


So what could they do to address this?


Well, they could start by looking at that territory level breakdown and see that Anthropologie is doing best in Seattle and LA.


Well, that's a really clear target for them.


You know, they can focus on press and outreach around Seattle and LA to make sure that they're really getting this cut through.


They could also consider doing things like sponsoring events there and definitely consider sponsoring CurvyCon.


Even though we don't have data for West Coast America, that seems like a fantastic opportunity.


They could also specifically target those territories with their social media ads.


So that lets them get a lot more cut through and lets them run a series of tests on what's working.


Because they're focusing on smaller populations, everything gets cheaper.


They can do more. They can also, again, overbid on plus size terms in PPC and they can create a plus size category page.


I think it's great that they don't want to segregate their audience, but they can have a plus size category page alongside the dressers page.


That, at the very least, is going to let them rank for J.Crew plus size, which is free traffic.


It's going to drive down costs for their PPC and social advertising.


It's an excuse to link to that page from the homepage, so now they can show different model types on the homepage.


And now they've got a targeted, well-linked to page targeting plus size terms, so they can start ranking organically for those plus size terms.


They could even decide to put a banner right at the top of the page saying, why are you here?


You don't have to shop in a separate category. Shop in our main dressers page. Everything is there.


That's turning a category page into a brand statement, which can get press interest and links, which will help them rank even better,


which is getting them more traffic, which is changing minds about what J.Crew does, which drives them branded traffic further down the line, so everything feeds into itself.


So Google Ads will give us city-level, product-level data.


It will give us insight into how people are interacting with the search engine around our site to see if our UX is broken.


It is paid. I mean, it costs us some money, not loads, but some, which means that we can only really get city-level data, otherwise it gets very expensive.


And we don't get any historic data like we do with Google Trends.


Now, I've spoken to you a lot about the kind of things we can do with this data, but I haven't told you really much how to set up the campaigns.


I've put out a bunch of fairly dry bullet points that you can go through here that will tell you how to get the results that you want.


But by means of wrap-up, whether we're using Google Trends or Google Ads, because they both have their uses,


we can use this data to tell our colleagues, our bosses, our companies what to put on billboards and where to put them,


when to judge ads by completely different metrics, what newspapers to target, social media to invest in, influencers to reach out to, emails to send,


when to judge organic sessions by completely different metrics, everything up to informing our five-year plan,


which will help us give our customers what they want rather than just what they know us for.


This is all ways to give your company direction.


If you're head of marketing, I've given you data your competitors don't have.


If you're head of SEO, I've given you a way to start expanding your role, owning these strategic conversations.


They're impacting your team already as it is. This is just a way for you to get credit for all that hard work.


But the most important thing here isn't, here are some tricks in Google Ads, here are some tricks in Google Trends.


The most important thing here is none of this is new to us.


This is looking at what people are searching, figuring out why we're not appearing and doing better.


That's our day-to-day. That is our skill set.


We are facing some real challenges as an industry.


But we're not just ready for them, we're made for them.


So let's go surprise some people.


10 Basic SEO Tips to Index & Rank New Content Faster - Whiteboard Friday

Howdy Moz fans, welcome to another edition of Whiteboard Friday.


I'm Cyrus Shepard, back in front of the Whiteboard, so excited to be here today.


We're talking about 10 tips to index and rank new content faster.


So you publish some new content on your blog, on your website, and you sit around and you


wait, and you wait for it to be in Google's index, you wait for it to rank.


It's a frustrating process, it can take weeks or months to see those rankings increase.


There are a few simple things we can do to help nudge Google along, to help them index


it and rank it faster.


Some very basic things, some more advanced things too, we're going to dive right in.


So basically, indexing content is not that hard in Google.


Google provides us with a number of tools.


The simplest and fastest is probably the URL inspection tool.


It's in the new search console, previously Fetch and Render.


As of this filming, both tools still exist, they're depreciating Fetch and Render, so


the new URL inspection tool allows you to submit a URL and tell Google to crawl it.


And when you do that, they put it in their priority crawl queue.


That just simply means Google has a list of URLs to crawl, it goes into the priority and


it's going to get crawled faster and indexed faster.


Another common technique is simply using site maps.


If you're not using site maps, it's one of the easiest, quickest ways to get your URLs


indexed.


When you have them in your site map, you want to let Google know that they're actually there,


and there's a number of different techniques that can actually optimize this process a


little bit more.


The first and the most basic one that everybody talks about is simply putting it in your robots.txt


file.


So in your robots.txt, you have a list of directives and at the end of your robots.txt,


you simply say site map and you tell Google where your site maps are.


You can do that for site map index files, you can list multiple site maps, it's really


easy.


You can also do it using the search console site map report, another report in the new


search console.


You can go in there, you can submit site maps, you can remove site maps, validate.


You can also do this via the search console API.


But a really cool way of informing Google of your site maps that a lot of people don't


use is simply pinging Google.


You can do this in your browser URL.


You simply type in google.com backslash ping and you put in the site map with the URL.


You can try this out right now with your current site maps, type it into the browser bar and


Google will instantly queue that site map for crawling.


All the URLs in there should get indexed quickly if they meet Google's quality standard.


So that's talking about indexing.


Now there's some other ways that you can get your content indexed faster and help it to


rank a little higher at the same time.


So when you publish new content, basic, if you do nothing else, you want to make sure


that you're linking from important pages.


Important pages may be your home page, adding links to the new content, your blog, your


resources page.


This is a basic step that you want to do.


You don't want to orphan those pages on your site with no incoming links.


And adding the links tells Google two things.


It says we need to crawl this link sometime in the future and then get to put in the regular


crawling queue.


But it also makes the link more important.


Google can say, well, we have important pages linking to this.


We have some quality signals to help us determine how to rank it.


So linking from important pages.


But a step that people oftentimes forget is not only link from your important pages, but


you want to go back to your older content and find relevant places to put those links.


A lot of people use link on their home page or link to older articles, but they forget


that step of going back to the older articles on your site and adding links to the new content.


Now what pages should you add from?


One of my favorite techniques is to use this search operator here where you type in the


keywords that your content is about and then you do a site example.com.


This allows you to find relevant pages on your site that are about your target keywords


and those make really good targets to add those links to from your older content.


Really obvious step, sharing socially.


When you have new content, sharing socially, there's a high correlation between social


shares and content ranking.


But especially when you share on content aggregators like Reddit, those create actual links for


Google to crawl.


And so Google can see those signals, see that social activity, sites like Reddit, Hacker


News, where they add actual links.


And that does the same thing as adding links from your own content, except it's even a


little better because it's external links, it's external signals.


So this is kind of advanced technique, which is a little controversial in terms of its


effectiveness but we see it anecdotally working time and time again and that's simply generating


traffic to the new content.


Now there's some debate whether direct traffic is a ranking signal.


There's some old Google patents that talk about measuring traffic and Google can certainly


measure traffic using Chrome so they can see where those sites are coming from.


But as an example, Facebook ads.


You launch some new content and you drive a massive amount of traffic to it via Facebook


ads.


So they're paying for that traffic but in theory, Google can see that traffic because


they're measuring things using the Chrome browser.


And when they see all that traffic going to the page, they can say, hey, maybe this is


a page that we need to have in our index and maybe we need to rank it appropriately.


So once we get our content indexed, talk about a few ideas for maybe ranking your content


faster.


Along with generating traffic to the URL, you can actually generate search clicks.


Now what do I mean by that?


So imagine you share a URL on Twitter.


Instead of sharing directly to the URL, you share to a Google search result.


So people click the link and you take them to a Google search result that has the keywords


you're trying to rank for and people search and they click on your result.


You see television commercials do this like in a Super Bowl commercial.


They'll say, Google this, go to Google and search for Toyota cars 2019.


And what this does, Google can see that searcher behavior instead of going directly to the


page, they're seeing people click on Google, choosing your result.


So this does a couple of things.


It helps increase your click-through rate, which may or may not be a ranking signal,


but it also helps you rank for auto suggest queries.


So when Google sees people search for best cars 2019 Toyota, that may appear in the suggest


bar, which also helps you to rank if you're ranking for those terms.


So generating search clicks instead of linking directly to your URL is one of those advanced


techniques that some SEOs use.


Target query deserves freshness.


So when you're creating the new content, you can help it to rank sooner if you pick terms


that Google thinks deserves freshness.


It's best if maybe if I just use a couple of examples here.


Consider a user searching for the term, cafes open Christmas 2019.


That's a result that Google wants to deliver a very fresh result for.


You want the freshest news about cafes or restaurants that are going to be open Christmas


So Google is going to preference pages that are created more recently.


So when you target those queries, you can maybe rank a little faster.


Compare that to a query like history of the Bible.


That's probably if you Google that right now, you'll probably find a lot of very old pages


with PDF pages.


Those results don't update much.


And that's going to be harder for you to crack into those SERPs with newer content.


So the way to tell this, simply type in the queries that you're trying to rank for and


see how old the most recent results are.


That will give you an indication of what Google thinks how much freshness this query deserves.


Use queries that deserve a little more freshness and you might be able to get in a little sooner.


Finally last tip.


This is something a lot of sites do and a lot of sites don't do because they're simply


not aware of it.


Leverage URL structure.


When Google sees a new URL, a new paged index, they don't have all the signals yet to rank


it.


So they have a lot of algorithms that try to guess where they should rank it.


And they've indicated in the past that they leverage the URL structure to determine some


of that.


So consider the New York Times puts all its book reviews under the same URL, newyorktimes.com


slash book reviews.


They have a lot of established ranking signals for all these URLs.


So when a new URL is published using this same structure, they can assign it some temporary


signals to rank it appropriately.


So if you have URLs that are high authority, maybe it's your blog, maybe it's your resources


on your site and you're leveraging an existing URL structure, new content published using


the same structure might have a little bit of a ranking advantage, at least in the short


run until Google can figure these things out.


So these are only a few of the ways to get your content indexed and ranking quicker.


By no means comprehensive list.


There's a lot of other ways.


We'd love to hear some of your ideas and tips.


Please let us know in the comments below.


And if you like this video, please share it for me.


Russ Jones: A Celebration of Life

Let's see how this goes.


Russ Jones: A Celebration of Life

Gotta figure out this lighting better.


Russ Jones: A Celebration of Life

Test from sitting for Whiteboard Friday.


Russ Jones: A Celebration of Life

Hopefully it works well.


Russ Jones: A Celebration of Life

Howdy, MOS fans.


Russ Jones: A Celebration of Life

I am Russ Jones, and I can't tell you how excited I am


Russ Jones: A Celebration of Life

for my first Whiteboard Friday.


Russ Jones: A Celebration of Life

Welcome to another great Whiteboard Friday.


Russ Jones: A Celebration of Life

Another Whiteboard Friday.


Russ Jones: A Celebration of Life

Great to be back here with you on Whiteboard Friday.


Russ Jones: A Celebration of Life

Normally at this time of night,


Russ Jones: A Celebration of Life

I figured I'd be having a scotch,


Russ Jones: A Celebration of Life

so maybe I'll start with that.


Russ Jones: A Celebration of Life

It's not just about the people who are in need.


Russ Jones: A Celebration of Life

It's about the people who are helping those in need.


Russ Jones: A Celebration of Life

And the deeper you can dig,


Russ Jones: A Celebration of Life

the better it's gonna be for them in the long.


Russ Jones: A Celebration of Life

So just try and stick with me


Russ Jones: A Celebration of Life

for just the next couple of minutes.


Russ Jones: A Celebration of Life

The beer is coming. Don't worry.


Russ Jones: A Celebration of Life

It's hard to believe my favorite task in MOS


Russ Jones: A Celebration of Life

is specifically to ruin Dr. Pete's day.


Russ Jones: A Celebration of Life

And he's done.


Russ Jones: A Celebration of Life

He's already given his talk, so he can't respond.


Russ Jones: A Celebration of Life

So that works great for me.


Russ Jones: A Celebration of Life

I brought my family up this time


Russ Jones: A Celebration of Life

to finally get to see Seattle,


Russ Jones: A Celebration of Life

and I go in there, and I'm like,


Russ Jones: A Celebration of Life

look, Daddy has got to talk


Russ Jones: A Celebration of Life

to like a thousand people in the morning.


Russ Jones: A Celebration of Life

You have got to go to sleep.


Russ Jones: A Celebration of Life

And my oldest looks at me, and she says,


Russ Jones: A Celebration of Life

a thousand people? That's a lot.


Russ Jones: A Celebration of Life

What if they don't like you?


Russ Jones: A Celebration of Life

I was like, sweetie, Daddy's a black hat.


Russ Jones: A Celebration of Life

He can always buy reviews. It'll be fine.


Russ Jones: A Celebration of Life

There are lots of things that I can think about


Russ Jones: A Celebration of Life

that are positive.


Russ Jones: A Celebration of Life

Made great friends.


Russ Jones: A Celebration of Life

I helped build businesses.


Russ Jones: A Celebration of Life

I've built software.


Russ Jones: A Celebration of Life

I helped other companies, small mom and pops.


Russ Jones: A Celebration of Life

And I think that that's the one takeaway


Russ Jones: A Celebration of Life

I want you guys to walk away with today.


Russ Jones: A Celebration of Life

It's really great to know


Russ Jones: A Celebration of Life

that you've got a lot of time ahead of you


Russ Jones: A Celebration of Life

to make a difference, to do it better.


Russ Jones: A Celebration of Life

It doesn't have to be the way it feels like every day.


Russ Jones: A Celebration of Life

It doesn't have to be a situation


Russ Jones: A Celebration of Life

where you feel like I've got to prove to Google,


Russ Jones: A Celebration of Life

I've got to make Google think that my site's the best.


Russ Jones: A Celebration of Life

Now, today, it has to happen immediately.


Russ Jones: A Celebration of Life

We need to stop sacrificing the long run for the short run.


Russ Jones: A Celebration of Life

So please, when you walk away from this conference,


Russ Jones: A Celebration of Life

I ask that you just remember one thing,


Russ Jones: A Celebration of Life

to be rather than to see it.


Russ Jones: A Celebration of Life

Thanks.


Russ Jones: A Celebration of Life

I want to end on a positive note.


Russ Jones: A Celebration of Life

These are difficult times.


Russ Jones: A Celebration of Life

But if there's one thing that I've seen,


Russ Jones: A Celebration of Life

I've seen time and time again,


Russ Jones: A Celebration of Life

is that people in our industry care


Russ Jones: A Celebration of Life


Better alternatives to Expert Content style content - Whiteboard Friday

Howdy, Moz fans, and welcome to another edition of Whiteboard Friday.


This week we're going to look at some better alternatives to the expert roundup style content


that's become extremely popular on the web, and there's a few reasons why it's popular.


Let's talk about why SEOs and content marketers do so many expert roundups, why this became


a popular content format.


Okay, it turns out if you've got a piece of content that's like 75 experts share their


favorite constitutional law cases, maybe you interviewed a bunch of constitutional law


scholars and you put together this article, there's a bunch of nice things that you actually


do get from this, which is why people use this format, right?


You kind of get automatic outreach, because if you talk to these people, you've had a


connection with them.


You built a little bit of a relationship.


There's now something of an incentive to share for these folks and the potential for a link.


All of those are sort of elements that people are looking for, well, that marketers are


looking for from their content.


The nice thing is you've got this long cadre of individuals who've contributed and they


create the content, which means you don't have to, saving you a bunch of time and energy,


right?


They become your amplifiers, so you can kind of sit back and relax when it comes time to


broadcast it out there.


You just tell them it's ready and they go and push it and they lend your content credibility.


So even if you don't have any credibility with your brand or with your website, they


deliver it for you.


You don't have to do that.


Those are all really nice things.


Don't get me wrong.


I understand why, but there are some big, big problems with expert roundup style content.


First one, like many of the easy to replicate tactics, expert roundups got spammed to heck,


They became way, way overdone.


I get emails like this, dear Fishkin, I round up you, right?


Do this then share.


Okay, bye spammy McSpams a lot.


Look Mr. McSpams a lot.


I appreciate how often you think of me.


I love that every day there's a couple of offers like this in my inbox.


I try to contribute to less than one every two or three weeks and only the ones that


look super credible and real interesting.


But geez, I mean, can you imagine if you are truly an expert who can lend credibility and


create lots of amplification, you're getting overwhelmed with these kinds of requests and


people are probably getting very tired of reading them, especially in certain market


segments where they've become way too overdone.


But even if, even if it's the case that you can get all these experts to contribute and,


you know, it is not overdone in your market space, there's two other big problems.


One, the content format is awful, awful for trying to get valuable and useful information.


It rarely actually satisfies either searchers or engines.


If you search for constitutional law cases and you see 75 experts share their favorite


constitutional law cases, you might click, but my God, have you gone through those types


of content?


Have you tried to read a lot of those roundups?


They are usually awful, just, just terrible.


You might get a nugget here or there, but there's a bunch of contributions that are,


you know, multiple paragraphs long and try to include links back to wherever the expert


is trying to get their links going.


There's a bunch of them that are short and meaningless.


Many of them overlap.


It's annoying.


It's bad, right?


It's not well curated.


It's not well put together.


There are exceptions.


Sometimes people put real effort into them and they get good.


But most of the time these are real bad things and you rarely see them in the search results.


So BuzzSumo did a great analysis of content that gets shares and gets links and gets rankings.


Guess what?


Did not fall into it.


Expert roundups.


Right?


Number three, right?


That's number three.


The links that the creators want from these roundups that they're hoping they're going


to get doesn't end up there most of the time.


What usually happens is you get a short traffic spike, some additional engagement, some additional


activity on mostly Twitter, sometimes a little bit Facebook or LinkedIn, but it's almost


all social activity and it's a very brief spike.


So what are some better alternatives?


What are some things we can do?


Well, I've got five for you.


First off, if you're going to be creating content that is, you know, a round or a roundup,


why not do almost exactly the same process but rather than asking a single question or


a set of questions that people are replying to, ask them to fill out a short survey with


a few data points because then you can create awesome graphs and visuals which have much


stronger link earning potential.


It's the same outreach effort but for much more compelling content that often does a


better job of ranking, is often more newsworthy and linkworthy.


I really, really like surveys and I think that they can work tremendously well if you


can put them together right.


Second, let's say you go, oh, Rand, that would be great but I want to survey people


about this thing and they won't give me the information that I'm looking for.


Never fear, you can aggregate public data.


So a lot of these pieces of information that may be interesting to your audience that you


could use to create cool visuals and graphs and charts and all that kind of thing and


trend lines are actually available on the web.


All you need to do is cite those sources, pull in that data, build it yourself and then


and then you can outreach to the people who are behind these companies or these organizations,


these individuals and then say, hey, I made this based on public data, can you correct


any errors?


Now you've got the outreach which can lead to the incentive to share and to build a link.


Very cool.


Experiments and case studies.


So this is taking a much smaller group saying I'm only going to work with this one person


or these couple of people or I'm going to do it myself, right.


And here's what Seattle's most influential law firm found when they challenged ten state


laws.


Oh, there you go, right.


Now I've got an interesting wholly formed case study.


I only had to work with one expert but chances are good that lots and lots of people will


be interested in this.


It's also excellent for newsworthiness.


You can also even get lots of press coverage in whatever industry you're in.


Fourth, if you're going to interview, if you're going to do a roundup style thing and you're


going to collect multiple opinions, if you can find a few points or a single subject


around which multiple experts have different opinions, that could be just two people, it


could be four or five, it could be seven or eight, but you're basically trying to create


this controversy.


So you're saying like here are these people on this side of this issue, here are these


people on this side of this issue.


Will Reynolds versus Rand Fishkin on link building.


I think we did a presentation like that in Minneapolis last year a couple of years ago


it was super fun, right?


Will and I got up on stage and we sort of debated with each other.


There were no losers in that debate, it was great.


This leverages the emotional response you're seeking of conflict, it creates more engaging


content by far and there's more incentive for the parties who participate to link and


share because they're sort of showing off their opinion and trying to make counterpoints


and you can get a lot of good things.


Number five, if you've decided you know what none of these formats or any other work,


I really, really want to do a roundup, I think it can work for me.


Okay, okay, but do me a favor and try something that is not just text, not just text.


Musly, M-U-Z-L-I is a newsletter I subscribe to in the design world that does lots of roundup


style content, but the roundups are all visuals, right?


They're visuals, they're like UI interactions and GIFs and animations and illustrations.


I actually really love those, those get great engagement and they rank by the way, they


rank quite well.