The Great UX Debate:

What Google Says vs What Google Does

SMX Advanced Europe 2019

UX

is a

ranking factor

UX

is a

ranking factor

How users see your site

UX

is a

ranking factor

UX

is a

ranking factor

Affects organic search performance

Not everyone agrees

Don’t worry. I have seen the reddit threads and twitter arguments

Later, I’ll get onto why I don’t care

Round up UX ranking factors

Make it actionable

Round up UX ranking factors

Make it actionable

Controversial

Accepted

CTR for algorithm evaluation

​

This has been confirmed repeatedly.

Paul Haahr

​

Worth following @haahr

​

His online bio dramatically understates his influence:

https://youtu.be/iJPu4vHETXw

“I’ve talked explicitly about using clickthrough rate for experiments”

https://youtu.be/eGSGZMI4Z_I?t=979

Controversial

Accepted

CTR for personalisation

​

Slightly more controversial as we don’t know exactly what they mean.

“We use it [CTR] for personalisation”

https://youtu.be/eGSGZMI4Z_I?t=991

Controversial

Accepted

Human UX ratings

​

Qualitative ratings of pages that rank: used to evaluate algorithms

“... make this page a frustratingly poor user experience”

​

-- reason for page quality rating

Search quality rater guidelines [PDF]

We also got to see the “leaked” UX playbooks come out of Google recently across a range of industries - getting deep into implementation details. [H/T @gfiorelli1]

Controversial

Accepted

ML models trained on ratings

​

E.g. Panda - trained to like the things humans like

Back in 2011, I was suggesting we run our own Panda-like quality surveys (WBF here, instructions here)

Sidenote: totally worth running these surveys for executive buy-in

Since Panda went real-time, quality issues don’t necessarily cause obvious drops correlated with algorithm history dates

We also asked for free-text feedback and found some surprising priorities from non-SEOs

“The reviews seem fake”

Trust is a huge deal for real-world users

“There's not enough information about the company and why I should use their products”

On a micro-site that doesn’t have an “about” page

“In this day and age every page that has anything at all to do with business should be https”

Security is a big deal in B2B - even without on-site purchases

“The pictures were of low quality and blurry”

We know this matters to users. It’s at the easier end of ML detection

Controversial

Accepted

Site speed measured directly

​

Real User Metrics including site speed from Chrome

https://moz.com/blog/google-chrome-usage-data-measure-site-speed

https://moz.com/blog/google-chrome-usage-data-measure-site-speed

First announced at SearchLove San Diego last year

“The Chrome User Experience Report is powered by real user measurement of key user experience metrics across the public web, aggregated from users who have opted-in to syncing their browsing history, have not set up a Sync passphrase, and have usage statistic reporting enabled.”

Chrome User Experience Report documentation and announcement

Q: Is CrUX data being used to assess page speed as a ranking factor?

​

A: “I don't remember where the data is coming from, but not our publicly available tools”

Soft denial by Gary Illyes in his AMA

Controversial

Accepted

CTR as direct ranking factor

​

This has been explicitly denied by Google reps.

https://www.slideshare.net/randfish/mad-science-experiments-in-seo-social-media

MozCon 2014

We’ve seen similar things in-person with experiments at conferences

[Note: others have tried to reproduce results more recently including this failure to do so by Bartosz]

More:

https://sparktoro.com/blog/queries-clicks-influence-googles-results/

But: not everyone agrees

https://www.reddit.com/r/TechSEO/comments/ao3fmk/i_am_gary_illyes_googles_chief_of_sunshine_and/

“In the most general, broadest way - without attempting to skirt, skip, duck or dive it, does Google use interactions of searchers to alter what position certain results may hold? Those actions could include, but aren't limited to; CTR, return and click another listing, adjusting the query, repeat search and repeat visit etc.”

​

-- /user/Darth_Autocrat

“we primarily use these things in evaluations.”

“Can you please confirm/deny whether [RankBrain] uses UX signals of any kind?”

​

-- /user/Darth_Autocrat

“Dwell time, CTR, whatever Fishkin's new theory is, those are generally made up crap”

“Dwell time, CTR, whatever Fishkin's new theory is, those are generally made up crap”

A little devil’s advocacy: are there any charitable interpretations?

​

​

Technically correct

Not exactly clickthrough rate

​

[Something more like “difference between expected and observed CTR”?]

Technically correct

Not exactly a “ranking factor”

​

[They call it something more like a “post-evaluation modifier” or something?]

Dodging the question

“primarily use”

​

[and maybe limiting the “crap” comments to RankBrain?]

Dodging the question

“primarily use”

​

[and maybe limiting the “crap” comments to RankBrain?]

Dodging the question

“primarily use”

​

[and maybe limiting the “crap” comments to RankBrain?]

1. They aren’t marketers

• They aren’t marketers
• Their interests aren’t aligned with yours

• They aren’t marketers
• Their interests aren’t aligned with yours
• They “don’t understand what it’s doing exactly”(*)

​

(*) Haahr on RankBrain

I gave a presentation here last year:

“From the horse’s mouth”

​

Slides & Video

But grrrr:

​

a) their job is supposed to be to help webmasters

and

​

b) calling it “made up crap” is unprofessional and discourages the vulnerable

• They aren’t marketers
• Their interests aren’t aligned with yours
• They “don’t understand what it’s doing exactly”(*)

​

(*) Haahr on RankBrain

Remember:

Some SERP interaction as some kind of ranking factor

​

My theory.

We sell custom cigar humidors. Our custom cigar humidors are handmade. If you’re thinking of buying a custom cigar humidor, please contact our custom cigar humidor specialists at custom.cigar.humidors@example.com

Keyword stuffing: Doesn’t mean keyword density is the correct metric

Controversial

Accepted

Controversial

Accepted

Controversial

Accepted

Future

Controversial

Accepted

Future

More intent measurement?

Controversial

Accepted

Future

More intent measurement?

​

[“Needs met” is already the primary metric]

Controversial

Accepted

Future

More intent measurement?

Measure task completion?

Controversial

Accepted

Future

More intent measurement?

Measure task completion?

Consider conversion rate?

So, given all of this, ALL we need to do is optimise for UX then?

Um. No.

Evidence later

Google makes mistakes

Haahr talked about evaluating algorithms on “next page click rate”

Adding white space gamed that metric

It’s also really hard, and people are confusing

“There are so many experiments that we’ve done that have very misleading live metrics that you really need to dig into”

“We had a long-running experiment that swapped results two and four”

“My guess is more people would scroll down and click on position three”

“...actually more people click on [position] one”

https://youtu.be/eGSGZMI4Z_I?t=1024

It is easy to think you have shown something, but find it full of confounding details when you dig into the individual results

We have seen the same thing with our testing

But we don’t need the futurist perspectives or the tinfoil hat conspiracy theories for my purposes here

We just need to know that UX is aligned with Google’s goals

1. Panda (ML-trained on UX signals)

​

2. Quant and qual measurement of algo changes using UX metrics

We just need to know that UX is aligned with Google’s goals

We should be building SEO hypotheses from UX fundamentals

If we’re talking UX, we’re talking CRO:

Conversion Rate Optimisation

Most organisations treat them entirely separately

SEO

CRO

Search Engine Optimisation

Conversion Rate Optimisation

But in truth there is a lot of overlap

SEO

CRO

While some SEO changes don’t impact CRO

SEO

Meta info - invisible to UX

CRO

And some CRO changes don’t impact SEO

SEO

Non-indexed pages

CRO

An awful lot of changes affect both

“How do we know the tests won’t hurt conversion rate?”

Never: “how do we know our CRO tests aren’t hurting rankings?”

Anyway. You ought to worry about BOTH

When CRO messes up SEO

When CRO messes up SEO

-25%!

I wrote a post at moz.com about how big an impact this can have

You think this is what happens as you roll out winning CRO tests

But you don’t realise that some of them are bad for search

What if there was a way to GET the best of both?

You can do better if you only roll out the true winners

SEO split testing is template based

Our solution: full funnel testing

Conversion rate optimisation focuses on converting more of your existing traffic (bottom of the funnel)

Conversion Rate Optimisation

Conversion rate optimisation focuses on converting more of your existing traffic (bottom of the funnel)

Search engine optimisation focuses on getting additional traffic to your website (top of funnel)

Conversion Rate Optimisation

Search Engine Optimisation

Conversion rate optimisation focuses on converting more of your existing traffic (bottom of the funnel)

Search engine optimisation focuses on getting additional traffic to your website (top of funnel)

Full funnel testing combines these approaches to do both at the same time resulting in higher ROI from testing

Conversion Rate Optimisation

Search Engine Optimisation

Full Funnel Testing

How does it work?

SEO split testing is template based

Here’s how it works - on an example website

Home

Animals

Cats

Dogs

Unicorns

Badgers

Cats

Cats

We want to test two templates for our animals pages

​

​

Cats

Dogs

Unicorns

Badgers

CRO test - different users see different templates on the same pages

1

Cats

Dogs

Unicorns

Badgers

CRO test - different users see different templates on the same pages

1

2

Cats

Badgers

Dogs

Unicorns

Cats

Badgers

SEO test - different users see the same templates on DIFFERENT pages

1

2

Cats

Badgers

Cats

Badgers

Cats

Badgers

Unicorns

Dogs

SEO test - different users see the same templates on DIFFERENT pages

Cats

Badgers

Unicorns

Dogs

Cats

Badgers

Unicorns

Dogs

1

2

The site looks like this during the test

Home

Animals

Cats

Countries

Badgers

SEO testing

CRO testing

vs.

Google’s view

Animals

Cats

Dogs

Badgers

A

B

B

A

User view

Animals

Cats

Dogs

Badgers

A

B

B

A

?????

Animals

Cats

Dogs

Badgers

A

B

B

A

Landing page established

What do unicorns eat

SEO test complete

Animals

Cats

Dogs

Badgers

A

B

B

A

What do unicorns eat

Cookied

A

A

A

A

What do unicorns eat

Animals

Cats

Dogs

Badgers

Second page view

Animals

Cats

Dogs

Badgers

A

A

A

A

What do unicorns eat

Third page view

Animals

Cats

Dogs

Badgers

A

A

A

A

What do unicorns eat

Sidenote:

This just makes better SEO tests anyway BTW

By avoiding an inconsistent user experience as they browse the site

In summary

Not everyone agrees

Controversial

Accepted

But we don’t need the futurist perspectives or the tinfoil hat conspiracy theories for my purposes here

We just need to know that UX is aligned with Google’s goals

1. Panda (ML-trained on UX signals)

​

2. Quant and qual measurement of algo changes using UX metrics

We just need to know that UX is aligned with Google’s goals

To know that we should be building SEO hypotheses from UX fundamentals

BUT: then we affect users and search

SEO split testing is template based

Which is why I believe the future of SEO is testing

Thank you

​

Will Critchlow, CEO Distilled

@willcritchlow

​

With help, ideas, slides and more from @craigbradford, @dom_woodman and @tomanthonyseo

Questions: @willcritchlow

Image credits

• Hornet
• Beach
• Paul Haahr
• Google
• Smile
• Pug
• Chrome
• Tinfoil hat
• Broken plate
• Confused monkey
• Unicyclist
• Horse

• Phone
• Devil cat
• Coloured smoke
• Berlin