Age of Article Warning:
This article was originally published 20 months ago. The tips and techniques explained may be outdated, or information may longer be applicable. Please consider this when viewing the below content.

This is the question I was asking myself a couple of days ago. I have a WordPress responsive website, and I know that it works well on mobile devices. So why would Google’s mobile friendly tester show my website as NOT friendly?

Firstly let’s explore why you need a mobile friendly website.

Google’s upcoming algorithm change

Well according to the Google Webmaster blog in a post titled “Finding more mobile friendly search results“, Google will very soon be making a change to the search results displayed on a mobile device.

From the article:
“Starting April 21, we will be expanding our use of mobile-friendliness as a ranking signal. This change will affect mobile searches in all languages worldwide and will have a significant impact in our search results. Consequently, users will find it easier to get relevant, high quality search results that are optimized for their devices”

The key words here are “… will affect mobile searches … worldwide … will have a significant impact in our search results …”

I think you get the picture that if you want to appear when someone is doing a Google search on a mobile device after 21st April 2015, then you need to be mobile friendly.

How to be mobile friendly

Well the preferred way to be mobile, according to Google is to have just one version of your website that is a Responsive Website Design.

What reasons does Google give for this (source: why responsive design):

easier for people to share a single link to a page on your website, rather than multiple links to the same content
easier for Google to index a single url than a separate desktop and mobile url
only have one page for the webmaster to maintain
saves on load time when redirection is required based on device type
improves Googlebot’s crawling efficiency as only one version of a page to be crawled and indexed so “can indirectly help Google index more of your site’s content and keep it appropriately fresh”

So if you are running a WordPress website, then simply choose a responsive theme. You can also use a mobile plugin that creates a mobile version of your website, however given the reasons listed above you would be better off using a responsive theme. There are plenty to choose from and sometimes the theme you are using may actually have a setting to turn on responsive design, so if you are not sure then check the documentation that comes with your theme, or alternatively contact your theme provider.

How to test if your website is now mobile friendly?

This step is nice and easy, since Google have provided us with a really easy tool to use. Simply visit test here and enter your home page url for Google to test.

Another way to test is by setting up your website with Google Webmaster Tools, and view the “mobile usability” section for any errors that Google has found.

What if I have a responsive website and Google is showing it as “not friendly”?

Well this is what I found out the other day. Even though my website is mobile friendly through testing on a range of mobiles and tablets, Google seems to be of the opinion that it is not.

Why is it so?

So I checked using the Chrome developer tools, and switched on the device mobile emulator to verify that the website was still appearing correctly on mobile devices. Yes that checked out ok.

Then I considered why the mobile testing tool might be showing a different result to Google. Reading on the page I realised that Google is trying to view the website exactly as a typical mobile user might see the page. So that means it needs access to all the stylesheets and javascript, images and other resources that might effect how a page looks.

Then it made sense. Based on the way that used to be recommended for WordPress websites, we use the robots.txt file to help prevent crawling and indexing of content that isn’t relevant for search results. In other words, we simply wanted Google and other search engines to crawl the text content of our websites, not all the other files. So the Googlebot was being blocked from crawling all the stylesheets and javascript, so it cannot therefore see the website exactly how it actually appears on a mobile device.

So even though the website is fully responsive, the reason why Google believes it is not is because the robots.txt file blocks the files that Google now wants to be able to view.

So what to do? Do I simply remove the robots.txt file and allow all crawling of the website? Do I amend the file and stop blocking all the files?

A new robots.txt file for Googlebot

I decided that the reason for blocking certain resources on the website is still relevant at this stage, though in the future this may change again. So the best way to fix the issue with Googlebot is to allow it to crawl the javascript, css, images and other important files. However I still may have some files loaded in my website that I don’t want crawled.

Thankfully we can target Googlebot in our robots.txt using “User-agent: Googlebot”. Also although the directive “Allow” is not recognised across all crawlers (so it shows as an error in robots txt testing tools), Google does actually recognise it, provided that it is “the URL path in of a subdirectory, within a blocked parent directory, that you want to unblock” (source: Webmaster tools help).

This is my revised robots.txt tool with an explanation below:

User-agent: Googlebot
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/plugins/
Allow: /wp-content/plugins/*.css
Allow: /wp-content/plugins/*.js
Disallow: /wp-content/mu-plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Allow: /wp-content/themes/*.css
Allow: /wp-content/themes/*.js
Allow: /wp-content/themes/*.gif
Allow: /wp-content/themes/*.png
Allow: /wp-content/themes/*.jpg
Disallow: /*.php$
Disallow: /*.inc$
Disallow: /*? 

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/mu-plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*?

Sitemap: http://www.davidtiong.com/sitemap.xml

Explanation:

  • Lines 1-17 are for Googlebot
  • Lines 19-31 are for all other bots
  • Line 33 is to inform search engines where my sitemap is located
  • Line 4 blocks the directory “plugins”, but lines 5-6 allow Googlebot to access css and js files in there
  • Line 9 blocks the directory “themes”, but lines 10-14 allow Googlebot to access css, js and image files in there

You can add other allow or disallow rules in there as needed using the same pattern, however remember that your robots.txt file is publicly available so be careful what you put in there, eg: don’t put important pathways in there to hidden files.

Submitting the new robots.txt file to Google and testing

Once you have uploaded your new robots.txt file to your website, you want to do the following:

1. Clear any caches to ensure your new robots.txt file is available

2. Purge any old copy of your robots.txt file from your CDN or Cloudflare if applicable

3. Test your robots.txt file for errors – try tool.motoricerca.info/robots-checker.phtml (remember that the allow directive will show as errors)

4. Tell Google about your new robots.txt file by going into your Google Webmaster Tools account, then navigate to Crawl > Robots.txt tester. Then click on the submit button, then next to 3 ask Google to update – click submit (see images below).

Submit robots.txt to Google

mobile friendly changes to robots.txt

5. Check that css, js and image files are not being blocked by Googlebot anymore by using the Robots.txt tester, view the images below as an example.

robots.txt tester

check Googlebot not blocked


6. Check that your website is now mobile friendly using the mobile friendly tester – www.google.com/webmasters/tools/mobile-friendly/.

7. Keep an eye on your Google webmaster tools account over the next couple of weeks to make sure that no new errors are showing up.

How did you go? If you need to print out this page to refer to, then click the button below.

Print a copy

Are you ready now for April 21?

Is my WordPress responsive website mobile friendly? was last modified: September 6th, 2016 by David Tiong
Is my WordPress responsive website mobile friendly?