Age of Article Warning:
This article was originally published 39 months ago. The information, tips and techniques explained may outdated. Examples shown on this page may no longer work. Please consider this when viewing the below content.
This is the question I was asking myself a couple of days ago. I have a WordPress responsive website, and I know that it works well on mobile devices. So why would Google’s mobile friendly tester show my website as NOT friendly?
Firstly let’s explore why you need a mobile friendly website.
Google’s upcoming algorithm change
Well according to the Google Webmaster blog in a post titled “Finding more mobile friendly search results“, Google will very soon be making a change to the search results displayed on a mobile device.
From the article:
“Starting April 21, we will be expanding our use of mobile-friendliness as a ranking signal. This change will affect mobile searches in all languages worldwide and will have a significant impact in our search results. Consequently, users will find it easier to get relevant, high quality search results that are optimized for their devices”
The key words here are “… will affect mobile searches … worldwide … will have a significant impact in our search results …”
I think you get the picture that if you want to appear when someone is doing a Google search on a mobile device after 21st April 2015, then you need to be mobile friendly.
How to be mobile friendly
Well the preferred way to be mobile, according to Google is to have just one version of your website that is a Responsive Website Design.
What reasons does Google give for this (source: why responsive design):
easier for people to share a single link to a page on your website, rather than multiple links to the same content
easier for Google to index a single url than a separate desktop and mobile url
only have one page for the webmaster to maintain
saves on load time when redirection is required based on device type
improves Googlebot’s crawling efficiency as only one version of a page to be crawled and indexed so “can indirectly help Google index more of your site’s content and keep it appropriately fresh”
So if you are running a WordPress website, then simply choose a responsive theme. You can also use a mobile plugin that creates a mobile version of your website, however given the reasons listed above you would be better off using a responsive theme. There are plenty to choose from and sometimes the theme you are using may actually have a setting to turn on responsive design, so if you are not sure then check the documentation that comes with your theme, or alternatively contact your theme provider.
How to test if your website is now mobile friendly?
This step is nice and easy, since Google have provided us with a really easy tool to use. Simply visit test here and enter your home page url for Google to test.
Another way to test is by setting up your website with Google Webmaster Tools, and view the “mobile usability” section for any errors that Google has found.
What if I have a responsive website and Google is showing it as “not friendly”?
Well this is what I found out the other day. Even though my website is mobile friendly through testing on a range of mobiles and tablets, Google seems to be of the opinion that it is not.
Why is it so?
So I checked using the Chrome developer tools, and switched on the device mobile emulator to verify that the website was still appearing correctly on mobile devices. Yes that checked out ok.
So even though the website is fully responsive, the reason why Google believes it is not is because the robots.txt file blocks the files that Google now wants to be able to view.
So what to do? Do I simply remove the robots.txt file and allow all crawling of the website? Do I amend the file and stop blocking all the files?
A new robots.txt file for Googlebot
Thankfully we can target Googlebot in our robots.txt using “User-agent: Googlebot”. Also although the directive “Allow” is not recognised across all crawlers (so it shows as an error in robots txt testing tools), Google does actually recognise it, provided that it is “the URL path in of a subdirectory, within a blocked parent directory, that you want to unblock” (source: Webmaster tools help).
This is my revised robots.txt tool with an explanation below:
User-agent: Googlebot Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Allow: /wp-content/plugins/*.css Allow: /wp-content/plugins/*.js Disallow: /wp-content/mu-plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Allow: /wp-content/themes/*.css Allow: /wp-content/themes/*.js Allow: /wp-content/themes/*.gif Allow: /wp-content/themes/*.png Allow: /wp-content/themes/*.jpg Disallow: /*.php$ Disallow: /*.inc$ Disallow: /*? User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/mu-plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Disallow: /*.php$ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*? Sitemap: https://www.davidtiong.com/sitemap.xml
- Lines 1-17 are for Googlebot
- Lines 19-31 are for all other bots
- Line 33 is to inform search engines where my sitemap is located
- Line 4 blocks the directory “plugins”, but lines 5-6 allow Googlebot to access css and js files in there
- Line 9 blocks the directory “themes”, but lines 10-14 allow Googlebot to access css, js and image files in there
You can add other allow or disallow rules in there as needed using the same pattern, however remember that your robots.txt file is publicly available so be careful what you put in there, eg: don’t put important pathways in there to hidden files.
Submitting the new robots.txt file to Google and testing
Once you have uploaded your new robots.txt file to your website, you want to do the following:
2. Purge any old copy of your robots.txt file from your CDN or Cloudflare if applicable
3. Test your robots.txt file for errors – try tool.motoricerca.info/robots-checker.phtml (remember that the allow directive will show as errors)
4. Tell Google about your new robots.txt file by going into your Google Webmaster Tools account, then navigate to Crawl > Robots.txt tester. Then click on the submit button, then next to 3 ask Google to update – click submit (see images below).
5. Check that css, js and image files are not being blocked by Googlebot anymore by using the Robots.txt tester, view the images below as an example.
6. Check that your website is now mobile friendly using the mobile friendly tester – www.google.com/webmasters/tools/mobile-friendly/.
7. Keep an eye on your Google webmaster tools account over the next couple of weeks to make sure that no new errors are showing up.
How did you go? If you need to print out this page to refer to, then click the button below.