BLOG

SHORT HILLS DESIGN BLOG

The Solution: Robots.txt for Wordpress - CSS and JS Update

Published:  July 31, 2015

Webmaster Tools/Search Console Notification:  CSS and JS Update and Fix

By now we've all read about the CSS and JS consideration regarding robots.txt.What I've attempted to do in this post is to share my thoughts on what the robots.txt should now look like for a generic Wordpress installation. If you would like more background on this issue - in some more technical detail - please read this blog post about robots.txt and CSS/JS.

What's the Fix?


There is a case to be made on both sides about the need to block: /wp-admin, and /wp-includes. There is a concern about having the contents of /wp-admin being indexed, and some risk for duplicate content. There is no one-size fits all robots.txt, and in fact there are some who recommend having an empty robots.txt file!
I do not have a final answer this aspect of the issue yet, but for now, the solution for this specific problem - at this time - is to add the following lines to the end of your robots.txt:

Allow: /*.css*
Allow: /*.js*

Re-Verify your robots.txt

Once you have made the change, make sure you resubmit your robots.txt so Google knows to "check you out" again. You can do that by logging into Webmaster Tools (search console) and navigating to "Crawl -> Fetch as Google". Click the "Fetch and Render" button, and run a test for "Desktop" and for "Mobile: Smartphone".

fetch-and-renderConclusion

Google has requested that website owners make .CSS and .JS files available for Googlebot in order to improve its search engine indexing. Accordingly, that's why we are making this change.
 

The Website Owner's Manual

Download our free 100+ page "website owner's manual" and learn what you and your team need to know about your website before you invest your marketing dollars

Download today

CONTACT US
© Copyright 2008-2021 - Short Hills Design, LLC - All Rights Reserved
envelopephone-handsetmap-marker linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram