By now we've all read about the CSS and JS consideration regarding robots.txt.What I've attempted to do in this post is to share my thoughts on what the robots.txt should now look like for a generic Wordpress installation. If you would like more background on this issue - in some more technical detail - please read this blog post about robots.txt and CSS/JS.
There is a case to be made on both sides about the need to block: /wp-admin, and /wp-includes. There is a concern about having the contents of /wp-admin being indexed, and some risk for duplicate content. There is no one-size fits all robots.txt, and in fact there are some who recommend having an empty robots.txt file!
I do not have a final answer this aspect of the issue yet, but for now, the solution for this specific problem - at this time - is to add the following lines to the end of your robots.txt:
Allow: /*.css* Allow: /*.js*
Once you have made the change, make sure you resubmit your robots.txt so Google knows to "check you out" again. You can do that by logging into Webmaster Tools (search console) and navigating to "Crawl -> Fetch as Google". Click the "Fetch and Render" button, and run a test for "Desktop" and for "Mobile: Smartphone".
ConclusionGoogle has requested that website owners make .CSS and .JS files available for Googlebot in order to improve its search engine indexing. Accordingly, that's why we are making this change.
