Google: Keep The Default Setting in Robots.txt

Google: Keep The Default Setting in Robots.txt

Somebody asked Google’s John Mueller in Google Webmaster Help thread about if one should allow or block access to the admin portion of their WordPress site.

John replied, that it would be better if you keep the default setting for your robots.txt access control unless you are aware of it.

John wrote:

For the most part, unless you’re aware of something specific that you need to change, I’d recommend leaving the robots.txt file to the default that your website’s content management system has. If you’re curious about how the robots.txt file works, you might want to take a look at https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt too.

You can see the whole discussion at Google Webmaster Help.

Source: http://bit.ly/2iHcElo