google webmasters search console

dear sirs,
Yesterday i received an email from google webmaster search console that there is an error (server 5xx) while fetching my new site (the url is wp-content/wp-themes/Newspaper.)

I added the following lines in robots.txt to avoid indexing this url
User-agent: Googlebot
Disallow: /wp-content/themes/Newspaper

Is this a correct step or must be allowed to google the theme folder?

Please let me know coz they stoped indexing my website in order to correct this.

Regards

I am plagued with the same error. Memory limit on the WordPress install is actually 512 MB. So this symptom that keeps showing up now — as I can see — at many of your users has a different cause.

When I try to ‘fetch as google’ the url wp-content/wp-themes/Newspaper in Search Console it gives me the information that it is ‘redirected’.
Checking the redirection givces the warning that that the address it is redirected to is unreachable. (It seems like for me that there si no difference in the original and redirected url. Also, I have not set a redirection for this url.)

Please fix this, my site has already fallen from Google search results because of this.

ok redirection comes from the difference between …/folder and …/folder/ path difference, probably not an issue.
Still it is interesting Google Search Tools would list this as a serious issue only for a few days (and it coincides the exact time the site has disappeared from Google search results)

as for temporal solution I tried adding this code to robots.txt:
Disallow: /wp-content/themes/Newspaper/index.php

but I am not sure it will not make the problem only worse.

Basically, when you change themes, or do a major update to WP and to caching plugins, etc., Google might be loading your sitemap or following a link, or your sitemap might actually stop working due to plugin, etc., and briefly Google is following links to whatever it can load via the regex/rewrite rules from WP in the htaccess file, and this can cause Google to try to load the /site/wp-content/themes/theme/index.php file which it should really never do. I’ve poked the WP folks numerous times the past decade to consider changing that to default.php vs index.php as it’s NOT actually an index for that directory, and should never be loaded by anybody.

Technically, one solution — for those who have the time and patience — is to go into the folder, and rename the index.php file as a copy to all the missing ‘default’ templates that WP would normally fall back on, so that WP has no need for ever using that file; then inside the index.php file put a redirect to http://yoursite.ooo — or simply remove the file entirely.

Also keep in mind getting a server error for that URL is actually correct — it’s an internal file and should never be loaded anyway; so Google is just letting you know that it temporarily got directed there during a crawl of your sitemap while the system was updating and it couldn’t go anyplace else and generated an error.

If you get one of these errors once in a while, no biggie; mark fixed and get on with your life. If you are getting them daily, you have a serious issue with your permalinks and configuration. If you see the errors in your domain/hosting error log, it may be people trying to hack the site and you can block those IPs from ever visiting your site again.

Upshot – nobody should be trying to access files directly in that folder, nor should Google, so a 500 error is just that; it’s just like getting a 404 error when you remove a page — it doesn’t hurt anything, or ruin your rankings, or anything else. That page is not part of your site, and not part of your navigation or site links or content.

It’s really a minor flaw in WP using ‘index.php’ for something that should be ‘default.php’ as default template when no other template exists. If there were no index.php file there, you’d get a 404 instead.

Not sure that helps; but I’ve posted about this topic several times the past 4+ years on here (I don’t work here, just a webdev dinosaur). 🙂

Thank you, that was a nice and thorough description, probably I panicked here for no reason. Just to make sure, I have tried this disallow thing in the robotics.txt for that very file in the theme folder and will see where it goes from there.
I am really sorry if I wasted the time of developers with a not too theme specific problem here. But maybe the thread will be useful for those who experience similar problems.

Again, thank you for the nice explanation, that was really useful, and a very clear one, too. Thanks for your time and effort!