Ie: https://www.interestinganswers.com/category/government/ and https://www.interestinganswers.com/category/sports , etc.
If not, is the best way to disallow this in robots.txt?
Just use Yoast SEO and there are several options in the plugin including to allow or disallow indexing of various elements. I do not allow indexing of my categories (this post might help you make a decision https://www.shoutmeloud.com/how-to-keep-wordpress-categories-and-tags-noindex.html).
Great thanks. I did not know that was an option within the plugin. I’ve been allowing these to index for about a year now and not really knowing it. I guess Google will pick up on this without me getting penalized,correct?
Just monitor your Google search console and read more about the topic online.
Thx. Should my robots file be updated as soon as I make the changes in yoast? It’s not being done.
Read the docs for Yoast, as good starting point. Most such plugins use a virtual robots.txt file.
Oh. I didnt see anything in Yoast about a virtual robots file. Now that I know that is a real thing, I will Google that. thanks
I didn’t see any mention of that. Oh well, We’ll see if it works.
You don’t need to use a robots.txt file….
Read the documentation on how to use Yoast, for more info.
https://yoast.com/wordpress-robots-txt-example/
Should you link to your XML Sitemap from your robots.txt?
We’ve always felt it pointless to link to your XML sitemap from your robots.txt file, because you should add your sitemap manually to your Google Search Console and Bing Webmaster Tools accounts and look at their feedback about it. This is why our Yoast SEO plugin doesn’t add it to your robots.txt. Don’t rely on search engines finding out about your XML sitemap through your robots.txt .
https://kb.yoast.com/kb/how-to-edit-robots-txt-through-yoast-seo/
