Hello !
A robots.txt file that is not valid may cause issues for search engines trying to crawl your website, as it may not understand the directives specified in the file. To resolve this issue, you can check the syntax of the robots.txt file for any errors or typos and correct them. Check this guide: https://www.semrush.com/blog/beginners-guide-robots-txt/?kw=&cmp=EE_SRCH_DSA_Blog_EN&label=dsa_pagefeed&Network=g&Device=c&utm_content=631551526530&kwid=dsa-1941932855757&cmpid=18361923498&agpid=143776559020&BU=Core&extid=60109334146&adpos=&gclid=Cj0KCQiAx6ugBhCcARIsAGNmMbhI00QYAnIDIL8ZySX3eFAAq2mze3hc8rsGiEHpLi-7okrAxvf2sKIaAhWDEALw_wcB
Upload the updated robots.txt file to the root directory of your website and test to make sure that it is accessible and contains the correct directives.
After every modification, you need to reindex your website on Google. It may take some more time.
Thank you !
But from what i read online this is a perfectly fine command ? and i shouldnt do anything ?
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Sitemap: https://beirutlab.com/sitemap.xml
Sitemap: https://beirutlab.com/sitemap.rss
Would you please explain ?
Hello!
Check this guide too: https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/
Upload again the file and make sure you re-index your website.
Thank you!
