Strange issue with search engine crawler in amp version of web site

Hello, we noticed there is a strange issue with AMP version of websites.

After debugging a few we found this path: plugins/td-composer/mobile/header-amp.php
The thing is that the search bar echoes this snippet of code:

<amp-img src=”{{post_thumb_url}}”

And then replaces the {{post_thumb_url}} at runtime. Unfortunately this confuses search engine crawlers that interpretes this as an URL so we found a lot of 404 like this:

https://www.mywebsite.com/path/to/article/{{post_thumb_url}}
https://www.mywebsite.com/path/to/article/{{post_cat_link}}
…

We did a fast research on google and found that other people have the same issue:

https://itmedialaw.com/en/page/3/?s=%7B%7Bpost_url%7D%7D%2Fpage%2F4%2F%7B%7Bpost_cat_link%7D%7D%2Fpage%2F5%2F%7B%7Btd_query_all_results%7D%7D%2F%7B%7Bpost_cat_link%7D%7D&amp

https://mbaretimes.com/search/%7B%7Bpost_cat_link%7D%7D/

https://universityobserver.ie/search/people/page/169/%7B%7Bpost_cat_link%7D%7D/%7B%7Bpost_url%7D%7D/%7B%7Bpost_cat_link%7D%7D

So, this is unlikely because of two factors:
1) CPU overload with huge amount of 404 (3-4 for each blog post, that is dramatic for a newspaper)
2) Waste of crawling budget for search engine => less SEO friendly website.

Please, can you investigate this bug and provide to fix in next Theme releases?
Thank you,

Silvio (Akaueb Srls)

• This topic was modified 6 years by Akaueb.
• This topic was modified 6 years by Akaueb.

Hi Calin,
thank you for response! I think this is irrelevant for bots, because also if we declare the “nofollow” directive it is… well a directive.

Bots like semrush ignores this directive and hits 404 page. The real matter is about performance because with something like 10.000 blog posts we will generate about 40.000 times the 404 page and this is an unnecessary overhead for CPU.

For Now we solved removing those snippets of code:
https://imgur.com/2kVdbeE
https://imgur.com/1lBmHy1

This disables the search feature and avoid Crawlers to follow those invalid url, but doing this make us necessary to remake this edit each time we update the theme.

As a compromise fix we will also like something like a WordPress filter to disable feature programmatically or a Switch in admin panel, we don’t know what is better for you, it doesn’t matter, our goal is only to inform you about this issue and avoid CPU performance waste on our server.

I’m sorry but we can’t find old 404 logs for this issue but I think that this Google result is eloquent about the real behavior of Crawlers:
https://www.google.com/search?q={{post_cat_link}}

As you see there are a lot of result from website with Newspaper Theme.
Let me know your technical feedback,

Thank you for your time,
Silvio

• This reply was modified 6 years by Akaueb.