issue with theme generating broken url to crawlers

Hi,

I spotted google and other crawlers started going for strange url

https://www.welovetennis.fr/%5C%22https:%5C/%5C/www.welovetennis.fr%5C/atp%5C/le-retour-de-nadal-et-federer-au-conseil-des-joueurs-bonne-ou-mauvaise-nouvelle%5C%22

As I couldn’t find where it was coming from, got a working copy

I’ve disabled all plugins and just kept td Composer and Cloud lib
(since reactivated wp no category base and categories order for the sake of making life easier)

and if I go for a

$ wget -e robots=off -r https://www2.welovetennis.fr/

after it crawls the categories appears those

https://www2.welovetennis.fr/%5C%22https:%5C/%5C/www2.welovetennis.fr%5C/atp%5C/le-retour-de-nadal-et-federer-au-conseil-des-joueurs-bonne-ou-mauvaise-nouvelle%5C%22

–2019-09-25 12:11:24– https://www2.welovetennis.fr/%5C%22https:%5C/%5C/www2.welovetennis.fr%5C/atp%5C/djokovic-se-dit-optimiste%5C%22
Réutilisation de la connexion existante à www2.welovetennis.fr:443.
requête HTTP transmise, en attente de la réponse… 301 Moved Permanently
Emplacement : https://www2.welovetennis.fr/%5C%22https:%5C/%5C/www2.welovetennis.fr%5C/atp%5C/djokovic-se-dit-optimiste%5C [suivant]
–2019-09-25 12:11:25– https://www2.welovetennis.fr/%5C%22https:%5C/%5C/www2.welovetennis.fr%5C/atp%5C/djokovic-se-dit-optimiste%5C
Réutilisation de la connexion existante à www2.welovetennis.fr:443.
requête HTTP transmise, en attente de la réponse… 404 Not Found
2019-09-25 12:11:25 erreur 404 : Not Found.

Any clue of where it could come from ?

update

urldecoded gives
\”https:\/\/www2.welovetennis.fr\/atp\/atp-zhuhai\/murray-retrouve-la-victoire\

scripts are not protected… gets trapped by crawlers … any solution ?

<a href=\”https:\/\/www2.welovetennis.fr\/atp\/atp-zhuhai\/murray-cest-lune-des-meilleures-victoires-que-jai-jamais-eues\” rel=\”bookmark\” class=\”td-image-wrap\” title=\”Murray: \u00ab\u00a0C’est l’une des m

• This topic was modified 6 years by cantoute.

Hi,

Yes I use a template from Cloud and I believe I’ve got something wrong on my header as it adds almost 1M of html when I compare http://www.welovetennis.fr and www2.welovetennis.fr for adding just 2 logos… I must have something in there. I’ll just recreate a header from scratch (I had based myself on Template 3 header of Cloud lib, I’ll try starting from blank template this time)

I didn’t see them in search console… yet…
But I could see yahoo, bing and google crawling them. And that yes is not good.
On a 150k pages website this is not helping us (and having wget doing the same is really not a good sign)

This is how I wasted a few hours.

I added this function in prod then put a tail -f on logs to check all was good
Then I spotted those broken url and thought my function was somehow responsable.

Then been hunting for couple hours checking all links. seeing wget -r was doing the same as the crawlers convinced me of investigating further.

// adapting ponctuation for french language
function filter_the_title_fix_ponctuation( $the_title, 5 ) {
//    $the_title = html_entity_decode($the_title);
//    $nbsp = html_entity_decode(" ");
    $thinsp = html_entity_decode(" ");
    $the_title = preg_replace('/ +([\?\!\:])/', $thinsp.'$1', $the_title);
    return $the_title;
}

Thanks for your help

PS
by the was, perhaps you guy could really benefit from using a private stackexchange … this forum isn’t very helpful when trying to find already answer questions…

• This reply was modified 6 years by cantoute.

Hi,

I now have google hitting me 50k /day on this type of urls

“https://www.welovetennis.fr/us-open/https :/www.welovetennis.fr/atp/atp-zhuhai”

Would be nice if had a look at the source of the page

https://www.welovetennis.fr/us-open/

and tell me if this does come from your theme or is it I should look elswere

To me it looks like I didn’t pick the right choice for the header (I based myself on cloud template model 3) and I would be better off using the “builtin” headers

I’m considering switching to Style 5

would that take all this json/js out of the way ?

to me it seems i’ll stay with the problem

the problem is to have href= or src= anywhere in the page (even in js comments it gets picked up) as the crawler scan the text without analyzing it’s structure to hunt for url.
Same happens with anything having //some.form/of/url

Now I have google crawling like crazy 100k pages / day on things we don’t want… getting really hard on the server… and we’ve got better things to ask google crawler to look for…

I have no issue giving you an access if that could help. But perhaps first if you could confirm that it’s the theme and no other plugin would be helpful.

Thanks

suggestions
“brake” anything like src= href= into ‘hr’ + ‘ef’ + ‘=’
or use a global var $hf=’href=’ $sc=’src=’ $ht=’http://&#8217;

or easy just all encode base64…

this can limit the problem at low cost (and won’t solve it completely) is the <![CDATA[

<script>
//<![CDATA[
…
<div class=\”td-module-image\”>\r\n <div class=\”td-module-thumb\”>edit<\/a><img width=\”218\” height=\”150\” class=\”entry-thumb\” src=\”https:\/\/www2.welovetennis.fr\/wp-content\/uploads\/2019\/09\/Tsitsipas1959-218×150.jpg\” srcset=\”https:\/\/www2.welovetennis.fr\/wp-content\/uploads\/2019\/09\/Tsitsipas1959-218×150.jpg 218w, https:\/\/www2.we

…
// ]]>
</script>

Hi,

For now I’ve solved the issue using a rewrite regex redirect rule taking much of the load off

But coming to my mind something I was thinking about… do you think it’s possible to imagine that this json data could come from an get call (from api ?)
That would take it out of the source but there is event a better benefit that could be made of it: it could then be possible to cache the menu separately… making it possible to cache post pages much longer and still having an up to date megamenu with the latest news… just one cache updated updates the menu of the entire site.
just a thought…

thanks for your attention

all the best

those url will not show in google console as they will be filtered out… but doesn’t mean google crawler isn’t hunting for them !!!

edit:
for now a rewrite rule lets google crawler get 301 fast… so this is under control

• This reply was modified 6 years by cantoute.

here problem solved…

the problem came from the “preloaded” megamenu that using template 3 from cloud templates and I couldn’t disable it.
Since then switched to theme headers and disabled preloading of the megamenu and the page weight dropped 🙂