yesterday i subscribed to stape, i uploaded 3 feeds in the merchant center and google started verifying the links. I specify that I have activated the Custom web GTM loader. After 24 hours I found myself with 65k requests on Stape:
While I have an average monthly traffic of 7-8k on the site:
Since we are based in Italy the 16k are justifiable as last night I also did the cache warmup by visiting 2830 links with a cronjob. But the 48k requests are clearly the google crawlers, even Awstats reports the last crawler access yesterday:
Therefore I wonder if google crawlers activate GTM server side and consume requests. If so, is there a way to avoid it?
We count any request to your server. So if google crawlers do request to your sGTM, then we also will count them.
I see from Screenshots that you are on the paid plan. That means you have access to log functionality. And you can check what exactly generated such many requests. From our experience, a lot of requests are used during the debugging process.
As I thought, Google bots are responsible by most requests sent to Stape, in fact in the logs these requests are with following User-Agent:
“request_User-Agent”: “Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)”
To remedy the problem I implemented the following solution to be applied in the GTM’s Client-Side container:
It should be possible to test the implementation by creating a custom user agent with one of the following extensions for Chrome:
Chrome UA Spoofer
Custom Useragent String
But as a comment explains in the following StackOverflow post:
The Useragent Change Thanks to The Extension" Custom Useragent String "Has Effect Only After The Page Is Rendered, Which Is Too Late
Therefore using these extensions the user agent is changed too late and all tags are already triggered. But when the User-Agent is actually that of AdsBot should work.
