Do google bots consume Stape requests?

Google Tag Manager Server Container
1

Dear Stape community,

yesterday i subscribed to stape, i uploaded 3 feeds in the merchant center and google started verifying the links. I specify that I have activated the Custom web GTM loader. After 24 hours I found myself with 65k requests on Stape:

Stape - Google Chrome 2022-02-20 11.29.44

While I have an average monthly traffic of 7-8k on the site:

users
users1619Ă—862 103 KB

also inspecting the requests of the last 24 hours on Cloudflare, there are 48k requests from the USA and 16k from Italy:

cloudflare requests
cloudflare requests1275Ă—380 17 KB

Since we are based in Italy the 16k are justifiable as last night I also did the cache warmup by visiting 2830 links with a cronjob. But the 48k requests are clearly the google crawlers, even Awstats reports the last crawler access yesterday:

awstats
awstats1626Ă—68 7.41 KB

Therefore I wonder if google crawlers activate GTM server side and consume requests. If so, is there a way to avoid it?

Thank you.

2

We count any request to your server. So if google crawlers do request to your sGTM, then we also will count them.

I see from Screenshots that you are on the paid plan. That means you have access to log functionality. And you can check what exactly generated such many requests. From our experience, a lot of requests are used during the debugging process.

1 Like
3

As I thought, Google bots are responsible by most requests sent to Stape, in fact in the logs these requests are with following User-Agent:

“request_User-Agent”: “Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)”

To remedy the problem I implemented the following solution to be applied in the GTM’s Client-Side container:

Create a GTM Variable For the User-Agent

  • Navigate to Variables
  • Click to create a new User-defined Variable
  • Name it: JAVASCRIPT - USER AGENT
  • Select type of JavaScript Variable
  • Enter navigator.userAgent
  • Save

variable-useragent
variable-useragent1123Ă—461 26.8 KB

Create a Blocking Trigger with The User-Agent Variable

  • Navigate to Triggers
  • Create New Trigger
  • Select Pageview
  • Change Toggle to Some Pageviews
  • Pick JAVASCRIPT - userAgent from the variable list
  • Select “contains”
  • Enter AdsBot in the value field
  • Save

exeption-useragent
exeption-useragent1110Ă—311 18.4 KB

  • Publish the container

Tests

It should be possible to test the implementation by creating a custom user agent with one of the following extensions for Chrome:

  • Chrome UA Spoofer
  • Custom Useragent String

But as a comment explains in the following StackOverflow post:

The Useragent Change Thanks to The Extension" Custom Useragent String "Has Effect Only After The Page Is Rendered, Which Is Too Late

Therefore using these extensions the user agent is changed too late and all tags are already triggered. But when the User-Agent is actually that of AdsBot should work.

Fingers crossed.

2 Likes