The Good Bot, the Bad Bot, and the Next Era of Commerce

Written by Gavin Mckew

Director of Shopify Practice

The Good Bot, the Bad Bot, and the Next Era of Commerce

For years, online stores treated bots as a nuisance. CAPTCHAs, IP blacklists, and fraud checks were designed to keep automated traffic out. But now, not all bots are bad. SEO crawlers, QA tools, and accessibility checkers are essential to how the web functions. And in the age of AI, a new category has emerged: answer engines and agentic shopping bots. These aren’t just crawling sites they want to interpret, summarise, and even transact.

This shift forces an uncomfortable question: “How do we distinguish between the good bots that help merchants and the bad bots that scrape, exploit, or bypass checkout altogether?”

Platforms like Shopify, working closely with Cloudflare, are laying down new rules to shape the answer.

The outcome will define who controls visibility, data, and checkout in the next era of commerce.

Cloudflare’s Permission First World

Cloudflare recently introduced a structural change to how bots interact with websites. The open web’s default used to be: anyone can crawl unless blocked. That assumption is gone. Now, new domains added to Cloudflare are set to block known AI crawlers unless the site owner explicitly allows them. It’s a move from open to permission first.

The list of AI crawlers targeted covers some of the biggest players in AI and search: Amazonbot, Applebot, GPTBot, ClaudeBot, TikTokSpider, Bytespider, and more. These crawlers are denied access by default, unless granted an explicit exception. In parallel, Cloudflare has introduced a “Pay Per Crawl” system, sending back 402 Payment Required responses to AI bots unless there’s an arrangement in place.

class= — Cloudflare crawler request routing

This enforcement doesn’t rely on politeness protocols like robots.txt. It’s edge-level control using fingerprinting, machine learning, and techniques like JA4 to distinguish bot traffic from real human sessions. Bots trying to mask themselves can still be identified and either throttled or blocked outright.

For Shopify merchants, this matters because Shopify storefront traffic is routed through Cloudflare. That means these AI blocking rules apply directly to Shopify stores, without the merchant necessarily realizing it.

Shopify’s Answer: Web Bot Auth

Shopify has responded by rolling out Web Bot Auth keys. These are cryptographic signatures that merchants can generate in their admin panel. A trusted crawler or tool can then present that key when accessing the store. If the signature matches, the bot is given free passage. If not, it risks being slowed, blocked, or denied.

In practice, this gives merchants more control, but it also hands them a new responsibility: deciding which bots to let in. Googlebot and Bingbot don’t need this; they remain whitelisted to ensure SEO works. But AI crawlers, experimental shopping agents, and even popular auditing tools may soon require explicit authorisation. The risk of staying closed is missing out on AI-driven visibility. The risk of opening up is feeding data into systems that may not yield a return on investment.

Protecting Checkout

One of the deeper drivers of these changes is the protection of the checkout process. Shopify’s strongest product is Shop Pay. It delivers conversion rates, wallets, BNPL options, and is strategically central to how Shopify anchors merchants to the platform. If AI agents could complete checkout flows on behalf of users without passing through Shopify’s rails, Shopify would lose control of its moat.

The nightmare scenario is an LLM or AI shopping assistant filling carts and submitting payments directly, bypassing all the guardrails. Fraud checks, device fingerprints, and CAPTCHAs are designed around human behaviour. Loosening them for “good” bots risks letting in “bad” bots. By insisting that agents prove themselves through Web Bot Auth and keeping checkout gated, Shopify can preserve Shop Pay as the mandatory route for transactions.

The Merchant’s Trade Off

Merchants are being pushed into a difficult middle ground. On one hand, they want discovery. Appearing in Google results remains crucial, and the same logic will apply to AI answer engines and shopping assistants. If their content isn’t accessible, they risk disappearing from the new discovery layer. On the other hand, they don’t want to give away product data, pricing, or FAQs to models that could aggregate and undercut them.

Some merchants may choose to gate even basic content like About Us pages, FAQs, and product details behind login or controlled APIs. Others will swing the opposite way, opening up access in the hope of being surfaced by emerging AI channels. Both choices carry risk. The balance is no longer about SEO versus no SEO; it’s about whether the AI economy values you as a data source or commoditises you.

The Layers of Control

What I think is happening under the hood is increasingly granular:

Thresholds: bots are allowed until they cross behavioural thresholds, at which point they’re throttled or blocked.
Proxy detection: Using methods like JA4 it would be possible for Shopify to identify things like using residential proxies and potentially apply different rule sets to good and bad bots alike
Predictions: My guess is that Web Bot Auth and default blocking are only phase one. Reporting tools, API level permissions, and monetisation layers are likely being developed.

My thoughts are that this creates a dynamic system rather than a static blocklist. Something like a bot that behaves well today may be throttled tomorrow if it overreaches.

Data Wars and Moat Building

This moment is part of a broader movement. Publishers, platforms, and infrastructure providers are all circling the same issue: how do you protect your data moat in an AI-first world? News publishers are gating content or striking licensing deals. Forums like Reddit and Stack Overflow have begun charging for access. Commerce platforms are doing the same, but with an added twist: they aren’t just defending content, they’re defending checkout rails.

The strategy is consistent: if AI firms want access, they need to come through sanctioned channels. No more free harvesting. Visibility becomes a currency, brokered through standards, APIs, and agreements.

Futures of Agentic Commerce

There are a few plausible outcomes:

Block: AI agents are prevented from completing checkout at all. They may help users browse, but transactions always require human confirmation.
Partner: Platforms offer sanctioned APIs and safe rails. AI agents can transact, but only by routing through Shop Pay or equivalent, where fraud protections and merchant rules apply.
Own: Platforms go further, setting themselves up as the central broker of agentic commerce, controlling not only checkout but discovery too.

The likely reality will be a hybrid of partnership and ownership. Total blocking would cut merchants off from new demand. Total openness would erode platform moats. The middle ground is to make AI play by platform rules.

Why This Matters

This is not just a technical tweak. It’s a strategic realignment of the web. The open, crawlable internet is giving way to a permissioned, negotiated layer where access depends on standards and agreements. Merchants, platforms, and AI firms will all have to adjust.

For merchants, the key question is no longer whether bots will interact with their stores. They will. The real question is: which bots should be allowed, under what terms, and through whose rails? The difference between a good bot and a bad bot is no longer binary it is strategic.

Conclusion

The arrival of Web Bot Auth, Cloudflare’s permission-first stance, and Shopify’s protective positioning all point in the same direction. The age of the good bot and the bad bot has begun. Commerce platforms are racing to define the standards for who gets in and who stays out. The outcome will shape visibility, discovery, and checkout for years to come.

We are watching the next era of commerce emerge in real time. The decisions being made now about access, gating, and standards will determine who holds power in an AI-driven shopping world. Merchants need to pay attention. The choice of which bots to trust is no longer background noise, it is a front-line decision in the fight for the future of eCommerce for both Product teams & Merchants.