SpotAisleBot

How it identifies itself

Every request carries this user agent:

Mozilla/5.0 (compatible; SpotAisleBot/1.0; +https://spotaisle.com/bot)

When it reads a store

• When the store's owner installs SpotAisle from the Shopify App Store. The app checks the store's product data and reads the store's public pages to compare them with it. Each check reads one page per product up to the owner's plan limit (section "How fast"), and the owner can run another check after making a change.
• When a store's owner has agreed to try our checks before installing. We run the same checks from our own computer, with the same limit on requests at a time and the same retries (section "How fast"). That check is not bound by a plan's limit: it reads every product in the store's public product list except gift cards and products priced at zero, each product's page and its public product data. The report is deleted from that computer within 24 hours of the check (if that computer is off, asleep or signed out at that point, within an hour of it being used again), and is sent to the owner by email or chat.
• Outside an install, only our own development and test stores, and a small number of stores whose owners have agreed to try our checks. We read them a few times a month, at the same low rate.

It never reads a store to train an AI, to build a list of products for anyone else, or to resell data. The results of a check are shown only to the store's owner: in the app, or by email or chat for a store whose owner agreed to try our checks.

What it reads

The home page, robots.txt, the sitemap, product pages, and Shopify's public product data (/meta.json, /products.json and /products/<handle>.js). It never opens the cart, the checkout, customer accounts, orders, or any page that needs a login; the one exception is our own development store, whose storefront password our test runs enter. It keeps the facts it checked (for example whether a product page carries structured data, and the price and stock it shows), not copies of your pages; those facts live on our servers, or, for a check a store's owner agreed to, in the report on the computer that ran it (deleted within 24 hours, or within an hour of that computer next being used if it was off, asleep or signed out) and in the email or chat message that sends it to the owner.

How fast

At most three requests at a time to one store, with two retries at most. A check of an installed store reads one page per product up to the plan's limit (25 products on the Free plan, 500 on Starter, 5,000 on Growth and Agency). Every check also reads up to 50 product sitemap files.

When a store answers 429, 430 or 503, or does not answer at all (a timeout or a network error), SpotAisleBot waits before each retry: the time in the Retry-After header, up to 30 seconds; if a store asks for a longer wait, SpotAisleBot stops and reports "couldn't check"; or 1 second and then 2 seconds when there is none; after the second retry it stops and reports "couldn't check".

How to stop it

Add this to your robots.txt:

User-agent: SpotAisleBot
Disallow: /

SpotAisleBot follows rules addressed to SpotAisleBot, and otherwise the rules for *. A page your robots.txt disallows is not read, and the check reports it as "not checked: your robots.txt asks our bot not to read this page". If your robots.txt cannot be read (a server error, a redirect, a file SpotAisleBot could not use, or the store limiting our requests), SpotAisleBot reads nothing on that store. If your robots.txt sets a Crawl-delay for SpotAisleBot, or otherwise for *, SpotAisleBot leaves at least that many seconds between its requests to your store, however long the delay; if a check cannot finish within its time at that pace, it stops and reports "couldn't check (the store asks for a slower crawl)" rather than crawl faster.

Questions

Email support@spotaisle.com with "SpotAisleBot" in the subject. We reply on weekdays, 9am to 6pm India time (UTC+5:30), usually within one working day.