You'll see "Website access required" when your site's security answers our crawler with a challenge or a refusal instead of your page. It can show on the setup's Business step, right after you enter your website, and on any audit that couldn't read the site. Your visitors aren't affected. Your firewall is doing its job, and it needs to be told our crawler is welcome.
Our crawler sends this user agent on every request:
Mozilla/5.0 SearchBlueprintBot/1.0 (+https://searchblueprint.io/bot)
Match on SearchBlueprintBot. The crawler runs on cloud servers in Singapore with no fixed IP address, which is why every step below allows it by user agent. It follows robots.txt and never fills in forms, and when it's refused it doesn't come back pretending to be a browser.
Find out what blocked it
The notice names the service when we could tell, for example Cloudflare or Sucuri. In Cloudflare, open your domain's security events (Analytics, Events tab) and filter on the user agent SearchBlueprintBot. Each event names the feature that acted on it.
Cloudflare
The step depends on your plan.
On the free plan
The free plan's bot protection is Bot Fight Mode, and Cloudflare doesn't let any rule skip it. The only way through is to switch it off while the audit runs:
- In Cloudflare, open your domain and go to Security, then Settings. Filter by Bot traffic and turn Bot Fight Mode off.
- Back in SearchBlueprint, press Test access again, then run the audit.
- Turn Bot Fight Mode back on when the audit finishes.
On a paid plan
Pro, Business and Enterprise plans use Super Bot Fight Mode, which a custom rule can skip, so you can let the crawler in for good:
- In Cloudflare, open your domain and go to Security, then Security rules, and create a custom rule.
- Match on the field User Agent with the operator contains and the value
SearchBlueprintBot. Set the action to Skip and tick All Super Bot Fight Mode rules, then deploy it. - If one of your own custom rules also blocks bots, tick All remaining custom rules too and move this rule to the top of the list.
Other Cloudflare settings that can block it
- Your own rules. A rule that blocks user agents containing "bot", blocks Singapore, or limits request rates can catch the crawler. The security event names the rule, so you can exempt
SearchBlueprintBotfrom it. - Under Attack mode. It checks every visitor before they reach a page, and the crawler can't pass that check. Turn it off from your domain's Overview page once the attack is over.
- AI crawler settings. Block AI bots and the newer AI bot policies are aimed at crawlers that train AI models, feed AI search or act as AI agents. SearchBlueprintBot reads your site for your own audit, so you shouldn't need to change them. If a security event shows one of them blocking it, tell us.
Other firewalls and hosts
In Sucuri, Wordfence, Imperva, Akamai, AWS WAF, the Vercel firewall or your host's own firewall, add an allow rule for user agents containing SearchBlueprintBot. Some hosts run a bot filter in front of yours. If you've allowed the crawler and it's still blocked, ask your host to allow it too.
On SiteGround, you don't need to change anything. Its bot filter refused our crawler's earlier user agent, and the current one gets through. If you asked SiteGround to switch off a security rule for SearchBlueprint, you can ask them to switch it back on.
If your robots.txt disallows SearchBlueprintBot, or disallows every crawler, the audits follow it. Remove that rule to let them in.
Check that it worked
Press Test access again, on the setup's Business step or on the audit card. It asks for your homepage as SearchBlueprintBot and costs nothing. When it gets through, run the audit.
Not technical? Use Copy instructions for your developer. It copies a message with the user agent and the steps for your firewall, ready to send to whoever looks after your site.
Limits and usage
- The crawler is identified by its user agent. There's no IP address to allow.
- It never sends a browser's user agent to get past a block. Getting in is your decision.
- An audit that couldn't read your site uses no credits, and its result says so. A Technical SEO audit also keeps your last complete report.
Can I build my strategy while the site blocks the crawler?
Yes. Keywords and competitors come from Google's data, so the strategy still runs. What needs the crawler is anything that reads your own pages: their current titles and descriptions, and the site audits.
If you run it while the crawler is blocked, the run is not wasted. Once you let the crawler in, your next strategy run for that website is free.
My report says the site was audited through a verified crawler. Do I need to do anything?
No. The audit read your site through Cloudflare's own verified crawler. The one limit is that checks needing your server's response headers didn't run.