What is Webz.io? How to allow or block it

At a glance

Operator: Webz.io
Category: Training
Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live.
User-Agent token: omgili
MetaPilot also counts these user agents as Webz.io: omgili, webz.io
robots.txt token: Omgilibot
Respects robots.txt: Documented as compliant
The robots.txt token and the user agents do not match: you will see omgili or webz.io in your logs, but the token to disallow is Omgilibot — a leftover from the Omgili-to-Webz.io rename. Webz.io no longer serves the old crawler information page it used to link from its user agent.

What Webz.io does

Webz.io is a web-data provider rather than an AI company: it crawls the open web — with a long-standing focus on forums, message boards and other discussion content — and sells the results as feeds and datasets. Buyers include companies that train models on them, so this is a training-adjacent crawler one step removed. It states its crawlers adhere to robots.txt exclusions.

Should you block Webz.io?

Blocking keeps your content out of a dataset resold to third parties you will never see, which is the whole reason to block it. As with any pure data broker there is no answer engine at the other end, so nothing about your search or AI visibility changes either way.

How to block Webz.io

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the Omgilibot token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block Webz.io

robots.txt

User-agent: Omgilibot
Disallow: /

Allow Webz.io explicitly

robots.txt

User-agent: Omgilibot
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles Webz.io

MetaPilot detects Webz.io in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Webz.io between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

ClaudeBot

Anthropic

Training

Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.

Applebot-Extended

Apple

Training

Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models.

Bytespider

ByteDance

Training

ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok).

CCBot (Common Crawl)

Common Crawl

Training

The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on.

Browse every AI crawler

FAQ

What is Webz.io?

Webz.io's crawler (historically "Omgilibot"), collecting public web and forum content for the datasets it sells.

Webz.io is a web-data provider rather than an AI company: it crawls the open web — with a long-standing focus on forums, message boards and other discussion content — and sells the results as feeds and datasets. Buyers include companies that train models on them, so this is a training-adjacent crawler one step removed. It states its crawlers adhere to robots.txt exclusions.

Does Webz.io respect robots.txt?

Yes — Webz.io documents that Webz.io honours robots.txt. A Disallow for the Omgilibot token is respected.

The robots.txt token and the user agents do not match: you will see omgili or webz.io in your logs, but the token to disallow is Omgilibot — a leftover from the Omgili-to-Webz.io rename. Webz.io no longer serves the old crawler information page it used to link from its user agent.

How do I block Webz.io?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: Omgilibot
Disallow: /

Use the Omgilibot token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking Webz.io hurt my traffic?

Blocking keeps your content out of a dataset resold to third parties you will never see, which is the whole reason to block it. As with any pure data broker there is no answer engine at the other end, so nothing about your search or AI visibility changes either way.