About our crawler
Reference for site owners and administrators. Last reviewed 22 September 2026.
- What it is
A fetcher run by the Cyprus Provider Index. It reads the public pages of websites recorded on the ICPAC register for the 683 firms in the index, to record which services and languages a firm names for itself.
- What it fetches
robots.txt first, then the home page and up to 20 HTML pages linked from it on the same host. One request at a time, at least 5 seconds apart. It keeps the extracted service and language names and the date of the visit; page text is kept for 90 days so a correction can be checked, then deleted.
- How often it visits
Once per firm every 90 days, or sooner when a firm asks for a fact to be re-read. 137 of the 683 firms have not had a website visit yet.
- What it ignores
- Anything disallowed in robots.txt
- Pages behind a login, forms, and search results
- Links off the firm’s own host
- Images, PDFs, scripts and fonts
- Client names, awards, testimonials and any claim about quality
- User agent
CyprusIndexBot/1.0 (+https://cyprus.merchantoolz.com/about-our-crawler)
Placeholder, pending a decision: the crawler does not yet announce itself under a name of our own. The exact string will be published here before it does.
- How to block it
Add these lines to your robots.txt:
User-agent: CyprusIndexBot Disallow: /
It takes effect at the next visit, at most 90 days away. Write to us and it takes effect within 2 working days. Anything already read from your site is deleted at the same time, and the profile then shows services as not recorded. Blocking the crawler does not remove the firm from the index; the register facts stay.
- Who to write to
[email protected] — a person reads it; reply within 2 working days.