For website owners
The RepBudResearch crawler
If you have seen RepBudResearch in your server logs, this is what it was doing.
What it is
RepBudResearch is RepBud’s web crawler. It reads the public websites of tour operators and travel agents to note which lodges, camps and hotels in Africa and the Indian Ocean they offer. RepBud uses these notes to help representatives see which operators feature a property and the properties around it.
What it reads
Pages about accommodation, itineraries and destinations, found through a site’s sitemap or the links on its home page. HTML pages only: it does not download documents, images or video, and it does not sign in, fill in forms or run scripts.
What it records
The operator’s business name, website and country; the name of the property; the address of the page it appeared on; and when it was seen. It does not record people’s names, email addresses, phone numbers, prices or the text of your pages. A property counts only where your site offers it (its own page, an accommodation list or an itinerary), not where an article mentions it.
How it behaves
- It identifies itself as
RepBudResearch/1.0 (+https://repbud.app/research). - It follows robots.txt: the rules for RepBudResearch, or for all agents where there are none.
- It asks for one page at a time from each site, two to three seconds apart, and reads a limited number of pages per visit.
- It slows down and then stops when a site answers 429 or 503.
- It visits each site about once a quarter.
Blocking it
Add this to your robots.txt and it will not read your site:
User-agent: RepBudResearch Disallow: /
Opting out or correcting a record
Email gw@repbud.app with your website’s address. We will switch your site off in the crawler and stop showing what it recorded, usually within a few working days. The same address takes corrections, for example a property you no longer offer.