For website owners

The RepBudResearch crawler

If you have seen RepBudResearch in your server logs, this is what it was doing.

What it is

RepBudResearch is RepBud’s web crawler. It reads the public websites of tour operators and travel agents to note which lodges, camps and hotels in Africa and the Indian Ocean they offer. RepBud uses these notes to help representatives see which operators feature a property and the properties around it.

What it reads

Pages about accommodation, itineraries and destinations, found through a site’s sitemap or the links on its home page. HTML pages only: it does not download documents, images or video, and it does not sign in, fill in forms or run scripts.

What it records

The operator’s business name, website and country; the name of the property; the address of the page it appeared on; and when it was seen. It does not record people’s names, email addresses, phone numbers, prices or the text of your pages. A property counts only where your site offers it (its own page, an accommodation list or an itinerary), not where an article mentions it.

How it behaves

  • It identifies itself as RepBudResearch/1.0 (+https://repbud.app/research).
  • It follows robots.txt: the rules for RepBudResearch, or for all agents where there are none.
  • It asks for one page at a time from each site, two to three seconds apart, and reads a limited number of pages per visit.
  • It slows down and then stops when a site answers 429 or 503.
  • It visits each site about once a quarter.

Blocking it

Add this to your robots.txt and it will not read your site:
User-agent: RepBudResearch
Disallow: /

Opting out or correcting a record

Email gw@repbud.app with your website’s address. We will switch your site off in the crawler and stop showing what it recorded, usually within a few working days. The same address takes corrections, for example a property you no longer offer.