ListspaceBot is how Listspace reads a web page when one of our users puts its link in an item. It fetches that one page to show a preview in the item: the page's title, description, image and site name, and for a book its ISBN, authors and a summary.
It identifies itself with this User-Agent: ListspaceBot/1.0 (+https://listspace.app/bot)
What it fetches
- Only a link a signed-in person added to one of their items, in its title, its description or as an attachment. It does not crawl and never follows other links on the page.
- One request for the page. When the page has no title or image tags but names an oEmbed address, it asks that address too.
- For a book, it asks the public Open Library API at openlibrary.org for the book's details.
- Only web pages (HTML). It reads at most 1 MB, gives up after 5 seconds and follows at most 3 redirects.
- It does not run scripts and does not download images. The preview image is loaded by the person's own browser, straight from your site, without a referrer.
How often
- Each address is fetched once and the preview is kept for 7 days, shared by everyone who adds the same link. A page that could not be read is tried again after a day at the earliest.
- A preview is stored with the item, so opening the item again does not fetch the page again.
- Each person can make at most 60 new previews in 10 minutes.
What a request looks like
- A plain GET over HTTPS or HTTP, on the standard ports 80 and 443 only.
- No cookies, no login and no Authorization header. It sends only User-Agent, Accept, Accept-Language and Accept-Encoding.
- Requests come from our server in Germany.
- It never fetches private, internal or local network addresses, also not after a redirect.
What we keep
The title, description and site name of the page, the addresses of its image, icon and canonical page, and for a book the Open Library record. Not the page itself.
robots.txt
ListspaceBot does not read robots.txt. Like a browser, it fetches a single page because a person asked for it, and it never crawls. To stop it, block the User-Agent ListspaceBot on your server, or write to us.