Sitemaps and robots.txt
Read and write sitemaps and read robots.txt files, then check whether a crawler may visit an URL. Typical uses: list the pages of a site to crawl, publish the sitemap of your site.
sitemap parsesitemap parsesitemap writesitemap writesitemap writerobots parserobots allowedrobots allowed
sitemap parse <text_or_path>
Description
Read a sitemap given as text (it starts with < or with http) or as a file, gzip included
Read a sitemap given as text (it starts with < or with http) or as a file, gzip included
Parameters
text_or_path: The sitemap, or the path of a .xml, .xml.gz or .txt file (same rights as file load) - String - required
text_or_path: The sitemap, or the path of a .xml, .xml.gz or .txt file (same rights as file load) - String - required
admin
sitemap parse "<?xml version=\"1.0\" encoding=\"UTF-8\"?><urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\"><url><loc>https://shop.example.com/</loc><lastmod>2026-11-10</lastmod><changefreq>daily</changefreq><priority>1.0</priority></url><url><loc>https://shop.example.com/winter</loc><lastmod>2026-11-09T16:30:00+00:00</lastmod></url><url><loc>https://shop.example.com/old</loc><changefreq>sometimes</changefreq><priority>5</priority></url></urlset>"
mentdb
{"type": "urlset", "count": 3, "urls": [{"loc": "https://shop.example.com/", "lastmod": "2026-11-10", "changefreq": "daily", "priority": 1}, {"loc": "https://shop.example.com/winter", "lastmod": "2026-11-09T16:30:00+00:00", "changefreq": null, "priority": null}, {"loc": "https://shop.example.com/old", "lastmod": null, "changefreq": "sometimes", "priority": 5}], "warnings": ["1 <changefreq> are not always, hourly, daily, weekly, monthly, yearly or never", "1 <priority> are not numbers from 0.0 to 1.0"], "gzip": false}
A sitemap index
admin
sitemap parse "<?xml version=\"1.0\" encoding=\"UTF-8\"?><sitemapindex xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\"><sitemap><loc>https://shop.example.com/sitemap-products.xml.gz</loc><lastmod>2026-11-10</lastmod></sitemap><sitemap><loc>https://shop.example.com/sitemap-blog.xml</loc></sitemap></sitemapindex>"
mentdb
{"type": "sitemapindex", "count": 2, "sitemaps": [{"loc": "https://shop.example.com/sitemap-products.xml.gz", "lastmod": "2026-11-10"}, {"loc": "https://shop.example.com/sitemap-blog.xml", "lastmod": null}], "warnings": [], "gzip": false}
A text sitemap
admin
sitemap parse "https://shop.example.com/ https://shop.example.com/winter "
mentdb
{"type": "text", "count": 2, "urls": [{"loc": "https://shop.example.com/", "lastmod": null, "changefreq": null, "priority": null}, {"loc": "https://shop.example.com/winter", "lastmod": null, "changefreq": null, "priority": null}], "warnings": [], "gzip": false}
sitemap parse <text_or_path> <json_params>
Description
Read a sitemap with parameters: source (auto, text or path), max_urls (100000, up to 1000000), max_bytes (50 MB, up to 512 MB, after decompression)
Read a sitemap with parameters: source (auto, text or path), max_urls (100000, up to 1000000), max_bytes (50 MB, up to 512 MB, after decompression)
Parameters
text_or_path: The sitemap or a file path - String - required
json_params: The parameters, as a JSON object - String - required
text_or_path: The sitemap or a file path - String - required
json_params: The parameters, as a JSON object - String - required
admin
sitemap write "[\"https://shop.example.com/\", {\"loc\": \"https://shop.example.com/winter\", \"lastmod\": \"2026-11-09\", \"changefreq\": \"weekly\", \"priority\": 0.8}, \"https://shop.example.com/café?size=M&color=blue\"]" "tmp/doc_sitemap.xml.gz" "{\"overwrite\": true}"; sitemap parse "tmp/doc_sitemap.xml.gz" "{\"max_urls\": 1}"
mentdb
{"type": "urlset", "count": 1, "urls": [{"loc": "https://shop.example.com/", "lastmod": null, "changefreq": null, "priority": null}], "warnings": ["only the first 1 entries were read (max_urls)"], "gzip": true}
Used in use cases (1 recipe)
Web & API developer › Plan a polite crawl of a site with its robots.txt and its sitemap
Web & API developer › Plan a polite crawl of a site with its robots.txt and its sitemap
sitemap write <json_urls>
Description
The sitemap XML of URLs (texts or {loc, lastmod, changefreq, priority}; the result of sitemap parse is accepted): 50000 URLs and 50 MB at most
The sitemap XML of URLs (texts or {loc, lastmod, changefreq, priority}; the result of sitemap parse is accepted): 50000 URLs and 50 MB at most
Parameters
json_urls: The URLs, as a JSON array - String - required
json_urls: The URLs, as a JSON array - String - required
admin
sitemap write "[\"https://shop.example.com/\", {\"loc\": \"https://shop.example.com/winter\", \"lastmod\": \"2026-11-09\", \"changefreq\": \"weekly\", \"priority\": 0.8}, \"https://shop.example.com/café?size=M&color=blue\"]"
mentdb
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://shop.example.com/</loc> </url> <url> <loc>https://shop.example.com/winter</loc> <lastmod>2026-11-09</lastmod> <changefreq>weekly</changefreq> <priority>0.8</priority> </url> <url> <loc>https://shop.example.com/caf%C3%A9?size=M&color=blue</loc> </url> </urlset>
sitemap write <json_urls> <path>
Description
Write a sitemap file, refused if it exists. Result: {index, urls, files: [{file, urls, bytes}]}
Write a sitemap file, refused if it exists. Result: {index, urls, files: [{file, urls, bytes}]}
Parameters
json_urls: The URLs, as a JSON array - String - required
path: The file to write (.xml, or .xml.gz for gzip) - String - required
json_urls: The URLs, as a JSON array - String - required
path: The file to write (.xml, or .xml.gz for gzip) - String - required
admin
sitemap write "[\"https://shop.example.com/\", {\"loc\": \"https://shop.example.com/winter\", \"lastmod\": \"2026-11-09\", \"changefreq\": \"weekly\", \"priority\": 0.8}, \"https://shop.example.com/café?size=M&color=blue\"]" "tmp/doc_sitemap.xml"
mentdb
{"index": null, "urls": 3, "files": [{"file": "tmp/doc_sitemap.xml", "urls": 3, "bytes": 420}]}
sitemap write <json_urls> <path> <json_params>
Description
Write sitemap files with parameters: overwrite (false), base_url (the public address of the folder, needed when the URLs are split), max_urls (50000 per file, 1 to 50000), max_bytes (50 MB per file, from 1000)
Write sitemap files with parameters: overwrite (false), base_url (the public address of the folder, needed when the URLs are split), max_urls (50000 per file, 1 to 50000), max_bytes (50 MB per file, from 1000)
Parameters
json_urls: The URLs, as a JSON array - String - required
path: The file to write (.xml or .xml.gz) - String - required
json_params: The parameters, as a JSON object - String - required
json_urls: The URLs, as a JSON array - String - required
path: The file to write (.xml or .xml.gz) - String - required
json_params: The parameters, as a JSON object - String - required
admin
sitemap write "[\"https://shop.example.com/\", {\"loc\": \"https://shop.example.com/winter\", \"lastmod\": \"2026-11-09\", \"changefreq\": \"weekly\", \"priority\": 0.8}, \"https://shop.example.com/café?size=M&color=blue\"]" "tmp/doc_sitemap.xml" "{\"overwrite\": true}"
mentdb
{"index": null, "urls": 3, "files": [{"file": "tmp/doc_sitemap.xml", "urls": 3, "bytes": 420}]}
5 URLs, 2 per file: an index and 3 sitemaps
admin
sitemap write "[\"https://shop.example.com/p1\", \"https://shop.example.com/p2\", \"https://shop.example.com/p3\", \"https://shop.example.com/p4\", \"https://shop.example.com/p5\"]" "tmp/doc_sitemap_split.xml" "{\"overwrite\": true, \"max_urls\": 2, \"base_url\": \"https://shop.example.com/maps/\"}"; sitemap parse "tmp/doc_sitemap_split.xml"
mentdb
{"type": "sitemapindex", "count": 3, "sitemaps": [{"loc": "https://shop.example.com/maps/doc_sitemap_split-1.xml", "lastmod": null}, {"loc": "https://shop.example.com/maps/doc_sitemap_split-2.xml", "lastmod": null}, {"loc": "https://shop.example.com/maps/doc_sitemap_split-3.xml", "lastmod": null}], "warnings": [], "gzip": false}
Split without base_url
admin
sitemap write "[\"https://shop.example.com/p1\", \"https://shop.example.com/p2\", \"https://shop.example.com/p3\", \"https://shop.example.com/p4\", \"https://shop.example.com/p5\"]" "tmp/doc_sitemap_split.xml" "{\"overwrite\": true, \"max_urls\": 2}"
mentdb
ERROR: Sorry, an error appears (the URLs need 3 sitemaps and an index: give base_url, the public address of the folder of the files).
robots parse <robots_text>
Description
The groups, sitemaps and other lines of a robots.txt. Result: {groups: [{user_agents, rules: [{type, path, line}], crawl_delay}], sitemaps, other, warnings}
The groups, sitemaps and other lines of a robots.txt. Result: {groups: [{user_agents, rules: [{type, path, line}], crawl_delay}], sitemaps, other, warnings}
Parameters
robots_text: The content of a robots.txt file - String - required
robots_text: The content of a robots.txt file - String - required
admin
robots parse "# robots.txt of shop.example.com User-agent: * Disallow: /cart Disallow: /*.pdf$ Allow: /cart/help User-agent: Googlebot User-agent: Bingbot Disallow: /search Crawl-delay: 2 Sitemap: https://shop.example.com/sitemap.xml"
mentdb
{"groups": [{"user_agents": ["*"], "rules": [{"type": "disallow", "path": "/cart", "line": 3}, {"type": "disallow", "path": "/*.pdf$", "line": 4}, {"type": "allow", "path": "/cart/help", "line": 5}], "crawl_delay": null}, {"user_agents": ["Googlebot", "Bingbot"], "rules": [{"type": "disallow", "path": "/search", "line": 9}], "crawl_delay": 2}], "sitemaps": ["https://shop.example.com/sitemap.xml"], "other": [], "warnings": []}
Used in use cases (1 recipe)
Web & API developer › Plan a polite crawl of a site with its robots.txt and its sitemap
Web & API developer › Plan a polite crawl of a site with its robots.txt and its sitemap
robots allowed <robots_text> <user_agent> <url_or_path>
Description
May this crawler visit this URL? Result: {user_agent, path, allowed, group, rule, crawl_delay, reason}
May this crawler visit this URL? Result: {user_agent, path, allowed, group, rule, crawl_delay, reason}
Parameters
robots_text: The content of a robots.txt file - String - required
user_agent: The product token of the crawler (MentDBBot, or a full user-agent: the first word is taken, without case) - String - required
url_or_path: An absolute URL or a path that starts with / - String - required
robots_text: The content of a robots.txt file - String - required
user_agent: The product token of the crawler (MentDBBot, or a full user-agent: the first word is taken, without case) - String - required
url_or_path: An absolute URL or a path that starts with / - String - required
admin
robots allowed "# robots.txt of shop.example.com User-agent: * Disallow: /cart Disallow: /*.pdf$ Allow: /cart/help User-agent: Googlebot User-agent: Bingbot Disallow: /search Crawl-delay: 2 Sitemap: https://shop.example.com/sitemap.xml" "MentDBBot/1.0" "https://shop.example.com/cart/checkout"
mentdb
{"user_agent": "mentdbbot", "path": "/cart/checkout", "allowed": false, "group": "*", "rule": {"type": "disallow", "path": "/cart", "line": 3}, "crawl_delay": null, "reason": "the longest matching rule is Disallow: /cart"}
The longer Allow wins
admin
robots allowed "# robots.txt of shop.example.com User-agent: * Disallow: /cart Disallow: /*.pdf$ Allow: /cart/help User-agent: Googlebot User-agent: Bingbot Disallow: /search Crawl-delay: 2 Sitemap: https://shop.example.com/sitemap.xml" "MentDBBot" "/cart/help/returns"
mentdb
{"user_agent": "mentdbbot", "path": "/cart/help/returns", "allowed": true, "group": "*", "rule": {"type": "allow", "path": "/cart/help", "line": 5}, "crawl_delay": null, "reason": "the longest matching rule is Allow: /cart/help"}
$ ends the pattern: /guide.pdf?v=2 is allowed
admin
robots allowed "# robots.txt of shop.example.com User-agent: * Disallow: /cart Disallow: /*.pdf$ Allow: /cart/help User-agent: Googlebot User-agent: Bingbot Disallow: /search Crawl-delay: 2 Sitemap: https://shop.example.com/sitemap.xml" "MentDBBot" "/guide.pdf?v=2"
mentdb
{"user_agent": "mentdbbot", "path": "/guide.pdf?v=2", "allowed": true, "group": "*", "rule": null, "crawl_delay": null, "reason": "no rule matches: allowed"}
Googlebot has its own group: the * rules do not apply to it
admin
robots allowed "# robots.txt of shop.example.com User-agent: * Disallow: /cart Disallow: /*.pdf$ Allow: /cart/help User-agent: Googlebot User-agent: Bingbot Disallow: /search Crawl-delay: 2 Sitemap: https://shop.example.com/sitemap.xml" "Googlebot/2.1 (+http://www.google.com/bot.html)" "/cart"
mentdb
{"user_agent": "googlebot", "path": "/cart", "allowed": true, "group": "googlebot", "rule": null, "crawl_delay": 2, "reason": "no rule matches: allowed"}
Non ASCII paths are compared encoded
admin
robots allowed "User-agent: * Disallow: /café" "MentDBBot" "/caf%C3%A9/menu"
mentdb
{"user_agent": "mentdbbot", "path": "/caf%C3%A9/menu", "allowed": false, "group": "*", "rule": {"type": "disallow", "path": "/café", "line": 2}, "crawl_delay": null, "reason": "the longest matching rule is Disallow: /café"}
An empty file allows everything
admin
robots allowed "" "MentDBBot" "/anything"
mentdb
{"user_agent": "mentdbbot", "path": "/anything", "allowed": true, "group": null, "rule": null, "crawl_delay": null, "reason": "no group for this user-agent and no * group: allowed"}
robots allowed <robots_text> <user_agent> <url_or_path> <json_params>
Description
May this crawler visit this URL, with parameters: http_status (the status of the download of robots.txt: 4xx allows everything, 5xx disallows everything)
May this crawler visit this URL, with parameters: http_status (the status of the download of robots.txt: 4xx allows everything, 5xx disallows everything)
Parameters
robots_text: The content of a robots.txt file - String - required
user_agent: The product token of the crawler (MentDBBot, or a full user-agent: the first word is taken, without case) - String - required
url_or_path: An absolute URL or a path that starts with / - String - required
json_params: The parameters, as a JSON object - String - required
robots_text: The content of a robots.txt file - String - required
user_agent: The product token of the crawler (MentDBBot, or a full user-agent: the first word is taken, without case) - String - required
url_or_path: An absolute URL or a path that starts with / - String - required
json_params: The parameters, as a JSON object - String - required
admin
robots allowed "" "MentDBBot" "/" "{\"http_status\": 503}"
mentdb
{"user_agent": "mentdbbot", "path": "/", "allowed": false, "group": null, "rule": null, "crawl_delay": null, "reason": "robots.txt unreachable (HTTP 503): everything is disallowed"}
robots.txt answered 404
admin
robots allowed "" "MentDBBot" "/" "{\"http_status\": 404}"
mentdb
{"user_agent": "mentdbbot", "path": "/", "allowed": true, "group": null, "rule": null, "crawl_delay": null, "reason": "robots.txt not available (HTTP 404): everything is allowed"}
Used in use cases (1 recipe)
Web & API developer › Plan a polite crawl of a site with its robots.txt and its sitemap
Web & API developer › Plan a polite crawl of a site with its robots.txt and its sitemap
