Sitemap Scraper - Find the Sitemap and Watch for New URLs

url-returned

One URL read out of the sitemap of a site: the address, the host, the path, how many levels deep it sits, the file extension where the URL has one, and the last-modified date, change frequency and priority exactly as the sitemap states them, with the date also given as an ISO instant. Each row also says which sitemap file it came from, how many index levels down that file was, whether it was served compressed, and how the sitemap was located in the first place: declared in robots.txt, found by trying a common location, or given directly. Rows that report a site with no sitemap, a sitemap the server refused, filters that matched nothing, or nothing new since the last monitoring run are returned with the reason and are never charged.