Discoverability
Store classification
Categories
No public values available.
Store labels
No public values available.
Catalog ID EFPWBHRAKmJ99AWKG
Public Apify Actor
trusted_tech_spot/robots-txt-sitemap-crawler
Parses robots.txt crawl rules and XML sitemaps. Detects disallowed paths, sitemap locations, AI web scraper permissions (GPTBot, ClaudeBot, CCBot), and index URL counts.
Active users / 30d
1
Total users
2
Runs / 30d
3
Total runs
7
Rating
—
No reviews
Bookmarks
0
Usage history
Latest captured Active users / 30d
1
+1 since 26 Sept
6 of 30 UTC days captured
Latest captured Runs / 30d
3
+2 since 28 Sept
4 of 30 UTC days captured
1
Users / 7d
Discoverability
Categories
No public values available.
Store labels
No public values available.
Monetization
Pay per event
2 charge events configured
Actor Start
apify-actor-start
Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).
$0.001
Robots.txt & Sitemap Audit
apify-default-dataset-item
Complete crawler permissions, AI bot rules, and sitemap URL count
1
Users / 30d
1
Users / 90d
$0.0018
Loading public Actor documentation
Reading the current public Store definition and reviews.
More from this builder