StackrateBot

What it does

StackrateBot reads public help-center articles published by software vendors covered in Stackrate reports. The articles are used as evidence when our analysis pipeline evaluates how a vendor handles a specific buyer requirement (for example, “how does NetSuite handle approvals across 12 subsidiaries”).

The bot makes one type of request: a help-center crawl, refreshed on a 7-day cycle. Each crawl reads only what the vendor publishes openly to the internet (no login, no API key). It does not click links, submit forms, or interact with anything that is not a public article URL.

What it does NOT do

• Reach gated or authenticated content. Articles requiring login are skipped.
• Attempt to bypass rate limits, robots.txt, captchas, or IP blocks.
• Republish vendor articles as standalone pages. We extract claims and cite the original URL.
• Train any general-purpose AI model on vendor content.

Robots.txt

StackrateBot reads robots.txt before every crawl. The bot honors:

• Disallow rules under User-agent: StackrateBot or the wildcard User-agent: * block.
• Crawl-delay directives in the matching block — article fetches are serialized at the declared interval.
• Longest-match Allow vs Disallow resolution (Google convention).

Every robots.txt response is stored alongside the date it was fetched. This is durable evidence of what the file said when we read it.

How to opt out

Add this to your robots.txt at the help-center origin:

User-agent: StackrateBot
Disallow: /

After your next robots.txt update, our 7-day refresh cycle will detect the change and stop fetching new articles. Articles already in our cache continue to be cited in existing reports but are labeled with the date of the last permitted crawl. See the archival policy below.

Archival snapshot policy

Help-center articles we read while access was openly permitted are retained as a dated snapshot. When a vendor later changes robots.txt to disallow us, we stop fetching new articles but the existing cache continues to be cited in reports — labeled with the date of the last permitted crawl, and labeled again to indicate access has since been restricted.

Snapshots stop influencing new reports after roughly six months. Beyond that point the “as of [date]” label is no longer a defensible freshness claim because help-center content has typically drifted enough that the snapshot misrepresents the vendor.

This mirrors the policy of archival services like the Internet Archive (Wayback Machine) and Common Crawl: once data was obtained legitimately, the snapshot is preserved, but always with the date of collection visible to the reader. The rationale is detailed on our methodology page.

If you want us to remove a specific snapshot

Email legal@stackrate.ai with the vendor name and the article URLs you would like removed. We will acknowledge within a few business days. Cease-and-desist letters from outside counsel are reviewed individually. Marketing-team removal requests for snapshots obtained while access was permitted are evaluated on the merits; the default is to honor with a vendor-response label rather than full deletion, unless the request cites a specific legal basis.

Verified vendors who upload content directly via the connector can withdraw their contributions at any time from the vendor portal. That content was given to us under a relationship; you can take it back. Public-crawl snapshots follow the archival policy above.

Contact

Bot-related questions: bot@stackrate.ai
Legal / removal requests: legal@stackrate.ai
Vendor portal (claim, connector, evidence upload): stackrate.ai/vendors