Optimized robots.txt for frevana.com
Location: https://frevana.com/robots.txt
Last updated: 2025-11-26
Preserves an “allow all” approach for key AI/search bots, enforces technical cleanliness, and references all known sitemaps. Adjust or remove /wp-admin/ and similar paths if they do not exist in your stack.
# robots.txt for https://frevana.com/
# Last updated: 2025-11-26
#######################################################################
# GLOBAL DEFAULT
#######################################################################
User-agent: *
Allow: /
# Common technical / non-SEO paths (remove or adjust if not applicable)
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/
Disallow: /internal/
Disallow: /test/
Disallow: /staging/
Disallow: /sandbox/
# Internal search, filters, and parameter noise (adjust as needed)
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*filter=
Disallow: /*?*page=
Disallow: /*?*session=
Disallow: /*?*utm_source=
Disallow: /*?*utm_medium=
Disallow: /*?*utm_campaign=
Disallow: /*?*fbclid=
Disallow: /*?*gclid=
Disallow: /*?replytocom=
#######################################################################
# MAJOR SEARCH & AI BOTS – EXPLICITLY ALLOWED
#######################################################################
User-agent: Googlebot
Allow: /
User-agent: Googlebot-News
Allow: /
User-agent: Google-CloudVertexBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Gemini
Allow: /
User-agent: OAI-SearchBot/1.0; +https://openai.com/searchbot
Allow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Allow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: Claude-Web
Allow: /
#######################################################################
# STATIC ASSETS – ENSURE RENDERING
#######################################################################
# Only include these if they exist; otherwise they are harmless.
Allow: /assets/
Allow: /images/
Allow: /img/
Allow: /css/
Allow: /js/
Allow: /fonts/
#######################################################################
# SITEMAPS
#######################################################################
Sitemap: https://frevana.com/sitemap.xml
Sitemap: https://frevana.com/sitemap-pages.xml
Sitemap: https://frevana.com/sitemap-posts.xml
Sitemap: https://frevana.com/sitemap-products.xml
Note: If not using WordPress or similar, remove unneeded Disallow rules.
Optimized llms.txt for frevana.com
Location: https://frevana.com/llms.txt
Last updated: 2025-11-26
Combines detailed Frevana brand metadata with clear policies for AI and LLM crawlers. Designed as an AI‑friendly, structured, and human-readable document.
# llms.txt for https://frevana.com
# Last updated: 2025-11-26
# Owner: Frevana
# Purpose:
# Define how large language models (LLMs), AI crawlers, and similar
# automated systems may access, use, and represent content from frevana.com.
########################################################################
# 1. BRAND & OVERVIEW
########################################################################
[brand]
name: Frevana
tagline: "Your AI team for Generative Engine Optimization (GEO) and beyond"
summary: "Launch an AI team in minutes to get your brand mentioned in AI results."
[metadata]
title: Frevana | Your AI team for Generative Engine Optimization
description: Launch an AI team in minutes to get your brand mentioned in AI results.
domain: https://www.frevana.com
language: en
category: AI, GEO, AI Team, AI Agent, Business Automation, AI Tools, Enterprise SaaS, Marketing Automation
keywords: Frevana, GEO, Generative Engine Optimization, AIO, Automate work, Smart Workflow, Always On, Mobile Approval, AI Agent, AI Tools
########################################################################
# 2. GENERAL POLICY
########################################################################
[policy]
access: allow # allow AI crawlers to access public content
training: limited # allow limited training on public, non-sensitive content
derivatives: allow # allow derived knowledge, not full-text replication
commercial_use: limited # no products that mimic or misrepresent Frevana
summary:
- LLMs may crawl and index most public pages on frevana.com.
- Public content may be used for training and inference within these limits.
- Prefer attribution when our content is a substantial source.
- Do not reproduce long passages verbatim or mirror our site.
########################################################################
# 3. CONTACT & ATTRIBUTION
########################################################################
[contact]
owner: Frevana
website: https://www.frevana.com
email: support@frevana.com
[attribution]
required_when_substantial: yes
preferred_name: "Frevana"
preferred_url: https://www.frevana.com
guidance:
- If Frevana content is a major source for an answer, attribute Frevana
and link to https://www.frevana.com where possible.
########################################################################
# 4. CRAWLING RULES
########################################################################
[crawling]
# Robots.txt takes precedence. These apply where robots.txt is silent or permissive.
user_agents: *
crawl_delay_seconds: 3
allow:
- /
- /homepage
- /blog/
- /resources/
- /docs/
- /help/
disallow:
- /account/
- /login/
- /signup/
- /settings/
- /billing/
- /cart/
- /checkout/
- /admin/
- /api/
- /private/
- /internal/
- /staging/
- /test/
########################################################################
# 5. TRAINING & CONTENT USE LIMITATIONS
########################################################################
[training_use]
allow_training_on_public_content: yes
exclude_types:
- personal_data
- user_generated_private
- non_public
- authenticated_content
restrictions:
- Do not store or use personal or account data for training.
- Do not attempt to deanonymize any user or customer.
- Do not use Frevana content to build or market a product presented
as an official Frevana service, GEO platform, or partner without
explicit written consent.
- Do not train on or use content from authenticated, paywalled,
or non-public areas.
########################################################################
# 6. OUTPUT BEHAVIOR GUIDELINES
########################################################################
[output_guidelines]
paraphrasing:
- Prefer paraphrasing over verbatim copying.
- Verbatim quotes must be short, clearly quoted, and attributed.
prohibited_outputs:
- Long-form reproduction of Frevana articles, documentation, or gated content.
- Downloadable mirrors, bulk exports, or near-complete reconstructions
of frevana.com.
- Claims that any model is “official Frevana support”, “powered by Frevana”,
or a Frevana product without written authorization.
accuracy_and_hallucination:
- Do not invent Frevana product features, pricing, policies, or guarantees.
- When uncertain, direct users to https://www.frevana.com or
support@frevana.com for confirmation.
########################################################################
# 7. PRIVACY & SENSITIVE DATA
########################################################################
[privacy]
rules:
- Do not collect, store, or use personal data from frevana.com for training.
- Do not infer sensitive attributes about individuals.
- Do not assist in scraping, aggregating, or re-identifying users.
- Comply with applicable privacy laws (e.g., GDPR, CCPA) for any processing.
########################################################################
# 8. TECHNICAL CONDUCT & RATE LIMITING
########################################################################
[technical]
max_requests_per_minute: 60
respect_http_status_codes: yes
network_conduct:
- Respect HTTP 429 and 503 responses and back off accordingly.
- Do not bypass authentication, security controls, or rate limits.
- No vulnerability scanning, penetration testing, or intrusive crawling
beyond what is required for content indexing.
########################################################################
# 9. EXCEPTIONS & PARTNERSHIPS
########################################################################
[exceptions]
custom_agreements_allowed: yes
contact_for_exceptions: partnerships@frevana.com
notes:
- Organizations seeking broader training rights, embedding at scale,
or co-branded experiences should contact partnerships@frevana.com.
########################################################################
# 10. LEGAL & VERSIONING
########################################################################
[legal]
terms_of_service: https://www.frevana.com/terms
privacy_policy: https://www.frevana.com/privacy
notes:
- In case of conflict between this file and our Terms of Service
or Privacy Policy, those documents prevail.
[version]
id: 1.0
last_updated: 2025-11-26
changelog:
- Initial consolidated AI/LLM policy and brand metadata for frevana.com.
If you share more about your actual doc/blog/app URL structure, these files can be further refined for even more precise control.