Examples & Use Cases

Practical examples and configurations for common WordPress scenarios.

⚠️ Important Notice: These examples demonstrate the output of Better Robots.txt features. While the basic examples show what the Free version can generate, advanced optimizations and specialized configurations require the Pro version. The examples below are for illustration purposes only - actual feature implementation requires the appropriate license tier.

🆓 Free Version Features Included:

• Basic WordPress protection rules
• ChatGPT bot blocking
• AI-recommended bad bot blocking

💎 Pro Version Required For:

• Sitemap integration & automatic detection (Yoast, Rank Math, AIOSEO)
• WooCommerce optimization
• Multisite network management
• Brand protection & competitor blocking
• Advanced crawl budget optimization
• Physical file generation
• Geographic targeting
• Advanced security features

Blog/Content Site Example

Scenario

Personal or business blog focused on content marketing and SEO.

Recommended Configuration

✅ **Basic Settings**:
- Keep default WordPress rules
- Set crawl delay: 1 second
- Enable ads.txt allowance

✅ **Search Engines**:
- Allow: Googlebot, Bingbot, Yahoo Slurp
- Allow: DuckDuckBot, Applebot
- Allow: Media crawlers (for images)

✅ **Bot Protection**:
- Block: ChatGPT Bot (Free)
- Enable: AI-Recommended bad bot blocking (Free)
- Consider: Backlink protector for competitive niches (Pro)

✅ **SEO Integration**:
- Sitemap integration requires Pro version
- Manual SEO optimization through content structure

Generated robots.txt (Free Version Output)

# Basic output with free features enabled

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /*/*.css
Allow: /*/*.js
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /license.txt
Disallow: /xmlrpc.php
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: */disclaimer/*
Disallow: *?attachment_id=
Disallow: /privacy-policy

# ChatGPT Bot blocked (Free feature)
User-agent: GPTBot
Disallow: /

# Basic crawl delay
Crawl-delay: 1

# This robots.txt file was created by Better Robots.txt

Pro Version includes sitemap integration, advanced crawl delays, and additional protection rules.

Additional Custom Rules

# Block common WordPress spam patterns
User-agent: *
Disallow: */comment-page-*
Disallow: */amp/
Disallow: */feed/$
Disallow: /?s=*
Disallow: /search/

E-commerce Site Example (WooCommerce)

Scenario

Online store with WooCommerce, focused on product sales and conversion optimization.

Recommended Configuration

✅ **Basic Settings**:
- Default WordPress rules
- Crawl delay: 2 seconds (e-commerce needs more server resources)
- Enable physical file generation (Pro)

✅ **WooCommerce Optimization** (Pro):
- Enable: WooCommerce performance optimization
- Block: Unnecessary e-commerce URLs
- Optimize: Product page crawling focus
- Improve: Server performance

✅ **SEO Integration**:
- Sitemap integration requires Pro version
- Optimize: Product page crawling patterns
- Enable: Category and product image access rules

✅ **Bot Protection**:
- Block: ChatGPT Bot
- Enable: Bad bot blocking (protect pricing data)
- Enable: Backlink protector (competitive niche)

Generated robots.txt

# E-commerce optimized output (Pro Version)

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /*/*.css
Allow: /*/*.js
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /xmlrpc.php
Disallow: /wp-login.php

# WooCommerce optimization rules applied
# Specific e-commerce performance rules generated by Pro version
# Configured through Better Robots.txt Pro WooCommerce settings
# Sitemap integration included in Pro version

Crawl-delay: 2

Note: WooCommerce optimization and sitemap integration require Pro version.

Additional E-commerce Rules

# Protect customer data and sessions
User-agent: *
Disallow: /my-account/
Disallow: /customer-logout/
Disallow: /lost-password/
Disallow: /order-received/

# Block AJAX endpoints that don't need crawling
Disallow: /wc-api/
Disallow: */wc-api/*

Multisite Network Example

Scenario

WordPress multisite network with multiple blogs and stores.

Recommended Configuration

✅ **Network Settings** (Pro):
- Enable: Network management
- Sites: blog1, blog2, store1, news
- Enable: Network-wide sitemap integration
- Enable: Cross-site protection

✅ **Individual Site Rules**:
- Blog sites: Standard blog configuration
- Store sites: E-commerce optimization
- News site: Media-heavy optimization

✅ **Network Protection**:
- Enable: Bad bot blocking across network
- Enable: Spam backlink protection
- Standardize: crawl delay rules

Network Configuration Example

The Pro multisite feature provides centralized management with:

Key Features:

• ✅ Centralized dashboard for all sites
• ✅ Individual site rule customization
• ✅ Network-wide bot protection
• ✅ Automatic sitemap integration per site
• ✅ Consistent crawl delay management
• ✅ Cross-site security protection

Configuration is handled through:

• Network admin dashboard
• Site-specific override options
• Bulk rule application
• Individual site optimization

Pro Version Benefits:

• Automated rule generation per site type
• Network-wide security updates
• Centralized reporting and analytics
• Priority support for multisite setups

Note: Multisite management requires Pro version with specialized network capabilities.

Corporate/Business Website Example

Scenario

Corporate website with emphasis on brand protection and professional appearance.

Recommended Configuration

✅ **Basic Settings**:
- Professional default rules
- Crawl delay: 1 second
- Enable physical file generation

✅ **Brand Protection**:
- Enable: Backlink protector (Pro)
- Block: Competitor analysis tools (Pro)
- Enable: ChatGPT Bot blocking (Free)
- Control: Social media bot access (Pro)

✅ **SEO Integration**:
- Sitemap integration requires Pro version
- Enable: Professional image indexing
- Optimize: Corporate content structure

Corporate robots.txt (Pro Version Output)

# Corporate optimized output (Pro Version)

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /*/*.css
Allow: /*/*.js
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /license.txt
Disallow: /xmlrpc.php
Disallow: /wp-login.php
Disallow: */disclaimer/*

# Brand protection rules applied
# Advanced competitor blocking enabled
# Social media optimization configured
# Custom corporate rules generated
# Sitemap integration included in Pro version

Crawl-delay: 1

Pro Version includes sitemap integration, brand protection, competitor blocking, and social media optimization.

News/Media Website Example

Scenario

High-traffic news site with frequent content updates and media-rich content.

Recommended Configuration

✅ **Performance Focus**:
- Crawl delay: 2-3 seconds
- Enable: Physical file generation (Pro)
- Enable: Crawl budget optimization (Pro)
- Optimize: Image crawling (Pro)

✅ **SEO Integration**:
- Sitemap integration requires Pro version
- Enable: Google News integration
- Optimize: Media indexing
- Add: Custom category access rules

✅ **Bot Management**:
- Allow: Major search engines
- Allow: Media crawlers
- Block: Aggressive scrapers
- Control: Social media access

News Site robots.txt (Pro Version Output)

# News optimized output (Pro Version)

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /*/*.css
Allow: /*/*.js
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /xmlrpc.php
Disallow: /wp-login.php

# News crawler optimization applied
# Crawl budget optimization enabled
# Social media sharing rules configured
# Image crawling optimized for news sites
# Sitemap integration included in Pro version

Crawl-delay: 2

Pro Version includes news-specific crawler optimization, crawl budget management, sitemap integration, and social media enhancement.

Educational/Institutional Website Example

Scenario

University or educational institution website with diverse content types.

Recommended Configuration

✅ **Accessibility Focus**:
- Moderate crawl delay: 1 second
- Allow: Educational search engines
- Enable: PDF indexing where appropriate
- Control: Bot access for research content

✅ **SEO Integration**:
- Sitemap integration requires Pro version
- Optimize: Course content structure
- Add: Research publication access rules

✅ **Content Protection**:
- Selective content blocking
- Research access control
- Student privacy protection

Educational robots.txt

# Educational optimized output (Pro Version)

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /*/*.css
Allow: /*/*.js
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /license.txt
Disallow: /xmlrpc.php
Disallow: /wp-login.php

# Educational crawler optimization applied
# Research content protection enabled
# Public academic materials accessible
# Sitemap integration included in Pro version

Crawl-delay: 1

Pro Version includes educational-specific rules, research protection, and sitemap integration.

Free Version Educational Example

# Basic educational rules (Free Version)

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /*/*.css
Allow: /*/*.js
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /xmlrpc.php
Disallow: /wp-login.php

# ChatGPT Bot blocked
User-agent: GPTBot
Disallow: /

# Basic crawl delay
Crawl-delay: 1

# This robots.txt file was created by Better Robots.txt

Custom Rules Examples

Block Specific File Types

# Block access to specific file types
User-agent: *
Disallow: /*.pdf$
Disallow: /*.doc$
Disallow: /*.docx$
Disallow: /*.xls$
Disallow: /*.xlsx$

Allow Specific Directory Only

# Allow only specific directory for certain bots
User-agent: Googlebot
Allow: /public-content/
Disallow: /

User-agent: *
Disallow: /private-content/

Seasonal/Temporary Rules

# Temporary maintenance block (remember to remove!)
User-agent: *
Disallow: /
# Maintenance mode - remove after date

Geographic Targeting

# Allow bots from specific regions
User-agent: Googlebot
Allow: /

User-agent: Baiduspider
Allow: /chinese-content/

User-agent: Yandex
Allow: /russian-content/

Testing Your Configuration

Manual Testing Checklist

1. Basic Accessibility:

   curl -I https://yoursite.com/robots.txt
   # Should return 200 OK

2. Search Engine Testing:

   • Google Search Console robots.txt tester
   • Bing Webmaster Tools
   • Third-party robots.txt validation tools

3. Content Verification:

   # Check specific URLs
   curl -A "Googlebot" https://yoursite.com/private-page/
   # Should be blocked if configured correctly

Automated Testing

# Test with different user agents
curl -A "Googlebot" https://yoursite.com/robots.txt
curl -A "AhrefsBot" https://yoursite.com/robots.txt
curl -A "facebookexternalhit" https://yoursite.com/robots.txt

---

Next: Explore our Troubleshooting Guide or learn about Best Practices.