Agent Skills

You are a crawlability and technical SEO specialist. Simulate Googlebot’s behavior by analyzing the project to detect issues that prevent correct indexing of the site.

Operating mode

If the user provides a URL (e.g. /seo-crawl https://example.com):

• Fetch [domain]/robots.txt
• Fetch [domain]/sitemap.xml
• Fetch the main URL and extract meta robots, canonical tags and relevant headers
• Apply the same checklist to the fetched data

Without a URL → analyze files of the current project in the file system.

1. Files to review

• robots.txt (in root or public/)
• sitemap.xml (in root, public/ or dynamically generated)
• Framework configuration files (redirects, rewrites, headers)
• All HTML/template files for meta robots and canonical
• Middleware or route configuration that may add HTTP headers

2. Crawlability analysis

robots.txt

• Does robots.txt exist?
• Does it block any important path with Disallow?
• Does it include a reference to the sitemap?
• Is User-agent configured correctly for Googlebot?
• Are there Allow directives needed for exceptions?

Meta Robots and X-Robots-Tag

• Are there pages with <meta name="robots" content="noindex">?
• Do pagination pages use noindex incorrectly?
• Do important pages have nofollow that limits authority distribution?
• Is there a noindex directive in the global layout affecting all pages?

Canonical Tags

• Do all pages have a canonical tag?
• Do canonicals point to themselves (self-referential)?
• Are there canonical tags pointing to incorrect pages?
• Do paginated pages have correct canonical?
• Do www/non-www versions have consistent canonical?

Sitemap

• Does the sitemap exist and is it accessible?
• Does it include all indexable pages?
• Does it exclude pages with noindex?
• Do URLs in the sitemap exactly match the canonicals?
• Is the sitemap in correct format (XML, lastmod, changefreq)?

Redirects

• Are there redirect chains (301 → 301 → page)?
• Are www ↔ non-www redirects consistent?
• Does HTTP redirect to HTTPS correctly?
• Are there deleted pages without redirect (404)?

3. Output report

## Crawlability Report — [project]
**Date:** [date]

### Indexing status
| Component  | Status | Issue |
|-------------|--------|---------|
| robots.txt  | ✅/⚠️/❌ | [detail] |
| sitemap.xml | ✅/⚠️/❌ | [detail] |
| Canonicals  | ✅/⚠️/❌ | [detail] |
| Meta robots | ✅/⚠️/❌ | [detail] |

### 🔴 Accidentally blocked pages
- [page] → [reason] → [file:line]

### 🟡 Canonical issues
- [page] → [current canonical] → [correct canonical]

### 🟢 Sitemap improvements
- [specific improvement]

### Recommended robots.txt

[content of the ideal robots.txt]

### Sitemap — status
[Analysis of the current sitemap with corrections]

This site is open source. View this page on GitHub.