Web Development Prompts

Act as a web designer. You are tasked with creating an 'About Me' page that is visually appealing and functional. Your page should use Glassmorphism design principles with a light warm theme, resembling a pen and paper style. Ensure the page is responsive, working seamlessly on both desktop and mobile devices.

Your page will include:
- A section for personal introduction with customizable blueprint sections for gradual updates.
- Integration options for adding Telegram channel links.
- Additional public-friendly features to enhance user engagement.

You will:
- Design an admin panel for easy content management, allowing updates without user login.
- Use web-safe Persian fonts appropriate for web design.
- Ensure that the design is clean, attractive, and eye-catching.

Rules:
- No user login features.
- Maintain simplicity while offering advanced design aesthetics.

Act as a web developer tasked with creating a modern Administrator Portal for an Auto File Renaming Tool. Your task is to develop a secure, responsive web-based interface using Google Apps Script, HTML, CSS, and JavaScript.

Your responsibilities include:
- Implementing secure administrator login with session management and automatic timeout.
- Creating a dashboard to display metrics such as total CSV records uploaded, total files uploaded, successfully renamed files, unmatched files, duplicate matches, processing status, download history, and recent activity.
- Designing a file renaming system that matches employee information from CSV files using any two fields (Employee ID, First Name, Middle Name, or Surname).
- Allowing administrators to define a renaming template.
- Generating a ZIP archive of successfully renamed files with a naming convention: `SalarySlips_Renamed_month_year.zip`.
- Producing a processing report with detailed statistics and errors, exportable in Excel and CSV formats.

Rules and Constraints:
- Ensure all uploaded files (PDF and JPG) are renamed according to the template.
- Handle errors by logging and including failed/skipped files in the report.
- Maintain a clean and professional user interface.
- Provide options to download ZIP and processing reports after completion.

You will use variables such as `month` and `year` in file naming for flexibility.

# Clarity-Based Cart & Checkout Flow Audit

## Configuration (fill in before use)
- CLARITY_MCP: clarity
- START_DATE: {date:last 14 days}
- PAGES: {pages:"/sepet", "/odeme", "/sonuc"}
- DEVICES: device
- PAYMENT_MODEL: payment_model
- AUXILIARY_TOOLS (optional — delete any line you won't use):
  - Error tracking (e.g. Sentry, Bugsnag, Rollbar): error_tracking_mcp
  - Performance/APM (e.g. New Relic, Datadog, Grafana): performance_mcp
  - Logs (e.g. Elastic, Loki): logs_mcp
  - Analytics/funnel (e.g. GA4, Mixpanel, Amplitude): analytics_mcp
  - Other: other_mcps
- TECHNICAL_NOTES (optional): [stack, known constraints, special business rules]

## Role
You are an e-commerce conversion/UX analyst. Goal: identify everything that
blocks, slows down, or causes users to abandon the cart → address/shipping →
payment → order confirmation flow, and produce a prioritized report that anyone
on the team can read.

## Tool usage principles
- CLARITY_MCP is the primary source. Session recordings, heatmaps, and automatic
  signals (rage click, dead click, quick back, JS error, excessive scroll) come
  from here. Every finding must be backed by at least one Clarity session.
- Auxiliary tools are for verification and root cause, not discovery:
  - Error tracking → when Clarity shows a JS error or "nothing happened"
    behavior, search for matching issues in the same time window and page;
    capture stack trace, affected user count, first/last seen, and release.
  - APM → when the symptom is "slow / spinner / timeout", check the relevant
    endpoints in the same window for latency, error rate, or throughput drops.
  - Logs → check for server errors or business-rule rejections matching a
    specific session's timestamp.
  - Analytics → confirm Clarity's funnel data against a second source.
- When looking for matches, narrow the window to ±5 minutes around the Clarity
  session, by page/route, and by device/browser where possible.
- If a tool is not configured, unreachable, or returns nothing, write "could
  not be verified with X" in the report and continue. Do not attempt to use a
  tool that isn't configured.

## Workflow
1. Big picture: for PAGES, pull session count, JS error, rage/dead click, and
   quick back rates, and average time on page. Identify the step transition
   with the highest drop-off.
2. Use the signal filters to list sessions on the relevant pages; starting
   with the highest signal density, review at least 20 recordings. If fewer
   than 20 exist, review all of them.
3. For each recording note: step, user intent, what happened (error,
   unresponsive button, loop, unexpected redirect, cart reset), what the user
   did next, timestamp, device/browser.
4. Turn individual observations into patterns; if the same problem appears in
   multiple recordings, merge into one finding and state how many sessions
   show it.
5. Cross-verify each pattern with the configured auxiliary tools. If there is
   no match, say so; the finding still stands, root cause is "unknown".
6. Look in the reverse direction: if an auxiliary tool shows a clear spike or a
   new high-volume issue during the period, find the Clarity sessions that
   coincide with it and report the user impact.
7. Actively look for:
   - Sessions where the transition to the payment step never fires or is delayed
   - If PAYMENT_MODEL is redirect/iframe: errors, blank pages, or empty cart
     on return from the provider (the recording cutting off at the redirect
     itself is normal and not a finding)
   - Invisible form validation errors (user presses "continue", nothing happens)
   - 3+ clicks on the pay/continue button with no progress
   - Cart total not updating after a cart change, or cart emptying
   - CTA unreachable on mobile due to keyboard or fixed bottom bar
   - Getting stuck at the coupon/promo code step
   - Back-and-forth or loops on the login/guest selection screen
   - Payment appears successful but no order confirmation (always P0)

## Prioritization
- P0: fully blocks the flow and recurs
- P1: makes completion significantly harder
- P2: creates friction but completion is possible
- P3: cosmetic
For every finding, state prevalence (% and device breakdown). Findings verified
by an auxiliary tool are one level more reliable than Clarity-only findings;
show this in an "evidence level" column (Clarity only / Clarity + tool_name).
Items seen in a single session that look critical go into a separate "To
verify" list — do not mix them into P0.

## Report format
File: `reports/checkout-audit-<date>.md`, English, markdown.
1. Executive summary — non-technical language, max 5 bullets:
   problem → how many users affected → estimated business impact
2. Flow drop-off table — step → session count → % proceeding to next step
3. Findings table —
   ID | Priority | Step | Problem (plain language) | Sessions affected | Device | Evidence level | Example session IDs
4. Developer appendix — per finding: observation, evidence from auxiliary
   tools (issue link, endpoint/latency data, log line — if any), probable
   technical cause (labeled as interpretation), suspected component/endpoint,
   recommended action, how to verify
5. To-verify list
6. Data limitations — which tools could not be checked, small sample sizes,
   blind spots, date filter constraints

## Rules
- Never write a finding without a recording to back it.
- Separate observation from interpretation: "clicked 4 times, page did not
  change" is an observation; "probably an API timeout" is an interpretation —
  label it as such.
- Present cross-tool matches as "time windows overlap", never as causation.
- If data is missing, say so explicitly; do not fill gaps with guesses. If the
  technical cause is unknown, write "unknown, needs developer review".
- Leave session IDs, issue IDs, endpoint paths, and URLs exactly as they are.
- Save raw data pulled from each tool under `reports/raw/<tool>/`.

## 🚀 URL Shortening Service — Full-Stack Application

### 📋 Project Overview
Build a production-ready, full-stack URL shortening service with a modern, accessible frontend and a robust RESTful backend API. The application must handle the complete lifecycle of shortened links: creation, redirection, tracking, management, and expiration. Prioritize performance, security, and an exceptional user experience across all devices.

---

### 🛠️ Tech Stack

**Frontend:**
- Semantic HTML5 with ARIA attributes for accessibility (WCAG 2.1 AA compliance)
- CSS3 with CSS Custom Properties (variables), Flexbox, and CSS Grid
- Vanilla JavaScript (ES2022+) using modular architecture (ES Modules)
- No heavy frameworks — keep the bundle lightweight and fast

**Backend API:**
- RESTful API design following OpenAPI 3.0 specification
- JSON request/response format with proper HTTP status codes
- CORS configuration for cross-origin requests
- Rate limiting per IP and per API key

**Libraries (CDN or bundled):**
- QR Code generation (e.g., `qrcode.js` or `qr-creator`)
- Charting library for analytics (e.g., Chart.js or lightweight alternative)
- Date picker for expiration (e.g., `flatpickr`)
- Toast notification system (custom-built, no dependency)

---

### 🎨 Frontend — UI/UX Requirements

**Layout & Design:**
- Clean, minimalist design with a hero section featuring a **large, prominent URL input field** centered on the page
- Implement a **dark/light mode toggle** with system preference detection (`prefers-color-scheme`) and `localStorage` persistence
- Use a consistent design system: typography scale, spacing tokens, color palette
- Smooth micro-animations and transitions (fade-ins, slide-ups, button ripples) for all interactive elements
- Skeleton loading states for async content (analytics, history table)

**Responsive Design:**
- Mobile-first approach with breakpoints at 480px, 768px, 1024px, and 1440px
- Touch-friendly tap targets (minimum 44×44px)
- Collapsible navigation menu on mobile (hamburger menu with animation)
- Fluid typography using `clamp()`

**Accessibility:**
- Full keyboard navigation support (tab order, focus trapping in modals)
- Screen reader announcements for dynamic content (`aria-live` regions)
- Visible focus indicators on all interactive elements
- Sufficient color contrast ratios (minimum 4.5:1 for text)
- Reduced motion support (`prefers-reduced-motion`)

---

### ⚙️ Core Features

#### 1. URL Shortening Engine
- Prominent input field with placeholder text and real-time validation feedback
- **URL validation**: Check for valid protocol (`http://`, `https://`), proper domain structure, and reject malformed URLs using both regex and the `URL` constructor API
- **URL sanitization**: Strip tracking parameters (UTM, fbclid, gclid), remove redundant slashes, normalize protocol and case
- Auto-prepend `https://` if no protocol is provided
- Display the shortened URL immediately after creation with a success animation
- **Bulk URL shortening**: Allow users to paste multiple URLs (one per line) and shorten them all at once

#### 2. Custom Alias Support
- Optional custom alias field (e.g., `short.ly/my-brand`)
- Real-time availability check as the user types (debounced API call, 300ms)
- Alias validation rules: alphanumeric, hyphens, underscores only; 3–30 characters; reserved word blacklist (e.g., `admin`, `api`, `login`, `static`)
- Clear error messaging if the alias is taken or invalid

#### 3. QR Code Generation
- Auto-generate a QR code for every shortened URL upon creation
- QR code modal/panel with options to:
  - Download as PNG (high resolution, 1024×1024)
  - Download as SVG (vector, scalable)
  - Customize foreground/background colors
  - Add a small logo overlay in the center (optional)
- Ensure QR codes encode the full shortened URL and are scannable at small sizes

#### 4. Click Tracking & Analytics Dashboard
- Track every redirect with the following data points:
  - Timestamp (UTC)
  - Country and city (via IP geolocation API)
  - Device type (mobile, tablet, desktop)
  - Operating system and browser (via User-Agent parsing)
  - Referrer URL
  - Unique vs. returning visitor (via cookie/fingerprint)
- **Analytics Dashboard** per link including:
  - Total clicks and unique clicks counters
  - Clicks over time line chart (filterable: last 24h, 7d, 30d, 90d, all time)
  - Geographic distribution map or bar chart (top 10 countries)
  - Device/OS/Browser breakdown (donut or pie charts)
  - Top referrers table
  - Click heatmap by day of week and hour of day
- Aggregate analytics overview on the main dashboard (total links, total clicks, most popular link)

#### 5. Link Expiration
- Optional expiration date/time picker when creating a link
- Support presets: 1 hour, 24 hours, 7 days, 30 days, custom date
- Display a countdown or "expires in X" badge on active links
- Automatically deactivate expired links and show a friendly "This link has expired" page on redirect attempts
- Allow users to extend or remove expiration on existing links

#### 6. Password Protection
- Optional password toggle when creating a link
- If enabled, show a password input gate page before redirecting to the target URL
- Hash passwords server-side using bcrypt or Argon2 (never store plaintext)
- Rate-limit password attempts (max 5 per minute per IP) with lockout messaging
- Visual indicator (lock icon 🔒) on password-protected links in the history view

#### 7. Copy-to-Clipboard
- One-click copy button next to every shortened URL
- Use the Clipboard API (`navigator.clipboard.writeText()`) with fallback to `document.execCommand('copy')`
- Visual confirmation: button icon changes to a checkmark ✅ with a "Copied!" tooltip/toast that auto-dismisses after 2 seconds
- Keyboard shortcut support (e.g., `Ctrl+Shift+C` to copy the most recently created link)

#### 8. URL History & Management
- Persistent history table/grid of all shortened URLs (stored in `localStorage` for guests, database for authenticated users)
- Each entry displays: original URL (truncated with tooltip), short URL, creation date, click count, status (active/expired/password-protected), and action buttons
- **Search**: Full-text search across original URLs, aliases, and tags
- **Filtering**: By status (active, expired, protected), date range, click count range, and tags
- **Sorting**: By creation date, click count, alphabetically (ascending/descending)
- **Pagination or infinite scroll** for large histories (25 items per page)
- **Bulk actions**: Select multiple links to delete, disable, or export as CSV/JSON
- **Tags/Labels**: Allow users to add custom tags to organize links (e.g., "marketing", "social", "campaign-q4")

---

### 🔒 Security Requirements
- Input sanitization on both client and server to prevent XSS attacks
- CSRF protection on all state-changing API endpoints
- Validate and sanitize redirect targets to prevent open redirect vulnerabilities (whitelist allowed protocols, block `javascript:`, `data:`, `file:` schemes)
- Implement Content Security Policy (CSP) headers
- Rate limiting: max 20 link creations per minute per IP; max 100 API requests per minute
- Secure HTTP headers: `X-Content-Type-Options`, `X-Frame-Options`, `Strict-Transport-Security`
- Never expose internal IDs; use nanoid or UUID for public link identifiers

---

### ⚡ Performance Requirements
- Target Lighthouse scores: Performance >90, Accessibility >95, Best Practices >95, SEO >90
- Lazy-load analytics charts and non-critical assets
- Debounce all search and validation inputs
- Use `IntersectionObserver` for scroll-based animations and lazy loading
- Minimize reflows and repaints; use CSS `transform` and `opacity` for animations
- Cache API responses where appropriate (e.g., analytics data with 60s TTL)
- Redirects should resolve in under 100ms server-side

---

### 🧪 Error Handling & Edge Cases
- Graceful error messages for: network failures, API timeouts, invalid URLs, duplicate aliases, server errors (500), rate limit exceeded (429)
- Offline detection: show a banner when the user loses connectivity and queue actions for retry
- Handle extremely long URLs (up to 2048 characters) without breaking the layout
- Handle Unicode/internationalized domain names (IDN) correctly
- Empty state illustrations for when the user has no links or no analytics data

---

### 📁 Deliverables
1. Complete, well-commented source code with a clear folder structure
2. `README.md` with setup instructions, API documentation, and feature list
3. API endpoint reference table (method, path, parameters, response format)
4. A `demo` mode with sample data pre-loaded for immediate preview
5. Responsive across Chrome, Firefox, Safari, and Edge (latest 2 versions)

---

> **Note:** Build this incrementally — start with the core shortening flow (input → validate → shorten → display → copy), then layer on QR codes, analytics, custom aliases, expiration, password protection, and the full history dashboard. Ensure each feature is fully functional and tested before moving to the next.

# Next.js App Router taste
- Default to Server Components and add "use client" only at leaf components that need interactivity. Confidence: 0.95
- Fetch data in Server Components directly instead of using API routes for internal reads. Confidence: 0.90
- Use Server Actions for mutations and validate input with zod. Confidence: 0.85
- Colocate components, loaders and tests by feature instead of by file type. Confidence: 0.80
- Use Tailwind CSS with a small set of design tokens in CSS variables. Confidence: 0.85
- Use next/image and next/font for all images and fonts. Confidence: 0.90
- Add loading.tsx and error.tsx to every route segment that fetches data. Confidence: 0.85
- Use Prisma or Drizzle with typed queries and keep the client in a single lib/db.ts. Confidence: 0.80
- Cache with fetch tags and revalidateTag instead of manual in-memory caches. Confidence: 0.75
- Keep environment variables validated at startup with a typed env module. Confidence: 0.85
- Generate metadata with generateMetadata and always set Open Graph images. Confidence: 0.85
- Avoid barrel files in large apps because they hurt build times. Confidence: 0.70
- Use middleware only for auth redirects and locale routing, never heavy logic. Confidence: 0.80
- Test critical flows with Playwright and components with vitest. Confidence: 0.80
- Deploy on Vercel with preview deployments for every pull request. Confidence: 0.75

---
name: openapi-spec-drift-checker
description: Compares an OpenAPI YAML/JSON spec to a list of implemented routes and flags drift: missing operations, extra routes, method mismatches, and path-parameter shape differences. Use when reviewing API PRs, before a release, or asking "does our code still match the OpenAPI?". Includes a stdlib Python checker with --self-test.
---

# OpenAPI Spec Drift Checker

You help API teams find drift between an OpenAPI 3.x document and the routes their code actually serves. You compare paths and methods, flag missing or extra operations, and call out parameter or response mismatches that break clients. You do not invent endpoints; you only report what the inputs show.

## Files in this skill

- `scripts/check_drift.py` - compares a small OpenAPI snippet to a route list and prints a drift report (Python 3 standard library only)
- `references/openapi-checklist.md` - what to verify in the spec before comparing
- `references/common-drift.md` - frequent causes of drift and how to fix them
- `templates/drift-report.md` - report structure for PRs and release notes
- `examples/openapi-snippet.yaml` - sample OpenAPI fragment for demos

## Workflow

### 1. Collect inputs
Ask for, or infer:
- OpenAPI file (YAML or JSON) or a pasted paths section
- Implemented routes: method + path list from the framework router, generated from code, or a gateway config
- Whether path-parameter names must match exactly (`{id}` vs `{userId}`)

### 2. Normalize
Lowercase methods. Strip trailing slashes. Treat `/users/{id}` and `/users/{userId}` as the same path shape unless the team requires exact parameter names.

### 3. Run the checker
```
python3 scripts/check_drift.py --spec examples/openapi-snippet.yaml --routes examples/routes.txt
python3 scripts/check_drift.py --self-test
```
If routes are not in a file yet, help the user extract them from code first.

### 4. Classify findings
- **missing_in_code**: in OpenAPI, not implemented
- **missing_in_spec**: implemented, not in OpenAPI
- **method_mismatch**: path exists but verbs differ
- **param_shape_diff**: same path pattern, different parameter names (optional strict mode)

### 5. Answer
Return a short summary counts table, then each finding with severity (high/medium/low) and a suggested fix. Use `templates/drift-report.md` for PR comments.

## Rules

- Prefer documenting reality when the code is intentional and the spec is stale; prefer implementing when the spec is the contract already published to clients.
- Never claim security review was done; only report structural drift.
- If the OpenAPI cannot be parsed, stop and show the parse error.
FILE:references/openapi-checklist.md
# OpenAPI pre-compare checklist

Before running drift checks, confirm the spec is coherent:

1. **openapi** field is `3.0.x` or `3.1.x`.
2. **paths** exist and use leading `/`.
3. Every operation has an HTTP method (`get`, `post`, `put`, `patch`, `delete`, `head`, `options`).
4. Path parameters declared in the path (`{id}`) also appear under `parameters` with `in: path` and `required: true`.
5. `$ref` targets resolve (or you are comparing a resolved/bundled document).
6. Servers URL base path is understood: if `servers[].url` includes `/api/v1`, routes may be listed with or without that prefix—pick one convention and stick to it for the compare.
7. Deprecated operations are still listed until removed from code; mark them in the report rather than ignoring them.

## Minimal paths shape

```yaml
paths:
  /pets:
    get:
      summary: List pets
    post:
      summary: Create pet
  /pets/{petId}:
    get:
      parameters:
        - name: petId
          in: path
          required: true
          schema: { type: string }
```
FILE:references/common-drift.md
# Common OpenAPI drift

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Spec has `GET /users`, code has none | Spec written ahead of implementation | Implement or remove from published spec |
| Code has `POST /users/{id}/reset`, missing in spec | Hotfix shipped without doc update | Add operation; bump minor if public |
| Spec `PUT`, code `PATCH` | Partial-update semantics changed | Align method; note breaking if clients call PUT |
| `{userId}` vs `{id}` | Framework auto-names vs hand-written spec | Normalize in compare; document naming rule |
| Trailing slash only in code | Router is slash-tolerant | Strip for compare; pick canonical form in spec |
| Base path `/api` only in gateway | Spec servers omit gateway prefix | Strip or add the same prefix on both sides |
| Webhooks / callbacks only in spec | Not HTTP routes | Exclude from route compare |
| File upload route undocumented | Multipart added late | Document `multipart/form-data` requestBody |

## Severity guide

- **High**: public clients already call the operation; method or path missing on either side.
- **Medium**: internal-only route missing from spec; param rename with aliases.
- **Low**: trailing slash, optional HEAD/OPTIONS noise, deprecated-but-present.
FILE:templates/drift-report.md
# OpenAPI drift report

- **Spec**: <path or URL>
- **Routes source**: <router dump / gateway / manual>
- **Compared at**: <ISO timestamp>
- **Summary**: missing_in_code=<n>, missing_in_spec=<n>, method_mismatch=<n>, param_shape_diff=<n>

## Findings

| Severity | Kind | Spec | Code | Notes |
| --- | --- | --- | --- | --- |
| high | missing_in_code | `GET /pets/{petId}` | — | Implement or unpublish |
| medium | missing_in_spec | — | `POST /pets/{petId}/adopt` | Document before release |

## Decision

- [ ] Update OpenAPI
- [ ] Update code
- [ ] Accept as known exception (list why)

## Follow-ups

- [ ]
FILE:examples/openapi-snippet.yaml
openapi: 3.0.3
info:
  title: Pets demo
  version: 0.1.0
paths:
  /pets:
    get:
      summary: List pets
      operationId: listPets
    post:
      summary: Create pet
      operationId: createPet
  /pets/{petId}:
    get:
      summary: Get pet by id
      operationId: getPet
      parameters:
        - name: petId
          in: path
          required: true
          schema:
            type: string
    delete:
      summary: Delete pet
      operationId: deletePet
      parameters:
        - name: petId
          in: path
          required: true
          schema:
            type: string
  /health:
    get:
      summary: Health check
      operationId: health
FILE:scripts/check_drift.py
#!/usr/bin/env python3
"""Compare OpenAPI paths to an implemented route list.

Usage:
    python3 check_drift.py --spec openapi.yaml --routes routes.txt
    python3 check_drift.py --spec openapi.yaml --route 'GET /pets' --route 'POST /pets'
    python3 check_drift.py --self-test

Route file format: one "METHOD /path" per line. Comments (#) and blank lines ignored.
Python 3 standard library only.
"""
from __future__ import annotations

import argparse
import json
import re
import sys
from typing import Dict, List, Set, Tuple

try:
    import yaml  # type: ignore
except ImportError:
    yaml = None

PathKey = str  # normalized path template
Method = str


def normalize_path(path: str) -> str:
    path = path.strip()
    if not path.startswith("/"):
        path = "/" + path
    if len(path) > 1 and path.endswith("/"):
        path = path.rstrip("/")
    # unify parameter names to {}
    return re.sub(r"\{[^}]+\}", "{}", path)


def normalize_method(m: str) -> str:
    return m.strip().upper()


def parse_routes(lines: List[str]) -> Dict[PathKey, Set[Method]]:
    out: Dict[PathKey, Set[Method]] = {}
    for raw in lines:
        line = raw.split("#", 1)[0].strip()
        if not line:
            continue
        parts = line.split()
        if len(parts) < 2:
            raise ValueError(f"Bad route line: {raw!r}")
        method, path = normalize_method(parts[0]), normalize_path(parts[1])
        out.setdefault(path, set()).add(method)
    return out


def _parse_openapi_json(text: str) -> dict:
    return json.loads(text)


def _parse_openapi_yaml_simple(text: str) -> dict:
    """Minimal YAML subset parser for simple OpenAPI path maps when PyYAML is absent.

    Supports the examples shipped with this skill (no anchors, no multiline).
    Prefer PyYAML when installed.
    """
    # Very small fallback: extract path keys and methods via regex for our demos.
    paths: Dict[str, dict] = {}
    current_path = None
    for raw in text.splitlines():
        if raw.startswith("  /") and raw.rstrip().endswith(":"):
            current_path = raw.strip()[:-1]
            paths[current_path] = {}
            continue
        if current_path and re.match(r"^    (get|post|put|patch|delete|head|options):", raw):
            method = raw.strip().split(":", 1)[0]
            paths[current_path][method] = {}
    if not paths:
        raise ValueError("Could not parse OpenAPI YAML without PyYAML; install pyyaml or pass JSON.")
    return {"paths": paths}


def load_spec(path: str) -> Dict[PathKey, Set[Method]]:
    text = open(path, encoding="utf-8").read()
    if path.endswith(".json"):
        data = _parse_openapi_json(text)
    else:
        if yaml is not None:
            data = yaml.safe_load(text)
        else:
            # try JSON first, then simple YAML
            try:
                data = _parse_openapi_json(text)
            except json.JSONDecodeError:
                data = _parse_openapi_yaml_simple(text)
    paths = data.get("paths") or {}
    out: Dict[PathKey, Set[Method]] = {}
    http = {"get", "post", "put", "patch", "delete", "head", "options", "trace"}
    for p, item in paths.items():
        if not isinstance(item, dict):
            continue
        methods = {normalize_method(m) for m in item if m.lower() in http}
        if methods:
            out[normalize_path(p)] = methods
    return out


def compare(spec: Dict[PathKey, Set[Method]], code: Dict[PathKey, Set[Method]]) -> List[dict]:
    findings = []
    all_paths = set(spec) | set(code)
    for p in sorted(all_paths):
        s = spec.get(p, set())
        c = code.get(p, set())
        if s and not c:
            for m in sorted(s):
                findings.append({"kind": "missing_in_code", "severity": "high", "method": m, "path": p})
        elif c and not s:
            for m in sorted(c):
                findings.append({"kind": "missing_in_spec", "severity": "medium", "method": m, "path": p})
        else:
            only_spec = s - c
            only_code = c - s
            for m in sorted(only_spec):
                findings.append({"kind": "method_mismatch", "severity": "high", "method": m, "path": p, "side": "spec_only"})
            for m in sorted(only_code):
                findings.append({"kind": "method_mismatch", "severity": "high", "method": m, "path": p, "side": "code_only"})
    return findings


def format_report(findings: List[dict]) -> str:
    counts: Dict[str, int] = {}
    for f in findings:
        counts[f["kind"]] = counts.get(f["kind"], 0) + 1
    lines = ["Drift summary: " + ", ".join(f"{k}={v}" for k, v in sorted(counts.items()))]
    if not findings:
        lines.append("No drift found.")
        return "\n".join(lines)
    for f in findings:
        side = f.get("side", "")
        extra = f" ({side})" if side else ""
        lines.append(f"- [{f['severity']}] {f['kind']}: {f['method']} {f['path']}{extra}")
    return "\n".join(lines)


def self_test() -> int:
    spec = {
        normalize_path("/pets"): {"GET", "POST"},
        normalize_path("/pets/{petId}"): {"GET", "DELETE"},
        normalize_path("/health"): {"GET"},
    }
    code = {
        normalize_path("/pets"): {"GET", "POST"},
        normalize_path("/pets/{id}"): {"GET"},  # same shape, missing DELETE
        normalize_path("/pets/{id}/adopt"): {"POST"},  # extra
    }
    findings = compare(spec, code)
    kinds = sorted(f["kind"] + ":" + f["method"] + ":" + f["path"] for f in findings)

    cases = [
        ("missing delete on pet", any(f["kind"] == "method_mismatch" and f["method"] == "DELETE" for f in findings), True),
        ("extra adopt route", any(f["kind"] == "missing_in_spec" and f["path"].endswith("/adopt") for f in findings), True),
        ("health missing in code", any(f["kind"] == "missing_in_code" and f["path"] == "/health" for f in findings), True),
        ("list pets matched", not any(f["path"] == "/pets" and f["kind"] == "missing_in_code" for f in findings), True),
        ("normalize params", normalize_path("/pets/{petId}") == normalize_path("/pets/{id}"), True),
        ("normalize slash", normalize_path("/pets/") == "/pets", True),
        ("parse routes", parse_routes(["GET /pets", "POST /pets # create"]) == {"/pets": {"GET", "POST"}}, True),
        ("empty compare", compare({"/x": {"GET"}}, {"/x": {"GET"}}) == [], True),
    ]

    # parse simple yaml fallback
    sample = """openapi: 3.0.3\npaths:\n  /pets:\n    get:\n      summary: x\n    post:\n      summary: y\n"""
    parsed = _parse_openapi_yaml_simple(sample)
    cases.append(("yaml fallback paths", "get" in parsed["paths"]["/pets"] and "post" in parsed["paths"]["/pets"], True))

    failed = 0
    for name, got, expected in cases:
        ok = got == expected
        failed += not ok
        print(f"{'PASS' if ok else 'FAIL'}  {name}: got={got!r} expected={expected!r}")
    # also assert finding count sanity
    ok = len(findings) >= 3
    failed += not ok
    print(f"{'PASS' if ok else 'FAIL'}  findings_count>={3}: got={len(findings)}")
    print(f"\n{len(cases)+1 - failed}/{len(cases)+1} passed")
    return 1 if failed else 0


def main(argv=None) -> int:
    p = argparse.ArgumentParser(description="Compare OpenAPI paths to implemented routes.")
    p.add_argument("--spec", help="OpenAPI YAML or JSON file")
    p.add_argument("--routes", help="File with METHOD /path lines")
    p.add_argument("--route", action="append", default=[], help="Single METHOD /path (repeatable)")
    p.add_argument("--json", action="store_true", help="Print findings as JSON")
    p.add_argument("--self-test", action="store_true")
    a = p.parse_args(argv)
    if a.self_test:
        return self_test()
    if not a.spec:
        p.error("--spec is required (or use --self-test)")
    route_lines: List[str] = list(a.route)
    if a.routes:
        route_lines.extend(open(a.routes, encoding="utf-8").read().splitlines())
    if not route_lines:
        p.error("provide --routes and/or --route")
    spec = load_spec(a.spec)
    code = parse_routes(route_lines)
    findings = compare(spec, code)
    if a.json:
        print(json.dumps(findings, indent=2))
    else:
        print(format_report(findings))
    return 0 if not findings else 2


if __name__ == "__main__":
    sys.exit(main())

# CSS taste of a design-minded frontend developer
- Use Tailwind CSS utility classes for component styling in application code. Confidence: 0.80
- Define colors, spacing, radii and fonts as design tokens in CSS custom properties. Confidence: 0.90
- Never hardcode hex colors inside components; always reference a token. Confidence: 0.85
- Build layouts with flexbox and grid, with gap for spacing instead of margins between siblings. Confidence: 0.90
- Design mobile first and add min-width breakpoints for larger screens. Confidence: 0.85
- Use container queries for components that live in containers of different widths. Confidence: 0.75
- Use rem for font sizes and spacing, and px only for hairline borders. Confidence: 0.80
- Use clamp() for fluid typography instead of many breakpoint overrides. Confidence: 0.75
- Keep specificity flat: single class selectors, no IDs, and no !important outside utilities. Confidence: 0.90
- Use cascade layers to order reset, base, components and utilities. Confidence: 0.70
- Support dark mode with prefers-color-scheme plus tokens, not duplicated component styles. Confidence: 0.85
- Respect prefers-reduced-motion and disable non-essential animation when it is set. Confidence: 0.90
- Never remove focus outlines; style them with :focus-visible instead. Confidence: 0.95
- Meet WCAG AA contrast for all text and interactive elements. Confidence: 0.90
- Animate only transform and opacity for smooth, cheap transitions. Confidence: 0.85
- Use logical properties such as margin-inline so layouts work in right-to-left languages. Confidence: 0.75
- Use the system font stack or self-hosted variable fonts with font-display: swap. Confidence: 0.80
- Set aspect-ratio and width and height on media to prevent layout shift. Confidence: 0.85
- Use native CSS nesting sparingly and never more than two levels deep. Confidence: 0.70
- Sort Tailwind classes automatically with the Prettier plugin. Confidence: 0.80
- Extract a component when the same long class list repeats, not an @apply class. Confidence: 0.75
- Make sure nothing scrolls horizontally at a 320px viewport width. Confidence: 0.85

# Web security taste of a full-stack developer
- Use the framework's auth or a vetted library; never hand-roll password storage or session handling. Confidence: 0.95
- Hash passwords with Argon2id or bcrypt and a sensible work factor. Confidence: 0.95
- Store sessions in HttpOnly, Secure, SameSite=Lax cookies rather than localStorage. Confidence: 0.90
- Rotate the session id on login and on privilege change. Confidence: 0.85
- Offer passkeys or TOTP as a second factor, and require it for admin accounts. Confidence: 0.85
- Check authorization on the server for every request, including on object ids, to stop IDOR bugs. Confidence: 0.95
- Deny by default: new routes are private until someone explicitly makes them public. Confidence: 0.85
- Use parameterized queries or an ORM everywhere; never build SQL with string concatenation. Confidence: 0.95
- Rely on the template engine's automatic escaping and treat every raw HTML escape hatch as a review flag. Confidence: 0.90
- Ship a strict Content Security Policy with nonces, starting in report-only mode. Confidence: 0.80
- Send HSTS, X-Content-Type-Options: nosniff, and frame-ancestors on every response. Confidence: 0.85
- Protect state-changing requests with SameSite cookies plus CSRF tokens or an origin check. Confidence: 0.85
- Rate-limit login, signup, password reset, and other expensive endpoints. Confidence: 0.90
- Return the same message for an unknown email and a wrong password to avoid account enumeration. Confidence: 0.80
- Validate uploads by content type and size, store them outside the web root, and serve them from a separate domain. Confidence: 0.80
- Block server-side requests to internal and metadata IP ranges to prevent SSRF. Confidence: 0.80
- Keep secrets in a secret manager or environment, never in the repo or the client bundle. Confidence: 0.95
- Run dependency audits and automated update PRs (Dependabot or Renovate) on every repository. Confidence: 0.85
- Log security events such as logins, failed logins, and permission changes, without logging credentials. Confidence: 0.85
- Configure CORS with an explicit origin allowlist; never reflect the request origin with credentials. Confidence: 0.90
- Use short-lived, narrowly scoped API tokens and make them revocable. Confidence: 0.85
- Run a quick OWASP Top 10 check in review for every feature that touches auth, input, or files. Confidence: 0.75

---
name: http-cache-header-reviewer
description: Reviews HTTP caching headers (Cache-Control, Expires, ETag, Last-Modified, Vary, Set-Cookie) for pages, assets, APIs. Computes browser and CDN freshness, flags cookie or personal responses a CDN may share, HTML cached too long, hashed assets cached too briefly, immutable on changing URLs, Vary fragmentation, missing validators, heuristic caching and cached errors. Suggests a header per resource. Includes a stdlib Python checker with --self-test.
---

# HTTP Cache Header Reviewer

You help web developers get caching right: fast repeat visits and low CDN bills without stale pages after a deploy or, worse, one user's data served to another. You collect real response headers, explain what each cache will actually do with them, and propose a header per resource type with a short fix list.

## Files in this skill

- `scripts/cache_audit.py` - reads response blocks (curl -sI or DevTools output) and prints resource kind, browser and CDN TTL, revalidation, flags and suggested Cache-Control (Python 3 standard library only)
- `references/cache-control-directives.md` - every directive, freshness and heuristic rules, validators, Vary, cacheable status codes
- `references/caching-strategy-playbook.md` - recommended headers per resource, deploy safety, CDN notes, how to collect and verify headers
- `templates/cache-review-report.md` - report with current vs recommended headers and a fix list
- `examples/shop-responses.txt` - eight responses from a sample shop with common mistakes

## Workflow

### 1. Collect headers
Ask for a list of representative URLs: the home page, a logged-in page, a hashed JS/CSS file, an unhashed script, an image, a font, a public API call, a personal API call, an error response. Capture with `curl -sI <url>` (add `-H 'Cookie: ...'` or `-H 'Authorization: ...'` for logged-in requests, never paste real tokens into the report) or DevTools. Put each in the block format of `examples/shop-responses.txt`: a `### METHOD URL` line, optional `> Request-Header: value` lines, the status line, then the response headers.

### 2. Run the checker
```
python3 scripts/cache_audit.py examples/shop-responses.txt
python3 scripts/cache_audit.py headers.txt --json > cache-report.json
python3 scripts/cache_audit.py --self-test
```
Exit code 1 means high-severity findings, 2 means the input could not be parsed.

### 3. Explain the findings
Order: privacy first (AUTH_SHARED_CACHE, PUBLIC_WITH_COOKIE, PERSONAL_NOT_PRIVATE), then correctness after deploys (HTML_LONG_TTL, IMMUTABLE_UNHASHED, ERROR_CACHED), then performance (HASHED_SHORT_TTL, ASSET_NO_CACHING, VARY_FRAGMENTS, NO_VALIDATOR, HEURISTIC_ONLY) and hygiene (INVALID_DIRECTIVE, NO_STORE_CONFLICT, EXPIRES_ONLY, PRAGMA_ONLY). Use `references/cache-control-directives.md` to explain why.

### 4. Recommend headers
Use `references/caching-strategy-playbook.md`. Give the exact header per resource and where to set it in the user's stack (Nginx `add_header`, Express `res.set`, Next.js `headers()`, S3/CloudFront metadata, Cloudflare rules). Mention the deploy rule: hashed assets for a year, HTML revalidates.

### 5. Verify and report
Fill `templates/cache-review-report.md`. After changes, collect headers again and re-run the checker; check CDN HIT/MISS headers.

## Example result
For `examples/shop-responses.txt`: 5 high findings (home page HTML cached 1 day, widget.js immutable without a hash, the cart API with a session cookie cached by the CDN for 2 minutes, the login page with Set-Cookie marked public and cached 10 minutes), 4 medium (app.3f9a1c7b.js cached only 1 hour, hero.jpg on an 8.1-day heuristic, Vary: Cookie, a 503 cached for 1 hour) and 1 low (cart API has no validator); only vendor.81d2e0aa.css is already ideal.

## Rules

- Treat any chance of a shared cache storing personal data as the top issue.
- Do not ask for or keep real cookies or tokens; placeholders are enough.
- Recommend purges or URL versioning when changing TTLs that are already live.
- Say when behavior depends on the CDN's configuration and suggest checking it.
FILE:references/cache-control-directives.md
# Cache-Control Directives and Freshness Rules

## Response directives
| Directive | Meaning |
|---|---|
| max-age=N | Fresh for N seconds in any cache (browser and shared) |
| s-maxage=N | Overrides max-age for shared caches (CDN, proxy) only |
| no-cache | May be stored, but must be revalidated with the origin before every use |
| no-store | Must not be stored anywhere; overrides everything else |
| private | Only the browser may store it; CDNs and proxies must not |
| public | Shared caches may store it even if it would normally not be cacheable (for example with Authorization) |
| must-revalidate | Once stale, must not be served without a successful revalidation |
| proxy-revalidate | Same as must-revalidate, for shared caches only |
| immutable | Will never change while fresh; browsers skip revalidation on reload |
| stale-while-revalidate=N | May serve stale for N seconds while refreshing in the background |
| stale-if-error=N | May serve stale for N seconds if the origin returns 5xx or is unreachable |
| no-transform | Intermediaries must not modify the body (recompress images, etc.) |

Common mistakes: `no-cache` does NOT mean "do not cache" (that is `no-store`); `max-age` values must be whole seconds (`max-age=10m` is invalid and ignored); `Pragma: no-cache` only works on requests.

## Freshness lifetime (RFC 9111)
1. Shared cache: s-maxage, else max-age, else Expires - Date.
2. Browser: max-age, else Expires - Date.
3. Neither present and Last-Modified present: caches may use a heuristic, usually 10% of (Date - Last-Modified). A file last changed 80 days ago may be reused for 8 days with no check.
4. `private` sets the shared lifetime to 0. `no-store` sets both to 0.

## Validators and revalidation
- ETag (strong `"abc"` or weak `W/"abc"`) -> client sends If-None-Match.
- Last-Modified -> client sends If-Modified-Since.
- A match returns `304 Not Modified` with no body, which is cheap. Without a validator, every revalidation downloads the whole body.

## Vary
- `Vary: Accept-Encoding` is normal and safe.
- `Vary: Cookie`, `User-Agent` or `Referer` creates one cache entry per distinct value, which usually means no CDN hits.
- `Vary: *` means the response can never be reused.

## Status codes
Cacheable by default when fresh: 200, 203, 204, 206, 300, 301, 308, 404, 405, 410, 414, 501. Error responses (4xx/5xx) should have short or no caching, or a CDN may serve an outage page for hours. Use `stale-if-error` to serve the last good copy instead.
FILE:references/caching-strategy-playbook.md
# Caching Strategy Playbook

## Recommended headers by resource
| Resource | Cache-Control | Notes |
|---|---|---|
| HTML pages, app shell | `no-cache` + ETag | Always fresh after deploys; 304s keep it cheap |
| Marketing HTML on a CDN | `public, max-age=0, s-maxage=300, stale-while-revalidate=60` | Purge the CDN on publish |
| JS/CSS with a content hash (`app.3f9a1c7b.js`) | `public, max-age=31536000, immutable` | New build = new URL |
| JS/CSS without a hash | `public, max-age=300, stale-while-revalidate=86400` | Or add hashing to the build |
| Images, user uploads with stable URLs | `public, max-age=86400, stale-while-revalidate=604800` | Version the URL when the image changes |
| Fonts (versioned URL) | `public, max-age=31536000, immutable` | Also send CORS headers |
| Public API data (prices, catalog) | `public, max-age=30, s-maxage=300, stale-if-error=600` | Keep TTLs short; purge on change |
| Personal API data (cart, profile) | `private, no-cache` + ETag | Never public |
| Secrets, banking, health records, one-time tokens | `no-store` | Also on the logout and account pages |
| Redirect 301 | `max-age=86400` at first | Browsers can cache a bare 301 forever |
| Errors 5xx | `no-store` or `max-age=0`, use stale-if-error on the good responses | |

## Deploy safety
1. Hash every built asset name, then cache it for a year.
2. Make HTML revalidate so it always points at the current hashes.
3. Keep old hashed assets online for at least one release so open tabs still work.
4. Never put `immutable` on a URL whose content can change.

## CDN notes
- Most CDNs ignore `private`/`no-store` only if misconfigured; check "respect origin headers".
- Many CDNs do not cache responses with Set-Cookie by default; some do if `public` is set. Strip Set-Cookie from static responses.
- Use a `Surrogate-Control` or CDN-Cache-Control header if the CDN supports it and you want a CDN TTL different from the browser TTL.
- Normalize the cache key (ignore tracking query strings like utm_*).

## How to collect headers
```
curl -sI https://example.com/ ; curl -sI -H 'Cookie: session=...' https://example.com/api/me
curl -s -o /dev/null -D - https://example.com/assets/app.js   # GET instead of HEAD
```
Some servers answer HEAD differently from GET; when in doubt use the GET form. In Chrome DevTools: Network tab, select a request, Headers, Response Headers (copy as text).

## Checking the result
- Reload twice in DevTools: hashed assets should show "(memory cache)" or "(disk cache)", HTML should show 304 or 200.
- CDN: look at its `Age`, `X-Cache` or `CF-Cache-Status` header (HIT / MISS / BYPASS).
FILE:templates/cache-review-report.md
# HTTP Cache Review - <site / service> - <date>

**Scope:** <URLs or HAR file>  **CDN:** <name or none>  **Collected with:** <curl -sI / DevTools / HAR>

## Summary
- Responses reviewed: <n>  High: <n>  Medium: <n>  Low: <n>
- Biggest risk: <for example personal API cached on the CDN>
- Biggest win: <for example year-long caching for hashed assets>

## Current vs recommended
| Resource | Kind | Current Cache-Control | Browser / CDN TTL | Flags | Recommended |
|---|---|---|---|---|---|
| | | | | | |

## Fix list
| Priority | Change | Where (server, framework, CDN rule) | Owner | Done |
|---|---|---|---|---|
| 1 | | | | [ ] |

## Verification
- [ ] Re-run `python3 scripts/cache_audit.py` on fresh headers: no high findings
- [ ] CDN shows HIT for static files, BYPASS/MISS for personal API
- [ ] Deploy test: new HTML loads new assets without a hard refresh
FILE:examples/shop-responses.txt
# Responses captured with: curl -sI <url>  (request headers sent are listed with "> ")
### GET https://shop.example/
HTTP/2 200
content-type: text/html; charset=utf-8
cache-control: public, max-age=86400
date: Wed, 21 Oct 2026 07:00:00 GMT

### GET https://shop.example/assets/app.3f9a1c7b.js
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=3600
etag: "3f9a1c7b"

### GET https://shop.example/assets/vendor.81d2e0aa.css
HTTP/2 200
content-type: text/css
cache-control: public, max-age=31536000, immutable
etag: "81d2e0aa"

### GET https://shop.example/js/widget.js
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=31536000, immutable

### GET https://shop.example/img/hero.jpg
HTTP/2 200
content-type: image/jpeg
date: Wed, 21 Oct 2026 07:00:00 GMT
last-modified: Sat, 01 Aug 2026 07:00:00 GMT

### GET https://shop.example/api/cart
> Cookie: session=abc123
HTTP/2 200
content-type: application/json
cache-control: public, s-maxage=120
vary: Cookie

### POST https://shop.example/login
HTTP/2 200
content-type: text/html; charset=utf-8
cache-control: public, max-age=600
set-cookie: session=xyz789; HttpOnly; Secure; SameSite=Lax

### GET https://shop.example/api/products?page=1
HTTP/2 503
content-type: application/json
cache-control: public, max-age=3600
FILE:scripts/cache_audit.py
#!/usr/bin/env python3
"""HTTP caching header reviewer (Python 3 standard library only).

Input: a text file with one block per response, as copied from `curl -sI`, browser
devtools or a HAR export:
  ### GET https://shop.example/assets/app.3f9a1c7b.js
  > Authorization: Bearer ...          (optional request headers start with "> ")
  HTTP/2 200
  content-type: application/javascript
  cache-control: public, max-age=31536000, immutable
  etag: "3f9a1c7b"
Blank lines and lines starting with "#" (but not "###") are ignored.

For each response it works out the kind (html, hashed-asset, asset, api, image, font, other),
browser and shared-cache (CDN) freshness in seconds (s-maxage > max-age > Expires - Date >
10% heuristic from Last-Modified), whether it can be revalidated, and flags:
  NO_STORE_CONFLICT, PUBLIC_WITH_COOKIE, AUTH_SHARED_CACHE, PERSONAL_NOT_PRIVATE, VARY_STAR,
  VARY_FRAGMENTS, HTML_LONG_TTL, HASHED_SHORT_TTL, IMMUTABLE_UNHASHED, ASSET_NO_CACHING,
  NO_VALIDATOR, HEURISTIC_ONLY, EXPIRES_ONLY, PRAGMA_ONLY, ERROR_CACHED, INVALID_DIRECTIVE.
It suggests a Cache-Control value per kind.
Usage:
  python3 cache_audit.py responses.txt
  python3 cache_audit.py responses.txt --json
  python3 cache_audit.py --self-test
Exit code: 0 OK, 1 high-severity flags found, 2 bad input.
"""
import argparse
import email.utils
import json
import re
import sys

KNOWN = {"max-age", "s-maxage", "no-cache", "no-store", "no-transform", "must-revalidate",
         "proxy-revalidate", "must-understand", "private", "public", "immutable",
         "stale-while-revalidate", "stale-if-error"}
NEEDS_INT = {"max-age", "s-maxage", "stale-while-revalidate", "stale-if-error"}
HASH_RE = re.compile(r"[.\-_]([0-9a-f]{8,}|[0-9A-Za-z_-]{8,}(?=\.[a-z0-9]+$))\.[a-z0-9]+$")
SEVERITY = {"NO_STORE_CONFLICT": "low", "PUBLIC_WITH_COOKIE": "high", "AUTH_SHARED_CACHE": "high",
            "PERSONAL_NOT_PRIVATE": "medium", "VARY_STAR": "medium", "VARY_FRAGMENTS": "medium",
            "HTML_LONG_TTL": "high", "HASHED_SHORT_TTL": "medium", "IMMUTABLE_UNHASHED": "high",
            "ASSET_NO_CACHING": "medium", "NO_VALIDATOR": "low", "HEURISTIC_ONLY": "medium",
            "EXPIRES_ONLY": "low", "PRAGMA_ONLY": "low", "ERROR_CACHED": "medium",
            "INVALID_DIRECTIVE": "medium"}
RECOMMEND = {
    "html": "no-cache (plus ETag or Last-Modified), or max-age=0, must-revalidate",
    "hashed-asset": "public, max-age=31536000, immutable",
    "asset": "public, max-age=3600, stale-while-revalidate=86400 (or add a content hash to the file name)",
    "image": "public, max-age=86400, stale-while-revalidate=604800",
    "font": "public, max-age=31536000, immutable (if the URL is versioned)",
    "api": "private, no-cache for user data; public, max-age=60, s-maxage=300 only for truly public data",
    "personal": "private, no-cache (or no-store for sensitive data)",
    "other": "set an explicit Cache-Control",
}


class CacheError(ValueError):
    pass


def parse_cc(value):
    """Return (directives dict, problems list)."""
    d, problems = {}, []
    for part in [p.strip() for p in value.split(",") if p.strip()]:
        k, eq, v = part.partition("=")
        k = k.strip().lower()
        v = v.strip().strip('"')
        if k not in KNOWN:
            problems.append(f"unknown directive {k!r}")
            continue
        if k in NEEDS_INT:
            if not re.fullmatch(r"\d+", v or ""):
                problems.append(f"{k} needs a whole number of seconds, got {v!r}")
                continue
            d[k] = int(v)
        else:
            d[k] = True if not eq else v
    return d, problems


def parse_blocks(text):
    blocks, cur = [], None
    for raw in text.splitlines():
        line = raw.rstrip("\r")
        if line.startswith("###"):
            cur = {"label": line[3:].strip(), "status": None, "headers": {}, "request": {}}
            blocks.append(cur)
            continue
        if not line.strip() or line.lstrip().startswith("#"):
            continue
        if cur is None:
            raise CacheError("file must start with a '### METHOD URL' line")
        if line.startswith(">"):
            k, _, v = line[1:].strip().partition(":")
            cur["request"][k.strip().lower()] = v.strip()
            continue
        m = re.match(r"HTTP/[\d.]+\s+(\d{3})", line)
        if m:
            cur["status"] = int(m.group(1))
            continue
        k, colon, v = line.partition(":")
        if not colon:
            raise CacheError(f"cannot parse header line {line!r} in block {cur['label']!r}")
        k = k.strip().lower()
        if k in cur["headers"] and k in ("cache-control", "vary"):
            cur["headers"][k] += ", " + v.strip()
        elif k == "set-cookie":
            cur["headers"].setdefault(k, []).append(v.strip())
        else:
            cur["headers"][k] = v.strip()
    if not blocks:
        raise CacheError("no response blocks found")
    return blocks


def url_of(label):
    parts = label.split()
    return parts[-1] if parts else ""


def kind_of(url, ctype):
    path = re.sub(r"[?#].*$", "", url)
    name = path.rsplit("/", 1)[-1].lower()
    ctype = ctype.lower()
    hashed = bool(HASH_RE.search(name)) and bool(re.search(r"\d", name))
    if "text/html" in ctype or name.endswith(".html") or (not ctype and "." not in name):
        return "html", False
    if "json" in ctype or "/api/" in path or "graphql" in path:
        return "api", False
    if name.endswith((".woff2", ".woff", ".ttf", ".otf")) or "font" in ctype:
        return "font", hashed
    if ctype.startswith("image/") or name.endswith((".png", ".jpg", ".jpeg", ".gif", ".webp", ".avif", ".svg", ".ico")):
        return ("hashed-asset" if hashed else "image"), hashed
    if name.endswith((".js", ".mjs", ".css", ".map", ".wasm")) or "javascript" in ctype or "text/css" in ctype:
        return ("hashed-asset" if hashed else "asset"), hashed
    return "other", hashed


def http_date(v):
    try:
        return email.utils.parsedate_to_datetime(v)
    except (TypeError, ValueError, IndexError):
        return None


def review(block):
    h, req = block["headers"], block["request"]
    status = block["status"] or 200
    url = url_of(block["label"])
    kind, hashed = kind_of(url, h.get("content-type", ""))
    cc, problems = parse_cc(h.get("cache-control", ""))
    flags = []

    def flag(code, detail):
        flags.append({"code": code, "severity": SEVERITY[code], "detail": detail})

    for p in problems:
        flag("INVALID_DIRECTIVE", p)
    cookies = h.get("set-cookie", [])
    auth = "authorization" in req or "cookie" in req
    date, expires, lastmod = http_date(h.get("date")), h.get("expires"), http_date(h.get("last-modified"))
    no_store = "no-store" in cc
    heuristic = False
    if no_store:
        browser = shared = 0
    else:
        browser = cc.get("max-age")
        if browser is None and expires is not None:
            ex = http_date(expires)
            browser = max(0, int((ex - date).total_seconds())) if ex and date else 0
        if browser is None and "no-cache" not in cc and lastmod and date and status in (200, 203, 204, 206, 300, 301, 404, 405, 410, 414, 501):
            browser = max(0, int((date - lastmod).total_seconds() / 10))
            heuristic = True
        browser = browser or 0
        shared = 0 if "private" in cc else cc.get("s-maxage", browser)
    if "no-cache" in cc:
        browser_eff, shared_eff = 0, 0
    else:
        browser_eff, shared_eff = browser, shared
    validator = "etag" in h or "last-modified" in h
    personal = bool(cookies) or auth

    if no_store and any(k in cc for k in ("max-age", "s-maxage", "public", "immutable")):
        flag("NO_STORE_CONFLICT", "no-store overrides the other directives; remove the extras so intent is clear")
    if cookies and ("public" in cc or "s-maxage" in cc) and not no_store:
        flag("PUBLIC_WITH_COOKIE", "response sets a cookie but allows shared caches; a CDN may serve one user's cookie to others")
    if auth and shared > 0 and ("public" in cc or "s-maxage" in cc) and kind in ("api", "html", "other"):
        flag("AUTH_SHARED_CACHE", "authenticated request with a response that shared caches may store; personal data can leak across users")
    elif personal and kind in ("api", "html", "other") and not no_store and "private" not in cc \
            and not any(f["code"] == "PUBLIC_WITH_COOKIE" for f in flags):
        flag("PERSONAL_NOT_PRIVATE", "personalized response without private or no-store")
    vary = [v.strip().lower() for v in h.get("vary", "").split(",") if v.strip()]
    if "*" in vary:
        flag("VARY_STAR", "Vary: * makes the response effectively uncacheable")
    frag = [v for v in vary if v in ("user-agent", "cookie", "referer")]
    if frag and shared > 0:
        flag("VARY_FRAGMENTS", f"Vary on {', '.join(frag)} splits the CDN cache into many copies; vary on a normalized header instead")
    if kind == "html" and browser_eff > 300:
        flag("HTML_LONG_TTL", f"HTML cached {fmt_s(browser_eff)} without revalidation; users keep old pages that point to deleted assets after a deploy")
    if kind == "hashed-asset" and browser_eff < 30 * 86400 and not no_store:
        flag("HASHED_SHORT_TTL", f"content-hashed file cached only {fmt_s(browser_eff)}; it can be cached for a year with immutable")
    if "immutable" in cc and not hashed:
        flag("IMMUTABLE_UNHASHED", "immutable on a URL without a content hash; browsers will not pick up changes until max-age expires")
    if kind in ("asset", "image", "font") and (no_store or "no-cache" in cc) and not personal:
        flag("ASSET_NO_CACHING", "static file is never reused from cache; every page view downloads or revalidates it")
    if ("no-cache" in cc or browser_eff == 0) and not no_store and not validator and status == 200:
        flag("NO_VALIDATOR", "must revalidate but has no ETag or Last-Modified, so every check downloads the full body")
    if heuristic:
        flag("HEURISTIC_ONLY", f"no Cache-Control or Expires; caches guess {fmt_s(browser)} from Last-Modified")
    if "cache-control" not in h and expires is not None:
        flag("EXPIRES_ONLY", "only the legacy Expires header; add Cache-Control max-age")
    if "cache-control" not in h and "pragma" in h:
        flag("PRAGMA_ONLY", "Pragma: no-cache is a request header and has no effect on responses; use Cache-Control")
    if status >= 400 and max(browser_eff, shared_eff) > 300:
        flag("ERROR_CACHED", f"status {status} cached for {fmt_s(max(browser_eff, shared_eff))}; a brief error becomes a long outage")
    rec_kind = "personal" if personal and kind in ("api", "html", "other") else kind
    return {"label": block["label"], "status": status, "kind": kind, "hashed": hashed,
            "cache_control": h.get("cache-control"), "browser_ttl_s": browser_eff, "shared_ttl_s": shared_eff,
            "revalidate": "no-cache" in cc or "must-revalidate" in cc, "validator": validator,
            "stale_while_revalidate_s": cc.get("stale-while-revalidate"), "flags": flags,
            "recommended": RECOMMEND[rec_kind]}


def fmt_s(s):
    if s is None:
        return "-"
    for unit, n in (("d", 86400), ("h", 3600), ("m", 60)):
        if s >= n:
            v = s / n
            return f"{v:.0f}{unit}" if v == int(v) else f"{v:.1f}{unit}"
    return f"{s}s"


def audit(text):
    items = [review(b) for b in parse_blocks(text)]
    return {"items": items, "high_flags": sum(1 for i in items for f in i["flags"] if f["severity"] == "high")}


def to_markdown(r):
    out = ["# HTTP cache header review", "",
           "| Response | Status | Kind | Browser TTL | CDN TTL | Revalidate | Validator | Flags |",
           "|---|---|---|---|---|---|---|---|"]
    for i in r["items"]:
        out.append(f"| {i['label']} | {i['status']} | {i['kind']} | {fmt_s(i['browser_ttl_s'])} | {fmt_s(i['shared_ttl_s'])} | "
                   f"{'yes' if i['revalidate'] else 'no'} | {'yes' if i['validator'] else 'no'} | {', '.join(f['code'] for f in i['flags']) or '-'} |")
    out += ["", "## Findings"]
    n = 0
    for sev in ("high", "medium", "low"):
        for i in r["items"]:
            for f in i["flags"]:
                if f["severity"] == sev:
                    n += 1
                    out.append(f"- **{sev.upper()} {f['code']}** {i['label']}: {f['detail']}")
    if not n:
        out.append("- No findings.")
    out += ["", "## Suggested Cache-Control"]
    for i in r["items"]:
        if i["flags"]:
            out.append(f"- {i['label']}: `{i['recommended']}` (now: `{i['cache_control'] or 'none'}`)")
    return "\n".join(out)


def self_test():
    cases = []

    def ok(name, cond):
        cases.append((name, bool(cond)))

    def one(url, *lines):
        return review(parse_blocks("### GET " + url + "\nHTTP/1.1 200 OK\n" + "\n".join(lines))[0])

    def codes(item):
        return {f["code"] for f in item["flags"]}

    ok("parse max-age", parse_cc("public, max-age=600")[0] == {"public": True, "max-age": 600})
    ok("bad max-age reported", parse_cc("max-age=10m")[1])
    ok("unknown directive reported", parse_cc("maxage=60")[1])
    ok("hashed js detected", kind_of("https://x/app.3f9a1c7b.js", "") == ("hashed-asset", True))
    ok("unhashed css", kind_of("https://x/styles.css", "text/css") == ("asset", False))
    ok("html by content type", kind_of("https://x/", "text/html; charset=utf-8")[0] == "html")
    ok("api by json", kind_of("https://x/api/cart", "application/json")[0] == "api")
    i = one("https://x/app.3f9a1c7b.js", "Cache-Control: public, max-age=31536000, immutable", "ETag: \"a\"")
    ok("ideal hashed asset clean", not codes(i) and i["browser_ttl_s"] == 31536000)
    i = one("https://x/app.3f9a1c7b.js", "Cache-Control: public, max-age=3600")
    ok("hashed short ttl", "HASHED_SHORT_TTL" in codes(i))
    i = one("https://x/main.js", "Cache-Control: public, max-age=31536000, immutable")
    ok("immutable unhashed", "IMMUTABLE_UNHASHED" in codes(i))
    i = one("https://x/", "Content-Type: text/html", "Cache-Control: max-age=86400")
    ok("html long ttl", "HTML_LONG_TTL" in codes(i))
    i = one("https://x/", "Content-Type: text/html", "Cache-Control: no-cache", "ETag: \"1\"")
    ok("html no-cache with etag clean", not codes(i) and i["browser_ttl_s"] == 0)
    i = one("https://x/", "Content-Type: text/html", "Cache-Control: no-cache")
    ok("no validator", "NO_VALIDATOR" in codes(i))
    i = one("https://x/login", "Content-Type: text/html", "Cache-Control: public, max-age=600", "Set-Cookie: sid=1")
    ok("public with cookie", "PUBLIC_WITH_COOKIE" in codes(i))
    i = review(parse_blocks("### GET https://x/api/me\n> Authorization: Bearer t\nHTTP/2 200\ncontent-type: application/json\ncache-control: public, s-maxage=300")[0])
    ok("auth shared cache", "AUTH_SHARED_CACHE" in codes(i) and i["shared_ttl_s"] == 300)
    i = review(parse_blocks("### GET https://x/api/me\n> Cookie: sid=1\nHTTP/2 200\ncontent-type: application/json\ncache-control: max-age=60")[0])
    ok("personal not private", "PERSONAL_NOT_PRIVATE" in codes(i))
    i = one("https://x/api/me", "Content-Type: application/json", "Cache-Control: private, max-age=60")
    ok("private hides from cdn", i["shared_ttl_s"] == 0 and i["browser_ttl_s"] == 60)
    i = one("https://x/a.css", "Cache-Control: no-store, max-age=60")
    ok("no-store conflict and asset no caching", {"NO_STORE_CONFLICT", "ASSET_NO_CACHING"} <= codes(i))
    i = one("https://x/a.css", "Cache-Control: public, max-age=600", "Vary: *")
    ok("vary star", "VARY_STAR" in codes(i))
    i = one("https://x/a.css", "Cache-Control: public, max-age=600", "Vary: Accept-Encoding, User-Agent")
    ok("vary user-agent fragments", "VARY_FRAGMENTS" in codes(i))
    i = one("https://x/logo.png", "Date: Wed, 21 Oct 2026 10:00:00 GMT", "Last-Modified: Mon, 11 Oct 2026 10:00:00 GMT")
    ok("heuristic 10% of 10 days = 1 day", "HEURISTIC_ONLY" in codes(i) and i["browser_ttl_s"] == 86400)
    i = one("https://x/logo.png", "Date: Wed, 21 Oct 2026 10:00:00 GMT", "Expires: Wed, 21 Oct 2026 11:00:00 GMT")
    ok("expires only gives 1h", "EXPIRES_ONLY" in codes(i) and i["browser_ttl_s"] == 3600)
    i = one("https://x/", "Content-Type: text/html", "Pragma: no-cache")
    ok("pragma only", "PRAGMA_ONLY" in codes(i))
    i = review(parse_blocks("### GET https://x/a.css\nHTTP/1.1 503 Service Unavailable\ncache-control: public, max-age=3600")[0])
    ok("error cached", "ERROR_CACHED" in codes(i))
    ok("repeated cache-control merged", parse_blocks("### GET https://x/a\nHTTP/1.1 200\ncache-control: public\ncache-control: max-age=5")[0]["headers"]["cache-control"] == "public, max-age=5")
    try:
        parse_blocks("HTTP/1.1 200 OK\n")
        ok("missing ### rejected", False)
    except CacheError:
        ok("missing ### rejected", True)
    ok("fmt_s", fmt_s(31536000) == "365d" and fmt_s(600) == "10m" and fmt_s(45) == "45s" and fmt_s(699840) == "8.1d")
    passed = sum(c for _, c in cases)
    for name, c in cases:
        print(("PASS " if c else "FAIL ") + name)
    print(f"{passed}/{len(cases)} passed")
    return 0 if passed == len(cases) else 1


def main(argv=None):
    ap = argparse.ArgumentParser(description="Review HTTP caching headers.")
    ap.add_argument("file", nargs="?", help="responses text file ('-' for stdin)")
    ap.add_argument("--json", action="store_true")
    ap.add_argument("--self-test", action="store_true")
    a = ap.parse_args(argv)
    if a.self_test:
        return self_test()
    if not a.file:
        ap.error("responses file required (or --self-test)")
    try:
        text = sys.stdin.read() if a.file == "-" else open(a.file, encoding="utf-8").read()
        r = audit(text)
    except (OSError, CacheError) as e:
        print(f"error: {e}", file=sys.stderr)
        return 2
    print(json.dumps(r, indent=2) if a.json else to_markdown(r))
    return 1 if r["high_flags"] else 0


if __name__ == "__main__":
    sys.exit(main())

# Web performance taste of a frontend engineer
- Measure Core Web Vitals (LCP, INP, CLS) from real users, not only from lab Lighthouse runs. Confidence: 0.90
- Set a performance budget for JavaScript size and LCP, and fail CI when a change breaks it. Confidence: 0.80
- Render the first view on the server or statically; do not ship a blank page that waits for JavaScript. Confidence: 0.90
- Ship less JavaScript before optimizing the JavaScript you ship. Confidence: 0.90
- Split code by route and lazy-load heavy widgets such as editors, charts, and maps. Confidence: 0.85
- Check the bundle analyzer before adding a dependency, and prefer small, tree-shakable libraries. Confidence: 0.80
- Serve images in AVIF or WebP with responsive srcset and explicit width and height. Confidence: 0.90
- Mark the LCP image with fetchpriority="high" and never lazy-load it. Confidence: 0.85
- Lazy-load below-the-fold images and iframes with loading="lazy". Confidence: 0.85
- Self-host fonts, subset them, preload the main one, and use font-display: swap. Confidence: 0.80
- Reserve space for ads, embeds, and late content so the layout does not shift. Confidence: 0.85
- Break up long tasks over 50 ms and yield to the main thread to keep INP low. Confidence: 0.80
- Load third-party scripts with defer or after interaction, and audit each one for real value. Confidence: 0.85
- Cache hashed static assets with Cache-Control: immutable for a year. Confidence: 0.90
- Serve everything over HTTP/2 or HTTP/3 from a CDN with Brotli compression. Confidence: 0.85
- Preconnect only to the one or two origins needed for the first render. Confidence: 0.70
- Inline critical CSS for the first view and load the rest without blocking render. Confidence: 0.65
- Virtualize long lists instead of rendering thousands of DOM nodes. Confidence: 0.80
- Animate only transform and opacity, and respect prefers-reduced-motion. Confidence: 0.85
- Avoid layout thrashing: batch DOM reads before DOM writes. Confidence: 0.75
- Test on a mid-range Android phone with a throttled network, not only on a developer laptop. Confidence: 0.90
- Treat a performance regression like a bug, with an owner and a fix date. Confidence: 0.80

# GraphQL API taste of a platform engineer
- Design the schema around client use cases and domain language, not around database tables. Confidence: 0.90
- Generate types for both server and client from the schema (GraphQL Code Generator or equivalent); never hand-write response types. Confidence: 0.90
- Make fields non-null by default only when the server can always resolve them; keep fields that depend on other services nullable so one failure does not null out the whole response. Confidence: 0.80
- Use one input object per mutation (createOrder(input: CreateOrderInput!)) and return a payload type, not the bare entity. Confidence: 0.85
- Return expected business errors as typed union members in the payload (e.g. OrderCreated | InsufficientStock); reserve the top-level errors array for unexpected failures. Confidence: 0.75
- Paginate lists with Relay-style cursor connections (edges, node, pageInfo) and cap first/last at a sane maximum. Confidence: 0.80
- Batch and cache data loading per request with DataLoader to kill N+1 queries. Confidence: 0.95
- Enforce query depth and complexity limits at the gateway. Confidence: 0.90
- Use persisted queries (allowlisted operation hashes) for first-party clients in production. Confidence: 0.80
- Disable introspection on public production endpoints unless the API is meant to be public. Confidence: 0.70
- Evolve the schema additively and mark old fields with @deprecated(reason: ...); never version the endpoint with /v2. Confidence: 0.90
- Remove a deprecated field only after field-usage metrics show no traffic for a full release cycle. Confidence: 0.85
- Run schema diff checks in CI and fail the build on breaking changes. Confidence: 0.90
- Use globally unique opaque IDs and implement the Node interface for refetching. Confidence: 0.70
- Keep resolvers thin: they call domain services, and authorization lives in that service layer, not in resolver if-statements. Confidence: 0.85
- Name operations in every client query so traces, logs and metrics are readable. Confidence: 0.90
- Colocate fragments with the UI components that use them. Confidence: 0.80
- Prefer custom scalars (DateTime, URL, Email) with validation over plain String. Confidence: 0.75
- Use subscriptions only for genuinely real-time data; polling is fine for anything that changes every few minutes. Confidence: 0.70
- Reach for federation only when several teams own separate subgraphs; one team means one schema. Confidence: 0.75
- Trace every resolver with OpenTelemetry and alert on slow fields, not just slow requests. Confidence: 0.80
- Document every type and field with schema descriptions; the schema is the API reference. Confidence: 0.85

# Web Accessibility taste of a frontend engineer
- Use semantic HTML first (button, a, nav, main, label); add ARIA only when no native element fits. Confidence: 0.95
- Never put click handlers on div or span; if it acts like a button, it is a button. Confidence: 0.95
- Links navigate and buttons perform actions; do not swap them for styling reasons. Confidence: 0.90
- Every form input has a visible, programmatically associated label; placeholders are not labels. Confidence: 0.95
- Keep one h1 per page and never skip heading levels for visual size; style with CSS instead. Confidence: 0.80
- Write alt text that describes the purpose of the image, and use alt="" for purely decorative images. Confidence: 0.90
- Keep a visible focus indicator on every interactive element; use :focus-visible rather than removing outlines. Confidence: 0.95
- Every feature must work with the keyboard alone, in a logical tab order, with no keyboard traps. Confidence: 0.95
- Avoid positive tabindex values; fix the DOM order instead. Confidence: 0.90
- Meet WCAG AA contrast: 4.5:1 for body text and 3:1 for large text and UI components. Confidence: 0.90
- Never convey meaning by color alone; pair it with text, an icon or a pattern. Confidence: 0.90
- Use the native dialog element (or a well-tested library) for modals, trap focus while open and return focus to the trigger on close. Confidence: 0.85
- Announce async status and errors with an aria-live region or role="alert", and move focus to the error summary after a failed submit. Confidence: 0.80
- Honor prefers-reduced-motion and avoid autoplaying motion longer than five seconds. Confidence: 0.85
- Make touch targets at least 24x24 CSS pixels, ideally 44x44. Confidence: 0.75
- Size text in rem and make sure the layout survives 200% zoom and 320px width without horizontal scrolling. Confidence: 0.85
- Add a "Skip to main content" link as the first focusable element. Confidence: 0.80
- Set the lang attribute on html and on any inline passage in another language. Confidence: 0.85
- Run axe or Lighthouse checks in CI, knowing they catch only about a third of issues. Confidence: 0.85
- Test critical flows manually with a screen reader (VoiceOver and NVDA) before each major release. Confidence: 0.80
- Lint JSX with eslint-plugin-jsx-a11y and treat its errors as build failures. Confidence: 0.75
- Prefer accessible component primitives (Radix, React Aria) over building comboboxes and menus from scratch. Confidence: 0.80

---
name: robots-txt-and-sitemap-checker
description: Checks a site's robots.txt and XML sitemap offline the way Google reads them. Picks the crawler's group, resolves longest-match Allow and Disallow with wildcards, and tests the URLs that must be crawlable or blocked. Flags whole-site blocks, sitemap URLs that robots.txt blocks, blocked CSS and JS, unsupported noindex and crawl-delay lines, off-site, duplicate or invalid sitemap entries. Includes a stdlib Python checker with --self-test.
---

# Robots.txt and Sitemap Checker

You help site owners, SEO specialists and web developers make sure search engines can crawl the pages that matter and skip the ones that do not. From a robots.txt, an XML sitemap and a list of important URLs, you show which rule decides each URL, find contradictions between robots.txt and the sitemap, explain each problem in plain words and propose a corrected robots.txt and sitemap fixes.

## Files in this skill

- `scripts/crawl_check.py` - parses robots.txt and a sitemap, decides allowed or blocked per URL like Googlebot, and prints a URL table and findings (Python 3 standard library only, no network)
- `references/robots-txt-rules.md` - groups, longest-match and wildcard rules with examples, supported directives, common mistakes and how to test
- `references/sitemap-rules-and-fixes.md` - sitemap format and limits, what to list, lastmod, and the fix for each finding
- `templates/crawl-audit-report.md` - an audit report with a proposed robots.txt to hand to the developer
- `examples/outdoor-shop-crawl-check.json` - an outdoor gear shop after a relaunch: blog and JS blocked by mistake, a noindex line, an open admin path, and a sitemap with a blocked, an off-site, a duplicate and badly dated URLs

## Workflow

### 1. Collect the files
Ask for the live robots.txt (`https://<host>/robots.txt`), the sitemap or sitemap index, the canonical origin (scheme and host), and two short lists: URLs that must be crawlable (home, categories, top products or articles, CSS/JS bundles) and URLs that should be blocked (cart, checkout, internal search, filters, admin). Ask which crawler matters most (default Googlebot).

### 2. Run the checker
```
python3 scripts/crawl_check.py examples/outdoor-shop-crawl-check.json --today 2026-10-23
python3 scripts/crawl_check.py --robots robots.txt --sitemap sitemap.xml --site https://www.example.com --allow /product/x --block /cart/
python3 scripts/crawl_check.py --self-test
```
A bundle JSON holds the site, crawler, robots.txt and sitemap lines and the URL lists. Exit code 1 means CRITICAL or HIGH findings, 2 means bad input.

### 3. Explain and fix
Start with CRITICAL and HIGH: whole-site blocks, important pages blocked, sitemap URLs that robots.txt blocks, off-site sitemap entries. For each, name the deciding line and give the corrected rule. Explain that robots.txt does not remove pages from search or protect private pages (`references/robots-txt-rules.md`), and point to the fix table in `references/sitemap-rules-and-fixes.md`.

### 4. Deliver
Fill `templates/crawl-audit-report.md` with the URL table, findings, a full corrected robots.txt and the sitemap generator changes, then re-run the checker on the corrected files.

## Example result
For `examples/outdoor-shop-crawl-check.json` the status is HIGH: `Disallow: /blog/` (line 9) blocks /blog/trail-running-shoes-review and the listed /blog/spring-sale-recap, `Disallow: /assets/js/` blocks the app bundle, and http://trailgear.example/backpacks/ in the sitemap is off-site. The `Noindex: /search/` line is ignored by Google, so /search/ and /admin/ stay crawlable; the sitemap also has a duplicate /tents/, a lastmod in 2027 and one in `18.10.2026` format.

## Rules

- Always name the exact robots.txt line that decides a URL, and the crawler it was evaluated for.
- Never suggest robots.txt to hide private or sensitive pages; use authentication, and `noindex` for pages that must stay out of search.
- Give a complete corrected robots.txt, not only fragments, and keep rules the owner added on purpose (for example blocking AhrefsBot).
- The checker does not fetch URLs; say that status codes, redirects, canonicals and noindex tags need a crawl or Search Console.
FILE:references/robots-txt-rules.md
# robots.txt Rules (RFC 9309 and Google)

## Where and what
- Lives at the root of each host and protocol: `https://www.example.com/robots.txt` covers only that host. `example.com`, `shop.example.com` and `http://` need their own file.
- Plain text, UTF-8. Google reads the first 500 KiB. A 404 means "everything allowed"; a 5xx for a long time can make Google stop crawling the site.
- robots.txt controls **crawling**, not indexing. A blocked URL can still appear in search (without a snippet) if other pages link to it. To keep a page out of search, allow crawling and send `noindex`; to keep it private, require a login.

## Groups
```
User-agent: Googlebot
User-agent: Bingbot
Disallow: /tmp/

User-agent: *
Disallow: /cart/
```
- Consecutive `User-agent` lines share one group. Rules belong to the group above them.
- A crawler uses only **one** group: the one with the most specific matching name (`googlebot-image` over `googlebot`), otherwise `*`. Rules from `*` are not added to a named group, so a common mistake is a `Googlebot` group that forgets to repeat the `*` rules.
- Groups with the same name are merged.

## Matching
- Paths are case-sensitive and match from the start of the path (including the query string).
- `*` matches any characters; `$` anchors the end of the URL.
- The **longest** matching rule wins; on equal length `Allow` wins. No matching rule means allowed.
- `Disallow:` with an empty value allows everything.

| Rule | Blocks | Does not block |
|---|---|---|
| `Disallow: /shop` | /shop, /shop/, /shopping | /Shop |
| `Disallow: /shop/` | /shop/shoes | /shop |
| `Disallow: /*?sort=` | /tents/?sort=price | /tents/?color=red&sort=price (needs `/*sort=`) |
| `Disallow: /*.pdf$` | /docs/a.pdf | /docs/a.pdf?v=2 |
| `Disallow: /blog/` + `Allow: /blog/guides/` | /blog/news | /blog/guides/x |

## Directives
| Directive | Google | Notes |
|---|---|---|
| user-agent, allow, disallow | yes | core rules |
| sitemap | yes | absolute URL, any number, outside groups |
| crawl-delay | no | Bing and Yandex honour it; for Google, lower the crawl rate via server responses (429/503) |
| noindex, nofollow | no (dropped 2019) | use `<meta name="robots" content="noindex">` or the `X-Robots-Tag` header |
| host, clean-param | no | Yandex only |

## Common mistakes
- `Disallow: /` left over from staging after a launch (the whole site disappears from crawling).
- Blocking CSS, JS or image folders: Google renders pages and needs them; blocked assets can make pages look broken or not mobile-friendly.
- Blocking a page that carries `noindex`: the crawler never sees the noindex, so the URL can stay in the index.
- Using robots.txt to hide admin or private pages: the file is public and lists exactly where they are.
- Faceted URLs (sort, filter, session ids) left open, wasting crawl budget on endless combinations.
- Typos (`Disalow`) are silently ignored.

## Testing
- Google Search Console: robots.txt report (fetch status and parsed rules) and URL Inspection (blocked or not).
- Locally: `python3 scripts/crawl_check.py --robots robots.txt --site https://www.example.com --allow /product/x --block /cart/`.
FILE:references/sitemap-rules-and-fixes.md
# XML Sitemap Rules and Fixes

## Format
```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/tents/ultralight-2p</loc>
    <lastmod>2026-10-18T09:30:00+03:00</lastmod>
  </url>
</urlset>
```
- At most **50,000 URLs** and **50 MB** uncompressed per file. Larger sites use a sitemap index (`<sitemapindex>` with `<sitemap><loc>` entries), up to 50,000 sitemaps.
- UTF-8, entity-escape `&`, `<`, `>`, `'`, `"` inside `<loc>`.
- `<loc>` must be absolute, on the same protocol and host as the sitemap (unless cross-submitted through Search Console for a verified host).

## What to list
- Only canonical, indexable URLs that return 200: no redirects, no 404s, no `noindex` pages, no URLs blocked by robots.txt, no parameter duplicates.
- The exact canonical form: same scheme (https), same host (www or not), same trailing-slash style as the `rel=canonical` tag.
- Each URL once.

## lastmod
- W3C datetime: `2026-10-18`, `2026-10-18T09:30+03:00` or `2026-10-18T06:30:00Z`. Formats like `18.10.2026` or `10/18/2026` are invalid.
- Set it to the last **meaningful** content change, not the build time. Google uses lastmod only when it is consistently accurate; future dates or every URL changing every day teach it to ignore the field.
- `<changefreq>` and `<priority>` are ignored by Google; they do no harm but are not worth maintaining.

## Telling crawlers about the sitemap
- `Sitemap: https://www.example.com/sitemap.xml` in robots.txt (absolute URL).
- Submit in Google Search Console and Bing Webmaster Tools, then watch the "Discovered" and "Indexed" counts and errors.

## Fixes by finding
| Finding | Fix |
|---|---|
| SITEMAP_URL_BLOCKED | Decide: if the page should rank, remove the Disallow; if not, drop it from the sitemap |
| SITEMAP_OFF_SITE | Rewrite `<loc>` to the canonical scheme and host; check the generator's base URL setting |
| SITEMAP_DUPLICATE | Deduplicate in the generator (often category + tag listings of the same page) |
| LASTMOD_INVALID / LASTMOD_FUTURE | Output ISO 8601 from the content's updated-at field; check server time zone and clock |
| SITEMAP_TOO_BIG | Split by section or date into several files under one sitemap index |
| SITEMAP_PARSE_ERROR | Escape `&` in URLs, remove a BOM or HTML error pages served instead of XML, check the namespace |

## Beyond this checker
The script works offline on the files you give it. Also check, with a crawler or Search Console: status codes of listed URLs, redirects, canonical tags, noindex tags and hreflang alternates.
FILE:templates/crawl-audit-report.md
# Crawl Audit - <site> - <date>

**Checked:** robots.txt (<fetched on / version>), <sitemap URL(s)>  **Crawler:** <Googlebot>  **Reviewer:** <name>

## Summary
- Status: <CRITICAL / HIGH / OK>
- robots.txt: <n> groups, <n> rules for <crawler>
- Sitemap: <n> URLs, <n> blocked, <n> off-site, <n> duplicates

## Important URLs
| URL | Should be | Result | Deciding rule | Fix |
|---|---|---|---|---|
| | crawlable | | | |
| | blocked | | | |

## Findings and fixes
| Severity | Finding | Detail | Fix | Owner |
|---|---|---|---|---|
| | | | | |

## Proposed robots.txt
```
User-agent: *
Disallow: ...
Allow: ...

Sitemap: https://...
```

## Follow-up
- Deploy the new robots.txt and sitemap on <date>; test in Search Console (robots.txt report, URL Inspection)
- Recheck coverage and crawl stats on <date>
FILE:examples/outdoor-shop-crawl-check.json
{
  "site": "https://www.trailgear.example",
  "user_agent": "Googlebot",
  "robots_txt": [
    "# robots.txt for www.trailgear.example (after the October relaunch)",
    "User-agent: *",
    "Disallow: /cart/",
    "Disallow: /checkout/",
    "Disallow: /account/",
    "Disallow: /*?sort=",
    "Disallow: /*?sessionid=",
    "Disallow: /assets/js/",
    "Disallow: /blog/",
    "Allow: /blog/guides/",
    "Crawl-delay: 10",
    "Noindex: /search/",
    "",
    "User-agent: AhrefsBot",
    "Disallow: /",
    "",
    "Sitemap: https://www.trailgear.example/sitemap.xml"
  ],
  "sitemap_xml": [
    "<?xml version=\"1.0\" encoding=\"UTF-8\"?>",
    "<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">",
    "  <url><loc>https://www.trailgear.example/</loc><lastmod>2026-10-20</lastmod></url>",
    "  <url><loc>https://www.trailgear.example/tents/</loc><lastmod>2026-10-18</lastmod></url>",
    "  <url><loc>https://www.trailgear.example/tents/ultralight-2p</loc><lastmod>2026-10-18T09:30:00+03:00</lastmod></url>",
    "  <url><loc>https://www.trailgear.example/tents/</loc><lastmod>2026-10-18</lastmod></url>",
    "  <url><loc>https://www.trailgear.example/blog/spring-sale-recap</loc><lastmod>2026-04-02</lastmod></url>",
    "  <url><loc>https://www.trailgear.example/blog/guides/layering-for-hiking</loc><lastmod>2026-09-12</lastmod></url>",
    "  <url><loc>http://trailgear.example/backpacks/</loc><lastmod>2026-10-01</lastmod></url>",
    "  <url><loc>https://www.trailgear.example/sleeping-bags/</loc><lastmod>18.10.2026</lastmod></url>",
    "  <url><loc>https://www.trailgear.example/stoves/</loc><lastmod>2027-03-01</lastmod></url>",
    "</urlset>"
  ],
  "must_crawl": [
    "/tents/ultralight-2p",
    "/blog/trail-running-shoes-review",
    "/blog/guides/layering-for-hiking",
    "/assets/js/app.js",
    "/tents/?color=green"
  ],
  "must_block": [
    "/checkout/payment",
    "/tents/?sort=price",
    "/search/?q=stove",
    "/admin/"
  ]
}
FILE:scripts/crawl_check.py
#!/usr/bin/env python3
"""robots.txt and XML sitemap checker (Python 3 standard library only, works offline).

Input, either:
  a JSON bundle  {"site": "https://www.example.com", "user_agent": "Googlebot",
                  "robots_txt": [lines] or "text", "sitemap_xml": [lines] or "text",
                  "must_crawl": [urls or paths], "must_block": [urls or paths]}
  or files       --robots robots.txt [--sitemap sitemap.xml] --site https://... [--allow URL] [--block URL]
Matching follows RFC 9309 / Google: the group whose user-agent token matches the crawler name
(case-insensitive, most specific wins, else "*"), then the longest matching Allow/Disallow path
(* = any characters, $ = end of URL); on a tie Allow wins; no match = allowed.
Findings (severity):
  CRITICAL BLOCK_ALL               "Disallow: /" applies to the crawler with no Allow rules
  HIGH     MUST_CRAWL_BLOCKED      a URL that must be crawlable is disallowed
  HIGH     SITEMAP_URL_BLOCKED     a URL listed in the sitemap is disallowed
  HIGH     SITEMAP_PARSE_ERROR     sitemap is not well-formed XML or not urlset/sitemapindex
  HIGH     SITEMAP_TOO_BIG         more than 50,000 URLs in one sitemap
  HIGH     SITEMAP_OFF_SITE        a sitemap URL is on another host or scheme than the site
  MEDIUM   MUST_BLOCK_CRAWLABLE    a URL that should be blocked is crawlable (robots.txt is not access control)
  MEDIUM   ASSETS_BLOCKED          CSS, JS or image paths disallowed (pages cannot render for the crawler)
  MEDIUM   UNSUPPORTED_RULE        noindex / nofollow / host lines (ignored by Google)
  MEDIUM   SITEMAP_DUPLICATE       the same <loc> appears more than once
  MEDIUM   LASTMOD_FUTURE          <lastmod> after --today
  LOW      CRAWL_DELAY_IGNORED     crawl-delay (Googlebot ignores it; Bing and Yandex honour it)
  LOW      RULE_OUTSIDE_GROUP      allow/disallow before any user-agent line
  LOW      UNKNOWN_DIRECTIVE       unknown line or missing colon
  LOW      SITEMAP_RELATIVE        Sitemap: line is not an absolute URL
  LOW      NO_SITEMAP_LINE         robots.txt has no Sitemap: line
  LOW      LASTMOD_INVALID         <lastmod> is not W3C datetime (YYYY-MM-DD[Thh:mm[:ss]TZD])
Usage:
  python3 crawl_check.py bundle.json [--today 2026-10-23] [--json]
  python3 crawl_check.py --robots robots.txt --sitemap sitemap.xml --site https://www.example.com --allow /shop/
  python3 crawl_check.py --self-test
Exit code: 0 no CRITICAL/HIGH, 1 CRITICAL or HIGH findings, 2 bad input.
"""
import argparse
import datetime as dt
import json
import re
import sys
import urllib.parse
import xml.etree.ElementTree as ET

SEV_ORDER = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3}
KNOWN = {"user-agent", "allow", "disallow", "sitemap", "crawl-delay", "noindex", "nofollow", "host", "clean-param"}
ASSET_RE = re.compile(r"(\.css|\.js|\.png|\.jpe?g|\.webp|\.svg|\.gif|/static/|/assets/|/_next/|/wp-includes/|/css/|/js/)", re.I)
LASTMOD_RE = re.compile(r"^\d{4}(-\d{2}(-\d{2}(T\d{2}:\d{2}(:\d{2}(\.\d+)?)?(Z|[+-]\d{2}:\d{2}))?)?)?$")
NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"


class DataError(ValueError):
    pass


def text_of(v):
    if v is None:
        return None
    return "\n".join(v) if isinstance(v, list) else str(v)


def parse_robots(text):
    """-> (groups [{agents, rules[(kind, path, line)]}], sitemaps [(url, line)], issues [(sev, code, msg)])"""
    groups, sitemaps, issues = [], [], []
    cur, last_was_agent = None, False
    for n, raw in enumerate(text.splitlines(), start=1):
        line = raw.split("#", 1)[0].strip()
        if not line:
            continue
        if ":" not in line:
            issues.append(("LOW", "UNKNOWN_DIRECTIVE", f"line {n}: no colon in {line!r}"))
            continue
        key, val = (x.strip() for x in line.split(":", 1))
        key = key.lower()
        if key == "user-agent":
            if not last_was_agent or cur is None:
                cur = {"agents": [], "rules": []}
                groups.append(cur)
            cur["agents"].append(val.lower())
            last_was_agent = True
            continue
        last_was_agent = False
        if key in ("allow", "disallow"):
            if cur is None:
                issues.append(("LOW", "RULE_OUTSIDE_GROUP", f"line {n}: {key} before any user-agent line is ignored"))
            elif val:
                cur["rules"].append((key, val, n))
        elif key == "sitemap":
            sitemaps.append((val, n))
            if not re.match(r"^https?://", val, re.I):
                issues.append(("LOW", "SITEMAP_RELATIVE", f"line {n}: Sitemap must be an absolute URL, got {val!r}"))
        elif key == "crawl-delay":
            issues.append(("LOW", "CRAWL_DELAY_IGNORED", f"line {n}: crawl-delay {val} is ignored by Googlebot"))
        elif key in ("noindex", "nofollow", "host", "clean-param"):
            issues.append(("MEDIUM", "UNSUPPORTED_RULE", f"line {n}: '{key}' in robots.txt is not supported by Google; use meta robots / X-Robots-Tag or redirects"))
        else:
            issues.append(("LOW", "UNKNOWN_DIRECTIVE", f"line {n}: unknown directive {key!r}"))
    return groups, sitemaps, issues


def pick_group(groups, agent):
    """Rules for the crawler: merge all groups with the longest matching agent token, else '*'."""
    a = agent.lower()
    best, rules = -1, []
    for g in groups:
        for tok in g["agents"]:
            if tok != "*" and tok and a.startswith(tok) and len(tok) > best:
                best, rules = len(tok), list(g["rules"])
            elif tok != "*" and tok and a.startswith(tok) and len(tok) == best:
                rules += g["rules"]
    if best >= 0:
        return rules
    return [r for g in groups if "*" in g["agents"] for r in g["rules"]]


def pattern_to_re(path):
    end = path.endswith("$")
    body = path[:-1] if end else path
    return re.compile("^" + ".*".join(re.escape(p) for p in body.split("*")) + ("$" if end else ""))


def path_of(url, site):
    u = urllib.parse.urlsplit(urllib.parse.urljoin(site + "/", url))
    return (u.path or "/") + (("?" + u.query) if u.query else "")


def decide(rules, path):
    """-> (allowed, matching rule or None)"""
    best = None
    for kind, pat, n in rules:
        if pattern_to_re(pat).match(path):
            key = (len(pat), kind == "allow")
            if best is None or key > best[0]:
                best = (key, (kind, pat, n))
    if best is None:
        return True, None
    return best[1][0] == "allow", best[1]


def parse_sitemap(text):
    try:
        root = ET.fromstring(text.strip())
    except ET.ParseError as e:
        raise DataError(f"sitemap XML error: {e}")
    tag = root.tag.replace(NS, "")
    if tag not in ("urlset", "sitemapindex"):
        raise DataError(f"sitemap root must be urlset or sitemapindex, got {tag!r}")
    child = "url" if tag == "urlset" else "sitemap"
    items = []
    for el in root.findall(NS + child) or root.findall(child):
        loc = el.findtext(NS + "loc") or el.findtext("loc") or ""
        lm = el.findtext(NS + "lastmod") or el.findtext("lastmod")
        items.append((loc.strip(), lm.strip() if lm else None))
    return tag, items


def check(bundle, today=None):
    site = (bundle.get("site") or "").rstrip("/")
    if not re.match(r"^https?://[^/]+$", site):
        raise DataError("site must be an origin like https://www.example.com")
    robots = text_of(bundle.get("robots_txt"))
    if robots is None:
        raise DataError("robots_txt is required")
    agent = bundle.get("user_agent") or "Googlebot"
    today = today or dt.date.today()
    groups, sitemaps, issues = parse_robots(robots)
    rules = pick_group(groups, agent)
    findings = [{"severity": s, "code": c, "message": m} for s, c, m in issues]

    def f(sev, code, msg):
        findings.append({"severity": sev, "code": code, "message": msg})

    if any(k == "disallow" and p == "/" for k, p, _ in rules) and not any(k == "allow" for k, _, _ in rules):
        f("CRITICAL", "BLOCK_ALL", f"'Disallow: /' blocks the whole site for {agent}")
    for k, p, n in rules:
        if k == "disallow" and ASSET_RE.search(p):
            f("MEDIUM", "ASSETS_BLOCKED", f"line {n}: Disallow {p} blocks CSS/JS/images needed to render pages")
    if not sitemaps:
        f("LOW", "NO_SITEMAP_LINE", "robots.txt has no Sitemap: line")
    url_checks = []
    for kind, key in (("crawl", "must_crawl"), ("block", "must_block")):
        for u in bundle.get(key) or []:
            p = path_of(u, site)
            allowed, rule = decide(rules, p)
            url_checks.append({"url": u, "path": p, "expect": kind, "allowed": allowed,
                               "rule": f"line {rule[2]}: {rule[0].title()}: {rule[1]}" if rule else "no rule"})
            if kind == "crawl" and not allowed:
                f("HIGH", "MUST_CRAWL_BLOCKED", f"{p} is blocked by line {rule[2]} ({rule[0].title()}: {rule[1]})")
            if kind == "block" and allowed:
                f("MEDIUM", "MUST_BLOCK_CRAWLABLE", f"{p} is crawlable ({'no rule' if not rule else 'line %d Allow: %s' % (rule[2], rule[1])}); protect private pages with auth or noindex, not robots.txt")
    sm_info = None
    sm_text = text_of(bundle.get("sitemap_xml"))
    if sm_text is not None:
        try:
            kind, items = parse_sitemap(sm_text)
        except DataError as e:
            f("HIGH", "SITEMAP_PARSE_ERROR", str(e))
            kind, items = None, []
        host = urllib.parse.urlsplit(site)
        seen, blocked, off = set(), 0, 0
        for loc, lm in items:
            u = urllib.parse.urlsplit(loc)
            if (u.scheme, u.netloc) != (host.scheme, host.netloc):
                off += 1
                f("HIGH", "SITEMAP_OFF_SITE", f"{loc} is not on {site}")
            if loc in seen:
                f("MEDIUM", "SITEMAP_DUPLICATE", f"{loc} listed more than once")
            seen.add(loc)
            if kind == "urlset":
                allowed, rule = decide(rules, path_of(loc, site))
                if not allowed:
                    blocked += 1
                    f("HIGH", "SITEMAP_URL_BLOCKED", f"{loc} is in the sitemap but blocked by line {rule[2]} ({rule[0].title()}: {rule[1]})")
            if lm is not None:
                if not LASTMOD_RE.match(lm):
                    f("LOW", "LASTMOD_INVALID", f"{loc}: lastmod {lm!r} is not W3C datetime")
                elif dt.date.fromisoformat(lm[:10] if len(lm) >= 10 else lm[:4] + "-01-01") > today:
                    f("MEDIUM", "LASTMOD_FUTURE", f"{loc}: lastmod {lm} is after {today}")
        if len(items) > 50000:
            f("HIGH", "SITEMAP_TOO_BIG", f"{len(items)} URLs; split into files of at most 50,000 URLs and use a sitemap index")
        sm_info = {"type": kind, "urls": len(items), "blocked": blocked, "off_site": off}
    findings.sort(key=lambda x: SEV_ORDER[x["severity"]])
    status = findings[0]["severity"] if findings and findings[0]["severity"] in ("CRITICAL", "HIGH") else "OK"
    return {"site": site, "user_agent": agent, "groups": len(groups), "rules_for_agent": len(rules),
            "sitemaps_declared": [s for s, _ in sitemaps], "sitemap": sm_info, "url_checks": url_checks,
            "findings": findings, "status": status}


def report(r):
    L = [f"# Crawl check for {r['site']} as {r['user_agent']}", "",
         f"robots.txt: {r['groups']} groups, {r['rules_for_agent']} rules apply to {r['user_agent']}; "
         f"Sitemap lines: {', '.join(r['sitemaps_declared']) or 'none'}"]
    if r["sitemap"]:
        s = r["sitemap"]
        L.append(f"Sitemap ({s['type']}): {s['urls']} URLs, {s['blocked']} blocked by robots.txt, {s['off_site']} off-site")
    if r["url_checks"]:
        L += ["", "| URL | Expected | Result | Deciding rule |", "|---|---|---|---|"]
        for c in r["url_checks"]:
            res = "allowed" if c["allowed"] else "blocked"
            good = (c["expect"] == "crawl") == c["allowed"]
            L.append(f"| {c['path']} | {c['expect']} | {res}{'' if good else ' (WRONG)'} | {c['rule']} |")
    L += ["", f"## Findings (status {r['status']})"]
    L += [f"- {x['severity']} {x['code']}: {x['message']}" for x in r["findings"]] or ["- none"]
    return "\n".join(L) + "\n"


def self_test():
    cases = []

    def ok(name, cond):
        cases.append(bool(cond))
        print(("PASS " if cond else "FAIL ") + name)

    def codes(r):
        return {x["code"] for x in r["findings"]}

    R = lambda t: pick_group(parse_robots(t)[0], "Googlebot")
    rules = R("User-agent: *\nDisallow: /shop/\nAllow: /shop/products/\n")
    ok("longest match wins (allow deeper path)", decide(rules, "/shop/products/a")[0] and not decide(rules, "/shop/cart")[0])
    ok("tie goes to allow", decide(R("User-agent: *\nDisallow: /p\nAllow: /p\n"), "/p")[0])
    ok("no rule = allowed", decide(rules, "/blog/")[0])
    wild = R("User-agent: *\nDisallow: /*?sort=\nDisallow: /*.pdf$\n")
    ok("* wildcard matches query", not decide(wild, "/shop/shoes?sort=price")[0])
    ok("$ anchors end", not decide(wild, "/files/a.pdf")[0] and decide(wild, "/files/a.pdf?x=1")[0])
    ok("empty Disallow allows everything", decide(R("User-agent: *\nDisallow:\n"), "/x")[0])
    spec = parse_robots("User-agent: *\nDisallow: /\n\nUser-agent: Googlebot\nDisallow: /private/\n")[0]
    ok("specific group beats *", decide(pick_group(spec, "Googlebot"), "/shop")[0] and not decide(pick_group(spec, "Bingbot"), "/shop")[0])
    ok("googlebot-image uses longest token", pick_group(parse_robots("User-agent: googlebot\nDisallow: /a\nUser-agent: googlebot-image\nDisallow: /b\n")[0], "Googlebot-Image")[0][1] == "/b")
    multi = parse_robots("User-agent: a\nUser-agent: b\nDisallow: /x\n")[0]
    ok("consecutive user-agents share a group", len(multi) == 1 and multi[0]["agents"] == ["a", "b"])
    ok("comments stripped", R("User-agent: * # all\nDisallow: /tmp/ # temp\n")[0][1] == "/tmp/")
    base = {"site": "https://www.shop.test", "robots_txt": "User-agent: *\nDisallow: /\n"}
    r = check(base, dt.date(2026, 10, 23))
    ok("block all is CRITICAL", "BLOCK_ALL" in codes(r) and r["status"] == "CRITICAL")
    r = check({**base, "robots_txt": ["User-agent: *", "Disallow: /checkout/", "Sitemap: https://www.shop.test/sitemap.xml"],
               "must_crawl": ["/checkout/thanks"], "must_block": ["/admin/"]}, dt.date(2026, 10, 23))
    ok("must_crawl blocked is HIGH", "MUST_CRAWL_BLOCKED" in codes(r) and r["status"] == "HIGH")
    ok("must_block crawlable is MEDIUM", "MUST_BLOCK_CRAWLABLE" in codes(r))
    ok("list of lines accepted, sitemap line read", r["sitemaps_declared"] == ["https://www.shop.test/sitemap.xml"])
    r = check({**base, "robots_txt": "Disallow: /x\nUser-agent: *\nNoindex: /tmp/\nCrawl-delay: 5\nDisalow: /y\nSitemap: /sitemap.xml\nDisallow: /assets/\n"}, dt.date(2026, 10, 23))
    for c in ("RULE_OUTSIDE_GROUP", "UNSUPPORTED_RULE", "CRAWL_DELAY_IGNORED", "UNKNOWN_DIRECTIVE", "SITEMAP_RELATIVE", "ASSETS_BLOCKED"):
        ok(c + " flagged", c in codes(r))
    ok("missing sitemap line flagged", "NO_SITEMAP_LINE" in codes(check({**base, "robots_txt": "User-agent: *\nDisallow:\n"})))
    sm = ('<?xml version="1.0"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
          "<url><loc>https://www.shop.test/a</loc><lastmod>2026-10-01</lastmod></url>"
          "<url><loc>https://www.shop.test/a</loc></url>"
          "<url><loc>http://www.shop.test/b</loc><lastmod>01/10/2026</lastmod></url>"
          "<url><loc>https://www.shop.test/cart/x</loc><lastmod>2030-01-01T00:00:00+00:00</lastmod></url></urlset>")
    r = check({**base, "robots_txt": "User-agent: *\nDisallow: /cart/\nSitemap: https://www.shop.test/sitemap.xml\n", "sitemap_xml": sm}, dt.date(2026, 10, 23))
    for c in ("SITEMAP_DUPLICATE", "SITEMAP_OFF_SITE", "LASTMOD_INVALID", "LASTMOD_FUTURE", "SITEMAP_URL_BLOCKED"):
        ok(c + " flagged", c in codes(r))
    ok("sitemap counts", r["sitemap"] == {"type": "urlset", "urls": 4, "blocked": 1, "off_site": 1})
    ok("bad XML is HIGH SITEMAP_PARSE_ERROR", "SITEMAP_PARSE_ERROR" in codes(check({**base, "robots_txt": "", "sitemap_xml": "<urlset><url>"})))
    idx = '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>https://www.shop.test/s1.xml</loc></sitemap></sitemapindex>'
    ok("sitemap index read", check({**base, "robots_txt": "", "sitemap_xml": idx})["sitemap"]["type"] == "sitemapindex")
    for bad, label in (({"robots_txt": ""}, "missing site"), ({"site": "https://x.test/path", "robots_txt": ""}, "site with path"),
                       ({"site": "https://x.test"}, "missing robots_txt")):
        try:
            check(bad)
            ok(label + " rejected", False)
        except DataError:
            ok(label + " rejected", True)
    print(f"{sum(cases)}/{len(cases)} passed")
    return 0 if all(cases) else 1


def main(argv=None):
    ap = argparse.ArgumentParser(description="Check robots.txt rules and an XML sitemap offline.")
    ap.add_argument("bundle", nargs="?", help="JSON bundle (see module docstring)")
    ap.add_argument("--robots", help="robots.txt file (instead of a bundle)")
    ap.add_argument("--sitemap", help="sitemap.xml file")
    ap.add_argument("--site", help="site origin, e.g. https://www.example.com")
    ap.add_argument("--agent", help="crawler name (default Googlebot)")
    ap.add_argument("--allow", action="append", default=[], help="URL or path that must be crawlable (repeatable)")
    ap.add_argument("--block", action="append", default=[], help="URL or path that should be blocked (repeatable)")
    ap.add_argument("--today", help="date for the future-lastmod check (default today)")
    ap.add_argument("--json", action="store_true")
    ap.add_argument("--self-test", action="store_true")
    a = ap.parse_args(argv)
    if a.self_test:
        return self_test()
    try:
        if a.bundle:
            b = json.load(open(a.bundle, encoding="utf-8"))
        elif a.robots:
            b = {"site": a.site, "robots_txt": open(a.robots, encoding="utf-8").read(),
                 "sitemap_xml": open(a.sitemap, encoding="utf-8").read() if a.sitemap else None,
                 "must_crawl": a.allow, "must_block": a.block}
        else:
            ap.error("give a JSON bundle, --robots FILE or --self-test")
        if a.agent:
            b["user_agent"] = a.agent
        today = dt.date.fromisoformat(a.today) if a.today else None
        r = check(b, today)
    except (OSError, ValueError, DataError) as e:
        print(f"error: {e}", file=sys.stderr)
        return 2
    print(json.dumps(r, indent=2) if a.json else report(r), end="\n" if a.json else "")
    return 1 if r["status"] in ("CRITICAL", "HIGH") else 0


if __name__ == "__main__":
    sys.exit(main())

# Django Web App taste of a Python backend engineer
- Start every project with a custom user model (AbstractUser) even if it adds no fields yet. Confidence: 0.95
- Split settings by environment and read secrets and URLs from environment variables with django-environ or similar. Confidence: 0.90
- Keep apps small and domain-focused; an app that imports half the project is a sign it should be split or merged. Confidence: 0.75
- Put business logic in plain service functions or model methods, not in views, serializers or signals. Confidence: 0.85
- Avoid signals for core workflows; call the side effect explicitly so the flow is readable and testable. Confidence: 0.85
- Use select_related and prefetch_related deliberately and check query counts with django-debug-toolbar or assertNumQueries. Confidence: 0.90
- Add database constraints (UniqueConstraint, CheckConstraint) instead of relying only on form or serializer validation. Confidence: 0.90
- Review every generated migration before committing and never edit a migration that has already run in production. Confidence: 0.90
- Write data migrations with RunPython using apps.get_model, never importing models directly. Confidence: 0.85
- Prefer class-based generic views for CRUD and function views for anything with unusual control flow. Confidence: 0.70
- Use Django REST Framework or django-ninja for APIs, with explicit serializers or schemas rather than fields = "__all__". Confidence: 0.85
- Wrap multi-step writes in transaction.atomic and enqueue background jobs with transaction.on_commit. Confidence: 0.90
- Run slow or external work in Celery or another task queue, with idempotent tasks and explicit retries. Confidence: 0.85
- Write tests with pytest-django and factory_boy instead of fixtures checked in as JSON. Confidence: 0.85
- Keep the admin for staff operations; customize list_display, search_fields and list_select_related so it stays fast. Confidence: 0.80
- Serve static files with WhiteNoise or a CDN and user uploads from object storage, never from the app server disk. Confidence: 0.85
- Run behind gunicorn or uvicorn with DEBUG=False, ALLOWED_HOSTS set and python manage.py check --deploy passing in CI. Confidence: 0.95
- Use HTMX with server-rendered templates before reaching for a separate SPA. Confidence: 0.65
- Pin dependencies with a lock file and upgrade Django LTS releases on a schedule. Confidence: 0.80
- Lint and format with Ruff and type check with mypy plus django-stubs. Confidence: 0.75

# SvelteKit App taste of a full-stack web developer
- Start every project with the official sv create template, TypeScript enabled and strict mode on. Confidence: 0.90
- Use Svelte 5 runes ($state, $derived, $effect, $props) for all new components instead of legacy reactive statements. Confidence: 0.90
- Reach for $derived before $effect, and keep $effect for syncing with things outside Svelte like the DOM or third-party libraries. Confidence: 0.90
- Load data in +page.server.ts when it needs secrets or the database, and use universal +page.ts only for public fetches. Confidence: 0.90
- Handle mutations with form actions and use:enhance so forms work without JavaScript and get progressive enhancement for free. Confidence: 0.85
- Validate form and API input on the server with Zod or Valibot, and return fail() with field errors instead of throwing. Confidence: 0.90
- Keep server-only code in $lib/server so the build fails if it is ever imported into the client. Confidence: 0.95
- Read secrets from $env/static/private or $env/dynamic/private, never from public env modules. Confidence: 0.95
- Put authentication and session lookup in hooks.server.ts and expose the user through event.locals. Confidence: 0.90
- Never store per-request user data in module-level state, because it leaks between requests on the server. Confidence: 0.95
- Share client state through context or small rune-based modules in .svelte.ts files, not a global store for everything. Confidence: 0.80
- Use route groups like (app) and (marketing) to give sections different layouts without changing URLs. Confidence: 0.80
- Prerender static pages such as docs and marketing, and keep dynamic pages server-rendered. Confidence: 0.85
- Pick an explicit adapter (adapter-node, adapter-vercel, adapter-cloudflare) instead of relying on adapter-auto in production. Confidence: 0.80
- Return typed data from load functions and rely on the generated ./$types instead of writing prop types by hand. Confidence: 0.90
- Use +error.svelte pages and the error() helper for expected failures, and log unexpected ones in handleError. Confidence: 0.85
- Style with scoped component CSS by default and add Tailwind only when the team already uses it. Confidence: 0.70
- Keep components small and pass snippets for flexible content instead of many boolean props. Confidence: 0.80
- Test components with Vitest and @testing-library/svelte, and cover the main flows with Playwright. Confidence: 0.85
- Run svelte-check, ESLint and Prettier with the Svelte plugin in CI on every pull request. Confidence: 0.85