crawl

⚠️ IMPORTANT: Store name is a POSITIONAL argument, NOT an option!

WRONG: crawl https://example.com --store=my-store
RIGHT: crawl https://example.com my-store

Crawling and indexing: $ARGUMENTS

node ${CLAUDE_PLUGIN_ROOT}/dist/index.js crawl $ARGUMENTS

The web pages will be crawled with Claude-driven intelligent link selection and optional natural language extraction, then indexed for searching. Requires Claude Code to be installed.

Note: The web store is auto-created if it doesn't exist. No need to create the store first.

Usage Examples

Intelligent crawl strategy:

/bluera-knowledge:crawl https://code.claude.com/docs/en/ claude-docs --crawl "all Getting Started pages"

With extraction:

/bluera-knowledge:crawl https://example.com/pricing pricing-store --extract "extract pricing and features"

Both strategy and extraction:

/bluera-knowledge:crawl https://docs.example.com my-docs --crawl "API reference pages" --extract "API endpoints and parameters"

Fast mode (axios-only, no JavaScript rendering):

/bluera-knowledge:crawl https://example.com/docs docs-store --fast --max-pages 20

Options

• --crawl <instruction> - Natural language instruction for which pages to crawl (e.g., "all Getting Started pages")
• --extract <instruction> - Natural language instruction for what content to extract (e.g., "extract API references")
• --max-pages <number> - Maximum number of pages to crawl (default: 50)
• --fast - Use fast axios-only mode instead of headless browser
  • Default behavior uses headless browser (Playwright via crawl4ai) for JavaScript-rendered sites
  • Use --fast when the target site doesn't use client-side rendering
  • Much faster than headless mode but may miss content from JavaScript-heavy sites