Spider The Web Crawler for AI Agents and LLMs

What is Spider?

Leverage a powerful data collecting solution engineered for exceptional speed and scalability. Built entirely in Rust, this platform provides next-generation performance, capable of crawling tens of thousands of pages rapidly in batch mode. It is specifically designed to enhance AI projects by providing efficiently gathered web data, aiming to significantly improve speed, productivity, and efficiency compared to standard scraping services, while also being more cost-effective.

The system offers seamless integration capabilities with a wide range of platforms, including major AI tools and services such as LangChain, LlamaIndex, CrewAI, FlowiseAI, AutoGen, and PhiData, ensuring data curation aligns perfectly with project requirements. It features concurrent streaming to save time and minimize bandwidth concerns, especially beneficial when crawling numerous websites. Users can obtain clean and formatted content in various formats like Markdown, HTML, or raw text, ideal for fine-tuning or training AI models. Additional performance boosts come from HTTP caching for repeated crawls and a 'Smart Mode' that dynamically utilizes Headless Chrome for pages requiring JavaScript rendering.

Features

High-Speed Crawling: Built in Rust for scalability and speed (crawls 20k+ pages in batch mode).
Concurrent Streaming: Efficiently streams results concurrently, saving time and bandwidth.
Multiple Response Formats: Outputs clean Markdown, HTML, raw text, JSON, JSONL, CSV, and XML.
Seamless Integrations: Compatible with LangChain, LlamaIndex, CrewAI, FlowiseAI, AutoGen, PhiData, and more.
Smart Mode: Dynamically switches to Headless Chrome for JavaScript-heavy pages.
AI Scraping (Beta): Enables custom browser scripting and data extraction using AI models.
HTTP Caching: Caches repeated page crawls to boost speed and reduce costs.
Cost-Effective: Offers significant cost savings compared to traditional scraping services.
Robots.txt Compliance: Adheres to robots.txt rules by default (can be disabled).

Use Cases

Gathering real-time web data for AI agents and LLMs.
Collecting formatted data (Markdown, text) for training AI models.
Executing large-scale web scraping projects efficiently.
Integrating web data extraction into automated data pipelines.
Building datasets for machine learning applications.
Automating data collection for market research and analysis.

FAQs

Why might a website crawl fail using Spider?

A crawl may fail if the website requires JavaScript rendering. Setting the request parameter to 'chrome' can often resolve this issue.
Can Spider crawl all pages on a website without needing a sitemap?

Yes, Spider is designed to accurately crawl all necessary content from a website even without a sitemap.
What data formats does Spider support for output?

Spider can output web data into HTML, raw text, and various markdown formats. For API responses, it supports JSON, JSONL, CSV, and XML.
How does Spider handle websites with dynamic content?

If you encounter issues with dynamic content, try setting the request parameter to 'chrome' or 'smart'. You might also need to set `disable_intercept` to true to allow third-party scripts.
Why might a crawl using Spider be slower than expected?

Slow crawls are often due to the website's robots.txt file specifying a crawl delay. Spider respects these delays, potentially up to 60 seconds, which can slow down the process.