Crawlforge MCP

crawlforge-mcp

CrawlForge MCP is a production-ready MCP server that gives AI agents the power to scrape websites, extract structured data, run deep research, bypass anti-bot detection, and process documents. It pac…

Quick Start

# Connect this server (installs CLI if needed)
npx -y smithery mcp add CrawlForgeDEV/crawlforge-mcp

# Browse available tools
npx -y smithery tool list CrawlForgeDEV/crawlforge-mcp

# Get full schema for a tool
npx -y smithery tool get CrawlForgeDEV/crawlforge-mcp fetch_url

# Call a tool
npx -y smithery tool call CrawlForgeDEV/crawlforge-mcp fetch_url '{}'

Direct MCP Connection

Endpoint: https://crawlforge-mcp--crawlforgedev.run.tools

Tools (20)

• fetch_url — Fetch content from a URL with optional headers and timeout
• extract_text — Extract clean text content from a webpage
• extract_links — Extract all links from a webpage with optional filtering
• extract_metadata — Extract metadata from a webpage (title, description, keywords, etc.)
• scrape_structured — Extract structured data from a webpage using CSS selectors
• search_web — Search the web using Google Search API (proxied through CrawlForge)
• crawl_deep — Crawl websites deeply using breadth-first search
• map_site — Discover and map website structure
• extract_content — Extract and analyze main content from web pages with enhanced readability detection
• process_document — Process documents from multiple sources and formats including PDFs and web pages
• summarize_content — Generate intelligent summaries of text content with configurable options
• analyze_content — Perform comprehensive content analysis including language detection and topic extraction
• extract_structured — Extract structured data from a webpage using LLM-powered analysis and a JSON Schema. Falls back to CSS selector extract…
• batch_scrape — Process multiple URLs simultaneously with support for async job management and webhook notifications
• scrape_with_actions — Execute browser action chains before scraping content, with form auto-fill and intermediate state capture
• deep_research — Conduct comprehensive multi-stage research with intelligent query expansion, source verification, and conflict detection
• track_changes — Enhanced content change tracking with baseline capture, comparison, scheduled monitoring, advanced comparison engine, a…
• generate_llms_txt — Analyze websites and generate standard-compliant LLMs.txt and LLMs-full.txt files defining AI model interaction guideli…
• stealth_mode — Advanced anti-detection browser management with stealth features, fingerprint randomization, and human behavior simulat…
• localization — Multi-language and geo-location management with country-specific settings, browser locale emulation, timezone spoofing,…

# Get full input/output schema for a tool
npx -y smithery tool get CrawlForgeDEV/crawlforge-mcp 

Prompts (1)

• getting-started (description)