MCP Crawler (Tai Dt)

🕷️ Selenium MCP Server

MCP server (stdio) sử dụng Selenium để crawl và trích xuất dữ liệu từ web pages bằng CSS/XPath selectors.

✨ Tính Năng

MCP Tools (17 tools)

REST API Endpoints

Server cũng cung cấp REST API đầy đủ qua FastAPI:

• POST /extract - Basic extraction
• POST /smart-extract - Smart extraction
• POST /analyze - Analyze page
• POST /crawl/pages - Crawl with pagination
• POST /crawl/infinite-scroll - Handle infinite scroll
• POST /crawl/categories - Crawl categories
• POST /crawl/deep - Deep crawl
• POST /discover - Discover site
• POST /products/details - Product details
• POST /screenshot - Capture screenshot
• POST /export - Export data
• GET /jobs - List jobs
• GET /outputs - List output files

🚀 Cài Đặt

# Clone repo
git clone https://github.com/Tai-DT/mcp-crawler.git
cd mcp-crawler

# Tạo virtual environment
python3 -m venv venv
source venv/bin/activate  # Linux/Mac
# hoặc: venv\Scripts\activate  # Windows

# Cài đặt dependencies
pip install -r requirements.txt

📦 Chạy Server

MCP Server (cho AI clients)

# Stdio mode (mặc định, cho MCP clients)
python3 selenium_mcp_server.py

# HTTP mode (cho debugging)
python3 selenium_mcp_server.py --transport streamable-http --host 127.0.0.1 --port 8000

REST API Server

python3 server_api.py
# API sẽ chạy tại http://localhost:8000

Docker

docker-compose up -d
# API: http://localhost:8000

⚙️ Cấu Hình MCP Client

VS Code (mcp.json)

{
  "servers": {
    "local/selenium-crawler": {
      "type": "stdio",
      "command": "/path/to/mcp-crawler/venv/bin/python",
      "args": ["/path/to/mcp-crawler/selenium_mcp_server.py"],
      "env": {}
    }
  }
}

Claude Desktop

{
  "mcpServers": {
    "selenium-crawler": {
      "command": "python3",
      "args": ["/path/to/mcp-crawler/selenium_mcp_server.py"]
    }
  }
}

📖 Ví Dụ Sử Dụng

Basic Extract

{
  "url": "https://example.com",
  "fields": {
    "title": "css:h1",
    "description": "css:p",
    "links": {"selector": "css:a", "attribute": "href", "multiple": true}
  }
}

Smart Extract Products

{
  "url": "https://shop.example.com/category",
  "extract_type": "products",
  "limit": 50,
  "normalize_prices": true
}

Deep Crawl

{
  "url": "https://shop.example.com/products",
  "follow_links": "products",
  "max_links": 20,
  "normalize_prices": true
}

Screenshot

{
  "url": "https://example.com",
  "full_page": true,
  "width": 1920,
  "height": 1080
}

Login and Extract

{
  "login_url": "https://example.com/login",
  "target_url": "https://example.com/dashboard",
  "username_selector": "css:#email",
  "password_selector": "css:#password",
  "submit_selector": "css:button[type='submit']",
  "username": "user@example.com",
  "password": "password123",
  "fields": {
    "welcome_message": "css:.welcome",
    "account_name": "css:.account-name"
  }
}

🔄 Retry & Error Handling

Server có built-in retry với exponential backoff:

• Max Retries: 3 lần
• Initial Delay: 1 giây
• Backoff Factor: 2x
• Rate Limiting: 30 requests/phút

📁 Cấu Trúc Dự Án

mcp-crawler/
├── selenium_mcp_server.py  # MCP Server chính (17 tools)
├── server_api.py           # REST API (FastAPI)
├── crawler_engine.py       # Crawler engine core
├── extraction_utils.py     # Tiện ích xử lý dữ liệu
├── config.py               # Configuration management
├── retry_handler.py        # Retry & rate limiting
├── job_manager.py          # Job tracking & persistence
├── requirements.txt        # Python dependencies
├── Dockerfile              # Docker image
├── docker-compose.yml      # Docker Compose
├── outputs/                # Extracted data
│   └── screenshots/        # Captured screenshots
├── jobs/                   # Job history
└── scripts/                # Test scripts

🧪 Testing

# Test crawler engine
python3 test_mcp.py

# Smoke test MCP server
python3 scripts/smoke_test_stdio.py

📝 License

MIT