Web Access MCP

Web访问MCP

提供web访问工具和SearXNG搜索集成的流式HTTP MCP服务器。

特性

• web_fetch_tool -获取网页并转换为Markdown
• extract_links_tool -从网页中提取所有链接
• 截屏工具 -截取网页截图
• 提取元数据工具 -从网页中提取元数据
• web_search_tool -使用SearXNG搜索网络
• image_search_tool -搜索图片
• 新闻搜索工具 -搜索新闻文章
• 视频搜索工具 -搜索视频

Docker 快速入门

1.先决条件

• Docker已安装

2.配置和启动

重要：跑步前，您必须编辑 start_docker.sh 并更改这些值：

# Generate a secret key
python -c "import secrets; print(secrets.token_urlsafe(32))"

# Edit start_docker.sh
nano start_docker.sh

更改这些行：

SEARXNG_SECRET="your_generated_secret_key_here"  # REQUIRED!
SEARXNG_BASE_URL="https://your-domain.com/searxng/"  # Optional, leave empty if not using reverse proxy

3.跑步

chmod +x start_docker.sh
./start_docker.sh

脚本将：

1. 检查是否更改了所需的值
2. 从模板生成SearXNG settings.yml
3. 创建Docker网络
4. 在端口8888上启动SearXNG
5. 在端口4568上构建并启动web访问mcp

4.验证

# Check services
docker ps

# Test SearXNG HTML
curl http://localhost:8888/

# Test SearXNG JSON API (should return search results)
curl -s "http://localhost:8888/search?q=test&format=json" | head -c 200

# Test web-access-mcp
curl -X POST http://localhost:4568/web-access-mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

访问URL

用法示例

列出可用工具

curl -X POST http://localhost:4568/web-access-mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

获取网页

curl -X POST http://localhost:4568/web-access-mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "jsonrpc":"2.0",
    "id":2,
    "method":"tools/call",
    "params":{
      "name":"web_fetch_tool",
      "arguments":{"url":"https://example.com"}
    }
  }'

搜索网络

curl -X POST http://localhost:4568/web-access-mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "jsonrpc":"2.0",
    "id":3,
    "method":"tools/call",
    "params":{
      "name":"web_search_tool",
      "arguments":{"query":"python tutorial","max_results":5}
    }
  }'

配置

环境变量

SearXNG配置

编辑 searxng/settings.yml 自定义：

• 搜索引擎
• 安全搜索
• 语言
• 用户界面主题

Apache2反向代理

要通过Apache2公开这些服务，请创建一个虚拟主机配置：

1.启用所需模块

sudo a2enmod proxy proxy_http proxy_wstunnel ssl rewrite headers
sudo systemctl restart apache2

2.创建虚拟主机

创建 /etc/apache2/sites-available/web-access.conf:

    ServerName your-domain.com

    # Redirect HTTP to HTTPS
    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

    ServerName your-domain.com

    # SSL Configuration (use Let's Encrypt or your certificate)
    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/your-domain.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/your-domain.com/privkey.pem

    # Security headers
    Header always set X-Content-Type-Options "nosniff"
    Header always set X-Frame-Options "SAMEORIGIN"
    Header always set X-XSS-Protection "1; mode=block"

    # SearXNG (http://localhost:8888)
    # Access: https://your-domain.com/searxng/
    ProxyPreserveHost On
    ProxyPass /searxng/ http://127.0.0.1:8888/
    ProxyPassReverse /searxng/ http://127.0.0.1:8888/

    # web-access-mcp (http://localhost:4568)
    # Access: https://your-domain.com/web-access-mcp
    ProxyPass /web-access-mcp http://127.0.0.1:4568/web-access-mcp
    ProxyPassReverse /web-access-mcp http://127.0.0.1:4568/web-access-mcp

    # WebSocket support for MCP Streamable HTTP
    RewriteEngine On
    RewriteCond %{HTTP:Upgrade} =websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteRule ^/web-access-mcp$ ws://127.0.0.1:4568/web-access-mcp [P,L]

    # Logging
    ErrorLog ${APACHE_LOG_DIR}/web-access-error.log
    CustomLog ${APACHE_LOG_DIR}/web-access-access.log combined

3.启用站点并重新启动

sudo a2ensite web-access
sudo systemctl restart apache2

4.更新sears网络基础URL

编辑 start_docker.sh 设置您的公共URL：

SEARXNG_BASE_URL="https://your-domain.com/searxng/"

然后重新启动服务：

docker stop searxng web-access-mcp
./start_docker.sh

5.防火墙（可选）

允许HTTP/HTTPS通过防火墙：

sudo ufw allow 'Apache Full'

安装后的公共URL

手动Docker命令

如果您更喜欢手动运行，则需要：

1. 生成 searxng/settings.yml 从模板：

# Copy template
cp searxng/settings.yml.template searxng/settings.yml

# Replace placeholder with your secret
SECRET=$(python -c "import secrets; print(secrets.token_urlsafe(32))")
sed -i "s/SEARXNG_SECRET_PLACEHOLDER/$SECRET/g" searxng/settings.yml

# (Optional) Set base_url
# sed -i "s|^server:|server:\n  base_url: \"https://your-domain.com/searxng/\"|g" searxng/settings.yml

1. 运行容器：

# Create network
docker network create web-access-net

# Start SearXNG
docker run -d \
  --name searxng \
  --network web-access-net \
  --user $(id -u):$(id -g) \
  -p 8888:8080 \
  -v $(pwd)/searxng:/etc/searxng:rw \
  searxng/searxng:latest

# Build and start web-access-mcp
docker build -t web-access-mcp .
docker run -d \
  --name web-access-mcp \
  --network web-access-net \
  -p 4568:4568 \
  -e SEARXNG_MCP_URL=http://searxng:8080 \
  web-access-mcp

测试

# Test all tools
python test_mcp_client.py

# Test specific tool
python test_mcp_client.py --tool web_search_tool

# Test against different URL
python test_mcp_client.py --url http://localhost:4568/web-access-mcp

发展

本地安装

pip install -e .
playwright install chromium

在本地运行

python -m web_access_mcp.server

运行测试

pip install -e ".[dev]"
pytest

常用命令

# View logs
docker logs -f searxng
docker logs -f web-access-mcp

# Restart services
docker restart searxng web-access-mcp

# Stop services
docker stop searxng web-access-mcp

# Start services
docker start searxng web-access-mcp

许可证

麻省理工学院