- name
- Web Crawling API
- description
- Fetch raw HTML, rendered HTML, or clean Markdown from public webpages through Just Serp API.
- author
- Just Serp API
- homepage
- https://justserpapi.com/?utm_source=clawhub.ai&utm_medium=referral&utm_campaign=justserpapi_web&utm_content=project_link
- metadata
- {"openclaw":{"homepage":"https://justserpapi.com/?utm_source=clawhub.ai&utm_medium=referral&utm_campaign=justserpapi_web&utm_content=project_link","primaryEnv":"JUST_SERP_API_KEY","requires":{"bins":["node"],"env":["JUST_SERP_API_KEY"]},"skillKey":"justserpapi_web"}}
Web Crawling
Use this skill when the user needs webpage retrieval rather than a search engine result. It fits crawling, scraping preparation, readable content extraction, and page structure inspection for a known URL.
When To Use It
- The user already has a target webpage URL and wants its raw HTML, rendered HTML, or cleaned Markdown.
- The task is about content extraction, page inspection, scrape preparation, or converting a page into LLM-friendly text.
- The user can provide a direct
urlto crawl. - The user needs page content from the source URL itself, not Google search results about that URL.
Representative Operations
html: Crawl Webpage (HTML) — Retrieve the raw HTML response for a page.renderedHtml: Crawl Webpage (Rendered HTML) — Retrieve DOM output after rendering for JavaScript-heavy pages.markdown: Crawl Webpage (Markdown) — Extract the main readable content as clean Markdown for summarization or downstream processing.
Request Pattern
- 3 read-only
GEToperations are available in this skill. - All operations require a direct
urlquery parameter. - No operation in this skill requires a request body.
- Choose
renderedHtmlfor dynamic pages,htmlfor raw source, andmarkdownfor readable content extraction.
How To Work
- Read
generated/operations.mdbefore choosing an endpoint. - Start with one of these operations when it matches the user's request:
html,renderedHtml,markdown. - Pick the smallest matching operation instead of guessing.
- Ask the user for any missing required parameter. Do not invent values.
- Call the helper with:
node {baseDir}/bin/run.mjs --operation "<operation-id>" --api-key "$JUST_SERP_API_KEY" --params-json '{"key":"value"}'Environment
- Required:
JUST_SERP_API_KEY - This skill uses
JUST_SERP_API_KEYonly for authenticated Just Serp API requests. - Keep
JUST_SERP_API_KEYprivate. Do not paste it into chat messages, screenshots, or logs. - Project site: Just Serp API.
- Authentication details: Just Serp API Docs.
Output Rules
- Start with what was fetched: raw HTML, rendered HTML, or cleaned Markdown.
- Echo the target URL so the crawl scope is explicit.
- For
markdown, surface the extracted readable content or key sections before raw JSON. - For HTML-oriented requests, mention whether the user asked for source HTML or rendered output.
- If the backend errors, include the backend payload and the exact operation ID.