Core Capabilities

Documented Crawl4AI surface

A consolidated view of the crawler, API, and MCP features that the console should represent.

port 11235health / ready / metrics / playgroundweekly buildsamd64 + arm64

API endpoints

crawl, markdown, schema, auth

MCP tools

md, html, screenshot, pdf, crawl

Feature groups

markdown, browser, throughput, media

Modes

sync, stream, tool-based

HTTP API

Primary server endpoints

/crawl

Non-streaming crawl job creation and execution

api

/deep

Deep crawl traversal with domain tracking

api

/crawl/job

Asynchronous crawl job with webhook delivery

api

/crawl/stream

Streaming crawl output as NDJSON

api

/health

Service health check

api

/ready

Readiness gate

api

/metrics

Prometheus metrics

api

/schema

Configuration schema discovery

api

/config

Browser, crawler, and LLM configuration defaults

api

/md/{url}

Markdown generation for a target URL

api

/llm/{url}

LLM extraction endpoint

api

/llm/job

Asynchronous LLM extraction with webhook delivery

api

/result/{url}

Full CrawlResult payload with HTML, links, media, and artifacts

api

/links/{url}

Direct link extraction view

api

/media/{url}

Direct media extraction view

api

/html/{url}

Preprocessed HTML extraction

api

/screenshot/{url}

Screenshot artifact generation

api

/pdf/{url}

PDF artifact generation

api

/execute_js

JavaScript execution hook

api

/session

Session state persistence across crawl steps

api

/ask

Library context question answering

api

/token

JWT token issuance

api

/mcp/sse

Server-sent MCP transport for external agents

api

MCP tools

Tools exposed to external agents

Generate markdown from web content

mcp

html

Extract preprocessed HTML

mcp

screenshot

Capture webpage screenshots

mcp

pdf

Generate PDF documents

mcp

execute_js

Run JavaScript on web pages

mcp

crawl

Perform multi-URL crawling

mcp

ask

Query the Crawl4AI library context

mcp

Markdown and extraction

Clean markdown generation for RAG pipelines

Structured extraction with CSS, XPath, and LLM-based schemas

Reusable extraction schemas for repeated jobs

Browser control

JavaScript execution hooks

Proxy configuration and header control

Session persistence and local storage reuse

SSL handling and stealth browser modes

Throughput and resilience

Parallel crawling with dispatchers

Adaptive rate limiting and memory-aware scheduling

Robots.txt compliance and backoff on 429/503 responses

Media and links

Link extraction for internal and external targets

Media handling with screenshot and PDF capture

Selective exclusion of external assets and images