Crawl4AI

Crawl4AI

Open-source LLM Friendly Web Crawler

v0.9.0
{{ backendOnline ? '✅ Backend terhubung — mode live' : '⚠️ Demo mode — backend tidak terdeteksi, gunakan simulated data' }}

🕷️ Web Crawler

📄 Results

⏱ {{ crawlResult.response_time || '?' }}ms 🔗 {{ crawlResult.links_count || 0 }} links 
{{ JSON.stringify(crawlResult, null, 2) }}

{{ crawlError }}

🔌 Model Context Protocol (MCP)

Connect Claude, VS Code, Cursor, or any MCP client to crawl4ai.

📡 SSE Endpoint

{{ backendUrl }}/mcp/sse

Server-Sent Events transport

🔗 WebSocket

{{ wsUrl }}/mcp/ws

WebSocket transport

MCP Tools

{{ tool.name }}

{{ tool.desc }}

tool

🔧 Claude Desktop Config

{{ JSON.stringify({mcpServers: {crawl4ai: {transport: "sse", url: backendUrl + "/mcp/sse"}}}, null, 2) }}

⚡ Zapier Integration

Dua skenario integrasi: Crawl4AI sebagai Action atau Trigger di Zapier.

SKENARIO 1 Zapier → Crawl4AI (Action)

Crawl website sebagai Action dalam Zap Anda. Cocok untuk: schedule rutin, trigger dari form, email, dll.

Setup di Zapier:

  1. Buat Zap baru → Trigger: pilih app apapun (Schedule, Email, Form, dll)
  2. Action: Webhooks by Zapier → POST
  3. URL: {{ backendUrl }}/webhook/catch
  4. Payload Type: JSON
  5. Data: {"url": "https://target.com", "formats": ["markdown"]}
  6. Test → response berisi content, response_time, links_count

Contoh: Google Form submission → Crawl website → Simpan hasil ke Google Sheets

SKENARIO 2 Crawl4AI → Zapier (Trigger)

Crawl4AI mengirim hasil crawl ke Zapier. Cocok untuk: notifikasi, alert, pipeline data otomatis.

Setup di Zapier:

  1. Buat Zap baru → Trigger: Webhooks by Zapier → Catch Hook
  2. Copy URL: https://hooks.zapier.com/hooks/catch/...
  3. Kirim request ke server dengan URL Zapier tersebut

Contoh Request:

curl -X POST {{ backendUrl }}/webhook/trigger \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "zapier_respond_url": "https://hooks.zapier.com/hooks/catch/xxx/yyy"
  }'

Deploy Webhook Server

Jalankan server webhook sendiri di Render / Railway / VPS. Source: zapier-webhook/ di repo.

Render: Root: zapier-webhook

Railway: gunicorn server:app

Lihat payload response webhook 
{
  "event": "crawl_complete",
  "task_id": "a1b2c3d4",
  "data": {
    "url": "https://example.com",
    "status": "success",
    "content": "# Page Title...",
    "format": "markdown",
    "response_time": 142,
    "links_count": 24
  },
  "timestamp": "2026-06-22T02:30:00Z"
}

Test Webhook Server

{{ zapierResult }}

{{ zapierError }}

📦 Deploy Your Own MCP Server

Jalankan MCP server crawl4ai sendiri di infrastruktur Anda.

🐳 Docker

docker build -t crawl4ai-mcp ./mcp-server
docker run -p 8000:8000 crawl4ai-mcp

☁️ Render

Deploy 1-click via Render:

# Set root: mcp-server
# Build: pip install -r requirements.txt
# Start: uvicorn main:app --host 0.0.0.0 --port $PORT

🐍 Python

cd mcp-server
pip install -r requirements.txt
python main.py

✅ Setelah deploy

  1. Masukkan URL server di field Backend URL (atas halaman)
  2. Klik ↻ cek untuk verifikasi koneksi
  3. Gunakan Crawl tab untuk test langsung
  4. Konfigurasi MCP client dengan endpoint SSE
  5. Integrasikan Zapier dengan webhook URL
{{ toast }}