@vibeflow-tools/web-crawl

@vibeflow-tools/web-crawl

Crawl any page, get clean markdown. Web crawler that extracts main content, strips boilerplate, and returns LLM-ready markdown — with an MCP server and CLI.

npm install -g @vibeflow-tools/web-crawl
web-crawl crawl https://example.com

---

Why It Matters

AI agents and LLMs work best with clean, focused content — not raw HTML full of nav bars, cookie banners, and footers. web-crawl fetches a page, extracts the main article via Mozilla Readability, converts it to clean markdown, and optionally filters to topic-relevant sections using an LLM.

• Clean markdown output — boilerplate stripped, GFM preserved
• LLM topic filtering — extract only the sections relevant to your query
• Browser rendering — handles JS-heavy pages via Playwright when needed
• Built-in caching — avoids re-fetching the same URL (8h TTL)
• Web search — search via SearXNG (70+ engines) or DuckDuckGo, no API key required

---

Quick Start

CLI (recommended)

# Install globally
npm install -g @vibeflow-tools/web-crawl

# Crawl a page
web-crawl crawl https://example.com

# Search the web
web-crawl search "how to use playwright"

# Crawl multiple pages
web-crawl crawl https://a.com https://b.com

# Filter to a specific topic
web-crawl crawl https://docs.example.com --filter "authentication setup"

# Output as JSON
web-crawl crawl https://example.com --format json

MCP Server

Note: The MCP server is not well tested. The CLI is the recommended tool.

Add to your editor's MCP config:

{
  "mcpServers": {
    "web-crawl": {
      "command": "web-crawl-mcp",
      "type": "stdio"
    }
  }
}

---

Commands

Command	Description
web-crawl crawl <url>	Crawl a URL and output clean markdown
web-crawl search <query>	Search the web via SearXNG or DuckDuckGo
web-crawl status	Show LLM config and cache status

web-crawl crawl <url> [options]

Crawl one or more URLs and output clean markdown.

web-crawl crawl https://example.com
web-crawl crawl https://a.com https://b.com          # multi-URL
web-crawl crawl https://docs.example.com --filter "auth"  # topic filter
web-crawl crawl https://example.com --browser         # force browser rendering
web-crawl crawl https://example.com --no-cache        # skip cache
web-crawl crawl https://example.com --meta            # include OG tags, description
web-crawl crawl https://example.com --format json     # JSON output

Flag	Default	Description
--filter <topic>	(none)	Filter content to topic-relevant sections via LLM
--no-cache	cache on	Disable result caching
--browser	auto	Force browser rendering (Playwright)
--no-browser	auto	Force HTTP-only, no browser
--cache-key <key>	(auto)	Custom cache key override
--retry <n>	0	Retry count on 429/503 errors
--timeout <ms>	20000	HTTP request timeout in ms
--meta	false	Include page metadata (OG tags, description, canonical)
--concurrent <n>	5	Max concurrent crawls for multi-URL mode
--format <fmt>	markdown	Output format: markdown or json

web-crawl search <query> [options]

Search the web via SearXNG (70+ engines aggregated) or DuckDuckGo. Free, no API key required.

web-crawl search "playwright browser automation"
web-crawl search "node.js best practices" --max-results 5
web-crawl search "typescript generics" --lang en-us
web-crawl search "react hooks" --engine ddg

Flag	Default	Description
--max-results <n>	10	Maximum number of results
--lang <code>	(none)	Language/region for results (e.g., de-at, en-us)
--engine <name>	searxng	Search engine: searxng or ddg
--format <fmt>	markdown	Output format: markdown or json

---

MCP Server

Note: The MCP server is not well tested. The CLI is the recommended tool.

The MCP server (web-crawl-mcp) exposes two tools over stdio:

crawl

Parameter	Type	Required	Description
url	string	Yes*	URL to crawl
urls	string[]	Yes*	Multiple URLs to crawl concurrently
filter_topic	string	No	Filter to topic-relevant sections via LLM
use_cache	boolean	No	Use cached result (default: true)
use_browser	boolean	No	Use browser rendering (default: auto)
cache_key	string	No	Custom cache key override
retry	number	No	Max retries on 429/503 errors
timeout_ms	number	No	HTTP timeout in ms
include_meta	boolean	No	Include page metadata
concurrency	number	No	Max concurrent crawls (default: 5)

*Either url or urls is required.

search

Parameter	Type	Required	Description
query	string	Yes	Search query
max_results	number	No	Maximum results (default: 10)
lang	string	No	Language/region (e.g., en-us)
engine	"searxng" | "ddg"	No	Search engine (default: searxng)

MCP Server Configuration

Flag	Description
--llm-enabled	Enable LLM topic filtering
--llm-url <url>	OpenAI-compatible endpoint
--llm-token <token>	API token
--llm-model <model>	Model name (default: gpt-4o)
--cache-ttl <seconds>	Cache TTL (default: 28800)
--http-timeout <ms>	HTTP timeout (default: 20000)
--browser-timeout <ms>	Browser timeout (default: 30000)
--no-browser-headless	Disable headless browser mode

---

Configuration

Environment Variables

Variable	Default	Description
CRAWL_MCP_LLM_ENABLED	false	Enable LLM topic filtering
CRAWL_MCP_LLM_URL	(none)	OpenAI-compatible chat completions endpoint
CRAWL_MCP_LLM_TOKEN	(none)	API token for the LLM endpoint
CRAWL_MCP_LLM_MODEL	gpt-4o	Model name for LLM filtering
CRAWL_MCP_CACHE_TTL_SECONDS	28800	Cache time-to-live (8 hours)
CRAWL_MCP_HTTP_TIMEOUT_MS	20000	HTTP request timeout
CRAWL_MCP_BROWSER_TIMEOUT_MS	30000	Browser page-load timeout
CRAWL_MCP_BROWSER_HEADLESS	true	Set to false for visible browser

A .env file in the current working directory is loaded automatically (does not overwrite existing env vars). CLI flags override environment variables.

Example

# Enable LLM topic filtering
CRAWL_MCP_LLM_ENABLED=true
CRAWL_MCP_LLM_URL=https://api.openai.com/v1/chat/completions
CRAWL_MCP_LLM_TOKEN=sk-...
CRAWL_MCP_LLM_MODEL=gpt-4o

Or via MCP server flags:

web-crawl-mcp \
  --llm-enabled \
  --llm-url https://api.openai.com/v1/chat/completions \
  --llm-token sk-... \
  --llm-model gpt-4o

---

How It Works

URL → Rate Limiter → HTTP Fetch (or Browser) → Readability → Turndown MD → Clean → [LLM Filter] → Cache → Return

1. Rate limiter — 2s minimum delay, 50-request session cap with 60s cooldown
2. Fetch — HTTP with gzip/brotli, redirect following, ETag/Last-Modified validation; or Playwright browser for JS-heavy pages
3. Extract — Mozilla Readability pulls main article content
4. Convert — Turndown + GFM produces clean markdown
5. Clean — strips nav, breadcrumbs, footers, and other boilerplate
6. Filter — optional LLM topic filtering (two-pass for structured content, single-pass for flat)
7. Cache — in-memory TTL cache with conditional re-fetching

---

Installation

npm install -g @vibeflow-tools/web-crawl

Or run without installing:

npx @vibeflow-tools/web-crawl crawl https://example.com

Requirements: Node.js >= 22

---

Contributing

pnpm install
pnpm --filter @vibeflow-tools/web-crawl run build
pnpm --filter @vibeflow-tools/web-crawl run test
pnpm --filter @vibeflow-tools/web-crawl run test:coverage
pnpm --filter @vibeflow-tools/web-crawl run mutation

---

License

Apache-2.0