What formats does the Web Crawl API return?

Every page comes back as cleaned, LLM-ready markdown, alongside structured JSON for its metadata, links, and contacts. You set the maximum characters kept per page, so you control how much content each result carries.

Can it crawl a whole site or just one page?

Both. Set maxCrawlDepth to 0 to scrape only the URLs you pass in, or raise it to spider the site, following links hop by hop. maxCrawlPages caps the total pages fetched, and the spider stays on the start URL's site.

Does it handle JavaScript-heavy sites?

Yes. The underlying engine renders dynamic pages and waits for content to load automatically, so you get the same markdown a browser would show. You do not configure a headless browser or manage proxies yourself.

How is this different from Firecrawl?

You get more than markdown: parsed links, harvested contacts with provenance, and an MCP server that exposes web.crawl as a native agent tool. It shares one harness and workspace with the SurfSense platform connectors, so a crawl and a Reddit or Maps pull live in the same workflow.

SurfSense

Connectors Pricing

SurfSense

Web Crawl connector

Web Crawl API: Turn Any Website into Agent-Ready Context

The SurfSense Web Crawl API points your AI agents at any URL and returns clean, LLM-ready markdown, structured links, and harvested contacts. Scrape a single page or spider a whole site, with the same agent harness that powers our platform connectors.

Start for free Read the docs GitHub

agent · surfsense

web.crawl({ startUrls: ["https://competitor.com"], maxCrawlDepth: 2,
  maxCrawlPages: 50 })

/pricing — Pro plan now $49/mo, up from $39markdown · depth 1 · successprice change
sales@competitor.comfound on 12 pages · site-widecontact
linkedin.com/in/their-head-of-salesteam page · anchor: 'VP Sales'social

50 pages · 41 markdown · 6 contacts · surfaced in 8.3s

What you can extract from Web Crawl

Give the API one or more start URLs. With depth 0 it scrapes just those pages; with a higher depth it spiders the site, bounded by your page cap, and returns one structured item per page.

Clean markdown

Cleaned, LLM-ready markdown for every page, truncated to the character limit you set.

Page metadata

Title, description, and other metadata, plus the URL, fetch status, and crawl depth.

Links

Every link with its anchor text and kind: internal, external, social, email, or tel.

Contacts

Emails, phone numbers, and social profiles harvested from each page's raw HTML.

Site-wide contacts

A deduplicated roll-up that separates company boilerplate from page-local people, with provenance.

Crawl control

Depth, page cap, and include or exclude URL regex patterns to keep the spider on target.

What teams do with the Web Crawl API

Deep research over the open web

Let an agent crawl dozens of sources on a topic, collect clean markdown, and synthesize an answer with citations. A deep-research API that reads the live web, not a stale index.

Competitor website monitoring

Crawl a rival's pricing, product, and changelog pages on a schedule and diff them. Catch a price change, a new feature, or a repositioning the day it ships, not the quarter after.

Lead and contact enrichment

Spider a company site and pull emails, phone numbers, and social profiles with provenance, so your agent knows which contact belongs to the company and which to a specific person.

RAG and knowledge-base ingestion

Turn any set of sites into clean markdown ready to embed. Feed a knowledge base or a RAG pipeline without writing a parser or babysitting a headless browser.

Call it from your code or your agent

One typed endpoint, one API key. Or add the SurfSense MCP server and let your agent call web.crawl as a native tool.

curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/web/crawl" \
  -H "Authorization: Bearer $SURFSENSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [
      "https://competitor.com"
    ],
    "maxCrawlDepth": 2,
    "maxCrawlPages": 50
  }'

Web Crawl API request and response schema

The exact contract behind POST /workspaces/{workspace_id}/scrapers/web/crawl. The same fields power the web.crawl MCP tool.

Request parameters

Only startUrls is required. With maxCrawlDepth 0 the call scrapes exactly those URLs; with a higher depth it spiders the site from them.

Request parameters
Field	Type	Description
`startUrls`	`string[]`required	Seed URLs to crawl, 1 to 20. With maxCrawlDepth 0 only these are fetched; otherwise they are the spider's entry points.
`maxCrawlDepth`	`integer` default `0`	Link-hops to follow from each start URL, 0 to 5. 0 scrapes only the start URLs; 1 also fetches their linked pages. The spider stays on the start URL's site.
`maxCrawlPages`	`integer` default `10`	Max pages to fetch in total, start URLs included, 1 to 200. The crawl stops at this ceiling.
`maxLength`	`integer` default `50000`	Max characters of cleaned markdown kept per page. Longer pages truncate.
`includeUrlPatterns`	`string[]` default `[]`	Regex patterns a discovered link must match to be followed. Empty follows every same-site link. Start URLs are always fetched. Max 25.
`excludeUrlPatterns`	`string[]` default `[]`	Regex patterns that exclude a discovered link from being followed. Wins over includeUrlPatterns. Max 25.

Response fields

The response is { items: [...], contacts: {...} }: one item per fetched page in crawl order, plus site-wide deduplicated contacts. Only successful pages are billed.

Response fields
Field	Type	Description
`items[].url / status`	`string`	The requested URL and its outcome: success, empty (fetched but no content), or failed.
`items[].markdown`	`string`	The page's content as cleaned markdown, ready for an LLM or a diff.
`items[].metadata`	`object`	Page metadata such as title and description.
`items[].crawl`	`object`	Crawl provenance: the URL actually loaded, its link depth, and the referrer page it was discovered on.
`items[].links`	`object[]`	Every link on the page with its anchor text and kind: internal, external, social, email, or tel.
`items[].contacts`	`object`	Emails, phones, and social profile URLs harvested from the page's raw HTML, including footer boilerplate the markdown omits.
`items[].error`	`string`	Failure reason when status is not success.
`contacts`	`object`	Site-wide contact rollup: every email, phone, and social URL deduplicated across pages, each with the pages it appeared on and a siteWide flag separating company boilerplate from page-local finds like team members.

A Firecrawl alternative with an agent harness

Commodity crawlers hand you markdown and leave the rest to you. SurfSense adds the harness, the contact extraction, and the platform connectors around it.

Feature	Commodity crawler	SurfSense
Output	Markdown or HTML you post-process yourself	Markdown plus parsed links, contacts, and site-wide roll-ups
Contacts	Not extracted; you write the regex	Emails, phones, and socials harvested with page provenance
Scope	One site at a time	Crawl the open web, then combine with platform connectors in one workflow
Pricing	Per-page credits that add up quickly	Pay per successful page, with a free tier to start
Agent-ready	An SDK you wire into your own harness	MCP server exposes web.crawl as a native tool

Web Crawl API: frequently asked questions

Point your agents at Web Crawl

The Web Crawl connector is one of many in the SurfSense competitive intelligence platform. Start free, no credit card required.

Start for free See pricing

SurfSense

Pages

Socials

Legal

SurfSense