AI Agent Web Fetcher at a glance
AI Agent Web Fetcher is a ready-to-run Apify actor from abotapi for collecting ai data. Its documented output includes url, title, description, markdown and loaded url, which you can export as JSON, CSV or Excel or read through the Apify API. It costs $2.00 per 1,000 results on the Apify Free plan and from $1.40 on paid plans, with smaller per-run or optional feature charges listed on Apify.
- Runs on
- Apify cloud: console, API, schedules and integrations
- Data type
- AI, Agents, Developer Tools
- Pricing
- $2.00 per 1,000 results (Free plan), from $1.40 on paid plans
- Output
- Documented below · JSON, CSV, Excel
- Inputs
- 21 options
- Run success
- 96.7% of completed public runs in the last 30 days
An advanced web fetcher that can fetch almost all websites and convert them to LLM-friendly Markdown format. Perfect for AI agents, RAG systems, and integration with search actors.
What AI Agent Web Fetcher can do
- Real browser rendering. With the default engine every page loads in a full headless browser, so JavaScript-rendered content, not just the raw HTML response, ends up in the output (engine set to http skips the browser for its first, plain request and has no JavaScript rendering).
- Clean Markdown, not raw HTML. Scripts, navigation, ads, forms and hidden elements are stripped before conversion, so downstream LLMs and agents get readable text instead of markup noise.
- More than Markdown. Title, description and author come back with every page; optional plain text, raw HTML, an HTML file and a screenshot in the key-value store, plus canonical URL, keywords, language, Open Graph, JSON-LD and response headers.
- Flexible URL input. Fetch one URL, a list of start URLs, or another actor's output dataset (link field from every item); optionally crawl same-site links, bounded by a page cap and glob patterns.
- Built for agent pipelines. Chain it after a search actor from the Integrations tab (it picks up that run's dataset automatically) or call the persistent Standby HTTP endpoint for real-time, low-latency fetches from your own backend or agent framework.
- Fails loudly, not quietly. A page that cannot be fetched within its time budget is reported as a failure in the run's status and log, never as a dataset item with empty content.
Choose what to collect
These inputs let you configure AI Agent Web Fetcher. Open the actor on Apify to enter your values and review the current options.
| Input | What it controls |
|---|---|
urlstring | The URL of the webpage to fetch (single URL mode). If the Start URLs list is left at its two built-in examples, only this URL is fetched (1 page, 1 result row); if you edit the list, this URL and the list are fetched together (duplicates removed). Clear this field to fetch the list alone. Exception: the default value https://www.google.com is not added to an edited list; to fetch google.com together with your own list, put it in the Start URLs list. |
startUrlsarray | Fetch several URLs in one run (used when no Dataset ID is given). Each page becomes one result row. Capped by Max pages. The form ships with two examples (a plain static page and a long article). If this list is left at those two examples, only the URL field above is fetched (1 page, 1 result row); if you edit the list, the URL field and your list are fetched together (duplicates removed). Clear the URL field to fetch just the two examples or your own list. Exception: the default value https://www.google.com is not added to an edited list; to fetch google.com together with your own list, put it in the Start URLs list. |
datasetIdstring | Dataset ID containing URLs to fetch. Used when integrating with search actors. Leave empty for single URL mode. |
urlFieldstring | Field name in dataset items containing the URL (default: 'link') |
maxUrlsinteger | Maximum number of URLs to fetch from dataset (1-100) |
maxCrawlDepthinteger | 0 (default) fetches only the given URLs. 1-10 also follows same-site links found on each page, that many levels deep. Every fetched page is one billed result row; failed pages (HTTP errors, non-HTML files) are skipped and never billed. (Not used with Dataset ID: dataset mode fetches only the dataset's links.) |
maxCrawlPagesinteger | Upper bound on pages fetched in a multi-URL or crawl run (1-1000). |
Show 14 more inputs
| Input | What it controls |
|---|---|
includeUrlGlobsarray | Crawl only links matching one of these glob patterns, e.g. https://example.com/blog/ (Not used with Dataset ID: dataset mode fetches only the dataset's links.) |
excludeUrlGlobsarray | Skip links matching any of these glob patterns. (Not used with Dataset ID: dataset mode fetches only the dataset's links.) |
waitForSelectorstring | CSS selector to wait for (up to 15 s) before reading the page. Uses the standard browser path instead of the stealth first path. |
customHttpHeadersobject | Extra request headers as a JSON object, e.g. {"Accept-Language": "de"}. Uses the standard browser path instead of the stealth first path. |
removeElementsCssSelectorstring | CSS selector of elements to drop from the page before the Markdown/text conversion (e.g. nav.sidebar). Empty keeps the built-in cleanup only. |
keepElementsCssSelectorstring | If set, only the matching elements are converted to Markdown/text. |
saveHtmlboolean | Adds the full page HTML to the html field of each row (larger dataset). |
saveHtmlAsFileboolean | Stores each page HTML as a file in the key-value store and puts its link in htmlUrl. |
saveScreenshotsboolean | Takes a screenshot per page, stores it in the key-value store and puts its link in screenshotUrl. Uses the standard browser path instead of the stealth first path. |
waitUntilstring | When to consider navigation succeeded. |
debugboolean | Adds a debug object (fetch seconds, HTML length, engine, links found) to each row. |
mcpConnectorsarray | Optionally send the scraped results into the apps you already use, via Model Context Protocol (MCP) connectors. Authorize a connector once under Apify -> Settings -> Integrations, then select it here. The connector receives a condensed, human-readable summary per item (title + key fields), not the full JSON; the complete record stays in the dataset. Leave empty to skip. Supported: Notion (https://mcp.notion.com/mcp), Linear (https://mcp.linear.app/sse), Airtable (https://mcp.airtable.com/mcp), Apify (https://mcp.apify.com). |
notionParentPageUrlstring | URL (or id) of the Notion page under which item pages are created. Required to enable the Notion export; ignored by other connectors. |
maxNotifyListingsinteger | Cap on items written to each connector per run. Does not affect the dataset. |
Fields in the output
The documented dataset includes the fields below. Availability can depend on the source page and selected mode.
urlURL (text)metadata.titleTitle (text)metadata.descriptionDescription (text)markdownMarkdown (text)crawl.loadedUrlLoaded URL (text)crawl.httpStatusCodeHTTP status (number)crawl.contentTypeContent type (text)crawl.depthDepth (number)crawl.loadedTimeLoaded at (date)metadata.canonicalUrlCanonical URL (text)metadata.languageCodeLanguage (text)metadata.keywordsKeywords (text)Show 1 more field
metadata.openGraphOpen Graph (object)Example output
An example from this actor’s documentation. Values are illustrative; this is not a fresh live result.
{
"url": "https://example.com/articles/global-markets-update",
"markdown": "# Global Markets Update\n\nA summary of today's session, including sector moves and a look ahead.\n\n## Today's Highlights\n\n- First point\n- Second point\n\nRead the [full report](https://example.com/reports/2026-q3).",
"metadata": {
"title": "Global Markets Update",
"description": "A daily summary of major market moves.",
"author": "Jane Analyst"
}
}Ways to use this data
- AI agent and RAG builders: convert any page into LLM-ready Markdown as live context for an agent, chatbot or retrieval index.
- Search-augmented apps: chain this actor after a web search actor so every result link is fetched and converted automatically, without writing glue code.
- Research and content teams: pull readable text from articles, docs and product pages without manually stripping ads and navigation.
- Backend and automation builders: call the Standby HTTP endpoint directly from your own service for on-demand, real-time page fetches.
Before you run
Start with a small input and check the resulting records against your expected fields. Source content and actor options can change; the current Apify listing is the reference for availability and pricing.
Questions about this actor
What data does AI Agent Web Fetcher return?
The documented output for AI Agent Web Fetcher includes url, title, description, markdown, loaded url, http status, content type and depth. Availability of each field depends on the source page and the selected mode.
How much does AI Agent Web Fetcher cost?
It costs $2.00 per 1,000 results on the Apify Free plan and from $1.40 on paid plans, with smaller per-run or optional feature charges listed on Apify. The Apify listing shows the current rate for every plan.
Can I run it on a schedule?
Yes, schedule it from the Schedules tab like any other actor. Every run is independent: it always (re)fetches the URLs or dataset you give it, there is no built-in change tracking or incremental memory, so add your own de-duplication if you only want new pages.
Why is the Markdown mostly empty or missing text I can see in my browser?
The default wait condition stops as soon as the initial page loads, so content a page injects later (infinite scroll, chat widgets, some single-page apps) may not be present yet; switch Wait Until to networkidle for pages like that. Scripts, navigation, ads, forms and hidden elements are also deliberately stripped before conversion, so a navigation-heavy page will look sparse in Markdown by design.
Why did my run fail instead of returning an empty dataset?
If a page cannot be fetched within its time budget, through both the primary and fallback browser attempts, that URL fails on its own without writing anything to the dataset, and the run's status message reports how many URLs succeeded and failed. If every URL in a run fails, the whole run fails rather than finishing with an empty dataset, so "nothing could be read" is never confused with "no data on this page". One exception: in batch mode, if no dataset item has a value in the Url Field you set, there is nothing to fetch, so the run finishes with an empty dataset and a warning in the log; check that Url Field matches your dataset.
