Web Scraper For Llms

What Web Scraper For Llms can do

• Two ways to collect pages. Scrape a fixed list of URLs, or turn on crawl mode and let the actor follow links from a seed page to discover and read the rest of a site.
• Markdown and HTML output. Choose one format or both, per run, so the content slots straight into a prompt, a vector store, or a content pipeline.
• Automatic content cleaning. Navigation, headers, footers, sidebars, ads and tracking elements are stripped by default, so what you get is the page's actual content, not the surrounding layout.
• Handles slow and script heavy pages. A page is read with a quick request first, and the actor automatically falls back to full page rendering when a site needs JavaScript to show its content or blocks a simple request.
• Page details with every record. Title, language, favicon and canonical URL come back alongside the content, no extra parsing required.
• Fine grained control when you need it. CSS selectors to keep or drop specific elements, regex patterns to include or exclude URLs during a crawl, and a selector to wait for before reading a page.

Choose what to collect

These inputs let you configure Web Scraper For Llms. Open the actor on Apify to enter your values and review the current options.

Fields in the output

The documented dataset includes the fields below. Availability can depend on the source page and selected mode.

Example output

An example from this actor’s documentation. Values are illustrative; this is not a fresh live result.

{
  "url": "https://example.com/blog/how-rag-pipelines-work",
  "title": "How RAG Pipelines Work",
  "description": null,
  "markdown": "# How RAG Pipelines Work\n\nRetrieval augmented generation combines a search step with a language model, so answers can draw on content the model was never trained on.",
  "html": "<article><h1>How RAG Pipelines Work</h1><p>Retrieval augmented generation combines a search step with a language model.</p></article>",
  "metadata": {
    "title": "How RAG Pipelines Work",
    "language": "en",
    "favicon": "https://example.com/favicon.ico",
    "canonical": "https://example.com/blog/how-rag-pipelines-work",
    "description": null,
    "openGraph": null,
    "twitter": null
  },
  "duration": 4210,
  "scrapedAt": "2026-02-24T03:36:28.990Z",
  "success": true,
  "error": null
}

Ways to use this data

• RAG pipelines and AI knowledge bases: turn a list of pages or a whole site into clean markdown chunks ready for embedding.
• Documentation and content migration: crawl a docs site or blog and export every page as markdown in one run.
• Research and monitoring: pull articles, product pages or listings on a recurring schedule and compare the results over time.
• AI agent tooling: give an agent a single action that reads any URL and returns clean text instead of raw HTML.

Before you run

Start with a small input and check the resulting records against your expected fields. Source content and actor options can change; the current Apify listing is the reference for availability and pricing.

Questions about this actor