AI tools for “crawl”
85 results (tools & skills) · 185 ms
Crawl docs, help centers, and websites into clean markdown for your AI support agent. Handles JS, CAPTCHAs, and proxies automatically. Pay per page, no subscription.
Search, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON from a site, crawl…
Crawl an existing website (capped, multi-page) and seed stardust/current/ with PRODUCT.md, DESIGN.md, DESIGN.json, a per-page inventory, and the consolidated brand surface — the captured design system, palette,…
Curate Gaia bot crawler branches into real-skill catalog entries and clean up processed bot/* branches. Use when the user says: "process bot branches", "curate the crawl output", "clean up bot proposals", "run bot…
AI-powered brand intelligence platform. Build knowledge graphs, score content with BRAND dimensions, and orchestrate workflows. Trusted by 500+ brands.
Context.dev is the web data API for AI agents: scrape any URL into LLM-ready Markdown, search the live web, crawl entire sites, and extract structured data with one REST API.
MCP server with 31 web scraping tools for AI agents. Turn any URL into clean markdown or JSON. Start free with 1,000 credits, no credit card required.
Preview Google Maps business data online without installing an extension. Run cloud jobs, save datasets, and export CSV, JSON, or Excel on eligible plans.
…"Google update hit me," "page speed," "core web vitals," "crawl errors," "indexing issues," "stuck on page 2," "striking distance keywords," "title tag rewrite," "low CTR," "local SEO," "Google Business Profile," or…
Search and AI crawler intelligence, verified from your server logs. Salience shows what Googlebot, GPTBot, ClaudeBot and 200+ crawlers requested, what they got back and where they wasted crawl budget.
…an asset, create a data product, run the data source crawl, delete a catalog asset, attach a conventions document to a table as a catalog skill asset, which assets in my project are missing descriptions or owners…
…by orchestrating the prep cascade — a full-inventory crawl (extract --prep), page-type and module-catalog confirmation (direct --prep), archetype prototypes plus design canon (prototype --prep), and asset…
…existing website through a four-phase pipeline — extract (crawl and capture the current site), direct (set a visual direction), prototype (generate redesigned HTML), and migrate (emit a deployable static site). Tracks…
Crawl and validate all internal and external links across an AEM Edge Delivery Services site. Uses the query index or sitemap to discover pages, extracts links from .plain.html renditions, checks HTTP status codes, and…
Transform any website into structured data with WaterCrawl. Built for developers, our modern web crawling framework makes data extraction easy and efficient.
Web scraping API to extract data from any website with one call. We handle proxies, browsers, CAPTCHAs and parsing. Get HTML, text, or AI-extracted JSON.
The web scraping API built for the AI era. Extract structured data from any website. No proxies, no selectors, no maintenance needed.
Discover the power of our Google Maps Scraper. Efficiently extract business data, reviews, and locations to enhance your marketing strategies and insights.
Extract structured JSON, Markdown, and SERP data from any website with XCrawl, the AI-ready web scraping API with built-in proxies, crawling, and automation.
The friendly web scraping platform for people and AI agents. Extract products, prices, images and reviews from any page in one click — or connect your AI agent and let it do the scraping for you.