ScrapeGraphAI Python SDK
Official Python SDK for the ScrapeGraphAI API.
Install
pip install scrapegraph-py
# or
uv add scrapegraph-py
Quick Start
from scrapegraph_py import ScrapeGraphAI
# reads SGAI_API_KEY from env, or pass explicitly: ScrapeGraphAI(api_key="...")
sgai = ScrapeGraphAI()
result = sgai.scrape("https://example.com")
if result.status == "success":
print(result.data["results"]["markdown"]["data"])
else:
print(result.error)
Every method returns ApiResult[T] — no exceptions to catch:
@dataclass
class ApiResult(Generic[T]):
status: Literal["success", "error"]
data: T | None
error: str | None
elapsed_ms: int
🆚 Open Source vs Managed API
This SDK is a client for the managed cloud API. ScrapeGraphAI also ships an open-source library you can run yourself. This table explains the difference so you can pick the right one.
|
Open Source (scrapegraphai) |
Managed API (this SDK) |
| What it is |
A Python library you run yourself |
A hosted cloud service you call via SDK |
| Where it runs |
Your own infrastructure (self-hosted) |
ScrapeGraphAI cloud |
| LLM |
Bring your own (OpenAI, Groq, Gemini, Azure, local via Ollama) |
Managed for you |
| Browser / JS rendering |
You configure it (Playwright) |
Managed (stealth, auto/fast/js modes) |
| Proxies & anti-bot |
Your responsibility |
Included |
| Scaling & maintenance |
Your responsibility |
Fully managed |
| Cost model |
LLM tokens + your own infra |
Pay-as-you-go credits |
| Auth |
Your own LLM keys |
SGAI_API_KEY |
| Capabilities |
Graph pipelines (SmartScraper, Search, Speech, ScriptCreator…) |
Scrape, Extract, Search, Crawl, Monitor, History |
| Setup effort |
More configuration |
Minimal — API key + one call |
| License |
MIT |
SDK is MIT; the API service is paid |
Choose the open-source library if you want full control, on-prem/self-hosted data, local LLMs (Ollama), or fine-grained cost tuning — and you're happy to manage browsers, proxies and scaling yourself.
Choose the managed API (this SDK) if you want zero infrastructure, managed JS rendering & anti-bot, built-in Crawl and scheduled Monitor jobs, and the fastest path to production — billed per credit.
API
scrape
Scrape a webpage in multiple formats (markdown, html, screenshot, json, etc).
from scrapegraph_py import (
ScrapeGraphAI, FetchConfig,
MarkdownFormatConfig, ScreenshotFormatConfig, JsonFormatConfig
)
sgai = ScrapeGraphAI()
res = sgai.scrape(
"https://example.com",
formats=[
MarkdownFormatConfig(mode="reader"),
ScreenshotFormatConfig(full_page=True, width=1440, height=900),
JsonFormatConfig(prompt="Extract product info"),
],
content_type="text/html", # optional, auto-detected
fetch_config=FetchConfig( # optional
mode="js", # "auto" | "fast" | "js"
stealth=True,
timeout=30000,
wait=2000,
scrolls=3,
headers={"Accept-Language": "en"},
cookies={"session": "abc"},
country="us",
),
)
Formats:
markdown — Clean markdown (modes: normal, reader, prune)
html — Raw HTML (modes: normal, reader, prune)
links — All links on the page
images — All image URLs
summary — AI-generated summary
json — Structured extraction with prompt/schema
branding — Brand colors, typography, logos
screenshot — Page screenshot (full_page, width, height, quality)
extract
Extract structured data from a URL, HTML, or markdown using AI.
from scrapegraph_py import ScrapeGraphAI
sgai = ScrapeGraphAI()
res = sgai.extract(
prompt="Extract product names and prices",
url="https://example.com",
schema={"type": "object", "properties": {...}}, # optional
mode="reader", # optional
# Or pass html/markdown directly instead of url
)
search
Search the web and optionally extract structured data.
from scrapegraph_py import ScrapeGraphAI
sgai = ScrapeGraphAI()
res = sgai.search(
"best programming languages 2024",
num_results=5, # 1-20, default 3
format="markdown", # "markdown" | "html"
prompt="Extract key points", # optional, for AI extraction
schema={...}, # optional
time_range="past_week", # optional
location_geo_code="us", # optional
allowed_types=["text/html", "application/pdf"], # optional MIME allowlist
)
By default search accepts every supported content type, including PDFs, and processes up to 25
pages per PDF. You do not need to send processors or max_pages for this default. Use
allowed_types to restrict accepted MIME types. Only configure processors to override the cap;
PdfProcessor() also defaults to 25, while max_pages accepts 1–500, or -1 for no page limit.
crawl
Crawl a website and its linked pages.
from scrapegraph_py import ScrapeGraphAI, ScrapeMarkdownFormatEntry
sgai = ScrapeGraphAI()
# Start a crawl
start = sgai.crawl.start(
"https://example.com",
formats=[ScrapeMarkdownFormatEntry()],
max_pages=50,
max_depth=2,
max_links_per_page=10,
include_patterns=["/blog/*"],
exclude_patterns=["/admin/*"],
)
# Check status
status = sgai.crawl.get(start.data["id"])
pages = sgai.crawl.pages(start.data["id"], cursor=0, limit=50)
# Control
sgai.crawl.stop(crawl_id)
sgai.crawl.resume(crawl_id)
sgai.crawl.delete(crawl_id)
monitor
Monitor a webpage for changes on a schedule.
from scrapegraph_py import ScrapeGraphAI, MarkdownFormatConfig
sgai = ScrapeGraphAI()
# Create a monitor
mon = sgai.monitor.create(
"https://example.com",
"0 * * * *", # cron expression
name="Price Monitor",
formats=[MarkdownFormatConfig()],
webhook_url="https://...", # optional
)
# Manage monitors
sgai.monitor.list()
sgai.monitor.get(cron_id)
sgai.monitor.update(cron_id, interval="0 */6 * * *")
sgai.monitor.pause(cron_id)
sgai.monitor.resume(cron_id)
sgai.monitor.delete(cron_id)
history
Fetch request history.
from scrapegraph_py import ScrapeGraphAI
sgai = ScrapeGraphAI()
history = sgai.history.list(
service="scrape", # optional filter
page=1,
limit=20,
)
entry = sgai.history.get("request-id")
credits / health
from scrapegraph_py import ScrapeGraphAI
sgai = ScrapeGraphAI()
credits = sgai.credits()
# { remaining: 1000, used: 500, plan: "pro", jobs: { crawl: {...}, monitor: {...} } }
health = sgai.health()
# { status: "ok", uptime: 12345 }
Async Client
All methods have async equivalents via AsyncScrapeGraphAI:
import asyncio
from scrapegraph_py import AsyncScrapeGraphAI
async def main():
async with AsyncScrapeGraphAI() as sgai:
result = await sgai.scrape("https://example.com")
if result.status == "success":
print(result.data["results"]["markdown"]["data"])
else:
print(result.error)
asyncio.run(main())
Async Extract
async with AsyncScrapeGraphAI() as sgai:
res = await sgai.extract(
prompt="Extract product names and prices",
url="https://example.com",
)
Async Search
async with AsyncScrapeGraphAI() as sgai:
res = await sgai.search("best programming languages 2024", num_results=5)
Async Crawl
async with AsyncScrapeGraphAI() as sgai:
start = await sgai.crawl.start("https://example.com", max_pages=50)
status = await sgai.crawl.get(start.data["id"])
pages = await sgai.crawl.pages(start.data["id"], cursor=0, limit=50)
Async Monitor
async with AsyncScrapeGraphAI() as sgai:
mon = await sgai.monitor.create(
"https://example.com",
"0 * * * *",
name="Price Monitor",
)
Examples
Sync Examples
Async Examples
Environment Variables
| Variable |
Description |
Default |
SGAI_API_KEY |
Your ScrapeGraphAI API key |
— |
SGAI_API_URL |
Override API base URL |
https://v2-api.scrapegraphai.com/api |
SGAI_DEBUG |
Enable debug logging ("1") |
off |
SGAI_TIMEOUT |
Request timeout in seconds |
120 |
License
MIT - ScrapeGraphAI