Apify-Haystack integration
The official integration of the Apify platform with Haystack.
Apify is a platform for web scraping, data extraction, and web automation tasks.
It provides serverless applications called Actors for tasks like crawling websites and scraping Facebook, Instagram, and Google results.
Haystack is an open-source framework for building LLM applications, RAG pipelines, and agents.
For a step-by-step guide, see the Haystack integration page in the Apify documentation.
Installation
The integration is available as the apify-haystack package on PyPI. It requires Python 3.11 or newer and Haystack 3.
pip install apify-haystack
Upgrading from 0.1.x? See UPGRADING.md.
Components
ApifyDatasetFromActorCall runs an Actor, waits for it to finish, and loads its default dataset.
ApifyDatasetFromTaskCall does the same for a saved Actor task.
ApifyDatasetLoader loads an existing dataset by its ID.
ApifyWebSearch searches the web with the RAG Web Browser Actor and returns the content of the top result pages.
ApifyActorTool is a tool for Haystack Agents that runs any Actor and returns its dataset items. It takes its parameters from the Actor's input schema, and it marks the items it hands to the LLM as untrusted external content to guard against prompt injection.
The dataset components turn dataset items into Haystack Document objects with a mapping function. Without one, they use default_dataset_mapping_function. For website and search Actors, it takes the item's markdown or text as the content and the page url, title, and description as meta. For other Actors, the content is the whole item as JSON.
Every component also has run_async, which Pipeline.run_async uses to run it without blocking the event loop.
When an Actor or task run finishes with any status other than SUCCEEDED, the component raises a RuntimeError.
The components read your Apify API token from the APIFY_TOKEN environment variable, or from APIFY_API_TOKEN if APIFY_TOKEN isn't set. To pass the token differently, use a Haystack Secret, for example apify_token=Secret.from_env_var('MY_APIFY_TOKEN').
Examples
The examples need an Apify account and API token. You can start with a free account at Apify and get your API token, then set it as the APIFY_TOKEN environment variable.
Crawl a website and load it as Documents
ApifyDatasetFromActorCall runs Website Content Crawler and loads the crawled pages as Document objects with the default mapping function:
from apify_haystack import ApifyDatasetFromActorCall
crawler = ApifyDatasetFromActorCall(
actor_id='apify/website-content-crawler',
run_input={
'maxCrawlPages': 3,
'startUrls': [{'url': 'https://haystack.deepset.ai/'}],
},
)
for document in crawler.run()['documents']:
print(document.meta['url'], document.content[:100])
Search the web
ApifyWebSearch returns the top result pages as Markdown Document objects, with the page url, title, and description in meta, and their URLs in links:
from apify_haystack import ApifyWebSearch
search = ApifyWebSearch(top_k=3)
result = search.run(query='What is Haystack?')
for document in result['documents']:
print(document.meta['url'], document.meta['title'])
Give an Agent access to an Actor
ApifyActorTool lets a Haystack Agent run an Actor. Limit dataset_items_limit to keep the tool output within the model's context window. For pay-per-event Actors, max_total_charge_usd caps what each run the Agent starts can cost:
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from apify_haystack import ApifyActorTool
tool = ApifyActorTool('apify/rag-web-browser', dataset_items_limit=3, max_total_charge_usd=0.5)
agent = Agent(chat_generator=OpenAIChatGenerator(), tools=[tool])
result = agent.run([ChatMessage.from_user('What is new in Haystack 3?')])
print(result['last_message'].text)
Save and load a pipeline
The components and the tool support Haystack serialization. The token is saved as the name of its environment variable, never as its value. Haystack 3 only loads classes and functions from trusted modules, so pass apify_haystack in allowed_modules, along with the module of your mapping function if you use a custom one:
from haystack import Pipeline
from apify_haystack import ApifyDatasetLoader
pipeline = Pipeline()
pipeline.add_component('loader', ApifyDatasetLoader('YOUR-DATASET-ID'))
yaml = pipeline.dumps()
restored = Pipeline.loads(yaml, allowed_modules=['apify_haystack'])
More examples
The examples directory has complete scripts:
Support
If you find a bug, please submit an issue on GitHub.
For questions, ask on Stack Overflow or join our Discord server.
Contributing
Code contributions are welcome.
If you have ideas for improvements, submit an issue or create a pull request.
For contribution guidelines, see CONTRIBUTING.md.
License
This project is licensed under the Apache License 2.0.