Skip to content

ShopSavvy for Apache Airflow

PythonBeta

An Apache Airflow provider package with a ShopSavvy hook, operators for product search, offers, and price history, plus a price sensor that waits for price drops. Build scheduled price monitoring and ETL pipelines.

View on GitHubGet Your API Key

Set up ShopSavvy for Apache Airflow

  1. 1

    Get a ShopSavvy Data API key

    Create a key at shopsavvy.com/data. Keys look like ss_live_ followed by 32 characters. Keep it out of source control.

  2. 2

    Install from GitHub

    The package in the install command (pip install airflow-provider-shopsavvy) is not on its public registry yet, so install from the source repository and follow its README:

    git clone https://github.com/shopsavvy/airflow-provider-shopsavvy.git
  3. 3

    Add your API key

    Set the SHOPSAVVY_API_KEY environment variable to your key (shell, .env file or Apache Airflow's secrets settings), or pass it to the client as the README shows.

  4. 4

    Make your first call

    From the shopsavvy/airflow-provider-shopsavvy README (Installation):

    # S3 output support
pip install "airflow-provider-shopsavvy[amazon]"

# Slack alert support (used in the price-monitor example DAG)
pip install "airflow-provider-shopsavvy[slack]"

    Copied from the repository README.

  5. 5

    Example DAGs

    From the shopsavvy/airflow-provider-shopsavvy README (Example DAGs):

    # airflow.cfg
[core]
dags_folder = /path/to/shopsavvy_provider/example_dags

    Copied from the repository README.

  6. 6

    Use your own connection ID

    From the shopsavvy/airflow-provider-shopsavvy README (Use your own connection ID):

    ShopSavvySearchOperator(
    task_id="search",
    query="laptops",
    conn_id="shopsavvy_prod",   # use a non-default connection
)

    Copied from the repository README.

Keys come with a Data API plan. See plans.

Features

ShopSavvyHook with Airflow connection management
SearchOperator, OffersOperator, PriceHistoryOperator
PriceSensor — waits for price to drop below threshold
Example DAGs for ETL and price monitoring
XCom integration for pipeline data passing

Use Cases

1.Schedule daily price data ETL to your warehouse
2.Monitor competitor prices with hourly checks
3.Trigger alerts when products drop below target prices
4.Build automated price intelligence pipelines

What data, how often, which direction

Data
Product records, current offers from tens of thousands of retailers and price history back to 2011, dense from 2019, from the ShopSavvy Data API. Prices are in US dollars.
How often
Every offer carries the time it was last checked. With the Data API you can schedule refreshes hourly, daily or weekly for retailers we can re-check through an official retailer API; other retailers update as ShopSavvy sees them.
Direction
ShopSavvy data flows into Apache Airflow. Requests carry only what you ask about (identifiers, search terms or schedules) and your API key.

FAQ

Do I need an API key?

Yes. The Apache Airflow integration calls the ShopSavvy Data API, which needs a Data API key. Create one at shopsavvy.com/data, then add it as the setup steps above show.

What data does it return?

ShopSavvy product records (title, brand, model, barcode, ASIN), current offers from tens of thousands of retailers with price, availability and condition, and price history back to 2011, dense from 2019. Products can be looked up by barcode, ASIN, URL or model number. Prices are in US dollars.

How fresh are the prices?

Every offer carries the time ShopSavvy last checked it, so you can see how current each price is. Through the Data API you can schedule refreshes hourly, daily or weekly for retailers we can re-check through an official retailer API; other retailers update as ShopSavvy sees them.

Is my API key stored securely?

The README has you supply it through the SHOPSAVVY_API_KEY environment variable or your own configuration, so it stays out of source code. The key is sent to the ShopSavvy API (api.shopsavvy.com) over HTTPS in the Authorization header. You can revoke a key and create a new one at shopsavvy.com/data/api-keys.

Can I install it from a package registry?

Not yet. The package named in its install command has not been published to its public registry, so install it from the GitHub repository (https://github.com/shopsavvy/airflow-provider-shopsavvy) for now.

Is it open source?

Yes. The source is on GitHub at https://github.com/shopsavvy/airflow-provider-shopsavvy under the MIT license.

Powered by the ShopSavvy Data API

This integration connects to the ShopSavvy Data API: product data and current prices from tens of thousands of retailers, across 30M+ products, with price history back to 2011, dense from 2019. Look up products by barcode, ASIN, URL or keyword, compare prices, read history and schedule refreshes.

Get StartedRead the Docs →

Related Integrations

Jupyter

Python

Magic commands for product search and pricing in Jupyter notebooks

Pandas

Python

Pandas accessor for enriching DataFrames with pricing data

Python SDK

Python

Official Python SDK for the ShopSavvy Data API

← All IntegrationsGet API Key →