WordPress Scraper: Plugins and Custom Content Parsers

Web scraping is one of the most useful ways to run a WordPress site on autopilot. A good WordPress scraper can keep a site filled with fresh content — currency rates, news, weather forecasts, supplier products and much more — without anyone touching the dashboard. In this guide we'll look at which ready-made plugins exist, when it makes sense to write your own parser in PHP, and walk through four practical scenarios for integrating scraping into WordPress.

If you want to go deeper into the parsing technology itself, see the separate guide on web scraping with PHP, which covers the principles, the libraries (cURL, Guzzle, Simple HTML DOM, Symfony DomCrawler) and the usual pitfalls. Here the focus is specifically on the WordPress integration.

What "scraping" means in a WordPress context

Web scraping is the automated collection of data from external sources: websites, APIs, RSS feeds, price lists. In WordPress you usually need to do more than just fetch that data — you also have to:

• store it in the database (as posts, products, meta fields or options);
• cache it so you don't hammer the external source or your own server;
• refresh it on a schedule (via wp_cron or the system cron);
• render it on the front end (through shortcodes, widgets or Gutenberg blocks).

That's exactly why a "WordPress scraper" isn't just a script that downloads something — it's a full module wired into the CMS ecosystem.

A tour of the existing plugins

Before you write your own solution, it's worth checking what the ready-made plugins can do. They cover a lot of common tasks without a single line of code, and for many sites a WordPress scraper plugin is all you'll ever need.

WP All Import

One of the most powerful import tools. It accepts XML, CSV and price-list formats, lets you visually map source fields to post, WooCommerce product or user fields, and supports scheduled imports plus updates to already-imported records. A great fit for the supplier-products scenario — but it needs the source to serve a structured feed.

Pros: flexible mapping, friendly UI, reliable updates. Cons: the advanced features are in the paid version; it doesn't "scrape" arbitrary HTML pages without a prepared feed.

Content Egg

An aggregator plugin for affiliate and content marketing: it can pull products, prices, reviews and other content from many sources and APIs. Often used for price comparison and populating product cards.

Pros: many ready-made source modules, output templates. Cons: built around specific use cases (affiliate, marketplaces), with its own output logic.

WP RSS Aggregator / Feedzy

Specialized plugins for importing RSS and Atom feeds. They can merge several feeds, filter by keywords, and turn feed items into WordPress posts.

Pros: simple setup for news and RSS, filtering and deduplication. Cons: limited to the feed format — no good for arbitrary HTML.

Universal scrapers (WP Content Pilot, scraper plugins)

There are plugins that can pull data from arbitrary pages using CSS selectors. These come closest to a "real" scraper, but any change to the source's markup breaks the configuration, and complex post-processing logic is hard to express in them.

When plugins aren't enough

Ready-made solutions work great as long as your task is standard. But the moment you hit a non-standard source, complex transformation logic, authentication on the supplier's side, or specific caching requirements, a plugin's flexibility runs out. That's when you write your own parser.

Your own PHP scraper inside WordPress

The big advantage of a custom solution is full control. You decide how to fetch the data, how to clean it, where to store it and when to refresh it. WordPress gives you a convenient toolkit for all of this, and you don't have to use a "raw" file_get_contents at all.

The building blocks any WordPress scraper stands on:

// 1. Fetching data — via WordPress's built-in HTTP API
$response = wp_remote_get( 'https://example.com/data', array(
    'timeout' => 15,
    'headers' => array( 'User-Agent' => 'MySiteBot/1.0' ),
) );

if ( is_wp_error( $response ) ) {
    error_log( 'Request error: ' . $response->get_error_message() );
    return;
}

$body = wp_remote_retrieve_body( $response );

// 2. Caching — via the Transients API, so you don't hit the source on every render
function my_get_cached_data() {
    $cache_key = 'my_parser_data';
    $data = get_transient( $cache_key );

    if ( false === $data ) {
        $data = my_fetch_and_parse(); // your parsing function
        set_transient( $cache_key, $data, HOUR_IN_SECONDS );
    }

    return $data;
}

// 3. Scheduled refresh — via WP-Cron
add_action( 'init', function () {
    if ( ! wp_next_scheduled( 'my_parser_update' ) ) {
        wp_schedule_event( time(), 'hourly', 'my_parser_update' );
    }
} );

add_action( 'my_parser_update', 'my_fetch_and_parse' );

A detailed walk through HTML-parsing libraries, encoding handling, anti-bot evasion and building a resilient parser lives in the separate web scraping with PHP guide. Below we build on those principles and show the WordPress-specific parts.

Now let's get to concrete scenarios.

Scenario 1. Scraping currency rates

A classic task for an online store or an informational site — show prices in several currencies and convert them automatically. A good free source is a central bank's published reference rates; the European Central Bank, for example, serves daily rates as XML at a stable URL. (A US-based store could just as easily use a currency API that returns JSON.)

The logic is simple: once a day we fetch the XML rates, parse them, store them in WordPress options, and use them when rendering prices.

function parse_currency_rates() {
    // ECB daily reference rates, free, no key. Rates are quoted per 1 EUR.
    $url = 'https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml';
    $response = wp_remote_get( $url, array( 'timeout' => 15 ) );

    if ( is_wp_error( $response ) ) {
        return;
    }

    $xml = simplexml_load_string( wp_remote_retrieve_body( $response ) );
    if ( false === $xml ) {
        return;
    }

    // ECB namespaces its Cube nodes, so we query them with XPath.
    $xml->registerXPathNamespace( 'ex', 'http://www.ecb.int/vocabulary/2002-08-01/eurofxref' );

    $rates = array( 'EUR' => 1.0 );
    foreach ( $xml->xpath( '//ex:Cube[@currency]' ) as $node ) {
        $code = (string) $node['currency'];   // USD, GBP, JPY ...
        $rates[ $code ] = (float) $node['rate']; // units per 1 EUR
    }

    update_option( 'my_currency_rates', $rates );
    update_option( 'my_currency_rates_updated', current_time( 'mysql' ) );
}
add_action( 'my_currency_update', 'parse_currency_rates' );

Converting a price from one currency to another (cross-rate via EUR, since the reference rates are EUR-based):

function convert_price( $amount, $from = 'USD', $to = 'GBP' ) {
    $rates = get_option( 'my_currency_rates', array() );
    if ( empty( $rates[ $from ] ) || empty( $rates[ $to ] ) ) {
        return $amount;
    }
    // amount_in_to = amount_in_from * (rate_to / rate_from)
    return round( $amount * ( $rates[ $to ] / $rates[ $from ] ), 2 );
}

And a shortcode to print a rate on a page:

add_shortcode( 'currency_rate', function ( $atts ) {
    $atts  = shortcode_atts( array( 'code' => 'USD' ), $atts );
    $rates = get_option( 'my_currency_rates', array() );
    $code  = strtoupper( $atts['code'] );

    return isset( $rates[ $code ] )
        ? esc_html( number_format( $rates[ $code ], 4 ) . ' per EUR' )
        : '—';
} );
// Usage: [currency_rate code="GBP"]

The full breakdown of central-bank XML/JSON formats, handling multiple currencies and building cross-rates is in the dedicated guide on scraping currency exchange rates.

Scenario 2. Scraping news and RSS

The second popular scenario is filling a site with news automatically. Here WordPress gives you a nice bonus: the core already ships a feed parser, fetch_feed(), built on the SimplePie library. It's the most reliable way to work with RSS without reinventing the wheel.

function import_news_from_rss( $feed_url ) {
    include_once ABSPATH . WPINC . '/feed.php';

    $feed = fetch_feed( $feed_url );
    if ( is_wp_error( $feed ) ) {
        return;
    }

    $max_items = $feed->get_item_quantity( 10 );
    $items     = $feed->get_items( 0, $max_items );

    foreach ( $items as $item ) {
        $guid = $item->get_id();

        // Dedup guard — look for a post with the same source GUID
        $existing = get_posts( array(
            'meta_key'   => '_source_guid',
            'meta_value' => $guid,
            'post_type'  => 'post',
            'fields'     => 'ids',
        ) );
        if ( $existing ) {
            continue;
        }

        $post_id = wp_insert_post( array(
            'post_title'   => sanitize_text_field( $item->get_title() ),
            'post_content' => wp_kses_post( $item->get_content() ),
            'post_date'    => $item->get_date( 'Y-m-d H:i:s' ),
            'post_status'  => 'draft', // hold for moderation to avoid junk
            'post_type'    => 'post',
        ) );

        if ( $post_id && ! is_wp_error( $post_id ) ) {
            update_post_meta( $post_id, '_source_guid', $guid );
        }
    }
}

Run the import on a schedule:

add_action( 'my_news_import', function () {
    import_news_from_rss( 'https://example.com/blog/' );
} );

If the source doesn't serve RSS and you have to pull the news straight out of the HTML by selectors, that's full-blown scraping with markup parsing. We handle those projects as done-for-you news scraping: extracting headlines, dates, images and body text from arbitrary sites, structure and all.

The finer points of feeds, the RSS 2.0 and Atom formats, filtering and merging several sources are covered in the separate RSS parsing article.

Scenario 3. Scraping a weather forecast

A weather widget is a frequent request for regional portals, hotel sites and travel agencies. The data usually comes from a weather API that returns JSON. The WordPress-specific task is to embed that widget cleanly: fetch the forecast, cache it briefly and render it through a shortcode or block.

function get_weather( $city = 'New York' ) {
    $cache_key = 'weather_' . sanitize_key( $city );
    $weather   = get_transient( $cache_key );

    if ( false !== $weather ) {
        return $weather; // serve from cache
    }

    $url = add_query_arg( array(
        'q'     => $city,
        'units' => 'metric',
        'lang'  => 'en',
        'appid' => 'YOUR_API_KEY',
    ), 'https://api.example-weather.com/data/forecast' );

    $response = wp_remote_get( $url, array( 'timeout' => 10 ) );
    if ( is_wp_error( $response ) ) {
        return null;
    }

    $data = json_decode( wp_remote_retrieve_body( $response ), true );

    $weather = array(
        'temp'        => $data['main']['temp'] ?? null,
        'description' => $data['weather'][0]['description'] ?? '',
        'icon'        => $data['weather'][0]['icon'] ?? '',
    );

    // Weather changes often — cache for 30 minutes
    set_transient( $cache_key, $weather, 30 * MINUTE_IN_SECONDS );

    return $weather;
}

A shortcode to drop the widget into any post or page:

add_shortcode( 'weather', function ( $atts ) {
    $atts    = shortcode_atts( array( 'city' => 'New York' ), $atts );
    $weather = get_weather( $atts['city'] );

    if ( empty( $weather ) ) {
        return '<span class="weather-error">Weather unavailable</span>';
    }

    return sprintf(
        '<div class="weather-widget">
            <span class="weather-city">%s</span>
            <span class="weather-temp">%s&deg;C</span>
            <span class="weather-desc">%s</span>
        </div>',
        esc_html( $atts['city'] ),
        esc_html( round( $weather['temp'] ) ),
        esc_html( $weather['description'] )
    );
} );
// Usage: [weather city="London"]

The WordPress-specific points here: a short cache TTL (weather goes stale fast), rendering through a shortcode/widget, and always escaping data before output.

A complete guide to weather-data sources, parsing multi-day forecasts and handling geolocation is in the scraping weather data article — here we looked at it purely in the context of embedding on a WordPress site.

Scenario 4. Scraping a supplier site and importing products into WooCommerce

The most involved scenario is filling an online store with a supplier's products automatically. The source can be a price list (CSV/XML) or the supplier's own site, which you have to parse out of the HTML. The result is imported into WooCommerce as products with prices, stock, images and categories.

Let's walk through importing from a structured feed (the most robust option):

function import_supplier_products() {
    // Guard against parallel runs
    if ( get_transient( 'import_lock' ) ) {
        return;
    }
    set_transient( 'import_lock', 1, 10 * MINUTE_IN_SECONDS );

    $response = wp_remote_get( 'https://supplier.example.com/price.xml', array(
        'timeout' => 60,
    ) );

    if ( is_wp_error( $response ) ) {
        delete_transient( 'import_lock' );
        return;
    }

    $xml = simplexml_load_string( wp_remote_retrieve_body( $response ) );

    foreach ( $xml->offer as $offer ) {
        $sku   = (string) $offer->sku;
        $name  = (string) $offer->name;
        $price = (float) $offer->price;
        $stock = (int) $offer->stock;

        upsert_product( $sku, $name, $price, $stock, (string) $offer->picture );
    }

    delete_transient( 'import_lock' );
}

The "create or update" (upsert) function — it looks up the product by SKU and either updates the existing one or creates a new one:

function upsert_product( $sku, $name, $price, $stock, $image_url = '' ) {
    $product_id = wc_get_product_id_by_sku( $sku );

    if ( $product_id ) {
        // Product exists — update only price and stock
        $product = wc_get_product( $product_id );
    } else {
        // New product
        $product = new WC_Product_Simple();
        $product->set_sku( $sku );
        $product->set_name( $name );

        // Sideload the image only on creation
        if ( $image_url ) {
            $attach_id = media_sideload_image( $image_url, 0, $name, 'id' );
            if ( ! is_wp_error( $attach_id ) ) {
                $product->set_image_id( $attach_id );
            }
        }
    }

    $product->set_regular_price( (string) $price );
    $product->set_stock_quantity( $stock );
    $product->set_manage_stock( true );
    $product->set_stock_status( $stock > 0 ? 'instock' : 'outofstock' );

    $product->save();
}

Running the import on a schedule (say, twice a day):

add_action( 'init', function () {
    if ( ! wp_next_scheduled( 'supplier_import_event' ) ) {
        wp_schedule_event( time(), 'twicedaily', 'supplier_import_event' );
    }
} );
add_action( 'supplier_import_event', 'import_supplier_products' );

What to watch for in this scenario:

• Identify by SKU. The SKU is the most reliable key for matching products between the supplier and your store.
• Sideload images once. media_sideload_image() is expensive; there's no need to repeat it on every update.
• Batch processing. Large price lists (tens of thousands of items) are better handled in batches, so you don't hit PHP's time and memory limits.
• Unpublish, don't delete. Products that disappear from the supplier's feed are better switched to "out of stock" than deleted outright.

If the supplier doesn't serve a feed and you have to parse product cards straight from the HTML pages of their site, it's a heavier task with its own quirks (pagination, lazy loading, anti-bot protection, sometimes rotating proxies). The principles of that kind of parsing are covered in detail in the web scraping with PHP guide.

Technical recommendations

A few universal rules that make any WordPress scraper more robust:

1. Use the WordPress HTTP API (wp_remote_get/wp_remote_post) instead of file_get_contents and raw cURL — it honors the CMS's proxy settings, timeouts and filters.
2. Cache through the Transients API. It reduces load on both your server and the source.
3. Don't parse on the fly during a page load. Heavy parsing should run in the background on cron; the front end just reads the ready data.
4. Replace WP-Cron with the system cron for heavy jobs: WP-Cron only fires when someone visits the site and isn't suited to a long import.
5. Always escape data before output (esc_html, esc_url, wp_kses_post) — an external source can't be treated as trusted.
6. Log errors and check is_wp_error() at every step: the source can be down, change format, or return junk.
7. Respect the legal and ethical rules: honor robots.txt, request limits and the source's terms of use.

Conclusion

Scraping is a powerful way to automate content on a WordPress site. For standard tasks, ready-made plugins like WP All Import or Feedzy are often enough. But as soon as the source or the processing logic gets non-standard, a custom PHP parser gives you far more control and flexibility.

We covered four practical scenarios — currency rates, news and RSS, weather forecasts and importing a supplier's products into WooCommerce — and showed how to wire each one into the WordPress ecosystem with the HTTP API, transients and cron.

Want to go deeper into the parsing technology? Start with the foundational web scraping with PHP guide. And if you'd rather just have a working scraper built for your task — from news scraping to a full supplier-catalog import and price monitoring — we'll take it off your plate as a done-for-you data extraction service.