WordPress Scraper: Plugins and Custom Content Parsers

A WordPress cron job fires at 03:00, reads forty items out of a news feed and inserts forty draft posts. It fires again at 04:00 and inserts the same forty again. A week later the site holds several thousand drafts of forty articles, and the duplicate check written to prevent exactly that has never matched once. The check calls get_posts() looking for a post that carries the same source GUID. get_posts() defaults to published posts, the imports are drafts, so the lookup returns an empty array every single time. That bug shipped in the earlier version of this article, and it is the most common way a WordPress scraper fails: no error, no warning, just a table that grows all night.

This guide covers what is worth automating inside WordPress, which ready-made plugins are alive in August 2026 and what they charge, and how to write the parsing yourself when the plugins run out. Everything below was checked against WordPress 7.0.4, released 12 August 2026, and WooCommerce 11.0.1. Plugin versions, install counts and prices were read from wordpress.org and from each vendor's own pricing page on 13 August 2026. Four scenarios follow with code: currency rates, news feeds, weather, and a supplier catalog landing in WooCommerce.

If you want to go deeper into the parsing technology itself, see the separate guide on web scraping with PHP, which covers the principles, the libraries (cURL, Guzzle, Simple HTML DOM, Symfony DomCrawler) and the usual pitfalls. Here the focus is specifically on the WordPress integration.

Two jobs that share one name

Importing data is not republishing prose. Both get called "scraping" and both end up in wp_posts, but they carry entirely different risk.

Exchange rates, forecasts, stock levels, your own supplier's price list: nobody holds a copyright on a number, no search engine cares that your euro rate matches everyone else's, and the page you build around the data is yours. Article bodies, product descriptions written by someone else, forum answers: those carry an owner and an opinion from Google.

That second half changed materially after this article was first written. With the March 2024 core update Google announced three new spam policies at once: scaled content abuse, expired domain abuse and site reputation abuse. Google said it expected the update and its earlier work to cut low-quality, unoriginal results by 40%; the April 2024 follow-up put the measured figure at 45%. Enforcement of the site reputation policy started on 5 May 2024. Google's spam policies page now carries a section headed Scraping, and lists reproducing feeds without unique benefit as an example of it.

The claim to drop is the "duplicate content penalty". It travels through every auto-blogging tutorial and it is not what Google documents. Google's own page on duplicate URLs describes canonical selection and consolidation, never a penalty. The real risk is narrower and worse: scraping and scaled content abuse are named spam categories, and spam categories carry manual actions. Google's helpful content guidance states the test directly: "If you use automation, including AI-generation, to produce content for the primary purpose of manipulating search rankings, that's a violation of our spam policies."

In the EU there is a second layer that has nothing to do with search. Article 15 of Directive (EU) 2019/790 gives press publishers the right "to authorise or prohibit the reproduction and making available to the public of their press publications". Recital 57 of the same directive says the right does not extend to acts of hyperlinking, nor to mere facts reported in press publications. Pulling a headline and a link is a different legal act from pulling the body.

The split runs through the rest of this page. Scenarios 1, 3 and 4 move data and are boring in the good sense. Scenario 2 moves prose, and every technical decision in it is downstream of one editorial question: does the visitor get something they could not get from the source. Full-body news scraping into a public archive is where the answer is usually no.

Numbers are safe. Sentences are not.

What a WordPress scraper actually has to do

Fetching is the easy part. Everything after it splits four ways, and that is where the time goes:

• Store it. Posts, WooCommerce products, post meta, options or a custom table. The choice decides what your queries will cost in two years.
• Cache it, so a page view does not become an outbound HTTP request and a visitor does not wait on someone else's server.
• Refresh it on a schedule, through wp_cron, a system cron, or a real job queue if the volume justifies one.
• Render it through shortcodes, widgets or blocks, with every value escaped on the way out.

A "WordPress scraper" is therefore never just a downloader. It is a module wired into four subsystems, each with a default that will surprise you at least once, and those defaults are the next section.

The plugins, read from their own pages on 13 August 2026

For a large share of sites a ready-made plugin is the whole answer, and writing PHP would waste a weekend. Versions, install counts and last-update dates below come from wordpress.org; prices come from each vendor's own pricing page.

Install counts on wordpress.org are buckets, not measurements: "40,000+" is the same label whether the real figure is 41,000 or 89,000. Only WP Automatic publishes an exact number, because CodeCanyon counts sales rather than installs, and those two things are not comparable.

WP All Import

The strongest of the group and the one worth the money if your source is structured. It takes XML, CSV, Excel and Google Sheets, gives you a drag-and-drop mapping screen from source fields to post fields, WooCommerce product fields, custom fields or users, and it handles scheduled re-imports that update records already imported instead of duplicating them. That update path is the part hobby scripts get wrong and the part you are actually paying for.

It does not scrape. Point it at an HTML product page and it has nothing to work with; it needs a feed, a file or a sheet. Every roundup that files WP All Import under "WordPress scraper plugins" is stretching the word.

Its recent changelog is unusually security-heavy: 4.0.1 is logged as "security improvement: fixes CVE-2026-2830", 4.1.0 as "security improvement: hardening of database queries", 4.1.1 as "security improvement: hardening of inline PHP execution restriction". Read that as a vendor responding to scrutiny rather than as a red flag, but read it.

Feedzy RSS Feeds and WP RSS Aggregator

The two serious RSS plugins, both at 40,000+ installs, both maintained within the last fortnight, both freemium with the useful half behind the paywall. Feedzy comes from Themeisle and prices at $99, $199 and $399 per year for 1, 3 and 50 sites. WP RSS Aggregator comes from RebelCode and prices at $99 Basic, $199 Pro and $269 Elite for a single site, with multi-site tiers above that.

The free versions display feeds. Turning feed items into real posts is a paid add-on in both, which is the feature almost everyone actually wants. Feedzy's free tier also caps you at one import project and one filter condition, enough to test and not enough to run. Both sell AI rewriting on top: Feedzy integrates OpenAI, WordAI and SpinnerChief, and WP RSS Aggregator sells AI credits by the year. If that is the feature that attracts you, reread the Google quote three sections up.

Feeds also lie about deduplication. The same story syndicated through three feeds arrives with three different GUIDs, three different titles and one body. Neither plugin fixes that for you.

Content Egg

An affiliate and price-comparison plugin rather than a general scraper, and the free version is far more capable than older reviews suggest: feed imports, price comparison and 25+ blocks at no cost, on PHP 8.0 or higher. Pro adds 28 affiliate network modules, raises the feed module cap from 3 to 50, and grants write permissions to AI agents that publish autonomously. Its output logic is built around affiliate product cards, and fighting that structure to produce something else is more work than starting from scratch.

WP Content Pilot

Pulls from 20-plus sources including RSS, YouTube, Flickr, Amazon, eBay, Reddit and Craigslist. Alive: 2.2.3, updated two days before this check, tested against WordPress 7.0.4. Small: 800+ installs and a 3.6-star average built from 23 five-star and 13 one-star reviews, which is the rating shape of a plugin that either fits your source exactly or does not work at all.

WP Automatic, and why plugin choice is a security decision

WP Automatic is the closest thing here to a general-purpose scraper: point it at arbitrary pages, define CSS selectors, get posts. It is also, by sales, the most widely deployed: 42,828 sales at $49 for a regular license, version 3.137.0 released 12 August 2026, from ValvePress.

It is also the cautionary tale. CVE-2024-27956, published 21 March 2024, is an unauthenticated SQL injection in ValvePress Automatic affecting every version up to and including 3.92.0. NIST scores it 9.8 critical; Patchstack, the assigning CNA, scores it 9.9. Attackers used it at scale within weeks of disclosure, creating administrator accounts on sites that had not updated.

The lesson generalizes past this one plugin. Every scraper plugin is, by design, code that accepts remote input, writes to your database and creates posts. That is the exact shape of an attack surface. Weigh patch cadence on the same side of the scale as price. The mirror image of the problem, keeping other people's scrapers off your own site, is covered in the anti-scraping WordPress plugins review.

Where the plugins run out

They cover the standard shapes well. They stop at a non-standard source, at transformation logic beyond find-and-replace, at authentication on the supplier's side, at a caching requirement someone else's UI does not expose, and at any source whose markup changes often enough to make a selector configuration a maintenance job. Past that line you write your own.

The four primitives you build on

The advantage of your own parser is control over all four subsystems. WordPress hands you decent tools for each, and there is no reason to reach for a raw file_get_contents.

Fetching. Use the HTTP API. It honors the site's proxy settings, respects filters other plugins have registered, and returns a WP_Error instead of a warning-and-false.

$response = wp_remote_get( 'https://example.com/data', array(
    'timeout'    => 15,   // the default is 5 seconds, which is not enough
    'user-agent' => 'MySiteBot/1.0 (+https://example.com/bot)',
) );

if ( is_wp_error( $response ) ) {
    error_log( 'Request error: ' . $response->get_error_message() );
    return;
}

$code = (int) wp_remote_retrieve_response_code( $response );
if ( 200 !== $code ) {
    error_log( "Source returned HTTP {$code}" );
    return;
}

$body = wp_remote_retrieve_body( $response );

Two defaults deserve attention. WP_Http::request() sets timeout to 5 seconds and httpversion to '1.0'; a slow supplier feed will trip the first one long before you suspect it. And if the URL comes from anywhere a user can influence, use wp_safe_remote_get() instead, which sets reject_unsafe_urls and validates the URL and every redirect it follows through wp_http_validate_url() to block server-side request forgery. wp_remote_get() leaves reject_unsafe_urls at false.

Caching. The Transients API is the right default, with one property people build on and should not:

function my_get_cached_data() {
    $cache_key = 'my_parser_data';
    $data      = get_transient( $cache_key );

    if ( false === $data ) {
        $data = my_fetch_and_parse();
        // An expiry is a maximum, never a minimum. This may be gone in a second.
        set_transient( $cache_key, $data, HOUR_IN_SECONDS );
    }

    return $data;
}

The Transients API documentation is blunt about it: "Transient expiration times are a maximum time. There is no minimum age. Transients might disappear one second after you set them, or 24 hours, but they will never be around after the expiration time." Names are capped at 172 characters. With a persistent object cache in play the value never touches wp_options at all, because set_transient() routes through wp_cache_set() when wp_using_ext_object_cache() is true. Without one it becomes two rows. Anything you cannot afford to lose mid-run belongs in a real table.

Scheduling. WP-Cron looks like cron and behaves like a to-do list checked by passing visitors:

add_action( 'init', function () {
    if ( ! wp_next_scheduled( 'my_parser_update' ) ) {
        wp_schedule_event( time(), 'hourly', 'my_parser_update' );
    }
} );

add_action( 'my_parser_update', 'my_fetch_and_parse' );

That snippet is correct and still a trap on a low-traffic site: with no visitors between 02:00 and 07:00, nothing runs until 07:00. The fix is two lines. Put define( 'DISABLE_WP_CRON', true ); in wp-config.php, then let the system scheduler drive it, which is exactly what the WordPress plugin handbook recommends:

0 * * * * wget --delete-after https://example.com/wp-cron.php

The wp_next_scheduled() guard matters more than it looks. Arguments have to match exactly for WordPress to consider two events the same, and a mismatch quietly schedules duplicates until the cron option grows into a performance problem of its own.

A detailed walk through HTML-parsing libraries, encoding handling, anti-bot evasion and building a parser that survives its source lives in the separate web scraping with PHP guide. Below we build on those principles and show the WordPress-specific parts.

Scenario 1. Scraping currency rates

A store or an information site shows prices in several currencies and converts them automatically. The European Central Bank publishes daily euro reference rates as XML, free, with no key and no registration, and its own page states the schedule: rates go out "around 16:00 CET every working day, except on TARGET closing days", covering 40 currencies.

That schedule is the first design decision. Fetching hourly buys you nothing. Reading eurofxref-hist-90d.xml on 13 August 2026 returned a newest entry of time="2026-08-12" with USD at 1.1545, because the 13 August rates had not been published yet. One request a day, scheduled after publication, is the correct shape.

function parse_currency_rates() {
    // ECB daily reference rates. Free, no key, quoted per 1 EUR.
    $url      = 'https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml';
    $response = wp_remote_get( $url, array( 'timeout' => 15 ) );

    if ( is_wp_error( $response ) ) {
        error_log( 'ECB fetch failed: ' . $response->get_error_message() );
        return;
    }
    if ( 200 !== (int) wp_remote_retrieve_response_code( $response ) ) {
        error_log( 'ECB returned HTTP ' . wp_remote_retrieve_response_code( $response ) );
        return;
    }

    $prev = libxml_use_internal_errors( true );
    $xml  = simplexml_load_string( wp_remote_retrieve_body( $response ) );
    libxml_use_internal_errors( $prev );

    if ( false === $xml ) {
        error_log( 'ECB XML did not parse' );
        return;
    }

    // The file declares a default namespace, so a plain //Cube matches nothing
    // and returns an empty array with no error. local-name() sidesteps prefixes.
    $nodes = $xml->xpath( '//*[local-name()="Cube"][@currency]' );

    if ( empty( $nodes ) ) {
        error_log( 'ECB XML parsed but contained no currency nodes' );
        return;
    }

    $rates = array( 'EUR' => 1.0 );
    foreach ( $nodes as $node ) {
        $rates[ (string) $node['currency'] ] = (float) $node['rate'];
    }

    // Third argument false: do not autoload this on every page request.
    update_option( 'my_currency_rates', $rates, false );
    update_option( 'my_currency_rates_updated', current_time( 'mysql' ), false );
}
add_action( 'my_currency_update', 'parse_currency_rates' );

This breaks silently, and here is exactly where. The ECB file opens with <gesmes:Envelope xmlns:gesmes="http://www.gesmes.org/xml/2002-08-01" xmlns="http://www.ecb.int/vocabulary/2002-08-01/eurofxref">. That second declaration puts every Cube element in a default namespace. SimpleXML's XPath then refuses to match //Cube and raises nothing: xpath() returns an empty array, the foreach runs zero times, and update_option() cheerfully stores an array with one entry in it. Your site shows EUR-only prices and no log line explains why. Registering the namespace with registerXPathNamespace() works too. local-name() works without keeping a URI you do not control in your source tree.

The earlier version of this article registered the namespace and never checked whether the node list came back empty. That check is the line that turns a silent failure into a logged one.

Now schedule it once, after publication:

add_action( 'init', function () {
    if ( ! wp_next_scheduled( 'my_currency_update' ) ) {
        // 17:30 UTC clears 16:00 in both CET and CEST.
        wp_schedule_event( strtotime( 'tomorrow 17:30 UTC' ), 'daily', 'my_currency_update' );
    }
} );

Converting between two non-euro currencies means going through EUR, since every published rate is quoted against it:

function convert_price( $amount, $from = 'USD', $to = 'GBP' ) {
    $rates = get_option( 'my_currency_rates', array() );

    if ( empty( $rates[ $from ] ) || empty( $rates[ $to ] ) ) {
        return null;   // null, not $amount: a wrong price is worse than no price
    }

    // amount_in_to = amount_in_from * (rate_to / rate_from)
    return round( $amount * ( $rates[ $to ] / $rates[ $from ] ), 2 );
}

Returning $amount unchanged on a missing rate, as the earlier version did, prints a dollar figure with a pound sign in front of it. Returning null forces the caller to decide. On a checkout page that difference is the difference between an incident and a fallback.

And a shortcode to print a rate:

add_shortcode( 'currency_rate', function ( $atts ) {
    $atts    = shortcode_atts( array( 'code' => 'USD' ), $atts, 'currency_rate' );
    $rates   = get_option( 'my_currency_rates', array() );
    $code    = strtoupper( preg_replace( '/[^A-Za-z]/', '', $atts['code'] ) );
    $updated = get_option( 'my_currency_rates_updated', '' );

    if ( ! isset( $rates[ $code ] ) ) {
        return '—';
    }

    return sprintf(
        '<span class="fx" title="%s">%s per EUR</span>',
        esc_attr( $updated ),
        esc_html( number_format( $rates[ $code ], 4 ) )
    );
} );
// Usage: [currency_rate code="GBP"]

Publishing the timestamp alongside the number, even in a title attribute, is worth the two lines. Stale financial data that looks fresh is the failure mode that costs money.

If you would rather not parse XML at all, Frankfurter wraps the same ECB data as JSON at https://api.frankfurter.dev/v2/rates, needs no API key, and states that it has no daily or monthly quotas, only abuse rate limiting. It puts a third party between you and the ECB, which is a real dependency, though it is open source and self-hostable via Docker. The full breakdown of central-bank formats, multi-currency handling and cross-rate construction is in the dedicated guide on scraping currency exchange rates.

Scenario 2. Scraping news and RSS

WordPress ships a feed parser, so nobody should be writing one. fetch_feed() wraps SimplePie, and the WordPress 7.0 branch bundles SimplePie 1.9.0, namespaced as SimplePie\SimplePie under wp-includes/SimplePie/src/. fetch_feed() caches responses for 12 hours by default, adjustable through the wp_feed_cache_transient_lifetime filter.

function import_news_from_rss( $feed_url ) {
    // No include_once needed: wp-includes/feed.php loads on every request.
    $feed = fetch_feed( $feed_url );

    if ( is_wp_error( $feed ) ) {
        error_log( 'Feed error: ' . $feed->get_error_message() );
        return;
    }

    $items = $feed->get_items( 0, $feed->get_item_quantity( 10 ) );

    foreach ( $items as $item ) {
        $guid = $item->get_id();

        // An explicit list, not 'any'. 'any' leaves out trashed posts, so
        // an item an editor threw away comes straight back on the next run.
        $existing = get_posts( array(
            'post_type'     => 'post',
            'post_status'   => array( 'publish', 'draft', 'pending', 'private', 'future', 'trash' ),
            'numberposts'   => 1,
            'fields'        => 'ids',
            'meta_key'      => '_source_guid',
            'meta_value'    => $guid,
            'no_found_rows' => true,
        ) );

        if ( $existing ) {
            continue;
        }

        $postarr = array(
            'post_title'   => wp_strip_all_tags( $item->get_title() ),
            'post_content' => wp_kses_post( $item->get_content() ),
            'post_status'  => 'draft',   // hold for moderation
            'post_type'    => 'post',
        );

        // SimplePie hands back UTC; let WordPress derive the local field.
        $gmt = $item->get_gmdate( 'Y-m-d H:i:s' );
        if ( $gmt ) {
            $postarr['post_date_gmt'] = $gmt;
            $postarr['post_date']     = get_date_from_gmt( $gmt );
        }

        $post_id = wp_insert_post( $postarr, true );

        if ( ! is_wp_error( $post_id ) ) {
            update_post_meta( $post_id, '_source_guid', $guid );
            update_post_meta( $post_id, '_source_url', esc_url_raw( $item->get_permalink() ) );
        }
    }
}

Three corrections against the earlier version of this article, and against most tutorials that copy it.

The include_once ABSPATH . WPINC . '/feed.php' line does nothing. WordPress loads feed.php during boot on every request, and the reference page for fetch_feed() carries a contributor note saying exactly that. The line is harmless and it is cargo.

The duplicate guard was broken. get_posts() defaults to numberposts => 5, post_type => 'post' and, through WP_Query, a post status of publish. The documentation states it plainly: only published posts will be found unless the status is specified. Inserting as draft and then searching among published posts means the guard never fires, and every cron run re-imports the whole feed. On an hourly schedule against a 40-item feed, that is 960 new drafts a day.

Dates were being written into the wrong field. post_date is the site-local column and post_date_gmt is the UTC one; writing a UTC value into post_date shifts every imported item by your timezone offset. Setting post_date_gmt and deriving the local value with get_date_from_gmt() is the correct direction.

Why wp_kses_post() appears explicitly. WordPress attaches its kses filters conditionally: kses_init() calls kses_init_filters() only if ( ! current_user_can( 'unfiltered_html' ) ). An import triggered by a logged-in administrator therefore runs with the filters removed, and raw feed HTML, script tags included, goes into the database untouched. Filtering explicitly costs one function call and removes the dependence on who happened to trigger the run.

Then schedule the import:

add_action( 'my_news_import', function () {
    import_news_from_rss( 'https://example.com/feed/' );
} );

If the source serves no feed and you have to pull headlines out of the markup by selectors, that is full HTML parsing with all of its maintenance attached. Getting structure out of arbitrary news pages, headlines and dates and images and body text, is a job with its own failure modes and its own upkeep.

The finer points of feeds, the RSS 2.0 and Atom formats, filtering and merging several sources are covered in the separate RSS parsing article, including what to do when a feed truncates bodies to a teaser.

Scenario 3. Scraping a weather forecast

Regional portals, hotels and travel sites all want a forecast widget. The WordPress-specific problem is not the API call; it is making sure a slow or dead weather service cannot take your site down with it.

The earlier version of this article used a placeholder host and an OpenWeatherMap-shaped JSON response, which meant nobody could run it. Here are the two real options, with limits read from each vendor's pricing page on 13 August 2026.

Open-Meteo needs no API key at all for non-commercial use, and publishes free-tier limits of 600 calls per minute, 5,000 per hour and 10,000 per day, roughly 300,000 a month. Commercial use requires a paid plan, a key, and attribution under CC BY 4.0. OpenWeather's free plan allows 60 calls per minute and one million per month across the Current Weather and 5-day/3-hour forecast APIs; its One Call product is pay-per-call with the first 1,000 calls per day free.

For a cached widget on a single site, either is free forever. Open-Meteo avoids the key-management problem, so the code below uses it.

function get_weather( $lat = 40.7128, $lon = -74.0060, $label = 'New York' ) {
    $cache_key = 'weather_' . md5( $lat . ',' . $lon );
    $weather   = get_transient( $cache_key );

    if ( false !== $weather ) {
        return $weather;   // may be an empty array: that is a cached failure
    }

    $url = add_query_arg( array(
        'latitude'  => $lat,
        'longitude' => $lon,
        'current'   => 'temperature_2m,weather_code',
        'timezone'  => 'auto',
    ), 'https://api.open-meteo.com/v1/forecast' );

    $response = wp_remote_get( $url, array( 'timeout' => 10 ) );

    if ( is_wp_error( $response ) || 200 !== (int) wp_remote_retrieve_response_code( $response ) ) {
        set_transient( $cache_key, array(), 5 * MINUTE_IN_SECONDS );
        return array();
    }

    $data = json_decode( wp_remote_retrieve_body( $response ), true );

    if ( ! isset( $data['current']['temperature_2m'] ) ) {
        set_transient( $cache_key, array(), 5 * MINUTE_IN_SECONDS );
        return array();
    }

    $weather = array(
        'label' => $label,
        'temp'  => (float) $data['current']['temperature_2m'],
        'code'  => (int) ( $data['current']['weather_code'] ?? 0 ),
        'time'  => (string) ( $data['current']['time'] ?? '' ),
    );

    set_transient( $cache_key, $weather, 30 * MINUTE_IN_SECONDS );

    return $weather;
}

Caching the failure is the whole point of those two extra set_transient() calls. Without them, an upstream outage turns into one outbound request per page view, each blocking a PHP worker for the full 10-second timeout. Five thousand page views during an outage is close to fourteen hours of blocked worker time. That is arithmetic rather than a measurement, but the shape holds: an uncached failure path converts someone else's downtime into yours.

add_shortcode( 'weather', function ( $atts ) {
    $atts = shortcode_atts( array(
        'lat'   => '40.7128',
        'lon'   => '-74.0060',
        'label' => 'New York',
    ), $atts, 'weather' );

    $w = get_weather( (float) $atts['lat'], (float) $atts['lon'], $atts['label'] );

    if ( empty( $w ) ) {
        return '<span class="weather-error">Weather unavailable</span>';
    }

    // WMO code table, abbreviated. Open-Meteo documents the full list.
    $labels = array(
        0  => 'clear sky',
        1  => 'mainly clear',
        2  => 'partly cloudy',
        3  => 'overcast',
        45 => 'fog',
        61 => 'light rain',
        71 => 'light snow',
        95 => 'thunderstorm',
    );
    $text = $labels[ $w['code'] ] ?? 'code ' . $w['code'];

    return sprintf(
        '<div class="weather-widget"><span class="weather-city">%s</span>'
        . '<span class="weather-temp">%s&deg;C</span>'
        . '<span class="weather-desc">%s</span></div>',
        esc_html( $w['label'] ),
        esc_html( round( $w['temp'] ) ),
        esc_html( $text )
    );
} );
// Usage: [weather lat="51.5072" lon="-0.1276" label="London"]

The unknown code falls through to printing the number rather than guessing at a description. A widget that says "code 82" is honest. A widget that says "clear sky" during a downpour is not.

A complete guide to weather-data sources, parsing multi-day forecasts and handling geolocation is in the scraping weather data article. Here we looked at it purely in the context of embedding on a WordPress site.

Scenario 4. Supplier products into WooCommerce

The heaviest scenario, and the one where WordPress stops being the interesting part. WooCommerce 11.0.1 sits on 7+ million installs, so the target is well documented. The difficulty is volume, images, and the fact that a bad import is visible to customers.

Check first whether you need code at all. WooCommerce ships a product CSV importer and exporter in core. Its documentation states that "the importer uses product IDs or SKUs in the CSV to match existing products", with an explicit "Update existing products" option. What it will not do is run on a schedule. If your supplier sends a file once a week and somebody is at a desk anyway, that is the entire project.

Everything below assumes you need it automatic.

function start_supplier_import() {
    // add_option() fails when the option already exists, which makes it a
    // better lock than a transient read-then-write between two cron runs.
    if ( ! add_option( 'supplier_import_lock', time(), '', false ) ) {
        return;
    }

    $response = wp_remote_get( 'https://supplier.example.com/price.xml', array(
        'timeout' => 60,
    ) );

    if ( is_wp_error( $response ) || 200 !== (int) wp_remote_retrieve_response_code( $response ) ) {
        delete_option( 'supplier_import_lock' );
        return;
    }

    $prev = libxml_use_internal_errors( true );
    $xml  = simplexml_load_string( wp_remote_retrieve_body( $response ) );
    libxml_use_internal_errors( $prev );

    if ( false === $xml || ! isset( $xml->offer ) ) {
        delete_option( 'supplier_import_lock' );
        error_log( 'Supplier feed unreadable' );
        return;
    }

    $offers = array();
    foreach ( $xml->offer as $offer ) {
        $sku = trim( (string) $offer->sku );
        if ( '' === $sku ) {
            continue;   // an offer without a SKU has no identity, so skip it
        }
        $offers[] = array(
            'sku'   => $sku,
            'name'  => (string) $offer->name,
            'price' => (float) $offer->price,
            'stock' => (int) $offer->stock,
            'image' => esc_url_raw( (string) $offer->picture ),
        );
    }

    $last = (int) get_option( 'supplier_last_count', 0 );

    if ( $last && count( $offers ) < $last * 0.7 ) {
        // A feed that shrinks by a third overnight is a supplier bug,
        // not a catalog change. Refuse it instead of acting on it.
        delete_option( 'supplier_import_lock' );
        error_log( sprintf( 'Supplier feed fell from %d to %d offers; aborting', $last, count( $offers ) ) );
        return;
    }

    update_option( 'supplier_last_count', count( $offers ), false );
    set_transient( 'supplier_offers', $offers, DAY_IN_SECONDS );
    as_schedule_single_action( time() + 10, 'supplier_import_batch', array( 0 ), 'supplier-import' );
}

The short-feed guard is worth more than it looks. A supplier whose export script dies halfway through serves you a perfectly valid XML file with 12 products in it. Without that comparison against the previous run, an obedient importer marks 19,988 products out of stock in one pass, and somebody finds out from the sales report.

The batch worker walks the cached offers 50 at a time through Action Scheduler, the job queue Automattic maintains and ships inside WooCommerce. Its own documentation reports processing queues in excess of 50,000 jobs and over 10,000 actions per hour. WP-Cron has no comparable figure because it has no queue.

add_action( 'supplier_import_batch', function ( $offset ) {
    $offers = get_transient( 'supplier_offers' );

    if ( ! is_array( $offers ) ) {
        // Transients have no minimum lifetime. Losing the working set
        // mid-run is a real outcome, so stop rather than half-import.
        delete_option( 'supplier_import_lock' );
        error_log( 'Supplier offers cache vanished at offset ' . $offset );
        return;
    }

    foreach ( array_slice( $offers, (int) $offset, 50 ) as $offer ) {
        upsert_product( $offer );
    }

    $next = (int) $offset + 50;

    if ( $next < count( $offers ) ) {
        as_schedule_single_action( time() + 5, 'supplier_import_batch', array( $next ), 'supplier-import' );
        return;
    }

    mark_missing_out_of_stock( wp_list_pluck( $offers, 'sku' ) );
    delete_transient( 'supplier_offers' );
    delete_option( 'supplier_import_lock' );
}, 10, 1 );

The upsert looks up an existing product by SKU, then writes only the fields that change:

function upsert_product( array $offer ) {
    $product_id = wc_get_product_id_by_sku( $offer['sku'] );

    if ( $product_id ) {
        $product = wc_get_product( $product_id );
        if ( ! $product ) {
            return;
        }
    } else {
        $product = new WC_Product_Simple();
        $product->set_sku( $offer['sku'] );
        $product->set_name( $offer['name'] );
        $product->set_status( 'draft' );   // a human approves new products

        // Sideload the image only on creation, never on a price update.
        if ( $offer['image'] ) {
            $attach_id = sideload_supplier_image( $offer['image'], $offer['name'] );
            if ( $attach_id ) {
                $product->set_image_id( $attach_id );
            }
        }
    }

    $product->set_regular_price( wc_format_decimal( $offer['price'] ) );
    $product->set_manage_stock( true );                       // before the quantity
    $product->set_stock_quantity( $offer['stock'] );
    $product->set_stock_status( $offer['stock'] > 0 ? 'instock' : 'outofstock' );

    $product->save();
}

function sideload_supplier_image( $url, $desc ) {
    // Admin-only files. They are not loaded in a cron request, and
    // media_sideload_image() then dies with an undefined function fatal.
    require_once ABSPATH . 'wp-admin/includes/file.php';
    require_once ABSPATH . 'wp-admin/includes/media.php';
    require_once ABSPATH . 'wp-admin/includes/image.php';

    $attach_id = media_sideload_image( $url, 0, $desc, 'id' );

    return is_wp_error( $attach_id ) ? 0 : (int) $attach_id;
}

Four things in that code are load-bearing:

• SKU is the identity. It is the only field both sides agree on. Since WooCommerce 9.1.0 there is also wc_get_product_id_by_global_unique_id() for GTIN, EAN, UPC and ISBN values, which is the better key when your supplier ships real barcodes; note that it returns int|null while wc_get_product_id_by_sku() returns int.
• Those three require_once lines are not decoration. media_sideload_image() lives in wp-admin/includes/media.php and depends on helpers from file.php and image.php. None of them load in a cron request. Omit the requires and the import runs perfectly by hand from the dashboard and fatals every single time it runs unattended, which is the worst possible failure schedule.
• set_manage_stock( true ) goes before set_stock_quantity(). Setting a quantity on a product that is not managing stock is an instruction WooCommerce has no reason to keep.
• New products arrive as drafts. A supplier typo becomes a public product page otherwise, and Google may index it before you notice.

Then the disappearance case, which is the one most imports get wrong:

function mark_missing_out_of_stock( array $seen_skus ) {
    $all = wc_get_products( array(
        'limit'  => -1,
        'return' => 'ids',
        'status' => 'publish',
    ) );

    foreach ( $all as $id ) {
        $product = wc_get_product( $id );
        if ( ! $product || in_array( $product->get_sku(), $seen_skus, true ) ) {
            continue;
        }
        $product->set_stock_status( 'outofstock' );
        $product->set_stock_quantity( 0 );
        $product->save();
    }
}

Products that vanish from a feed are usually out of stock, not discontinued. Deleting them destroys URLs that rank, reviews, and every order line that references them. Switching them to out of stock costs nothing and is reversible.

If the supplier serves no feed and you have to parse product cards out of their HTML, the job grows pagination, lazy loading, anti-bot protection and sometimes rotating proxies. Those principles are covered in detail in the web scraping with PHP guide, and the operational side of keeping a catalog current is in the price monitoring write-up.

What breaks between one page and ten thousand

Every scenario above works on a laptop against a small feed. Here is what changes when the numbers grow, in rough order of when you will hit it.

Meta lookups stop being free. WordPress indexes wp_postmeta on post_id and on meta_key(191), and on nothing else. meta_value is a longtext column with no index at all. The GUID dedup query in scenario 2 therefore scans every row sharing that meta key. At 500 imported posts nobody notices. At 200,000 it is the slowest query on the site, running once per feed item. Move the identifier to an indexed column when it starts to hurt: wp_posts carries KEY post_name (post_name(191)), so a hashed source key stored as post_name is an indexed lookup.

Images cost more than the download. WordPress scales anything over 2,560 pixels on a side, a threshold introduced in 5.3 and filterable through big_image_size_threshold, then generates the registered thumbnail sizes on top of the scaled original. One sideloaded product photo is therefore several files, several image-library operations, and disk. Multiply by a 20,000-product catalog before you decide to import images at all, then decide whether hotlinking the supplier's CDN is honest and permitted in your case.

One-shot imports run out of clock. PHP's max_execution_time and memory_limit end a monolithic loop somewhere in the middle, with no record of where. Twenty thousand upserts at 0.4 seconds each is over two hours. That figure is arithmetic rather than a measurement, since your per-item cost depends on the object cache, the plugins hooked into save_post and whether images are involved. Anything past a few thousand items needs batching with a persisted cursor, which is what the Action Scheduler pattern above provides.

WP-Cron overlaps itself. Two visitors landing simultaneously on a site with a due event can both spawn a run. WordPress guards this imperfectly, which is why the import above takes a lock before it does anything. Use add_option() rather than a transient for locks: it fails when the option already exists, whereas a read-then-write on a transient has a window between the two operations.

wp_options fills with debris. Transients with an expiry are stored with autoload off, so they do not slow every page load, but they do accumulate, and the Transients API documentation admits WordPress "infrequently cleans out expired transients". A parser that caches per-item results under unique keys can add thousands of rows a day. Either use a persistent object cache, which keeps them out of the database entirely, or delete your own transients when you are done with them.

Nothing tells you when it stops. A parser that quietly imports zero items looks identical to a parser with nothing to import. Log the count on every run, and alert on the count being zero twice in a row rather than on exceptions. Most scraper outages are not crashes; they are successful requests returning a changed page.

Six failure modes, and the front end looks fine in every one of them.

Where this approach stops working

JavaScript-rendered sources. wp_remote_get() retrieves HTML, not a rendered page. If the prices arrive through XHR after load, you get an empty container. Find the underlying JSON endpoint in the network tab, which is usually cleaner than the HTML anyway; if there is none you need a headless browser, and a headless browser does not belong on the same box as your WordPress site.

Bot management. Cloudflare announced on 1 July 2025 that it was changing the default for new domains to block AI crawlers unless they pay for content. Whatever your view of the policy, the operational consequence is concrete: default-deny is now a checkbox that many site owners never had to think about, and a polite PHP script with a custom user agent gets a 403 more often than it did in 2023.

Authentication and captchas. A supplier portal behind a login is a contractual question before it is a technical one. If your contract grants access, ask for an API or a feed. Vendors hand those out more readily than people expect, and a feed is cheaper for both sides than you scraping their catalog.

Shared hosting. Long-running imports, generous timeouts and image processing are the three things shared plans limit hardest. An import that works on your laptop and dies at 30 seconds in production is a hosting problem wearing a code costume.

Volume you cannot babysit. One feed on a cron schedule is maintenance you can absorb. Forty sources, each changing markup on its own timetable, is a job. That is the point where a managed extraction service or a data-as-a-service feed costs less than the developer time it replaces, because the maintenance moves off your plate along with the code.

The case for not scraping at all. If the source publishes an API, use it. It is faster, it is stable across redesigns, and it removes the legal ambiguity entirely. Working against an API instead of markup is covered in the API scraping guide.

Rules worth keeping

1. Use the HTTP API, and raise the timeout. The default 5 seconds is right for a health check and wrong for a supplier feed. When the URL is influenced by user input, switch to wp_safe_remote_get() and get validation on every redirect hop for free.
2. Never parse during a page load. Heavy work belongs on a schedule; the front end reads prepared data and nothing else. This single rule prevents most of the incidents in this article.
3. Replace WP-Cron with system cron for anything heavy. For genuine volume, use Action Scheduler instead, which gives you retries, groups, a queue and an admin screen showing what failed.
4. Escape on output, filter on input, and do both explicitly. External data is never trusted, and WordPress's automatic kses filtering depends on the capabilities of whoever triggered the run.
5. Check is_wp_error() and the HTTP status code. A 200 with an error page in the body is the common case, not the exotic one.
6. Log the count, not just the exception. Zero items imported is the signal that matters.
7. Honor robots.txt, rate limits and terms of use, and keep a contact URL in your user agent so an administrator can reach you before they reach for a block.

Summary

For a standard source, a maintained plugin is the right answer and the prices above are the whole cost. WP All Import for structured feeds, Feedzy or WP RSS Aggregator for RSS, Content Egg for affiliate catalogs. The moment the source or the transformation stops being standard, a custom parser built on the HTTP API, transients and a real scheduler gives you control that no settings screen can.

The four scenarios differ less than they look. Each is a fetch, a parse, a store and a render, and each fails in the same three places: a silent parse that returns nothing, a duplicate guard that never matches, a schedule that never fires. Check the node count after every XPath query. Search every post status when you check for duplicates. Watch the import count rather than the error log. Each scenario has a companion guide that goes past the WordPress wiring, on currency rates, RSS parsing and weather data.

The judgment call sits above the code. Rates, forecasts and your own supplier's stock levels are yours to automate freely. Somebody else's article is a copyright question and, since March 2024, a named spam category. Start with the web scraping with PHP guide for the parsing technology itself, and keep the two categories apart in your head before you write the cron job.