OpenCart Product Scraper: Import Supplier Catalogs

A supplier sends a catalog with 4,300 SKUs and mentions that prices move every Monday. Typing one product into the OpenCart admin takes about three minutes once you count the name, the description, two photos, the model code and the price. Three minutes across 4,300 products is 215 hours of somebody's month, and by the time you finish, the prices you typed first are six weeks old. That arithmetic is the whole reason an OpenCart product scraper exists.

The fetching part is rarely what goes wrong. Downloading a product page and pulling three fields out of it is an afternoon of work. What goes wrong is the writing: OpenCart moved its SEO URLs to a different table in 2017, moved SKUs to a different table in 2025, sets a non-default SQL mode on every connection it opens, and hides any product whose date_available is NULL. Every one of those is silent. You see 4,300 rows in the database and an empty storefront, or 4,300 products priced at one thousandth of what they should cost.

This guide is rebuilt against the current code. Everything below was checked on 13 August 2026 against the OpenCart source tree, the vendor's own download page and marketplace, and the pricing pages of the tools named. Both live branches shipped a release two days before that date: 4.1.0.4 and 3.0.5.1, both on 11 August 2026. OpenCart is not the largest platform on the web, and W3Techs put it at 1.9% of all e-commerce systems and 0.3% of all websites in its August 2026 survey. It is, however, one of the few where writing to the database directly is still the normal way to bulk-load a catalog.

Three things the earlier version of this guide got wrong

Corrections first, because two of them shipped bad data rather than an error message.

The url_alias table does not exist on OpenCart 3. The earlier text said SEO URLs live in url_alias on both 2.x and 3.x, and that OpenCart 4 moved them into seo_url. The move happened much earlier. We pulled the installer schema for every tagged release: url_alias is present in 1.5.6.4 and 2.3.0.2, and gone in 3.0.0.0, released 19 June 2017, which ships seo_url instead. An INSERT INTO oc_url_alias against any OpenCart 3 store fails with "Table doesn't exist". What OpenCart 4 changed was the shape of seo_url, not its existence.

The price parser returned wrong numbers instead of failing. Both the Python and the PHP versions stripped everything except digits and dots. On a German supplier page reading 1.234,56 EUR the Python one produced 1.23456. After conversion and markup that product lands in your store at $1.78 instead of $1,781.00, exactly one thousandth of the right price, with no exception raised and nothing in the log. The price section below has the full measured table across ten locale formats. The PHP version was wrong on a different set of inputs, including plain US formatting.

SKU is no longer a column on the newest branch. From 4.1.0.1 (March 2025) OpenCart writes SKU, UPC, EAN, JAN, ISBN and MPN into a separate oc_product_code table. The legacy columns still sit in oc_product, and the admin no longer reads them. A scraper that writes sku into oc_product on a 4.1 store produces products whose SKU field looks empty in the admin, and deduplication by SKU quietly matches nothing on the next run.

None of these break loudly. That is the pattern worth internalising before you write a line of import code.

Which OpenCart are you writing into

A scraper ends up either writing rows or generating an import file, and the schema underneath has moved four times. Pin the version before you pin anything else.

The history is older than most write-ups claim. Wikipedia's entry traces the name to a 1998 project by Christopher G. Mann for Walnut Creek CDROM, first published in May 1999 and abandoned by 2000; the domain lapsed, and Daniel Kerr rebuilt the thing in PHP, shipping 1.1.1 on Google Code in February 2009. The article's infobox still lists 4.1.0.3 from March 2025 as current, which is a decent illustration of how quickly this kind of fact rots.

Here is what the current tree actually looks like, read from the installer schema of each tag:

Release dates come from OpenCart's own download history, read on 13 August 2026. Do not use the repository's CHANGELOG.md for this: the file on master stops at v4.0.2.2 from April 2023 and has not been touched through four subsequent releases.

OpenCart 3 is not legacy. This is the single most useful thing to know before choosing a target. The 3.0.x branch got 3.0.4.1 in May 2025, 3.0.5.0 in January 2026 and 3.0.5.1 on the same day as the newest 4.x release. If a client is on 3.0.x, there is no urgency argument for migrating first and importing later.

OpenCart 4.1 is where the schema diverges most. Beyond the SKU move, oc_product gained master_id, variant and override for the variant system, and the whole database is InnoDB. The practical consequence is pleasant: multi-statement imports are transactional, so a half-written product can be rolled back. On 3.0.x you get MyISAM and no rollback at all, which changes how you order your writes.

The 1.5 and 2.x branches still turn up in the wild and share the url_alias shape. Treat them as a single case in your code and move on.

Forks and community distributions copy the schema of whichever OpenCart version they branched from, but they patch models and controllers freely. Check the tables yourself rather than trusting the version string in the footer.

Practical takeaway: before you write or buy a scraper, pin down the exact store version (the admin dashboard footer, or define('VERSION', ...) in the store's index.php) and the table prefix (oc_ by default, routinely changed). Then run one query: SHOW TABLES LIKE '%seo_url%'. Empty result means 1.5 or 2.x. A row means 3.x or later, and SHOW COLUMNS FROM oc_seo_url tells you which of the two shapes you are looking at.

The connection settings OpenCart changes and your script does not

This section did not exist in the earlier version of this guide, and it explains more failed imports than any other single fact.

Open upload/system/library/db/mysqli.php in either branch. Every OpenCart connection runs this immediately after connecting:

SET SESSION sql_mode = 'NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION'

On 4.1.0.4 there is a second line, SET FOREIGN_KEY_CHECKS = 0. The installer goes further still and runs SET @@session.sql_mode = '' before creating tables.

What that means for you. MySQL 8.4 ships STRICT_TRANS_TABLES, NO_ZERO_DATE, NO_ZERO_IN_DATE and ONLY_FULL_GROUP_BY in its default sql_mode. OpenCart replaces all of that with two harmless modes for the duration of its session. Your PyMySQL or PDO connection does not: it inherits the server default, strict mode included. So OpenCart's own code and your code are talking to the same tables under different rules, and the rules matter.

MySQL's manual defines the trap precisely: "A value is missing when a new row to be inserted does not contain a value for a non-NULL column that has no explicit DEFAULT clause in its definition." In strict mode that is an error, not a warning. The sql_mode reference spells out the nontransactional case too, which is the one you hit on OpenCart 3: the statement aborts and the table is unchanged.

On OpenCart 3.0.5.1, oc_product has thirteen such columns: model, sku, upc, ean, jan, isbn, mpn, location, stock_status_id, manufacturer_id, tax_class_id, date_added, date_modified. The earlier version of the import code in this guide supplied nine columns and omitted nine of those thirteen. On a stock MySQL 8 server it fails on the first row with error 1364, "Field 'upc' doesn't have a default value". On a hosting account still running non-strict mode it works. Same code, same store, opposite outcomes, which is why this bug survives so long in circulation.

On OpenCart 4.1.0.4 the same omission does not raise anything, and that is worse. The 4.x schema is generated from upload/system/helper/db_schema.php, where nearly every field carries a default and almost nothing is marked not_null. Omitted columns become NULL. Including date_available, which has neither a default nor a NOT NULL flag. And the storefront model in both branches selects products with p.date_available <= NOW(). NULL compared to anything is NULL, never true, so the product is invisible in the catalog while sitting perfectly visible in the admin list. Every "I imported 4,000 products and the category page is empty" thread ends here.

Two more differences worth carrying in your head. OpenCart 3's driver calls set_charset('utf8') against utf8mb4 tables, so anything outside the basic multilingual plane, emoji in particular, is mangled by OpenCart's own reads and writes even when your scraper inserts it correctly over a utf8mb4 connection. OpenCart 4 calls set_charset('utf8mb4') and behaves. And the foreign keys declared in db_schema.php are never created: the loop that would apply them sits inside a /* */ block in the installer, right under the comment // Setup foreign keys. Do not count on the database to catch a dangling category_id for you.

The fix is one line at the top of your importer, copying what OpenCart itself does, plus the discipline of supplying every column. Both appear in the code below.

Ready-made solutions, priced on 13 August 2026

If the store is standard and the budget is thin, you may not need to write anything. The market splits in two.

Import modules that live inside OpenCart

These do not crawl anything. They take a prepared CSV, XLS, XLSX or XML and load it into the catalog, which means they also handle the schema differences for you. Prices below are from the OpenCart marketplace on 13 August 2026:

One correction to the earlier version of this article, which called Export/Import Tool "low-cost" and recommended a module called CSV Price Pro at "around $20 per domain". Export/Import Tool is free. A marketplace search for CSV Price Pro on 13 August 2026 returns no module of that name at all, so treat any listicle still recommending it as untested since roughly 2019.

The pattern that keeps working is unglamorous: the scraper builds a file, the module loads it. You get to eyeball 4,300 rows in a spreadsheet before they touch the database, the module owns the version-specific SQL, and a bad run costs you a re-upload rather than a restore from backup.

General-purpose scrapers that export a file

• Octoparse is the cloud point-and-click option. The free plan covers 10 tasks and 50,000 exported rows a month, which is enough to test whether the supplier's markup is regular. Standard is $69/month, or about $58/month billed annually, and adds cloud extraction with three concurrent runs, IP rotation and scheduling. Professional is $249/month, or about $209 annually, for 250 tasks and 20 concurrent cloud runs.
• Web Scraper keeps its browser extension free and charges for the cloud: Project at $50/month for 5,000 URL credits and two concurrent scrapers, Professional at $100/month for 20,000 credits, Scale from $200/month. Residential proxies are an add-on at $2.50/GB.
• ParseHub still appears in every roundup. Its marketing site now renders entirely client-side and returns no readable content to a plain HTTP fetch, so we could not confirm current pricing from the vendor; the help centre at help.parsehub.com is intact but carries no dates. Verify tier and price yourself before budgeting around it.
• Managed data-as-a-service providers hand you a scheduled feed and you never run a crawler at all. That is the right shape when the supplier count grows faster than your maintenance budget.

When does a ready-made solution fall short? When the supplier's markup is irregular enough that point-and-click selectors miss half the fields, when the site has real anti-scraping protection, when your margin logic is per-category rather than flat, when you need conversion at your own rate rather than a static one, when the source category tree has to map onto yours, or when stock has to resynchronise every few hours. At that point the tooling costs more to bend than to replace.

Reading a supplier that also runs OpenCart

A quarter of the small distributors you will scrape run the same platform you are importing into. That is worth exploiting, and no other guide on this query mentions it.

Pagination has an off switch. The category controller in both 3.0.5.1 and 4.1.0.4 reads limit straight from the query string and passes it to the model with no upper bound:

if (isset($this->request->get['limit']) && (int)$this->request->get['limit']) {
    $limit = (int)$this->request->get['limit'];
} else {
    $limit = $this->config->get('config_pagination');
}

So index.php?route=product/category&path=20&limit=500 returns five hundred products in a single response. A 60-page crawl collapses into one request. Be decent about it: a limit of 500 makes the supplier's server build a large page, so use it once to enumerate, not once per minute.

Product URLs are predictable. Even on a store with SEO URLs enabled, index.php?route=product/product&product_id=1234 resolves. If you know the ID range, you do not need to discover links at all.

The sitemap feed may hand you the whole catalog. OpenCart 3 ships a Google Sitemap feed at index.php?route=extension/feed/google_sitemap, active whenever the feed_google_sitemap_status setting is on. Reading the controller source, it emits a <loc> for every product, a <lastmod> taken from product.date_modified, and the popup-sized image URL. That is a complete product index plus change dates, which turns an incremental sync into a diff instead of a crawl. OpenCart 4 renamed extension routes to the extension/<vendor>/<type>/<code> form, so probe extension/opencart/feed/google_sitemap there. Check for a plain /sitemap.xml first, and see crawling a sitemap for the general shape of that approach.

Check for structured data before you write a single CSS selector. Most modern OpenCart themes emit JSON-LD with Product and Offer objects. Schema.org is explicit about the format: "Use '.' (Unicode 'FULL STOP' (U+002E)) rather than ',' to indicate a decimal point. Avoid using these symbols as a readability separator." A price read from schema.org/Offer is already machine-formatted and carries priceCurrency next to it. Every locale problem in the next section disappears if the page gives you this and you take it.

Four probes, and they cover most OpenCart suppliers.

The price parser that lies quietly

Here is the code the earlier version of this article recommended, in both languages:

# Python, as previously published
digits = "".join(ch for ch in price_raw.text if ch.isdigit() or ch == ".")
price_source = float(digits) if digits else None

// PHP, as previously published
$digits = preg_replace('/[^0-9.]/', '', str_replace(',', '.', $priceRaw));
$priceSource = $digits !== '' ? (float)$digits : null;

We ran both against ten price strings copied from the formats real European and North American catalogs use, alongside a locale-aware replacement. The table is measured output, not a prediction:

Three of ten correct for the Python version, five of ten for the PHP one. Read the bold cells again: those are not crashes. They are numbers, of the right general shape, that go into oc_product.price and onto a live product page.

The failure that costs money is the thousandths one. At the EUR/USD rate of the day this article was checked, 1.154098, and a 25% markup, a correctly parsed 1234.56 becomes $1,781.00. The mis-parsed 1.23456 becomes $1.78. Exactly a factor of a thousand, on every German, Dutch, Spanish and Italian supplier page you touch. If any of those orders are fulfilled before somebody notices, the loss is not a bug report, it is inventory.

Two of the ValueErrors are worth knowing about too. 19.99 EUR incl. VAT fails because the full stop in incl. survives the filter, producing 19.99.. In the published loop that exception is swallowed by a broad except Exception, printed as "Error on ", and the product is skipped. Silent data loss instead of silent corruption, which is an improvement of degree only.

A parser that survives this decides the decimal separator by position rather than by guessing:

import re

def parse_price(text, decimal=None):
    """Parse a human-formatted price. `decimal` forces ',' or '.' when you know the locale."""
    s = text.replace(" ", " ").replace(" ", " ").replace("'", "'")
    m = re.search(r"\d[\d\s.,']*", s)
    if not m:
        return None
    tok = m.group(0).strip().replace(" ", "").replace("'", "")   # kill thousands spacers

    if decimal is None:
        last_dot, last_comma = tok.rfind("."), tok.rfind(",")
        if last_dot >= 0 and last_comma >= 0:
            decimal = "." if last_dot > last_comma else ","      # rightmost one wins
        elif last_dot >= 0 or last_comma >= 0:
            sep = "." if last_dot >= 0 else ","
            tail = len(tok) - tok.rfind(sep) - 1
            # a single separator with exactly three digits after it is a thousands mark
            decimal = sep if (tok.count(sep) == 1 and tail != 3) else None

    if decimal:
        tok = tok.replace("." if decimal == "," else ",", "").replace(decimal, ".")
    else:
        tok = tok.replace(".", "").replace(",", "")

    try:
        return float(tok)
    except ValueError:
        return None

Nine of ten. The one it still gets wrong is the last row, and no numeric heuristic can fix that one: Was $59.99 Now $39.99 needs a selector precise enough to reach the current price, or JSON-LD. When you know the supplier's locale, pass decimal="," and remove the guessing entirely.

Assert before you import. A scraper that has run correctly for six months will meet a supplier redesign eventually. Two cheap guards catch nearly all of it: reject any price outside a sane band for the category, and compare the new price against the last stored one, flagging anything that moved by more than an order of magnitude instead of writing it.

Currency: there is no such thing as a live rate

The earlier version of this guide converted "at the live rate". No free source gives you one.

The endpoint the code uses, open.er-api.com, is alive and still key-free. We called it on 13 August 2026 and it answered "result": "success", naming exchangerate-api.com as the provider. The response carries its own honesty in two fields: time_last_update_utc read Tue, 11 Aug 2026 23:57:32 +0000 and time_next_update_utc read Thu, 13 Aug 2026 00:25:22 +0000. That is a once-a-day refresh. The EUR to USD figure it returned was 1.154098.

Cross-check it against the European Central Bank's daily reference rates, which are free, need no key, and are published once per TARGET business day in the late afternoon Frankfurt time. Its file for 12 August 2026 gives USD at 1.1545. The two sources are 0.03% apart, far inside any sane markup, and neither is live. For a euro-denominated supplier the ECB file is the better source: it is a primary reference, it does not rate-limit you, and it is stable enough to cache for a day without thinking about it. Our notes on scraping exchange rates cover fallbacks when you need more than one currency pair.

You may not need conversion in the scraper at all. OpenCart carries a currency table with a value double(15,8) per currency and converts at display time. If your store's base currency matches the supplier's, store the supplier price untouched and let OpenCart present it. Conversion at import is only required when the base currencies differ, because product.price is always in the store's default currency.

Round late, and not to two places. oc_product.price is decimal(15,4). Rounding to two decimals during import throws away precision OpenCart is keeping for you, and it does so before tax and per-customer group discounts are applied. Compute in Decimal, store four places, let the storefront round for display.

Log the rate with the batch. The most useful table in an import pipeline is not in OpenCart at all:

CREATE TABLE import_source (
  product_id      INT NOT NULL,
  source_url      VARCHAR(512) NOT NULL,
  source_price    DECIMAL(15,4) NOT NULL,
  source_currency CHAR(3) NOT NULL,
  fx_rate         DECIMAL(18,8) NOT NULL,
  fx_source       VARCHAR(32) NOT NULL,
  markup          DECIMAL(6,4) NOT NULL,
  imported_at     DATETIME NOT NULL,
  PRIMARY KEY (product_id, imported_at)
) ENGINE=InnoDB;

Six months later, when somebody asks why a product costs what it costs, that table answers in one query. Without it you are re-deriving a rate from a date, which is guesswork.

Example: a custom scraper with import and currency conversion

A teaching-grade scraper in Python that crawls a supplier catalog, extracts the fields, converts the currency with a defined markup, and writes into OpenCart with the version differences handled.

This code is for learning. Before running it against a live store, always back up the database, confirm that scraping the source site does not violate its terms or the law, and test everything on a copy of the store first.

Stack and setup

pip install requests beautifulsoup4 pymysql lxml python-slugify

Written and run against Python 3.11, with the versions PyPI served on 13 August 2026: requests 2.34.2, beautifulsoup4 4.15.0, pymysql 1.2.0, lxml 6.1.1, python-slugify 8.0.4. Note that requests 2.34.2 requires Python 3.10 or newer, so a scraper inherited from an old server may need its interpreter moved first.

Step 1. The currency converter

The rate is fetched once a day, because the source updates once a day. Money is handled in Decimal end to end.

import requests
from datetime import date
from decimal import Decimal, ROUND_HALF_UP

class CurrencyConverter:
    """Fetches a daily FX rate and converts a supplier price into the store currency."""

    API_URL = "https://open.er-api.com/v6/latest/{base}"

    def __init__(self, source_currency="EUR", store_currency="USD", markup=Decimal("1.20")):
        # markup=1.20 -> the store adds 20% on top of the converted price
        self.source_currency = source_currency
        self.store_currency = store_currency
        self.markup = Decimal(markup)
        self._rate = None
        self._rate_date = None

    def _fetch_rate(self):
        """How many units of the store currency equal 1 unit of the source currency."""
        resp = requests.get(self.API_URL.format(base=self.source_currency), timeout=15)
        resp.raise_for_status()
        data = resp.json()
        if data.get("result") != "success":
            raise RuntimeError(f"FX API error: {data.get('error-type', 'unknown')}")
        rate = data["rates"].get(self.store_currency)
        if rate is None:
            raise ValueError(f"Currency {self.store_currency} not in FX response")
        # keep the timestamp: it is the difference between a rate and a guess
        self.rate_stamp = data.get("time_last_update_utc", "")
        return Decimal(str(rate))

    @property
    def rate(self):
        if self._rate is None or self._rate_date != date.today():
            self._rate = self._fetch_rate()
            self._rate_date = date.today()
        return self._rate

    def convert(self, amount):
        """Supplier price -> store price, four decimals, which is what the column holds."""
        if amount is None:
            return None
        price = Decimal(str(amount)) * self.rate * self.markup
        return price.quantize(Decimal("0.0001"), rounding=ROUND_HALF_UP)

Step 2. The supplier card parser

Structured data first, CSS selectors second. The selectors below are placeholders you replace per source site.

import json
import time
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

class SupplierParser:
    def __init__(self, base_url, converter, delay=1.0, locale_decimal=None):
        self.base_url = base_url
        self.converter = converter
        self.delay = delay              # pause between requests
        self.locale_decimal = locale_decimal   # "," for most of continental Europe
        self.session = requests.Session()
        self.session.headers.update({
            "User-Agent": "Mozilla/5.0 (compatible; CatalogImporter/1.0)"
        })

    def _get_soup(self, url):
        resp = self.session.get(url, timeout=20)
        resp.raise_for_status()
        return BeautifulSoup(resp.text, "lxml")

    def parse_catalog(self, catalog_path, max_pages=5, limit=None):
        """Collect product links. On an OpenCart source, pass limit=500 and stop after one page."""
        product_urls = []
        for page in range(1, max_pages + 1):
            url = urljoin(self.base_url, f"{catalog_path}?page={page}")
            if limit:
                url += f"&limit={limit}"
            soup = self._get_soup(url)
            cards = soup.select(".product-card a.product-link")
            if not cards:
                break
            for a in cards:
                product_urls.append(urljoin(self.base_url, a["href"]))
            time.sleep(self.delay)
        return product_urls

    def _json_ld_offer(self, soup):
        """Return (price, currency) from schema.org markup, which is always dot-decimal."""
        for tag in soup.find_all("script", type="application/ld+json"):
            try:
                data = json.loads(tag.string or "{}")
            except ValueError:
                continue
            for node in (data if isinstance(data, list) else [data]):
                if not isinstance(node, dict) or node.get("@type") != "Product":
                    continue
                offer = node.get("offers") or {}
                if isinstance(offer, list):
                    offer = offer[0] if offer else {}
                if offer.get("price") is not None:
                    return float(offer["price"]), offer.get("priceCurrency")
        return None, None

    def parse_product(self, url):
        soup = self._get_soup(url)

        name = soup.select_one("h1.product-title")
        description = soup.select_one(".product-description")
        sku = soup.select_one(".sku")
        image = soup.select_one(".product-gallery img")

        price_source, currency = self._json_ld_offer(soup)
        if price_source is None:
            price_raw = soup.select_one(".price .value")
            price_source = parse_price(price_raw.text, self.locale_decimal) if price_raw else None

        time.sleep(self.delay)

        return {
            "name": name.text.strip() if name else "Untitled",
            "sku": sku.text.strip() if sku else "",
            "description": description.decode_contents().strip() if description else "",
            "image_url": urljoin(self.base_url, image["src"]) if image else "",
            "price_source": price_source,
            "source_currency": currency or self.converter.source_currency,
            "price": self.converter.convert(price_source),
            "source_url": url,
        }

Step 3. Import into the OpenCart database

This is where the version matters. The importer takes a seo_mode and supplies every column that has no default, and it opens the session the way OpenCart opens its own.

import pymysql
from datetime import date, datetime
from decimal import Decimal
from slugify import slugify

# every column present in both 3.0.x and 4.1.x, in one list
PRODUCT_COLUMNS = (
    "model", "sku", "upc", "ean", "jan", "isbn", "mpn", "location",
    "quantity", "stock_status_id", "image", "manufacturer_id", "shipping",
    "price", "points", "tax_class_id", "date_available", "weight",
    "weight_class_id", "length", "width", "height", "length_class_id",
    "subtract", "minimum", "sort_order", "status", "date_added", "date_modified",
)

class OpenCartImporter:
    def __init__(self, db_config, prefix="oc_", store_id=0, language_id=1,
                 seo_mode="seo_url_query", sku_in_product_code=False):
        # seo_mode: "url_alias" (1.5, 2.x) | "seo_url_query" (3.x) | "seo_url_kv" (4.x)
        # sku_in_product_code: True from 4.1.0.1 onwards
        self.conn = pymysql.connect(**db_config, charset="utf8mb4",
                                    cursorclass=pymysql.cursors.DictCursor,
                                    autocommit=False)
        with self.conn.cursor() as cur:
            # the exact session mode OpenCart sets on every connection it opens
            cur.execute("SET SESSION sql_mode = 'NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION'")
        self.prefix = prefix
        self.store_id = store_id
        self.language_id = language_id
        self.seo_mode = seo_mode
        self.sku_in_product_code = sku_in_product_code

    def find_by_sku(self, sku):
        """Dedup key. On 4.1+ the SKU lives in product_code, not in product."""
        px = self.prefix
        with self.conn.cursor() as cur:
            if self.sku_in_product_code:
                cur.execute(f"SELECT product_id FROM {px}product_code "
                            f"WHERE code = 'sku' AND value = %s LIMIT 1", (sku,))
            else:
                cur.execute(f"SELECT product_id FROM {px}product "
                            f"WHERE sku = %s LIMIT 1", (sku,))
            row = cur.fetchone()
        return row["product_id"] if row else None

    def import_product(self, p, category_id=None, stock_status_id=7):
        px = self.prefix
        now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        cur = self.conn.cursor()

        values = (
            p["sku"] or p["name"][:64],  # model
            "" if self.sku_in_product_code else p["sku"],
            "", "", "", "", "", "",      # upc, ean, jan, isbn, mpn, location
            100,                         # quantity
            stock_status_id,             # 7 = In Stock in a default install; check oc_stock_status
            "",                          # image, filled in after the download
            0, 1,                        # manufacturer_id, shipping
            p["price"], 0, 0,            # price, points, tax_class_id
            date.today().isoformat(),    # date_available is a DATE column, and NULL hides the product
            Decimal("0"), 0,             # weight, weight_class_id
            Decimal("0"), Decimal("0"), Decimal("0"), 0,
            1, 1, 0, 1,                  # subtract, minimum, sort_order, status
            now, now,
        )
        placeholders = ", ".join(["%s"] * len(PRODUCT_COLUMNS))
        cur.execute(f"INSERT INTO {px}product ({', '.join(PRODUCT_COLUMNS)}) "
                    f"VALUES ({placeholders})", values)
        product_id = cur.lastrowid

        # description: tag and meta_keyword are NOT NULL with no default on 3.x
        cur.execute(f"""
            INSERT INTO {px}product_description
                (product_id, language_id, name, description, tag,
                 meta_title, meta_description, meta_keyword)
            VALUES (%s, %s, %s, %s, '', %s, %s, '')
        """, (product_id, self.language_id, p["name"], p["description"],
              p["name"][:255], p["name"][:255]))

        cur.execute(f"INSERT INTO {px}product_to_store (product_id, store_id) VALUES (%s, %s)",
                    (product_id, self.store_id))

        if category_id:
            cur.execute(f"INSERT INTO {px}product_to_category (product_id, category_id) "
                        f"VALUES (%s, %s)", (product_id, category_id))

        if self.sku_in_product_code and p["sku"]:
            cur.execute(f"INSERT INTO {px}product_code (product_id, code, value) "
                        f"VALUES (%s, 'sku', %s)", (product_id, p["sku"]))

        self._write_seo_url(cur, product_id, p["name"])
        self.conn.commit()
        return product_id

    def close(self):
        self.conn.close()

The SEO URL is the part that has to branch, and each branch matches what OpenCart's own admin model writes:

    def _write_seo_url(self, cur, product_id, name):
        px = self.prefix
        keyword = f"{slugify(name) or 'product'}-{product_id}"

        if self.seo_mode == "url_alias":            # OpenCart 1.5 and 2.x
            cur.execute(f"INSERT INTO {px}url_alias (query, keyword) VALUES (%s, %s)",
                        (f"product_id={product_id}", keyword))

        elif self.seo_mode == "seo_url_query":      # OpenCart 3.0.x
            cur.execute(f"""INSERT INTO {px}seo_url (store_id, language_id, query, keyword)
                            VALUES (%s, %s, %s, %s)""",
                        (self.store_id, self.language_id,
                         f"product_id={product_id}", keyword))

        elif self.seo_mode == "seo_url_kv":         # OpenCart 4.x
            cur.execute(f"""INSERT INTO {px}seo_url
                            (store_id, language_id, `key`, `value`, keyword, sort_order)
                            VALUES (%s, %s, 'product_id', %s, %s, 0)""",
                        (self.store_id, self.language_id, str(product_id), keyword))
        else:
            raise ValueError(f"unknown seo_mode {self.seo_mode!r}")

key and value are reserved words in MySQL, which is why they carry backticks. Leave them bare and you get a syntax error pointing at the wrong part of the statement.

Step 4. Wire it together

from decimal import Decimal

def main():
    converter = CurrencyConverter("EUR", "USD", markup=Decimal("1.25"))
    parser = SupplierParser("https://supplier-example.com", converter,
                            delay=1.0, locale_decimal=",")

    importer = OpenCartImporter(
        db_config={
            "host": "localhost",
            "user": "db_user",
            "password": "db_password",
            "database": "opencart_db",
        },
        prefix="oc_",
        store_id=0,
        language_id=1,
        seo_mode="seo_url_query",     # 3.0.x
        sku_in_product_code=False,    # True on 4.1.0.1 and later
    )

    product_urls = parser.parse_catalog("/catalog/category-1", max_pages=3)
    print(f"Products found: {len(product_urls)}")

    imported = skipped = 0
    for url in product_urls:
        try:
            product = parser.parse_product(url)
            if product["price"] is None:
                print(f"Skipped (no price): {url}")
                skipped += 1
                continue
            if product["sku"] and importer.find_by_sku(product["sku"]):
                print(f"Skipped (exists): {product['sku']}")
                skipped += 1
                continue
            pid = importer.import_product(product, category_id=20)
            imported += 1
            print(f"[{imported}] {product['name']} - "
                  f"{product['price_source']} {product['source_currency']} "
                  f"-> {product['price']} at {converter.rate} (ID {pid})")
        except Exception as e:
            print(f"Error on {url}: {type(e).__name__}: {e}")

    importer.close()
    print(f"Done. Imported: {imported}, skipped: {skipped}")

if __name__ == "__main__":
    main()

Printing the rate alongside every converted price costs one variable and saves the argument about where a number came from.

What breaks at ten thousand products

A run of fifty products tells you nothing about a run of ten thousand. Four things change shape.

Parsing stops being free. We measured the three usual ways of reading a catalog page on a single machine: a 184,076-byte page carrying 600 product cards, median of 25 runs, Python 3.11.15 on x86-64 Linux with beautifulsoup4 4.15.0 and lxml 6.1.1.

Measured on one machine, on one synthetic page shaped like a supplier catalog. Your absolute numbers will differ; the ratios hold. BeautifulSoup buys you readable selectors for roughly four times the CPU of raw lxml. On a nightly job that is fine. Inside a loop that also renders JavaScript, it is the difference between finishing before the store opens and not.

Politeness dominates the wall clock. The code above sleeps one second per product page and one per catalog page. Ten thousand products is about three hours of sleeping plus fetch time, single-threaded, before anything is parsed. That is the correct default. The lever to pull is the catalog side, where &limit=500 removes 95% of the listing requests, not the per-product delay.

Images are their own project. Suppliers routinely return 403 to a bare image request while serving the same file happily to a browser that sends a Referer from the product page. You need the referer, a real user agent, a size sanity check, and a deterministic filename so a re-run does not duplicate every photo. Then image/catalog/... on disk, the relative path into product.image, and one row per extra photo in oc_product_image. A headless browser is the fallback when the CDN insists on a session cookie.

Failure has to be resumable. OpenCart 3 is MyISAM, so commit() does nothing and a crash between the product row and the description row leaves an orphan. Keep your own progress table in InnoDB, write the source URL before you fetch it and the result after, and make the whole run restartable from the last completed URL. On OpenCart 4 the tables are InnoDB and a real transaction per product is available, which is the one place where 4.x is genuinely easier to import into.

Updates, not inserts, are the steady state. After the first load, every run should match on SKU and update price, quantity and date_modified rather than insert. That is what turns a scrape into synchronisation, and it is the difference between a script you run once and a pipeline you own. Remember which table the SKU lives in on your branch, or the match silently fails and you insert duplicates for the rest of the year.

Blocking arrives around here too. When a supplier starts returning 429s at page 400, rotating proxies and slower pacing beat cleverness, and the same import logic works whether the HTML came from your IP or someone else's.

The same scraper in PHP, on the store's own server

OpenCart is PHP on MySQL, so the scraper can live on the same hosting with no extra runtime. That buys you real things: nothing to install, credentials reusable from the store's own config.php, cron already available, and the option of writing through OpenCart's models instead of raw SQL.

Mind the PHP version. OpenCart 4's repository README carries a php >= 8.0 badge, and the code below needs 8.0 or newer as well, for constructor property promotion, str_starts_with and the nullsafe operator. Plenty of OpenCart 3 stores still run PHP 7.4, where this is a parse error rather than a runtime one. Check php.net's supported versions before you commit to a target: 8.0 and 8.1 are end of life, 8.2 is security-only until 31 December 2026, and 8.4 is supported into 2028.

For HTML parsing we use the built-in DOMDocument with DOMXPath, cURL for requests and PDO for the database. On PHP 8.4 and newer there is a better option: Dom\HTMLDocument::createFromString() parses to the HTML5 specification and handles encoding itself, which retires the <?xml encoding="UTF-8"> hack below.

Step 1. The currency converter

<?php
class CurrencyConverter
{
    private const API_URL = 'https://open.er-api.com/v6/latest/%s';

    private ?float $rate = null;
    private ?string $rateDate = null;
    public ?string $rateStamp = null;   // what the API says its rate is from

    public function __construct(
        private string $sourceCurrency = 'EUR',
        private string $storeCurrency = 'USD',
        private float $markup = 1.20
    ) {}

    private function fetchRate(): float
    {
        $json = file_get_contents(sprintf(self::API_URL, $this->sourceCurrency));
        $data = json_decode($json, true);

        if (($data['result'] ?? '') !== 'success') {
            throw new RuntimeException('FX API error: ' . ($data['error-type'] ?? 'unknown'));
        }
        $rate = $data['rates'][$this->storeCurrency] ?? null;
        if ($rate === null) {
            throw new RuntimeException("Currency {$this->storeCurrency} not in FX response");
        }
        $this->rateStamp = $data['time_last_update_utc'] ?? '';
        return (float)$rate;
    }

    public function getRate(): float
    {
        $today = date('Y-m-d');
        if ($this->rate === null || $this->rateDate !== $today) {
            $this->rate = $this->fetchRate();
            $this->rateDate = $today;
        }
        return $this->rate;
    }

    public function convert(?float $amount): ?string
    {
        if ($amount === null) {
            return null;
        }
        // four decimals, because oc_product.price is decimal(15,4)
        return number_format($amount * $this->getRate() * $this->markup, 4, '.', '');
    }
}

Step 2. Price parsing that matches the Python version

The published PHP one-liner turned $1,234.56 into 1.234. This is the locale-aware replacement:

<?php
function parse_price(string $text, ?string $decimal = null): ?float
{
    $s = str_replace(["\xc2\xa0", "\xe2\x80\xaf", "\xe2\x80\x99"], [' ', ' ', "'"], $text);
    if (!preg_match("/\d[\d\s.,']*/u", $s, $m)) {
        return null;
    }
    $tok = str_replace([' ', "'"], '', trim($m[0]));

    if ($decimal === null) {
        $lastDot   = strrpos($tok, '.');
        $lastComma = strrpos($tok, ',');
        if ($lastDot !== false && $lastComma !== false) {
            $decimal = $lastDot > $lastComma ? '.' : ',';
        } elseif ($lastDot !== false || $lastComma !== false) {
            $sep  = $lastDot !== false ? '.' : ',';
            $tail = strlen($tok) - strrpos($tok, $sep) - 1;
            $decimal = (substr_count($tok, $sep) === 1 && $tail !== 3) ? $sep : null;
        }
    }

    if ($decimal !== null) {
        $tok = str_replace($decimal === ',' ? '.' : ',', '', $tok);
        $tok = str_replace($decimal, '.', $tok);
    } else {
        $tok = str_replace(['.', ','], '', $tok);
    }

    return is_numeric($tok) ? (float)$tok : null;
}

is_numeric before the cast is the important half. PHP's float cast takes the leading numeric prefix and returns it without complaint, which is exactly how 59.9939.99 became 59.9939 in the table above.

Step 3. The supplier card parser

<?php
class SupplierParser
{
    public function __construct(
        private string $baseUrl,
        private CurrencyConverter $converter,
        private float $delay = 1.0,
        private ?string $localeDecimal = null
    ) {}

    private function getHtml(string $url): string
    {
        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_FOLLOWLOCATION => true,
            CURLOPT_TIMEOUT        => 20,
            CURLOPT_USERAGENT      => 'Mozilla/5.0 (compatible; CatalogImporter/1.0)',
        ]);
        $html = curl_exec($ch);
        if ($html === false) {
            throw new RuntimeException('cURL: ' . curl_error($ch));
        }
        curl_close($ch);
        return $html;
    }

    private function makeXPath(string $html): DOMXPath
    {
        $doc = new DOMDocument();
        libxml_use_internal_errors(true);   // silence messy-markup warnings
        $doc->loadHTML('<?xml encoding="UTF-8">' . $html);
        libxml_clear_errors();
        return new DOMXPath($doc);
    }

    private function absolute(string $href): string
    {
        return str_starts_with($href, 'http')
            ? $href
            : rtrim($this->baseUrl, '/') . '/' . ltrim($href, '/');
    }

    /** Collect product links. Pass $limit on an OpenCart source and one page is enough. */
    public function parseCatalog(string $catalogPath, int $maxPages = 5, ?int $limit = null): array
    {
        $urls = [];
        for ($page = 1; $page <= $maxPages; $page++) {
            $url = $this->absolute("{$catalogPath}?page={$page}") . ($limit ? "&limit={$limit}" : '');
            $xpath = $this->makeXPath($this->getHtml($url));
            $links = $xpath->query("//div[contains(@class,'product-card')]//a[contains(@class,'product-link')]");

            if ($links->length === 0) {
                break;
            }
            foreach ($links as $a) {
                $urls[] = $this->absolute($a->getAttribute('href'));
            }
            usleep((int)($this->delay * 1_000_000));
        }
        return $urls;
    }

    public function parseProduct(string $url): array
    {
        $xpath = $this->makeXPath($this->getHtml($url));
        $text = fn(string $q) => trim($xpath->query($q)->item(0)?->textContent ?? '');

        $name        = $text("//h1[contains(@class,'product-title')]") ?: 'Untitled';
        $sku         = $text("//*[contains(@class,'sku')]");
        $description = $text("//*[contains(@class,'product-description')]");
        $priceRaw    = $text("//*[contains(@class,'price')]//*[contains(@class,'value')]");

        $imgNode  = $xpath->query("//*[contains(@class,'product-gallery')]//img")->item(0);
        $imageUrl = $imgNode ? $this->absolute($imgNode->getAttribute('src')) : '';

        $priceSource = $priceRaw !== '' ? parse_price($priceRaw, $this->localeDecimal) : null;

        usleep((int)($this->delay * 1_000_000));

        return [
            'name'         => $name,
            'sku'          => $sku,
            'description'  => $description,
            'image_url'    => $imageUrl,
            'price_source' => $priceSource,
            'price'        => $this->converter->convert($priceSource),
            'source_url'   => $url,
        ];
    }
}

Step 4. Import, with the same version handling

<?php
class OpenCartImporter
{
    private PDO $pdo;

    public function __construct(
        array $dbConfig,
        private string $prefix = 'oc_',
        private int $storeId = 0,
        private int $languageId = 1,
        private string $seoMode = 'seo_url_query',   // url_alias | seo_url_query | seo_url_kv
        private bool $skuInProductCode = false       // true from 4.1.0.1
    ) {
        $dsn = "mysql:host={$dbConfig['host']};dbname={$dbConfig['database']};charset=utf8mb4";
        $this->pdo = new PDO($dsn, $dbConfig['user'], $dbConfig['password'], [
            PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
        ]);
        // match OpenCart's own session, or strict mode rejects the insert on 3.x
        $this->pdo->exec("SET SESSION sql_mode = 'NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION'");
    }

    private function seoKeyword(string $name): string
    {
        $slug = strtolower(trim($name));
        $slug = preg_replace('/[^a-z0-9\s-]/', '', $slug);
        $slug = preg_replace('/\s+/', '-', $slug);
        return $slug !== '' ? $slug : 'product';
    }

    public function importProduct(array $p, ?int $categoryId = null, int $stockStatusId = 7): int
    {
        $now   = date('Y-m-d H:i:s');
        $today = date('Y-m-d');
        $px    = $this->prefix;

        $stmt = $this->pdo->prepare("
            INSERT INTO {$px}product
                (model, sku, upc, ean, jan, isbn, mpn, location,
                 quantity, stock_status_id, image, manufacturer_id, shipping,
                 price, points, tax_class_id, date_available,
                 weight, weight_class_id, length, width, height, length_class_id,
                 subtract, minimum, sort_order, status, date_added, date_modified)
            VALUES (:model, :sku, '', '', '', '', '', '',
                    100, :stock, '', 0, 1,
                    :price, 0, 0, :available,
                    0, 0, 0, 0, 0, 0,
                    1, 1, 0, 1, :added, :modified)
        ");
        $stmt->execute([
            ':model'     => $p['sku'] ?: mb_substr($p['name'], 0, 64),
            ':sku'       => $this->skuInProductCode ? '' : $p['sku'],
            ':stock'     => $stockStatusId,
            ':price'     => $p['price'],
            ':available' => $today,       // never leave this NULL: the storefront filters on it
            ':added'     => $now,
            ':modified'  => $now,
        ]);
        $productId = (int)$this->pdo->lastInsertId();

        $this->pdo->prepare("
            INSERT INTO {$px}product_description
                (product_id, language_id, name, description, tag,
                 meta_title, meta_description, meta_keyword)
            VALUES (?, ?, ?, ?, '', ?, ?, '')
        ")->execute([
            $productId, $this->languageId, $p['name'], $p['description'],
            mb_substr($p['name'], 0, 255), mb_substr($p['name'], 0, 255),
        ]);

        $this->pdo->prepare("INSERT INTO {$px}product_to_store (product_id, store_id) VALUES (?, ?)")
                  ->execute([$productId, $this->storeId]);

        if ($categoryId !== null) {
            $this->pdo->prepare("INSERT INTO {$px}product_to_category (product_id, category_id) VALUES (?, ?)")
                      ->execute([$productId, $categoryId]);
        }

        if ($this->skuInProductCode && $p['sku'] !== '') {
            $this->pdo->prepare("INSERT INTO {$px}product_code (product_id, code, value) VALUES (?, 'sku', ?)")
                      ->execute([$productId, $p['sku']]);
        }

        $keyword = $this->seoKeyword($p['name']) . '-' . $productId;

        if ($this->seoMode === 'url_alias') {                 // 1.5 and 2.x
            $this->pdo->prepare("INSERT INTO {$px}url_alias (query, keyword) VALUES (?, ?)")
                      ->execute(["product_id={$productId}", $keyword]);
        } elseif ($this->seoMode === 'seo_url_query') {       // 3.0.x
            $this->pdo->prepare("INSERT INTO {$px}seo_url (store_id, language_id, query, keyword)
                                 VALUES (?, ?, ?, ?)")
                      ->execute([$this->storeId, $this->languageId, "product_id={$productId}", $keyword]);
        } else {                                             // 4.x
            $this->pdo->prepare("INSERT INTO {$px}seo_url
                                 (store_id, language_id, `key`, `value`, keyword, sort_order)
                                 VALUES (?, ?, 'product_id', ?, ?, 0)")
                      ->execute([$this->storeId, $this->languageId, (string)$productId, $keyword]);
        }

        return $productId;
    }
}

Step 5. Entry point and cron

<?php
require 'CurrencyConverter.php';
require 'SupplierParser.php';
require 'OpenCartImporter.php';

$converter = new CurrencyConverter('EUR', 'USD', 1.25);
$parser    = new SupplierParser('https://supplier-example.com', $converter, 1.0, ',');

$importer = new OpenCartImporter([
    'host'     => 'localhost',
    'user'     => 'db_user',
    'password' => 'db_password',
    'database' => 'opencart_db',
], prefix: 'oc_', storeId: 0, languageId: 1,
   seoMode: 'seo_url_query', skuInProductCode: false);

$urls = $parser->parseCatalog('/catalog/category-1', maxPages: 3);
echo 'Products found: ' . count($urls) . PHP_EOL;

$imported = 0;
foreach ($urls as $url) {
    try {
        $product = $parser->parseProduct($url);
        if ($product['price'] === null) {
            echo "Skipped (no price): {$url}" . PHP_EOL;
            continue;
        }
        $pid = $importer->importProduct($product, categoryId: 20);
        $imported++;
        printf("[%d] %s - %s EUR at %s -> %s (ID %d)%s",
            $imported, $product['name'], $product['price_source'],
            $converter->getRate(), $product['price'], $pid, PHP_EOL);
    } catch (Throwable $e) {
        echo "Error on {$url}: " . get_class($e) . ': ' . $e->getMessage() . PHP_EOL;
    }
}

echo "Done. Imported: {$imported}" . PHP_EOL;
echo "FX rate dated: {$converter->rateStamp}" . PHP_EOL;

Scheduling is one crontab line:

# sync the catalog with the supplier every day at 4 a.m.
0 4 * * * /usr/bin/php /var/www/opencart/parser/run.php >> /var/www/opencart/parser/parser.log 2>&1

You can pull database credentials straight from OpenCart: require '/path/to/opencart/config.php'; gives you DB_HOSTNAME, DB_USERNAME, DB_PASSWORD, DB_DATABASE and DB_PREFIX, so no password is duplicated in the scraper. On OpenCart 4 the same file also defines DB_DRIVER and the storefront and admin each carry their own config.

Everything from the "what breaks at scale" section applies here unchanged: images, deduplication, attributes and options, the category tree, cache clearing after a bulk load.

Where this approach stops working

Four boundaries, and knowing them in advance is worth more than any of the code above.

The catalog is rendered client-side. If the product grid arrives as JSON and is painted by JavaScript, DOMDocument and BeautifulSoup see an empty shell. The right move is usually to call the JSON endpoint the page itself calls, which is faster and more stable than any DOM. Failing that, a headless browser, at roughly ten to thirty times the cost per page.

Prices live behind a login. Distributor pricing is routinely per-account, and the public number is a list price you must not import. Scraping your own authenticated session is a contractual question, not a technical one: read the reseller agreement before you automate it. This is the most common reason a supplier import turns into a manual CSV exchange instead.

The supplier fights back. Rate limiting, bot detection and IP bans arrive in that order. Slower crawls and honest pacing solve most of it; the rest is proxy rotation and session management. Sites that fight back seriously are where a managed extraction service earns its keep, because the maintenance load, not the code, is what makes these projects expensive.

The data is not yours to republish. Product photographs and marketing descriptions are copyrighted works, usually owned by the manufacturer rather than the distributor you scraped them from. A supplier agreement that permits reselling goods does not automatically permit republishing their catalog text. Import specifications and prices, which are facts, and either license the descriptions or write your own. Also honour robots.txt and keep the request rate at a level you would be comfortable explaining.

Two limits on what we could verify for this article. OpenCart's marketplace listings show a headline price but not a licence term, so the "per domain" language circulating about paid import modules could not be confirmed from the vendor pages. And ParseHub's pricing page serves no content to a non-browser client, so its current tiers are unknown to us rather than unchanged.

The bottom line

For a standard store, ready-made still wins: the free Export/Import Tool, or a $20 to $99 module from the marketplace, fed by a CSV that Octoparse or the Web Scraper extension produced. You will spend an afternoon and no maintenance.

Write your own when the source needs judgment rather than selectors: irregular markup, per-category margins, conversion at a rate you can defend six months later, stock resynchronising several times a day, or a store whose OpenCart version the modules do not cover. The code is not the hard part. The hard part is the four silent failures this article exists to name: the wrong SEO table, the strict-mode INSERT, the NULL date_available that hides everything you imported, and the price parser that turns 1.234,56 into 1.23. Get those right and the rest is plumbing.

The same reasoning transfers to other platforms with an accessible schema, and the Bitrix product scraper guide covers that side. If running the crawler is the part you would rather not own, a scheduled data feed delivers the catalog and leaves the import to you.