Filling a store by hand is the most expensive and tedious part of any launch. When a supplier has thousands of SKUs and prices change every week, automation stops being optional. That is where an OpenCart product scraper comes in: a program that collects product data from a source site (a supplier, a distributor, a marketplace) and loads it into your store, complete with names, descriptions, photos, specifications and, crucially, correctly recalculated prices.
In this guide we will look at how OpenCart versions differ from a scraping point of view, what ready-made solutions exist, and then build a custom OpenCart parser in Python that collects products from a supplier site and converts the currency on the fly.
Why the OpenCart version matters
A scraper ultimately writes data into the database or generates an import file, and both the database schema and the SEO-URL logic differ between OpenCart releases. Ignore this and you are guaranteed either broken products or import errors.
OpenCart first appeared back in 2005 and has been through several major reworks since: version 1.5 (2011) brought a new layout editor and admin panel, the 2.x branch (2014) added a modular architecture and a better extension system, version 3.0 (2017) introduced multi-store and the Marketplace, and the 4.x branch modernized the codebase and brought OCMOD back. Every one of these branches is still running in live projects today.
Here is what a scraper needs to account for:
OpenCart 1.5.x is a legacy branch that still turns up. The table structure is simple, but several tables use different field names, and almost no modern import modules support it.
OpenCart 2.x is the most common "old but working" branch. Many of the battle-tested import modules live here. SEO URLs are stored in the url_alias table.
OpenCart 3.x is the most popular version for production stores at the time of writing. Its structure is close to 2.x, and SEO URLs are also kept in url_alias. Most ready-made scrapers and modules target 2.x-3.x specifically.
OpenCart 4.x is the current branch. It brings one fundamental change: SEO URLs have moved out of url_alias into a seo_url table (keyed by store_id and language_id), and some controllers changed too. A scraper written for OC3 will not write correct SEO-friendly URLs on OC4 without edits.
Forks and distributions (community rebuilds of OpenCart) share the same data structure as the OpenCart version they are based on, but their bundled modules sometimes behave differently, so always test them separately.
Practical takeaway: before you write or buy a scraper, pin down the exact store version (shown at the bottom of the admin panel or in the core
index.php) and the table prefix in the database (oc_by default, but often customized).
Ready-made solutions: what the market offers
If the budget is tight and the store is fairly standard, you may not need to write your own scraper at all. There are plenty of off-the-shelf tools, and they fall into a few categories.
Import modules inside OpenCart
These extensions do not crawl third-party sites themselves, but they take a prepared file (CSV / XLS / XLSX / XML) and load it into the catalog:
- Export/Import Tool is a popular, low-cost module that comfortably handles imports of several thousand products (limits depend on your hosting). A solid starting point.
- CSV Price Pro import/export is a paid module (around $20 per domain) that is more flexible about mapping fields.
- Advanced CSV/XML import modules on the OpenCart marketplace add scheduling, incremental updates and richer field mapping for OC 2-4.
The pattern "the scraper builds a file, the import module loads it" is the most reliable one: you get to eyeball the data before uploading and you never touch the database directly.
General-purpose scrapers that export to a file
These are the tools that actually crawl the source sites and produce a clean CSV/Excel you then feed to an import module:
- Octoparse and ParseHub are visual, no-code cloud scrapers with point-and-click selectors, pagination handling and scheduled runs.
- WebHarvy and the Web Scraper browser extension are lightweight desktop/browser options for smaller catalogs.
- Managed data-as-a-service providers deliver a ready feed on a schedule so you never run the crawler yourself.
When does a ready-made solution fall short? When the supplier has non-trivial markup or anti-scraping protection, when you need specific margin logic, currency conversion at your own rate, mapping to your own category tree, or regular stock synchronization. In those cases you build a scraper for the job, which is what we do next.
Example: a custom scraper with import and currency conversion
Let us build a teaching-grade scraper in Python that:
- crawls the supplier catalog and collects product cards;
- extracts the name, price, description, SKU and image;
- converts the price from the supplier's currency into the store's currency at the live rate with a defined markup;
- imports the products straight into the OpenCart database.
This code is for learning. Before running it against a live store, always back up the database, confirm that scraping the source site does not violate its terms or the law, and test everything on a copy of the store first.
Stack and setup
pip install requests beautifulsoup4 pymysql lxml python-slugify
requests+beautifulsoup4to download and parse HTML;pymysqlto write into the OpenCart database;- exchange rates come from a free public API (this example uses
open.er-api.com, which needs no key; swap in any provider you like). See our notes on scraping exchange rates if you need a fallback source.
Step 1. The currency converter
First, a standalone module that fetches the rate and recalculates the price with a markup. The rate is cached so we do not hit the API for every product.
import requests
from datetime import date
class CurrencyConverter:
"""Fetches live FX rates and converts a supplier price into the store currency."""
API_URL = "https://open.er-api.com/v6/latest/{base}"
def __init__(self, source_currency="EUR", store_currency="USD", markup=1.20):
# markup=1.20 -> the store adds a 20% margin on top of the converted price
self.source_currency = source_currency
self.store_currency = store_currency
self.markup = markup
self._rate = None
self._rate_date = None
def _fetch_rate(self):
"""How many units of the store currency equal 1 unit of the source currency."""
resp = requests.get(self.API_URL.format(base=self.source_currency), timeout=15)
resp.raise_for_status()
data = resp.json()
if data.get("result") != "success":
raise RuntimeError(f"FX API error: {data.get('error-type', 'unknown')}")
rate = data["rates"].get(self.store_currency)
if rate is None:
raise ValueError(f"Currency {self.store_currency} not found in FX response")
return float(rate)
@property
def rate(self):
# cache the rate for the current day
if self._rate is None or self._rate_date != date.today():
self._rate = self._fetch_rate()
self._rate_date = date.today()
return self._rate
def convert(self, amount):
"""Supplier price -> store price (with the store margin applied)."""
if amount is None:
return None
price = float(amount) * self.rate * self.markup
return round(price, 2)
Step 2. The supplier card parser
The selectors (.product-card, .price and so on) are placeholders: for each source site you pick them individually by opening the page source.
import time
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
class SupplierParser:
def __init__(self, base_url, converter, delay=1.0):
self.base_url = base_url
self.converter = converter
self.delay = delay # pause between requests so we do not hammer the source
self.session = requests.Session()
self.session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; CatalogImporter/1.0)"
})
def _get_soup(self, url):
resp = self.session.get(url, timeout=20)
resp.raise_for_status()
return BeautifulSoup(resp.text, "lxml")
def parse_catalog(self, catalog_path, max_pages=5):
"""Collect product links across paginated catalog pages."""
product_urls = []
for page in range(1, max_pages + 1):
url = urljoin(self.base_url, f"{catalog_path}?page={page}")
soup = self._get_soup(url)
cards = soup.select(".product-card a.product-link")
if not cards:
break # ran out of pages
for a in cards:
product_urls.append(urljoin(self.base_url, a["href"]))
time.sleep(self.delay)
return product_urls
def parse_product(self, url):
"""Parse a single product card."""
soup = self._get_soup(url)
name = soup.select_one("h1.product-title")
price_raw = soup.select_one(".price .value")
description = soup.select_one(".product-description")
sku = soup.select_one(".sku")
image = soup.select_one(".product-gallery img")
# strip the price of spaces and currency symbols
price_source = None
if price_raw:
digits = "".join(ch for ch in price_raw.text if ch.isdigit() or ch == ".")
price_source = float(digits) if digits else None
time.sleep(self.delay)
return {
"name": name.text.strip() if name else "Untitled",
"sku": sku.text.strip() if sku else "",
"description": description.decode_contents().strip() if description else "",
"image_url": urljoin(self.base_url, image["src"]) if image else "",
"price_source": price_source,
# this is where the currency conversion happens:
"price": self.converter.convert(price_source),
"source_url": url,
}
Step 3. Import into the OpenCart database
Here we apply what we discussed at the start: the structure depends on the version. This example targets OpenCart 3.x; the key OC4 difference is called out in a comment.
import pymysql
from datetime import datetime
from slugify import slugify # pip install python-slugify
class OpenCartImporter:
def __init__(self, db_config, prefix="oc_", store_id=0, language_id=1):
self.conn = pymysql.connect(**db_config, charset="utf8mb4",
cursorclass=pymysql.cursors.DictCursor)
self.prefix = prefix
self.store_id = store_id
self.language_id = language_id
def _seo_keyword(self, name):
return slugify(name) or "product"
def import_product(self, p, category_id=None):
cur = self.conn.cursor()
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
px = self.prefix
# 1) main product table
cur.execute(f"""
INSERT INTO {px}product
(model, sku, quantity, price, image, status,
date_added, date_modified, date_available)
VALUES (%s, %s, %s, %s, %s, 1, %s, %s, %s)
""", (p["sku"] or p["name"][:64], p["sku"], 100,
p["price"], "", now, now, now))
product_id = cur.lastrowid
# 2) description (name, text, meta) - per language
cur.execute(f"""
INSERT INTO {px}product_description
(product_id, language_id, name, description,
meta_title, meta_description)
VALUES (%s, %s, %s, %s, %s, %s)
""", (product_id, self.language_id, p["name"],
p["description"], p["name"], p["name"]))
# 3) link to the store
cur.execute(f"""
INSERT INTO {px}product_to_store (product_id, store_id)
VALUES (%s, %s)
""", (product_id, self.store_id))
# 4) link to a category (if provided)
if category_id:
cur.execute(f"""
INSERT INTO {px}product_to_category (product_id, category_id)
VALUES (%s, %s)
""", (product_id, category_id))
# 5) SEO URL
keyword = f"{self._seo_keyword(p['name'])}-{product_id}"
# --- OpenCart 3.x: url_alias table ---
cur.execute(f"""
INSERT INTO {px}url_alias (query, keyword)
VALUES (%s, %s)
""", (f"product_id={product_id}", keyword))
# --- OpenCart 4.x would be (seo_url table):
# cur.execute(f'''INSERT INTO {px}seo_url
# (store_id, language_id, `key`, `value`, keyword)
# VALUES (%s, %s, "product_id", %s, %s)''',
# (self.store_id, self.language_id, product_id, keyword))
self.conn.commit()
return product_id
def close(self):
self.conn.close()
Step 4. Wire it all together
def main():
converter = CurrencyConverter(source_currency="EUR", store_currency="USD", markup=1.25)
parser = SupplierParser("https://supplier-example.com", converter, delay=1.0)
importer = OpenCartImporter(
db_config={
"host": "localhost",
"user": "db_user",
"password": "db_password",
"database": "opencart_db",
},
prefix="oc_",
store_id=0,
language_id=1,
)
product_urls = parser.parse_catalog("/catalog/category-1", max_pages=3)
print(f"Products found: {len(product_urls)}")
imported = 0
for url in product_urls:
try:
product = parser.parse_product(url)
if product["price"] is None:
print(f"Skipped (no price): {url}")
continue
pid = importer.import_product(product, category_id=20)
imported += 1
print(f"[{imported}] {product['name']} - "
f"{product['price_source']} {converter.source_currency} "
f"-> ${product['price']} (ID {pid})")
except Exception as e:
print(f"Error on {url}: {e}")
importer.close()
print(f"Done. Imported: {imported}")
if __name__ == "__main__":
main()
What was left out of scope but matters in a real project
The teaching example is deliberately simplified. A production scraper also needs to handle:
- Downloading and linking images - save the photo into
image/catalog/...and write the path intoproduct.imageandoc_product_image. Source sites often return 403 on direct downloads, so you need a correct referer and headers. A headless browser sometimes helps here. - Deduplication - check by
sku/modelwhether the product already exists and update the price/stock instead of creating duplicates. This is also what turns a one-off scrape into ongoing synchronization. - Attributes and options -
oc_product_attribute,oc_product_optionwith their own relationship structure. - Categories - auto-building the category tree from the source structure (
oc_category,oc_category_description,oc_category_path). - Cache clearing and rebuilding SEO indexes after a bulk import.
- Rate and markup - keep them in config and log which rate each batch was converted at, so prices are reproducible and transparent.
- Ethics and legality - respect
robots.txt, use reasonable delays, and honor the source site's terms. When a site fights back, rotating proxies and careful pacing keep you unblocked.
The same scraper in PHP - right on the store's server
Python is convenient, but it needs its own environment. OpenCart itself is written in PHP and runs on MySQL, which means the scraper can be built on the same stack and run directly on the store's hosting, with no Python or extra runtimes. The upsides:
- nothing extra to install - PHP is already on the server;
- you can reuse OpenCart's
config.phpso you do not duplicate database credentials; - it is easy to schedule with cron for regular synchronization;
- if you want, the script can hook into the OpenCart core and write products through its models rather than straight into tables.
For HTML parsing we use the built-in DOMDocument + DOMXPath (no third-party libraries), cURL for requests, and PDO for the database.
Step 1. The currency converter
<?php
class CurrencyConverter
{
private const API_URL = 'https://open.er-api.com/v6/latest/%s';
private ?float $rate = null;
private ?string $rateDate = null;
public function __construct(
private string $sourceCurrency = 'EUR',
private string $storeCurrency = 'USD',
private float $markup = 1.20 // store margin: +20% on top of the rate
) {}
private function fetchRate(): float
{
$json = file_get_contents(sprintf(self::API_URL, $this->sourceCurrency));
$data = json_decode($json, true);
if (($data['result'] ?? '') !== 'success') {
throw new RuntimeException('FX API error: ' . ($data['error-type'] ?? 'unknown'));
}
$rate = $data['rates'][$this->storeCurrency] ?? null;
if ($rate === null) {
throw new RuntimeException("Currency {$this->storeCurrency} not found in FX response");
}
return (float)$rate;
}
public function getRate(): float
{
$today = date('Y-m-d');
if ($this->rate === null || $this->rateDate !== $today) {
$this->rate = $this->fetchRate();
$this->rateDate = $today;
}
return $this->rate;
}
public function convert(?float $amount): ?float
{
if ($amount === null) {
return null;
}
return round($amount * $this->getRate() * $this->markup, 2);
}
}
Step 2. The supplier card parser
<?php
class SupplierParser
{
public function __construct(
private string $baseUrl,
private CurrencyConverter $converter,
private float $delay = 1.0 // pause between requests, seconds
) {}
private function getHtml(string $url): string
{
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_TIMEOUT => 20,
CURLOPT_USERAGENT => 'Mozilla/5.0 (compatible; CatalogImporter/1.0)',
]);
$html = curl_exec($ch);
if ($html === false) {
throw new RuntimeException('cURL: ' . curl_error($ch));
}
curl_close($ch);
return $html;
}
private function makeXPath(string $html): DOMXPath
{
$doc = new DOMDocument();
libxml_use_internal_errors(true); // silence messy-markup warnings
$doc->loadHTML('<?xml encoding="UTF-8">' . $html);
libxml_clear_errors();
return new DOMXPath($doc);
}
private function absolute(string $href): string
{
return str_starts_with($href, 'http') ? $href : rtrim($this->baseUrl, '/') . '/' . ltrim($href, '/');
}
/** Collect product links across paginated catalog pages. */
public function parseCatalog(string $catalogPath, int $maxPages = 5): array
{
$urls = [];
for ($page = 1; $page <= $maxPages; $page++) {
$html = $this->getHtml($this->absolute("{$catalogPath}?page={$page}"));
$xpath = $this->makeXPath($html);
$links = $xpath->query("//div[contains(@class,'product-card')]//a[contains(@class,'product-link')]");
if ($links->length === 0) {
break; // ran out of pages
}
foreach ($links as $a) {
$urls[] = $this->absolute($a->getAttribute('href'));
}
usleep((int)($this->delay * 1_000_000));
}
return $urls;
}
/** Parse a single product card. */
public function parseProduct(string $url): array
{
$xpath = $this->makeXPath($this->getHtml($url));
$text = fn(string $q) => trim($xpath->query($q)->item(0)?->textContent ?? '');
$name = $text("//h1[contains(@class,'product-title')]") ?: 'Untitled';
$sku = $text("//*[contains(@class,'sku')]");
$description = $text("//*[contains(@class,'product-description')]");
$priceRaw = $text("//*[contains(@class,'price')]//*[contains(@class,'value')]");
$imgNode = $xpath->query("//*[contains(@class,'product-gallery')]//img")->item(0);
$imageUrl = $imgNode ? $this->absolute($imgNode->getAttribute('src')) : '';
// strip the price of spaces and currency symbols
$priceSource = null;
if ($priceRaw !== '') {
$digits = preg_replace('/[^0-9.]/', '', str_replace(',', '.', $priceRaw));
$priceSource = $digits !== '' ? (float)$digits : null;
}
usleep((int)($this->delay * 1_000_000));
return [
'name' => $name,
'sku' => $sku,
'description' => $description,
'image_url' => $imageUrl,
'price_source' => $priceSource,
// currency conversion:
'price' => $this->converter->convert($priceSource),
'source_url' => $url,
];
}
}
Step 3. Import into the OpenCart database
<?php
class OpenCartImporter
{
private PDO $pdo;
public function __construct(
array $dbConfig,
private string $prefix = 'oc_',
private int $storeId = 0,
private int $languageId = 1
) {
$dsn = "mysql:host={$dbConfig['host']};dbname={$dbConfig['database']};charset=utf8mb4";
$this->pdo = new PDO($dsn, $dbConfig['user'], $dbConfig['password'], [
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
]);
}
private function seoKeyword(string $name): string
{
// simple ASCII slug for the SEO-friendly URL
$slug = strtolower(trim($name));
$slug = preg_replace('/[^a-z0-9\s-]/', '', $slug);
$slug = preg_replace('/\s+/', '-', $slug);
return $slug !== '' ? $slug : 'product';
}
public function importProduct(array $p, ?int $categoryId = null): int
{
$px = $this->prefix;
$now = date('Y-m-d H:i:s');
// 1) main product table
$stmt = $this->pdo->prepare("
INSERT INTO {$px}product
(model, sku, quantity, price, image, status,
date_added, date_modified, date_available)
VALUES (:model, :sku, 100, :price, '', 1, :added, :modified, :available)
");
$stmt->execute([
':model' => $p['sku'] ?: mb_substr($p['name'], 0, 64),
':sku' => $p['sku'],
':price' => $p['price'],
':added' => $now,
':modified' => $now,
':available' => $now,
]);
$productId = (int)$this->pdo->lastInsertId();
// 2) description (name, text, meta)
$this->pdo->prepare("
INSERT INTO {$px}product_description
(product_id, language_id, name, description, meta_title, meta_description)
VALUES (?, ?, ?, ?, ?, ?)
")->execute([
$productId, $this->languageId, $p['name'],
$p['description'], $p['name'], $p['name'],
]);
// 3) link to the store
$this->pdo->prepare("
INSERT INTO {$px}product_to_store (product_id, store_id) VALUES (?, ?)
")->execute([$productId, $this->storeId]);
// 4) link to a category
if ($categoryId !== null) {
$this->pdo->prepare("
INSERT INTO {$px}product_to_category (product_id, category_id) VALUES (?, ?)
")->execute([$productId, $categoryId]);
}
// 5) SEO URL
$keyword = $this->seoKeyword($p['name']) . '-' . $productId;
// --- OpenCart 3.x: url_alias table ---
$this->pdo->prepare("
INSERT INTO {$px}url_alias (query, keyword) VALUES (?, ?)
")->execute(["product_id={$productId}", $keyword]);
// --- OpenCart 4.x would be (seo_url table):
// INSERT INTO {$px}seo_url (store_id, language_id, `key`, `value`, keyword)
// VALUES (:store, :lang, 'product_id', :pid, :keyword)
return $productId;
}
}
Step 4. Entry point and cron scheduling
<?php
require 'CurrencyConverter.php';
require 'SupplierParser.php';
require 'OpenCartImporter.php';
$converter = new CurrencyConverter('EUR', 'USD', 1.25);
$parser = new SupplierParser('https://supplier-example.com', $converter, 1.0);
// credentials can live in a config or be pulled from OpenCart's config.php
$importer = new OpenCartImporter([
'host' => 'localhost',
'user' => 'db_user',
'password' => 'db_password',
'database' => 'opencart_db',
], prefix: 'oc_', storeId: 0, languageId: 1);
$urls = $parser->parseCatalog('/catalog/category-1', maxPages: 3);
echo 'Products found: ' . count($urls) . PHP_EOL;
$imported = 0;
foreach ($urls as $url) {
try {
$product = $parser->parseProduct($url);
if ($product['price'] === null) {
echo "Skipped (no price): {$url}" . PHP_EOL;
continue;
}
$pid = $importer->importProduct($product, categoryId: 20);
$imported++;
printf("[%d] %s - %s EUR -> \$%s (ID %d)%s",
$imported, $product['name'], $product['price_source'], $product['price'], $pid, PHP_EOL);
} catch (Throwable $e) {
echo "Error on {$url}: {$e->getMessage()}" . PHP_EOL;
}
}
echo "Done. Imported: {$imported}" . PHP_EOL;
Setting it to run daily is a single crontab line:
# sync the catalog with the supplier every day at 4 a.m.
0 4 * * * /usr/bin/php /var/www/opencart/parser/run.php >> /var/www/opencart/parser/parser.log 2>&1
You can pull database credentials straight from OpenCart like this:
require '/path/to/opencart/config.php';- after that the constantsDB_HOSTNAME,DB_USERNAME,DB_PASSWORD,DB_DATABASEandDB_PREFIXbecome available, and you never duplicate passwords in the scraper.
All the "out of scope" caveats (images, deduplication, attributes, cache, legality) apply equally to the PHP version.
The bottom line
For a typical store, the combination "ready-made scraper + import module" is often enough: Export/Import Tool, CSV Price Pro, Octoparse or ParseHub feeding a CSV will cover most scenarios. But the moment you get a non-standard source, your own margin logic, currency conversion at the live rate, or regular stock synchronization tied to a specific OpenCart version, a purpose-built scraper is both cheaper and more reliable in the long run.
If you need to fill or synchronize an OpenCart store (or any other platform), pull data from supplier sites, and set up automatic price updates with currency conversion, our team can design and deploy it as a done-for-you data extraction service built for your exact OpenCart version and workflow. Prefer to skip the infrastructure entirely? We can also deliver the catalog as a scheduled data feed.