Filling a catalog by hand is slow and expensive, especially when a supplier carries thousands of SKUs and their prices and stock levels change every week. The fix is a 1C-Bitrix product scraper: a program that collects products from a supplier's site (or their price file) and loads them into your store with names, descriptions, photos, properties and — critically — correctly recalculated prices.
This guide breaks down how Bitrix editions differ from a scraping standpoint, which ready-made solutions exist, and how to write your own product scraper that collects supplier data and converts currency in the same pass — using the currency module built into Bitrix itself.
Why the Bitrix edition and architecture matter
Unlike many content-management systems, in Bitrix you should never write products straight into the database. The storage model is intricate — information blocks, the commercial catalog, price types, warehouses, trade offers — and a raw INSERT will almost certainly break indexes and business logic. The correct path is to work through the Bitrix API. So for a scraper, what matters is less the "kernel version" and more the product edition and which modules ship with it.
1C-Bitrix: Site Management comes in five main editions — Start, Standard, Small Business, Business and Enterprise. A full-blown online store and the modules a scraper needs first appear in Small Business.
Start and Standard are for content sites. There is no Commercial Catalog module in the base package, so you can't assemble a real store without custom work. There is nowhere to scrape catalog products into.
Small Business is the first "store" edition. It includes the Online Store and Commercial Catalog modules and supports integration with ERP/accounting systems and marketplace feeds (Google Shopping, Amazon and the like). It works with a single price type and a single warehouse — enough for a mid-sized store scraper.
Business is the extended store: multiple price types (wholesale/retail-data-services/), multi-warehouse stock, cumulative discounts, sets and bundles. If you scrape both wholesale and retail prices, or split stock across warehouses, the scraper has to be able to write that — which means you specifically need Business.
Enterprise targets large, high-load projects. The logic for a scraper is the same as Business, but with added performance requirements for mass imports.
Worth calling out separately: Bitrix24: Online Store + CRM is a different product (the store is tied to CRM entities and the CRM catalog). A scraper for it is written differently than one for Site Management — keep that in mind if you're actually on Bitrix24.
A few more things a scraper must understand about Bitrix:
- Information blocks (infoblocks) — products are stored as elements of the catalog infoblock; infoblock sections are your categories.
- Trade offers (SKUs / offers) — product variants (color, size) live in a separate linked infoblock. If your supplier has variations, the scraper has to create both the product and its trade offers.
- Currency module — Bitrix has a built-in currency module that can auto-pull exchange rates on a schedule via an agent. That's handy: you can do conversion with Bitrix's own tools instead of inventing your own rate source.
- CommerceML — the standard XML exchange format for syncing with ERP/accounting systems. The stock catalog import runs on it too.
Practical takeaway: before you build anything, pin down the edition, the catalog infoblock ID, the property schema, and whether trade offers and multiple price types are in use. All of your import logic depends on it.
Ready-made solutions: what's on the market
If your store is standard, you may not need to write anything at all.
Bitrix's native import tools
Small Business and Business ship with catalog import out of the box:
- CommerceML import — the primary sync channel with ERP/accounting: products, prices, stock, properties. If your supplier can emit CommerceML, this is the most "native" route.
- CSV import — under Commercial Catalog → Export/Import. It accepts a prepared CSV with products, prices and properties.
On their own, these tools don't crawl third-party sites — they need a ready file. So they're usually paired with a scraper: the scraper builds the CSV/XML, the native import loads it. This is the safest option for the kernel.
Modules from the Bitrix Marketplace
These are extensions that both scrape and load the result into infoblocks right from the admin panel:
- Bulk product import / site scraper modules — parse sites straight from the admin and add products to an infoblock: import from all categories, auto-creation of sections and properties, multithreading, CSV import. Many vendors also offer a supplier-API connection.
- Content scraper modules (e.g. from vendors like Sotbit) — integrate supplier catalogs into your store from a catalog site, Excel, XML, YML or CSV; load into infoblocks and highload blocks, work through proxies, log everything, and run in a limited demo mode.
Tip: most such modules have a demo mode (usually a few products/pages). Always test it on your specific donor site before buying — plenty of sites don't scrape "out of the box."
External programs
- Desktop scraping suites (Datacol and similar) ship with dozens of ready presets for popular stores. They collect data from sites, price files and marketplaces, process it (translation, uniqueness, markup) and load it into Bitrix semi-automatically (via Excel + native import) or fully automatically.
When do ready-made tools fall short? Non-standard markup or anti-bot protection on the donor, your own markup logic, currency conversion at your own rate, mapping to your property and trade-offer schema, regular stock synchronization. That's when people build a scraper for the specific task — which is what we'll do next.
Example 1: a PHP scraper via the Bitrix API (with currency conversion)
Because Bitrix is written in PHP, the most correct scraper is a PHP script that bootstraps the Bitrix kernel and writes products through its API. We'll do currency conversion with the built-in currency module and apply the markup ourselves.
This code is for learning. Before running against a live store, back it up, confirm that scraping the donor doesn't violate its terms or the law, and test on a copy.
Step 0. Loading the kernel and modules
The script runs from the console or via cron. It bootstraps Bitrix and loads the modules it needs.
<?php
// so the script works from console/cron, not only in a web context
$_SERVER['DOCUMENT_ROOT'] = '/var/www/bitrix';
define('NO_KEEP_STATISTIC', true);
define('NOT_CHECK_PERMISSIONS', true);
define('BX_NO_ACCELERATOR_RESET', true);
require $_SERVER['DOCUMENT_ROOT'] . '/bitrix/modules/main/include/prolog_before.php';
use Bitrix\Main\Loader;
Loader::includeModule('iblock'); // infoblocks (catalog)
Loader::includeModule('catalog'); // commercial catalog (prices, stock)
Loader::includeModule('currency'); // currency module (exchange rates)
Step 1. A currency converter on the built-in module
Bitrix already stores and (via an agent) refreshes exchange rates. We take the rate from the currency module through CCurrencyRates::ConvertCurrency and add the markup separately. Here the supplier quotes in EUR and we sell in USD.
<?php
class CurrencyConverter
{
public function __construct(
private string $from = 'EUR',
private string $to = 'USD',
private float $markup = 1.20 // store markup: +20%
) {}
/** Supplier price -> store price at the Bitrix rate + markup. */
public function convert(?float $amount): ?float
{
if ($amount === null) {
return null;
}
// ConvertCurrency pulls the current rate from the currency module
$converted = \CCurrencyRates::ConvertCurrency($amount, $this->from, $this->to);
return round($converted * $this->markup, 2);
}
}
Step 2. The supplier card parser
For HTML parsing we use the built-in DOMDocument + DOMXPath and cURL — no third-party libraries. The selectors are placeholders: you tune them per donor site.
<?php
class SupplierParser
{
public function __construct(
private string $baseUrl,
private CurrencyConverter $converter,
private float $delay = 1.0
) {}
private function getHtml(string $url): string
{
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_TIMEOUT => 20,
CURLOPT_USERAGENT => 'Mozilla/5.0 (compatible; CatalogImporter/1.0)',
]);
$html = curl_exec($ch);
if ($html === false) {
throw new RuntimeException('cURL: ' . curl_error($ch));
}
curl_close($ch);
return $html;
}
private function xpath(string $html): DOMXPath
{
$doc = new DOMDocument();
libxml_use_internal_errors(true);
$doc->loadHTML('<?xml encoding="UTF-8">' . $html);
libxml_clear_errors();
return new DOMXPath($doc);
}
private function absolute(string $href): string
{
return str_starts_with($href, 'http')
? $href
: rtrim($this->baseUrl, '/') . '/' . ltrim($href, '/');
}
/** Product links from paginated catalog pages. */
public function parseCatalog(string $path, int $maxPages = 5): array
{
$urls = [];
for ($page = 1; $page <= $maxPages; $page++) {
$xp = $this->xpath($this->getHtml($this->absolute("{$path}?page={$page}")));
$links = $xp->query("//div[contains(@class,'product-card')]//a[contains(@class,'product-link')]");
if ($links->length === 0) {
break;
}
foreach ($links as $a) {
$urls[] = $this->absolute($a->getAttribute('href'));
}
usleep((int)($this->delay * 1_000_000));
}
return $urls;
}
/** Parse a single product card. */
public function parseProduct(string $url): array
{
$xp = $this->xpath($this->getHtml($url));
$text = fn(string $q) => trim($xp->query($q)->item(0)?->textContent ?? '');
$name = $text("//h1[contains(@class,'product-title')]") ?: 'Untitled';
$sku = $text("//*[contains(@class,'sku')]");
$descr = $text("//*[contains(@class,'product-description')]");
$priceRaw = $text("//*[contains(@class,'price')]//*[contains(@class,'value')]");
$imgNode = $xp->query("//*[contains(@class,'product-gallery')]//img")->item(0);
$imageUrl = $imgNode ? $this->absolute($imgNode->getAttribute('src')) : '';
$priceSource = null;
if ($priceRaw !== '') {
$digits = preg_replace('/[^0-9.]/', '', str_replace(',', '.', $priceRaw));
$priceSource = $digits !== '' ? (float)$digits : null;
}
usleep((int)($this->delay * 1_000_000));
return [
'name' => $name,
'sku' => $sku,
'description' => $descr,
'image_url' => $imageUrl,
'price_source' => $priceSource,
'price' => $this->converter->convert($priceSource), // conversion
'source_url' => $url,
];
}
}
Step 3. Importing into the catalog via the Bitrix API
Here's the key difference from other CMSs: we write not to the database directly but through CIBlockElement (the infoblock element), CCatalogProduct (the commercial-catalog card) and CPrice (the price). The image is downloaded with CFile::MakeFileArray. We de-dupe by external code (XML_ID).
<?php
class BitrixImporter
{
public function __construct(
private int $iblockId, // catalog infoblock ID
private int $sectionId, // default section (category)
private string $priceCurrency = 'USD'
) {}
/** Look up a product by external code so we don't create duplicates. */
private function findByXmlId(string $xmlId): ?int
{
$res = \CIBlockElement::GetList(
[],
['IBLOCK_ID' => $this->iblockId, 'XML_ID' => $xmlId],
false, false, ['ID']
);
$row = $res->Fetch();
return $row ? (int)$row['ID'] : null;
}
public function import(array $p): int
{
$el = new \CIBlockElement();
$xmlId = $p['sku'] !== '' ? $p['sku'] : md5($p['source_url']);
$fields = [
'IBLOCK_ID' => $this->iblockId,
'IBLOCK_SECTION_ID' => $this->sectionId,
'NAME' => $p['name'],
'XML_ID' => $xmlId, // external code = donor SKU
'DETAIL_TEXT' => $p['description'],
'DETAIL_TEXT_TYPE' => 'html',
'ACTIVE' => 'Y',
];
// download and attach the image (if the donor serves one)
if ($p['image_url'] !== '') {
$file = \CFile::MakeFileArray($p['image_url']);
if ($file) {
$fields['DETAIL_PICTURE'] = $file;
}
}
$existingId = $this->findByXmlId($xmlId);
if ($existingId) {
// product already exists — update it (without overwriting the image)
unset($fields['DETAIL_PICTURE'], $fields['XML_ID']);
$el->Update($existingId, $fields);
$productId = $existingId;
} else {
$productId = $el->Add($fields);
if (!$productId) {
throw new RuntimeException('CIBlockElement: ' . $el->LAST_ERROR);
}
}
// commercial-catalog card (stock)
\CCatalogProduct::Add([
'ID' => $productId,
'QUANTITY' => 100,
'AVAILABLE' => 'Y',
]);
// base price (already converted and marked up)
if ($p['price'] !== null) {
\CPrice::SetBasePrice($productId, $p['price'], $this->priceCurrency);
}
return (int)$productId;
}
}
Step 4. Entry point and running on cron
<?php
// after the kernel bootstrap block (Step 0) and the classes above:
$converter = new CurrencyConverter('EUR', 'USD', 1.25);
$parser = new SupplierParser('https://supplier-example.com', $converter, 1.0);
$importer = new BitrixImporter(iblockId: 12, sectionId: 47, priceCurrency: 'USD');
$urls = $parser->parseCatalog('/catalog/category-1', maxPages: 3);
echo 'Products found: ' . count($urls) . PHP_EOL;
$imported = 0;
foreach ($urls as $url) {
try {
$product = $parser->parseProduct($url);
if ($product['price'] === null) {
echo "Skipped (no price): {$url}" . PHP_EOL;
continue;
}
$id = $importer->import($product);
$imported++;
printf("[%d] %s — %s EUR -> %s USD (ID %d)%s",
$imported, $product['name'], $product['price_source'], $product['price'], $id, PHP_EOL);
} catch (Throwable $e) {
echo "Error on {$url}: {$e->getMessage()}" . PHP_EOL;
}
}
echo "Done. Imported/updated: {$imported}" . PHP_EOL;
Auto-run once a day — one line in crontab:
# every day at 4:00, sync the catalog with the supplier
0 4 * * * /usr/bin/php /var/www/bitrix/local/parser/run.php >> /var/www/bitrix/local/parser/parser.log 2>&1
An alternative to cron is a Bitrix agent (
CAgent::AddAgent): it fires on site hits and is handy when you have no access to the system scheduler. But for heavy scraping, cron is more reliable — it keeps the load off your visitors.
Example 2: collect in Python → CSV → native import
If you'd rather not bootstrap the kernel and touch live code, you can decouple scraping from importing: collect the data with any convenient tool (Python, say), dump it to CSV, and let the native Commercial Catalog → Export/Import handle loading. That way the scraper doesn't depend on Bitrix or its updates at all.
Here we do currency conversion ourselves — pull a live FX rate and apply the markup (same as the PHP version, but on the collector's side). We use the free, key-less Frankfurter API, which serves European Central Bank reference rates.
import csv
import time
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
from datetime import date
class FxRate:
"""Live FX rate with a one-day cache (ECB reference rates)."""
URL = "https://api.frankfurter.dev/v1/latest"
def __init__(self, base="EUR", quote="USD"):
self.base = base
self.quote = quote
self._rate = None
self._date = None
def rate(self):
if self._rate is None or self._date != date.today():
r = requests.get(
self.URL, params={"base": self.base, "symbols": self.quote}, timeout=15
)
self._rate = r.json()["rates"][self.quote]
self._date = date.today()
return self._rate
def parse_and_export(base_url, catalog_path, out_csv,
base_ccy="EUR", quote_ccy="USD",
markup=1.25, max_pages=3, delay=1.0):
fx = FxRate(base_ccy, quote_ccy)
session = requests.Session()
session.headers["User-Agent"] = "Mozilla/5.0 (compatible; CatalogImporter/1.0)"
rows = []
for page in range(1, max_pages + 1):
soup = BeautifulSoup(
session.get(f"{base_url}{catalog_path}?page={page}", timeout=20).text, "lxml"
)
cards = soup.select(".product-card a.product-link")
if not cards:
break
for a in cards:
url = urljoin(base_url, a["href"])
ps = BeautifulSoup(session.get(url, timeout=20).text, "lxml")
name = ps.select_one("h1.product-title")
sku = ps.select_one(".sku")
descr = ps.select_one(".product-description")
price_el = ps.select_one(".price .value")
img = ps.select_one(".product-gallery img")
price_src = None
if price_el:
digits = "".join(c for c in price_el.text if c.isdigit() or c == ".")
price_src = float(digits) if digits else None
# currency conversion: FX rate * markup
price_usd = round(price_src * fx.rate() * markup, 2) if price_src else ""
rows.append({
"XML_ID": sku.text.strip() if sku else url,
"NAME": name.text.strip() if name else "Untitled",
"DETAIL_TEXT": descr.decode_contents().strip() if descr else "",
"PRICE": price_usd,
"CURRENCY": quote_ccy,
"PICTURE": urljoin(base_url, img["src"]) if img else "",
})
time.sleep(delay)
time.sleep(delay)
# CSV for the native Commercial Catalog import (delimiter ;)
with open(out_csv, "w", newline="", encoding="utf-8-sig") as f:
writer = csv.DictWriter(f, fieldnames=rows[0].keys(), delimiter=";")
writer.writeheader()
writer.writerows(rows)
print(f"Done. Rows written: {len(rows)} -> {out_csv}")
if __name__ == "__main__":
parse_and_export(
base_url="https://supplier-example.com",
catalog_path="/catalog/category-1",
out_csv="import.csv",
)
From there the CSV is loaded in the admin: Store → Settings → Module settings → Commercial Catalog → Export/Import → CSV import, where you map columns to product fields (external code, name, description, price, currency, image). The upside — the scraper doesn't depend on the Bitrix kernel; the downside — loading is a separate step, not on the fly.
What matters in a real project
The learning examples are deliberately simplified. A production Bitrix scraper also needs to handle:
- Trade offers (SKUs) — if a product has variations, create elements in the offers infoblock, link them to the product, and hang the price/stock on the offers rather than the product.
- Catalog properties — map the donor's characteristics to infoblock properties (
CIBlockElement::SetPropertyValues), auto-creating list values where needed. - Sections — auto-build the category tree from the donor's structure via
CIBlockSection. - Multiple price types and warehouses — for the Business/Enterprise editions: wholesale/retail-data-services prices, per-warehouse stock.
- Deduplication and syncing — update price/stock by
XML_IDrather than spawning duplicates; that's what turns a one-off scrape into a recurring sync. - Rate and markup — keep them in config and log which rate a batch was recalculated at, so prices are reproducible.
- Ethics and legality — respect
robots.txt, pause between requests, and honor the donor site's terms of use. If you're weighing what's permissible, see our note on the legality of web scraping.
To go deeper on any of the moving parts, see our guides on handling pagination, rotating proxies, and scraping dynamic JavaScript pages.
Wrapping up
For a typical store, you can often get by with "scraper builds a file + native CommerceML/CSV import" or a ready-made Marketplace module. But the moment you hit a non-standard donor, your own property and trade-offer schema, multiple price types, currency conversion at a live rate, or regular stock synchronization tuned to your Bitrix edition — a purpose-built scraper written through the Bitrix API is the reliable option.
If you need to fill or sync a store on 1C-Bitrix (or any other platform), pull data from supplier sites, and set up automatic price updates with currency conversion, our team can design and ship it as a done-for-you online store monitoring and data extraction service. We'll build the scraper around your edition and your requirements.